It takes me less time to write a Pig script than get a Spark program running for batch ETL workloads. Compared to Spark, Pig has a steeper learning curve because it employs a proprietary programming language. In one script and one fine, it can handle both Map Reduce and Hadoop. …
Apache Pig is best suited for ETL-based data processes. It is good in performance in handling and analyzing a large amount of data. it gives faster results than any other similar tool. It is easy to implement and any user with some initial training or some prior SQL knowledge can work on it. Apache Pig is proud to have a large community base globally.
Google BigQuery really shines in scenarios requiring real-time analytics on large data streams and predictive analytics with its machine learning integration. Teams have been using it extensively all over. However, it may not be the best fit for organizations dealing with small datasets because of the higher costs. And also, it might not be the best fit for highly complex data transformations, where simpler or more specialized solutions could be more appropriate.
Its serverless architecture and underlying Dremel technology are incredibly fast even on complex datasets. I can get answers to my questions almost instantly, without waiting hours for traditional data warehouses to churn through the data.
Previously, our data was scattered across various databases and spreadsheets and getting a holistic view was pretty difficult. Google BigQuery acts as a central repository and consolidates everything in one place to join data sets and find hidden patterns.
Running reports on our old systems used to take forever. Google BigQuery's crazy fast query speed lets us get insights from massive datasets in seconds.
It is challenging to predict costs due to BigQuery's pay-per-query pricing model. User-friendly cost estimation tools, along with improved budget alerting features, could help users better manage and predict expenses.
The BigQuery interface is less intuitive. A more user-friendly interface, enhanced documentation, and built-in tutorial systems could make BigQuery more accessible to a broader audience.
We have to use this product as its a 3rd party supplier choice to utilise this product for their data side backend so will not be likely we will move away from this product in the future unless the 3rd party supplier decides to change data vendors.
web UI is easy and convenient. Many RDBMS clients such as aqua data studio, Dbeaver data grid, and others connect. Range of well-documented APIs available. The range of features keeps expanding, increasing similar features to traditional RDBMS such as Oracle and DB2
BigQuery can be difficult to support because it is so solid as a product. Many of the issues you will see are related to your own data sets, however you may see issues importing data and managing jobs. If this occurs, it can be a challenge to get to speak to the correct person who can help you.
Apache Pig might help to start things faster at first and it was one of the best tool years back but it lacks important features that are needed in the data engineering world right now. Pig also has a steeper learning curve since it uses a proprietary language compared to Spark which can be coded with Python, Java.
I have used Snowflake and DataGrip for data retrieval as well as Google BigQuery and can say that all these tools compete for head to head. It is very difficult to say which is better than the other but some features provided by Google BigQuery give it an edge over the others. For example, the reliability of Google is unmatchable by others. One thing that I really like is the ability to integrate Data Studio so easily with Google BigQuery.
Google Support has kindly provide individual support and consultants to assist with the integration work. In the circumstance where the consultants are not present to support with the work, Google Support Helpline will always be available to answer to the queries without having to wait for more than 3 days.
Higher learning curve than other similar technologies so on-boarding new engineers or change ownership of Apache Pig code tends to be a bit of a headache
Once the language is learned and understood it can be relatively straightforward to write simple Pig scripts so development can go relatively quickly with a skilled team
As distributed technologies grow and improve, overall Apache Pig feels left in the dust and is more legacy code to support than something to actively develop with.
Pricing has been very reasonable for us. The first 10 GB of storage is free each month and costs start at 2 cents per GB per month after that. For example, if you store 1 terabyte (TB) for a month, then the cost would be $20. Streaming data inserts start at 1 cent per 200 megabytes (MBs). The first 1 TB of queries is free, with additional analysis at $5 per TB thereafter. Meta data operations are free.
Big Query helps reduce the bar for data analytics, ML and AI. BQ takes care of mundane tasks and streamlines for easy data processing, consumption. The most impressive thing is the ML and AI integration as SQL functions, so the need for moving data around is minimized.
The visuals of ML models is very helpful to fine tune training, model building and prediction, etc.
