Amazon Kinesis vs. Google BigQuery

Amazon Kinesis is a great replacement for Kafka and it works better whenever the components of the solution are AWS based. Best if extended fan-out is not required, but still price-performance ratio is very good for simplifying maintenance.

I would go with a different option if the systems to be connected are legacy, for instance in the case of traditional messaging clients.

Incentivized

Event-based data can be captured seamlessly from our data layers (and exported to Google BigQuery). When events like page-views, clicks, add-to-cart are tracked, Google BigQuery can help efficiently with running queries to observe patterns in user behaviour. That intermediate step of trying to "untangle" event data is resolved by Google BigQuery. A scenario where it could possibly be less appropriate is when analysing "granular" details (like small changes to a database happening very frequently).

Incentivized

• Processing huge loads of data
• Integrating well with IoT Platform on Amazon
• Integration with overall AWS Ecosystem
• Scalability

Incentivized

• Realtime integration with Google Sheets.
• GSheet data can be linked to a BigQuery table and the data in that sheet is ingested in realtime into BigQuery. It's a live 'sync' which means it supports insertions, deletions, and alterations. The only limitation here is the schema'; this remains static once the table is created.
• Seamless integration with other GCP products.
• A simple pipeline might look like this:-
• GForms -> GSheets -> BigQuery -> Looker
• It all links up really well and with ease.
• One instance holds many projects.
• Separating data into datamarts or datameshes is really easy in BigQuery, since one BigQuery instance can hold multiple projects; which are isolated collections of datasets.

Incentivized

• Not a queue system, so little visibility into "backlog" if there is any
• Confusing terminology to make sure events aren't missed
• Sometimes didn't seem to trigger Lambda functions, or dropped events when a lot came in

Incentivized

• Please expand the availability of documentation, tutorials, and community forums to provide developers with comprehensive support and guidance on using Google BigQuery effectively for their projects.
• If possible, simplify the pricing model and provide clearer cost breakdowns to help users understand and plan for expenses when using Google BigQuery. Also, some cost reduction is welcome.
• It still misses the process of importing data into Google BigQuery. Probably, by improving compatibility with different data formats and sources and reducing the complexity of data ingestion workflows, it can be made to work.

Incentivized

We have to use this product as its a 3rd party supplier choice to utilise this product for their data side backend so will not be likely we will move away from this product in the future unless the 3rd party supplier decides to change data vendors.

Incentivized

I think overall it is easy to use. I haven't done anything from the development side but an more of an end user of reporting tables built in Google BigQuery. I connect data visualization tools like Tableau or Power BI to the BigQuery reporting tables to analyze trends and create complex dashboards.

Incentivized

I have never had any significant issues with Google Big Query. It always seems to be up and running properly when I need it. I cannot recall any times where I received any kind of application errors or unplanned outages. If there were any they were resolved quickly by my IT team so I didn't notice them.

Incentivized

I think Google Big Query's performance is in the acceptable range. Sometimes larger datasets are somewhat sluggish to load but for most of our applications it performs at a reasonable speed. We do have some reports that include a lot of complex calculations and others that run on granular store level data that so sometimes take a bit longer to load which can be frustrating.

Incentivized

The documentation was confusing and lacked examples. The streams suddenly stopped working with no explanation and there was no information in the logs. All these were more difficult when dealing with enhanced fan-out. In fact, we were about to abort the usage of Kinesis due to a misunderstanding with enhanced fan-out.

Incentivized

BigQuery can be difficult to support because it is so solid as a product. Many of the issues you will see are related to your own data sets, however you may see issues importing data and managing jobs. If this occurs, it can be a challenge to get to speak to the correct person who can help you.

Incentivized

The main benefit was around set up - incredibly easy to just start using Kinesis. Kinesis is a real-time data processing platform, while Kafka is more of a message queue system. If you only need a message queue from a limited source, Kafka may do the job. More complex use cases, with low latency, higher volume of data, real time decisions and integration with multiple sources and destination at a decent price, Kinesis is better.

Incentivized

PowerBI can connect to GA4 for example but the data processing is more complicated and it takes longer to create dashboards. Azure is great once the data import has been configured but it's not an easy task for small businesses as it is with BigQuery.

Incentivized

None so far. Very satisfied with the transparency on contract terms and pricing model.

We have continued to expand out use of Google Big Query over the years. I'd say its flexibility and scalability is actually quite good. It also integrates well with other tools like Tableau and Power BI. It has served the needs of multiple data sources across multiple departments within my company.

Incentivized

Google Support has kindly provide individual support and consultants to assist with the integration work. In the circumstance where the consultants are not present to support with the work, Google Support Helpline will always be available to answer to the queries without having to wait for more than 3 days.

• Caused us to need to re-engineer some basic re-try logic
• Caused us to drop some content without knowing it
• Made monitoring much more difficult
• We eventually switched back to SQS because Kinesis is not the same as a Queue system

Incentivized

• Previously, running complex queries on our on-premise data warehouse could take hours. Google BigQuery processes the same queries in minutes. We estimate it saves our team at least 25% of their time.
• We can target our marketing campaigns very easily and understand our customer behaviour. It lets us personalize marketing campaigns and product recommendations and experience at least a 20% improvement in overall campaign performance.
• Now, we only pay for the resources we use. Saved $1 million annually on data infrastructure and data storage costs compared to our previous solution.

Incentivized