Amazon Kinesis vs. Google Cloud Dataflow vs. Spotfire Streaming

Amazon Kinesis is a great replacement for Kafka and it works better whenever the components of the solution are AWS based. Best if extended fan-out is not required, but still price-performance ratio is very good for simplifying maintenance.

I would go with a different option if the systems to be connected are legacy, for instance in the case of traditional messaging clients.

Incentivized

It is best in cases where you have batch as well as streaming data. Also in some cases where you have batch data right now and in future you will get streaming data. In those cases Dataflow is very good. Also in cases where most of your infra is on GCP. It might not be good when you already are on AWS or Azure. And also you want in-depth control over security and management. Then you can directly use Apache beam over Dataflow.

Incentivized

Taking data from various sources including files, databases, web services, applying some complex rules, transforming, aggregating and producing a result. This is what Spotfire Streaming does best.
- If one needs connectivity to special services as secured databases or web services, building interactive web apps, those are probably tasks that shall be addressed with different tools.

Incentivized

• Processing huge loads of data
• Integrating well with IoT Platform on Amazon
• Integration with overall AWS Ecosystem
• Scalability

Incentivized

• Streaming, Real time work load
• Batch processing
• Auto scaling
• flexible pricing

• Processing events in real-time with real low latency and high throughput.
• 100% visual program language, which can be extended by common languages like Java, Python and .NET.
• Reduced time to prototype, create an application and deployment, which reduces the software lifecycle.
• Real robust engine and server. Barely heard of customers having issues in production.

• Not a queue system, so little visibility into "backlog" if there is any
• Confusing terminology to make sure events aren't missed
• Sometimes didn't seem to trigger Lambda functions, or dropped events when a lot came in

Incentivized

• More templates for Bigquery and App Engine. There is only limited options for templates so the things we use can limit.
• I would like native connectors for Excel (XLSX) to reduce the need for custom wrappers in financial pipelines.
• Debugging Google Cloud Dataflow using only logs in Cloud Logging can be overwhelming sometimes, and it’s not always obvious which specific element in the flow caused a failure. IT uses a lot of time.

Incentivized

• Being a niche tool, there's not much community support
• It is on prem, making it slow to boot. not cloud native (It could be that it is our org's issue)
• The dev environment is very tough to understand for large projects due to the wire style UI/UX

Incentivized

It really saved a lot of time and it's flexibility really can give you infra which is future-proof for most of the use cases may it be streaming or batch data. And with this you can avoid use of resource-heavy big data offerings.

Incentivized

The usability is good in terms that it gets well integrated with the Spotfire suite but the only few issues I have is the tough UI/UX (learning curve, if the project is huge) and unable to find many users and devs to help with the queries. At the end it is solely based on the documentation provided which is never enough

Incentivized

The documentation was confusing and lacked examples. The streams suddenly stopped working with no explanation and there was no information in the logs. All these were more difficult when dealing with enhanced fan-out. In fact, we were about to abort the usage of Kinesis due to a misunderstanding with enhanced fan-out.

Incentivized

Spotfire Streaming support is prompt and to the point. They help with best practices and learning from existing projects.

Incentivized

The main benefit was around set up - incredibly easy to just start using Kinesis. Kinesis is a real-time data processing platform, while Kafka is more of a message queue system. If you only need a message queue from a limited source, Kafka may do the job. More complex use cases, with low latency, higher volume of data, real time decisions and integration with multiple sources and destination at a decent price, Kinesis is better.

Incentivized

Google Cloud Dataproc Cloud Datafusion

We are using Dataflow (by Google).The development time in Spotfire Streaming is definitely shorter because its GUI based. Dataflow handles late arrivals after the window closes, not sure Spotfire Streaming can do that. Dataflow can run GCP as a managed service which is why we chose that tool for our new product.

Incentivized

• Caused us to need to re-engineer some basic re-try logic
• Caused us to drop some content without knowing it
• Made monitoring much more difficult
• We eventually switched back to SQS because Kinesis is not the same as a Queue system

Incentivized

• cost saving from managing our own data center for ETL servers
• consumption based pricing
• with auto scaling feature, we were able to expand components to support work load

• While we haven't specifically integrated Spotfire Streaming into our product development, it has allowed us to see the benefits of real-time streaming data.
• We have much more visibility into how our longer term roadmap will look and what we should focus on.

Incentivized