Google Cloud Dataflow vs. IBM Streams (discontinued)

It is best in cases where you have batch as well as streaming data. Also in some cases where you have batch data right now and in future you will get streaming data. In those cases Dataflow is very good. Also in cases where most of your infra is on GCP. It might not be good when you already are on AWS or Azure. And also you want in-depth control over security and management. Then you can directly use Apache beam over Dataflow.

Incentivized

Like the name says, it is good for streaming data and analyzing. It is great to look at tuples at a fast rate, filtering, calling other sources to enrich data, can call APIs, etc. Could do better for ingest use cases, can do better with guaranteed delivery, etc.

Incentivized

• Streaming, Real time work load
• Batch processing
• Auto scaling
• flexible pricing

• IBM Streams is well suited for providing wire-speed real-time end-to-end processing with sub-millisecond latency.
• Streams is amazingly computationally efficient. In other words, you can typically do much more processing with a given amount of hardware than other technologies. In a recent linear-road benchmark Streams based application was able to provide greater capability than the Hadoop-based implementation using 10x less hardware. So even when latency isn't critical, using Streams might still make sense for reducing operational cost.
• Streams comes out of the box with a large and comprehensive set of tested and optimized toolkits. Leveraging these toolkits not only reduces the development time and cost but also helps reduce project risk by eliminating the need for custom code which likely has not seen as much time in test or production.
• In addition to the out of the box toolkits, there is an active developer community contributing additional specialized packages.

Incentivized

• More templates for Bigquery and App Engine. There is only limited options for templates so the things we use can limit.
• I would like native connectors for Excel (XLSX) to reduce the need for custom wrappers in financial pipelines.
• Debugging Google Cloud Dataflow using only logs in Cloud Logging can be overwhelming sometimes, and it’s not always obvious which specific element in the flow caused a failure. IT uses a lot of time.

Incentivized

• Documentation could be more extensive, with more examples, although overall this is not too bad compared to some of the alternative solutions.
• Seems expensive to use in production.

Incentivized

It really saved a lot of time and it's flexibility really can give you infra which is future-proof for most of the use cases may it be streaming or batch data. And with this you can avoid use of resource-heavy big data offerings.

Incentivized

Google Cloud Dataproc Cloud Datafusion

There are well explained tutorials to get the user started. If you are looking for business application ideas, the user community offers a diversity of applications. It is very easy to launch applications on the cloud and can integrate with other analytic tools available on Watson Studio. It takes away the burden of the technology so that users can focus on business innovations.

Incentivized

• cost saving from managing our own data center for ETL servers
• consumption based pricing
• with auto scaling feature, we were able to expand components to support work load

• Ability to do more with less
• Admins and data analyst can now focus on more thinking tasks
• No negative impacts yet

Incentivized