Item: Google Cloud Dataflow
Rating: 8
Author: Sathish Palanivel(He/Him)

Use Cases and Deployment Scope

In our company we are using Google Cloud Dataflow to create data pipe lines for data transformation and ingestion use cases before loading data into database. Flexibility to create our own flex templates for any special case handling. Capability to fit streaming and batch data loads are some benefits. We have some real time loads, which Dataflow helps alot.

Pros and Cons

Streaming, Real time work load
Batch processing
Auto scaling
flexible pricing

inbuild template options can be expanded
more data connector options
easy of use

Return on Investment

cost saving from managing our own data center for ETL servers
consumption based pricing
with auto scaling feature, we were able to expand components to support work load

Alternatives Considered

Google Cloud Dataproc
Cloud Datafusion

Key Insights

Do you think Google Cloud Dataflow delivers good value for the price?

Yes

Are you happy with Google Cloud Dataflow's feature set?

Yes

Did Google Cloud Dataflow live up to sales and marketing promises?

Yes

Did implementation of Google Cloud Dataflow go as expected?

Yes

Would you buy Google Cloud Dataflow again?

Yes

Other Software Used

Google BigQuery, erwin Data Modeler, Microsoft Teams

Likelihood to Recommend

Based on my experience, streaming / real time / machine learning / AI type of processing and batch processing which needs less transformation are very well suited. Work load that needs complex transformation / multiple hops gets very complicated to implement. New feature like Dataflow SQL option will come in handy for sql heavy users.

Google Managed data processing service

Overall Satisfaction with Google Cloud Dataflow