Amazon Kinesis vs. StreamSets DataOps Platform

Amazon Kinesis is a great replacement for Kafka and it works better whenever the components of the solution are AWS based. Best if extended fan-out is not required, but still price-performance ratio is very good for simplifying maintenance.

I would go with a different option if the systems to be connected are legacy, for instance in the case of traditional messaging clients.

Incentivized

Majorly for all Batch and Streaming Scenarios we are designing StreamSets pipelines, few best suited and tried out use cases below : 1. JDBC to ADLS data transfer based on source refresh frequency. 2. Kafka to GCS. 3. Kafka to Azure Event. 4. Hub HDFS to ADLS data transfer. 5. Schema generation to generate Avro. The easy to design Canvas, Scheduling Jobs, Fragment creation and utilization, an inbuilt wide range of Stage availability makes it an even more favorable tool for me to design data engineering pipelines.

Incentivized

• Processing huge loads of data
• Integrating well with IoT Platform on Amazon
• Integration with overall AWS Ecosystem
• Scalability

Incentivized

• A easy to use canvas to create Data Engineering Pipeline.
• A wide range of available Stages ie. Sources, Processors, Executors, and Destinations.
• Supports both Batch and Streaming Pipelines.
• Scheduling is way easier than cron.
• Integration with Key-Vaults for Secrets Fetching.

Incentivized

• Not a queue system, so little visibility into "backlog" if there is any
• Confusing terminology to make sure events aren't missed
• Sometimes didn't seem to trigger Lambda functions, or dropped events when a lot came in

Incentivized

• Monitoring/Visualization can be improvised and enhanced a lot (e.g. to monitor a Job to see what happened 7 days back with data transfer).
• The logging mechanism can be simplified (Logs can be filtered with "ERROR", "DEBUG", "ALL" etc but still takes some time to get familiar for understanding).
• Auto Scalability for heavy load transfer (Taking much time for >5 million record transfer from JDBC to ADLS destination in Avro file transfer).
• There should be a concept of creating Global variables which is missing.

Incentivized

The documentation was confusing and lacked examples. The streams suddenly stopped working with no explanation and there was no information in the logs. All these were more difficult when dealing with enhanced fan-out. In fact, we were about to abort the usage of Kinesis due to a misunderstanding with enhanced fan-out.

Incentivized

The main benefit was around set up - incredibly easy to just start using Kinesis. Kinesis is a real-time data processing platform, while Kafka is more of a message queue system. If you only need a message queue from a limited source, Kafka may do the job. More complex use cases, with low latency, higher volume of data, real time decisions and integration with multiple sources and destination at a decent price, Kinesis is better.

Incentivized

StreamSets is a one-stop solution to design Data engineering Pipelines and doesn't require deep Programming knowledge, It's so user-friendly that anyone in Team can contribute to the Idea of pipeline design. In Hadoop One has to be programming proficient to use its various components like Hive, HDFS, Kafka, etc but in StreamSets all these stages are built-in and ready to use with minor configuration.

Incentivized

• Caused us to need to re-engineer some basic re-try logic
• Caused us to drop some content without knowing it
• Made monitoring much more difficult
• We eventually switched back to SQS because Kinesis is not the same as a Queue system

Incentivized

• Simplified Improvised Overall data ingestion and Integration Process.
• Support to various Hetrogenous Source systems like RDBMS< Kafka, Salesforce, Key Vault.
• Secure, easy to launch Integration tool.

Incentivized