IBM Event Streams vs. StreamSets DataOps Platform

We used it in two cases, and it went pretty good with both. We used it for analytics and logging alongside event-based communication between different tech-stacks. One thing that users need to keep in mind is that it has a limited knowledgebase and will need a good understanding of Kafka to use it efficiently. So choose it carefully and when its needed

Incentivized

Majorly for all Batch and Streaming Scenarios we are designing StreamSets pipelines, few best suited and tried out use cases below : 1. JDBC to ADLS data transfer based on source refresh frequency. 2. Kafka to GCS. 3. Kafka to Azure Event. 4. Hub HDFS to ADLS data transfer. 5. Schema generation to generate Avro. The easy to design Canvas, Scheduling Jobs, Fragment creation and utilization, an inbuilt wide range of Stage availability makes it an even more favorable tool for me to design data engineering pipelines.

Incentivized

• It is adaptive and helps us create more engaging experiences on our platforms.
• The Key metrics dashboard is rich with insights.

Incentivized

• A easy to use canvas to create Data Engineering Pipeline.
• A wide range of available Stages ie. Sources, Processors, Executors, and Destinations.
• Supports both Batch and Streaming Pipelines.
• Scheduling is way easier than cron.
• Integration with Key-Vaults for Secrets Fetching.

Incentivized

• Provide Capabilities to connect the Event Streams via REST Proxy.
• Schema Registry to handle Avro Formats.
• Provide Kafka Connect Sink & Source Connectors.

Incentivized

• Monitoring/Visualization can be improvised and enhanced a lot (e.g. to monitor a Job to see what happened 7 days back with data transfer).
• The logging mechanism can be simplified (Logs can be filtered with "ERROR", "DEBUG", "ALL" etc but still takes some time to get familiar for understanding).
• Auto Scalability for heavy load transfer (Taking much time for >5 million record transfer from JDBC to ADLS destination in Avro file transfer).
• There should be a concept of creating Global variables which is missing.

Incentivized

The product was very user friendly and extremely easy to get started with. The documentation is excellent and the free tier makes it very easy to get started with without having to make deep or long term financial commitments.

Incentivized

I met with the support team and they have deep technical and development understanding of the needs and the problems which IBM Event Streams addresses. If you are looking for a product backed by a highly technical support team then IBM Event Streams is probably the best choice. I was specifically impressed by the level of technical understanding my support team demonstrated.

Incentivized

In Event Streams, applications send data by creating a message and sending it to a topic. To receive messages, applications subscribe to a topic. High availability and reliability. Event Streams offers a highly available and reliable Apache Kafka service running on IBM Cloud. Event Streams. Event Streams stores three replicas of your data to ensure the highest level of resilience across three availability zones.

Incentivized

StreamSets is a one-stop solution to design Data engineering Pipelines and doesn't require deep Programming knowledge, It's so user-friendly that anyone in Team can contribute to the Idea of pipeline design. In Hadoop One has to be programming proficient to use its various components like Hive, HDFS, Kafka, etc but in StreamSets all these stages are built-in and ready to use with minor configuration.

Incentivized

• In using downstreams, the minimal features and the rate of releases were slow, makes us feel that there's no upgrades and other than that there's poor marketing of the product.
• The adoption around the service is low, requires focused marketing.
• Lack of visibility into topic depth , Monitoring capabilities

Incentivized

• Simplified Improvised Overall data ingestion and Integration Process.
• Support to various Hetrogenous Source systems like RDBMS< Kafka, Salesforce, Key Vault.
• Secure, easy to launch Integration tool.

Incentivized