Apache Kafka vs. IBM App Connect

Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.

IBM App Connect is well-suited to serve as a central integration hub, particularly for scenarios involving data transformation, complex routing logic, and dynamic backend routing. It excels at enabling legacy system modernization and supports real-time, event-driven architectures effectively. However, it is less appropriate for simple point-to-point integrations or for use cases requiring workflow process management and human task orchestration, where BPM or lightweight automation tools may be more suitable.

• Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
• Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
• Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).

Incentivized

• Just the ability to display and consume data through a single dashboard makes this a great application for our business purposes.
• With the ability to consume their exposed API, data validation and manipulation becomes a breeze.

Incentivized

• Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
• Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
• Learning curve around creation of broker and topics could be simplified

• Documentation - Documentation for older versions of the product was more detailed, with more examples.
• Sometimes not very intuitive, you need a certain learning curve.
• Some concepts are difficult to grasp or explain, due to "legacy" reasons (e.g. when to use DFDL vs MRM?)

Kafka is quickly becoming core product of the organization, indeed it is replacing older messaging systems. No better alternatives found yet

Incentivized

It is the best on-premise application to cloud integration in the market. I guess IBM is planning to integrate IBM App Connect with the IBM API Connect solution.

Incentivized

Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file

Incentivized

You can do some really powerful things with this system. The overall design is an attempt to make configurable some of the routine tasks/common functionality, but allow for development/customization of the core of the application.

Incentivized

Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.

Incentivized

Usually, the IBM Ops team provides a resolution or a response for 80% of defects raised in my project. There is one which has been open on their end for more than 3 months. With literally no response even after multiple follow-ups.

Incentivized

I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.

Incentivized

We did not select Cast Iron as our iPaaS solution, it was the weakest competitor in the field that we evaluated. Our experience was that it was not nearly as easy to learn, without in-depth training and guidance, and the developer UI was extremely buggy. We subjected each of the vendors to a battery of integrations, from simple to challenging, and it fell short on each one. One of the most simple integrations was grabbing a CSV file from an FTP source, parsing the data, doing a small amount of transformation, then inserting that data into an Azure MSSQL DB. After 2 hours on the phone with the Cast Iron support team, we were still unable to get this working.

Incentivized

• Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
• Positive: it's scalable so we can develop small and scale for real-world scenarios
• Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.

Incentivized

• I don't see any negative Impact .
• I like using this tool to do my job.
• I am comfortable using this tool.

Incentivized