Apache Kafka vs. IBM InfoSphere Information Server vs. RabbitMQ

For brokering messages, Confluent Kafka is well suited since it offers a managed solution ready to use. Scenarios where the solution is not very well suited are for example, where pricing is an issue. The solution costs quite a lot for basic usage (for example: for 3 clusters, pricing is above 100k$ a year).

Incentivized

You can use infosphere: -If you have multiple targets and source systems and they are different than each other. -If your infostructure is so big and unplaned well so you can't find what you want to see. -If your databases not so strong to process your data

Incentivized

If you are looking for a message broker, RabbitMQ is pretty good. Its API lets you create tons of queues on demand and publish to all of them at once, while you can have 10+ consumers on each queue. It also does a good job of absorbing bursts of traffic. We've seen our queues get backed up to 3 million messages with no problem. In the modern era of GDPR, you may run into problems with keeping messages encrypted out of the box in-flight and at-rest with RabbitMQ. Not saying it's impossible, but it's tough to set up and you have to pay a high overload.

Incentivized

• Apache Kafka is able to handle a large number of I/Os (writes) using 3-4 cheap servers.
• It scales very well over large workloads and can handle extreme-scale deployments (eg. Linkedin with 300 billion user events each day).
• The same Kafka setup can be used as a messaging bus, storage system or a log aggregator making it easy to maintain as one system feeding multiple applications.

Incentivized

• It is very strong to make transformations/data derivations
• It is very easy to connect to various external data sources. It has an interface (stages) for each connection that simplifies the task
• It is a stable platform. And that parallelism helps make it fast for loading, if the process is well designed

• What RabbitMQ does well is what it's advertised to do. It is good at providing lots of high volume, high availability queue. We've seen it handle upwards of 10 million messages in its queues, spread out over 200 queues before its publish/consume rates dipped. So yeah, it can definitely handle a lot of messages and a lot of queues. Depending on the size of the machine RabbitMQ is running on, I'm sure it can handle more.
• Decent number of plugins! Want a plugin that gives you an interface to view all the queues and see their publish/consume rates? Yes, there's one for that. Want a plugin to "shovel" messages from one queue to another in an emergency? Check. Want a plugin that does extra logging for all the messages received? Got you covered!
• Lots of configuration possibilities. We've tuned over 100 settings over the past year to get the performance and reliability just right. This could be a downside though--it's pretty confusing and some settings were hard to understand.

Incentivized

• The Kafka Tool is a community-made Java application that looks and feels from the past century.
• Logging can be confusing. This certainly shows when we have to do troubleshooting.
• Hybrid scenarios - pub/sub, but there are services in and outside a Kubernetes cluster. Then there are a ~3 options, but only 2 (the harder ones) are production-safe.

Incentivized

• Lack of a strong web development environment.
• Metadata propagation in Jobs is somewhat complex.
• The possibility to develop jobs in Parallel and/or Server Engines is confusing.

• It breaks communication if we don't acknowledge early. In some cases our work items are time consuming that will take a time and in that scenario we are getting errors that RabbitMQ broke the channel. It will be good if RabbitMQ provides two acknowledgements, one is for that it has been received at client side and second ack is client is completed the processing part.

Incentivized

Kafka has suited our use case very well so far. Going forward we are planning to expand our platform manifold so the load on Kafka and our reliance on Kafka is going to increase only.

• Scale of implementation
• IBM techsupport

Incentivized

Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file

Incentivized

RabbitMQ is very easy to configure for all supported languages (Python, Java, etc.). I have personally used it on Raspberry Pi devices via a Flask Python API as well as in Java applications. I was able to learn it quickly and now have full mastery of it. I highly recommend it for any IoT project.

Incentivized

Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.

Incentivized

I gave it a 10 but we do not have a support contract with any company for RabbitMQ so there is no official support in that regard. However, there is a community and questions asked on StackOverflow or any other major question and answer site will usually get a response.

Incentivized

Apache Kafka is built for scale. From high throughput and real-time data streaming, it has a strong advantage over RabbitMQ with its low latency. This put Apache Kafka at the forefront as the platform of choice for large datasets messaging and ensuring scalability when data scale up tremendously. RabbitMQ however has its strengths in traditional messaging. Routing and message delivery reliability are the bedrock of RabbitMQ and this is where RabbitMQ excels. In my previous workplace, RabbitMQ was of choice as reliability matters more than scale. In two words. Apache Kafka for scale, RabbitMQ for reliability. And for cloud deployment and large dataset messaging in what I am doing now, Apache Kafka is the default choice.

I particularly believe that Information Server, especially DataStage, is superior in many aspects to the Oracle Data Integrator tool. Several market analysts such as Gartner and / or Forrester better position DataStage on the Oracle solution.

It is very easy to use as it has a simple function to connect and use RabbitMQ. It is having Fast Learning curve, Any newbies can learn it in a week or month. It is having proper documentation, we are able to find all the details about its functionality and usage of it. The Features of RabbitMQ are providing are matching with our business requirements.

Incentivized

• Positive: bursts of traffic on special holidays are easy to handle because Kafka can absorb and buffer all the messages we need to process long enough to let an understaffed set of back-end services catch up on processing. Hard to put a number to it but we probably save $5k a month having fewer machines running.
• Positive: makes decoupling the web and API services from the deeper back-end services easier by providing topics as an interface. This allowed us to split up our teams and have them develop independently of each other, speeding up software development.
• Negative: our engineers have made mistakes such as accidentally dropping a few thousand messages due to the CLI being confusing to use, and as a result a customer lost some of their precious data. I'd say that was more our fault than Kafka's though.

Incentivized

• Information Server can positively impact the costs of companies by increasing the productivity of development and therefore reduce their time and costs. It is estimated that DataStage can increase a developer's productivity by 40% on average.
• Better data governance
• Improve data quality and reduce bad data impacts

Incentivized

• Earlier we had a problem with missing work items with our own implementation but later using RabbitMQ is solved a problem. Now our job processing mechanism is highly reliable.
• We also had a problem with scaling, processing 1k work items per second. RabbitMQ helped us to scale well with increasing work items.

Incentivized