Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
IBM MQ
Score 9.0 out of 10
N/A
IBM MQ (formerly WebSphere MQ and MQSeries) is messaging middleware.
Confluent Cloud is still based on Apache Kafka but it has a subscription fee so, from a long term perspective, it is wiser to deploy your own Kafka instance that spans public and private cloud. Amazon Kinesis, Google Cloud Pub/Sub do not do well for a very number of messages …
Kafka is faster and more scalable, also "free" as opensource (albeit we deploy using a commercial distribution). Infrastructure tends to be cheaper. On the other hand, projects must adapt to Kafka APIs that sometimes change and BAU increases until a major 1.x version comes out …
Apache Kafka may be a better option in comparison with IBM MQ its real-time data streaming and large data payload service. It depends upon the specific requirement and meets those needs. MuleSoft any point platform is very easy to connect to various other types of platforms in …
Kafka is renowned for its impressive throughput, fault tolerance, and real-time data streaming capabilities. Nonetheless, IBM MQ remains the preferred choice due to its unwavering commitment to guaranteed delivery and exceptional reliability. Fault-Tolerant Architectures of IBM …
We found IBM MQ very easy to get started and quick to learn by the new users with a short learning curve and seamlessly integrates with IBM products, and quick to perform self-service analytics and make informed business decisions. IBM MQ is also very straightforward in …
IBM MQ is very stable and a proven product compared to other Messaging platforms available. Performance was better than WSO2 product and also the RabbitMQ. Though Kafka and IBM MQ is not directly comparable, Kafka is more suited for event based systems and also where there is …
Apache Kafka and IBM MQ are both messaging queue tools built to help IT systems communicate with each other in an asynchronous manner. Apache Kafka is designed to enable the streaming of real time data feeds and is an open source tool that users can access for free. IBM MQ is a traditional message queue system that allows multiple subscribers to pull messages from the end of the queue. Both tools are most popular with mid-sized businesses and large enterprises that are more likely to have complex IT infrastructures that need to communicate with each other.
Features
Both Apache Kafka and IBM MQ allow systems to send messages to each other asynchronously, but they also have a few standout features that set them apart from each other.
Apache Kafka utilized pull based communication, meaning that the receiving system requests a message from the producing system. This method of communication makes Apache Kafka faster than most traditional message queue systems. As businesses add more nodes to Apache Kafka, they will also find that it scales well, and notice few performance dips. Lastly, messages in Apache Kafka are not deleted upon the receiving system reading them, so it is easier to log events using Apache Kafka compared to other options.
IBM MQ is a more traditional message queue system that uses push based communication, in which a message producing system pushes its message into the queue and any receiver can consume it. This type of communication allows multiple systems to pull the same message from the queue at once. IBM MQ also includes several advanced features for security and message simplification.
Limitations
Though Apache Kafka and IBM MQ can both act as effective message borokers, they also have some limitations that are important to consider.
Apache Kafka can get a message from one system to it’s receiver quickly compared to traditional message queue tools, but each receiver must make a request for the message, rather than the producing system placing the message into an accessible queue. Additionally, while Apache Kafka can be used to log events and scales well, it doesn’t include as many granular features for security and message simplification. Apache Kafka is ideal for teams that value speed and performance highly.
IBM MQ is a robust traditional message queue system, but it doesn’t match the speed of Apache Kafka. Users should expect messages to take longer to complete in IBM MQ and will have a harder time using it to log events. As a result, IBM MQ is an ideal choice for businesses with complex IT infrastructures that often send messages from one system to many other systems, and who can benefit from granular customization features.
Pricing
Apache Kafka is an open source tool, so its pricing depends on the hosting service. Businesses should expect to pay at least $0.42 per hour, with that rate increasing as messaging needs increase.
IBM MQ is provided through IBM’s cloud service, pricing is dependent on the amount of client connections needed and users can reach out to the vendor for a detailed quote.
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
In the context of Internet of Things (IoT) applications, IBM MQ plays a pivotal role in managing the substantial data streams emanating from interconnected devices. Its primary function is to guarantee the dependable transmission and processing of data, catering to a diverse range of IoT use cases, including but not limited to smart city initiatives, healthcare monitoring systems, and industrial automation solutions. In the telecommunications sector, IBM MQ is employed for message routing, call detail record (CDR) processing, and network management to ensure real-time data exchange and fault tolerance. When managing the supply chain and logistics, IBM MQ is used to ensure timely and accurate communication between different entities, including suppliers, warehouses, and transportation providers. IBM MQ can be cost-prohibitive for smaller organizations due to licensing and maintenance costs. In such cases, open-source or lightweight messaging solutions may be more appropriate. For scenarios requiring extremely low-latency, real-time data exchange, and high throughput, other messaging technologies, like Apache Kafka, may be more suitable due to their specialized design for such use cases.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
The documentation is very clear,It is understandable and the support helps to configure it in the best way.
Server guidelines make it possible to get the most out of work management. It's broad, we can work with different operating systems, I really recommend using linux.
It is highly compatible with systems, brockers, applications, and data accumulation programs, it is possible to configure everything so that after the installation of programs, they can communicate with each other and then throw data to an external program that accumulates it and represents in clear details of steps to follow and make business decisions.
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
There is limitation on number of svrconn connections you can have to MQ on the mainframe which has been an major issue for us. This has been an issue for us for over 4 years and still no fix although I am aware IBM have been working on a solution over the last year.
When upgrading to MQ V9.3 on our MQ appliances there is no fall-back option. This was the same for MQ V9.2 upgrade from MQ V9.0. For production upgrades this I believe is not acceptable.
AMS is not supplied as part of the standard mainframe MQ licence. You need an extra licence. IBM tell customers how important security and protecting data is yet they still want to charge for this software. The cost of MQ on the mainframe is not cheap so I would expect AMS to be part of the base product.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
The messages are delivered instantly with this software and it integrates with our technology stack, in terms of availability we only had one failure when we were doing some testing and integration with third parties, the features of this software make it always available and its deployment is easy for the company, it does not generate expenses due to failures
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
There are very specific things that must be elevated to more specialized areas of support, but the common support is very agile when receiving questions or when we leave concerns in real time. I recommend the support of the program in this regard.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
We found IBM MQ very easy to get started and quick to learn by the new users with a short learning curve and seamlessly integrates with IBM products, and quick to perform self-service analytics and make informed business decisions. IBM MQ is also very straightforward in creating simple and best reports, which are very profitable and productive.
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
