Amazon Kinesis is a streaming analytics suite for data intake from video or other disparate sources and applying analytics for machine learning (ML) and business intelligence.
$0.01
per GB data ingested / consumed
RabbitMQ
Score 8.4 out of 10
N/A
RabbitMQ, an open source message broker, is part of Pivotal Software, a VMware company acquired in 2019, and supports message queue, multiple messaging protocols, and more.
RabbitMQ is available open source, however VMware also offers a range of commercial services for RabbitMQ; these are available as part of the Pivotal App Suite.
Amazon Kinesis is a great replacement for Kafka and it works better whenever the components of the solution are AWS based. Best if extended fan-out is not required, but still price-performance ratio is very good for simplifying maintenance.
I would go with a different option if the systems to be connected are legacy, for instance in the case of traditional messaging clients.
It is highly recommended that if you have microservices architecture and if you want to solve 2 phase commit issue, you should use RabbitMQ for communication between microservices. It is a quick and reliable mode of communication between microservices. It is also helpful if you want to implement a job and worker mechanism. You can push the jobs into RabbitMQ and that will be sent to the consumer. It is highly reliable so you won't miss any jobs and you can also implement a retry of jobs with the dead letter queue feature. It will be also helpful in time-consuming API. You can put time-consuming items into a queue so they will be processed later and your API will be quick.
What RabbitMQ does well is what it's advertised to do. It is good at providing lots of high volume, high availability queue. We've seen it handle upwards of 10 million messages in its queues, spread out over 200 queues before its publish/consume rates dipped. So yeah, it can definitely handle a lot of messages and a lot of queues. Depending on the size of the machine RabbitMQ is running on, I'm sure it can handle more.
Decent number of plugins! Want a plugin that gives you an interface to view all the queues and see their publish/consume rates? Yes, there's one for that. Want a plugin to "shovel" messages from one queue to another in an emergency? Check. Want a plugin that does extra logging for all the messages received? Got you covered!
Lots of configuration possibilities. We've tuned over 100 settings over the past year to get the performance and reliability just right. This could be a downside though--it's pretty confusing and some settings were hard to understand.
It breaks communication if we don't acknowledge early. In some cases our work items are time consuming that will take a time and in that scenario we are getting errors that RabbitMQ broke the channel. It will be good if RabbitMQ provides two acknowledgements, one is for that it has been received at client side and second ack is client is completed the processing part.
RabbitMQ is very usable if you are a programmer or DevOps engineer. You can setup and configure a messaging system without any programmatic knowledge either through an admin console plugin or through a command-line interface. It's very easy to spin up additional consumers when volume is heavy and it's very easy to manage those consumers either through automated scripting or through their admin console. Because it's language agnostic it integrates with any system supporting AMQP.
The documentation was confusing and lacked examples. The streams suddenly stopped working with no explanation and there was no information in the logs. All these were more difficult when dealing with enhanced fan-out. In fact, we were about to abort the usage of Kinesis due to a misunderstanding with enhanced fan-out.
RabbitMQ is more software than service so there's no real customer service to speak of unless you go with a provider such as CloudAMQP. So I'll just speak on CloudAMQP. Their customer support is only okay: they only do it over email. They frequently gloss over our support tickets and half answer them without delving deeply or investigating our issues. Their response times are pretty reasonable though.
The main benefit was around set up - incredibly easy to just start using Kinesis. Kinesis is a real-time data processing platform, while Kafka is more of a message queue system. If you only need a message queue from a limited source, Kafka may do the job. More complex use cases, with low latency, higher volume of data, real time decisions and integration with multiple sources and destination at a decent price, Kinesis is better.
RabbitMQ has a few advantages over Azure Service Bus 1) RMQ handles substantially larger files - ASB tops out at 100MB, we use RabbitMQfor files over 200MB 2) RabbitMQ can be easily setup on prem - Azure Service Bus is cloud only 3) RabbitMQ exchanges are easier to configure over ASB subscriptions ASB has a few advantages too 1) Cloud based - just a few mouse clicks and you're up and running
Earlier we had a problem with missing work items with our own implementation but later using RabbitMQ is solved a problem. Now our job processing mechanism is highly reliable.
We also had a problem with scaling, processing 1k work items per second. RabbitMQ helped us to scale well with increasing work items.
