Amazon Kinesis vs. Amazon Simple Queue Service (SQS)

Amazon Kinesis is a great replacement for Kafka and it works better whenever the components of the solution are AWS based. Best if extended fan-out is not required, but still price-performance ratio is very good for simplifying maintenance.

I would go with a different option if the systems to be connected are legacy, for instance in the case of traditional messaging clients.

Incentivized

If you are looking to build something that just requires a simple queue service (as the name implies) this is great for it. You might look elsewhere though if you get into more complicated needs. This is also very well suited if you are already using other services with AWS and intend to fully build whatever you are building in AWS. If you are looking for a mixed environment -- SQS is not for you

Incentivized

• Processing huge loads of data
• Integrating well with IoT Platform on Amazon
• Integration with overall AWS Ecosystem
• Scalability

Incentivized

• It provides an always-available serverless queue for workflows or mission-critical processes.
• Is extremely low cost and overall costs to our environments have been negligible.

Incentivized

• Not a queue system, so little visibility into "backlog" if there is any
• Confusing terminology to make sure events aren't missed
• Sometimes didn't seem to trigger Lambda functions, or dropped events when a lot came in

Incentivized

• Almost all of the functionality has been covered by SQS, but they could improve the throughput time.
• Also, they could provide built-in Cloud Watch, so that we can easily configure it without any external efforts.

Incentivized

The documentation was confusing and lacked examples. The streams suddenly stopped working with no explanation and there was no information in the logs. All these were more difficult when dealing with enhanced fan-out. In fact, we were about to abort the usage of Kinesis due to a misunderstanding with enhanced fan-out.

Incentivized

Online blogging and documentation for SQS is great. There are many examples of implementing it and if you look hard enough, more than likely there are examples that meet the exact case with which you are working

Incentivized

The main benefit was around set up - incredibly easy to just start using Kinesis. Kinesis is a real-time data processing platform, while Kafka is more of a message queue system. If you only need a message queue from a limited source, Kafka may do the job. More complex use cases, with low latency, higher volume of data, real time decisions and integration with multiple sources and destination at a decent price, Kinesis is better.

Incentivized

The most comparable products are RabbitMQ, and perhaps ActiveMQ. Until recently, AWS did not offer a managed ActiveMQ product. Running RabbitMQ will never be to my team's competitive advantage; we wanted a managed service.

Incentivized

• Caused us to need to re-engineer some basic re-try logic
• Caused us to drop some content without knowing it
• Made monitoring much more difficult
• We eventually switched back to SQS because Kinesis is not the same as a Queue system

Incentivized

• Positive impact - time allocation towards different features
• Negative impact - too many resources dedicated towards debugging
• Positive impact - less manual labor during testing

Incentivized