Amazon Web Services (AWS) Provides the Amazon Simple Queue Service (SQS), a managed message queue service which supports the safe decoupling and distribution of different components in a cloud infrastructure and cloud applications.
$0
per GB
Apache Kafka
Score 8.4 out of 10
N/A
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
Pricing
Amazon Simple Queue Service (SQS)
Apache Kafka
Editions & Modules
All Data Transfer In
$0.00
per GB
Standard Queue
$0.00000004
per request
FIFO Queue
$0.00000005
per request
No answers on this topic
Offerings
Pricing Offerings
Amazon SQS
Apache Kafka
Free Trial
No
No
Free/Freemium Version
Yes
No
Premium Consulting/Integration Services
No
No
Entry-level Setup Fee
No setup fee
No setup fee
Additional Details
—
—
More Pricing Information
Community Pulse
Amazon Simple Queue Service (SQS)
Apache Kafka
Considered Both Products
Amazon SQS
Verified User
Engineer
Chose Amazon Simple Queue Service (SQS)
To be blunt: Amazon SQS was the simplest to implement given our requirements. Other services in this space work just as well, and SQS does not have any benefits outside of being the easiest to implement when using an otherwise fully AWS stack. AWS itself even has other …
Amazon SQS stacks up with the best of them as most of their products do. The only issue comparatively that I’ve had with this service, in particular, is the silently failing messages and then allocation of time to dedicate to debugging when the issue of why a message got stuck …
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond …
If you are looking to build something that just requires a simple queue service (as the name implies) this is great for it. You might look elsewhere though if you get into more complicated needs. This is also very well suited if you are already using other services with AWS and intend to fully build whatever you are building in AWS. If you are looking for a mixed environment -- SQS is not for you
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
Online blogging and documentation for SQS is great. There are many examples of implementing it and if you look hard enough, more than likely there are examples that meet the exact case with which you are working
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
The most comparable products are RabbitMQ, and perhaps ActiveMQ. Until recently, AWS did not offer a managed ActiveMQ product. Running RabbitMQ will never be to my team's competitive advantage; we wanted a managed service.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
