Apache Kafka vs. Redis Software

For brokering messages, Confluent Kafka is well suited since it offers a managed solution ready to use. Scenarios where the solution is not very well suited are for example, where pricing is an issue. The solution costs quite a lot for basic usage (for example: for 3 clusters, pricing is above 100k$ a year).

Incentivized

Perfect solution for caching needs. If you have a bottleneck due to frequent data access to your database, then Redis can really help you by diverting those traffic away from your database. Its key/value pair structure also makes data lookup very efficient, providing excellent performance.

Incentivized

• Apache Kafka is able to handle a large number of I/Os (writes) using 3-4 cheap servers.
• It scales very well over large workloads and can handle extreme-scale deployments (eg. Linkedin with 300 billion user events each day).
• The same Kafka setup can be used as a messaging bus, storage system or a log aggregator making it easy to maintain as one system feeding multiple applications.

Incentivized

• Easy for developers to understand. Unlike Riak, which I've used in the past, it's fast without having to worry about eventual consistency.
• Reliable. With a proper multi-node configuration, it can handle failover instantly.
• Configurable. We primarily still use Memcache for caching but one of the teams uses Redis for both long-term storage and temporary expiry keys without taking on another external dependency.
• Fast. We process tens of thousands of RPS and it doesn't skip a beat.

Incentivized

• The Kafka Tool is a community-made Java application that looks and feels from the past century.
• Logging can be confusing. This certainly shows when we have to do troubleshooting.
• Hybrid scenarios - pub/sub, but there are services in and outside a Kubernetes cluster. Then there are a ~3 options, but only 2 (the harder ones) are production-safe.

Incentivized

• Redis is super fast but it comes with a cost. Whole dataset resides in RAM. So it can be costly as primary memory is more costly, then secondary ones.
• Persistence issues: To achieve it, Redis uses a memory dump to create a persistence snapshot, that's cool. But it requires some Linux Kernel tweaking to avoid performance degradation while the Redis server process is forking. This further causes latency.
• Master-slave structure side effect: Master-slave architecture comes with its own side effects. Please note that there will be only one master with multiple slaves for replication. All writing goes to the master, which creates more load on the master node. So, when the master goes down, the whole architecture does.

Incentivized

Kafka has suited our use case very well so far. Going forward we are planning to expand our platform manifold so the load on Kafka and our reliance on Kafka is going to increase only.

We will definitely continue using Redis because: 1. It is free and open source. 2. We already use it in so many applications, it will be hard for us to let go. 3. There isn't another competitive product that we know of that gives a better performance. 4. We never had any major issues with Redis, so no point turning our backs.

Incentivized

Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file

Incentivized

It is quite simple to set up for the purpose of managing user sessions in the backend. It can be easily integrated with other products or technologies, such as Spring in Java. If you need to actually display the data stored in Redis in your application this is a bit difficult to understand initially but is possible.

Incentivized

Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.

Incentivized

The support team has always been excellent in handling our mostly questions, rarely problems. They are responsive, find the solution and get us moving forward again. I have never had to escalate a case with them. They have always solved our problems in a very timely manner. I highly commend the support team.

Incentivized

Whitelisting of the AWS lambda functions.

Incentivized

Apache Kafka is built for scale. From high throughput and real-time data streaming, it has a strong advantage over RabbitMQ with its low latency. This put Apache Kafka at the forefront as the platform of choice for large datasets messaging and ensuring scalability when data scale up tremendously. RabbitMQ however has its strengths in traditional messaging. Routing and message delivery reliability are the bedrock of RabbitMQ and this is where RabbitMQ excels. In my previous workplace, RabbitMQ was of choice as reliability matters more than scale. In two words. Apache Kafka for scale, RabbitMQ for reliability. And for cloud deployment and large dataset messaging in what I am doing now, Apache Kafka is the default choice.

UI isn't that great compared to the other competitors. The management of our memcached cluster was becoming pretty complicated as the application grew in size. Redis is a much better option compared to memcached. Redis is bit unreliable compared to the alternative RabbitMQ especially when it needs to be integrated with Celery.

Incentivized

• Positive: bursts of traffic on special holidays are easy to handle because Kafka can absorb and buffer all the messages we need to process long enough to let an understaffed set of back-end services catch up on processing. Hard to put a number to it but we probably save $5k a month having fewer machines running.
• Positive: makes decoupling the web and API services from the deeper back-end services easier by providing topics as an interface. This allowed us to split up our teams and have them develop independently of each other, speeding up software development.
• Negative: our engineers have made mistakes such as accidentally dropping a few thousand messages due to the CLI being confusing to use, and as a result a customer lost some of their precious data. I'd say that was more our fault than Kafka's though.

Incentivized

• Existing tools like Redisson that were built over Redis reduced dev time in solving challenging problems, which had a positive impact on ROI.
• We initially misused Redis for persistent storage which had a negative impact on ROI because we were paying a lot for inactive users.
• The increased performance we achieved using Redis in areas like locking helped us improve the performance of our system reducing the likelihood of system timeouts.

Incentivized