Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
Redis™*
Score 9.0 out of 10
N/A
Redis is an open source in-memory data structure server and NoSQL database.
For us, Kafka really doesn't have a 1:1 alternative. We have used ActiveMQ extensively and we still use it as a lighter option for small messages. The situation is similar with Redis - although it could be used like a Kafka alternative, we do use it just as a per-component …
Memcached didn't offer the possibility to distribute data easily across nodes. Memcached was more for near cache features whereas we were looking for a distributed cache. Hazelcast and ActiveSpaces are considered as data grid with a powerful approach while distributing data …
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
Redis has been a great investment for our organization as we needed a solution for high speed data caching. The ramp up and integration was quite easy. Redis handles automatic failover internally, so no crashes provides high availability. On the fly scaling scale to more/less cores and memory as and when needed.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
Easy for developers to understand. Unlike Riak, which I've used in the past, it's fast without having to worry about eventual consistency.
Reliable. With a proper multi-node configuration, it can handle failover instantly.
Configurable. We primarily still use Memcache for caching but one of the teams uses Redis for both long-term storage and temporary expiry keys without taking on another external dependency.
Fast. We process tens of thousands of RPS and it doesn't skip a beat.
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
We had some difficulty scaling Redis without it becoming prohibitively expensive.
Redis has very simple search capabilities, which means its not suitable for all use cases.
Redis doesn't have good native support for storing data in object form and many libraries built over it return data as a string, meaning you need build your own serialization layer over it.
We will definitely continue using Redis because: 1. It is free and open source. 2. We already use it in so many applications, it will be hard for us to let go. 3. There isn't another competitive product that we know of that gives a better performance. 4. We never had any major issues with Redis, so no point turning our backs.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
It is quite simple to set up for the purpose of managing user sessions in the backend. It can be easily integrated with other products or technologies, such as Spring in Java. If you need to actually display the data stored in Redis in your application this is a bit difficult to understand initially but is possible.
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
The support team has always been excellent in handling our mostly questions, rarely problems. They are responsive, find the solution and get us moving forward again. I have never had to escalate a case with them. They have always solved our problems in a very timely manner. I highly commend the support team.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
We are big users of MySQL and PostgreSQL. We were looking at replacing our aging web page caching technology and found that we could do it in SQL, but there was a NoSQL movement happening at the time. We dabbled a bit in the NoSQL scene just to get an idea of what it was about and whether it was for us. We tried a bunch, but I can only seem to remember Mongo and Couch. Mongo had big issues early on that drove us to Redis and we couldn't quite figure out how to deploy couch.
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
Redis has helped us increase our throughput and server data to a growing amount of traffic while keeping our app fast. We couldn't have grown without the ability to easily cache data that Redis provides.
Redis has helped us decrease the load on our database. By being able to scale up and cache important data, we reduce the load on our database reducing costs and infra issues.
Running a Redis node on something like AWS can be costly, but it is often a requirement for scaling a company. If you need data quickly and your business is already a positive ROI, Redis is worth the investment.
