Item: Apache Kafka
Rating: 8
Author: Borislav Traykov

Overall Satisfaction with Apache Kafka

Use Cases and Deployment Scope

Kafka is an event streaming platform and this is exactly the purpose we use it for in our company. Application data-in-transit goes into Kafka, which generates an even, and all relevant applications (consumers) get notified and then consume said messages. We are really happy with the volume of data we get through and the speed that we get from Kafka. It's used in multiple 1st and 3rd party components of the applications we develop in the entire company. It addresses data proliferation and notifications. If not for Kafka, we'd have to invent a pub/sub model (which multiple people have in the past in this company) - those are complex, hard to maintain, extend and customize. Kafka is fair well documented and used so there is a lot of info about multiple use cases online.

Pros and Cons

Pros

The pub/sub model
Quick data transfer - regardless of volume (if you have enough resources)
Ability to transfer large amounts of data consistently (non-binary)

Cons

The Kafka Tool is a community-made Java application that looks and feels from the past century.
Logging can be confusing. This certainly shows when we have to do troubleshooting.
Hybrid scenarios - pub/sub, but there are services in and outside a Kubernetes cluster. Then there are a ~3 options, but only 2 (the harder ones) are production-safe.

Return on Investment

Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.

Alternatives Considered

Redis and Enterprise Fluentd

For us, Kafka really doesn't have a 1:1 alternative. We have used ActiveMQ extensively and we still use it as a lighter option for small messages. The situation is similar with Redis - although it could be used like a Kafka alternative, we do use it just as a per-component queue which other components rarely access. Kafka is just easier for our group of developers to use and encompass more use cases without having to introduce more tools into the mix.

Key Insights

Do you think Apache Kafka delivers good value for the price?

Yes

Are you happy with Apache Kafka's feature set?

Yes

Did Apache Kafka live up to sales and marketing promises?

Yes

Did implementation of Apache Kafka go as expected?

Yes

Would you buy Apache Kafka again?

Yes

Other Software Used

Microsoft Teams, Jira Software, Atlassian Confluence

Likelihood to Recommend

Pub/sub model when more services are involved.
A lot of of technologies know how to work with Kafka. There are Kafka libraries for all general-purpose languages.
Quick and reliable data transit and notifications.
Kafka can have a big memory and/or disk footprint depending on your scenario. Be prepared to delegate resources if your amount of data gets more and more. Kafka is lean by default, but it does require memory (in-mem storage) and disk (offloading) to keep your data.
Kafka has a lot of configuration options - be sure to check them if you need to fit Kafka into a specific scenario.
The Kafka Tools looks ancient, but it does what it's supposed to.
If your developers are debugging, they may unintentionally "steal" events/data from a given queue as they would probably register as a consumer. This is very nasty especially when dealing with a living system There are ways to avoid this, but people need to be aware that it can happen.

Comments

Please log in to join the conversation

Apache Kafka is awesome! Tricky sometimes, but we love it!