Kafka for tracking changes
May 30, 2023

Kafka for tracking changes

Animesh Kumar | TrustRadius Reviewer
Score 8 out of 10
Vetted Review
Verified User

Overall Satisfaction with Apache Kafka

We use Apache Kafka to stream order information across systems. An order may go through certain updates through its lifecycle. These updates need to be communicated to the systems in near real time and we rely on Kafka for this.Our business use case is to take these orders up with the insurance companies for approval and thus the order information need to be up to date. Kafka has been excellent at doing this so far.

Pros

  • Receiving messages from publisher and sending to consumer in FIFO manner
  • Handling of errors using Dead Letter Queue when message could not be consumed on the consumer end
  • Fault tolerance

Cons

  • Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
  • Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
  • Learning curve around creation of broker and topics could be simplified
  • High throughput
  • Low latency
  • Fault tolerance
  • We are able to submit orders to the insurance companies with almost 100% accuracy because we receive Kafka updates in almost real time
  • We are getting notified of error scenarios separately because of our Dead Letter Queue implementation so that we can handle those cases
  • There is certain engineering effort being spent to maintain Kafka
Apache Kafka can work at a higher scale as compared to SQS. It can work with higher size per message and millions of messages per second. Moreover it can be scaled horizontally by adding more brokers to the cluster. SQS is good enough for simple use cases like making a task async by passing it to the worker app or delaying a task execution by certain time duration but not advisable for heavy load systems.

Do you think Apache Kafka delivers good value for the price?

Yes

Are you happy with Apache Kafka's feature set?

Yes

Did Apache Kafka live up to sales and marketing promises?

Yes

Did implementation of Apache Kafka go as expected?

Yes

Would you buy Apache Kafka again?

Yes

Kafka is well suited in scenarios where a message need to be sent to another system in fault tolerant manner. It is useful when the message size could be large and large number of messages could be floating around.
It would be less appropriate or rather an overkill to use Kafka in scenarios where we are sending short messages to offload certain tasks(like invoice generation and sending email) to a worker(like celery). For such use cases, simple queueing solutions like Amazon SQS should suffice.

Using Apache Kafka

1000 - We have an engineering team of about 1500 engineers and almost 2/3 of them use Kafka for some or other use case.
50 - Support for Kafka comes from within our developer community. So the people managing Kafka are engineers only.
  • Publishing messages to other services to inform them of modification in an entity
  • Publishing messages to Camunda to mark a user task as complete
Kafka has suited our use case very well so far. Going forward we are planning to expand our platform manifold so the load on Kafka and our reliance on Kafka is going to increase only.

Comments

More Reviews of Apache Kafka

  1. Home
  2. Message Queue Software
  3. Apache Kafka Reviews
  4. User Review of Apache Kafka