Apache Kafka

Apache Kafka is really the bedrock of all things streaming and data processing. I cannot imagine if there is any other product that does it better. My last 2 companies used it, and my current one does so as well. If you want your data stream to be organized and sent, Apache Kafka has become the tool of choice. I have dabbled in Azure EventHubs as well, if you are into opensource data streaming, Apache Kafka will take you where you need to be for data lakes and the amount of data that is streamed for the cybersecurity industry that my company is in. Without Apache Kafka, there is no way that my company products can handle the volume of data that we process for our customers.

We use Apache Kafka as message broker between our two client facing applications. We used ActiveMQ before but it had shortfalls of high availability and clustering. Kafka solved it on both fronts and gives a good business continuity.

We use Apache Kafka as an event bus for all our async activities & Micro Service Communication, like sending emails, SMS, and notifications between services and consumers and for event & data processing.

We use Apache Kafka to stream order information across systems. An order may go through certain updates through its lifecycle. These updates need to be communicated to the systems in near real time and we rely on Kafka for this.Our business use case is to take these orders up with the insurance companies for approval and thus the order information need to be up to date. Kafka has been excellent at doing this so far.

Currently consulting and implementing for a bank, we use a cloud-native Kafka solution (Confluent Kafka) for brokering. The solution is well documented, and liked by the developers but lacks certain technical aspects to improve usability and administration.

Apache Kafka is the most powerful and scalable streaming framework on the market. We have used Apache Kafka as a part of many real-time analytics solutions. It has a great performance [and is] easy to integrate with big data technologies like Spark. Due to its distributed nature, Apache Kafka is capable of operating very quickly and can handle millions of messages every second.

Kafka is an event streaming platform and this is exactly the purpose we use it for in our company. Application data-in-transit goes into Kafka, which generates an even, and all relevant applications (consumers) get notified and then consume said messages. We are really happy with the volume of data we get through and the speed that we get from Kafka. It's used in multiple 1st and 3rd party components of the applications we develop in the entire company. It addresses data proliferation and notifications. If not for Kafka, we'd have to invent a pub/sub model (which multiple people have in the past in this company) - those are complex, hard to maintain, extend and customize. Kafka is fair well documented and used so there is a lot of info about multiple use cases online.

We use Kafka as the queuing mechanism for records in an indexing pipeline. Previous to using Kafka we were working with tables in SQL Server to handle a queue in a situation that SQL is not really designed for. Kafka provides a simple and efficient system that does the job it was intended for, queuing and maintaining records in a queue, and works very well. We use Kafka for several processes in our organization that require records to be stored and be processed by dedicated servers.

My application was dependent on other applications to generate data and those data were needed to be processed immediately. And, processed data were published for other applications. Moreover, data load was very high nearly a hundred thousand a day. And, consumed data may be replayed in the future if required. So, after carefully considering several messaging queues we finally decided to continue with Apache Kafka.

Kafka is being used for our IoT data flows as the middle layer to transport data and make it available for consumption. We are implementing it slowly starting project by project and plan to use it globally.

It is being used for the product mainly. We have huge data pipelines running which depend on Apache Kafka. It is being used for more than 5 years now and we are really happy with the performance and the reliability Apache Kafka has to offer. The experience has been excellent.

Kafka is being used for sending log information in real time and there[fore] can monitor apps and send these events to feed other apps. It's the core for send[ing] and receiv[ing] messages due to quantity of messages per second. Helps us to scale and manage the common errors in this type of problem.

Apache Kafka is used by our company as the "next generation" of messaging/data-streaming pipeline solutions, to replace our old legacy JMS-based messaging solution and enable the modern streaming API based applications. When it is used for messaging purposes, we shift the responsibility of data replay from the message source (publisher application) to the message destination (consumer application). This flexibility resolved the legacy issue of sources replaying the messages but impacting all subscribers to the same topic. When Kafka is used as the streaming pipeline, it is integrated seamlessly with the Spark/Spring Stream-based analytic solutions, as it is also a kind of distributed storage.

We used it for event logging. It was used for application log collection. Was used with exception tracking and with core microservices of the web application. It helped us reduce cost and simplified operational monitoring.

Apache Kafka is used as a stream/message ingestion engine for all the customer-facing apps including some internal streams company-wide. It is used to ingest close to 2-5 million small (few bytes) messages per second that are then used for internal analytics and decision making in realtime and feed analytics backend (Tibco Spotfire).

We use Kafka for two key features: (1) keeping a buffer of all the incoming records that need to be stored in our data infrastructure, and (2) having a way to replay messages in case our data infrastructure loses some data.
The reason we need to buffer is that when our traffic spikes, we can have up to 1 million messages coming in that need to be processed in some form or fashion. To expect the back-end service to support that is crazy. Instead, we dump them into Kafka to give our data infrastructure time to ingest them. As for replaying events, sometimes the ingestion pipeline fails and drops some messages. I know - that's a huge mistake on our engineering team's part - but when it does happen Kafka has the ability to rewind and replay messages, resulting in delayed processing but no data loss.

We are using Kafka as an ingress and egress queue for data being saved into a big data system. Kafka is also being used as a queue for frontend applications to use in order to retrieve data and analytics from MapR and HortonWorks.

Apache Kafka is becoming the new standard for messaging at our organization. Originally we limited the use to big data environments and projects but as the technology is becoming more mature we think it will eventually replace classical messaging software.