Apache Kafka

Kafka is being used for our IoT data flows as the middle layer to transport data and make it available for consumption. We are implementing it slowly starting project by project and plan to use it globally.

My application was dependent on other applications to generate data and those data were needed to be processed immediately. And, processed data were published for other applications. Moreover, data load was very high nearly a hundred thousand a day. And, consumed data may be replayed in the future if required. So, after carefully considering several messaging queues we finally decided to continue with Apache Kafka.

We used it for event logging. It was used for application log collection. Was used with exception tracking and with core microservices of the web application. It helped us reduce cost and simplified operational monitoring.

Apache Kafka is really the bedrock of all things streaming and data processing. I cannot imagine if there is any other product that does it better. My last 2 companies used it, and my current one does so as well. If you want your data stream to be organized and sent, Apache Kafka has become the tool of choice. I have dabbled in Azure EventHubs as well, if you are into opensource data streaming, Apache Kafka will take you where you need to be for data lakes and the amount of data that is streamed for the cybersecurity industry that my company is in. Without Apache Kafka, there is no way that my company products can handle the volume of data that we process for our customers.

Apache Kafka is becoming the new standard for messaging at our organization. Originally we limited the use to big data environments and projects but as the technology is becoming more mature we think it will eventually replace classical messaging software.