Apache Kafka vs. Grafana

Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.

Grafana is a one stop solution for all application monitoring needs. In our organisation, we use nodeJs. We run it using pm2. Since we didn't use Grafana it was hard for us to know if an app has stopped working unless we find it ourselves because our other app components were failing, or someone called us up and told about the situation. We were already using Grafana for monitoring our Ubuntu servers, but we hadn't had set it up for our app monitoring. Somehow we decided to monitor our nodeJs app with Grafana. It's a really good decision we made. We used Grafana nodeJs module "prom-client" for our node app which brought us relief from app failing situations. Since we have implemented Grafana for our Node app, it has helped us to monitor every health aspect of our node app. Now we have set up alerts based on heap-memory, so when heap memory goes beyond a set threshold we get notified and take the right steps ultimately saving our app from crashing and of course from losing business and reputation.

Incentivized

• Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
• Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
• Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).

Incentivized

• It visualizes metrics very well coming from any well known data source.
• It sends alerts to collaboration channels when a threshold is breeched.
• Graphs and dashboards are portable (Graph-as-a-Code).

• Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
• Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
• Learning curve around creation of broker and topics could be simplified

• Functions to customize values
• Improved user experience

Incentivized

Kafka is quickly becoming core product of the organization, indeed it is replacing older messaging systems. No better alternatives found yet

Incentivized

Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file

Incentivized

Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.

Incentivized

I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.

Incentivized

Grafana has a direct plugin to Icinga monitoring solution and allowed for easy configuration for us. At the time of implementation, other services did not have such an integration. As we already had a very customized and heavily introduced monitoring solution in place, we needed settings that could plug into the system quickly and efficiently. This was the case with Grafana and allowed us to have all the integrations we needed with services such as Icinga

Incentivized

• Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
• Positive: it's scalable so we can develop small and scale for real-world scenarios
• Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.

Incentivized

• Helps us to keep our application and server up all time
• Dashboards are easy to share with others

Incentivized