Likelihood to Recommend Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
Read full review Perfect for projects where
Elasticsearch makes sense: if you decide to employ ES in a project, then you will almost inevitably use LogStash, and you should anyways. Such projects would include: 1. Data Science (reading, recording or measure web-based Analytics, Metrics) 2. Web Scraping (which was one of our earlier projects involving LogStash) 3. Syslog-ng Management: While I did point out that it can be a bit of an electric boo-ga-loo in finding an errant configuration item, it is still worth it to implement Syslog-ng management via LogStash: being able to fine-tune your log messages and then pipe them to other sources, depending on the data being read in, is incredibly powerful, and I would say is exemplar of what modern Computer Science looks like: Less Specialization in mathematics, and more specialization in storing and recording data (i.e. Less Engineering, and more Design).
Read full review Pros Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak. Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered) Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many). Read full review Logstash design is definitely perfect for the use case of ELK. Logstash has "drivers" using which it can inject from virtually any source. This takes the headache from source to implement those "drivers" to store data to ES. Logstash is fast, very fast. As per my observance, you don't need more than 1 or 2 servers for even big size projects. Data in different shape, size, and formats? No worries, Logstash can handle it. It lets you write simple rules to programmatically take decisions real-time on data. You can change your data on the fly! This is the CORE power of Logstash. The concept is similar to Kafka streams, the difference being the source and destination are application and ES respectively. Read full review Cons Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great. Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed. Learning curve around creation of broker and topics could be simplified Read full review Since it's a Java product, JVM tuning must be done for handling high-load. The persistent queue feature is nice, but I feel like most companies would want to use Kafka as a general storage location for persistent messages for all consumers to use. Using some pipeline of "Kafka input -> filter plugins -> Kafka output" seems like a good solution for data enrichment without needing to maintain a custom Kafka consumer to accomplish a similar feature. I would like to see more documentation around creating a distributed Logstash cluster because I imagine for high ingestion use cases, that would be necessary. Read full review Likelihood to Renew Kafka is quickly becoming core product of the organization, indeed it is replacing older messaging systems. No better alternatives found yet
Read full review Usability Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
Read full review Support Rating Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
Read full review Alternatives Considered I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
Read full review MongoDB and
Azure SQL Database are just that: Databases, and they allow you to pipe data into a database, which means that alot of the log filtering becomes a simple exercise of querying information from a DBMS. However, LogStash was chosen for it's ease of integration into our choice of using ELK
Elasticsearch is an obvious inclusion: Using Logstash with it's native DevOps stack its really rational
Read full review Return on Investment Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily. Positive: it's scalable so we can develop small and scale for real-world scenarios Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort. Read full review Positive: Learning curve was relatively easy for our team. We were up and running within a sprint. Positive: Managing Logstash has generally been easy. We configure it, and usually, don't have to worry about misbehavior. Negative: Updating/Rehydrating Logstash servers have been little challenging. We sometimes even loose data while Logstash is down. It requires more in-depth research and experiments to figure the fine-grained details. Negative: This is now one more application/skill/server to manage. Like any other servers, it requires proper grooming or else you will get in trouble. This is also a single point of failure which can have the ability to make other servers useless if it is not running. Read full review ScreenShots