Apache Flume vs. Splunk Enterprise

Apache Flume is well suited when the use case is log data ingestion and aggregate only, for example for compliance of configuration management. It is not well suited where you need a general-purpose real-time data ingestion pipeline that can receive log data and other forms of data streams (eg IoT, messages).

Incentivized

Pros: Splunk is very well suited if you have multiple log sources of related data. All of them can be correlated and tasks can be automated based on the requirement. Other than alerts, Splunk can also run a specific script of your choice, based on some defined conditions. Cons: If you have a few logs but a large number of log sources, Splunk can be very expensive.

Incentivized

• Multiple sources of data (sources) and destinations (sinks) that allows you to move data form and to any relevant data storage
• It is very easy to setup and run
• Very open to personalization, you can create filters, enrichment, new sources and destinations

Incentivized

• Real-time + Scheduled alerts - i-e you can set up alerts which are actively monitoring your logs
• Pretty good response time for search results. With our key/value logging, Splunk makes it blazing fast to query the data.
• Dashboards provide insights into historical data
• Love how Splunk indexes all of the data and provides keys to search on

Incentivized

• It is very specific for log data ingestion so it is pretty hard to use for anything else besides log data
• Data replication is not built in and needs to be added on top of Apache Flume (not a hard job to do though)

Incentivized

• At times some queries can run slowly if indices are not on a portion of the query you use.
• Setup time initially can be difficult if your logs aren't stored in common locations or in a common way to write the log.
• Ability to ingest logs from different locations without having to change code to put logs in a certain place (pro and con).
• Searches can be a bit more difficult to look through if your log isn't pulled in a manner that is easy to read through splunk.

Incentivized

We are using Splunk extensively in our projects and we have recently upgraded to Splunk version 6.0 which is quite efficient and giving expected results. We keep track of updates and new features Splunk introduces periodically and try to introduce those features in our day to day activities for improvement in our reporting system and other tasks.

You can literally throw in a single word into Splunk and it will pull back all instances of that word across all of your logs for the time span you select (provided you have permission to see that data). We have several users who have taken a few of the free courses from Splunk that are able to pull data out of it everyday with little help at all.

Incentivized

When properly setup and configured, Splunk is extremely reliable.

Incentivized

Apache Flume is open-source so support is limited. Never the less, it has great documentation and best practices documents from their end-users so it is not hard to use, setup and configure.

Incentivized

Splunk maintains a well resourced support system that has been consistent since we purchased the product. They help out in a timely manner and provide expert level information as needed. We typically open cases online and communicate when possible via e-mail and are able to resolve most issues with that method.

Incentivized

The online course was simple clear and described the main capabilities of the solution. There is also an initial module that can be done for free so anyone can familiarize themselves with the functionality of this solution. On the other hand, however, there could be more free online courses. Maybe even with a certificate, this would broaden the group of people who are familiar with the platform while increasing familiarity with the solution itself.

Smooth without too many major issues.

Incentivized

Apache Flume is a very good solution when your project is not very complex at transformation and enrichment, and good if you have an external management suite like Cloudera, Hortonworks, etc. But it is not a real EAI or ETL like AB Initio or Attunity so
you need to know exactly what you want. On the other hand being an opensource project give Apache a lot of room to personalize thanks to its plug-able architecture and has a very nice performance having a very low CPU and Memory footprint, a single server can do the job on many occasions, as opposed to the multi-server architecture of paid products.

Incentivized

I wanted to learn a new language that I can quickly master and implement. Splunk is easy, fun to use and best of all, it can be developed in hours not days or weeks. Splunk is fundamentally a programming language that is minimal but yet powerful enough to collect, analyze and visualize data.

Incentivized

Splunk can scale in to the petabyte per day range which of course is awesome

• Flume has simplified a lot many of our ingest procedures, easier to deploy and integrate than a classical EAI, reducing the time to market
• But opposed to EAIs if the project starts to grow in complexity Apache Flume project may not be as suitable

Incentivized

• Overall very positive. It has provided visibility to what is going on within our network.
• One drawback is the time it takes to get up to speed with the application, but this is up to the user, and Splunk education is excellent.
• In my field, IT Security, there are few other friends to have in your back pocket better than Splunk. They are just that good.

Incentivized