Apache Flume for log aggregation and compliance monitoring in real-time
Apache Flume is used for aggregating and analyzing log data in near-real-time across the organization for compliance purposes with a goal …
Apache Flume
Apache Flume
Apache Flume
Overview
What is Apache Flume?
Apache Flume is a product enabling the flow of logs and other data into a Hadoop environment.
Read moreRecent Reviews
Product Details
- About
- Tech Details
What is Apache Flume?
Apache Flume is a product enabling the flow of logs and other data into a Hadoop environment.
Apache Flume Technical Details
| Operating Systems | Unspecified |
|---|---|
| Mobile Application | No |
Comparisons
View all alternativesCompare with
Reviews and Ratings
 (9)
Attribute Ratings
Reviews
(1-2 of 2)- Popular Filters
Companies can't remove reviews or game the system. Here's why
Apache Flume is used for aggregating and analyzing log data in near-real-time across the organization for compliance purposes with a goal to generate monthly compliance reports based on log data.
- Apache Flume being a log-centric system, it is able to parse and aggregate log data very well.
- It is easy to customize it for different source (producers) for log data ingestion as well as for sinks (consumers).
- It is very specific for log data ingestion so it is pretty hard to use for anything else besides log data
- Data replication is not built in and needs to be added on top of Apache Flume (not a hard job to do though)
- Positive impact on ROI due to a reduction in manual labor to generate and maintain compliance reports based on logs.
- Positive impact on the business objective by reducing the need for provisioning compute for log aggregate IT stack in advance but adding on an as-needed basis.
- TIBCO Streaming (StreamBase), Apache Kafka, Google Cloud Pub/Sub, IBM MQ and Apama Streaming Analytics
Apache Flume is on par with Scribe with similar functions. Apache Kafka is a generation purpose while Apache Flume is specific to log aggregation. Google Pub/Sub and IBM MQ are costlier than Apache Flume ( open source ) and have a lot more cost associated with them. Apama Streaming Analytics and Tibco Steaming are more comprehensive streaming solutions than Apache Flume so for deeper performance guarantees, it is easier to use Apache Flume.
November 03, 2017
Apache Flume, the way your information flows
Apache Flume is a key software piece in BigData environments, we have used it along with CDC (Change Data Capture) to ingest near real time database changes into Kafka so the data is available for realtime analysis, machine learning, dynamic dashboards and so
on.
We have successfully integrated also Apache Flume in log acquisition solutions (mainly PaaS and Docker) where application log is difficult access.
We have successfully integrated also Apache Flume in log acquisition solutions (mainly PaaS and Docker) where application log is difficult access.
- Multiple sources of data (sources) and destinations (sinks) that allows you to move data form and to any relevant data storage
- It is very easy to setup and run
- Very open to personalization, you can create filters, enrichment, new sources and destinations
- Apache Flume develops new functionality at a slower pace than other OpenSource projects, it is well behing Kafka and has some compatibiliy issues with latest releases
- It lack HA or FT, it relies on third party management software like Hortonworks or Cloudera
- Flume has simplified a lot many of our ingest procedures, easier to deploy and integrate than a classical EAI, reducing the time to market
- But opposed to EAIs if the project starts to grow in complexity Apache Flume project may not be as suitable
Apache Flume is a very good solution when your project is not very complex at transformation and enrichment, and good if you have an external management suite like Cloudera, Hortonworks, etc. But it is not a real EAI or ETL like AB Initio or Attunity so
you need to know exactly what you want.
On the other hand being an opensource project give Apache a lot of room to personalize thanks to its plug-able architecture and has a very nice performance having a very low CPU and Memory footprint, a single server can do the job on many occasions, as opposed to the multi-server architecture of paid products.
On the other hand being an opensource project give Apache a lot of room to personalize thanks to its plug-able architecture and has a very nice performance having a very low CPU and Memory footprint, a single server can do the job on many occasions, as opposed to the multi-server architecture of paid products.