Apache Flume vs. Azure HDInsight

Apache Flume is well suited when the use case is log data ingestion and aggregate only, for example for compliance of configuration management. It is not well suited where you need a general-purpose real-time data ingestion pipeline that can receive log data and other forms of data streams (eg IoT, messages).

Incentivized

Well suited: A tiny-mid sized company with no immediate plans of growing the volume of their data processing, that can afford long response times from support. Also it helps if you are not prone to put your hands on Linux and Spark configuration. In fact, it can make things go really faster if you also work with the bundle-in Jupyter. And, if you need to perform some diagnostics and / or administrative tasks, that's full of tools to find an understand the Root Cause. Ideal for non experts. Less appropriate: Big Data company, intense on demand cluster creation, mission critical, costs reduction, latest versions of libraries required, sophisticate customizations required.

Incentivized

• Multiple sources of data (sources) and destinations (sinks) that allows you to move data form and to any relevant data storage
• It is very easy to setup and run
• Very open to personalization, you can create filters, enrichment, new sources and destinations

Incentivized

• Data is presented without interfering others (IT or other dept).
• Data is managed properly and is available for retrievable any time.
• Legacy use of CD/DVD and Pendrive are not required.

Incentivized

• It is very specific for log data ingestion so it is pretty hard to use for anything else besides log data
• Data replication is not built in and needs to be added on top of Apache Flume (not a hard job to do though)

Incentivized

• The only problem I have come across is when loading large volumes of data I sometimes get an error message, I assume this means something is corrupt from within. I would love a way for this to be resolved without having to start over.

Incentivized

Azure HDInsight is usable on the top of Azure Data Lake and gives us the benefit of analyzing large scale data workload in Hadoop. Usability and support from Microsoft are outstanding.

Incentivized

Apache Flume is open-source so support is limited. Never the less, it has great documentation and best practices documents from their end-users so it is not hard to use, setup and configure.

Incentivized

Inexpert, isolated teams... not good for support an excessively complex platform. Lots of weeks or months for a complex problem troubleshoot. Many time lost stuck on MindTree, before the case was finally escalated with Microsoft!

Incentivized

Apache Flume is a very good solution when your project is not very complex at transformation and enrichment, and good if you have an external management suite like Cloudera, Hortonworks, etc. But it is not a real EAI or ETL like AB Initio or Attunity so
you need to know exactly what you want. On the other hand being an opensource project give Apache a lot of room to personalize thanks to its plug-able architecture and has a very nice performance having a very low CPU and Memory footprint, a single server can do the job on many occasions, as opposed to the multi-server architecture of paid products.

Incentivized

At this time I have not used any other similar products... I am open to it but Azure HDInsight and its components really work well for our organization.

Incentivized

• Flume has simplified a lot many of our ingest procedures, easier to deploy and integrate than a classical EAI, reducing the time to market
• But opposed to EAIs if the project starts to grow in complexity Apache Flume project may not be as suitable

Incentivized

• ROI is of course there, as no legacy software for data presentation.
• No manual intervention for data retrieval.
• Data is available anywhere as requested.

Incentivized