A newbie's look at Hadoop
June 03, 2016
A newbie's look at Hadoop
Score 8 out of 10
Vetted Review
Verified User
Modules Used
- Hadoop Common
- Hadoop Distributed File System
- Hadoop MapReduce
- Yarn, Oozie, Spark, Hive, Hue
Overall Satisfaction with Hadoop
We needed a robust/redundant system to run multiple simultaneous jobs for our ETL pipeline, this needed distributed storage space, integration with Windows AD user accounts and the ability to expand when needed with little to no downtime.
We are using Cloudera 5.6 to orchestrate the install (along with puppet) and manage the hadoop cluster.
We are using Cloudera 5.6 to orchestrate the install (along with puppet) and manage the hadoop cluster.
- The distributed replicated HDFS filesystem allows for fault tolerance and the ability to use low cost JBOD arrays for data storage.
- Yarn with MapReduce2 gives us a job slot scheduler to fully utilize available compute resources while providing HA and resource management.
- The hadoop ecosystem allows for the use of many different technologies all using the same compute resources so that your spark, samza, camus, pig and oozie jobs can happily co-exist on the same infrastructure.
- Without Cloudera as a management interface the hadoop components are much harder to manage to ensure consistency across a cluster.
- The calculations of hardware resources to job slots/resource management can be quite an exercise in finding that "sweet spot" with your applications, a more transparent way of figuring this out would be welcome.
- A lot of the roles and management pieces are written in java, which from an administration perspective can have there own issues with garbage collection and memory management.
- With our current platform (and budget) hadoop is really the only option at this time to gain access to the capacity and technologies we require.
- So far the only real investment has been hardware and man hours, especially in the initial learning and deployment phase.
As I am new to the hadoop ecosystem I have not used or evaluated any other similar products at this time. This was handed to me from a previous much older installation that was very under utilized. Our new platform will be working the new cluster much harder with jobs that run indefinitely. I'm not sure that any of the other "big data" technologies out there have as many certified components or work with such a diverse collection but as I said I am pretty new to this and so have only tertiary knowledge of competing products.