Apache Hadoop vs. Apache Spark vs. IBM Cloud Object Storage

Altogether, I want to say that Apache Hadoop is well-suited to a larger and unstructured data flow like an aggregation of web traffic or even advertising. I think Apache Hadoop is great when you literally have petabytes of data that need to be stored and processed on an ongoing basis. Also, I would recommend that the software should be supplemented with a faster and interactive database for a better querying service. Lastly, it's very cost-effective so it is good to give it a shot before coming to any conclusion.

Incentivized

Well suited: To most of the local run of datasets and non-prod systems - scalability is not a problem at all. Including data from multiple types of data sources is an added advantage. MLlib is a decently nice built-in library that can be used for most of the ML tasks. Less appropriate: We had to work on a RecSys where the music dataset that we used was around 300+Gb in size. We faced memory-based issues. Few times we also got memory errors. Also the MLlib library does not have support for advanced analytics and deep-learning frameworks support. Understanding the internals of the working of Apache Spark for beginners is highly not possible.

Incentivized

In my experience, IBM Cloud Object Storage is well suited for projects like the one I am working on. This includes the use of natural language classification and the uploading of data to train a machine learning model for tag suggestions based on a body of text. Using IBM Cloud Object Storage has helped with this greatly. IBM Cloud Object Storage has also been great for Big Data Analytics thanks to its scalablilty and ease of use for large datasets. Alongside IBM Watson and our team's internal big data tools we've managed to process and analyze data more efficiently, leading to key insights that have driven business value for our clients.

• Handles large amounts of unstructured data well, for business level purposes
• Is a good catchall because of this design, i.e. what does not fit into our vertical tables fits here.
• Decent for large ETL pipelines and logging free-for-alls because of this, also.

Incentivized

• Rich APIs for data transformation making for very each to transform and prepare data in a distributed environment without worrying about memory issues
• Faster in execution times compare to Hadoop and PIG Latin
• Easy SQL interface to the same data set for people who are comfortable to explore data in a declarative manner
• Interoperability between SQL and Scala / Python style of munging data

Incentivized

• IBM Cloud Object Storage is an excellent choice for disaster recovery and backup solutions. Its high durability and geographic redundancy ensure that our backup data is safe and can be quickly restored in case of a disaster. This capability is crucial for maintaining our business continuity and minimizing downtime. We have deployed our loads in an IKS cluster distributed in 3 different AZs with stateful data allocated in COS.
• We have a video streaming application and need to store and deliver a vast library of video content to millions of users worldwide, so we store our data in COS, which is cheap and reliable.
• We have a bunch of data that must be analyzed and stored in datasets for fraud detection, risk management, and customer insights. In these cases, this data is moved from Onprem to IBM Cloud so we can use cheap storage like COS.

Incentivized

• Less organizational support system. Bugs need to be fixed and outside help take a long time to push updates
• Not for small data sets
• Data security needs to be ramped up
• Failure in NameNode has no replication which takes a lot of time to recover

Incentivized

• Memory management. Very weak on that.
• PySpark not as robust as scala with spark.
• spark master HA is needed. Not as HA as it should be.
• Locality should not be a necessity, but does help improvement. But would prefer no locality

Incentivized

• Searching and retrieving—full-text search or metadata search—is one of the significant areas of improvement. It isn't easy to search for data with this.
• Integration with other IBM cloud services is trickier. For example, integrating this with API Connect to access the data from API will be difficult for users.
• Support - I think you should have more support community.

Incentivized

Hadoop is organization-independent and can be used for various purposes ranging from archiving to reporting and can make use of economic, commodity hardware. There is also a lot of saving in terms of licensing costs - since most of the Hadoop ecosystem is available as open-source and is free

Capacity of computing data in cluster and fast speed.

because it is a robust, safe and flexible product

Incentivized

As Hadoop enterprise licensed version is quite fine tuned and easy to use makes it good choice for Hadoop administrators. It’s scalability and integration with Kerberos is good option for authentication and authorisation. installation can be improved. logging can be improved so that it become easier for debugging purposes. parallel processing of data is achieved easily.

Incentivized

If the team looking to use Apache Spark is not used to debug and tweak settings for jobs to ensure maximum optimizations, it can be frustrating. However, the documentation and the support of the community on the internet can help resolve most issues. Moreover, it is highly configurable and it integrates with different tools (eg: it can be used by dbt core), which increase the scenarios where it can be used

Incentivized

For my use cases, it has been a very smooth experience. Even my new colleagues have been able to get on top of things very quickly. This shows how easy it is to work with

Incentivized

We rarely face downtime or access issues with IBM Cloud Object Storage. It’s mostly available when we need it, even during peak hours or heavy data loads.

Incentivized

I would give it a 9 because it works smooth with our AI and analytics tools, no major slowdown. Pages and dashboards load fine most of the time, and reports finish in decent time even when data is heavy.

Incentivized

It's a great value for what you pay, and most Data Base Administrators (DBAs) can walk in and use it without substantial training. I tend to dabble on the analyst side, so querying the data I need feels like it can take forever, especially on higher traffic days like Monday.

Incentivized

1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.

I have been working in IT sector for more than 15 years. I have worked with various vendors. IBM's sales team, support team have been really helpful. After we start to use their product, their UX design team also contacted us to get feedback from us. They are really interested about our experience.

Incentivized

Hadoop is a complex topic and best suited for classrom training. Online training are a waste of time and money.

I just researching and applying the tools on their platforms to ensure a good learning path, based on my needs. Reading the documentation related with resources, tools. Is too big, but I am trying to know more about it every day. It is a good way to know more about their resources. A new way to attract new customers. At the end of the day, we are all involved in improvement and automation of our tasks and resources for customers and end-users.

Incentivized

Yes Our organization used IBM professional services to implement IBM object storage because of its data consistency and multiple way to upload and download data and its encryption security features. Also that its brand matter for the any organization to secure the layer and storage. It sis also verify that application and system are compatibale for this product

Incentivized

Not used any other product than Hadoop and I don't think our company will switch to any other product, as Hadoop is providing excellent results. Our company is growing rapidly, Hadoop helps to keep up our performance and meet customer expectations. We also use HDFS which provides very high bandwidth to support MapReduce workloads.

Incentivized

Spark in comparison to similar technologies ends up being a one stop shop. You can achieve so much with this one framework instead of having to stitch and weave multiple technologies from the Hadoop stack, all while getting incredibility performance, minimal boilerplate, and getting the ability to write your application in the language of your choosing.

Incentivized

Amazon S3 is a great service to safely back up your data where redundancy is guaranteed, and the cost is fair. In the past I have used Amazon S3 for data that we backup and hope we never need to access, but in the case of a catastrophic or even small slip of the finger with the delete command, we know our data and our client's data is safely backed up by Amazon S3. Amazon S3 service is a good option, but based on the features it provides compared with IBM Cloud Object Storage, it is less suitable. IBM Cloud Object Storage is also integrated with more services, like IBM Cloud SQL and IBM Aspera, which AWS does not provide to transfer files at maximum speed in the world.

Incentivized

Scaling up the number of users can lead to significant increases in licensing costs, which, while not a technical limitation, can be a practical constraint for some organizations

Incentivized

• There are many advantages of Hadoop as first it has made the management and processing of extremely colossal data very easy and has simplified the lives of so many people including me.
• Hadoop is quite interesting due to its new and improved features plus innovative functions.

Incentivized

• Business leaders are able to take data driven decisions
• Business users are able access to data in near real time now . Before using spark, they had to wait for at least 24 hours for data to be available
• Business is able come up with new product ideas

Incentivized

• This allows us to recommend a platform to our clients that will quickly help them create new, efficient business processes with very little development.
• This saves clients hours and days of manual analysis of images, allowing the system to do the work when attaching Object Storage to models.
• There is a learning curve in utilizing the storage and the modeling, but once up and running, it works well during deployment.

Incentivized