Apache Spark vs. Hive

Apache Spark has rich APIs for regular data transformations or for ML workloads or for graph workloads, whereas other systems may not such a wide range of support. Choose it when you need to perform data transformations for big data as offline jobs, whereas use MongoDB-like distributed database systems for more realtime queries.

Incentivized

Hive is great for managing projects with your team. Assigning tasks is simple enough using Hive. It helps manage team goals for the projects. We are able to create reports (via the dashboard) for the progress and updates to provide to the team based on completed stages. Works great for bigger projects.

Incentivized

• It performs a conventional disk-based process when the data sets are too large to fit into memory, which is very useful because, regardless of the size of the data, it is always possible to store them.
• It has great speed and ability to join multiple types of databases and run different types of analysis applications. This functionality is super useful as it reduces work times
• Apache Spark uses the data storage model of Hadoop and can be integrated with other big data frameworks such as HBase, MongoDB, and Cassandra. This is very useful because it is compatible with multiple frameworks that the company has, and thus allows us to unify all the processes.

Incentivized

• Data warehousing: Hive is often used as a data warehousing platform, allowing users to store and analyze large amounts of structured and semi-structured data. It is especially good at handling data that is too large to be stored and analyzed on a single machine, and supports a wide variety of data formats.
• Batch processing: Hive is designed for batch processing of large datasets, making it well-suited for tasks such as data ETL (extract, transform, load), data cleansing, and data aggregation.
• Data transformation: Hive allows users to perform data transformations and manipulations using custom scripts written in Java, Python, or other programming languages. This can be useful for tasks such as data cleansing, data aggregation, and data transformation.
• Integration with other tools: Hive integrates with a wide variety of other tools and services in the Hadoop ecosystem, such as Pig, Spark, and HBase, allowing users to perform a wide range of data analysis and management tasks.

Incentivized

• Memory management. Very weak on that.
• PySpark not as robust as scala with spark.
• spark master HA is needed. Not as HA as it should be.
• Locality should not be a necessity, but does help improvement. But would prefer no locality

Incentivized

• Organizing tasks by assignees could be better. It's a little cumbersome to check off each person you want. Can you group these?
• I don't really use any view besides task view. Is there something better I could be using?
• It would be nice if attachments showed up in a nicer format, maybe with a preview?

Incentivized

Capacity of computing data in cluster and fast speed.

If the team looking to use Apache Spark is not used to debug and tweak settings for jobs to ensure maximum optimizations, it can be frustrating. However, the documentation and the support of the community on the internet can help resolve most issues. Moreover, it is highly configurable and it integrates with different tools (eg: it can be used by dbt core), which increase the scenarios where it can be used

Incentivized

Its a easy tool, the best way to organize the workflow but has room for more improvements.

1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.

Our CSR is easily accessible and they have support built into the app itself. They also have a pretty robust support site. We also took advantage of the free trial and learned so much by putting Hive through the paces and figuring out the best way to mold it to our needs.

Incentivized

We used Surprise Kit for one of the other research works. It is more fine-tuned to Recommendation systems and their algorithms. Apache Spark has MLlib for majority of ML problems. Where as software like Surprse Kit - it suitable for a specific task of Recommendations only

Incentivized

One key difference between Hive and Spark is the way they process data. Hive is a batch-oriented system, which means that it is designed to process large amounts of data in a batch mode rather than in real-time. In contrast, Spark is a real-time processing platform that is designed to handle streaming data and support interactive queries. Another difference is the way they execute queries. Hive uses a SQL-like query language called HiveQL, while Spark supports a wide range of languages and APIs, including SQL, Python, Scala, and R. But we chose Hive due to its simple queries on large datasets and for data warehousing tasks.

Incentivized

• Faster turn around on feature development, we have seen a noticeable improvement in our agile development since using Spark.
• Easy adoption, having multiple departments use the same underlying technology even if the use cases are very different allows for more commonality amongst applications which definitely makes the operations team happy.
• Performance, we have been able to make some applications run over 20x faster since switching to Spark. This has saved us time, headaches, and operating costs.

Incentivized

• I've gotten to know my colleagues better, knowing their roles makes it faster to contact them to complete tasks and that speed makes us optimize and earn better results
• The jobs speed made us focus on optimization and customization for the client, and that in a better treatment by the client and better revenue
• We can understand which tasks takes more time and to stimate better what we can ask for

Incentivized