Apache Spark Reviews

DataFrames, DataSets, and RDDs.

Spark has in-built Machine Learning library which scales and integrates with existing tools.

Cluster management for ETL.

Data processing engine for our data lake.

Very good tool to process big datasets.

Inbuilt fault tolerance.

Supports multiple languages.

Supports advanced analytics.

A large number of libraries available -- GraphX, Spark SQL, Spark Streaming, etc.

Customizable, it integrates with Jupyter notebooks which was really helpful for our team.

Easy to use and implement.

It allows us to quickly build microservices.

It is very fast.

It is gaining usability now that the PySpark community is growing and more functions are being developed.

Programmers can run different languages on the servers.

Apache Spark makes processing very large data sets possible. It handles these data sets in a fairly quick manner.

Apache Spark does a fairly good job implementing machine learning models for larger data sets.

Apache Spark seems to be a rapidly advancing software, with the new features making the software ever more straight-forward to use.

in memory data engine and hence faster processing

does well to lay on top of hadoop file system for big data analytics

very good tool for streaming data

It performs a conventional disk-based process when the data sets are too large to fit into memory, which is very useful because, regardless of the size of the data, it is always possible to store them.

It has great speed and ability to join multiple types of databases and run different types of analysis applications. This functionality is super useful as it reduces work times

Apache Spark uses the data storage model of Hadoop and can be integrated with other big data frameworks such as HBase, MongoDB, and Cassandra. This is very useful because it is compatible with multiple frameworks that the company has, and thus allows us to unify all the processes.

Rich APIs for data transformation making for very each to transform and prepare data in a distributed environment without worrying about memory issues

Faster in execution times compare to Hadoop and PIG Latin

Easy SQL interface to the same data set for people who are comfortable to explore data in a declarative manner

Interoperability between SQL and Scala / Python style of munging data

Machine Learning.

Data Analysis

WorkFlow process (faster than MapReduce).

SQL connector to multiple data sources

Easy ELT Process

Easy clustering on cloud

Amazing speed

Batch & real time processing

Spark uses Scala which is a functional programming language and easy to use language. Syntax is simpler and human readable.

It can be used to run transformations on huge data on different cluster parallelly. It automatically optimizes the process to get output efficiently in less time.

It also provides machine learning API for data science applications and also Spark SQL to query fast for data analysis.

I also use Zeppelin online tool which is used to fast query and very helpful for BI guys to visualize query outputs.

It makes the ETL process very simple when compared to SQL SERVER and MYSQL ETL tools.

It's very fast and has many machine learning algorithms which can be used for data science problems.

It is easily implemented on a cloud cluster.

Scale from local machine to full cluster. You can run a standalone, single cluster simply by starting up a Spark Shell or submitting an application to test an algorithm, then it quickly can be transferred and configured to run in a distributed environment.

Provides multiple APIs. Most people I know use Python and/or Java as their main programming language. Data scientists who are familiar with NumPy and SciPy can quickly become comfortable with Spark, while Java developers would best served using Java 8 and the new features that it provides. Scala, on the other hand, is a mix between the Java and Python styles of writing Spark code, in my opinion.

Plentiful learning resources. The Learning Spark book is a good introduction to the mechanics of Spark although written for Spark 1.3, and the current version is 2.0. The GitHub repository for the book contains all the code examples that are discussed, plus the Spark website is also filled with useful information that is simple to navigate.

Ease of use, the Spark API allows for minimal boilerplate and can be written in a variety of languages including Python, Scala, and Java.

Performance, for most applications we have found that jobs are more performant running via Spark than other distributed processing technologies like Map-Reduce, Hive, and Pig.

Flexibility, the frameworks comes with support for streaming, batch processing, sql queries, machine learning, etc. It can be used in a variety of applications without needing to integrate a lot of other distributed processing technologies.

We used to make our batch processing faster. Spark is faster in batch processing than MapReduce with it in memory computing

Spark will run along with other tools in the Hadoop ecosystem including Hive and Pig

Spark supports both batch and real-time processing

Apache Spark has Machine Learning Algorithms support

Great APIs and tools.

Scale.

Speed for iterative algorithms.