How does Apache Spark perform against competing tools? I think Apache Spark does well in processing large volumes of data. The machine learning models also seem to be easier to program and interpret. With that said, the programming side of Apache Spark seems more difficult …
We are using MySQL as SaaS that is readily available to us in the cloud. In terms of ease of use, it is comparable and sometimes even easier with the available detailed user guides. I am guessing the vast presence of MySQL on the internet is because of its open source legacy.
Well suited: To most of the local run of datasets and non-prod systems - scalability is not a problem at all. Including data from multiple types of data sources is an added advantage. MLlib is a decently nice built-in library that can be used for most of the ML tasks. Less appropriate: We had to work on a RecSys where the music dataset that we used was around 300+Gb in size. We faced memory-based issues. Few times we also got memory errors. Also the MLlib library does not have support for advanced analytics and deep-learning frameworks support. Understanding the internals of the working of Apache Spark for beginners is highly not possible.
From my own perspective and the tasks that I perform on a daily basis, MySQL is perfect. It has a reasonable footprint, is fast enough and offers the security and flexibility I need. Everyone has their preferred applications and, no doubt, for larger data warehouses or more intensive applications, MySQL may have its limits, but for the area that I operate in, it's a great match.
Security: is embedded at each level in MySQL. Authentication mechanisms are in place for configuring user access and even service account access to applications. MySQL is secure enough under the hood to store your sensitive information. Also, additional plugins are available that sit on top of MySQL for even tighter security.
Widely adopted: MySQL is used across the industry and is trusted the most. Therefore, if you face any problems, simply Google it and you shall land in plenty of forums. This is a great relief as when you are in a need of help, you can find it right in your browser.
Lightweight application: MySQL is not a heavy application. However, the data you store in the database can get heavy with time, but as in the configuration and MySql application files, those are not very heavy and can easily be installed on legacy systems as well.
Although you can add the data you require as more and more data is added, the fixity of it becomes more critical.
As the demand, size, and use of the system increase, you may also need to change or acquire more equipment on your servers, although this is an internal inconvenience for the company.
For teaching Databases and SQL, I would definitely continue to use MySQL. It provides a good, solid foundation to learn about databases. Also to learn about the SQL language and how it works with the creation, insertion, deletion, updating, and manipulation of data, tables, and databases. This SQL language is a foundation and can be used to learn many other database related concepts.
The only thing I dislike about spark's usability is the learning curve, there are many actions and transformations, however, its wide-range of uses for ETL processing, facility to integrate and it's multi-language support make this library a powerhouse for your data science solutions. It has especially aided us with its lightning-fast processing times.
I give MySQL a 9/10 overall because I really like it but I feel like there are a lot of tech people who would hate it if I gave it a 10/10. I've never had any problems with it or reached any of its limitations but I know a few people who have so I can't give it a 10/10 based on those complaints.
1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.
The support staff is friendly, knowledgeable, and efficient. I only had to get part way through my explanations before they had a solution. They will walk you through a fix or actually connect in and fix the problem for you--or would if you can allow it. I've done it both ways with them. They are always forthcoming with 'how to do this if it happens again' information. I love working with MySQL support.
All the above systems work quite well on big data transformations whereas Spark really shines with its bigger API support and its ability to read from and write to multiple data sources. Using Spark one can easily switch between declarative versus imperative versus functional type programming easily based on the situation. Also it doesn't need special data ingestion or indexing pre-processing like Presto. Combining it with Jupyter Notebooks (https://github.com/jupyter-incubator/sparkmagic), one can develop the Spark code in an interactive manner in Scala or Python
MongoDB has a dynamic schema for how data is stored in 'documents' whereas MySQL is more structured with tables, columns, and rows. MongoDB was built for high availability whereas MySQL can be a challenge when it comes to replication of the data and making everything redundant in the event of a DR or outage.
Faster turn around on feature development, we have seen a noticeable improvement in our agile development since using Spark.
Easy adoption, having multiple departments use the same underlying technology even if the use cases are very different allows for more commonality amongst applications which definitely makes the operations team happy.
Performance, we have been able to make some applications run over 20x faster since switching to Spark. This has saved us time, headaches, and operating costs.
