Well suited: To most of the local run of datasets and non-prod systems - scalability is not a problem at all. Including data from multiple types of data sources is an added advantage. MLlib is a decently nice built-in library that can be used for most of the ML tasks. Less appropriate: We had to work on a RecSys where the music dataset that we used was around 300+Gb in size. We faced memory-based issues. Few times we also got memory errors. Also the MLlib library does not have support for advanced analytics and deep-learning frameworks support. Understanding the internals of the working of Apache Spark for beginners is highly not possible.
If you have a cluster of nodes with MariaDB MaxScale and you want all the nodes of the cluster to have a similar load and not to be be penalized for queries or writes to the database, you can mount the MaxScale product in front of the MariaDB cluster. MaxScale will balance the requests based on what is being sent to each node to have an equitable load and will cache the queries that it sends to each one. This optimizes the response time to database queries and spreads the load out among all nodes in a similar way.
If the team looking to use Apache Spark is not used to debug and tweak settings for jobs to ensure maximum optimizations, it can be frustrating. However, the documentation and the support of the community on the internet can help resolve most issues. Moreover, it is highly configurable and it integrates with different tools (eg: it can be used by dbt core), which increase the scenarios where it can be used
MariaDB MaxScale is a powerful tool and easy to use. It has helped us a lot to improve the performance of our database queries. It implements a security layer that acts as a firewall for the databases, masks the data, or limits the results of the queries. It also integrates easily with Kafka.
1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.
We have launched several inquiries to MariaDB MaxScale support, and they have always responded very quickly. They also want to hold frequent meetings with the client to get their opinion understand how they can help. I see a very human support that is concerned about the customer.
Spark in comparison to similar technologies ends up being a one stop shop. You can achieve so much with this one framework instead of having to stitch and weave multiple technologies from the Hadoop stack, all while getting incredibility performance, minimal boilerplate, and getting the ability to write your application in the language of your choosing.
ProxySLQ allows many simultaneous connections and allows the cache of queries in memory but it does not have high availability or scalability natively, only through external tools. HAProxy is not able to perform load balancing in an optimal way.
Instead, MariaDB MaxScale allows high availability, scalability, and data replication to external systems such as Kafka. In addition, MaxScale has a monitor that allows you to see the status of the set of databases.
