vs MapRedce, it was faster and easier to manage. Especially for Machine Learning, where MapReduce is lacking. Also Apache Storm was slower and didn't scale as much as Spark does. Spark elasticity was easier to apply compared to storm and MapReduce. managing resources for …
Well suited: To most of the local run of datasets and non-prod systems - scalability is not a problem at all. Including data from multiple types of data sources is an added advantage. MLlib is a decently nice built-in library that can be used for most of the ML tasks. Less appropriate: We had to work on a RecSys where the music dataset that we used was around 300+Gb in size. We faced memory-based issues. Few times we also got memory errors. Also the MLlib library does not have support for advanced analytics and deep-learning frameworks support. Understanding the internals of the working of Apache Spark for beginners is highly not possible.
K8s should be avoided - If your application works well without being converted into microservices-based architecture & fits correctly in a VM, needs less scaling, have a fixed traffic pattern then it is better to keep away from Kubernetes. Otherwise, the operational challenges & technical expertise will add a lot to the OPEX. Also, if you're the one who thinks that containers consume fewer resources as compared to VMs then this is not true. As soon as you convert your application to a microservice-based architecture, a lot of components will add up, shooting your resource consumption even higher than VMs so, please beware. Kubernetes is a good choice - When the application needs quick scaling, is already in microservice-based architecture, has no fixed traffic pattern, most of the employees already have desired skills.
Local development, Kubernetes does tend to be a bit complicated and unnecessary in environments where all development is done locally.
The need for add-ons, Helm is almost required when running Kubernetes. This brings a whole new tool to manage and learn before a developer can really start to use Kubernetes effectively.
Finicy configmap schemes. Kubernetes configmaps often have environment breaking hangups. The fail safes surrounding configmaps are sadly lacking.
The Kubernetes is going to be highly likely renewed as the technologies that will be placed on top of it are long term as of planning. There shouldn't be any last minute changes in the adoption and I do not anticipate sudden change of the core underlying technology. It is just that the slow process of technology adoption that makes it hard to switch to something else.
If the team looking to use Apache Spark is not used to debug and tweak settings for jobs to ensure maximum optimizations, it can be frustrating. However, the documentation and the support of the community on the internet can help resolve most issues. Moreover, it is highly configurable and it integrates with different tools (eg: it can be used by dbt core), which increase the scenarios where it can be used
It is an eminently usable platform. However, its popularity is overshadowed by its complexity. To properly leverage the capabilities and possibilities of Kubernetes as a platform, you need to have excellent understanding of your use case, even better understanding of whether you even need Kubernetes, and if yes - be ready to invest in good engineering support for the platform itself
1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.
Spark in comparison to similar technologies ends up being a one stop shop. You can achieve so much with this one framework instead of having to stitch and weave multiple technologies from the Hadoop stack, all while getting incredibility performance, minimal boilerplate, and getting the ability to write your application in the language of your choosing.
Most of the required features for any orchestration tool or framework, which is provided by Kubernetes. After understanding all modules and features of the K8S, it is the best fit for us as compared with others out there.
