Apache Spark vs. AWS Lambda

Well suited: To most of the local run of datasets and non-prod systems - scalability is not a problem at all. Including data from multiple types of data sources is an added advantage. MLlib is a decently nice built-in library that can be used for most of the ML tasks. Less appropriate: We had to work on a RecSys where the music dataset that we used was around 300+Gb in size. We faced memory-based issues. Few times we also got memory errors. Also the MLlib library does not have support for advanced analytics and deep-learning frameworks support. Understanding the internals of the working of Apache Spark for beginners is highly not possible.

Incentivized

[AWS Lambda] is very well suited for the projects that doesn't have any infra but needs it where short running processes are required. But if your application need to run continuously than this might not be the very apt tool for you.

Incentivized

• Apache Spark makes processing very large data sets possible. It handles these data sets in a fairly quick manner.
• Apache Spark does a fairly good job implementing machine learning models for larger data sets.
• Apache Spark seems to be a rapidly advancing software, with the new features making the software ever more straight-forward to use.

Incentivized

• Lambda provides multiple methods for triggering functions, this includes AWS resources and services and external triggers like APIs and CLI calls.
• The compute provided my Lambda is largely hands off for operations teams. Once the function is deployed, the management overhead is minimal since there are no servers to maintain.
• Lambda's pricing can be very cost effective given that users are only charged for the time the function runs and associated costs like network or storage if those are used. A function that executes quickly and is not called often can cost next to nothing.

Incentivized

• Memory management. Very weak on that.
• PySpark not as robust as scala with spark.
• spark master HA is needed. Not as HA as it should be.
• Locality should not be a necessity, but does help improvement. But would prefer no locality

Incentivized

• Putting a significant portion of your codebase into AWS Lambda and taking advantage of the high level of integration with other AWS services comes with the risk of vendor lock-in.
• While the AWS Lambda environment is "not your problem," it's also not at your disposal to extend or modify, nor does it preserve state between function executions.
• AWS Lambda functions are subject to strict time limitations, and will be aborted if they exceed five minutes of execution time. This can be a problem for some longer-running tasks that are otherwise well-suited to serverless delivery.

Incentivized

Capacity of computing data in cluster and fast speed.

The only thing I dislike about spark's usability is the learning curve, there are many actions and transformations, however, its wide-range of uses for ETL processing, facility to integrate and it's multi-language support make this library a powerhouse for your data science solutions. It has especially aided us with its lightning-fast processing times.

Incentivized

I give it a seven is usability because it's AWS. Their UI's are always clunkier than the competition and their documentation is rather cumbersome. There's SO MUCH to dig through and it's a gamble if you actually end up finding the corresponding info if it will actually help. Like I said before, going to google with a specific problem is likely a better route because AWS is quite ubiquitous and chances are you're not the first to encounter the problem. That being said, using SAM (Serverless application model) and it's SAM Local environment makes running local instances of your Lambdas in dev environments painless and quite fun. Using Nodejs + Lambda + SAM Local + VS Code debugger = AWESOME.

Incentivized

1. It integrates very well with scala or python. 2. It's very easy to understand SQL interoperability. 3. Apache is way faster than the other competitive technologies. 4. The support from the Apache community is very huge for Spark. 5. Execution times are faster as compared to others. 6. There are a large number of forums available for Apache Spark. 7. The code availability for Apache Spark is simpler and easy to gain access to. 8. Many organizations use Apache Spark, so many solutions are available for existing applications.

I have not needed support for AWS Lambda, since it is already using Python, which has resources all over the internet. AWS blog posts have information about how to install some libraries, which is necessary for some more complex operations, but this is available online and didn't require specific customer support for.

Incentivized

All the above systems work quite well on big data transformations whereas Spark really shines with its bigger API support and its ability to read from and write to multiple data sources. Using Spark one can easily switch between declarative versus imperative versus functional type programming easily based on the situation. Also it doesn't need special data ingestion or indexing pre-processing like Presto. Combining it with Jupyter Notebooks (https://github.com/jupyter-incubator/sparkmagic), one can develop the Spark code in an interactive manner in Scala or Python

Incentivized

Azure Functions is another product that provides lambda functionality, but the documentation for some of Azure's products is quite hard to read. Additionally, AWS Lambda was one of the first cloud computing products on a large cloud service that implemented lambda functions, so they have had the most time to develop the product, increase the quality of service, and extend functionality to more languages. Amazon, by far, has the best service for Lambda that I know.

Incentivized

• Faster turn around on feature development, we have seen a noticeable improvement in our agile development since using Spark.
• Easy adoption, having multiple departments use the same underlying technology even if the use cases are very different allows for more commonality amongst applications which definitely makes the operations team happy.
• Performance, we have been able to make some applications run over 20x faster since switching to Spark. This has saved us time, headaches, and operating costs.

Incentivized

• I was able to perform a lot of processing on data delivered from my website and little or no cost. This was a big plus to me.
• Programming AWS Lambda is quite easy once you understand the time limits to the functions.
• AWS Lambda has really good integration with the AWS S3 storage system. This a very good method of delivering data to be processed and a good place to pick it up after processing.

Incentivized