Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. And FlinkCEP is the Complex Event Processing (CEP) library implemented on top of Flink. Users can detect event patterns in streams of events.
N/A
AWS Lambda
Score 8.8 out of 10
N/A
AWS Lambda is a serverless computing platform that lets users run code without provisioning or managing servers. With Lambda, users can run code for virtually any type of app or backend service—all with zero administration. It takes of requirements to run and scale code with high availability.
In well-suited scenarios, I would recommend using Apache Flink when you need to perform real-time analytics on streaming data, such as monitoring user activities, analyzing IoT device data, or processing financial transactions in real-time. It is also a good choice in scenarios where fault tolerance and consistency are crucial. I would not recommend it for simple batch processing pipelines or for teams that aren't experienced, as it might be overkill, and the steep learning curve may not justify the investment.
[AWS Lambda] is very well suited for the projects that doesn't have any infra but needs it where short running processes are required. But if your application need to run continuously than this might not be the very apt tool for you.
Lambda provides multiple methods for triggering functions, this includes AWS resources and services and external triggers like APIs and CLI calls.
The compute provided my Lambda is largely hands off for operations teams. Once the function is deployed, the management overhead is minimal since there are no servers to maintain.
Lambda's pricing can be very cost effective given that users are only charged for the time the function runs and associated costs like network or storage if those are used. A function that executes quickly and is not called often can cost next to nothing.
Python/SQL API, since both are relatively new, still misses a few features in comparison with the Java/Scala option
Steep Learning Curve, it's documentation could be improved to something more user-friendly, and it could also discuss more theoretical concepts than just coding
Putting a significant portion of your codebase into AWS Lambda and taking advantage of the high level of integration with other AWS services comes with the risk of vendor lock-in.
While the AWS Lambda environment is "not your problem," it's also not at your disposal to extend or modify, nor does it preserve state between function executions.
AWS Lambda functions are subject to strict time limitations, and will be aborted if they exceed five minutes of execution time. This can be a problem for some longer-running tasks that are otherwise well-suited to serverless delivery.
I give it a seven is usability because it's AWS. Their UI's are always clunkier than the competition and their documentation is rather cumbersome. There's SO MUCH to dig through and it's a gamble if you actually end up finding the corresponding info if it will actually help. Like I said before, going to google with a specific problem is likely a better route because AWS is quite ubiquitous and chances are you're not the first to encounter the problem. That being said, using SAM (Serverless application model) and it's SAM Local environment makes running local instances of your Lambdas in dev environments painless and quite fun. Using Nodejs + Lambda + SAM Local + VS Code debugger = AWESOME.
I have not needed support for AWS Lambda, since it is already using Python, which has resources all over the internet. AWS blog posts have information about how to install some libraries, which is necessary for some more complex operations, but this is available online and didn't require specific customer support for.
Apache Spark is more user-friendly and features higher-level APIs. However, it was initially built for batch processing and only more recently gained streaming capabilities. In contrast, Apache Flink processes streaming data natively. Therefore, in terms of low latency and fault tolerance, Apache Flink takes the lead. However, Spark has a larger community and a decidedly lower learning curve.
Azure Functions is another product that provides lambda functionality, but the documentation for some of Azure's products is quite hard to read. Additionally, AWS Lambda was one of the first cloud computing products on a large cloud service that implemented lambda functions, so they have had the most time to develop the product, increase the quality of service, and extend functionality to more languages. Amazon, by far, has the best service for Lambda that I know.
I was able to perform a lot of processing on data delivered from my website and little or no cost. This was a big plus to me.
Programming AWS Lambda is quite easy once you understand the time limits to the functions.
AWS Lambda has really good integration with the AWS S3 storage system. This a very good method of delivering data to be processed and a good place to pick it up after processing.
