AWS Elastic Beanstalk vs. Databricks Data Intelligence Platform

I have been using AWS Elastic Beanstalk for more than 5 years, and it has made our life so easy and hassle-free. Here are some scenarios where it excels -

• I have been using different AWS services like EC2, S3, Cloudfront, Serverless, etc. And Elastic Beanstalk makes our lives easier by tieing each service together and making the deployment a smooth process.
• N number of integrations with different CI/CD pipelines make this most engineer's favourite service.
• Scalability & Security comes with the service, which makes it the absolute perfect product for your business.

Personally, I haven't found any situations where it's not appropriate for the use cases it can be used. The pricing is also very cost-effective.

Incentivized

Medium to Large data throughput shops will benefit the most from Databricks Spark processing. Smaller use cases may find the barrier to entry a bit too high for casual use cases. Some of the overhead to kicking off a Spark compute job can actually lead to your workloads taking longer, but past a certain point the performance returns cannot be beat.

Incentivized

• Getting a project set up using the console or CLI is easy compared to other [computing] platforms.
• AWS Elastic Beanstalk supports a variety of programming languages so teams can experiment with different frameworks but still use the same compute platform for rapid prototyping.
• Common application architectures can be referenced as patterns during project [setup].
• Multiple environments can be deployed for an application giving more flexibility for experimentation.

Incentivized

• Process raw data in One Lake (S3) env to relational tables and views
• Share notebooks with our business analysts so that they can use the queries and generate value out of the data
• Try out PySpark and Spark SQL queries on raw data before using them in our Spark jobs
• Modern day ETL operations made easy using Databricks. Provide access mechanism for different set of customers

Incentivized

• Limited to the frameworks and configurations that AWS supports. There is no native way to use Elastic Beanstalk to deploy a Go application behind Nginx, for example.
• It's not always clear what's changed on an underlying system when AWS updates an EB stack; the new version is announced, but AWS does not say what specifically changed in the underlying configuration. This can have unintended consequences and result in additional work in order to figure out what changes were made.

Incentivized

• Sometimes, when multiple jobs depend on each other in different environments, it is not always easy to see the full workflow in one place.
• It is sometimes difficult to determine which job or cluster contributes more to the overall cost.
• For beginners, cluster configuration may be a little difficult. So more recommendation in the platform can help.

Incentivized

As our technology grows, it makes more sense to individually provision each server rather than have it done via beanstalk. There are several reasons to do so, which I cannot explain without further diving into the architecture itself, but I can tell you this. With automation, you also loose the flexibility to morph the system for your specific needs. So if you expect that in future you need more customization to your deployment process, then there is a good chance that you might try to do things individually rather than use an automation like beanstalk.

Incentivized

The overall usability is good enough, as far as the scaling, interactive UI and logging system is concerned, could do a lot better when it comes to the efficiency, in case of complicated node logics and complicated node architectures. It can have better software compatibility and can try to support collaboration with more softwares

Incentivized

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

As I described earlier it has been really cost effective and really easy for fellow developers who don't want to waste weeks and weeks into learning and manually deploying stuff which basically takes month to create and go live with the Minimal viable product (MVP). With AWS Beanstalk within a week a developer can go live with the Minimal viable product easily.

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

- Do as many experiments as you can before you commit on using beanstalk or other AWS features. - Keep future state in mind. Think through what comes next, and if that is technically possible to do so. - Always factor in cost in terms of scaling. - We learned a valuable lesson when we wanted to go multi-region, because then we realized many things needs to change in code. So if you plan on using this a lot, factor multiple regions.

Incentivized

We also use Heroku and it is a great platform for smaller projects and light Node.js services, but we have found that in terms of cost, the Elastic Beanstalk option is more affordable for the projects that we undertake. The fact that it sits inside of the greater AWS Cloud offering also compels us to use it, since integration is simpler. We have also evaluated Microsoft Azure and gave up trying to get an extremely basic implementation up and running after a few days of struggling with its mediocre user interface and constant issues with documentation being outdated. The authentication model is also badly broken and trying to manage resources is a pain. One cannot compare Azure with anything that Amazon has created in the cloud space since Azure really isn't a mature platform and we are always left wanting when we have to interface with it.

Incentivized

The most important differentiating factor for Databricks Lakehouse Platform from these other platforms is support for ACID transactions and the time travel feature. Also, native integration with managed MLflow is a plus. EMR, Cloudera, and Hortonworks are not as optimized when it comes to Spark Job Execution. Other platforms need to be self-managed, which is another huge hassle.

Incentivized

• till now we had not Calculated ROI as the project is still evolving and we had to keep on changing the environment implementation
• it meets our purpose of quick deployment as compared to on-premises deployment
• till now we look good as we also controlled our expenses which increased suddenly in the middle of deployment activity

Incentivized

• The ability to spin up a BIG Data platform with little infrastructure overhead allows us to focus on business value not admin
• DB has the ability to terminate/time out instances which helps manage cost.
• The ability to quickly access typical hard to build data scenarios easily is a strength.

Incentivized