AWS Elastic Beanstalk vs. dbt

I have been using AWS Elastic Beanstalk for more than 5 years, and it has made our life so easy and hassle-free. Here are some scenarios where it excels -

• I have been using different AWS services like EC2, S3, Cloudfront, Serverless, etc. And Elastic Beanstalk makes our lives easier by tieing each service together and making the deployment a smooth process.
• N number of integrations with different CI/CD pipelines make this most engineer's favourite service.
• Scalability & Security comes with the service, which makes it the absolute perfect product for your business.

Personally, I haven't found any situations where it's not appropriate for the use cases it can be used. The pricing is also very cost-effective.

Incentivized

The prerequisite is that you have a supported database/data warehouse and have already found a way to ingest your raw data. Then dbt is very well suited to manage your transformation logic if the people using it are familiar with SQL. If you want to benefit from bringing engineering practices to data, dbt is a great fit. It can bring CI/CD practices, version control, automated testing, documentation generation, etc. It is not so well suited if the people managing the transformation logic do not like to code (in SQL) but prefer graphical user interfaces.

Incentivized

• Getting a project set up using the console or CLI is easy compared to other [computing] platforms.
• AWS Elastic Beanstalk supports a variety of programming languages so teams can experiment with different frameworks but still use the same compute platform for rapid prototyping.
• Common application architectures can be referenced as patterns during project [setup].
• Multiple environments can be deployed for an application giving more flexibility for experimentation.

Incentivized

• dbt supports version control through GIT, this allows teams to collaborate and track the data transformation logic.
• dbt allows us to build data models which helps to break complex transformation logic into simple and smaller logic.
• dbt is completely based on SQL which allows data analyst and data engineers to build the transformation logic.
• dbt can be easily integrated with snowflake.

Incentivized

• Limited to the frameworks and configurations that AWS supports. There is no native way to use Elastic Beanstalk to deploy a Go application behind Nginx, for example.
• It's not always clear what's changed on an underlying system when AWS updates an EB stack; the new version is announced, but AWS does not say what specifically changed in the underlying configuration. This can have unintended consequences and result in additional work in order to figure out what changes were made.

Incentivized

• Field-level lineage (currently at table level)
• Documentation inheritance - if a field is documented the downstream field of the same name could inherit the doc info
• Adding python model support (in beta now)

Incentivized

As our technology grows, it makes more sense to individually provision each server rather than have it done via beanstalk. There are several reasons to do so, which I cannot explain without further diving into the architecture itself, but I can tell you this. With automation, you also loose the flexibility to morph the system for your specific needs. So if you expect that in future you need more customization to your deployment process, then there is a good chance that you might try to do things individually rather than use an automation like beanstalk.

Incentivized

The overall usability is good enough, as far as the scaling, interactive UI and logging system is concerned, could do a lot better when it comes to the efficiency, in case of complicated node logics and complicated node architectures. It can have better software compatibility and can try to support collaboration with more softwares

Incentivized

dbt is very easy to use. Basically if you can write SQL, you will be able to use dbt to get what you need done. Of course more advanced users with more technical skills can do more things.

Incentivized

As I described earlier it has been really cost effective and really easy for fellow developers who don't want to waste weeks and weeks into learning and manually deploying stuff which basically takes month to create and go live with the Minimal viable product (MVP). With AWS Beanstalk within a week a developer can go live with the Minimal viable product easily.

Incentivized

- Do as many experiments as you can before you commit on using beanstalk or other AWS features. - Keep future state in mind. Think through what comes next, and if that is technically possible to do so. - Always factor in cost in terms of scaling. - We learned a valuable lesson when we wanted to go multi-region, because then we realized many things needs to change in code. So if you plan on using this a lot, factor multiple regions.

Incentivized

We also use Heroku and it is a great platform for smaller projects and light Node.js services, but we have found that in terms of cost, the Elastic Beanstalk option is more affordable for the projects that we undertake. The fact that it sits inside of the greater AWS Cloud offering also compels us to use it, since integration is simpler. We have also evaluated Microsoft Azure and gave up trying to get an extremely basic implementation up and running after a few days of struggling with its mediocre user interface and constant issues with documentation being outdated. The authentication model is also badly broken and trying to manage resources is a pain. One cannot compare Azure with anything that Amazon has created in the cloud space since Azure really isn't a mature platform and we are always left wanting when we have to interface with it.

Incentivized

I actually don't know what the alternative to dbt is. I'm sure one must exist other than more 'roll your own' options like Apache Airflow, say, bu tin terms of super easy managed/cloud data transforms, dbt really does seem to be THE tool to use. It's $50/month per dev, BUT there's a FREE version for 1 dev seat with no read-only access for anyone else, so you can always start with that and then buy yourself a seat later.

Incentivized

• till now we had not Calculated ROI as the project is still evolving and we had to keep on changing the environment implementation
• it meets our purpose of quick deployment as compared to on-premises deployment
• till now we look good as we also controlled our expenses which increased suddenly in the middle of deployment activity

Incentivized

• Simplified our BI layer for faster load times
• Increased the quality of data reaching our end users
• Makes complex transformations manageable

Incentivized