dbt

Dbt is a revelation for the analytics/BI engineering space. The seamless version control and 'don't repeat yourself' tools make your data and analytics pipelines WAY more reliable and efficient. Build data like a developer!

If you can load your data first into your warehouse, dbt is excellent. It does the T(ransformation) part of ELT brilliantly but does not do the E(xtract) or L(oad) part. If you know SQL or your development team knows SQL, it's a framework and extension around that. So, it's easy to learn and easy to hire people with that technical skill (as opposed to specific Informatica, SnapLogic, etc. experience). dbt uses plain text files and integrates with GitHub. You can easily see the changes made between versions. In GUI-based UIs it was always hard to tell what someone had changed. Each "model" is essentially a "SELECT" statement. You never need to do a "CREATE TABLE" or "CREATE VIEW" - it's all done for you, leaving you to work on the business logic. Instead of saying "FROM specific_db.schema.table" you indicate "FROM ref('my_other_model')". It creates an internal dependency diagram you can view in a DAG. When you deploy, the dependencies work like magic in your various environments. They also have great documentation, an active slack community, training, and support. I like the enhancements they have been making and I believe they are headed in a good direction.

DBT seems to work well when your data needs arise from your production environment. The IDE allows for integration with a GitHub repository but the current setup makes it a little complicated if you need to develop in other environments for system integration & user acceptance testing. However, the tool does perform its duties well & works with current modern tools such as Snowflake.

I use dbt on the T part of ETL in my data tech stack. It is amazing and also does the change control quite well. So, to summarize - ingest, do a multitude of transforms, spit out to data mart. Allows various business logic applications to data to happen simultaneously and well tracked for the data marts. Not very well suited so a single data mart, low scale data volume.

dbt (data build tool) has the capability to make your data models more accessible; other teams can read documentation, follow along the lineage, and even collaborate to make changes themselves dbt (data build tool) also has the capability to easily increase your database cost and write complex data models. The key to mitigating this risk is to adhere to best practices from the community and within your organization. Look to your data engineering teams to help guide scalable and efficient dbt (data build tool) processes and listen to your analysts for building well-documented and reusable data models.

It is great for transforming data already loaded into a cloud data warehouse

Very well suited for Data Engineering and Analytics Engineering. Works very well in modern data stacks with other tools such as Fivetran and Snowflake. Not appropriate for teams with little SQL experience or who require no-code analytic/engineering options.