dbt vs. IBM DataStage

If you can load your data first into your warehouse, dbt is excellent. It does the T(ransformation) part of ELT brilliantly but does not do the E(xtract) or L(oad) part. If you know SQL or your development team knows SQL, it's a framework and extension around that. So, it's easy to learn and easy to hire people with that technical skill (as opposed to specific Informatica, SnapLogic, etc. experience). dbt uses plain text files and integrates with GitHub. You can easily see the changes made between versions. In GUI-based UIs it was always hard to tell what someone had changed. Each "model" is essentially a "SELECT" statement. You never need to do a "CREATE TABLE" or "CREATE VIEW" - it's all done for you, leaving you to work on the business logic. Instead of saying "FROM specific_db.schema.table" you indicate "FROM ref('my_other_model')". It creates an internal dependency diagram you can view in a DAG. When you deploy, the dependencies work like magic in your various environments. They also have great documentation, an active slack community, training, and support. I like the enhancements they have been making and I believe they are headed in a good direction.

Incentivized

Excellent Cloud data mapping tool and easy creating multiple project data analytics in real-time and the report distribution are excellent via this IBM product. Easy tool to provide data visualization and the integration is effective and helpful to migrating huge amounts of data across other platforms and different websites insights gathering.

Incentivized

• user experience makes it easy to work with SQL and version control
• customer success team and the dbt (data build tool) community help establish best practices
• thorough and clear documentation

• Data movement
• Seamless integration of scripts and etl jobs
• Descriptive logging
• Ability to work with myriad of data assets
• Direct integration for Governance catalog

Incentivized

• Slow load times of the dbt cloud environment (they're working on it via a new UI though)
• More out-of-the-box solutions for managing procedures, functions, etc would be nice to have, but honestly, it's pretty easy to figure out how to adapt dbt macros

Incentivized

• Connector Stages to Snowflake on the cloud. We had some issues initially but since then had been corrected.
• Accessing tool from a browser (zero foot-print). Currently we need to either install locally or connect to a server to do ETL work.
• Diversify ways of authenticating users.

Incentivized

Because it is robust, and it is being continuously improved. DS is one of the most used and recognized tools in the market. Large companies have implemented it in the first instance to develop their DW, but finding the advantages it has, they could use it for other types of projects such as migrations, application feeding, etc.

Incentivized

It could load thousands of records in seconds. But in the Parallel version, you need to understand how to particionate the data. If you use the algorithms erroneously, or the functionalities that it gives for the parsing of data, the performance can fall drastically, even with few records. It is necessary to have people with experience to be able to determine which algorithm to use and understand why.

Incentivized

I believe that IBM generally has one of the worst and most complex assistance systems (physical and online) that exists.

Incentivized

Most ETL pipeline products have a T layer, but dbt just does it better. The transformation is on steroids compared to the others. Also, just allows much more Adhoc solutions for very specific projects. Those ETL tools are probably better on the T part if you don't need too many transforms - also dbt is pretty much free dependent on how you work it, also extremely scalable.

Incentivized

It's obvious since they both are from the same vendors and it makes it easier and can get better rates for licensing. Also, sales rapes are very helpful in case of escalations and critical issues.

Incentivized

• Simplified our BI layer for faster load times
• Increased the quality of data reaching our end users
• Makes complex transformations manageable

Incentivized

• Reduce development time by 65% compared with hand coding.
• Reduces ETL process maintenance times.
• Better data governance for technical and non-technical people.
• Improve time to market for initiatives that require data integration.

Incentivized