dbt vs. HCL DataConnect

If you can load your data first into your warehouse, dbt is excellent. It does the T(ransformation) part of ELT brilliantly but does not do the E(xtract) or L(oad) part. If you know SQL or your development team knows SQL, it's a framework and extension around that. So, it's easy to learn and easy to hire people with that technical skill (as opposed to specific Informatica, SnapLogic, etc. experience). dbt uses plain text files and integrates with GitHub. You can easily see the changes made between versions. In GUI-based UIs it was always hard to tell what someone had changed. Each "model" is essentially a "SELECT" statement. You never need to do a "CREATE TABLE" or "CREATE VIEW" - it's all done for you, leaving you to work on the business logic. Instead of saying "FROM specific_db.schema.table" you indicate "FROM ref('my_other_model')". It creates an internal dependency diagram you can view in a DAG. When you deploy, the dependencies work like magic in your various environments. They also have great documentation, an active slack community, training, and support. I like the enhancements they have been making and I believe they are headed in a good direction.

Incentivized

The choice to use Actian DataConnect really comes down to pricing and the type of integration project you are performing, specifically with the requirements, whether you want the integration to be on-premise or in the cloud, can you install agents on your servers etc. For integration tools it runs head to head with its competitors.

Incentivized

• user experience makes it easy to work with SQL and version control
• customer success team and the dbt (data build tool) community help establish best practices
• thorough and clear documentation

• Map multiple source and destination tables - what would be done in many steps in SSIS can be done in one or two "maps" in Data Integrator.
• Platform independent, cloud based, no client software.
• Conversion of data types, write flat files - this helps us with integrating systems.
• Fairly straightforward to use if the user has ETL experience.

Incentivized

• Slow load times of the dbt cloud environment (they're working on it via a new UI though)
• More out-of-the-box solutions for managing procedures, functions, etc would be nice to have, but honestly, it's pretty easy to figure out how to adapt dbt macros

Incentivized

• Integration Manager needs better scheduling improvements. Like dependencies, and not running a process again if the prior run is still running.
• New connector development does not seem a priority.

Incentivized

Most ETL pipeline products have a T layer, but dbt just does it better. The transformation is on steroids compared to the others. Also, just allows much more Adhoc solutions for very specific projects. Those ETL tools are probably better on the T part if you don't need too many transforms - also dbt is pretty much free dependent on how you work it, also extremely scalable.

Incentivized

Data Integrator was initially a part of a package deal for us with our new Student Information System. It was chosen prior to my arrival at the institution. Since I discovered we have SQL Server licenses, we have decided to use SSIS instead since I have a background in the MSBI stack and SSIS appears to be more straightforward (process flow philosophy vs "map" philosophy.)

Incentivized

• Simplified our BI layer for faster load times
• Increased the quality of data reaching our end users
• Makes complex transformations manageable

Incentivized

• Our customer likes the consistent reliable integrations we have built so we [got] a definite return on investment.
• Problems are fairly easy to handle and rework.

Incentivized