Dataiku is a French startup and its product, DSS, is a challenger to market incumbents and features some visual tools to assist in building workflows.
N/A
dbt
Score 9.4 out of 10
N/A
dbt is an SQL development environment, developed by Fishtown Analytics, now known as dbt Labs. The vendor states that with dbt, analysts take ownership of the entire analytics engineering workflow, from writing data transformation code to deployment and documentation. dbt Core is distributed under the Apache 2.0 license, and paid Teams and Enterprise editions are available.
Dataiku DSS is very well suited to handle large datasets and projects which requires a huge team to deliver results. This allows users to collaborate with each other while working on individual tasks. The workflow is easily streamlined and every action is backed up, allowing users to revert to specific tasks whenever required. While Dataiku DSS works seamlessly with all types of projects dealing with structured datasets, I haven't come across projects using Dataiku dealing with images/audio signals. But a workaround would be to store the images as vectors and perform the necessary tasks.
If you can load your data first into your warehouse, dbt is excellent. It does the T(ransformation) part of ELT brilliantly but does not do the E(xtract) or L(oad) part. If you know SQL or your development team knows SQL, it's a framework and extension around that. So, it's easy to learn and easy to hire people with that technical skill (as opposed to specific Informatica, SnapLogic, etc. experience). dbt uses plain text files and integrates with GitHub. You can easily see the changes made between versions. In GUI-based UIs it was always hard to tell what someone had changed. Each "model" is essentially a "SELECT" statement. You never need to do a "CREATE TABLE" or "CREATE VIEW" - it's all done for you, leaving you to work on the business logic. Instead of saying "FROM specific_db.schema.table" you indicate "FROM ref('my_other_model')". It creates an internal dependency diagram you can view in a DAG. When you deploy, the dependencies work like magic in your various environments. They also have great documentation, an active slack community, training, and support. I like the enhancements they have been making and I believe they are headed in a good direction.
Slow load times of the dbt cloud environment (they're working on it via a new UI though)
More out-of-the-box solutions for managing procedures, functions, etc would be nice to have, but honestly, it's pretty easy to figure out how to adapt dbt macros
As I have described earlier, the intuitiveness of this tool makes it great as well as the variety of users that can use this tool. Also, the plugins available in their repository provide solutions to various data science problems.
The support team is very helpful, and even when we discover the missing features, after providing enough rational reasons and requirements, they put into it their development pipeline for the future release.
Strictly for Data Science operations, Anaconda can be considered as a subset of Dataiku DSS. While Anaconda supports Python and R programming languages, Dataiku also provides this facility, but also provides GUI to creates models with just a click of a button. This provides the flexibility to users who do not wish to alter the model hyperparameters in greater depths. Writing codes to extract meaningful data is time consuming compared to Dataiku's ability to perform feature engineering and data transformation through click of a button.
Most ETL pipeline products have a T layer, but dbt just does it better. The transformation is on steroids compared to the others. Also, just allows much more Adhoc solutions for very specific projects. Those ETL tools are probably better on the T part if you don't need too many transforms - also dbt is pretty much free dependent on how you work it, also extremely scalable.
