Apache Airflow vs. Collibra Platform

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

Collibra is well suited where you have multiple reporting environments and multiple source systems. Collibra works well in our environment because we can delegate roles and administration to departments or it can be used in a centralized environment. The ability to customize attributes and assets as well and integrate using the rest api is very important to us.

Incentivized

• In charge of the ETL processes.
• As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.

Incentivized

• Data Lineage and traceability
• Easy to customize
• Easy tool to use for business users
• Workflow
• Out of the box metrics

Incentivized

• they should bring in some time based scheduling too not only event based
• they do not store the metadata due to which we are not able to analyze the workflows
• they only support python as of now for scripted pipeline writing

Incentivized

• Data Quality and Data Profiling
• Mobile interface

Incentivized

Easy to learn Easy to use Robust workflow orchestration framework Good in dependent job management

Incentivized

There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.

Incentivized

Collibra offers more features [than alternatives] and is easy for business users to use. We did a proof of concept with Collibra using the Cloud service that allowed us to kick the tires and get a comfort level before we made the investment. At the time of our purchase the market was very immature and is continuing to evolve.

Incentivized

• A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
• Allowed us to implement complex data pipelines easily and at a relatively low cost

Incentivized

• ROI is hard to measure because there are so many soft benefits that come from the tool. We do have a variety of metrics that we track using Collibra to better gauge the maturity of data governance.

Incentivized