Apache Airflow vs. Azure Databricks

Overview
ProductRatingMost Used ByProduct SummaryStarting Price
Apache Airflow
Score 8.6 out of 10
N/A
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.N/A
Azure Databricks
Score 8.4 out of 10
N/A
Azure Databricks is a service available on Microsoft's Azure platform and suite of products. It provides the latest versions of Apache Spark so users can integrate with open source libraries, or spin up clusters and build in a fully managed Apache Spark environment with the global scale and availability of Azure. Clusters are set up, configured, and fine-tuned to ensure reliability and performance without the need for monitoring. The solution includes autoscaling and auto-termination to improve…N/A
Pricing
Apache AirflowAzure Databricks
Editions & Modules
No answers on this topic
No answers on this topic
Offerings
Pricing Offerings
Apache AirflowAzure Databricks
Free Trial
NoNo
Free/Freemium Version
YesNo
Premium Consulting/Integration Services
NoNo
Entry-level Setup FeeNo setup feeNo setup fee
Additional Details——
More Pricing Information
Community Pulse
Apache AirflowAzure Databricks
Top Pros

No answers on this topic

Top Cons

No answers on this topic

Features
Apache AirflowAzure Databricks
Workload Automation
Comparison of Workload Automation features of Product A and Product B
Apache Airflow
9.7
10 Ratings
16% above category average
Azure Databricks
-
Ratings
Multi-platform scheduling9.910 Ratings00 Ratings
Central monitoring9.910 Ratings00 Ratings
Logging9.910 Ratings00 Ratings
Alerts and notifications9.810 Ratings00 Ratings
Analysis and visualization9.810 Ratings00 Ratings
Application integration9.010 Ratings00 Ratings
Platform Connectivity
Comparison of Platform Connectivity features of Product A and Product B
Apache Airflow
-
Ratings
Azure Databricks
8.6
2 Ratings
3% above category average
Connect to Multiple Data Sources00 Ratings7.82 Ratings
Extend Existing Data Sources00 Ratings9.02 Ratings
Automatic Data Format Detection00 Ratings9.42 Ratings
MDM Integration00 Ratings8.01 Ratings
Data Exploration
Comparison of Data Exploration features of Product A and Product B
Apache Airflow
-
Ratings
Azure Databricks
5.4
2 Ratings
43% below category average
Visualization00 Ratings5.12 Ratings
Interactive Data Analysis00 Ratings5.72 Ratings
Data Preparation
Comparison of Data Preparation features of Product A and Product B
Apache Airflow
-
Ratings
Azure Databricks
8.2
2 Ratings
0% above category average
Interactive Data Cleaning and Enrichment00 Ratings7.02 Ratings
Data Transformations00 Ratings8.62 Ratings
Data Encryption00 Ratings9.42 Ratings
Built-in Processors00 Ratings7.92 Ratings
Platform Data Modeling
Comparison of Platform Data Modeling features of Product A and Product B
Apache Airflow
-
Ratings
Azure Databricks
8.6
2 Ratings
2% above category average
Multiple Model Development Languages and Tools00 Ratings8.92 Ratings
Automated Machine Learning00 Ratings8.62 Ratings
Single platform for multiple model development00 Ratings8.42 Ratings
Self-Service Model Delivery00 Ratings8.42 Ratings
Model Deployment
Comparison of Model Deployment features of Product A and Product B
Apache Airflow
-
Ratings
Azure Databricks
8.7
2 Ratings
2% above category average
Flexible Model Publishing Options00 Ratings8.02 Ratings
Security, Governance, and Cost Controls00 Ratings9.42 Ratings
Best Alternatives
Apache AirflowAzure Databricks
Small Businesses

No answers on this topic

Jupyter Notebook
Jupyter Notebook
Score 9.2 out of 10
Medium-sized Companies
ActiveBatch Workload Automation
ActiveBatch Workload Automation
Score 8.1 out of 10
Posit
Posit
Score 9.8 out of 10
Enterprises
Control-M
Control-M
Score 9.3 out of 10
Posit
Posit
Score 9.8 out of 10
All AlternativesView all alternativesView all alternatives
User Ratings
Apache AirflowAzure Databricks
Likelihood to Recommend
9.0
(10 ratings)
9.1
(3 ratings)
Usability
10.0
(1 ratings)
8.0
(1 ratings)
User Testimonials
Apache AirflowAzure Databricks
Likelihood to Recommend
Apache
For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.
Read full review
Microsoft
Suppose you have multiple data sources and you want to bring the data into one place, transform it and make it into a data model. Azure Databricks is a perfectly suited solution for this. Leverage spark JDBC or any external cloud based tool (ADG, AWS Glue) to bring the data into a cloud storage. From there, Azure Databricks can handle everything. The data can be ingested by Azure Databricks into a 3 Layer architecture based on the delta lake tables. The first layer, raw layer, has the raw as is data from source. The enrich layer, acts as the cleaning and filtering layer to clean the data at an individual table level. The gold layer, is the final layer responsible for a data model. This acts as the serving layer for BI For BI needs, if you need simple dashboards, you can leverage Azure Databricks BI to create them with a simple click! For complex dashboards, just like any sql db, you can hook it with a simple JDBC string to any external BI tool.
Read full review
Pros
Apache
  • In charge of the ETL processes.
  • As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.
Read full review
Microsoft
  • SQL
  • Data management
  • Data access
Read full review
Cons
Apache
  • they should bring in some time based scheduling too not only event based
  • they do not store the metadata due to which we are not able to analyze the workflows
  • they only support python as of now for scripted pipeline writing
Read full review
Microsoft
  • Their pipeline workflow orchestration is pretty primitive. Lacks some common features
  • Workspace UI and navigation requires steep learning curve
  • Personally, I am not fond of their autosave feature. Its dangerous for production level notebooks scripts
Read full review
Usability
Apache
Easy to learn Easy to use Robust workflow orchestration framework Good in dependent job management
Read full review
Microsoft
Based on my extensive use of Azure Databricks for the past 3.5 years, it has evolved into a beautiful amalgamation of all the data domains and needs. From a data analyst, to a data engineer, to a data scientist, it jas got them all! Being language agnostic and focused on easy to use UI based control, it is a dream to use for every Data related personnel across all experience levels!
Read full review
Alternatives Considered
Apache
There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.
Read full review
Microsoft
Against all the tools I have used, Azure Databricks is by far the most superior of them all! Why, you ask? The UI is modern, the features are never ending and they keep adding new features. And to quote Apple, "It just works!" Far ahead of the competition, the delta lakehouse platform also fares better than it counterparts of Iceberg implementation or a loosely bound Delta Lake implementation of Synapse
Read full review
Return on Investment
Apache
  • A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
  • Allowed us to implement complex data pipelines easily and at a relatively low cost
Read full review
Microsoft
  • Helped reduce time for collecting data
  • Reduced cost in maintaining multiple data sources
  • Access for multiple users and management of users/data in a single platform
Read full review
ScreenShots