Skip to main content
TrustRadius
Apache Airflow

Apache Airflow

Overview

What is Apache Airflow?

Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s…

Read more

Learn from top reviewers

Scalable Scheduling Framework and Orchestration tool

Rating: 10 out of 10
January 7, 2025
IncentivizedVetted ReviewVerified User
We are using Apache Airflow as an orchestration tool in data engineering workflows in gaming product. We are scheduling multiple jobs i.e hourly / daily / weekly / monthly. ...
Continue reading
Anshuman Varshney
Engineer - Engineering
Engineering Manager
Gameskraft
Apache Airflow6 years of experience

Apache Airflow for Startups

Rating: 8 out of 10
December 18, 2024
IncentivizedVetted ReviewVerified User
Used Airflow for Analytics & Reporting
Continue reading
Alok Pabalkar
Team Lead - Research & Development
Co-Founder & CTO
Mini Venture Lab Private Limited
Apache Airflow1 year of experience

Apache AirFlow - Love the Features, Love the Reliability.. Love if the UI get modenized!

Rating: 7 out of 10
June 26, 2022
IncentivizedVetted ReviewVerified User
We use apache airflow as part of our DAG scheduler and health monitoring tool. It serves as a core component in ensuring our scheduled jobs are run, the ability to allow us to...
Continue reading
Victor Tay
Engineer - Product Management
Engineer
IronNet CyberSecurity
Apache Airflow1 year of experience

A great solution to help orchestrate workflows and pipelines

Rating: 9 out of 10
June 22, 2022
IncentivizedVetted ReviewVerified User
Apache airflow is a great way to orchestrate workflows and build enterprise data pipelines. It is very easy to configure and setup and would be my go to solution for orchestra...
Continue reading
Nick Waters
Consultant - Sales
Solutions Engineer
Datameer
Apache Airflow1 year of experience

A very nice job scheduler of DAGs that could become even better

Rating: 7 out of 10
June 13, 2022
IncentivizedVetted ReviewVerified User
We use apache Airflow in GCP as part of Cloud Composer to run all our ETL jobs.
Continue reading
Nic Georgakopoulos
Manager - Engineering
DevOps
Deloitte Digital
Apache Airflow3 years of experience
Log in with LinkedIn to see content from your network
Return to navigation

Pricing

View all pricing
Apache Airflow

Apache Airflow

N/A
Unavailable

What is Apache Airflow?

Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top…

Entry-level set up fee?

  • No setup fee

Offerings

  • Free Trial
  • Free/Freemium Version
  • Premium Consulting/Integration Services

Would you like us to let the vendor know that you want pricing?

30 people also want pricing

Alternatives Pricing

Control-M

Control-M

N/A
Unavailable
What is Control-M?

Control-M from BMC is a platform for integrating, automating, and orchestrating application and data workflows in production across complex hybrid technology ecosystems. It provides deep operational capabilities, delivering speed, scale, security, and governance.

Appy Pie

Appy Pie

$16
per app/per month
What is Appy Pie?

Appy Pie is a diversified no-code development platform. It offers app and web development, helpdesk support, chatbot building, design features, and integration that are helpful when starting, running, or growing a business.

Return to navigation

Product Demos

Getting Started with Apache Airflow

YouTube

Apache Airflow | Build your custom operator for twitter API

YouTube
Return to navigation

Features

Workload Automation

Workload automation tools manage event-based scheduling and resource management across a wide variety of applications, databases and architectures

9.8
Avg 8.3
Return to navigation

Product Details

What is Apache Airflow?

Apache Airflow Video

What's coming in Airflow 2.0?

Apache Airflow Technical Details

Operating SystemsUnspecified
Mobile ApplicationNo

Frequently Asked Questions

Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.

Reviewers rate Multi-platform scheduling highest, with a score of 10.

The most common users of Apache Airflow are from Enterprises (1,001+ employees).
Return to navigation

Comparisons

View all alternatives
Return to navigation

Reviews From Top Reviewers

View all reviews
(1-4 of 4)

Apache Airflow

Rating: 8 out of 10
Incentivized
April 04, 2022
PM
PRABHAT MISHRA
Senior Devops Engineer
LEAD School (Education Management, 1001-5000 employees)
Vetted Review
Verified User
Apache Airflow
1 year of experience
View profile
Use Cases and Deployment Scope
We are using apache airflow for managing the ETL pipelines. We are using programmatically to monitor the data pipeline. I have been helping the data team in creating the pipeline using apache airflow.
Pros
  • We are using for the workflow management system
  • managing the etl pipelines.
  • We can manage the task scheduling as code & need not monitor as there is no data in & out.
Cons
  • they should bring in some time based scheduling too not only event based
  • they do not store the metadata due to which we are not able to analyze the workflows
  • they only support python as of now for scripted pipeline writing
Likelihood to Recommend
We were using it for managing the workflows for the etl pipelines as code so Airflow came as very helpful.
Apache Airflow Feature Ratings
Workload Automation (6)
83.33333333333334%
8.3
Multi-platform scheduling
80%
8.0
Central monitoring
70%
7.0
Logging
80%
8.0
Alerts and notifications
90%
9.0
Analysis and visualization
90%
9.0
Application integration
90%
9.0
Return on Investment
  • We had a better understanding of data as ETL pipelines were giving data using airflow
  • We were able to automate most of the ETL pipelines so it reduced manual efforts
  • Airflow UI was extremely helpful which made it easy to understand
Alternatives Considered
Airflow was best suited in my use case for designing the ETL pipelines in a scripted manner for workflows & the UI was very good & easy to use.
Other Software Used

We used it to manage processes for etl pipelines

Rating: 9 out of 10
Incentivized
July 05, 2022
Verified User
Engineer in Information Technology
Information Technology & Services Company, 10,001+ employees
Vetted Review
Verified User
Apache Airflow
3 years of experience
Use Cases and Deployment Scope
We use Apache Airflow to streamline the data pipelines, create workflows according to the needs of the project and overall monitoring of the functionality itself. In addition, we are using Apache Airflow to solve the problem of retrieving data from Hive before creating the workflow in its entirety. It's also utilized for automation.
Pros
  • In charge of the ETL processes.
  • As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.
Cons
  • There is no way to assess the processes because they do not keep the metadata.
  • Python is currently the only language supported for creating programmed pipelines.
  • They need to implement both event-based and time-based scheduling.
Likelihood to Recommend
I handle our pipeline scheduling and monitoring. I had minimal problems with Apache Airflow. It's well-suited for data engineers who are responsible for the creation of the data workflows. It is also best suited for the scheduling of the workflow; it allows us to execute Python scripts as well. Finally, Apache Airflow is best suited for the circumstances in which we need a scalable solution.
Apache Airflow Feature Ratings
Workload Automation (6)
95%
9.5
Multi-platform scheduling
100%
10.0
Central monitoring
100%
10.0
Logging
90%
9.0
Alerts and notifications
90%
9.0
Analysis and visualization
100%
10.0
Application integration
90%
9.0
Return on Investment
  • Most of the ETL processes were automated, cutting down on human labor.
  • Apache Airflow's user interface (UI) was very informative and straightforward.
  • Since ETL processes were providing data via airflow, we were able to gain a deeper comprehension of the data at hand.
Alternatives Considered
Multiple DAGs can be orchestrated simultaneously at varying times, and runs can be reproduced or replicated with relative ease. Overall, utilizing Apache Airflow is easier to use than other solutions now on the market. It is simple to integrate in Apache Airflow, and the workflow can be monitored and scheduling can be done quickly using Apache Airflow. We advocate using this tool for automating the data pipeline or process.
Other Software Used

A very nice job scheduler of DAGs that could become even better

Rating: 7 out of 10
Incentivized
June 13, 2022
NG
Nic Georgakopoulos
DevOps
Deloitte Digital (Internet, 1001-5000 employees)
Vetted Review
Verified User
Apache Airflow
3 years of experience
Use Cases and Deployment Scope
We use apache Airflow in GCP as part of Cloud Composer to run all our ETL jobs.
Pros
  • schedule jobs
  • graphing job flow and dependencies and retries
  • Nice UI for visualization
Cons
  • Instead of using a Storage bucket as a source, will be nice if the DAGs can be pulled by a private git repo directly
  • Upgrade process could be smoother
Likelihood to Recommend
If you are using GCP, you can use Apache Airflow very easily by using Cloud Composer which is the managed service for Airflow. If you need to deploy it yourself, installation and setup could be tricky.
Apache Airflow Feature Ratings
Workload Automation (6)
65%
6.5
Multi-platform scheduling
90%
9.0
Central monitoring
70%
7.0
Logging
50%
5.0
Alerts and notifications
50%
5.0
Analysis and visualization
60%
6.0
Application integration
70%
7.0
Return on Investment
  • Triggering ETL jobs based on events
  • running DAGs in a scheduled fashion

Apache AirFlow - Love the Features, Love the Reliability.. Love if the UI get modenized!

Rating: 7 out of 10
Incentivized
June 26, 2022
VT
Victor Tay
Engineer
IronNet CyberSecurity (Computer Software, 201-500 employees)
Vetted Review
Verified User
Apache Airflow
1 year of experience
Use Cases and Deployment Scope
We use apache airflow as part of our DAG scheduler and health monitoring tool. It serves as a core component in ensuring our scheduled jobs are run, the ability to allow us to inspect jobs successes and failures, and as a troubleshooting tool in an event of job errors/failures. It has been a core tool and we are happy with what it does.
Pros
  • Job scheduling - Pretty straightforward in terms of UI.
  • Job monitoring - Dashboard is as straightforward as it gets.
  • Troubleshooting jobs - ability to dive into detailed errors and navigate the job workflow.
Cons
  • UI/Dashboard can be updated to be customisable, and jobs summary in groups of errors/failures/success, instead of each job, so that a summary of errors can be used as a starting point for reviewing them.
  • Navigation - It's a bit dated. Could do with more modern web navigation UX. i.e. sidebars navigation instead of browser back/forward.
  • Again core functional reorg in terms of UX. Navigation can be improved for core functions as well, instead of discovery.
Likelihood to Recommend
For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.
Apache Airflow Feature Ratings
Workload Automation (6)
88.33333333333334%
8.8
Multi-platform scheduling
100%
10.0
Central monitoring
100%
10.0
Logging
100%
10.0
Alerts and notifications
80%
8.0
Analysis and visualization
70%
7.0
Application integration
80%
8.0
Return on Investment
  • It is a good workflow job scheduler.
  • It meets all, if not most of our organization product requirements.
  • AirFlow stability in terms of the product reliability is unmatched.
Alternatives Considered
Using Jenkins and Kafka, it is not for the same purpose, although it might be similar. I would say AirFlow is really what it says on the can - workflow management. For our organisation, the purpose is clear. So long your aim is to have a rich workflow scheduler and job management, AirFlow is the go-to. Use the tool for what it's meant for, and it will meet your need for sure.
Other Software Used
Jenkins, Apache Kafka, Redis™*
Return to navigation