Apache Airflow for Startups
Used Airflow for Analytics & Reporting
Overview
What is Apache Airflow?
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s…
Recent Reviews
Awards
Products that are considered exceptional by their customers based on a variety of criteria win TrustRadius awards. Learn more about the types of TrustRadius awards to make the best purchase decision. More about TrustRadius Awards
Popular Features
- Multi-platform scheduling (9)8.888%
- Central monitoring (9)8.484%
- Logging (9)8.181%
- Alerts and notifications (9)7.979%
Reviewer Pros & Cons
Pricing
N/A
Unavailable
What is Apache Airflow?
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top…
Entry-level set up fee?
- No setup fee
Offerings
- Free Trial
- Free/Freemium Version
- Premium Consulting/Integration Services
Would you like us to let the vendor know that you want pricing?
28 people also want pricing
Alternatives Pricing
N/A
Unavailable
What is Control-M?
Control-M from BMC is a platform for integrating, automating, and orchestrating application and data workflows in production across complex hybrid technology ecosystems. It provides deep operational capabilities, delivering speed, scale, security, and governance.
What is Superblocks?
Superblocks is an IDE for internal tooling – a programmable set of building blocks for developers to create mission-critical internal operational software. The Superblocks Application Builder to assemble flexible components and connect to databases and APIs. Users can create REST, GraphQL, and gPRC…
Product Demos
Getting Started with Apache Airflow
YouTube
Apache Airflow | Build your custom operator for twitter API
YouTube
Features
Return to navigation
Product Details
- About
- Tech Details
- FAQs
What is Apache Airflow?
Apache Airflow Video
What's coming in Airflow 2.0?
Apache Airflow Technical Details
| Operating Systems | Unspecified |
|---|---|
| Mobile Application | No |
Frequently Asked Questions
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.
Reviewers rate Multi-platform scheduling highest, with a score of 8.8.
The most common users of Apache Airflow are from Enterprises (1,001+ employees).
Comparisons
Compare with
Reviews and Ratings
(35)
Community Insights
TrustRadius Insights are summaries of user sentiment data from TrustRadius reviews and, when necessary, 3rd-party data sources. Have feedback on this content? Let us know!
- Business Problems Solved
- Recommendations
Apache Airflow has proven to be a versatile solution for managing and orchestrating various data tasks. Users have utilized this product as a core component for scheduling and monitoring scheduled jobs, inspecting job successes and failures, and troubleshooting errors or failures. It has also been extensively employed in GCP as part of Cloud Composer for running ETL jobs, streamlining data pipelines, and creating workflows for analytics and reporting.
Reviewers have found Apache Airflow to be an easy-to-configure and setup solution, making it ideal for orchestrating data flows and building enterprise data pipelines. Its ability to integrate with third-party solutions via APIs allows for seamless data access and integration. Users have also appreciated the product's capability to manage ETL pipelines and programmatically monitor data pipelines.
Another valuable use case of Apache Airflow is its role in creating workflows, orchestrating data pipelines, and automating tasks. Its flexibility has been particularly beneficial when dealing with complex data pipelines from diverse sources. Furthermore, the product has been effective in performing data integration in AWS S3 region, connecting to relational databases, executing data extracts, and compiling them into multiple flat file segments.
Apache Airflow brings standardization and modularity to data pipelines, enabling the implementation of complex pipelines and facilitating the sharing of data with partners as well as scoring machine learning models. Overall, users have found this product to be a valuable tool for managing data tasks efficiently and effectively.
Based on user reviews, here are the most common recommendations for Apache Airflow:
-
Read the documentation and take an introduction course to fully understand Airflow's behavior and close any knowledge gaps.
-
Consider Airflow as a first choice for ETL tasks that require programming. However, keep in mind that the coding aspect may not be suitable for all ETL engineers.
-
Replace cron jobs with Airflow for better results, utilizing its scheduling and dependency management features.
Overall, these recommendations emphasize the importance of familiarizing oneself with the documentation, leveraging Airflow's capabilities for programming-centric ETL tasks, and using it to replace traditional cron jobs.
Reviews
(1-3 of 3)Companies can't remove reviews or game the system. Here's why
June 26, 2022
Apache AirFlow - Love the Features, Love the Reliability.. Love if the UI get modenized!
We use apache airflow as part of our DAG scheduler and health monitoring tool. It serves as a core component in ensuring our scheduled jobs are run, the ability to allow us to inspect jobs successes and failures, and as a troubleshooting tool in an event of job errors/failures. It has been a core tool and we are happy with what it does.
- Job scheduling - Pretty straightforward in terms of UI.
- Job monitoring - Dashboard is as straightforward as it gets.
- Troubleshooting jobs - ability to dive into detailed errors and navigate the job workflow.
- UI/Dashboard can be updated to be customisable, and jobs summary in groups of errors/failures/success, instead of each job, so that a summary of errors can be used as a starting point for reviewing them.
- Navigation - It's a bit dated. Could do with more modern web navigation UX. i.e. sidebars navigation instead of browser back/forward.
- Again core functional reorg in terms of UX. Navigation can be improved for core functions as well, instead of discovery.
88.33333333333334%
8.8
- It is a good workflow job scheduler.
- It meets all, if not most of our organization product requirements.
- AirFlow stability in terms of the product reliability is unmatched.
- Jenkins and Apache Kafka
Using Jenkins and Kafka, it is not for the same purpose, although it might be similar. I would say AirFlow is really what it says on the can - workflow management. For our organisation, the purpose is clear. So long your aim is to have a rich workflow scheduler and job management, AirFlow is the go-to. Use the tool for what it's meant for, and it will meet your need for sure.
May 05, 2022
Apache Airflow for Automation and scheduling
We are using Apache Airflow for streamline the data pipelines, creating the workflow, Schedule the workflow as per the need, and also monitor the same, we are solving the problem of fetching the data from hive and then created the complete workflow and also we are using for automation as well.
- Smart Automation
- Highly Scalable
- Complex Workflow
- Easy Integration with other system
- Documentation part
- GUI can be improved
- Reliability issues
85%
8.5
- Overall positive impact using apache airflow
- Easy to use
- Easy to integrate which saves time
Overall using Apache Airflow is easy to use compare than other other tools available in the market, It is easy to integrate in apache airflow and the workflow can be monitored and scheduling can be done easily using apache airflow, recommend this tool for Automating the data pipeline or workflow.
April 04, 2022
Apache Airflow
We are using apache airflow for managing the ETL pipelines. We are using programmatically to monitor the data pipeline. I have been helping the data team in creating the pipeline using apache airflow.
- We are using for the workflow management system
- managing the etl pipelines.
- We can manage the task scheduling as code & need not monitor as there is no data in & out.
- they should bring in some time based scheduling too not only event based
- they do not store the metadata due to which we are not able to analyze the workflows
- they only support python as of now for scripted pipeline writing
83.33333333333334%
8.3
- We had a better understanding of data as ETL pipelines were giving data using airflow
- We were able to automate most of the ETL pipelines so it reduced manual efforts
- Airflow UI was extremely helpful which made it easy to understand
Airflow was best suited in my use case for designing the ETL pipelines in a scripted manner for workflows & the UI was very good & easy to use.