Apache Airflow for Startups
Used Airflow for Analytics & Reporting
Overview
What is Apache Airflow?
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s…
Recent Reviews
Awards
Products that are considered exceptional by their customers based on a variety of criteria win TrustRadius awards. Learn more about the types of TrustRadius awards to make the best purchase decision. More about TrustRadius Awards
Popular Features
- Multi-platform scheduling (9)8.888%
- Central monitoring (9)8.484%
- Logging (9)8.181%
- Alerts and notifications (9)7.979%
Reviewer Pros & Cons
Pricing
N/A
Unavailable
What is Apache Airflow?
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top…
Entry-level set up fee?
- No setup fee
Offerings
- Free Trial
- Free/Freemium Version
- Premium Consulting/Integration Services
Would you like us to let the vendor know that you want pricing?
28 people also want pricing
Alternatives Pricing
N/A
Unavailable
What is Control-M?
Control-M from BMC is a platform for integrating, automating, and orchestrating application and data workflows in production across complex hybrid technology ecosystems. It provides deep operational capabilities, delivering speed, scale, security, and governance.
What is Superblocks?
Superblocks is an IDE for internal tooling – a programmable set of building blocks for developers to create mission-critical internal operational software. The Superblocks Application Builder to assemble flexible components and connect to databases and APIs. Users can create REST, GraphQL, and gPRC…
Product Demos
Getting Started with Apache Airflow
YouTube
Apache Airflow | Build your custom operator for twitter API
YouTube
Features
Return to navigation
Product Details
- About
- Tech Details
- FAQs
What is Apache Airflow?
Apache Airflow Video
What's coming in Airflow 2.0?
Apache Airflow Technical Details
| Operating Systems | Unspecified |
|---|---|
| Mobile Application | No |
Frequently Asked Questions
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.
Reviewers rate Multi-platform scheduling highest, with a score of 8.8.
The most common users of Apache Airflow are from Enterprises (1,001+ employees).
Comparisons
Compare with
Reviews and Ratings
(35)
Community Insights
TrustRadius Insights are summaries of user sentiment data from TrustRadius reviews and, when necessary, 3rd-party data sources. Have feedback on this content? Let us know!
- Business Problems Solved
- Recommendations
Apache Airflow has proven to be a versatile solution for managing and orchestrating various data tasks. Users have utilized this product as a core component for scheduling and monitoring scheduled jobs, inspecting job successes and failures, and troubleshooting errors or failures. It has also been extensively employed in GCP as part of Cloud Composer for running ETL jobs, streamlining data pipelines, and creating workflows for analytics and reporting.
Reviewers have found Apache Airflow to be an easy-to-configure and setup solution, making it ideal for orchestrating data flows and building enterprise data pipelines. Its ability to integrate with third-party solutions via APIs allows for seamless data access and integration. Users have also appreciated the product's capability to manage ETL pipelines and programmatically monitor data pipelines.
Another valuable use case of Apache Airflow is its role in creating workflows, orchestrating data pipelines, and automating tasks. Its flexibility has been particularly beneficial when dealing with complex data pipelines from diverse sources. Furthermore, the product has been effective in performing data integration in AWS S3 region, connecting to relational databases, executing data extracts, and compiling them into multiple flat file segments.
Apache Airflow brings standardization and modularity to data pipelines, enabling the implementation of complex pipelines and facilitating the sharing of data with partners as well as scoring machine learning models. Overall, users have found this product to be a valuable tool for managing data tasks efficiently and effectively.
Based on user reviews, here are the most common recommendations for Apache Airflow:
-
Read the documentation and take an introduction course to fully understand Airflow's behavior and close any knowledge gaps.
-
Consider Airflow as a first choice for ETL tasks that require programming. However, keep in mind that the coding aspect may not be suitable for all ETL engineers.
-
Replace cron jobs with Airflow for better results, utilizing its scheduling and dependency management features.
Overall, these recommendations emphasize the importance of familiarizing oneself with the documentation, leveraging Airflow's capabilities for programming-centric ETL tasks, and using it to replace traditional cron jobs.
Reviews
(1-6 of 6)Companies can't remove reviews or game the system. Here's why
July 05, 2022
We used it to manage processes for etl pipelines
We use Apache Airflow to streamline the data pipelines, create workflows according to the needs of the project and overall monitoring of the functionality itself. In addition, we are using Apache Airflow to solve the problem of retrieving data from Hive before creating the workflow in its entirety. It's also utilized for automation.
- In charge of the ETL processes.
- As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.
- There is no way to assess the processes because they do not keep the metadata.
- Python is currently the only language supported for creating programmed pipelines.
- They need to implement both event-based and time-based scheduling.
June 24, 2022
Apache Airflow is flawless
We use Apache Airflow to perform data integration in AWS S3 region. With this we are able to connect to a relational database, easily execute data extracts, and compile them all in multiple flat file segments. Airflow brings a lot of standardization as well as modularity. We also use it to send data to partners and score ML models. It allows us to implement complex data pipelines easily.
- Multiple helpful features
- Very intuitive flow charts
- Reruns and backfills are very easy
- SLA and DAGs are easy to set up
- Potentially a steep learning curve
- The browser UI could do with a few enhancements
Apache airflow is a great way to orchestrate workflows and build enterprise data pipelines. It is very easy to configure and setup and would be my go to solution for orchestrating data flows. We use Airflow to integrate our solution via APIs and allow third party solutions to access our solution and data held within in it.
- Orchestrate workflows
- Visualise workflows easily using DAG
- Integrate 3rd party data sources
- Visualisation UI could be improved in my opinion.
- Enterprise features
- Performance improvements in bigger deployments.
June 21, 2022
Apache Airflow software
Apache Airflow is used for the scheduling and orchestration of data pipelines or workflows. Orchestration of data pipelines refers to the sequencing, coordination, scheduling, and managing of complex data pipelines from diverse sources. It is also helpful when your data pipelines change slowly (days or weeks – not hours or minutes), are related to a specific time interval, or are pre-scheduled.
- Scheduling of data pipelines or workflows.
- Orchestration of data pipelines or workflows.
- Not intuitive for new users.
- Setting up Airflow architecture for production is NOT easy.
May 05, 2022
Apache Airflow for Automation and scheduling
We are using Apache Airflow for streamline the data pipelines, creating the workflow, Schedule the workflow as per the need, and also monitor the same, we are solving the problem of fetching the data from hive and then created the complete workflow and also we are using for automation as well.
- Smart Automation
- Highly Scalable
- Complex Workflow
- Easy Integration with other system
- Documentation part
- GUI can be improved
- Reliability issues
April 04, 2022
Apache Airflow
We are using apache airflow for managing the ETL pipelines. We are using programmatically to monitor the data pipeline. I have been helping the data team in creating the pipeline using apache airflow.
- We are using for the workflow management system
- managing the etl pipelines.
- We can manage the task scheduling as code & need not monitor as there is no data in & out.
- they should bring in some time based scheduling too not only event based
- they do not store the metadata due to which we are not able to analyze the workflows
- they only support python as of now for scripted pipeline writing