Apache Airflow for Startups
Used Airflow for Analytics & Reporting
Overview
What is Apache Airflow?
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s…
Recent Reviews
Awards
Products that are considered exceptional by their customers based on a variety of criteria win TrustRadius awards. Learn more about the types of TrustRadius awards to make the best purchase decision. More about TrustRadius Awards
Popular Features
- Multi-platform scheduling (9)8.888%
- Central monitoring (9)8.484%
- Logging (9)8.181%
- Alerts and notifications (9)7.979%
Reviewer Pros & Cons
Pricing
N/A
Unavailable
What is Apache Airflow?
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top…
Entry-level set up fee?
- No setup fee
Offerings
- Free Trial
- Free/Freemium Version
- Premium Consulting/Integration Services
Would you like us to let the vendor know that you want pricing?
28 people also want pricing
Alternatives Pricing
N/A
Unavailable
What is Control-M?
Control-M from BMC is a platform for integrating, automating, and orchestrating application and data workflows in production across complex hybrid technology ecosystems. It provides deep operational capabilities, delivering speed, scale, security, and governance.
What is Superblocks?
Superblocks is an IDE for internal tooling – a programmable set of building blocks for developers to create mission-critical internal operational software. The Superblocks Application Builder to assemble flexible components and connect to databases and APIs. Users can create REST, GraphQL, and gPRC…
Product Demos
Getting Started with Apache Airflow
YouTube
Apache Airflow | Build your custom operator for twitter API
YouTube
Features
Return to navigation
Product Details
- About
- Tech Details
- FAQs
What is Apache Airflow?
Apache Airflow Video
What's coming in Airflow 2.0?
Apache Airflow Technical Details
| Operating Systems | Unspecified |
|---|---|
| Mobile Application | No |
Frequently Asked Questions
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.
Reviewers rate Multi-platform scheduling highest, with a score of 8.8.
The most common users of Apache Airflow are from Enterprises (1,001+ employees).
Comparisons
Compare with
Reviews and Ratings
(35)
Community Insights
TrustRadius Insights are summaries of user sentiment data from TrustRadius reviews and, when necessary, 3rd-party data sources. Have feedback on this content? Let us know!
- Business Problems Solved
- Recommendations
Apache Airflow has proven to be a versatile solution for managing and orchestrating various data tasks. Users have utilized this product as a core component for scheduling and monitoring scheduled jobs, inspecting job successes and failures, and troubleshooting errors or failures. It has also been extensively employed in GCP as part of Cloud Composer for running ETL jobs, streamlining data pipelines, and creating workflows for analytics and reporting.
Reviewers have found Apache Airflow to be an easy-to-configure and setup solution, making it ideal for orchestrating data flows and building enterprise data pipelines. Its ability to integrate with third-party solutions via APIs allows for seamless data access and integration. Users have also appreciated the product's capability to manage ETL pipelines and programmatically monitor data pipelines.
Another valuable use case of Apache Airflow is its role in creating workflows, orchestrating data pipelines, and automating tasks. Its flexibility has been particularly beneficial when dealing with complex data pipelines from diverse sources. Furthermore, the product has been effective in performing data integration in AWS S3 region, connecting to relational databases, executing data extracts, and compiling them into multiple flat file segments.
Apache Airflow brings standardization and modularity to data pipelines, enabling the implementation of complex pipelines and facilitating the sharing of data with partners as well as scoring machine learning models. Overall, users have found this product to be a valuable tool for managing data tasks efficiently and effectively.
Based on user reviews, here are the most common recommendations for Apache Airflow:
-
Read the documentation and take an introduction course to fully understand Airflow's behavior and close any knowledge gaps.
-
Consider Airflow as a first choice for ETL tasks that require programming. However, keep in mind that the coding aspect may not be suitable for all ETL engineers.
-
Replace cron jobs with Airflow for better results, utilizing its scheduling and dependency management features.
Overall, these recommendations emphasize the importance of familiarizing oneself with the documentation, leveraging Airflow's capabilities for programming-centric ETL tasks, and using it to replace traditional cron jobs.
Reviews
(1-9 of 9)Companies can't remove reviews or game the system. Here's why
March 06, 2024
Apache Airflow for Startups
- Reports
- Sending Bulk Email/Notification
- Processing from different data sources
- Improve the GUI Control Panel
- Provide more example and documentation
- Improvement in debugging
July 05, 2022
We used it to manage processes for etl pipelines
- In charge of the ETL processes.
- As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.
- There is no way to assess the processes because they do not keep the metadata.
- Python is currently the only language supported for creating programmed pipelines.
- They need to implement both event-based and time-based scheduling.
June 26, 2022
Apache AirFlow - Love the Features, Love the Reliability.. Love if the UI get modenized!
- Job scheduling - Pretty straightforward in terms of UI.
- Job monitoring - Dashboard is as straightforward as it gets.
- Troubleshooting jobs - ability to dive into detailed errors and navigate the job workflow.
- UI/Dashboard can be updated to be customisable, and jobs summary in groups of errors/failures/success, instead of each job, so that a summary of errors can be used as a starting point for reviewing them.
- Navigation - It's a bit dated. Could do with more modern web navigation UX. i.e. sidebars navigation instead of browser back/forward.
- Again core functional reorg in terms of UX. Navigation can be improved for core functions as well, instead of discovery.
June 24, 2022
Apache Airflow is flawless
- Multiple helpful features
- Very intuitive flow charts
- Reruns and backfills are very easy
- SLA and DAGs are easy to set up
- Potentially a steep learning curve
- The browser UI could do with a few enhancements
- Orchestrate workflows
- Visualise workflows easily using DAG
- Integrate 3rd party data sources
- Visualisation UI could be improved in my opinion.
- Enterprise features
- Performance improvements in bigger deployments.
June 21, 2022
Apache Airflow software
- Scheduling of data pipelines or workflows.
- Orchestration of data pipelines or workflows.
- Not intuitive for new users.
- Setting up Airflow architecture for production is NOT easy.
- schedule jobs
- graphing job flow and dependencies and retries
- Nice UI for visualization
- Instead of using a Storage bucket as a source, will be nice if the DAGs can be pulled by a private git repo directly
- Upgrade process could be smoother
May 05, 2022
Apache Airflow for Automation and scheduling
- Smart Automation
- Highly Scalable
- Complex Workflow
- Easy Integration with other system
- Documentation part
- GUI can be improved
- Reliability issues
April 04, 2022
Apache Airflow
- We are using for the workflow management system
- managing the etl pipelines.
- We can manage the task scheduling as code & need not monitor as there is no data in & out.
- they should bring in some time based scheduling too not only event based
- they do not store the metadata due to which we are not able to analyze the workflows
- they only support python as of now for scripted pipeline writing