Item: Apache Airflow
Rating: 8
Author: Verified User

Overall Satisfaction with Apache Airflow

Use Cases and Deployment Scope

Apache Airflow is a best orchestrator in market. It gives us to flexibility to orchestrate our data engineering workflows with various levels of modifications possible through python programming. It allows us to connect with various cloud providers like Google, AWS and Azure which enables the teams to work in cross cloud environment.

Pros and Cons

Pros

Provides Connection to different Cloud Providers
Good Access Management
Good User Interface for Users to interact with. If we need to pause , trigger manually , mark any task as successful etc

Cons

A local "dry run" or IDE plugin that can validate and simulate DAG execution without needing a full environment.
Better feedback on DAG parse errors in the UI or CLI.
Navigating large DAGs with hundreds of tasks can be slow and hard to understand visually.

Return on Investment

Apache Airflow various options to interact with different databases around controlled by the business since we get the flexibility to write in python.
Since Apache Airflow requires python programming hence onboarding people it takes time to onboard the data pipelines because it requires some development effort
Apache Airflow makes monitoring easy for all the stake holders as business can see their pipelines running in UI

Usability

For its capability to connect with multicloud environments. Access Control management is something that we don't get in all the schedulers and orchestrators. But although it provides so many flexibility and options to due to python , some level of knowledge of python is needed to be able to build workflows.

Alternatives Considered

AWS Step Functions

Step functions are only available in AWS but Apache Airflow provides cross cloud access. Apache Airflow also provides flexibility to pause, start and re-trigger dags. Provides executors where we can run in-house calculations if needed and which requires no integration with other tools.

Key Insights

Do you think Apache Airflow delivers good value for the price?

Yes

Are you happy with Apache Airflow's feature set?

Yes

Did Apache Airflow live up to sales and marketing promises?

Yes

Did implementation of Apache Airflow go as expected?

Yes

Would you buy Apache Airflow again?

Yes

Other Software Used

Docker, Kubernetes, DBeaver

Likelihood to Recommend

Apache Airflow is well suited when we want to connect to different cloud providers from same interface. It is useful when we have to run or batch pipelines where we pull data from transactional systems and apply business rules.It provides rich scheduling with the help of cron expressions to the batch pipelines however it is not good for real time processing because the task have overhead and take several seconds or minutes to start

Apache Airflow Feature Ratings

Comments

Please log in to join the conversation

Apache Airflow master of Schedulers and Orchestrator