Item: Apache Airflow
Rating: 7
Author: Nic Georgakopoulos

Use Cases and Deployment Scope

We use apache Airflow in GCP as part of Cloud Composer to run all our ETL jobs.

Pros and Cons

schedule jobs
graphing job flow and dependencies and retries
Nice UI for visualization

Instead of using a Storage bucket as a source, will be nice if the DAGs can be pulled by a private git repo directly
Upgrade process could be smoother

Return on Investment

Triggering ETL jobs based on events
running DAGs in a scheduled fashion

Key Insights

Do you think Apache Airflow delivers good value for the price?

Yes

Are you happy with Apache Airflow's feature set?

Yes

Did Apache Airflow live up to sales and marketing promises?

I wasn't involved with the selection/purchase process

Did implementation of Apache Airflow go as expected?

Yes

Would you buy Apache Airflow again?

Yes

If you are using GCP, you can use Apache Airflow very easily by using Cloud Composer which is the managed service for Airflow. If you need to deploy it yourself, installation and setup could be tricky.

A very nice job scheduler of DAGs that could become even better

Overall Satisfaction with Apache Airflow