Item: Apache Airflow
Rating: 7
Author: Victor Tay

Use Cases and Deployment Scope

We use apache airflow as part of our DAG scheduler and health monitoring tool. It serves as a core component in ensuring our scheduled jobs are run, the ability to allow us to inspect jobs successes and failures, and as a troubleshooting tool in an event of job errors/failures. It has been a core tool and we are happy with what it does.

Pros and Cons

Job scheduling - Pretty straightforward in terms of UI.
Job monitoring - Dashboard is as straightforward as it gets.
Troubleshooting jobs - ability to dive into detailed errors and navigate the job workflow.

UI/Dashboard can be updated to be customisable, and jobs summary in groups of errors/failures/success, instead of each job, so that a summary of errors can be used as a starting point for reviewing them.
Navigation - It's a bit dated. Could do with more modern web navigation UX. i.e. sidebars navigation instead of browser back/forward.
Again core functional reorg in terms of UX. Navigation can be improved for core functions as well, instead of discovery.

Return on Investment

It is a good workflow job scheduler.
It meets all, if not most of our organization product requirements.
AirFlow stability in terms of the product reliability is unmatched.

Alternatives Considered

Jenkins and Apache Kafka

Using Jenkins and Kafka, it is not for the same purpose, although it might be similar. I would say AirFlow is really what it says on the can - workflow management. For our organisation, the purpose is clear. So long your aim is to have a rich workflow scheduler and job management, AirFlow is the go-to. Use the tool for what it's meant for, and it will meet your need for sure.

Key Insights

Do you think Apache Airflow delivers good value for the price?

Yes

Are you happy with Apache Airflow's feature set?

Yes

Did Apache Airflow live up to sales and marketing promises?

I wasn't involved with the selection/purchase process

Did implementation of Apache Airflow go as expected?

Yes

Would you buy Apache Airflow again?

Yes

Other Software Used

Jenkins, Apache Kafka, Redis™*

Likelihood to Recommend

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Apache AirFlow - Love the Features, Love the Reliability.. Love if the UI get modenized!

Overall Satisfaction with Apache Airflow