Apache Airflow vs. AWS Batch

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

More appropriate if you have a tech group that can use more of the AWS Batch rather than one or 2 things. It works great for me, but there was a huge learning curve the first week of using it. Now, I love it - and I hope to dig deep into other parts not just S3.

Incentivized

• In charge of the ETL processes.
• As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.

Incentivized

• S3 amount of files
• Easy to share files to other sites
• Like that the files and folders can be open and public or completely private
• I like that you can have double or mirroring in different Data Centers for Data Recovery

Incentivized

• they should bring in some time based scheduling too not only event based
• they do not store the metadata due to which we are not able to analyze the workflows
• they only support python as of now for scripted pipeline writing

Incentivized

• Jobs monitoring dashboards are not matured
• Documentation and support is something which can be improved
• Sometime i faced the slow response or slow in performance i would say

Incentivized

There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.

Incentivized

We wanted to start everything on a scale & with fewer resources to manage the underlying infrastructure.

Incentivized

• A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
• Allowed us to implement complex data pipelines easily and at a relatively low cost

Incentivized

• Overall over business is able to save the cost
• Saved our times to improve the existing process
• Able to integrate with other applications as well, so that is plus point

Incentivized