Apache Airflow vs. Docker

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

You are going to be able to find the most resources and examples using Docker whenever you are working with a container orchestration software like Kubernetes. There will always some entropy when you run in a container, a containerized application will never be as purely performant as an app running directly on the OS. However, in most scenarios this loss will be negligible to the time saved in deployment, monitoring, etc.

Incentivized

• In charge of the ETL processes.
• As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.

Incentivized

• Packaging of application to limit the space occupied
• Ease of running the application
• Provide multiple ways to handle the application issues and integration of different components like pipeline, ansible, terraform etc

• they should bring in some time based scheduling too not only event based
• they do not store the metadata due to which we are not able to analyze the workflows
• they only support python as of now for scripted pipeline writing

Incentivized

• Docker hub image retention policy can be relaxed
• Docker hub policies can be more developer friendly
• Docker CLI help section can be improved
• Image and container storage (local) management can be optimized

Incentivized

Easy to learn Easy to use Robust workflow orchestration framework Good in dependent job management

Incentivized

I have been using Docker for more than 3 years and it really simplifies the modern application development and deployment. I like the ability of Docker to improve efficiency, portability and scalability for developers and operations teams. Another reason for giving this rating is because Docker integrates CI/CD pipelines very well

Incentivized

Haven't seen any outages, fatal/unrecoverable errors in my usage so far. Enough said.

Incentivized

Docker Desktop. The CPU high usage is a known issue. Needs fixing. Otherwise, it is great overall. Would not use anything else still.

Incentivized

There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.

Incentivized

The reason why we are still using Docker right now is due to that is the best among its peers and suits our needs the best. However, the trend we foresee for the future might indicate Amazon lambda could potentially fit our needs to code enviornmentless in the near future.

Incentivized

It is the only tool in our toolset that has not [had] any issues so far. That is really a mark of reliability, and it's a testimony to how well the product is made, and a tool that does its job well is a tool well worth having. It is the base tool that I would say any organisation must have if they do scalable deployment.

Incentivized

• A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
• Allowed us to implement complex data pipelines easily and at a relatively low cost

Incentivized

• Reduces the number of virtual machine which impacted our quarterly billing
• Using docker with proxy we run multiple application on same port on same host.
• impact on billing is we have to provide docker training to the people who are working on it.