Apache Airflow vs. Apache Camel

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

Message brokering across different systems, with transactionality and the ability to have fine tuned control over what happens using Java (or other languages), instead of a heavy, proprietary languages. One situation that it doesn't fit very well (as far as I have experienced) is when your workflow requires significant data mapping. While possible when using Java tooling, some other visual data mapping tools in other integration frameworks are easier to work with.

Incentivized

• In charge of the ETL processes.
• As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.

Incentivized

• Camel has an easy learning curve. It is fairly well documented and there are about 5-6 books on Camel.
• There is a large user group and blogs devoted to all things Camel and the developers of Camel provide quick answers and have also been very quick to patch Camel, when bugs are reported.
• Camel integrates well with well known frameworks like Spring, and other middleware products like Apache Karaf and Servicemix.
• There are over 150 components for the Camel framework that help integrate with diverse software platforms.
• Camel is also good for creating microservices.

Incentivized

• they should bring in some time based scheduling too not only event based
• they do not store the metadata due to which we are not able to analyze the workflows
• they only support python as of now for scripted pipeline writing

Incentivized

• didn't work well when our developers tried to transform heavy data sets
• Apache Camel's whole logic is based on java so team needs to have a great skill set in java
• if there are a handful of workflows then Apache Camel's full potential can't be realized

Incentivized

Easy to learn Easy to use Robust workflow orchestration framework Good in dependent job management

Incentivized

There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.

Incentivized

If you are looking for a Java-based open source low cost equivalent to webMethods or Azure Logic Apps, Apache Camel is an excellent choice as it is mature and widely deployed, and included in many vendored Java application servers too such as Redhat JBoss EAP. Apache Camel is lacking on the GUI tooling side compared to commercial products such as webMethods or Azure Logic Apps.

Incentivized

• A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
• Allowed us to implement complex data pipelines easily and at a relatively low cost

Incentivized

• Very fast time to market in that so many components are available to use immediately.
• Error handling mechanisms and patterns of practice are robust and easy to use which in turn has made our application more robust from the start, so fewer bugs.
• However, testing and debugging routes is more challenging than working is standard Java so that takes more time (less time than writing the components from scratch).
• Most people don't know Camel coming in and many junior developers find it overwhelming and are not enthusiastic to learn it. So finding people that want to develop/maintain it is a challenge.

Incentivized