Apache Airflow vs. Confluent

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

If you have a need to stream data, real time or segmented structured data then Confluent is a great platform to do so with. You won't run into packet transfer size limitations that other platforms have. Flexibility in on-prem, cloud, and managed cloud offerings makes it very flexible no matter how you choose to implement.

Incentivized

• In charge of the ETL processes.
• As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.

Incentivized

• Products work great.
• Training is available.
• Customer support is good.

Incentivized

• they should bring in some time based scheduling too not only event based
• they do not store the metadata due to which we are not able to analyze the workflows
• they only support python as of now for scripted pipeline writing

Incentivized

• Cloud based Azure platform features for Confluent lacks behind AWS And GCP

Incentivized

Easy to learn Easy to use Robust workflow orchestration framework Good in dependent job management

Incentivized

The support from the Confluent platform is great and satisfying. We have been working with Confluent for more than a year now. They sent out resident architects to help us set up Confluent cluster on our cloud and help us troubleshoot problems we have encountered. Overall, it has been a great experience working with the Confluent Platform.

Incentivized

There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.

Incentivized

For our use case it was very important that the technology we were working with fit into our Azure architecture, and met our data processing size requirements to stream data within certain SLAs. Confluent more than met our performance requirements and compared to the others scale options and cost to run it was more than financially viable as a platform solution to our global operations.

Incentivized

• A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
• Allowed us to implement complex data pipelines easily and at a relatively low cost

Incentivized

• It enables us to develop event driven application.
• It increases our ability to handle streaming data.
• It reduces latency of communication.

Incentivized