Apache Airflow vs. Grafana Loki

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

In our application there are many points from where logs are coming so in order to go to each and every application and check logs its very overhead so we are using Grafana Loki for the logs gathering and monitoring.

Incentivized

• In charge of the ETL processes.
• As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.

Incentivized

• Access to many open-source dashboards, access to add many data-sources to gather and visualize data from.
• Grafana Loki does well gathering of logs from various data-sources, we can also filter the logs based on our needs.
• One stop solution for all the logs and monitoring.

Incentivized

• they should bring in some time based scheduling too not only event based
• they do not store the metadata due to which we are not able to analyze the workflows
• they only support python as of now for scripted pipeline writing

Incentivized

• We can modify the logs directly from UI of Grafana Loki
• Better and simplified options for logs filtering
• Easy usability with SMTP configuration and other system level configuration

Incentivized

Easy to learn Easy to use Robust workflow orchestration framework Good in dependent job management

Incentivized

There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.

Incentivized

First and foremost if Grafana Loki is based on CNCF open source projects so organizations can get freedom to choice to configure it at your own other main thing is Grafana Loki is totally free of cost and we can deploy it on our infrastructure. On compared with other managed services like Datadog, New Relic it is very expensive and we also don't have much control on the tools we use.

Incentivized

• A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
• Allowed us to implement complex data pipelines easily and at a relatively low cost

Incentivized

• Only indexes the metadata
• Have to manage it by ourselves compare to other available managed monitoring and log observability solutions
• Dedicated person or team of SRE to manage the monitoring and observability solutions

Incentivized