Apache Airflow vs. Panoply

For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.

Incentivized

Panoply is well suited for companies with large amounts of data that do not want all users of the company to see or view. It is appropriate for companies that have to store data for several years in case clients need the data. It is not made to constantly be downloading or looking at the documents.

Incentivized

• In charge of the ETL processes.
• As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.

Incentivized

• Ease of data integrations. Importing data into the data warehouse is a point-and-click effort.
• Automated data modeling. The data is modeled in a way that facilitates analysis without having to look into it myself.
• Query speed. Automated query optimization has really sped up our dashboards.

• they should bring in some time based scheduling too not only event based
• they do not store the metadata due to which we are not able to analyze the workflows
• they only support python as of now for scripted pipeline writing

Incentivized

• The user interface could use some improvement for analytical tools. Apart from that, it’s great.

It’s a good choice for building a quick, multipurpose data stack for a variety of businesses needs.

There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.

Incentivized

Panoply has much more storage and better organization tools for searching. There are better folders and tools for keeping documents secure and stored for a long period of time. It is better to offload data in mass than singular or Zip drive documents like you would do in Google Drive.

Incentivized

• A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
• Allowed us to implement complex data pipelines easily and at a relatively low cost

Incentivized

• Saved time on Redshift setup
• Saved support costs (monetary and temporal) by not having to set up on-premise
• Widens the BI tool use in organization; lowers the barrier to entry and makes organization more productive in analytical terms