Azure Data Factory vs. Azure Pipelines

Azure Data Factory is a great data integration tool for developing a cloud data platform, especially within the Azure ecosystem. Azure Data Factory is very good for the Data Ingestion part. It can work for simple data transformation with its Data Flow, but it will also need cluster configuration, and there is some cost. Also, it is an excellent tool for orchestrating data pipelines. But for complex data transformations, you may need to use technologies like Databricks and PySpark.

Incentivized

With a fully Microsoft Azure based workflow - Azure Pipelines makes absolute sense. Azure Pipelines are robust and work very well with SonarQube for test coverage and are shared with our developers. This prevents the developers for pushing code without unit tests across our backend and frontend platforms. We have reduced our instances of manual regression tests especially when there are multiple teams working across the same repositories.

Incentivized

• It allows copying data from various types of data sources like on-premise files, Azure Database, Excel, JSON, Azure Synapse, API, etc. to the desired destination.
• We can use linked service in multiple pipeline/data load.
• It also allows the running of SSIS & SSMS packages which makes it an easy-to-use ETL & ELT tool.

Incentivized

• Code Integration/Code Deployment
• Azure Engine Auto Scaling Up with help of Pipeline
• Managing Version Control and deploy in rollback with just one click

Incentivized

• Limited source/sink (target) connectors depending on which area of Azure Data Factory you are using.
• Does not yet have parity with SSIS as far as the transforms available.

Incentivized

• Error messaging when team members don't have permissions

Incentivized

So far product has performed as expected. We were noticing some performance issues, but they were largely Synapse related. This has led to a shift from Synapse to Databricks. Overall this has delayed our analytic platform. Once databricks becomes fully operational, Azure Data Factory will be critical to our environment and future success.

Incentivized

We have not had need to engage with Microsoft much on Azure Data Factory, but they have been responsive and helpful when needed. This being said, we have not had a major emergency or outage requiring their intervention. The score of seven is a representation that they have done well for now, but have not proved out their support for a significant issue

Incentivized

Azure Data Factory is more of a universal pipeline. SAP BW is a tool offering good SAP connectivity but very limited third-party connectivity. The same is the case with BW4hana. SAP Datasphere is offering better connectivity with SAP sources, but not so good when compared to adf. Power Center of Informatica is a legacy tool, and Anaplan is a planning tool with limited connectivity options.

Incentivized

We have used the GitHub CI/CD. Earlier we were using the Azure Pipelines but after GitHub had their actions, we integrated that for CI/CD. It runs the tests and makes a production build which can be live. GitHub CI/CD is more useful because we have to make script only once then just by few changes we can deploy it onto Azure, AWS, Google anywhere so we found it more convenient

Incentivized

• Cost Savings: By automating our ETL processes with Azure Data Factory, we've reduced manual data handling by approximately 60%. This translates to savings from reduced man-hours and the overhead of maintaining legacy systems.
• Timeliness: Our report generation time has reduced by 70% with Azure Data Factory's scheduled pipelines. Faster insights mean quicker decisions for us, enabling our teams to capitalize on time-sensitive opportunities. We can easily share the data visualizations to all stakeholders.

Incentivized

• We don't have to deploy changes manually
• But because of this that we automated some tasks, we need to be still aware of some edge cases we can meet and which can cause a pipelines failures
• We can deploy changes pretty fast

Incentivized