Azure Data Factory an Universal pipe
Overall Satisfaction with Azure Data Factory
We live in a world where half of the data for analytics come from SAP and half from non SAP sources. We use Azure Data Factory to load non SAP data from different source systems into Azure lake house. The project follows medallion architecture where Azure Data Factory takes data from multiple sources and stores them in the bronze layer of the medallion architecture. Since our SAP Datasphere has limitations connecting to non SAP sources as good and native like Azure Data Factory, we use Azure Data Factory for these scenarios. Further modelling of data in the next layers (silver layer and gold layer) is done using Azure Data Bricks, where the final data product is created. The Azure Data Factory also helps in applying transformations which loading the data from different source systems. Datasphere often relies on ODBC/JDBC/OData connectivity, whereas Azure Data Factory provides maintenance-free connectors for our web applications, like partner portal, cloud applications like one crm, on-prem Oracle systems, and also to NoSQL dbs like MongoDB. To summarize Azure Data Factory is used in our organisation to ingest non SAP data from different sources into our Bronze layer for the Databricks to further clean and curate the data for data product creation. Without Azure data factory connecting the data from different source wouldnt been possible because SAP Datasphere has limitations when it comes to connection with non SAP source systems
Pros
- Connectivity with other cloud environment like Salesforce
- Connectivity with non structured data and big data systems
- Reduces data islands
- Azure Data Factory handles perfectly the huge volume of data in JSON format from our global apps and services
Cons
- The error details where there is an error while processing the files is not clear
- Connectivity with s4 system is not so good compared to Datasphere
- Since Azure Data Factory just transfers data it lacks the capacity to identify the wrongness in the data. It is just a dumb data transfer tool from point A to B
- Drag and drop interface is a positive feature which allows end users to create data pipelines
- The advantage of no or low code is causing spaghetti situation sometimes
- Cost efficient as this is serverless and use only for pay is a interesting
- No meta data or governance of data
Azure Data Factory is more of a universal pipeline. SAP BW is a tool offering good SAP connectivity but very limited third-party connectivity. The same is the case with BW4hana. SAP Datasphere is offering better connectivity with SAP sources, but not so good when compared to adf. Power Center of Informatica is a legacy tool, and Anaplan is a planning tool with limited connectivity options.
Do you think Azure Data Factory delivers good value for the price?
Yes
Are you happy with Azure Data Factory's feature set?
Yes
Did Azure Data Factory live up to sales and marketing promises?
Yes
Did implementation of Azure Data Factory go as expected?
Yes
Would you buy Azure Data Factory again?
Yes


Comments
Please log in to join the conversation