Microsoft's Azure Data Factory is a service built for all data integration needs and skill levels. It is designed to allow the user to easily construct ETL and ELT processes code-free within the intuitive visual environment, or write one's own code. Visually integrate data sources using more than 80 natively built and maintenance-free connectors at no added cost. Focus on data—the serverless integration service does the rest.
N/A
Azure Virtual Machines
Score 7.9 out of 10
N/A
Virtual Machines (VMs) are available on Microsoft Azure, providing what is built as a low-cost, per-second compute service, available via Windows or Linux.
Best scenario is for ETL process. The flexibility and connectivity is outstanding. For our environment, SAP data connectivity with Azure Data Factory offers very limited features compared to SAP Data Sphere. Due to the limited modelling capacity of the tool, we use Databricks for data modelling and cleaning. Usage of multiple tools could have been avoided if adf has modelling capabilities.
If you want to host a dedicated Windows server on the cloud, and especially if you want to integrate it with your on premises Active Directory, Azure Virtual Machines should be your first choice. Obviously running Linux on Azure works very well too, but given Azure's pricing is not the cheapest, there are other providers out there that have a better cost-benefit ratio for Linux. That said, hosting Windows on Azure can be affordable (especially when compared to other providers) if you plan your licensing, topology, and application architecture correctly.
When demand is high, we scale the service out, eg During a Football Match.
When a football match is over and the throughput of data from OPTA drops we save by the service scaling back in.
Our App Service Plans along with the Clean C# code are lightening fast giving a good customer experience.
When producing the TV Guide information and a program overruns its scheduled time, a client can instantly be updated to the new programming schedule as our change is instant and its in the right place for all the clients to download and adjust their television guides appropriately to send out to the public giving a 24x7 uptime service that is precise and accurate and resilient to outages due to failover zones around the world.
Granularity of Errors: Sometimes, Azure Data Factory provides error messages that are too generic or vague for us, making it challenging to pinpoint the exact cause of a pipeline failure. Enhanced error messages with more actionable details would greatly assist us as users in debugging their pipelines.
Pipeline Design UI: In my experience, the visual interface for designing pipelines, especially when dealing with complex workflows or numerous activities, can become cluttered. I think a more intuitive and scalable design interface would improve usability. In my opinion, features like zoom, better alignment tools, or grouping capabilities could make managing intricate designs more manageable.
Native Support: While Azure Data Factory does support incremental data loads, in my experience, the setup can be somewhat manual and complex. I think native and more straightforward support for Change Data Capture, especially from popular databases, would simplify the process of capturing and processing only the changed data, making regular data updates more efficient
Pricing can be a concern if you are truly agnostic to which cloud you are building your particular solution in.
The UI, as is the case with any cloud provider, is crowded.
As with any cloud provider, it can be difficult to tune in exactly the right amount of servers for your needs...you might find yourself under/overprovisioning.
So far product has performed as expected. We were noticing some performance issues, but they were largely Synapse related. This has led to a shift from Synapse to Databricks. Overall this has delayed our analytic platform. Once databricks becomes fully operational, Azure Data Factory will be critical to our environment and future success.
No VM console, weak management interface, changing CPU/memory is not straightforward. On the positive side, basic RDP functionality is good to have. As long as things are working, the ability to host Windows VMs is appreciated.
We have not had need to engage with Microsoft much on Azure Data Factory, but they have been responsive and helpful when needed. This being said, we have not had a major emergency or outage requiring their intervention. The score of seven is a representation that they have done well for now, but have not proved out their support for a significant issue
I give the overall support for Azure Virtual Machines a 7 because I think while the overall support do a great job there are still areas that it could improve on such as efficiency and speed. So while I only give it a 7 and it has some issues it is still better than the overall support at Amazon EC2 Auto Scaling.
Azure Data Factory helps us automate to schedule jobs as per customer demands to make ETL triggers when the need arises. Anyone can define the workflow with the Azure Data Factory UI designer tool and easily test the systems. It helped us automate the same workflow with programming languages like Python or automation tools like ansible. Numerous options for connectivity be it a database or storage account helps us move data transfer to the cloud or on-premise systems.
Azure Virtual Machines offer unparalleled flexibility in provisioning, managing and upgrading the VM instances, both manually and programmatically. AVM offer very granular billing options and enables high costs optimisations (while still being costly). The other competitors I mentioned are very good at offering dead-cheap VMs. But if you need anything beyond that, especially for big computing, you need Azure Virtual Machines.
It's so easy to spin up new instances, that it becomes also to easy to have to many of them to manage. Many teams end up with a couple of hundreds of VMs after a short while, making the whole thing very hard to maneuver
Azure VMs are the next step for us to rely on Onprem servers, and leaving the management of the infrastructure to the professionals
The ease of use, is also important when our main focus is to deliver new applications and integrations fast, and not having to worry about infrastructure. We sell bottles, not CPUs
