Microsoft's Azure Data Factory is a service built for all data integration needs and skill levels. It is designed to allow the user to easily construct ETL and ELT processes code-free within the intuitive visual environment, or write one's own code. Visually integrate data sources using more than 80 natively built and maintenance-free connectors at no added cost. Focus on data—the serverless integration service does the rest.
N/A
Azure Virtual Machines
Score 7.9 out of 10
N/A
Virtual Machines (VMs) are available on Microsoft Azure, providing what is built as a low-cost, per-second compute service, available via Windows or Linux.
Well-suited Scenarios for Azure Data Factory (ADF): When an organization has data sources spread across on-premises databases and cloud storage solutions, I think Azure Data Factory is excellent for integrating these sources. Azure Data Factory's integration with Azure Databricks allows it to handle large-scale data transformations effectively, leveraging the power of distributed processing. For regular ETL or ELT processes that need to run at specific intervals (daily, weekly, etc.), I think Azure Data Factory's scheduling capabilities are very handy. Less Appropriate Scenarios for Azure Data Factory: Real-time Data Streaming - Azure Data Factory is primarily batch-oriented. Simple Data Copy Tasks - For straightforward data copy tasks without the need for transformation or complex workflows, in my opinion, using Azure Data Factory might be overkill; simpler tools or scripts could suffice. Advanced Data Science Workflows: While Azure Data Factory can handle data prep and transformation, in my experience, it's not designed for in-depth data science tasks. I think for advanced analytics, machine learning, or statistical modeling, integration with specialized tools would be necessary.
If you want to host a dedicated Windows server on the cloud, and especially if you want to integrate it with your on premises Active Directory, Azure Virtual Machines should be your first choice. Obviously running Linux on Azure works very well too, but given Azure's pricing is not the cheapest, there are other providers out there that have a better cost-benefit ratio for Linux. That said, hosting Windows on Azure can be affordable (especially when compared to other providers) if you plan your licensing, topology, and application architecture correctly.
It allows copying data from various types of data sources like on-premise files, Azure Database, Excel, JSON, Azure Synapse, API, etc. to the desired destination.
We can use linked service in multiple pipeline/data load.
It also allows the running of SSIS & SSMS packages which makes it an easy-to-use ETL & ELT tool.
When demand is high, we scale the service out, eg During a Football Match.
When a football match is over and the throughput of data from OPTA drops we save by the service scaling back in.
Our App Service Plans along with the Clean C# code are lightening fast giving a good customer experience.
When producing the TV Guide information and a program overruns its scheduled time, a client can instantly be updated to the new programming schedule as our change is instant and its in the right place for all the clients to download and adjust their television guides appropriately to send out to the public giving a 24x7 uptime service that is precise and accurate and resilient to outages due to failover zones around the world.
Pricing can be a concern if you are truly agnostic to which cloud you are building your particular solution in.
The UI, as is the case with any cloud provider, is crowded.
As with any cloud provider, it can be difficult to tune in exactly the right amount of servers for your needs...you might find yourself under/overprovisioning.
So far product has performed as expected. We were noticing some performance issues, but they were largely Synapse related. This has led to a shift from Synapse to Databricks. Overall this has delayed our analytic platform. Once databricks becomes fully operational, Azure Data Factory will be critical to our environment and future success.
No VM console, weak management interface, changing CPU/memory is not straightforward. On the positive side, basic RDP functionality is good to have. As long as things are working, the ability to host Windows VMs is appreciated.
We have not had need to engage with Microsoft much on Azure Data Factory, but they have been responsive and helpful when needed. This being said, we have not had a major emergency or outage requiring their intervention. The score of seven is a representation that they have done well for now, but have not proved out their support for a significant issue
I give the overall support for Azure Virtual Machines a 7 because I think while the overall support do a great job there are still areas that it could improve on such as efficiency and speed. So while I only give it a 7 and it has some issues it is still better than the overall support at Amazon EC2 Auto Scaling.
The easy integration with other Microsoft software as well as high processing speed, very flexible cost, and high level of security of Microsoft Azure products and services stack up against other similar products.
Azure Virtual Machines offer unparalleled flexibility in provisioning, managing and upgrading the VM instances, both manually and programmatically. AVM offer very granular billing options and enables high costs optimisations (while still being costly). The other competitors I mentioned are very good at offering dead-cheap VMs. But if you need anything beyond that, especially for big computing, you need Azure Virtual Machines.
It's so easy to spin up new instances, that it becomes also to easy to have to many of them to manage. Many teams end up with a couple of hundreds of VMs after a short while, making the whole thing very hard to maneuver
Azure VMs are the next step for us to rely on Onprem servers, and leaving the management of the infrastructure to the professionals
The ease of use, is also important when our main focus is to deliver new applications and integrations fast, and not having to worry about infrastructure. We sell bottles, not CPUs
