Azure Data Lake Storage

We have used both Hadoop and GCS buckets for our storage needs of very large healthcare data. In terms of comparison with the Hadoop distributed Files system, Azure Data Lake Storage always stands in a far better position due to easy integration with various latest and widely used Data engineering and Data Science tech stack like Azure Databricks, Data factory. In comparison to GCS, I like the UI and search feature of GCS buckets far better than Azure Data Lake Storage.

Azure Data Lake Storage from a functionality perspective is a much easier solution to work with. It's implementation from Amazon EMR went smooth, and continued usage is definitely better. However, Amazon EMR was significantly cheaper overall between the high transaction fees and cost of storage due to growth. The two both have their advantages and disadvantages, but the functionality of Azure Data Lake Storage outweighed it's cost.

We chose Azure Data Lake due to the fact that it was already a product under the Azure application suite. We didn't have to focus on integrating another 3rd party application within our environment. Also due to the fact Azure Data Lake scales its storage pools very efficiently, we decided to reduce Snowflake to a backup solution in case Azure data lake no longer worked out for us.

We decided long ago to develop for the Azure platform, so we only evaluate products from within Azure. And Azure Data Lake Storage is really the dominant offering within its space. But to give you a comparison, previously we used to use Azure SQL Database for our analytical workloads and then we decided to migrate because maintaining >1TB database was quite a challenge, both in terms of governance and cost.

Microsoft solutions provide great harmony in end-to-end data value creation, and Azure Data Lake Storage is highly compatible with other analytical solutions, e.g., Azure Data Factory and Databricks. So I would say that it is at the heart of the analytical solution in the company, from ADF to ingesting the data and creating insightful analytics in Azure Databricks.

AWS charges you on an hourly basis but Azure has a pricing model of per minute charge. In terms of short term subscriptions, Azure has more flexibility but it is more expensive. Azure has a much better hybrid cloud support in comparison with AWS. AWS provides direct connections whereas Azure express provides routing.

I am much more familiar with Snowflake. I thought it was fairly straightforward to use and did not have to learn much syntax. With Azure Data Lake Storage, I have had to learn some new syntax and thought there was a steeper learning curve. We selected it because of cost savings.

Simpler to use, in my opinion. It is also slightly cheaper.

We looked at the Amazon solution and it did not play as well with our existing tools and added a layer of maintenance that we were not willing to take on at the time. We thought that our Microsoft contract and support were good and that our internal team had the knowledge to support this over the long term vs outsourcing with Amazon Web Services.

The Azure Data Lake solution is designed for organizations that want to take advantage of big data. It provides a data platform that can help developers, data scientists, and analysts store data of any size and format and perform all types of processing and analytics across multiple platforms and programming languages. It can work with your existing solutions, such as identity management and security solutions. It also integrates with other data warehouses and cloud environments. It can be useful for organizations that need the above softwares.