Azure Data Lake Analytics

To make our work more accessible and more efficient, Azure Data Lake has provided fast access to and analysis of data. This product solves our need for quick reporting on cross-platform applications and bulk data from partners. We can manage on-premises access and roles because the analytics service integrates with Azure Active Directory. There are no clusters, virtual machines, or servers to manage, maintain, or fine-tune—the utility of a highly adaptable, Azure Blob Storage-based information lake that is also secure. Azure Data Lake Analytics' simple interface makes it a reliable and easy-to-use program for beginners. SQL benefits are combined with user code flexibility through the inclusion of U-SQL. Scalable distributed runtime for U-SQL allows us to analyze data across SQL Servers in Azure (SQL database and data warehouse) in a streamlined manner.

My primary use case in using and investigating Azure Data Lake Analytics was in comparing how it fulfilled aggregate build in our data lake environment compared to how Databricks solved for our initial use cases. At the time, in building out a raw, refined, and curated zone before landing data in a warehouse multiple bidirectional transformation processes run between the Refined to Curated and then ultimately Warehouse layer. Key was scale, cost, and performance as compared to what can be done in processing aggregates via Databricks and opposite that ELT to a warehouse like Snowflake instead of load from lake to Microsoft Synapse.

We have Azure Storage Blobs, which is traditionally one of the many ways that we would and probably more often and not the default way where we would store data. We make our data workforce by putting in Azure Storage Blobs We use the Azure SQL Database, a traditional SQL-based database. Microsoft makes that available to us in the Azure platform and we can host our data there. We also have a SQL Database, running an Azure on a Virtual Machine, if we don't want to use the Azure base SQL DB directly.

We have been using Azure Data Lake Analytics as one of our data lakes, we are collecting data from many different sources, storing it on the data lake, and processing this data. As result, we have Business Intelligence tools connected to this result which we use to present some KPIs.

Used Azure Data Lake Analytics while working for a CPG major to store/process/analyze large volumes of data (daily cadence). Used Python as a programming language for processing the stored data. Also, with fluctuating data volume across weekdays/weekends, ADL analytics was helpful in processing data on demand, and scale instantly, thereby enabling us to pay for the services used/rendered.

We utilize this solution for reporting on our storage usage.