Azure Databricks vs. pandas

Centralised notebooks are out directly into production. This can lead to poorly engineered code. It is very good for fast queries and our data team are always able to provide what we ask for. It is a big cost to our business so it is important it runs efficiently and returns on our investment.

Incentivized

Pandas are great for quick and relatively simple analytics and visualizations

Pandas work well for exploratory ad-hoc analytic work

But , We had little success in implementing complicated predictive analytics. And large data sizes can be a problem.

Incentivized

• Data Processing and Transformations based on Spark
• Delta Lakehouse when clubbed with an external cloud storage
• Governance using Unity Catalog to unify IAM
• Delta Live Tables is a product, which although relatively newer, has a great potential with the visuals of a pipeline.

Incentivized

• It is easy to do statistical analysis
• It is easy to clean the data
• It is easy to produce graphs and charts to visualize

Incentivized

• Intuitive interface
• Ease of use
• Providing FAQ or QRGs

Incentivized

• There are a lot of libraries and ways to do visualization. Sometimes it is very confusing.
• Error handling can be a challenge. Sometimes the error messages do not provide valuable clues for the debugging.
• In our case, there are a bunch of different frameworks and libraries working together. I would rather work with one framework, well tuned for my use case

Incentivized

The developers are able to switch between Python and SQL in the Notebook which allows the collaboration of SQL analyst and Data scientist. The integration of Mosaic AI allows users to write complex codes in natural languages. Unity catalog has centralized the security and governance features and simplified the process of maintaining it

Incentivized

Over the years, we tried a lot of different frameworks and tools, homegrown and commercial. Pandas provide the best results.

It is lightweight, flexible and easy to implement.

Incentivized

I have found Azure Databricks to be much better than Snowflake for handling bigger, diverse data types. Snowflake is much simpler and better for smaller warehousing. The real time processing is much better in Azure Databricks and we have much more language options. Snowflake is more expensive but simpler to use. Both are great for different needs.

Incentivized

All these frameworks are great for gathering data and providing some initial analysis. But for real performance debugging work one needs more than tools provided by this tools. That's where the pandas excel.

Incentivized

• The support team is amazing, they help you at every stage of the projects, from sales to delivery.
• On a framework level, it has had an amazing impact and has reduced the clients overall data platform costs by a staggering 65%
• There has been a 40% Manual work requirement on average for the clients when they move to Azure Databricks Data Platform

Incentivized

• Performance debugging was time consuming and mostly poorly automated exploratory process. Once we started use pandas for these tasks, it really moved the needle. Pandas are instrumental to provide actionable insights. As a result we were able to improve notably cloud software resource utilization and performance
• Analytics implemented with pandas allow us to detect and. address problems in our APIs before they are notable to our customers

Incentivized