Azure Databricks vs. TensorFlow

Centralised notebooks are out directly into production. This can lead to poorly engineered code. It is very good for fast queries and our data team are always able to provide what we ask for. It is a big cost to our business so it is important it runs efficiently and returns on our investment.

Incentivized

TensorFlow is great for most deep learning purposes. This is especially true in two domains: 1. Computer vision: image classification, object detection and image generation via generative adversarial networks 2. Natural language processing: text classification and generation. The good community support often means that a lot of off-the-shelf models can be used to prove a concept or test an idea quickly. That, and Google's promotion of Colab means that ideas can be shared quite freely. Training, visualizing and debugging models is very easy in TensorFlow, compared to other platforms (especially the good old Caffe days). In terms of productionizing, it's a bit of a mixed bag. In our case, most of our feature building is performed via Apache Spark. This means having to convert Parquet (columnar optimized) files to a TensorFlow friendly format i.e., protobufs. The lack of good JVM bindings mean that our projects end up being a mix of Python and Scala. This makes it hard to reuse some of the tooling and support we wrote in Scala. This is where MXNet shines better (though its Scala API could do with more work).

Incentivized

• Data Processing and Transformations based on Spark
• Delta Lakehouse when clubbed with an external cloud storage
• Governance using Unity Catalog to unify IAM
• Delta Live Tables is a product, which although relatively newer, has a great potential with the visuals of a pipeline.

Incentivized

• A vast library of functions for all kinds of tasks - Text, Images, Tabular, Video etc.
• Amazing community helps developers obtain knowledge faster and get unblocked in this active development space.
• Integration of high-level libraries like Keras and Estimators make it really simple for a beginner to get started with neural network based models.

Incentivized

• Intuitive interface
• Ease of use
• Providing FAQ or QRGs

Incentivized

• RNNs are still a bit lacking, compared to Theano.
• Cannot handle sequence inputs
• Theano is perhaps a bit faster and eats up less memory than TensorFlow on a given GPU, perhaps due to element-wise ops. Tensorflow wins for multi-GPU and “compilation” time.

The developers are able to switch between Python and SQL in the Notebook which allows the collaboration of SQL analyst and Data scientist. The integration of Mosaic AI allows users to write complex codes in natural languages. Unity catalog has centralized the security and governance features and simplified the process of maintaining it

Incentivized

Support of multiple components and ease of development.

Incentivized

Community support for TensorFlow is great. There's a huge community that truly loves the platform and there are many examples of development in TensorFlow. Often, when a new good technique is published, there will be a TensorFlow implementation not long after. This makes it quick to ally the latest techniques from academia straight to production-grade systems. Tooling around TensorFlow is also good. TensorBoard has been such a useful tool, I can't imagine how hard it would be to debug a deep neural network gone wrong without TensorBoard.

Incentivized

Use of cloud for better execution power is recommended.

Incentivized

I have found Azure Databricks to be much better than Snowflake for handling bigger, diverse data types. Snowflake is much simpler and better for smaller warehousing. The real time processing is much better in Azure Databricks and we have much more language options. Snowflake is more expensive but simpler to use. Both are great for different needs.

Incentivized

Keras is built on top of TensorFlow, but it is much simpler to use and more Python style friendly, so if you don't want to focus on too many details or control and not focus on some advanced features, Keras is one of the best options, but as far as if you want to dig into more, for sure TensorFlow is the right choice

Incentivized

• The support team is amazing, they help you at every stage of the projects, from sales to delivery.
• On a framework level, it has had an amazing impact and has reduced the clients overall data platform costs by a staggering 65%
• There has been a 40% Manual work requirement on average for the clients when they move to Azure Databricks Data Platform

Incentivized

• Learning is s bit difficult takes lot of time.
• Developing or implementing the whole neural network is time consuming with this, as you have to write everything.
• Once you have learned this, it make your job very easy of getting the good result.

Incentivized