Databricks Data Intelligence Platform vs. Vertex AI

Medium to Large data throughput shops will benefit the most from Databricks Spark processing. Smaller use cases may find the barrier to entry a bit too high for casual use cases. Some of the overhead to kicking off a Spark compute job can actually lead to your workloads taking longer, but past a certain point the performance returns cannot be beat.

Incentivized

We needed to build some ML models and Vertex AI is one such place where all the functionalities are consolidated for model deployment, tweaking and monitoring. And although it is costly, it is compatible with google cloud services and is scalable which makes it the perfect tool for us.

Incentivized

• Process raw data in One Lake (S3) env to relational tables and views
• Share notebooks with our business analysts so that they can use the queries and generate value out of the data
• Try out PySpark and Spark SQL queries on raw data before using them in our Spark jobs
• Modern day ETL operations made easy using Databricks. Provide access mechanism for different set of customers

Incentivized

• Vertex AI comes with support for LOTs of LLMs out of the box
• MLOps tools are available that help to standardize operational aspects
• Document AI is an out of the box feature that works just perfectly for our use cases of automating lots to tedious data extraction tasks from images as well as papers

Incentivized

• Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
• Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
• Visualization in MLFLOW experiment can be enhanced

Incentivized

• Customization of AutoML models - A must needed capability to be able to tweak hyperparameters and also working with different models
• Model Explainability -Providing more comprehensive explanations about how models are utilizing features could be very beneficial
• Model versioning and experiments tracking - Enhancing the versioning capability could be good for end users

Incentivized

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

Vertex AI is very user friendly and it is very fast. We can spin up instances pretty quickly and it allows to work with GPU instances without much planning. Pages load up very quickly and tasks are competed pretty much in time frame. It integrates with other systems also very well.

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

The most important differentiating factor for Databricks Lakehouse Platform from these other platforms is support for ACID transactions and the time travel feature. Also, native integration with managed MLflow is a plus. EMR, Cloudera, and Hortonworks are not as optimized when it comes to Spark Job Execution. Other platforms need to be self-managed, which is another huge hassle.

Incentivized

Vertex AI is much more accessible to non-developers than IBM's product. Moreover, Vertex AI integrates well with other Google products, enhancing its capabilities. A big plus is its integration with cloud storage, that allows for better management and access of data. In all honesty, it wasn't much of a difficult choice to choose Vertex AI.

Incentivized

• The ability to spin up a BIG Data platform with little infrastructure overhead allows us to focus on business value not admin
• DB has the ability to terminate/time out instances which helps manage cost.
• The ability to quickly access typical hard to build data scenarios easily is a strength.

Incentivized

• It is pay as you go model so it'll save more cost of your org. In our case previously we used to incurred 1-2L/Month now we are reduced it to 80k-1L.
• It'll help you save your model training & model selection time as it provides pre-trained models in autoML.
• It'll help you in terms of Security wherein we can use row level security access to authorized persons.

Incentivized