Databricks Data Intelligence Platform vs. Delta Lake vs. IBM watsonx.data

Medium to Large data throughput shops will benefit the most from Databricks Spark processing. Smaller use cases may find the barrier to entry a bit too high for casual use cases. Some of the overhead to kicking off a Spark compute job can actually lead to your workloads taking longer, but past a certain point the performance returns cannot be beat.

Incentivized

IBM watsonx.data is well suited for use cases were you have to combine various data sources to build a lakehouse. It provides a secure framework to gather data and provide access to it to build ML/AI models. It allows users to focus on prompts and business logic than spend time on data engineering.

Incentivized

• Process raw data in One Lake (S3) env to relational tables and views
• Share notebooks with our business analysts so that they can use the queries and generate value out of the data
• Try out PySpark and Spark SQL queries on raw data before using them in our Spark jobs
• Modern day ETL operations made easy using Databricks. Provide access mechanism for different set of customers

Incentivized

• It doesn't just store data but unlocks potential. I am able to analyse a vast amount of information, identify trends, and predict future outcomes.
• It not only gives me high quality but accessible data as well. It handles missing values, outliers and feature engineering with case.

Incentivized

• Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
• Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
• Visualization in MLFLOW experiment can be enhanced

Incentivized

• Integration complexity with Security Tools while watsonx.Data is well-suited for native tools, but integration with third-party security tools requires custom connectors or manual ETL pipelines. which leads to an increase in setup time.
• User interface and query time can be improved.

Incentivized

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

I can give it 10/10 due to its impact in data analysis management. This is the right software for driving business insights and enhancing effective decision making. The infrastructure has the formal tools for preparing data before using it to make critical decisions. The NLP has enhanced standard analysis of unstructured data from social media websites.

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

The most important differentiating factor for Databricks Lakehouse Platform from these other platforms is support for ACID transactions and the time travel feature. Also, native integration with managed MLflow is a plus. EMR, Cloudera, and Hortonworks are not as optimized when it comes to Spark Job Execution. Other platforms need to be self-managed, which is another huge hassle.

Incentivized

Pinecone and IBM watsonx.data (Milvus in our case) both work great as a full-managed cloud-based vector database. We selected IBM watsonx.data because it integrates well with watson.ai and is a little more beginner friendly than Pinecone, but I think both are great anyway.

• The ability to spin up a BIG Data platform with little infrastructure overhead allows us to focus on business value not admin
• DB has the ability to terminate/time out instances which helps manage cost.
• The ability to quickly access typical hard to build data scenarios easily is a strength.

Incentivized

• for one automation project, we managed to cut cloud storage costs by a third through IBM watsonx.data's lakehouse optimization
• data integration projects have had a 20 % reduction in turnaround times. Can only imagine how that will improve with the Claude partnership

Incentivized