Azure HDInsight vs. Databricks Data Intelligence Platform vs. IBM watsonx.data

Well suited: A tiny-mid sized company with no immediate plans of growing the volume of their data processing, that can afford long response times from support. Also it helps if you are not prone to put your hands on Linux and Spark configuration. In fact, it can make things go really faster if you also work with the bundle-in Jupyter. And, if you need to perform some diagnostics and / or administrative tasks, that's full of tools to find an understand the Root Cause. Ideal for non experts. Less appropriate: Big Data company, intense on demand cluster creation, mission critical, costs reduction, latest versions of libraries required, sophisticate customizations required.

Incentivized

Medium to Large data throughput shops will benefit the most from Databricks Spark processing. Smaller use cases may find the barrier to entry a bit too high for casual use cases. Some of the overhead to kicking off a Spark compute job can actually lead to your workloads taking longer, but past a certain point the performance returns cannot be beat.

Incentivized

Real-time transaction processing (both reads and writes) is where DataStax Enterprise shines. It's very fast with linear scalability should more resources be needed. Additional nodes are added very easily. DataStax Enterprise on its own (without Solr or Spark enabled) isn't well suited for long complicated reports. The data model doesn't support joining multiple tables together which is common in BI reporting.

• Data is presented without interfering others (IT or other dept).
• Data is managed properly and is available for retrievable any time.
• Legacy use of CD/DVD and Pendrive are not required.

Incentivized

• Process raw data in One Lake (S3) env to relational tables and views
• Share notebooks with our business analysts so that they can use the queries and generate value out of the data
• Try out PySpark and Spark SQL queries on raw data before using them in our Spark jobs
• Modern day ETL operations made easy using Databricks. Provide access mechanism for different set of customers

Incentivized

• Datastax Cassandra provides high availability and good performance for a database. It is built on top of open source Apache Cassandra so you can always somewhat understand the internal functioning and why.
• Datastax Cassandra is fairly simple to start using, you can install/setup your cluster and be productive in 1 day.
• Datastax Cassandra provides a lot of good detailed documentation, and when starting, the detailed free videos on the Datastax site and documentation are very helpful.
• Datastax Enterprise Edition of Cassandra provides more tools, good support, and quick response SLA for enterprise business support.

Incentivized

• The only problem I have come across is when loading large volumes of data I sometimes get an error message, I assume this means something is corrupt from within. I would love a way for this to be resolved without having to start over.

Incentivized

• Sometimes, when multiple jobs depend on each other in different environments, it is not always easy to see the full workflow in one place.
• It is sometimes difficult to determine which job or cluster contributes more to the overall cost.
• For beginners, cluster configuration may be a little difficult. So more recommendation in the platform can help.

Incentivized

• Integration complexity with Security Tools while watsonx.Data is well-suited for native tools, but integration with third-party security tools requires custom connectors or manual ETL pipelines. which leads to an increase in setup time.
• User interface and query time can be improved.

Incentivized

As an open source technology Cassandra can be readily used with or without any commercial support. DataStax provides value-added services and features, and in the end it is up to individual situations to strike a balance between the desirability of such support/service versus the associated cost.

Incentivized

Azure HDInsight is usable on the top of Azure Data Lake and gives us the benefit of analyzing large scale data workload in Hadoop. Usability and support from Microsoft are outstanding.

Incentivized

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

DataStax has a good community built around it and has amazing scalability options. Though the initial setup is a bit costly, in the long run, it makes up for it. It also has powerful monitoring tools and a clean UI.

Incentivized

good recovery features

Incentivized

scalable product

Incentivized

Inexpert, isolated teams... not good for support an excessively complex platform. Lots of weeks or months for a complex problem troubleshoot. Many time lost stuck on MindTree, before the case was finally escalated with Microsoft!

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

We have had a few situations where we caused an outage or something has gone wrong and we are able to get a support person to offer live help within minutes. The escalation process is excellent - the best I've seen - and the support team is incredibly strong. Outside of emergencies, the team is very helpful with general questions and working through data model exercises and the subscription I believe still comes with some hours to help get the data model reviewed.

easy to follow documentation, support is there when needed

Incentivized

use saas service

Incentivized

At this time I have not used any other similar products... I am open to it but Azure HDInsight and its components really work well for our organization.

Incentivized

The most important differentiating factor for Databricks Lakehouse Platform from these other platforms is support for ACID transactions and the time travel feature. Also, native integration with managed MLflow is a plus. EMR, Cloudera, and Hortonworks are not as optimized when it comes to Spark Job Execution. Other platforms need to be self-managed, which is another huge hassle.

Incentivized

Pinecone and IBM watsonx.data (Milvus in our case) both work great as a full-managed cloud-based vector database. We selected IBM watsonx.data because it integrates well with watson.ai and is a little more beginner friendly than Pinecone, but I think both are great anyway.

cognos integration works great

Incentivized

• ROI is of course there, as no legacy software for data presentation.
• No manual intervention for data retrieval.
• Data is available anywhere as requested.

Incentivized

• The ability to spin up a BIG Data platform with little infrastructure overhead allows us to focus on business value not admin
• DB has the ability to terminate/time out instances which helps manage cost.
• The ability to quickly access typical hard to build data scenarios easily is a strength.

Incentivized

• for one automation project, we managed to cut cloud storage costs by a third through IBM watsonx.data's lakehouse optimization
• data integration projects have had a 20 % reduction in turnaround times. Can only imagine how that will improve with the Claude partnership

Incentivized