Databricks Lakehouse Platform vs. Qubole

If you need a managed big data megastore, which has native integration with highly optimized Apache Spark Engine and native integration with MLflow, go for Databricks Lakehouse Platform. The Databricks Lakehouse Platform is a breeze to use and analytics capabilities are supported out of the box. You will find it a bit difficult to manage code in notebooks but you will get used to it soon.

Incentivized

I find Qubole is well suited for getting started analyzing data in the cloud without being locked in to a specific cloud vendor's tooling other than the underlying filesystem. Since the data itself is not isolated to any Qubole cluster, it can be easily be collected back into a cloud-vendor's specific tools for further analysis, therefore I find it complementary to any offerings such as Amazon EMR or Google DataProc.

Incentivized

• Process raw data in One Lake (S3) env to relational tables and views
• Share notebooks with our business analysts so that they can use the queries and generate value out of the data
• Try out PySpark and Spark SQL queries on raw data before using them in our Spark jobs
• Modern day ETL operations made easy using Databricks. Provide access mechanism for different set of customers

Incentivized

• From a UI perspective, I find Qubole's closest comparison to Cloudera's HUE; it provides a one-stop shop for all data browsing and querying needs.
• Auto scaling groups and auto-terminating clusters provides cost savings for idle resources.
• Qubole fits itself well into the open-source data science market by providing a choice of tools that aren't tied to a specific cloud vendor.

Incentivized

• Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
• Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
• Visualization in MLFLOW experiment can be enhanced

Incentivized

• Providing an open selection of all cloud provider instance types with no explanation as to their ideal use cases causes too much confusion for new users setting up a new cluster. For example, not everyone knows that Amazon's R or X-series models are memory optimized, while the C and M-series are for general computation.
• I would like to see more ETL tools provided other than DistCP that allow one to move data between Hadoop Filesystems.
• From the cluster administration side, onboarding of new users for large companies seems troublesome, especially when trying to create individual cluster per team within the company. Having the ability to debug and share code/queries between users of other teams / clusters should also be possible.

Incentivized

Personally, I have no issues using Amazon EMR with Hue and Zeppelin, for example, for data science and exploratory analysis. The benefits to using Qubole are that it offers additional tooling that may not be available in other cloud providers without manual installation and also offers auto-terminating instances and scaling groups.

Incentivized

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

Compared to Synapse & Snowflake, Databricks provides a much better development experience, and deeper configuration capabilities. It works out-of-the-box but still allows you intricate customisation of the environment. I find Databricks very flexible and resilient at the same time while Synapse and Snowflake feel more limited in terms of configuration and connectivity to external tools.

Qubole was decided on by upper management rather than these competitive offerings. I find that Databricks has a better Spark offering compared to Qubole's Zeppelin notebooks.

Incentivized

• The ability to spin up a BIG Data platform with little infrastructure overhead allows us to focus on business value not admin
• DB has the ability to terminate/time out instances which helps manage cost.
• The ability to quickly access typical hard to build data scenarios easily is a strength.

Incentivized

• We like to say that Qubole has allowed for "data democratization", meaning that each team is responsible for their own set of tooling and use cases rather than being limited by versions established by products such as Hortonworks HDP or Cloudera CDH
• One negative impact is that users have over-provisioned clusters without realizing it, and end up paying for it. When setting up a new cluster, there are too many choices to pick from, and data scientists may not understand the instance types or hardware specs for the datasets they need to operate on.

Incentivized