Databricks Data Intelligence Platform vs. IBM ILOG CPLEX Optimization Studio

If you need a managed big data megastore, which has native integration with highly optimized Apache Spark Engine and native integration with MLflow, go for Databricks Lakehouse Platform. The Databricks Lakehouse Platform is a breeze to use and analytics capabilities are supported out of the box. You will find it a bit difficult to manage code in notebooks but you will get used to it soon.

Incentivized

In my opinon, if the problem is less than 5000 variables, one should try to solve with free available solver rather than directly going for a commercial license of IBM CPLEX Optimization Studio. In my opinion, if the priority is not in terms of solving time with higher number of variables, even then one can go for free solvers like CBC, IPOPT, SCIP. In my opinion, if priority is solving time and number of variables is also high, only in that case one should prefer going for a commercial license.

• First, it handles large amounts of data. We run daily and weekly jobs that process a lot of records. Databricks manages it very well, with no issues, if the cluster is set up properly.
• Second, it really works well for incremental updates. We load only new or changed data, which makes it easy to update existing tables without duplicating records.
• Third, job scheduling is useful. We can schedule the jobs easily and monitor them. The best part is that we can retry or repair the failed runs.
• The last one is about the notebook interface that I really love. It makes development and debugging easy. We can test logic step by step, validate data, and fix all our issues.

Incentivized

• Linear Programming
• Mixed-Integer Linear Programming
• Non-Linear Convex-Optimization
• Visualization
• Shadow Price Analysis
• Parameter Tuning

• Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
• Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
• Visualization in MLFLOW experiment can be enhanced

Incentivized

• Data handling from different sources like Note Pad, etc.
• Large size of MILP problems.
• Various parameters to set.

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

It's nice to use and with good optimization.

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

Honestly, to say, I never contacted CPLEX but used its forum to know/clarify any issues I faced.

Databricks is a true all-in-one platform, and at the time of implementation, it had more features available to us, making it a clear choice over Snowflake. Moving our workloads from local computing to the servers in Databricks gave our start-up staff a great quality of life boost.

Incentivized

Compared with MATLAB, CPLEX is a more user-friendly and simpler structure for writing models. This one also has a good return on investment.

• ROI for us has been tremendous. Time to market by processing raw data in our big data infrastructure has been pretty fast.
• Non engineers can easily use Databricks, hence helping business customers.
• Thousands of different data combinations can easily be joined and used by our data teams.

Incentivized

• Better for price.
• Many model parameters/features.
• No visualization.