Databricks Data Intelligence Platform vs. IBM StreamSets

If you need a managed big data megastore, which has native integration with highly optimized Apache Spark Engine and native integration with MLflow, go for Databricks Lakehouse Platform. The Databricks Lakehouse Platform is a breeze to use and analytics capabilities are supported out of the box. You will find it a bit difficult to manage code in notebooks but you will get used to it soon.

Incentivized

When you are dealing with a data warehouse and want to find an easy way to integrate applications and expose data in real-time, then IBM StreamSets is the best tool to go for. I'm using it for the same purpose in my applications. This tool will be well-suited for someone with a proper technical background. Though IBM StreamSets UI is mostly drag and drop, advanced configurations require technical expertise or support to do the initial setup.

Incentivized

• First, it handles large amounts of data. We run daily and weekly jobs that process a lot of records. Databricks manages it very well, with no issues, if the cluster is set up properly.
• Second, it really works well for incremental updates. We load only new or changed data, which makes it easy to update existing tables without duplicating records.
• Third, job scheduling is useful. We can schedule the jobs easily and monitor them. The best part is that we can retry or repair the failed runs.
• The last one is about the notebook interface that I really love. It makes development and debugging easy. We can test logic step by step, validate data, and fix all our issues.

Incentivized

• It makes building data pipelines quite super intuitive even for non coders.
• Ir also handles real time data ingestion effortlessly so I always have up to date information for my reports.
• It's great at monitoring data quality as well.

Incentivized

• Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
• Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
• Visualization in MLFLOW experiment can be enhanced

Incentivized

• Where the person's skillsets in data analysis is not of an expert.
• Data monitoring and analysis.
• Customer data for better customer acquisition

Incentivized

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

because i think that overall the solution is having a positive impact on the business, it allows multiple benefits in simplification of the tasks and is capable of doing multiple process that are usually done by a combination of man and systems, reducing the time and effort required to have the data.

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

Databricks is a true all-in-one platform, and at the time of implementation, it had more features available to us, making it a clear choice over Snowflake. Moving our workloads from local computing to the servers in Databricks gave our start-up staff a great quality of life boost.

Incentivized

At Unify Logistics, we chose IBM StreamSets over Fivetran for its flexibility in handling complex, real time data pipelines across hybrid environments. While Fivetran offers simplicity and fast setup, StreamSets provides deeper customization, better data drift handling, and stronger support for dynamic logistics workflows.

Incentivized

• ROI for us has been tremendous. Time to market by processing raw data in our big data infrastructure has been pretty fast.
• Non engineers can easily use Databricks, hence helping business customers.
• Thousands of different data combinations can easily be joined and used by our data teams.

Incentivized

• Huge reduction in maintenance as the StreamSets pipelines are resilient.
• Eliminated the need for other data ingestion tools, saving us hundreds of thousands annually.

Incentivized