TrustRadius Insights for Databricks Data Intelligence Platform are summaries of user sentiment data from TrustRadius reviews and, when necessary, third party data sources.
Pros
User-Friendly SQL: Users have found the SQL in Databricks to be user-friendly, allowing them to easily write and execute queries. Several reviewers have praised the intuitive nature of the SQL interface, making it accessible for users of different skill levels.
Enhanced Collaboration: The enhanced collaboration between data science and data engineering teams is seen as a positive feature by many users. They appreciate how Databricks facilitates seamless communication and knowledge sharing among team members, ultimately leading to improved productivity and efficiency.
Versatile Integration: The integration with multiple Git providers and the merge assistant is highly valued by users. This feature allows for smooth version control and simplifies the collaborative development process. With this capability, developers can easily manage their codebase, track changes, resolve conflicts, and ensure a streamlined workflow.
I use Databricks every day. We use multiple environments, such as dev, stage, and prod. Our primary use case is to get data from SnapLogic into the Bronze layer of Databricks. So, it handles both full and incremental loads as per the use case. Our workflows also support versioning across releases. This is really good for minimizing prod risks. We also perform many transformations on silver tables for end-use data products in the gold layer.
Pros
First, it handles large amounts of data. We run daily and weekly jobs that process a lot of records. Databricks manages it very well, with no issues, if the cluster is set up properly.
Second, it really works well for incremental updates. We load only new or changed data, which makes it easy to update existing tables without duplicating records.
Third, job scheduling is useful. We can schedule the jobs easily and monitor them. The best part is that we can retry or repair the failed runs.
The last one is about the notebook interface that I really love. It makes development and debugging easy. We can test logic step by step, validate data, and fix all our issues.
Cons
Sometimes, when multiple jobs depend on each other in different environments, it is not always easy to see the full workflow in one place.
It is sometimes difficult to determine which job or cluster contributes more to the overall cost.
For beginners, cluster configuration may be a little difficult. So more recommendation in the platform can help.
Likelihood to Recommend
Table merges: When we have to update existing tables with new records, Databricks makes it very simple and also reliable. The notebook environment helps multiple team members work together, test logic, and debug issues very quickly. It also works well when we need separate environments for dev and prod. Jobs can be tested safely in dev before moving to prod.
VU
Verified User
Engineer in Information Technology (201-500 employees)
Databricks is used to run all the data engineering, data science, analytics jobs related to spark , feature engineering jobs of data science and all the model training jobs of datascience. Databricks SQLwarehouse is also used as the main compute for serving the analytics powering the queries between 10 minutes to 60 minutes
Pros
Databricks provides an amazing notebook which serves and the backbone of unified analytics for all things data engineering, data science, and analytics
the performance of Databricks SQL Warehouse is top notch and very hard to beat by many competitors , and amazing performance optimization for delta file format
Databricks' support for iceberg format is also very good allowing for no vendor lock - in for delta format
Cons
The Unity catalog is restrictive in terms of policy definition due to user-defined functions compared to market alternatives like Apache Ranger or Open Policy Agent.
Unity Catalog has a table quota limitation where a schema with a higher number of tables requires increasing the table quota repeatedly , which is not an issue if you use Hive Metastore.
SQL Warehouse as a solution is amazing and performant, but there should have been some support to add Spark plugins or extensions so that there is some room for changes or adding custom dependencies
Likelihood to Recommend
for a new team with less number of users Databricks outshines due to the unified analytics , and provided all solutions from data ingestion to data lake ETL , to data consumption with a modern notebook . The Databricks GPU and Spark runtimes are also very well maintained and are regularly updated. So there is very little maintenance. If someone is using Databricks but does not use Unity Catalog, then they miss out on a lot of advanced features like SQL Warehouse serverless. Making Unity Catalog optional would have been easier for non-Unity Catalog customers to use the full capabilities of Databricks and the newer features that require Unity Catalog
In our organization, we use the Databricks Data Intelligence Platform as the main platform for building and managing data products. I use Databricks for creating notebooks for data transformation and creating data products and redirect them to project repositories and jobs scheduling using Databricks Workflows. And create business data products as delta tables in catalog (unity). It helps in solving big data manipulation/handling and jobs management.
Pros
Large Data Processing:- It handles large volumes of data efficiently using Spark for transforming data fulfilling business purposes and handles the jobs smoothly using clusters, workers.
Notebook-Oriented Development:- Databricks notebooks make development easy and flexible of data transformations using SQL, Python and R. Helps in testing the notebooks before deploying
Data Governance:- Provides data governance providing unity catalog for managing permissions security.
Cons
Job Monitoring and Alerts:- A better visual dashboard for pipeline tracking dependencies and failures would improve visibility.
Serverless limitation:- The sql variable set up using 'set' in notebooks is limited in serverless and can't be initialize, could be improved
Likelihood to Recommend
Based on my experience, Databricks Platform is well suited for huge-scale data processing and building end-to-end data pipelines. It works very well for developing complete data products using bronze, silver, and gold architecture.Well suited for faster development with the help of integrated Ai assistant. The notebook environment allows us to quickly develop, test and deploy.
Databricks is the primary data platform where we land, standardize, clean, transform, and clean our data sources. We utilize the Workflows feature to automate reoccurring tasks and have built internal applications around the reusable workflows. We use the dashboard feature internally to allow customer success teams and business analysts to keep tabs on the performance and outputs of our products. The workloads are orchestrated in Databricks but executed within our own AWS accounts, allowing us to stay compliant with our stringent security requirements.
Pros
Thoughtful application of AI assistants during the coding and analysis steps.
Intuitive UI for users of varying skill sets.
Frequently updated documentation.
Cons
Greater support for non spark workloads.
Ability to host JAR files on serverless endpoints.
Likelihood to Recommend
Medium to Large data throughput shops will benefit the most from Databricks Spark processing. Smaller use cases may find the barrier to entry a bit too high for casual use cases. Some of the overhead to kicking off a Spark compute job can actually lead to your workloads taking longer, but past a certain point the performance returns cannot be beat.
I use Databricks Lakehouse Platform in my Data Scienc & AI consulting company to help various business entities with data-driven solutions. The platform can handle large and complex data sets and enable us to build and deploy applications using the latest technologies. The opennness of Databricks allows us to seamlessly integrate and adapt to our clients requirements : * Creating dashboards with Tableau, Redash, Qlik, * Feed their CRM tool like Salesforce, SAP, * developing chatbots for Knowledge Management * Serve ML models behind API endpoints. Databricks Lakehouse Platform is a versatile and open product that saves us a lot of time, help us control cloud cost and human resources energy !
Pros
Enhanced Data Science & Data Engineering collaboration
Multiple Git providers integration with merge assistant
Cons
VsCode IDE support for local development
Python SDK for Workflows
Poetry support
Likelihood to Recommend
Databricks shines when you are working with a growing team of multiple data professions. By providing an easy to instantiate common workspace for Data Engineers, Data Scientist, ML Engineers and Data Analyst, fully integrated with Active Directory security, it makes your data projects more likely to go to production. No need to switch between tools, to transfer the data, the Unity Catalog will centralize all the assets and all your data citizens will find it in a second and can benefit from the Spark engine whatever language they use.
It would be less appropriate for very small data projects as the entry cost may be high. Yet, if the data is meant to grow, Databricks will horizontally scale without requiring a re-write of your codebase
I use Databricks Lakehouse Platform to build a data-science based solutions that adress many problems in my business. This includes: increment our data in the lake house and use Databricks Lakehouse Platform computational capabilities to analyze and feature engineer our data, build different machine learning model and track different experiment and finally register our trained model that can be used by the business.
Pros
MLFLOW Experiment
MLFLOW Registry
Databricks Lakehouse Platform Notebook
Cons
Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
Visualization in MLFLOW experiment can be enhanced
Likelihood to Recommend
Well Suited: Dealing with big data and being able to train different models that address many problems in my business. In addition to its computational capabilities, using Databricks Lakehouse Platform allowed us to do all development in one platform. Less Appropriate: Having a small dataset that doesn't need parallel processing. Local development is easier to develop and track so if no parallelization is needed (data is not big or parallelized computations is not required), I prefer local development.
We used Databricks Lakehouse platform for running all our Machine Learning workloads as well as storing large amounts of data in our data lake backend. The data stored in the databricks lakehouse was used to train state-of-the-art ML and Deep Learning models on text and image datasets. Databricks' Spark jobs as well as Delta Lake Lakehouse backend is well equipped for these kinds of tasks.
Pros
Very well optimized Spark Jobs Execution Engine.
Time travel in Databricks Lakehouse Platform allows you to version your datasets.
Newly integrated Analytics feature allows you to build visualization dashboards.
Native integration with managed MLflow service.
Cons
Running MLflow jobs remotely is extremely cluttered and needs to be simplified.
All the runnable code has to stay in Notebooks which are not very production-friendly.
File management on DBFS can be improved.
Likelihood to Recommend
If you need a managed big data megastore, which has native integration with highly optimized Apache Spark Engine and native integration with MLflow, go for Databricks Lakehouse Platform. The Databricks Lakehouse Platform is a breeze to use and analytics capabilities are supported out of the box. You will find it a bit difficult to manage code in notebooks but you will get used to it soon.
This product is used for Data Science project development, from data analysis/wrangling to feature creation, to training, to finetuning and to model test and validation, and finally to deployment. While Databricks is used by many users, we also use GitHub and code Q/A to promote a code in production. This is one of the advantages of Databricks is the integration part, not only Git but whether you use it on Azure or AWS, you can also leverage the power of the integrated Machine Learning in those platforms, such as auto ml or Azure ML.
Pros
Data Science code agnostic (SQL, R, Pyton, Pyspark, Scala)
Customer Service with REAL support from data eng. and data scientist
Integration with many technology : Tableau, Azure, AWS, Spark, etc.
Cons
Visualization
Collaboration
Likelihood to Recommend
Currently the best Data Science tool for a large-scale company that needs strong tech support once and a while. The performance and the connectivity/integration with a large bread of tools and platform is also important when you don't want to change all your stack. DataBricks is a great non-drage and drops tool for real Data Scientist that knows their things.
It is currently used by our Data and Product teams in order to perform deep dives analysis on how our current metrics are performing (KPIs, OKRs), to develop tools for metric predictions based on data models in languages such as SQL and Python while mixing them and giving to the entire company visibility of the results with graphs via shared workspaces
Pros
Cross company shared workspaces for unified comprehension of the data
Combining different languages such as SQL and Python in one single space in order to make data analysis
Quick execution of highly complex queries
Cons
How graphs are created, it requires a certain level of expertise in the platform and it could be more intuitive and user friendly
More guidance on the basics, since some of the new users come from different platforms expecting a similar UI
An option where all the tables are shown with their respective fields, when a DB is selected for a query
Likelihood to Recommend
I reckon is an amazing platform for users with a certain level of expertise for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, also it is very useful when it comes to cross company shared workspaces for unified comprehension of the data.
it is less appropriate for users who don't have full knowledge of the tables they are going to query on and need more support on the data, since the platform doesn't give an option to see what are the fields in a table before even querying it
We currently use the Databricks Lakehouse Platform for a client. My team specifically uses it to data-mine, create reports and analytics for the client. Depending on where the data is stored, various Analytics teams in my company use different platforms - GCP, AWS, Databricks, etc.
Pros
Scheduling jobs to automate queries
User friendly - a new user can easily navigate through SQL/Python queries
Options to code in multiple languages (SQL, Python, Scala, R) and easy to switch with the use of the % operator
Cons
Errors can be difficult to understand at times
Session resets automatically at times, which leads to the temporary tables being wiped out from memory
Git connections are dicey
Very inconsistent with job success/failure notification emails
Likelihood to Recommend
Databricks is great for beginner as well as advanced coders. The interface is extremely user-friendly and the learning curve is quite short. It is well suited for automation where we can have scripts running late at night when the load is less and wake up to an email notification of success or failure. It is also well suited for writing codes that require the use of multiple languages (in some cases of data modeling)
The ability to store temporary/permanent tables on data lakes is a fabulous feature as well. PySpark is an excellent language to learn and it works really fast with large datasets.