Good platform for Data engineers.
Rating: 9 out of 10
IncentivizedUse Cases and Deployment Scope
I use Databricks every day. We use multiple environments, such as dev, stage, and prod. Our primary use case is to get data from SnapLogic into the Bronze layer of Databricks. So, it handles both full and incremental loads as per the use case. Our workflows also support versioning across releases. This is really good for minimizing prod risks. We also perform many transformations on silver tables for end-use data products in the gold layer.
Pros
- First, it handles large amounts of data. We run daily and weekly jobs that process a lot of records. Databricks manages it very well, with no issues, if the cluster is set up properly.
- Second, it really works well for incremental updates. We load only new or changed data, which makes it easy to update existing tables without duplicating records.
- Third, job scheduling is useful. We can schedule the jobs easily and monitor them. The best part is that we can retry or repair the failed runs.
- The last one is about the notebook interface that I really love. It makes development and debugging easy. We can test logic step by step, validate data, and fix all our issues.
Cons
- Sometimes, when multiple jobs depend on each other in different environments, it is not always easy to see the full workflow in one place.
- It is sometimes difficult to determine which job or cluster contributes more to the overall cost.
- For beginners, cluster configuration may be a little difficult. So more recommendation in the platform can help.
Likelihood to Recommend
Table merges: When we have to update existing tables with new records, Databricks makes it very simple and also reliable. The notebook environment helps multiple team members work together, test logic, and debug issues very quickly. It also works well when we need separate environments for dev and prod. Jobs can be tested safely in dev before moving to prod.
Vetted Review
2 years of experience