Azure Databricks vs. PostgreSQL

Suppose you have multiple data sources and you want to bring the data into one place, transform it and make it into a data model. Azure Databricks is a perfectly suited solution for this. Leverage spark JDBC or any external cloud based tool (ADG, AWS Glue) to bring the data into a cloud storage. From there, Azure Databricks can handle everything. The data can be ingested by Azure Databricks into a 3 Layer architecture based on the delta lake tables. The first layer, raw layer, has the raw as is data from source. The enrich layer, acts as the cleaning and filtering layer to clean the data at an individual table level. The gold layer, is the final layer responsible for a data model. This acts as the serving layer for BI For BI needs, if you need simple dashboards, you can leverage Azure Databricks BI to create them with a simple click! For complex dashboards, just like any sql db, you can hook it with a simple JDBC string to any external BI tool.

Incentivized

PostgreSQL, unlike other databases, is user-friendly and uses an open-source database. Ideal for relational databases, they can be accessed when speed and efficiency are required. It enables high-availability and disaster recovery replication from instance to instance. PostgreSQL can store data in a JSON format, including hashes, keys, and values. Multi-platform compatibility is also a big selling point. We could, however, use all the DBMS’s cores. While it works well in fast environments, it can be problematic in slower ones or cause multiple master replication.

• SQL
• Data management
• Data access

Incentivized

• The stability it offers, its speed of response and its resource management is excellent even in complex database environments and with low-resource machines.
• The large amount of resources it has in addition to the many own and third-party tools that are compatible that make productivity greatly increase.
• The adaptability in various environments, whether distributed or not, [is a] complete set of configuration options which allows to greatly customize the work configuration according to the needs that are required.
• The excellent handling of referential and transactional integrity, its internal security scheme, the ease with which we can create backups are some of the strengths that can be mentioned.

Incentivized

• Their pipeline workflow orchestration is pretty primitive. Lacks some common features
• Workspace UI and navigation requires steep learning curve
• Personally, I am not fond of their autosave feature. Its dangerous for production level notebooks scripts

Incentivized

• The query syntax for JSON fields is unwieldy when you start getting into complex queries with many joins.
• I wish there was a distinction (a flag) you could set for automated scripts vs working in the psql CLI, which would provide an 'Are you sure you want to do X?' type prompt if your query is likely to affect more than a certain number of rows. Especially on updates/deletes. Setting the flag in the headless(scripted) flow would disable the prompt.
• Better documentation around JSON and Array aggregation, with more examples of how the data is transformed.

Incentivized

As a needed software for day to day development activities

Incentivized

Based on my extensive use of Azure Databricks for the past 3.5 years, it has evolved into a beautiful amalgamation of all the data domains and needs. From a data analyst, to a data engineer, to a data scientist, it jas got them all! Being language agnostic and focused on easy to use UI based control, it is a dream to use for every Data related personnel across all experience levels!

Incentivized

Postgresql is the best tool out there for relational data so I have to give it a high rating when it comes to analytics, data availability and consistency, so on and so forth. SQL is also a relatively consistent language so when it comes to building new tables and loading data in from the OLTP database, there are enough tools where we can perform ETL on a scalable basis.

Incentivized

PostgreSQL's availability is top notch. Apart from connection time-out for an idle user, the database is super reliable.

Incentivized

The data queries are relatively quick for a small to medium sized table. With complex joins, and a wide and deep table however, the performance of the query has room for improvement.

Incentivized

There are several companies that you can contract for technical support, like EnterpriseDB or Percona, both first level in expertise and commitment to the software.

But we do not have contracts with them, we have done all the way from googling to forums, and never have a problem that we cannot resolve or pass around. And for dozens of projects and more than 15 years now.

Incentivized

The online training is request based. Had there been recorded videos available online for potential users to benefit from, I could have rated it higher. The online documentation however is very helpful. The online documentation PDF is downloadable and allows users to pace their own learning. With examples and code snippets, the documentation is great starting point.

Incentivized

The online documentation of the PostgreSQL product is elaborate and takes users step by step.

Incentivized

Against all the tools I have used, Azure Databricks is by far the most superior of them all! Why, you ask? The UI is modern, the features are never ending and they keep adding new features. And to quote Apple, "It just works!" Far ahead of the competition, the delta lakehouse platform also fares better than it counterparts of Iceberg implementation or a loosely bound Delta Lake implementation of Synapse

Incentivized

Postgres stacks up just [fine] along the other big players in the RDBMS world. It's very popular for a reason. It's very close to MySQL in terms of cost and features - I'd pick either solution and be just as happy. Compared to Oracle it is a MUCH cheaper solution that is just as usable.

Incentivized

The DB is reliable, scalable, easy to use and resolves most DB needs

Incentivized

• Helped reduce time for collecting data
• Reduced cost in maintaining multiple data sources
• Access for multiple users and management of users/data in a single platform

Incentivized

• The user-role system has saved us tons of time and thus money. As I mentioned in the "Use Case" section, Postgres is not only used by engineering but also finance to measure how much to charge customers and customer support to debug customer issues. Sure, it's not easy for non-technical employees to psql in and view raw tables, but it has saved engineering hundreds of man-hours that would have had to be spent on building equivalent tools to serve finance or customer support.
• It provides incredibly trustworthy storage for wherever customer data dumped in. In our 6 years of Postgres existence, we have not lost a byte of customer data due to Postgres messing up a transaction or during the multiple times the hard-drives failed (thanks to ACID compliance!).
• This is less significant, but Postgres is also quite easy to manage (unless you are going above and beyond to squeeze out every last bit of performance). There's not much to configure, and the out of the box settings are quite sane. That has saved us engineers lots of time that would have gone into Postgres administration.

Incentivized