Azure Databricks vs. Informatica Cloud Data Quality

Suppose you have multiple data sources and you want to bring the data into one place, transform it and make it into a data model. Azure Databricks is a perfectly suited solution for this. Leverage spark JDBC or any external cloud based tool (ADG, AWS Glue) to bring the data into a cloud storage. From there, Azure Databricks can handle everything. The data can be ingested by Azure Databricks into a 3 Layer architecture based on the delta lake tables. The first layer, raw layer, has the raw as is data from source. The enrich layer, acts as the cleaning and filtering layer to clean the data at an individual table level. The gold layer, is the final layer responsible for a data model. This acts as the serving layer for BI For BI needs, if you need simple dashboards, you can leverage Azure Databricks BI to create them with a simple click! For complex dashboards, just like any sql db, you can hook it with a simple JDBC string to any external BI tool.

Incentivized

We used Informatica Data Quality to measure the "Data Quality Score" of internal and external reports at my company. Business users set up data profiling and prepared detailed analysis documents for business analysts. and developers developed Data Quality Mapplets for other IT teams to import their Informatica Power Center repositories. Results are stored in a centralized data quality space and then reported and summarized to related business users in detailed ways. At the end of each project, we are now able to place a "Data Quality Score" watermark score on each report involved.

Incentivized

• Unity Catalog
• Data Federation in Lakehouse Architecture
• Integration of Mosaic AI in the SQL Layer

Incentivized

• Watch the data real time- After creating the job the data quality engine checks and run the custom rules creating a navigation window at the bottom for review and accessing the data right away.
• Character Set Mapping
• Makes sense of our own data, which in turn gives us confidence that we can provide to the end users. IDQ helped us with erroneous data in accounting and HR for accurate and immaculate reports

• Their pipeline workflow orchestration is pretty primitive. Lacks some common features
• Workspace UI and navigation requires steep learning curve
• Personally, I am not fond of their autosave feature. Its dangerous for production level notebooks scripts

Incentivized

• Several partnerships diminishing the value of technologies
• Unable to get list of objects from Repository (like sources & targets) that don't have any dependency
• Scheduling: The built-in scheduling tool has many constraints such as handling Unix/VB scripts etc. Most enterprises use third party tools for this.

I gave a rating of 8 due to the fact that we use Informatica for both our data quality product and ETL product. Having both integrated makes it so much easier. Microsoft had a similar product of finding duplicates, but at the time it didn't seem mature enough. The usability also in IDQ was pretty easy to navigate and use.

Great for what we use day to day and does what we need it to do. Cost management is not fully developed across the UX and gets expensive very quickly for developing projects. Integrated very well with our Microsoft stack and can be worked on collaboratively which works well for us.

Incentivized

Easy to use not only for developers but also business users

Incentivized

The application works well except an occasional error out while using the system. It usually gets fixed when restarting the Infa server

Incentivized

Performance works just fine. It was able to load 200+ business terms, 150+ DQ automation, etc. very well.

Incentivized

I have found Azure Databricks to be much better than Snowflake for handling bigger, diverse data types. Snowflake is much simpler and better for smaller warehousing. The real time processing is much better in Azure Databricks and we have much more language options. Snowflake is more expensive but simpler to use. Both are great for different needs.

Incentivized

Informatica Data Quality has a wide range of cleansing features, that are detailed, professional, and accurate in scaling down the required database. Further, Informatica Data Quality ensures there is proper collaboration, and this fosters businesses to have the freedom of working closely with several programs. Finally, Informatica Data Quality design is authentic and allows personalization.

Incentivized

Scalability works as expected and it is truly an enterprise system.

Incentivized

• Helped reduce time for collecting data
• Reduced cost in maintaining multiple data sources
• Access for multiple users and management of users/data in a single platform

Incentivized

• Integration with tools like PowerCenter helped faster delivery of product, and at the same time conversion
• Reduce overall project cost due to bad data , bad quality, exceptions identified nearing go-live and post production
• Employee efficiency is increased exponentially due to more automated, customized tool