Databricks Data Intelligence Platform vs. Informatica Cloud Data Quality

Medium to Large data throughput shops will benefit the most from Databricks Spark processing. Smaller use cases may find the barrier to entry a bit too high for casual use cases. Some of the overhead to kicking off a Spark compute job can actually lead to your workloads taking longer, but past a certain point the performance returns cannot be beat.

Incentivized

For effective data collaboration, systematic verification of customer information, and address, among others, Informatica Data Quality is a fruitful application to consider. Besides, Informatica Data Quality controls quality through a cleansing process, giving the company a professional outline of candid data profiling and reputable analytics. Finally, Informatica Data Quality allows the simplistic navigation of content, with a dashboard that supports predictability.

Incentivized

• Process raw data in One Lake (S3) env to relational tables and views
• Share notebooks with our business analysts so that they can use the queries and generate value out of the data
• Try out PySpark and Spark SQL queries on raw data before using them in our Spark jobs
• Modern day ETL operations made easy using Databricks. Provide access mechanism for different set of customers

Incentivized

• The matching algorithms in IDQ are very powerful if you understand the different types that they offer (e.g., Hamming Distance, Jaro, Bigram, etc..). We had to play around with it to see which best suit our own needs of identifying and eliminating duplicate customers. Setting up the whole process (e.g., creating the KeyGenerator Transformation, setting up the matching threshold, etc..) can be somewhat time consuming and a challenge if you don't first standardize your data.
• The integration with PowerCenter is great if you have both. You can either import your mappings directly to PowerCenter or to an XML file. The only downside is that some of the transformations are unique to IDQ, so you are not really able to edit them once in PowerCenter.
• The standardizer transformation was key in helping us standardize our customer data (e.g., names, addresses, etc..). It was helpful due to having create a reference table containing the standardized value and the associated unstandardized values. What was great was that if you used Informatica Analyst, a business analyst could login and correct any of the values.

• Connect my local code in Visual code to my Databricks Lakehouse Platform cluster so I can run the code on the cluster. The old databricks-connect approach has many bugs and is hard to set up. The new Databricks Lakehouse Platform extension on Visual Code, doesn't allow the developers to debug their code line by line (only we can run the code).
• Maybe have a specific Databricks Lakehouse Platform IDE that can be used by Databricks Lakehouse Platform users to develop locally.
• Visualization in MLFLOW experiment can be enhanced

Incentivized

• Several partnerships diminishing the value of technologies
• Unable to get list of objects from Repository (like sources & targets) that don't have any dependency
• Scheduling: The built-in scheduling tool has many constraints such as handling Unix/VB scripts etc. Most enterprises use third party tools for this.

As pointed out earlier, due all the robust features IDQ has, our use f the product is successful and stable. IDQ is being used in multiple sources (from CRM application and in batch mode). As this is an iterative process, we are looking to improve our system efficiency using IDQ.

Because it is an amazing platform for designing experiments and delivering a deep dive analysis that requires execution of highly complex queries, as well as it allows to share the information and insights across the company with their shared workspaces, while keeping it secured.

in terms of graph generation and interaction it could improve their UI and UX

Incentivized

Easy to use not only for developers but also business users

Incentivized

The application works well except an occasional error out while using the system. It usually gets fixed when restarting the Infa server

Incentivized

Performance works just fine. It was able to load 200+ business terms, 150+ DQ automation, etc. very well.

Incentivized

One of the best customer and technology support that I have ever experienced in my career. You pay for what you get and you get the Rolls Royce. It reminds me of the customer support of SAS in the 2000s when the tools were reaching some limits and their engineer wanted to know more about what we were doing, long before "data science" was even a name. Databricks truly embraces the partnership with their customer and help them on any given challenge.

The most important differentiating factor for Databricks Lakehouse Platform from these other platforms is support for ACID transactions and the time travel feature. Also, native integration with managed MLflow is a plus. EMR, Cloudera, and Hortonworks are not as optimized when it comes to Spark Job Execution. Other platforms need to be self-managed, which is another huge hassle.

Incentivized

Has a suite of applications and components that we can integrate with Informatica Power center to deliver enterprise-strength data quality capabilities in a wide range of scenarios. Provides comprehensive and modular support for all data and all use cases whether in small or complex projects. Streamlined data discovery with a broad and deep lineup of enterprises.

Incentivized

Scalability works as expected and it is truly an enterprise system.

Incentivized

• The ability to spin up a BIG Data platform with little infrastructure overhead allows us to focus on business value not admin
• DB has the ability to terminate/time out instances which helps manage cost.
• The ability to quickly access typical hard to build data scenarios easily is a strength.

Incentivized

• Integration with tools like PowerCenter helped faster delivery of product, and at the same time conversion
• Reduce overall project cost due to bad data , bad quality, exceptions identified nearing go-live and post production
• Employee efficiency is increased exponentially due to more automated, customized tool