Data Integration Tools Overview
What are Data Integration Tools?
The need for data integration emerges from complex data center environments where multiple different systems are creating large volumes of data. This data must be understood in aggregate, rather than in isolation. Data integration is nothing more than a technique and technology for providing a unified and consistent view of enterprise-wide data.
Data Integration Tools Features & Capabilities
Ability to process data from a wide variety of sources such as mainframes, enterprise applications, spreadsheets, proprietary databases, etc.
Ability to process unstructured data from social media, email, web pages, etc.
Syntactic and semantic checks to make sure data conforms to business rules and policies
Deduplication and removal of incorrectly or improperly formatted data
Support for metadata
Types of Data Integration
There are several different approaches to achieving this goal which are quite different to each other and essentially solve slightly different problems: The main technologies for data integration are Extract, Transform Load (ETL), Enterprise Application Integration (EAI), and Enterprise Information Integration (EII), or data virtualization as it is more often called today.
Products listed in this category belong to the ETL data integration approach. Unlike the other listed approaches, ETL is designed for data migration and integration of large volumes of data to provide a basis for decision-making.
What is ETL?
ETL is a process whereby large volumes of required data are extracted from various databases and converted into a common format. The data is then cleaned, and loaded into the specialized reporting database called a data warehouse. It is then available for standard reporting purposes.
The data used in ETL can come from any source including flat files, Excel data, application data like CRM or ERP data, or mainframe application data. Perhaps the most difficult part of the process is the “Transform” component. Here, not only must the data be cleansed and any duplicates removed, but the software also has to resolve data consistency issues. It applies rules to consistently convert data to the appropriate form for the data warehouse or repository.
Once the data has been loaded into a data warehouse it is available for querying by business intelligence front-end processes that can pull consolidated data into reports and dashboards.
Shortcomings of Data Warehouses
One shortcoming of the data warehouse approach is that the data is not always current. Data warehouses pull data from databases periodically in batches, not in real time. If the data in the source database has changed, this might not be reflected in the data in the warehouse. Various strategies can be employed to achieve “real-time ETL”, although some of them place a significant load on the database. This can have performance repercussions.
The simplest thing to do is simply increase the frequency of batch updates to near real-time processing. But there are other solutions including continuously feeding the database using real-time data transport technologies, the use of staging tables, or a real-time data cache.
Pricing Information
Enterprise-level data integration tools can be very expensive with some products costing upwards of $10,000 per user per year. On top of that, you may need to pay for professional services to get up and running. SMB solutions are significantly cheaper than this.
