Talend Data Integration

We use the commercial Talend Data Integration and some open studio versions of their various tool sets. We primarily use it to consolidate data from several different databases. We are up to 30 across the four subject areas we refresh daily in our data lake / warehouse. The tools allow ease of access to various databases, including Oracle, SQL Server, Postgres, and Snowflake while providing access to blob storage on AWS and Google Cloud Platform.

We use Talend Data Integration in Data Engineering and Profiling departments to develop ETL. We use it to move data from relational data bases to mongodb.

The free version of this software is infuriating to use for anything but the most simple projects. There is no good way to implement source control without paying for the software which is outrageously expensive per seat. Talend uses the Eclipse environment but disables any useful functionality for development. They insert keys into every text file - these key values change constantly so using a source control system like Git is next to impossible. With some difficulty, you can set the application up so you can compare versions of your project side-by-side, but you can not copy Talend objects across instances of Talend and it is not possible to open two projects up at the same time in one instance.

Additionally, projects open in new versions of Talend - even minor updates - are completely incompatible with older versions. As a result, you get stuck on the lowest common version. In our case, that version was discontinued without notification from the open-source repository. Totally unacceptable.

Yes, the open-source version is free, but it is little more than a demo if you are going to do anything complex or mission critical. If you can not compare versions of projects with one another this tool is useless and they are really only using open source as a sales tool.

Talend Data Integration is used across our organization. We perform Extract, Transform, and Load using Talend. We integrate various systems such as Oracle Database, Clarify, Salesforce, Sugar CRM, SQL DB, MQ, Stibo Step, FTP, Netezza, and Files using Talend Data Integration. We also migrate a lot of data between the system on a one-time basis.

Talend Data Integration is primarily used by the data warehousing team. While the management of data warehousing in handled by a single team, they work with data sourced from and sent to all departments across the entire organisation - this makes Talend Data Integration a critical software component for our use cases.

We have a high amount of excel files coming in from different departments. They all have a different data structure. It was very difficult integrating all the types of data, with different data sizes. We used Talend to integrate data in one format. We had to perform data analysis, pattern recognition. It had a big impact on overall efficiency. We also found several bottlenecks, and narrowed our focus area.

We use the big data edition of Talend Integration Suite. It is used by the Business Intelligence department for extract transform load (ETL). There are more than 400 jobs that pull data or push data from various sources. Talend is used to load data into the enterprise data warehouse. (EDW). I use the big data edition to pull data from hive cluster. There are complex sources like highly nested JSON and XML that can be easily parsed using Talend components. The best thing I like about Talend is the ease of use/debug and ease of deployment. Behind the scenes it uses Java coding, so whatever you do in the interface can be easily visible under code. Packaging and deployment of code is easy too in any environment (Windows or Mac or Linux) with any version of Java compatibility.

Talend is used to build a corporate data warehouse solution which centralizes all relevant company intel into a single repository. Talend is used for development only by the IT department (as expected) but the information it gathers is used for the whole company. The main problem which talend addresses is to unify different sources of intel by retrieving data from databases, files, web services, the internet, etc.

Talend Integration Suite has been chosen as the enterprise data integration tool for our client. We were using Informatica Power Center before that. Currently we are recreating the existing power center mapping in Talend to decommission Informatica. All new projects are using Talend DI. Talend has data integration, big data integration, data profiling and master data management components which are widely used in our projects based upon the data requirements.