Talend Data Integration

We use the commercial Talend Data Integration and some open studio versions of their various tool sets. We primarily use it to consolidate data from several different databases. We are up to 30 across the four subject areas we refresh daily in our data lake / warehouse. The tools allow ease of access to various databases, including Oracle, SQL Server, Postgres, and Snowflake while providing access to blob storage on AWS and Google Cloud Platform.

We use Talend to transform data from non-relational databases into relational databases. Talend is extremely robust and versatile, so much so that we have even migrated from other ETL tools and trained employees in order to use Talend. It is versatile in the sense of readability, it can read many more types of files and you are able to interact much more with the data.

The free version of this software is infuriating to use for anything but the most simple projects. There is no good way to implement source control without paying for the software which is outrageously expensive per seat. Talend uses the Eclipse environment but disables any useful functionality for development. They insert keys into every text file - these key values change constantly so using a source control system like Git is next to impossible. With some difficulty, you can set the application up so you can compare versions of your project side-by-side, but you can not copy Talend objects across instances of Talend and it is not possible to open two projects up at the same time in one instance.

Additionally, projects open in new versions of Talend - even minor updates - are completely incompatible with older versions. As a result, you get stuck on the lowest common version. In our case, that version was discontinued without notification from the open-source repository. Totally unacceptable.

Yes, the open-source version is free, but it is little more than a demo if you are going to do anything complex or mission critical. If you can not compare versions of projects with one another this tool is useless and they are really only using open source as a sales tool.

Talend Data Integration is primarily used by the data warehousing team. While the management of data warehousing in handled by a single team, they work with data sourced from and sent to all departments across the entire organisation - this makes Talend Data Integration a critical software component for our use cases.

Talend is used to build a corporate data warehouse solution which centralizes all relevant company intel into a single repository. Talend is used for development only by the IT department (as expected) but the information it gathers is used for the whole company. The main problem which talend addresses is to unify different sources of intel by retrieving data from databases, files, web services, the internet, etc.