Item: Matillion
Rating: 8
Author: Polina Moore

Use Cases and Deployment Scope

We use Matillion as the new ETL tool for building Data infrastructure for the company. Our Data team is limited to 2 Matillion users but the Data infrastructure will be used by the BI team to help all areas of the business. So far, we've used it for various data loading tasks but are starting to use for the new Data Warehouse processes we'll be building.

Pros and Cons

Data Mapping - it is really good at recognizing the source and destination data types and does a great job at converting data types.
User-Friendly Interface - development is very easy within the tool and very intuitive for Data experts.
Troubleshooting - for the most part, it gives informative messages when errors occur and it is extremely useful to see data transformation 'in-flight' through data sampling.

Deployment to other Environment - this is not a smooth process and needs improvements. Better integration with code repository and smooth deployment of changes is needed (for example, just exporting variable default values is very clunky).
Documentation - for the version of Matillion we are using, we can't export documentation for our jobs. Since this is the level of functionality we need, it sounds unreasonable that to get a couple of more features (one of them being Documentation), we would need to upgrade to a higher level of license that would cost twice as much).
Running queries on other databases - although we primarily want to load data into Redshift, we sometimes need to run quick queries as part of the process against non-Redshift databases (that are not straight loads from the DB to Redshift). Although we can do this through Python, it would be a lot more helpful to be able to do this within Matillion components.

Return on Investment

Too early in the process to tell but it has made a lot of development easier and faster compared to other tools used in this company.

Usability

The user interface is great for development, the processes of deploying jobs to Prod are not. Great to have Python to complement functionality that's missing but seems like having to use it too much sometimes as the functionality needed is missing (for example, even the simple tasks of deleting files at the source once processed).

Time to Value

We started testing and developing the initial jobs right after standing Matillion up. We are continuously developing new jobs as we keep moving through the data projects.

Scalability

We have run into certain limitations already that we didn't expect to hit until the volume of jobs we have has risen more significantly. Because of budget concerns, we would like to stay within the license we currently have but not sure when we would hit the point where a lot of degradation occurs.

Alternatives Considered

In this company, we went through review of other products that work with Redshift. We have used Talend as the previous ETL tool.

Other Software Used

Talend Data Integration, Microsoft Power BI, Atom

Likelihood to Recommend

For a more comprehensive tool that is designed to work with Redshift, Matillion currently fills a niche with no good alternatives out there. If we were going to use a different database, there are other ETL tools that would perform that job and have a wider range of functionality. We have use cases to parse XML data and load it and although we have only started testing that functionality, we are not convinced Matillion has the most user-friendly interface for that (or that it will work in the end for huge XML files). For most of our processes of consolidating data (loading it from various sources and files), Matillion works really well.

Very user-friendly product that fills a niche for AWS Integration.

Overall Satisfaction with Matillion