Item: Matillion
Rating: 10
Author: Nick Wardle

Use Cases and Deployment Scope

Matillion is a fundamental part of our ETL pipeline creation and management. It is the solution we have chosen that enables myriad external data sources (APIs, CSVs, data lakes etc.) to be ingested into our data warehouse. As a tool for its specific purpose, it is primarily used by the Data Science team, but the resulting data is disseminated to multiple departments via dashboards and reports across the whole company.

Pros and Cons

It provides a simple and intuitive visual interface for ETL creation and management that is also highly flexible and extensible i.e. integrating custom Python scripts
It makes complex, advanced workflows, e.g. updating, scheduling, error handling & reporting, simple with pre-existing drag-n-drop "feature blocks"
It makes the creation of a clear, hierarchical architecture across all pipelines simple and easily replicable by all members of the team
Matillion's support team is highly responsive and always do their best to address issues as quickly as possible

Whilst the pre-existing "feature blocks" do make creating new API integrations simple, fine tuning the integration or ironing out minor bugs requires conversations with Matillion support (however, they are always very quick to respond and have always fixed our issues)
Minor feature request: It would be nice to have a way to pass the output of a Python script directly to a subsequent SQL "block"

Return on Investment

We don't have ROI data to share at this stage.

Usability

Matillion takes the various aspects of ETL creation and management and presents them separately and clearly, allowing for different levels of user to engage with only the parts of the product that they need to. The visual interface for pipeline building is intuitive and clear. The fully flexible folder structure for projects and pipelines is ideal for setting up the organisation of your whole ETL environment exactly as you want it.

Time to Value

We took on Matillion at the very beginning of a new data warehouse project, so it was a longer set up than if we were just swapping out an existing ETL solution. However, we were able to create an end-to-end POC within a few weeks.

Scalability

We haven't hit any issues with scalability yet. However, as everything is tied to instance sizes for our implementation, we will inevitably find out how this goes as we ramp up the amount of data coming through the pipelines.

Alternatives Considered

Matillion is the only dedicated ETL solution we have used in the current data team. However, we decided to use Matillion because the BI front-end solution we also chose at the time, Periscope, recommended them to us and have a solid working relationship with them. So far, both solutions have worked seamlessly together.

Other Software Used

Amazon Redshift, Periscope Data

Likelihood to Recommend

Matillion is an excellent solution for teams wanting to create and manage ETL pipelines for integration of external data sources into, for example, a data warehouse. There are many pre-existing "feature blocks" for most of the major data providers (Facebook, Google Analytics, Big Query, Twitter and many more) which makes setting up pipelines for those APIs trivial. For data sources that do not have a pre-existing "block", Python can be integrated directly with SQL in the pipelines allowing for custom integrations.

Matillion makes ETL pipeline creation and management easy and extensible

Overall Satisfaction with Matillion