Matillion ETL in Healthcare

We utilize Matillion as our primary ETL tool. Due to the complexity of numerous healthcare data sources, we not only chose Matillion for it security and flexibility, but also for its ease of use. One of the main challenges that we face is the different source systems that we pull data from - both internally and externally. It is necessary for the organization to transform both sets of data simultaneously to ensure that we have a standardized EDW. In addition, we have also built template jobs and projects that we are able to utilize when bringing on new data sources. One of the key features that we routinely utilize is the ability to SFTP data files from an AWS instance to a secure SFTP server. While the native object within Matillion is used a vast majority of the time, we also integrated Python script to accomplish more complex cases. We currently utilize the on-prem server side version of Matillion, however we will be transitioning to the cloud-based version of Matillion. This is a game changer for us as we do not want to manage versioning of our Matillion instance.

The most difficult challenge that we faced was working with our legal team to ensure that data was protected and that nothing important was exposed. Within a week, we were able to get Matillion setup, installed and functioning in basic mode. Whenever we bring in new data sources, it may be necessary to ask external vendors to whitelist Matillion's public IP address, but that has not prevented us from moving very quickly.