AWS Glue vs. Azure Data Factory vs. Matillion

One of AWS Glue's most notable features that aid in the creation and transformation of data is its data catalog. Support, scheduling, and the automation of the data schema recognition make it superior to its competitors aside from that. It also integrates perfectly with other AWS tools. The main restriction may be integrated with systems outside of the AWS environment. It functions flawlessly with the current AWS services but not with other goods. Another potential restriction that comes to mind is that glue operates on a spark, which means the engineer needs to be conversant in the language.

Incentivized

Best scenario is for ETL process. The flexibility and connectivity is outstanding. For our environment, SAP data connectivity with Azure Data Factory offers very limited features compared to SAP Data Sphere. Due to the limited modelling capacity of the tool, we use Databricks for data modelling and cleaning. Usage of multiple tools could have been avoided if adf has modelling capabilities.

Incentivized

Great: Need to query simpler APIs, or utilize well known services such as GSheets etc.? Matillion has got some of the best and easiest to use connectors out there. Not so great: Do you need have a competent CI/CD flow that you will be able to update / compare from Matillion as well as other sources at the same time? Good luck, you will need to be extra careful, as you might have to have a deeper dive into your servers Terminal each time you have a git conflict.

Incentivized

• It is extremely fast, easy, and self-intuitive. Though it is a suite of services, it requires pretty less time to get control over it.
• As it is a managed service, one need not take care of a lot of underlying details. The identification of data schema, code generation, customization, and orchestration of the different job components allows the developers to focus on the core business problem without worrying about infrastructure issues.
• It is a pay-as-you-go service. So, there is no need to provide any capacity in advance. So, it makes scheduling much easier.

Incentivized

• Data Ingestion - it works very well with numerous data sources.
• Data pipeline orchestration: It is a generic, popular tool for orchestrating data pipelines.
• Works well in Azure ecosystem, Azure services and data platforms like Databricks.
• It is a serverless and scalable solution for cloud data integration.

Incentivized

• The user interface of your data pipelines makes it easier for people who aren’t as techy as data engineers to observe what's going on.
• Customer support is quick, not always as efficient as you would want it to be, but still.
• Nice documentation available.

Incentivized

• In-Stream schema registries feature people can not use this more efficiently
• in Connections feature they can add more connectors as well
• The crucial problem with AWS Glue is that it only works with AWS.

Incentivized

• Granularity of Errors: Sometimes, Azure Data Factory provides error messages that are too generic or vague for us, making it challenging to pinpoint the exact cause of a pipeline failure. Enhanced error messages with more actionable details would greatly assist us as users in debugging their pipelines.
• Pipeline Design UI: In my experience, the visual interface for designing pipelines, especially when dealing with complex workflows or numerous activities, can become cluttered. I think a more intuitive and scalable design interface would improve usability. In my opinion, features like zoom, better alignment tools, or grouping capabilities could make managing intricate designs more manageable.
• Native Support: While Azure Data Factory does support incremental data loads, in my experience, the setup can be somewhat manual and complex. I think native and more straightforward support for Change Data Capture, especially from popular databases, would simplify the process of capturing and processing only the changed data, making regular data updates more efficient

Incentivized

• Matillion is brilliant at importing data -- it would be amazing to have more ways to export data, from emailed exports to API pushes.
• Any Python that takes more than a few lines of code requires an external server to run it. It would be great to have more integration (perhaps in a connected virtual environment) to easily integrate customized code.
• Troubleshooting server logs requires quite a bit of technical expertise. More human readable detailed error handling would be greatly appreciated.

Incentivized

With the current experience of Matillion, we are likely to renew with the current feature option but will also look for improvement in various areas including scalability and dependability. 1. Connectors: It offers various connectors option but isn't full proof which we will be looking forward as we grow. 2. Scalability: As usage increase, we want Matillion system to be more stable.

Incentivized

While easy to set up and manage monitoring for large datasets, its complexity can be a barrier for new users. Integration with AWS Ecosystem, Managed Monitoring, Dashboards and monitoring tools for AWS Glue are generally easy to set up and maintain, Automated Data Pipelines. Automates data pipeline creation, making it efficient for certain data integration

Incentivized

So far product has performed as expected. We were noticing some performance issues, but they were largely Synapse related. This has led to a shift from Synapse to Databricks. Overall this has delayed our analytic platform. Once databricks becomes fully operational, Azure Data Factory will be critical to our environment and future success.

Incentivized

We are able to bring on new resources and teach them how to use Matillion without having to invest a significant amount of time. We prefer looking for resources with any type of ETL skill-set and feel that they can learn Matillion without problem. In addition, the prebuilt objects cover more than 95% of our use cases and we do not have to build much from scratch.

Amazon responds in good time once the ticket has been generated but needs to generate tickets frequent because very few sample codes are available, and it's not cover all the scenarios.

Incentivized

We have not had need to engage with Microsoft much on Azure Data Factory, but they have been responsive and helpful when needed. This being said, we have not had a major emergency or outage requiring their intervention. The score of seven is a representation that they have done well for now, but have not proved out their support for a significant issue

Incentivized

Overall, I've found Matillion to be responsive and considerate. I feel like they value us as a customer even when I know they have customers who spend more on the product than we do. That speaks to a motive higher than money. They want to make a good product and a good experience for their customers. If I have any complaint, it's that support sometimes feels community-oriented. It isn't always immediately clear to me that my support requests are going to a support engineer and not to the community at large. Usually, though, after a bit of conversation, it's clear that Matillion is watching and responding. And responses are generally quick in coming.

Incentivized

We were able to control on access and built various enviroment for implementation

Incentivized

AWS Glue is a fully managed ETL service that automates many ETL tasks, making it easier to set AWS Glue simplifies ETL through a visual interface and automated code generation.

Azure Data Factory helps us automate to schedule jobs as per customer demands to make ETL triggers when the need arises. Anyone can define the workflow with the Azure Data Factory UI designer tool and easily test the systems. It helped us automate the same workflow with programming languages like Python or automation tools like ansible. Numerous options for connectivity be it a database or storage account helps us move data transfer to the cloud or on-premise systems.

Incentivized

Fivetran offers a managed service and pre-configured schemas/models for data loading, which means much less administrative work for initial setup and ongoing maintenance. But it comes at a much higher price tag. So, knowing where your sweet spot is in the build vs. buy spectrum is essential to deciding which tool fits better. For the transformation part, dbt is purely (SQL-) code-based. So, it is mainly whether your developers prefer a GUI or code-based approach.

Incentivized

We're using Matillion on EC2 instances, and we have about 20 projects for our clients in the same instance. Sometimes, we're struggling to manage schedules for all projects because thread management is not visible, and we can't see the process at the instance level.

Incentivized

• We are using GLUE for our ETL purpose. it’s ease with other our AWS services makes our ROI, 100% ROI.
• One missing piece was compatibility with other data source for which we found a work around and made our data source as S3 only, so our dependencies on other data source is also reducing

Incentivized

• Facilitate better decision-making and improve business processes.
• Optimize business process outcomes by increasing internal efficiency and operational effectiveness.
• Boosts revenue growth while improving business process agility.

Incentivized

• Matillion has been the backbone of my company's analytical functionalities for 10+ years, so it has a good ROI.
• The price is ok for what our company built with it, but it starts to be less competitive if the tool is not used at its fullest.

Incentivized