Analytic Solver vs. AWS Glue vs. dbt

Based on my limited experience and use, and therefore limited global knowledge of the software, I would recommend it especially if the data that will be used as inputs to the model has previously worked on a spreadsheet such as Excel. I would also recommend it to analyze problems of medium and small size. Given the experience I have had when I have used it with large problems, there have been noticeable decreases in the speed of response (which are not associated with the size of the system of equations involved in the calculation). Excellent for processing linear programming models.

Incentivized

One of AWS Glue's most notable features that aid in the creation and transformation of data is its data catalog. Support, scheduling, and the automation of the data schema recognition make it superior to its competitors aside from that. It also integrates perfectly with other AWS tools. The main restriction may be integrated with systems outside of the AWS environment. It functions flawlessly with the current AWS services but not with other goods. Another potential restriction that comes to mind is that glue operates on a spark, which means the engineer needs to be conversant in the language.

Incentivized

The prerequisite is that you have a supported database/data warehouse and have already found a way to ingest your raw data. Then dbt is very well suited to manage your transformation logic if the people using it are familiar with SQL. If you want to benefit from bringing engineering practices to data, dbt is a great fit. It can bring CI/CD practices, version control, automated testing, documentation generation, etc. It is not so well suited if the people managing the transformation logic do not like to code (in SQL) but prefer graphical user interfaces.

Incentivized

• Easy to use and learn.
• Strong optimization platform for business and supply chain problems which can be modeled in Excel
• It can handle multiple objectives and more than 8000 decision variables in a problem.
• Lots of tutorial and examples are available to solve the problem

Incentivized

• It is extremely fast, easy, and self-intuitive. Though it is a suite of services, it requires pretty less time to get control over it.
• As it is a managed service, one need not take care of a lot of underlying details. The identification of data schema, code generation, customization, and orchestration of the different job components allows the developers to focus on the core business problem without worrying about infrastructure issues.
• It is a pay-as-you-go service. So, there is no need to provide any capacity in advance. So, it makes scheduling much easier.

Incentivized

• dbt supports version control through GIT, this allows teams to collaborate and track the data transformation logic.
• dbt allows us to build data models which helps to break complex transformation logic into simple and smaller logic.
• dbt is completely based on SQL which allows data analyst and data engineers to build the transformation logic.
• dbt can be easily integrated with snowflake.

Incentivized

• On the few occasions when I have used it to deal with problems of optimization of relatively large parameters (with a large number of restrictions and decision variables), the program has been slower, not substantially but slower, than programs such as the WinQsb, even when the latter runs on 32-bit machines and not 64. That has caught my attention, even though it is not a real problem for the uses I give to the program.
• Given my partial function as a university professor, it has been much more effective and practical to use other software, due to the limited options that the educational license associated with the software has.

Incentivized

• In-Stream schema registries feature people can not use this more efficiently
• in Connections feature they can add more connectors as well
• The crucial problem with AWS Glue is that it only works with AWS.

Incentivized

• Field-level lineage (currently at table level)
• Documentation inheritance - if a field is documented the downstream field of the same name could inherit the doc info
• Adding python model support (in beta now)

Incentivized

While easy to set up and manage monitoring for large datasets, its complexity can be a barrier for new users. Integration with AWS Ecosystem, Managed Monitoring, Dashboards and monitoring tools for AWS Glue are generally easy to set up and maintain, Automated Data Pipelines. Automates data pipeline creation, making it efficient for certain data integration

Incentivized

dbt is very easy to use. Basically if you can write SQL, you will be able to use dbt to get what you need done. Of course more advanced users with more technical skills can do more things.

Incentivized

Amazon responds in good time once the ticket has been generated but needs to generate tickets frequent because very few sample codes are available, and it's not cover all the scenarios.

Incentivized

We believe in building the models in Excel. A limitation with Excel is that Excel Solver can not take more than 200 decision variables with multiple constraints. It is cheap in terms of license and maintenance fees against other softwares which are available in the market.

Incentivized

AWS Glue is a fully managed ETL service that automates many ETL tasks, making it easier to set AWS Glue simplifies ETL through a visual interface and automated code generation.

I actually don't know what the alternative to dbt is. I'm sure one must exist other than more 'roll your own' options like Apache Airflow, say, bu tin terms of super easy managed/cloud data transforms, dbt really does seem to be THE tool to use. It's $50/month per dev, BUT there's a FREE version for 1 dev seat with no read-only access for anyone else, so you can always start with that and then buy yourself a seat later.

Incentivized

• - It has allowed finding ways to optimize (minimizing costs or times) the field processes involved in various projects.
• It has even allowed, in specific cases where it was used for that purpose, to optimize the allocation of resources (people) to work in different jobs that present weekly variations of the activity that these people must perform.
• It has allowed the sensitivity analysis of projects to changes in the decision variables related to them, which, and in very dynamic and changing environments, resulted in substantial decreases in money losses.

Incentivized

• We are using GLUE for our ETL purpose. it’s ease with other our AWS services makes our ROI, 100% ROI.
• One missing piece was compatibility with other data source for which we found a work around and made our data source as S3 only, so our dependencies on other data source is also reducing

Incentivized

• Simplified our BI layer for faster load times
• Increased the quality of data reaching our end users
• Makes complex transformations manageable

Incentivized