AWS Glue vs. AWS IoT Core vs. dbt

One of AWS Glue's most notable features that aid in the creation and transformation of data is its data catalog. Support, scheduling, and the automation of the data schema recognition make it superior to its competitors aside from that. It also integrates perfectly with other AWS tools. The main restriction may be integrated with systems outside of the AWS environment. It functions flawlessly with the current AWS services but not with other goods. Another potential restriction that comes to mind is that glue operates on a spark, which means the engineer needs to be conversant in the language.

Incentivized

End-to-end encryption is an amazing feature because we use IoT to connect to various devices in order to gather data/ stats in real-time. We're able to publish solutions with ease and at a faster rate because of AWS IoT Core. However, its inability to interact with other IoT tools is a big con that I would like them to improve upon.

Incentivized

The prerequisite is that you have a supported database/data warehouse and have already found a way to ingest your raw data. Then dbt is very well suited to manage your transformation logic if the people using it are familiar with SQL. If you want to benefit from bringing engineering practices to data, dbt is a great fit. It can bring CI/CD practices, version control, automated testing, documentation generation, etc. It is not so well suited if the people managing the transformation logic do not like to code (in SQL) but prefer graphical user interfaces.

Incentivized

• It is extremely fast, easy, and self-intuitive. Though it is a suite of services, it requires pretty less time to get control over it.
• As it is a managed service, one need not take care of a lot of underlying details. The identification of data schema, code generation, customization, and orchestration of the different job components allows the developers to focus on the core business problem without worrying about infrastructure issues.
• It is a pay-as-you-go service. So, there is no need to provide any capacity in advance. So, it makes scheduling much easier.

Incentivized

• AWS IoT Core integrates power analytics and an AI solution for sensor data processing.
• Great protocol support including HTTPs and MQTT required to connect resource constraint IoT devices to cloud
• High security standards during data transmission
• Scalability and a great user community

Incentivized

• dbt supports version control through GIT, this allows teams to collaborate and track the data transformation logic.
• dbt allows us to build data models which helps to break complex transformation logic into simple and smaller logic.
• dbt is completely based on SQL which allows data analyst and data engineers to build the transformation logic.
• dbt can be easily integrated with snowflake.

Incentivized

• In-Stream schema registries feature people can not use this more efficiently
• in Connections feature they can add more connectors as well
• The crucial problem with AWS Glue is that it only works with AWS.

Incentivized

• Ability to integrate with Ethercalc software solution
• Ability to introduce NFC toolchains and systems which could be utilized in iot modular gateway solutions like drones
• Ability to reduce latency as the solution scales is very much needed
• Hybrid solution which can integrate other existing iot solutions into AWS iot core

Incentivized

• Field-level lineage (currently at table level)
• Documentation inheritance - if a field is documented the downstream field of the same name could inherit the doc info
• Adding python model support (in beta now)

Incentivized

While easy to set up and manage monitoring for large datasets, its complexity can be a barrier for new users. Integration with AWS Ecosystem, Managed Monitoring, Dashboards and monitoring tools for AWS Glue are generally easy to set up and maintain, Automated Data Pipelines. Automates data pipeline creation, making it efficient for certain data integration

Incentivized

I give AWS IoT Core's overall usability this rating because it is very easy to use and is enjoyed by all of our staff. The only problem is that it sometimes glitches and it freezes a lot. So overall, the usability of AWS IoT Core is very good, and we will continue to use it.

Incentivized

dbt is very easy to use. Basically if you can write SQL, you will be able to use dbt to get what you need done. Of course more advanced users with more technical skills can do more things.

Incentivized

Amazon responds in good time once the ticket has been generated but needs to generate tickets frequent because very few sample codes are available, and it's not cover all the scenarios.

Incentivized

It covers all the aspects of IoT services required for an IoT company. It supports all the industry-wide protocols for secure data transmission and integrates powerful AL and ML technology for data analytics. For data storage, Amazon S3 is a great solution. Strong tech support and user community. Since it is widely used as compared to other products, there is an abundance of training and learning material on the web.

Incentivized

AWS Glue is a fully managed ETL service that automates many ETL tasks, making it easier to set AWS Glue simplifies ETL through a visual interface and automated code generation.

Azure IoT service provides more or less the same services as compared to AWS IoT core, however the costing of AWS lead us to continued usage of IoT core over Azure IoT services. Also, considering our existing technology stack is on AWS, it was a natural selection for better integration and ease of use.

Incentivized

I actually don't know what the alternative to dbt is. I'm sure one must exist other than more 'roll your own' options like Apache Airflow, say, bu tin terms of super easy managed/cloud data transforms, dbt really does seem to be THE tool to use. It's $50/month per dev, BUT there's a FREE version for 1 dev seat with no read-only access for anyone else, so you can always start with that and then buy yourself a seat later.

Incentivized

• We are using GLUE for our ETL purpose. it’s ease with other our AWS services makes our ROI, 100% ROI.
• One missing piece was compatibility with other data source for which we found a work around and made our data source as S3 only, so our dependencies on other data source is also reducing

Incentivized

• Gives confidence to prototype iot solutions across lots of devices in little time
• Ability to meet requirements for managing a variety of communication protocols
• Learning iot analytics and visualization for faster insights enablement
• Onboarding devices and managing quickly without building inhouse solutions

Incentivized

• Simplified our BI layer for faster load times
• Increased the quality of data reaching our end users
• Makes complex transformations manageable

Incentivized