AWS Data Exchange vs. dbt

AWS Data Exchange fits best for scenarios where you have datasets that you would like to sell and you want to deliver it to anyone who would like to purchase it. It really beats having to set up downloads via your own website or portal. However, it can get complicated to manage if you're trying to deliver a dataset a client has already paid for.

Incentivized

The prerequisite is that you have a supported database/data warehouse and have already found a way to ingest your raw data. Then dbt is very well suited to manage your transformation logic if the people using it are familiar with SQL. If you want to benefit from bringing engineering practices to data, dbt is a great fit. It can bring CI/CD practices, version control, automated testing, documentation generation, etc. It is not so well suited if the people managing the transformation logic do not like to code (in SQL) but prefer graphical user interfaces.

Incentivized

• Simplified data delivery
• Ability to create any amount of data products
• Ability to integrate payment plans with data products
• Tracking data downloads and users
• Integration with other AWS data services

Incentivized

• dbt supports version control through GIT, this allows teams to collaborate and track the data transformation logic.
• dbt allows us to build data models which helps to break complex transformation logic into simple and smaller logic.
• dbt is completely based on SQL which allows data analyst and data engineers to build the transformation logic.
• dbt can be easily integrated with snowflake.

Incentivized

• Integration with more data sources
• Ability to deliver data to clients without AWS accounts
• Inclusion of direct data downloads in addition to asynchronous methods

Incentivized

• Field-level lineage (currently at table level)
• Documentation inheritance - if a field is documented the downstream field of the same name could inherit the doc info
• Adding python model support (in beta now)

Incentivized

There have been a lot of problems with ADX. First, the entire system is incredibly clunky from beginning to end.First, by AWS's own admission they're missing a lot of "tablestakes functionality" like the ability to see who is coming to your pages, more flexibility to edit and update your listings, the ability to create a storefront or catalog that actually tries to sell your products. All-in-all you're flying completely blind with AWS. In our convos with other sellers we strongly believe very little organic traffic is flowing through the AWS exchange. For the headache, it's not worth the time or the effort. It's very difficult to market or sell your products.We've also had a number of simple UX bugs where they just don't accurately reflect the attributes of your product. For instance for an S3 bucket they had "+metered costs" displayed to one of our buyers in the price. This of course caused a lot of confusion. They also misrepresented the historical revisions that were available in our product sets because of another UX bug. It's difficult to know what other things in the UX are also broken and incongruent.We also did have a purchase, but the seller is completely at their whim at providing you fake emails, fake company names, fake use cases because AWS hasn't thought through simple workflows like "why even have subscription confirmation if I can fake literally everything about a subscription request." So as a result we're now in an endless, timewasting, unhelpful thread with AWS support trying to get payment. They're confused of what to do and we feel completely lost.Lastly, the AWS team has been abysmal in addressing our concerns. Conversations with them result in a laundry list of excuses of why simple functionalities are so hard (including just having accurate documentation). It was a very frustrating and unproductive call. Our objective of our call was to help us see that ADX is a well-resourced and well-visioned product. Ultimately they couldn't clearly articulate who they built the exchange for both on the seller side and the buyer side.Don't waste your time. This is at best a very foggy experiment. Look at other sellers, they have a lot of free pages to try to get attention, but then have smart tactics to divert transactions away from the ADX. Ultimately, smart move. Why give 8-10% of your cut to a product that is basically bare-bones infrastructure.

dbt is very easy to use. Basically if you can write SQL, you will be able to use dbt to get what you need done. Of course more advanced users with more technical skills can do more things.

Incentivized

I actually don't know what the alternative to dbt is. I'm sure one must exist other than more 'roll your own' options like Apache Airflow, say, bu tin terms of super easy managed/cloud data transforms, dbt really does seem to be THE tool to use. It's $50/month per dev, BUT there's a FREE version for 1 dev seat with no read-only access for anyone else, so you can always start with that and then buy yourself a seat later.

Incentivized

• Reduced time to publish datasets for sale by more than 80%
• Increased net profit from dataset sales by ~10%
• Reduced data delivery time to clients by 15%

Incentivized

• Simplified our BI layer for faster load times
• Increased the quality of data reaching our end users
• Makes complex transformations manageable

Incentivized