If you want to stream high volumes of data, be it for ETL streaming or event sourcing, Google Cloud Pub/Sub is your go-to tool. It's easy to learn, easy to observe its metrics and scales with ease without additional configuration so if you have more producers of consumers, all you need to do is to deploy on k8s your solutions so that you can perform autoscaling on your pods to adjust to the data volume. The DLQ is also very transparent and easy to configure. Your code will have no logic whatsoever regarding orchestrating pubsub, you just plug and play. However, if you are not in the Google Cloud Pub/Sub environment, you might have trouble or be most likely unable to use it since I think it's a product of Google Cloud.
BizTalk is well suited as middleware. Where you wish to translate an input file into an output file and send it to some endpoint. In our case, we used it to convert and send files to SAP. In many ways, it very flexible, and you can do almost anything you want with it. In many ways, it's a better solution than your SAP XI or PI as middleware, since it's much less expensive, and allows you do interface with non-SAP systems.
With a pub/sub architecture the consumer is decoupled in time from the publisher i.e. if the consumer goes down, it can replay any events that occurred during its downtime.
It also allows consumer to throttle and batch incoming data providing much needed flexibility while working with multiple types of data sources
A simple and easy to use UI on cloud console for setup and debugging
It enables event-driven architectures and asynchronous parallel processing, while improving performance, reliability and scalability
It is very user friendly. Users can change rules during run time and change workflow.
Huge capacity for queueing messages. It supports all types of adapters like Oracle, Salesforce, SMTP, FTP, etc. Also users can built custom adaptors.
If users want to dynamically deploy their solution without any downtime, this is a perfect solution. BizTalk will be a good fit, especially for public-facing websites.
Well-proven in the market. I used it when developing a website for Virgin Trains, catering more than 800K user requests per day.
It serves all of our purposes in the most transparent way I can imagine, after seeing other message queueing providers, I can only attest to its quality.
It has many libraries in many languages, google provides either good guides or they're AI generated code libraries that are easy to understand. It has very good observability too.
They have decent documentation, but you need to pay for support. We weren't able to answer all our questions with the documentation and didn't have time to setup support before we needed it so I can't give it a higher rating but I think it tends to be a bit slow unless you're a GCP enterprise support customer.
BizTalk Server has been supported for more than 15 years. It is well proven in the market. Microsoft has provided excellent support with technical issues.
Having used Amazon Web Services SNS & SQS I can say that even if the latter may offer more features, Google Cloud Pub/Sub is easier to use. On the other hand, usage of SNS & SQS as well as documentation and troubleshooting is easier with the AWS solution. Since we are not using GCP only for Pub/Sub the choice depends on other variables.
BizTalk was selected here mainly because it is easy to integrate to a .NET application (most of them are Web Service, WCF SOAP, WCF REST and Web API) and many backend databases are Microsoft SQL Server. Another benefit is that the monitoring job is easy to set up and centralize with other .NET application monitoring jobs.
You can just plug in consumers at will and it will respond, there's no need for further configuration or introducing new concepts. You have a queue, if it's slow, you plug in more consumers to process more messages: simple as that.
A positive impact has been the quicker turnaround time of a part request and that part showing up in SAP using Biztalk as middleware.
A somewhat negative impact has been the somewhat insufficient error logging/message capture settings that Biztalk provide. This has caused occasional delays when attempted to create parts for the business.
A somewhat negative impact has been the need to have a specialized developer who understands Biztalk to troubleshoot issues with the Biztalk and SAP interaction when creating parts, and when adding new fields to the parts.
