Google Cloud Pub/Sub

If you want to stream high volumes of data, be it for ETL streaming or event sourcing, Google Cloud Pub/Sub is your go-to tool. It's easy to learn, easy to observe its metrics and scales with ease without additional configuration so if you have more producers of consumers, all you need to do is to deploy on k8s your solutions so that you can perform autoscaling on your pods to adjust to the data volume. The DLQ is also very transparent and easy to configure. Your code will have no logic whatsoever regarding orchestrating pubsub, you just plug and play.

However, if you are not in the Google Cloud Pub/Sub environment, you might have trouble or be most likely unable to use it since I think it's a product of Google Cloud.

Using Google Cloud Pub/Sub will mainly depend on the cloud platform used. Our client didn't choose GCP for Google Cloud Pub/Sub, if we went with AWS we would be using SNS/SQS (obviously). However, Google Cloud Pub/Sub is a better solution in the GCP services compared to self-managed solutions such as RabbitMQ for instance (it is managed by GCP and integrates with GCP solutions).

Google Cloud Pub/Sub is ideal for any real-time data project, especially with large datasets in the terabytes per day range. We have used it successfully in a variety of use cases ranging from recommendations engines (where we’d need to ingest in real time people’s clickstream) to more ops related Kubernetes cluster provisioning and monitoring (based on Kubernetes control plane events).

Pub/Sub is perfect for all distributed systems. Whether it be services talking to each other, being the middleman between applications and Cloud Functions, or sending messages between standalone VMs that rely on one another, Pub/Sub makes it easy to communicate between pieces. It is ideal as a publication/subscription software as a service.

Well Suited for

• One Source, multi subscribers scenario, where there is no issue with errors on multiple source datasets and anyone who is subscribed to a Topic receives the messages
• The load of the messages are totally comfortable, the workspace inside the platform, the functions of messaging and calls of the properties are easy to use
  
• Reliable and Scalable model for creating and maintaining a big data pipeline

Less Appropriate for:

• Ingesting multiple datasets from same data sources at the same time
• Many Google Cloud Pub/Sub classes are concrete and not interfaces, making them harder to plan for when writing unit or integration tests

Every Google Play billing console user who handles subscriptions at scale should definitely adopt pub/sub for seamless processing.

Scenarios where GCP Pub/Sub is well suited:
- Companies processing large amounts of data where eventual consistency is acceptable and the ability to recover from loss desirable.
- Companies already working with other GCP products (it integrates well with other products in the ecosystem).

Scenarios where GCP Pub/Sub is less well suited:
- If you're not queuing a huge amount of data it's probably overkill.