Google Cloud Pub/Sub Reviews and Ratings

Use Cases and Deployment Scope

We used Google Cloud Pub/Sub to solve ETL/Streaming and real-time processing problems for high volumes of data. We used it either to fill datalakes, process and store in warehouses or data marts and also for processing events, either using JSON or protobuf.

This was integrated for many languages such as python, java, golang and kotlin. We had configured kubernetes auto scaling system based on some Google Cloud Pub/Sub metrics which worked very well. The main observed metrics for alerts and overall health indicator of our systems were both the size of each queue and the oldest message in queue, either indicating a high volume jam or some random specific error for a single message, respectively.

We had to handle idempotency since a duplicated message delivery is a possibility, this was usually paired and a Redis Cache to guarantee idempotency for a reasonable time window.

Likelihood to Recommend

If you want to stream high volumes of data, be it for ETL streaming or event sourcing, Google Cloud Pub/Sub is your go-to tool. It's easy to learn, easy to observe its metrics and scales with ease without additional configuration so if you have more producers of consumers, all you need to do is to deploy on k8s your solutions so that you can perform autoscaling on your pods to adjust to the data volume. The DLQ is also very transparent and easy to configure. Your code will have no logic whatsoever regarding orchestrating pubsub, you just plug and play.

However, if you are not in the Google Cloud Pub/Sub environment, you might have trouble or be most likely unable to use it since I think it's a product of Google Cloud.

Use Cases and Deployment Scope

We mainly use Google Cloud Pub/Sub with cloud functions and google scheduler to handle FinOps subjects. For example :

- Stopping a Google Kubernetes Engine (GKE) using a cloud scheduler that publishes to a pub/sub topic which then triggers a python function that changes GKE configurations.

- Stopping a Google SQL database using Google scheduler and Google Cloud Pub/Sub.

Likelihood to Recommend

Using Google Cloud Pub/Sub will mainly depend on the cloud platform used. Our client didn't choose GCP for Google Cloud Pub/Sub, if we went with AWS we would be using SNS/SQS (obviously). However, Google Cloud Pub/Sub is a better solution in the GCP services compared to self-managed solutions such as RabbitMQ for instance (it is managed by GCP and integrates with GCP solutions).

Use Cases and Deployment Scope

Fully managed messaging middleware useful for service integration. We have mainly used it for large scale logs from Kubernetes clusters. Pub/Sub proved to be easy to use, with a clear, user-friendly UI, well documented SDK, built-in elasticity and was the foundation for a real time data management system. Overall, very pleased with Google Cloud Pub/Sub.

Likelihood to Recommend

Google Cloud Pub/Sub is ideal for any real-time data project, especially with large datasets in the terabytes per day range.

We have used it successfully in a variety of use cases ranging from recommendations engines (where we’d need to ingest in real time people’s clickstream) to more ops related Kubernetes cluster provisioning and monitoring (based on Kubernetes control plane events).

Use Cases and Deployment Scope

Google [Cloud] Pub/Sub is used for high Volume, high speed data stream from multiple data sources towards a Data Platform. This is setup by Central Data and Analytics team to serve business use cases for most of the departments like Marketing, Product, Sales, Editorial etc. within our organization

Business Problems addressed

<ul><li>Identify Non App users , their inactivity and show such users a personalized promo on the website when they visit</li><li>Identify low engaged users overall, ingest their past usage patterns and cluster them in lookalike user segments to show them next best actions</li><li>Capture users' behavior and interactions as "Topics" in Pub/Sub and show them next best action as appropriate</li></ul>

Likelihood to Recommend

Well Suited for

<ul><li>One Source, multi subscribers scenario, where there is no issue with errors on multiple source datasets and anyone who is subscribed to a Topic receives the messages</li><li>The load of the messages are totally comfortable, the workspace inside the platform, the functions of messaging and calls of the properties are easy to use

</li><li>Reliable and Scalable model for creating and maintaining a big data pipeline</li></ul>Less Appropriate for:

<ul><li>Ingesting multiple datasets from same data sources at the same time</li><li>Many Google Cloud Pub/Sub classes are concrete and not interfaces, making them harder to plan for when writing unit or integration tests</li></ul>

Use Cases and Deployment Scope

Google Cloud Pub/Sub is used for processing scheduled tasks, sending notifications to clients and other background job execution. It helps in the point-in-time queue to send notifications to customers. As a team lead, I helped my team to transition

from the legacy queue system to the Google Cloud Pub/Sub real-time system to capture Play billing

alerts in real time.

Likelihood to Recommend

Every Google Play billing console user who handles subscriptions at scale should definitely adopt pub/sub for seamless processing.

Use Cases and Deployment Scope

Our company currently uses Pub/Sub to reliably queue up messages that arrive rapidly from multiple instances. It's a very technical product and really only used by the development team rather than company-wide. The primary problem it addresses for us is its ability to handle scale and resilience in the case of errors.

Likelihood to Recommend

Scenarios where GCP Pub/Sub is well suited:

- Companies processing large amounts of data where eventual consistency is acceptable and the ability to recover from loss desirable.

- Companies already working with other GCP products (it integrates well with other products in the ecosystem).

Scenarios where GCP Pub/Sub is less well suited:

- If you're not queuing a huge amount of data it's probably overkill.

Use Cases and Deployment Scope

We have a highly distributed system of batch jobs. We use Google Cloud Pub/Sub in order to communicate between our front-end, orchestration software, and VMs running batch workloads. It serves as a key piece under the hood for running jobs for scientists and it allows us to distribute our workloads across specialized VMs.

Likelihood to Recommend

Pub/Sub is perfect for all distributed systems. Whether it be services talking to each other, being the middleman between applications and Cloud Functions, or sending messages between standalone VMs that rely on one another, Pub/Sub makes it easy to communicate between pieces. It is ideal as a publication/subscription software as a service.