Google Cloud Pub/Sub

We used Google Cloud Pub/Sub to solve ETL/Streaming and real-time processing problems for high volumes of data. We used it either to fill datalakes, process and store in warehouses or data marts and also for processing events, either using JSON or protobuf.

This was integrated for many languages such as python, java, golang and kotlin. We had configured kubernetes auto scaling system based on some Google Cloud Pub/Sub metrics which worked very well. The main observed metrics for alerts and overall health indicator of our systems were both the size of each queue and the oldest message in queue, either indicating a high volume jam or some random specific error for a single message, respectively.

We had to handle idempotency since a duplicated message delivery is a possibility, this was usually paired and a Redis Cache to guarantee idempotency for a reasonable time window.

We mainly use Google Cloud Pub/Sub with cloud functions and google scheduler to handle FinOps subjects. For example :
- Stopping a Google Kubernetes Engine (GKE) using a cloud scheduler that publishes to a pub/sub topic which then triggers a python function that changes GKE configurations.
- Stopping a Google SQL database using Google scheduler and Google Cloud Pub/Sub.

Fully Managed messaging middleware useful for service integration. We have mainly used it for large scale logs from Kubernetes clusters. Pub/Sub proved to be easy to use, with a clear, user-friendly UI, well documented SDK, built-in elasticity and was the foundation for a real time data management system. Overall, very pleased with Google Cloud Pub/Sub.

We have a highly distributed system of batch jobs. We use Google Cloud Pub/Sub in order to communicate between our front-end, orchestration software, and VMs running batch workloads. It serves as a key piece under the hood for running jobs for scientists and it allows us to distribute our workloads across specialized VMs.

Google [Cloud] Pub/Sub is used for high Volume, high speed data stream from multiple data sources towards a Data Platform. This is setup by Central Data and Analytics team to serve business use cases for most of the departments like Marketing, Product, Sales, Editorial etc. within our organization

Business Problems addressed

• Identify Non App users , their inactivity and show such users a personalized promo on the website when they visit
• Identify low engaged users overall, ingest their past usage patterns and cluster them in lookalike user segments to show them next best actions
• Capture users' behavior and interactions as "Topics" in Pub/Sub and show them next best action as appropriate

Google Cloud Pub/Sub is used for processing scheduled tasks, sending notifications to clients and other background job execution. It helps in the point-in-time queue to send notifications to customers. As a team lead, I helped my team to transition from the legacy queue system to the Google Cloud Pub/Sub real-time system to capture Play billing alerts in real time.

Our company currently uses Pub/Sub to reliably queue up messages that arrive rapidly from multiple instances. It's a very technical product and really only used by the development team rather than company-wide. The primary problem it addresses for us is its ability to handle scale and resilience in the case of errors.