What is Google Cloud Pub/Sub?

Google offers Cloud Pub/Sub, a managed message oriented middleware supporting many-to-many asynchronous messaging between applications.

What is Google Cloud Pub/Sub's best feature?

Reviewers rate Support Rating highest, with a score of 9.8.

Who uses Google Cloud Pub/Sub?

The most common users of Google Cloud Pub/Sub are from Enterprises (1,001+ employees).

Google Cloud Pub/Sub Usability, Reviews & Ratings 2024

Operating Systems	Unspecified
Mobile Application	No

April 04, 2023

Google Cloud Pub/Sub, the jewel of streaming data

Cézar Augusto Nascimento e Silva

Solutions Architect, Tech Lead, Software Engineer

Stilingue (Telecommunications, 201-500 employees)

Score 10 out of 10

Vetted Review

Verified User

Use Cases and Deployment Scope

We used Google Cloud Pub/Sub to solve ETL/Streaming and real-time processing problems for high volumes of data. We used it either to fill datalakes, process and store in warehouses or data marts and also for processing events, either using JSON or protobuf.

This was integrated for many languages such as python, java, golang and kotlin. We had configured kubernetes auto scaling system based on some Google Cloud Pub/Sub metrics which worked very well. The main observed metrics for alerts and overall health indicator of our systems were both the size of each queue and the oldest message in queue, either indicating a high volume jam or some random specific error for a single message, respectively.

We had to handle idempotency since a duplicated message delivery is a possibility, this was usually paired and a Redis Cache to guarantee idempotency for a reasonable time window.

Pros and Cons

Data Streaming
Even Sourcing
Protobuf message format
Scalability
Easy to Use
Observability
Integrated Dead Letter Queue (DLQ) functionality

Deliver Once (idempotency) - currently in preview
Vendor locked to Google

Likelihood to Recommend

If you want to stream high volumes of data, be it for ETL streaming or event sourcing, Google Cloud Pub/Sub is your go-to tool. It's easy to learn, easy to observe its metrics and scales with ease without additional configuration so if you have more producers of consumers, all you need to do is to deploy on k8s your solutions so that you can perform autoscaling on your pods to adjust to the data volume. The DLQ is also very transparent and easy to configure. Your code will have no logic whatsoever regarding orchestrating pubsub, you just plug and play.

However, if you are not in the Google Cloud Pub/Sub environment, you might have trouble or be most likely unable to use it since I think it's a product of Google Cloud.

Most Important Features

DLQ (Dead Letter Queues)
Scalability
Delivering backoff for failed messages

Return on Investment

Scalable System
Better Alerts (observability)
Auto Scaling

Alternatives Considered

Apache Kafka

Kafka looks like and ordered queue, there no deliver backoff, so if a message has a problem, it doesn't advance to the next one. Google Cloud Pub/Sub looks like more a SET of messages, and kafka like a LIST. In kafka a same message will repeat instantaneously while it is being NACKED, on the other hand Google Cloud Pub/Sub will just deliver another and apply a backoff to the previous NACKED one, never stopping on a single message forever. Dead Letter Queues are innate to Google Cloud Pub/Sub while in Kafka is not a feature at all. One can configure the maximum amount of NACKs before sending a message to the DLQ, this is very powerful if combined with exponential backoff. Google Cloud Pub/Sub lets users scale consumers at will with no constraint, Kafka has a rather convoluted concept of partitions and consumer groups that requires one to plan ahead how many consumers they would like to plug to the queue because these constraints prevent a free scaling of consumers. If you plan for N consumers, you have to probably stick to that N forever or recreate your Kafka topic, in Confluent Cloud this is somewhat solved, but looks like a solution to a problem that should even exist in the first place.

Other Software Used

MongoDB, HashiCorp Consul, HashiCorp Vault, Istio

Users and Roles

100

Data and AI engineers

Support Headcount Required

SRE (Site Reliability Engineer)

Business Processes Supported

ETL data Streaming
Event Sourcing

Likelihood to Renew

It serves all of our purposes in the most transparent way I can imagine, after seeing other message queueing providers, I can only attest to its quality.

Product Configurability

It's just right, everything is configurable and very easy to understand.

Configuration Lessons

Always set a good exponential backoff to avoid processing the same failed message over and over. Consider using performatic data formats like protobuf instead of simple JSON when dealing with predictable structures of data.

UI Customization

No - we have not done any customization to the interface

Extensibility and Customization

No - the product does not support adding custom code

Usability

It has many libraries in many languages, google provides either good guides or they're AI generated code libraries that are easy to understand. It has very good observability too.

Easy Tasks

Dead Letter Queue (DLQ)
Exponential Backoff

Difficult Tasks

Priority Queueing
Deliver Once

Scalability

You can just plug in consumers at will and it will respond, there's no need for further configuration or introducing new concepts. You have a queue, if it's slow, you plug in more consumers to process more messages: simple as that.

Reliability and Availability

I have never faced a single problem in 4 years.

Performance

It's very fast, can be even better if you use protobuf.

October 26, 2020

Reliable Pub/Sub Vendor

Verified User

Engineer in Engineering

Computer Software Company, 1001-5000 employees

Score 10 out of 10

Vetted Review

Verified User

Use Cases and Deployment Scope

Google Cloud Pub/Sub is used for processing scheduled tasks, sending notifications to clients and other background job execution. It helps in the point-in-time queue to send notifications to customers. As a team lead, I helped my team to transition from the legacy queue system to the Google Cloud Pub/Sub real-time system to capture Play billing alerts in real time.