Google Cloud Pub/Sub

Score8.8 out of 10

28 Reviews and Ratings

What is Google Cloud Pub/Sub?

Google offers Cloud Pub/Sub, a managed message oriented middleware supporting many-to-many asynchronous messaging between applications.

Categories & Use Cases

Message Oriented Middleware

Verified User

Team Lead in Engineering (1-10 employees employees)

Use Cases and Deployment Scope

<div>We use Google Cloud Pub/Sub to communicate between micro services. By using a Pub/Sub topic, we can delegate not so time critical tasks to be processed asynchronously by another micro service.</div><div>

</div><div>Using a BigQuery subscription, we can automatically write events sent to a Pub/Sub topic into a BigQuery table.</div>

Pros

Push subscription that can trigger a HTTP endpoint
Supports retries
BigQuery subscription for writing into a BigQuery table

Cons

Allow write and read of messages more than 10 MB
Support rate at which HTTP endpoints are triggered to reduce scale requirements on downstream services
Support replay of messages

Return on Investment

Using a BigQuery subscription reduces development effort.
Since there is no rate control built into Pub/Sub it can result in overload of downstream services (including databases) when the incoming traffic is bursty in nature. We had to built a custom solution to address this scenario.
Helps integration of services spread across multiple cloud environments.

Usability

Alternatives Considered

Amazon Simple Queue Service (SQS) and Apache Kafka

Other Software Used

AlloyDB, ExpressVPN, Sprinto, Google BigQuery, Firebase, Sentry

Cézar Augusto Nascimento e Silva View profile

Solutions Architect, Tech Lead, Software Engineer in Information Technology at Stilingue (201-500 employees employees)

Use Cases and Deployment Scope

We used Google Cloud Pub/Sub to solve ETL/Streaming and real-time processing problems for high volumes of data. We used it either to fill datalakes, process and store in warehouses or data marts and also for processing events, either using JSON or protobuf.

This was integrated for many languages such as python, java, golang and kotlin. We had configured kubernetes auto scaling system based on some Google Cloud Pub/Sub metrics which worked very well. The main observed metrics for alerts and overall health indicator of our systems were both the size of each queue and the oldest message in queue, either indicating a high volume jam or some random specific error for a single message, respectively.

We had to handle idempotency since a duplicated message delivery is a possibility, this was usually paired and a Redis Cache to guarantee idempotency for a reasonable time window.

Pros

Data Streaming
Even Sourcing
Protobuf message format
Scalability
Easy to Use
Observability
Integrated Dead Letter Queue (DLQ) functionality

Cons

Deliver Once (idempotency) - currently in preview
Vendor locked to Google

Most Important Features

DLQ (Dead Letter Queues)
Scalability
Delivering backoff for failed messages

Return on Investment

Scalable System
Better Alerts (observability)
Auto Scaling

Alternatives Considered

Apache Kafka

Other Software Used

MongoDB, HashiCorp Consul, HashiCorp Vault, Istio

Usability

Mehdi Bettiche

Senior Consultant in Information Technology at Wavestone (1001-5000 employees employees)

Use Cases and Deployment Scope

We mainly use Google Cloud Pub/Sub with cloud functions and google scheduler to handle FinOps subjects. For example :

- Stopping a Google Kubernetes Engine (GKE) using a cloud scheduler that publishes to a pub/sub topic which then triggers a python function that changes GKE configurations.

- Stopping a Google SQL database using Google scheduler and Google Cloud Pub/Sub.

Pros

Pub/Sub UI
Pub/Sub subscriptions & message retrieval.
Integration with BigQuery & Cloud Storage.

Cons

Debugging
Error screening

Most Important Features

Handling messages with a managed solution.
Integration with other GCP services.

Return on Investment

Since it integrates in our FinOps solution, we have positive impact on the billing.

Alternatives Considered

Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS) and RabbitMQ

Other Software Used

Google Kubernetes Engine, Google Cloud SQL, Google Cloud Functions

Cameron Gable View profile

Software Engineer in Research & Development at Serimmune Inc. (11-50 employees employees)

Pros

A hands-off approach to publishing messages and subscribing to topics.
Easy to use APIs.
Useful, simple UI on cloud console to send messages for debugging, etc.

Cons

I can't think of missing functionality, but added functionality is always nice!

Return on Investment

Allowed the use of specialized VMs
Saves costs and time

Alternatives Considered

Google Compute Engine

Other Software Used

Google Cloud Storage, Stackdriver, Google Cloud Dataflow, Google Cloud SQL

Eric Mason

Lead Developer in Engineering at SaaSquatch (11-50 employees employees)

Pros

Highly scalable.
Queues messages for up to 7 days if not acknowledged.

Cons

Would be nice if the queue could be extended beyond 7 days.
We found it a bit tricky replay unacknowledged messages when needed.

Return on Investment

Pub/Sub has helped avoid data loss improving our customer value prop.
Pub/Sub has reduced development time that would otherwise be needed to build a highly scalable queue.

Alternatives Considered

RabbitMQ, Apache Kafka and Amazon Kinesis

Other Software Used

MongoDB, Redis, Google Cloud Storage