Google Cloud Pub/Sub, the jewel of streaming data
We used Google Cloud Pub/Sub to solve ETL/Streaming and real-time processing problems for high volumes of data. We used it either to fill …
Overview
What is Google Cloud Pub/Sub?
Google offers Cloud Pub/Sub, a managed message oriented middleware supporting many-to-many asynchronous messaging between applications.
Recent Reviews
Awards
Products that are considered exceptional by their customers based on a variety of criteria win TrustRadius awards. Learn more about the types of TrustRadius awards to make the best purchase decision. More about TrustRadius Awards
Product Details
- About
- Tech Details
- FAQs
What is Google Cloud Pub/Sub?
Google Cloud Pub/Sub Technical Details
| Operating Systems | Unspecified |
|---|---|
| Mobile Application | No |
Frequently Asked Questions
Google offers Cloud Pub/Sub, a managed message oriented middleware supporting many-to-many asynchronous messaging between applications.
Reviewers rate Support Rating highest, with a score of 9.8.
The most common users of Google Cloud Pub/Sub are from Enterprises (1,001+ employees).
Comparisons
Compare with
Reviews and Ratings
(23)Attribute Ratings
Reviews
(1-7 of 7)Companies can't remove reviews or game the system. Here's why
April 04, 2023
Google Cloud Pub/Sub, the jewel of streaming data
Score 10 out of 10
Vetted Review
Verified User
We used Google Cloud Pub/Sub to solve ETL/Streaming and real-time processing problems for high volumes of data. We used it either to fill datalakes, process and store in warehouses or data marts and also for processing events, either using JSON or protobuf.
This was integrated for many languages such as python, java, golang and kotlin. We had configured kubernetes auto scaling system based on some Google Cloud Pub/Sub metrics which worked very well. The main observed metrics for alerts and overall health indicator of our systems were both the size of each queue and the oldest message in queue, either indicating a high volume jam or some random specific error for a single message, respectively.
We had to handle idempotency since a duplicated message delivery is a possibility, this was usually paired and a Redis Cache to guarantee idempotency for a reasonable time window.
This was integrated for many languages such as python, java, golang and kotlin. We had configured kubernetes auto scaling system based on some Google Cloud Pub/Sub metrics which worked very well. The main observed metrics for alerts and overall health indicator of our systems were both the size of each queue and the oldest message in queue, either indicating a high volume jam or some random specific error for a single message, respectively.
We had to handle idempotency since a duplicated message delivery is a possibility, this was usually paired and a Redis Cache to guarantee idempotency for a reasonable time window.
- Data Streaming
- Even Sourcing
- Protobuf message format
- Scalability
- Easy to Use
- Observability
- Integrated Dead Letter Queue (DLQ) functionality
- Deliver Once (idempotency) - currently in preview
- Vendor locked to Google
July 19, 2022
Google Pub/Sub review & usage with FinOps
We mainly use Google Cloud Pub/Sub with cloud functions and google scheduler to handle FinOps subjects. For example :
- Stopping a Google Kubernetes Engine (GKE) using a cloud scheduler that publishes to a pub/sub topic which then triggers a python function that changes GKE configurations.
- Stopping a Google SQL database using Google scheduler and Google Cloud Pub/Sub.
- Stopping a Google Kubernetes Engine (GKE) using a cloud scheduler that publishes to a pub/sub topic which then triggers a python function that changes GKE configurations.
- Stopping a Google SQL database using Google scheduler and Google Cloud Pub/Sub.
- Pub/Sub UI
- Pub/Sub subscriptions & message retrieval.
- Integration with BigQuery & Cloud Storage.
- Debugging
- Error screening
July 14, 2022
Overview of Google Cloud Pub/Sub
Fully Managed messaging middleware useful for service integration. We have mainly used it for large scale logs from Kubernetes clusters. Pub/Sub proved to be easy to use, with a clear, user-friendly UI, well documented SDK, built-in elasticity and was the foundation for a real time data management system. Overall, very pleased with Google Cloud Pub/Sub.
- Messaging
- Scalable
- No-ops
- Secure
- Only usable within GCP
July 22, 2021
Messaging Service is Great for Distributed Systems
We have a highly distributed system of batch jobs. We use Google Cloud Pub/Sub in order to communicate between our front-end, orchestration software, and VMs running batch workloads. It serves as a key piece under the hood for running jobs for scientists and it allows us to distribute our workloads across specialized VMs.
- A hands-off approach to publishing messages and subscribing to topics.
- Easy to use APIs.
- Useful, simple UI on cloud console to send messages for debugging, etc.
- I can't think of missing functionality, but added functionality is always nice!
May 26, 2021
Great Asynchronous Data Messaging Services, Reliable, Scalable and fully managed by Google
Google [Cloud] Pub/Sub is used for high Volume, high speed data stream from multiple data sources towards a Data Platform. This is setup by Central Data and Analytics team to serve business use cases for most of the departments like Marketing, Product, Sales, Editorial etc. within our organization
Business Problems addressed
Business Problems addressed
- Identify Non App users , their inactivity and show such users a personalized promo on the website when they visit
- Identify low engaged users overall, ingest their past usage patterns and cluster them in lookalike user segments to show them next best actions
- Capture users' behavior and interactions as "Topics" in Pub/Sub and show them next best action as appropriate
- With a pub/sub architecture the consumer is decoupled in time from the publisher i.e. if the consumer goes down, it can replay any events that occurred during its downtime.
- It also allows consumer to throttle and batch incoming data providing much needed flexibility while working with multiple types of data sources
- A simple and easy to use UI on cloud console for setup and debugging
- It enables event-driven architectures and asynchronous parallel processing, while improving performance, reliability and scalability
- It is limited to work with the same platform but with different datasets at the same time, you must request a prior security authentication.
- It can sometimes lead to unexpected charges, as Pub/Sub will automatically keep on retrying messages continuously, even if failures are due to permanent code-level issues.
- Message re-deliveries don't apply for ingested services like with Python based client. Push messages tried to be delivered immediately and if your service is busy dealing with some other task, it won't be done OR goes into a queue
October 26, 2020
Reliable Pub/Sub Vendor
Score 10 out of 10
Vetted Review
Verified User
Google Cloud Pub/Sub is used for processing scheduled tasks, sending notifications to clients and other background job execution. It helps in the point-in-time queue to send notifications to customers. As a team lead, I helped my team to transition
from the legacy queue system to the Google Cloud Pub/Sub real-time system to capture Play billing
alerts in real time.
- Queue Caching
- Notification Service
- Background Task
- Priority Queue is a missing feature
December 14, 2019
Pub/Sub, simple + reliable + scalable = a great product
Our company currently uses Pub/Sub to reliably queue up messages that arrive rapidly from multiple instances. It's a very technical product and really only used by the development team rather than company-wide. The primary problem it addresses for us is its ability to handle scale and resilience in the case of errors.
- Highly scalable.
- Queues messages for up to 7 days if not acknowledged.
- Would be nice if the queue could be extended beyond 7 days.
- We found it a bit tricky replay unacknowledged messages when needed.