Google Cloud Pub/Sub, In Depth: Topics, Subscriptions, Delivery & Ordering

Pub/Sub is the messaging backbone of Google Cloud. Whenever one part of a system needs to tell another part that something happened, and you do not want the sender to wait around for, or even know about, the receiver, Pub/Sub is the service that carries the message. It is a fully managed, globally available, horizontally scalable publish/subscribe system: publishers send messages to a topic, and Pub/Sub durably stores each message and delivers a copy to every subscription attached to that topic. The publisher and the subscriber never talk to each other directly, never need to be online at the same time, and never need to agree on how fast each can go. That decoupling — in space (who), in time (when), and in rate (how fast) — is the whole point.

People reach for Pub/Sub in three broad situations. The first is event-driven architecture: a service emits “order placed”, “file uploaded”, “user signed up” events and any number of downstream services react. The second is stream ingestion: millions of IoT, clickstream, or log events per second flowing into BigQuery, Dataflow, or Cloud Storage. The third is work distribution / task queues: a producer enqueues units of work and a fleet of workers pulls them off and processes them, scaling the fleet up or down with the backlog. Pub/Sub handles all three with the same primitives, which is why it shows up in almost every non-trivial GCP design.

This lesson is the exhaustive version. By the end you will know every subscription type, every delivery guarantee and the exact knobs that control it (acknowledgement deadlines, ordering keys, retry policies, dead-letter topics, flow control), how retention, seek, and snapshots let you replay history, how schemas enforce message structure, how filtering trims what a subscription receives, and when to choose Pub/Sub versus its cheaper cousin Pub/Sub Lite. It is written for someone who can deploy a Cloud Run service but has never operated a message bus, and it goes deep enough to carry you through an Associate Cloud Engineer or Professional Data Engineer exam and into production.

Learning objectives

After working through this lesson you will be able to:

• Explain the Pub/Sub data model — projects, topics, subscriptions, messages, and the publisher/subscriber decoupling — and reason about where a message physically lives at each stage.
• Choose correctly between the four subscription delivery types (pull, push, BigQuery, Cloud Storage) and configure each with every relevant setting.
• Configure and reason about delivery guarantees: at-least-once versus exactly-once, ordering keys, acknowledgement deadlines and extension, nacks, dead-letter topics, retry policies, and subscriber flow control.
• Use message retention, seek, and snapshots to replay or discard messages, and explain the difference between the two replay mechanisms.
• Attach and evolve schemas (Avro / Protocol Buffers), apply subscription filters, and decide between Pub/Sub and Pub/Sub Lite on cost and operational grounds.
• Operate Pub/Sub day-to-day with the gcloud pubsub CLI: build a working pipeline, validate it, monitor backlog, and tear it down cleanly.

Prerequisites & where this fits

You need a Google Cloud project with billing enabled (the Free Tier and the $300 new-customer credit comfortably cover this lesson — Pub/Sub also has a standing 10 GiB/month free throughput allowance), the Pub/Sub API enabled, the gcloud CLI installed and initialised, and an IAM role that lets you administer Pub/Sub (Owner, Editor, or the granular roles/pubsub.editor). A working mental model of IAM (service accounts and roles) helps, because push delivery and BigQuery/Cloud Storage subscriptions all hinge on granting the right service account the right permission.

In the Google Cloud Zero-to-Hero course this is the Integration module. It follows the BigQuery deep dive (Pub/Sub frequently feeds BigQuery, directly or via Dataflow) and precedes the Cloud KMS & Secret Manager lesson. The companion advanced lesson, Pub/Sub Delivery Guarantees: Exactly-Once, Ordering Keys, Dead-Letter, and Flow Control, drills into the reliability mechanics with production-grade subscriber code; this lesson is the broad, every-option foundation underneath it.

Core concepts

Five terms carry the whole service. Get these straight and everything else is configuration.

A topic is a named resource you publish to. It is purely a routing point — a topic on its own does not store messages for delivery; it fans them out to subscriptions. Its full name is projects/PROJECT_ID/topics/TOPIC_ID. A topic can have a schema attached (to validate message bodies) and a list of message storage policies (to restrict which regions messages may be persisted in, for data-residency compliance).

A message is the unit of data: a binary payload (the data field, up to 10 MB) plus optional attributes (a string-to-string map of metadata, e.g. eventType=order.created), an ordering key (optional), and Pub/Sub-assigned fields — a unique message ID and a publish timestamp. Attributes are how you filter and route without parsing the payload.

A subscription is the resource a consumer reads from, and it is where messages are actually stored and tracked. This is the single most important thing to internalise: each subscription is an independent queue. When a message is published to a topic, Pub/Sub writes a separate copy into the backlog of every subscription attached at that moment. Subscription A acknowledging a message has no effect on subscription B’s copy. Two subscriptions on the same topic let two independent consumers each get the full stream — that is the “fan-out” pattern. A subscription has exactly one topic; a topic can have many subscriptions (up to 10,000).

A publisher is any client (your app, another GCP service via Eventarc, a Cloud Scheduler job) that sends messages to a topic using the publish API. The client library batches and compresses messages for efficiency and retries on transient failures.

A subscriber is any client or service that receives messages from a subscription. How it receives them — by asking (pull) or by being called (push) or by Pub/Sub writing directly to a sink (BigQuery / Cloud Storage) — is the subscription type, and it is chosen per subscription, not per topic.

The lifecycle of a single message, end to end: a publisher sends it to a topic → Pub/Sub durably stores a copy in every attached subscription’s backlog (synchronously replicated across zones in the region before the publish is acknowledged) → a subscriber receives a copy → the subscriber acknowledges (acks) it, telling Pub/Sub “I have it, do not send it again” → Pub/Sub removes it from that subscription’s backlog. If the subscriber does not ack within the acknowledgement deadline, Pub/Sub redelivers. If a message is acked by nobody and reaches the message retention duration, it is dropped. That cycle — publish, store-per-subscription, deliver, ack, expire — is the entire engine.

Creating a topic: every setting

You can create a topic in the Console (Pub/Sub → Topics → Create topic) or with the CLI. Here is every field the create flow offers, with the what / choices / default / when / trade-off / gotcha treatment.

gcloud pubsub topics create orders \
  --message-retention-duration=7d \
  --message-storage-policy-allowed-regions=us-central1,us-east1 \
  --schema=order-schema --message-encoding=JSON \
  --kms-key-name=projects/$PROJECT/locations/us-central1/keyRings/pubsub/cryptoKeys/topic-key

After creation you can change retention duration, schema settings, message storage policy, CMEK key, and labels (gcloud pubsub topics update). You cannot rename a topic or change its ID.

Creating a subscription: the settings common to every type

Subscriptions are where most configuration lives. A handful of settings apply to all subscription types; the delivery-specific ones come in the next sections. Create one with:

gcloud pubsub subscriptions create orders-worker \
  --topic=orders \
  --ack-deadline=30 \
  --message-retention-duration=7d \
  --retain-acked-messages \
  --expiration-period=31d \
  --min-retry-delay=10s --max-retry-delay=600s \
  --dead-letter-topic=orders-dlq --max-delivery-attempts=5 \
  --enable-message-ordering \
  --enable-exactly-once-delivery

Most of these are mutable later via gcloud pubsub subscriptions update — the notable immutable ones are the ID, the topic, the ordering setting, and the filter. Now the four delivery types.

Subscription type 1 — Pull

A pull subscription is the default and the most flexible. Your subscriber code opens a connection and asks Pub/Sub for messages. There are two flavours:

• Unary / synchronous pull (subscriptions pull, or the Pull RPC): you request up to N messages, get whatever is available right now, process, and ack. Simple, but you poll. Good for low-volume or batch-style draining; the CLI uses this.
• StreamingPull (what the high-level client libraries use): a long-lived bidirectional gRPC stream. Pub/Sub pushes messages down the open stream as they arrive, your handler runs, and acks flow back up. This is the high-throughput path and what you use in production — you write a message handler callback and the library manages the stream, concurrency, ack-deadline extension, and flow control for you.

Pull’s superpower is flow control — the subscriber dictates the rate. The client library lets you cap how many messages and how many bytes are outstanding (received but not yet acked) at once:

from google.cloud import pubsub_v1
flow = pubsub_v1.types.FlowControl(max_messages=1000, max_bytes=100 * 1024 * 1024)
sub = pubsub_v1.SubscriberClient()
path = sub.subscription_path(PROJECT, "orders-worker")
def handle(msg):
    process(msg.data)        # your work
    msg.ack()                # or msg.nack() to request immediate redelivery
future = sub.subscribe(path, callback=handle, flow_control=flow)
future.result()

When the outstanding count hits the cap, the library stops accepting new messages until your handler acks some, which naturally throttles a fleet of workers to exactly what they can handle. Each ack() removes the message; each nack() asks Pub/Sub to redeliver immediately (subject to the retry policy). The library also automatically extends the ack deadline while your handler is still running, up to maxAckExtensionPeriod, so long-running work does not trigger a spurious redelivery.

When to choose pull: high or bursty throughput, a worker fleet you scale yourself, work that takes a while per message, or any case where the consumer must control its own pace. It is the default for a reason.

Subscription type 2 — Push

A push subscription inverts control: Pub/Sub sends each message to an HTTPS endpoint you provide, as an HTTP POST, and your endpoint replies 2xx to ack or non-2xx (or times out) to nack. There is no client library to run and no connection to maintain — your endpoint just has to exist and respond. The endpoint is typically a Cloud Run service, a Cloud Functions function, an App Engine handler, or any HTTPS server.

Push’s defining mechanism is flow control via the success rate: Pub/Sub uses a slow-start algorithm, ramping the push rate up while your endpoint returns 2xx quickly and backing off when it sees errors or slow responses. You do not set a numeric rate; you influence it by how fast and how reliably you ack.

The ack deadline still applies: if your endpoint does not respond 2xx within the deadline (default 10 s, raise it for slow handlers), Pub/Sub treats it as a nack and redelivers per the retry policy. Push does not support exactly-once delivery — it is strictly at-least-once, so your endpoint must be idempotent.

When to choose push: serverless, event-driven targets that should scale to zero and wake on demand (Cloud Run / Cloud Functions), webhook-style integrations, and cases where you would rather not run a long-lived subscriber process. Avoid push for very high sustained throughput (pull scales better), long per-message processing (you would push the ack deadline high and risk duplicates), or strict ordering plus exactly-once (push supports neither as well as pull). Eventarc, incidentally, is built on push subscriptions under the hood — it provisions a push subscription that calls your Cloud Run/Functions target when an event source fires.

Subscription type 3 — BigQuery

A BigQuery subscription writes messages directly into a BigQuery table — no subscriber code, no Dataflow job, no glue. Pub/Sub becomes the streaming-ingest pipe straight into your warehouse. This is the simplest possible “events to analytics” path and it is free of the message-delivery (subscriber) pricing; you pay BigQuery storage and the BigQuery Storage Write API.

gcloud pubsub subscriptions create orders-to-bq \
  --topic=orders \
  --bigquery-table=PROJECT:dataset.orders_raw \
  --use-topic-schema \
  --write-metadata \
  --drop-unknown-fields

Permissions are the classic trip-up: the Pub/Sub service agent (service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com) needs roles/bigquery.dataEditor on the dataset and roles/bigquery.metadataViewer. BigQuery subscriptions are at-least-once (duplicates possible — dedupe on message_id in queries) and do not support ordering. When to choose: raw event capture into BigQuery for analytics where you do not need transformation in flight (if you do, send to a topic that Dataflow reads).

Subscription type 4 — Cloud Storage

A Cloud Storage subscription batches messages and writes them as files (objects) into a GCS bucket — again with no subscriber code. It is the managed way to archive a stream to cheap object storage, or to land data for batch processing.

gcloud pubsub subscriptions create orders-to-gcs \
  --topic=orders \
  --cloud-storage-bucket=my-events-archive \
  --cloud-storage-file-prefix=orders/ \
  --cloud-storage-file-suffix=.json \
  --cloud-storage-max-bytes=10MB \
  --cloud-storage-max-duration=5m \
  --cloud-storage-output-format=text

Cloud Storage subscriptions are at-least-once and unordered. When to choose: long-term archival of a stream, building a data lake landing zone, or feeding batch jobs that read files. It is the cheapest way to durably keep every message you ever published.

Here is how the four types compare at a glance — this table is the one to memorise:

Delivery guarantees: at-least-once vs exactly-once

By default every Pub/Sub subscription is at-least-once: Pub/Sub guarantees it will deliver each message at least one time, but under retries, deadline expiries, or rare internal events it may deliver the same message more than once. The practical consequence is non-negotiable: at-least-once subscribers must be idempotent — processing the same message twice must produce the same result as processing it once (e.g. upsert by a business key, or dedupe on the message ID). This is not a bug to work around; it is the contract, and it is what lets the system be so available and scalable.

Exactly-once delivery is an opt-in mode (--enable-exactly-once-delivery) available on pull subscriptions only. It does not magically make duplicates impossible across your entire system; what it guarantees is, within the subscription: (1) an acknowledged message is not redelivered, and (2) acks become confirmed operations — ack() returns success only once Pub/Sub has durably recorded it, so you know whether the ack “took”. It also introduces the concept of ack IDs that can expire: if your processing runs long and the ack deadline lapses, the original ack ID becomes invalid, and acking it returns a failure rather than silently doing nothing — telling you the message will be redelivered so you must not commit its side effects. Exactly-once typically has slightly higher latency and lower per-subscription throughput than at-least-once, and it works hand-in-hand with ordering. The deep mechanics — idempotent ack handling, the AckResponse futures, and combining it with ordering and dead-lettering — are covered in the exactly-once / ordering / dead-letter lesson.

Acknowledgement deadlines, nacks, and deadline extension

The acknowledgement deadline is the contract clock. When Pub/Sub delivers a message, it starts a timer (the ack deadline, 10–600 s, default 10 s). If you ack before it expires, the message is done. If the timer expires first, Pub/Sub assumes delivery failed and redelivers (counting against delivery attempts). A nack is an explicit “I could not process this, send it again now” — equivalent to letting the deadline expire, but immediate.

The tension is obvious: set the deadline too short and slow work gets redelivered (duplicates, wasted effort); set it too long and a genuinely dead subscriber holds messages hostage for minutes. The resolution is dynamic ack-deadline extension: the high-level client libraries watch how long your handler actually takes and silently extend the deadline (via ModifyAckDeadline) in the background, up to a configurable maximum (maxAckExtensionPeriod, default 60 minutes). So in practice you set a modest base deadline (say 30 s), let the library extend it for the occasional long message, and only a truly stuck handler ever hits the ceiling. If you write a raw pull loop yourself, you are responsible for calling ModifyAckDeadline — forgetting this is a top cause of “why am I getting duplicates” tickets.

Ordering keys

By default Pub/Sub does not guarantee order — it optimises for throughput and availability, and messages can arrive out of the order they were published. When order matters (a sequence of edits to the same document, state transitions for one order, events for one user), you use ordering keys.

An ordering key is a string you set on each published message. The guarantee is scoped per key: messages with the same ordering key, published to the same region, are delivered in publish order — provided the subscription has ordering enabled (--enable-message-ordering, set at creation, immutable). Messages with different keys have no ordering relationship and still flow in parallel, so you keep most of the throughput. There is no global ordering; order is always per-key.

from google.cloud import pubsub_v1
opts = pubsub_v1.types.PublisherOptions(enable_message_ordering=True)
pub = pubsub_v1.PublisherClient(publisher_options=opts)
topic = pub.topic_path(PROJECT, "orders")
pub.publish(topic, b'{"state":"placed"}', ordering_key="order-123")
pub.publish(topic, b'{"state":"shipped"}', ordering_key="order-123")  # delivered after "placed"

Two important consequences. First, ordering and throughput trade off: all messages for one key are processed serially, so a single hot key is a serial bottleneck — design your keys to spread load (use the order ID, the user ID, the device ID; not a constant). Second, ordering interacts with failure: if a message for a key cannot be delivered/acked, Pub/Sub pauses delivery for that key until it is resolved (it will not skip ahead and break order). With ordering plus a dead-letter topic, a poison message is dead-lettered and the key resumes. If you enabled ordering on the publisher and a publish fails, you must resume_publish(ordering_key) before publishing more for that key.

Dead-letter topics and retry policies

What happens to a message that simply cannot be processed — malformed payload, a bug, a permanently-missing dependency? Without intervention it redelivers forever (until retention expires), wasting resources and possibly blocking an ordered key. The fix is a dead-letter topic (DLQ).

Configure a subscription with --dead-letter-topic and --max-delivery-attempts (5–100). After Pub/Sub has delivered a message that many times without an ack, it stops retrying on the main subscription and forwards the message to the dead-letter topic instead, stamping attributes with the original subscription and delivery-attempt count. You attach a subscription to the DLQ to inspect, alert on, fix, and optionally re-publish the poison messages once the bug is fixed. The DLQ turns “infinite retries of a bad message” into “a small, observable pile of failures you handle on your own schedule”.

DLQ has two IAM requirements that bite everyone the first time. The Pub/Sub service agent needs:

• roles/pubsub.publisher on the dead-letter topic (so Pub/Sub can write failures into it), and
• roles/pubsub.subscriber on the source subscription (so Pub/Sub can ack the message off the source once it has been dead-lettered).

SA="service-$(gcloud projects describe $PROJECT --format='value(projectNumber)')@gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud pubsub topics add-iam-policy-binding orders-dlq \
  --member="serviceAccount:$SA" --role=roles/pubsub.publisher
gcloud pubsub subscriptions add-iam-policy-binding orders-worker \
  --member="serviceAccount:$SA" --role=roles/pubsub.subscriber

The retry policy is the orthogonal control governing the gap between redeliveries. The default is immediate redelivery, which can hammer a struggling downstream. Switch to exponential backoff with --min-retry-delay and --max-retry-delay (each 0–600 s): Pub/Sub waits min, then doubles up to max between attempts, giving a flaky dependency room to recover. Retry policy and dead-lettering compose: back off between attempts, and after max-delivery-attempts give up to the DLQ.

Flow control

Flow control is how a subscriber protects itself from being overwhelmed by a backlog — it bounds the work in flight. For pull it is explicit and client-side: set max_messages and max_bytes (outstanding, i.e. received-but-not-yet-acked) in the client library; when the cap is reached the library stops fetching until acks free up capacity, so a fleet of workers self-throttles to exactly its capacity and memory stays bounded. For push it is implicit and server-side: Pub/Sub’s slow-start ramps the rate up while your endpoint stays fast and healthy and backs off on errors/latency — you tune it indirectly through how quickly and reliably you ack. Either way, the backlog itself is durable in Pub/Sub, so throttling never loses data; it just paces delivery. (Flow control mechanics and tuning are expanded in the exactly-once / ordering / dead-letter / flow-control lesson.)

Retention, seek, and snapshots

Three features let you control the past — keep messages around and rewind or fast-forward a subscription.

Retention comes in two layers. Subscription retention (10 min – 31 days, default 7 days) is how long unacknowledged messages persist in a subscription’s backlog; with --retain-acked-messages it also keeps acked messages for the same window so you can replay them. Topic retention (10 min – 31 days, off by default) keeps published messages at the topic so that a brand-new subscription created later can seek back and read history from before it existed — without it, a new subscription only sees messages published after its creation.

Seek moves a subscription’s acknowledgement state to a point in time or to a snapshot, in a single operation:

• seek --time=TIMESTAMP rewinds (mark everything after the timestamp as unacked, i.e. replay from then — bounded by what is retained) or fast-forwards (mark everything before the timestamp as acked, i.e. purge/skip the backlog up to then). To purge a backlog entirely, seek to now.
• seek --snapshot=SNAPSHOT jumps to a previously captured snapshot.

# Replay everything from the last hour
gcloud pubsub subscriptions seek orders-worker \
  --time=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)
# Purge: skip the whole current backlog
gcloud pubsub subscriptions seek orders-worker --time=$(date -u +%Y-%m-%dT%H:%M:%SZ)

Snapshots capture the acknowledgement state of a subscription at a moment in time so you can return to it later. The classic use is a safe deploy: before rolling out a risky new subscriber version, snapshot the subscription; if the new version mishandles messages, seek back to the snapshot and replay everything from that point with the fixed code — nothing is lost. A snapshot retains the backlog it references for up to 7 days (it holds messages alive even if normal retention would have dropped them).

gcloud pubsub snapshots create pre-deploy-snap --subscription=orders-worker
# ... deploy v2, something's wrong ...
gcloud pubsub subscriptions seek orders-worker --snapshot=pre-deploy-snap

Seek vs snapshot, the exam distinction: seek-to-time replays based on the retention window and needs no prior action; a snapshot must be taken in advance but captures the exact ack state at that instant and is the right tool when you want to mark a known-good point before a risky change.

Schemas

By default a Pub/Sub payload is an opaque blob — publishers and subscribers must agree on its shape out of band, and nothing stops a publisher sending garbage. A schema makes that contract explicit and enforced. You define a schema as Avro or Protocol Buffers, attach it to a topic (with a message encoding of JSON or BINARY), and from then on Pub/Sub validates every published message against it and rejects non-conforming ones at publish time — bad data never enters the pipeline.

gcloud pubsub schemas create order-schema \
  --type=AVRO \
  --definition='{"type":"record","name":"Order","fields":[
    {"name":"orderId","type":"string"},
    {"name":"amount","type":"double"}]}'
gcloud pubsub topics create orders --schema=order-schema --message-encoding=JSON

Schemas support revisions for safe evolution: you add a new revision and Pub/Sub can accept a range of revisions on the topic, so producers and consumers can migrate independently as long as changes are compatible (adding optional fields is safe; removing required ones is not). Schemas pair naturally with BigQuery subscriptions (--use-topic-schema maps fields to columns) and with strongly-typed consumers. When to use: any pipeline with more than one team or service touching the messages — the schema is the documented, enforced contract.

Filtering

A filter lets a subscription receive only the messages it cares about, evaluated server-side on message attributes — so unmatched messages are never delivered, never stored against that subscription, and (importantly) not billed as delivered throughput. You set the filter when creating the subscription, and it is immutable thereafter (to change it, recreate the subscription).

gcloud pubsub subscriptions create high-value-orders \
  --topic=orders \
  --message-filter='attributes.eventType = "order.created" AND attributes.tier = "gold"'

The filter language works on the attributes map: equality (attributes.key = "v"), inequality (!=), existence (attributes:key), prefix (hasPrefix(attributes.key, "ord")), and boolean combinations (AND, OR, NOT). It operates only on attributes, not on the payload body — so publish the fields you want to route on as attributes. The pattern is powerful: one topic carries a mixed stream, and several filtered subscriptions each peel off their slice (orders to fulfilment, refunds to finance) without any consumer-side discarding. Gotcha: because the filter is fixed at creation and applied before delivery, a message that matches no subscription filter is simply dropped for that subscription — make sure your filters collectively cover everything you must not lose, or keep an unfiltered “catch-all” subscription.

Pub/Sub vs Pub/Sub Lite

Google offers a second, cheaper messaging product, Pub/Sub Lite, and the exam loves to test the distinction. They solve the same problem with very different trade-offs.

Pub/Sub (the service this lesson covers) is fully managed and serverless: it auto-scales capacity, is global (publish in one region, subscribe in another), and you pay per data volume with no capacity to provision. You never think about throughput limits in normal operation.

Pub/Sub Lite is a cost-optimised, zonal/regional service where you provision capacity in advance (publish and subscribe throughput, and storage, as explicit units) — and because you pre-provision rather than pay for elastic on-demand capacity, it can be dramatically cheaper (often ~80–90%) for predictable, high-volume workloads. The cost is operational: you must capacity-plan, it is not global, and it lacks several Pub/Sub conveniences (no push subscriptions, no exactly-once, no filtering, no BigQuery/Cloud Storage subscriptions, weaker built-in dead-lettering).

Note (2026): Google has announced the deprecation of Pub/Sub Lite, steering predictable-throughput workloads toward Pub/Sub or Managed Service for Apache Kafka. Treat Pub/Sub as the default for new designs; understand Lite mainly for the conceptual contrast (provisioned vs serverless capacity) the exam still references.

Eventarc integration

You will frequently meet Pub/Sub indirectly through Eventarc, GCP’s standardised eventing layer. Eventarc routes events from sources (Cloud Storage object events, Audit Log events from ~140 services, custom sources) to targets (Cloud Run, GKE, Workflows) using Pub/Sub as the transport underneath — for many event types it provisions a Pub/Sub topic and a push subscription automatically. You can also publish your own events to a Pub/Sub topic and have Eventarc deliver them. The takeaway: understanding Pub/Sub subscriptions (especially push and OIDC auth) directly explains how Eventarc behaves, and you can drop to raw Pub/Sub whenever Eventarc’s abstraction is too coarse.

Diagram: the Pub/Sub data flow

The diagram traces a message from publishers into a topic, fanning out to the four subscription types — pull (your worker fleet with flow control), push (an HTTPS endpoint such as Cloud Run), BigQuery (straight into a table), and Cloud Storage (batched into objects) — and shows the reliability path off to the side: ack deadline → retry policy → dead-letter topic, with retention/seek/snapshots governing replay.

Hands-on lab

This lab builds a complete pull pipeline, exercises delivery and dead-lettering, then a snapshot/seek replay, entirely within the Free Tier. Set your project first.

export PROJECT=$(gcloud config get-value project)
export NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)')
gcloud services enable pubsub.googleapis.com

1. Create a topic, a working subscription, and a dead-letter topic + DLQ subscription.

gcloud pubsub topics create lab-orders
gcloud pubsub topics create lab-orders-dlq
gcloud pubsub subscriptions create lab-dlq-sub --topic=lab-orders-dlq

gcloud pubsub subscriptions create lab-worker \
  --topic=lab-orders \
  --ack-deadline=20 \
  --message-retention-duration=1d \
  --retain-acked-messages \
  --dead-letter-topic=lab-orders-dlq \
  --max-delivery-attempts=5 \
  --min-retry-delay=10s --max-retry-delay=60s

2. Grant the Pub/Sub service agent the DLQ permissions (without these, dead-lettering silently fails).

SA="service-${NUM}@gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud pubsub topics add-iam-policy-binding lab-orders-dlq \
  --member="serviceAccount:$SA" --role=roles/pubsub.publisher
gcloud pubsub subscriptions add-iam-policy-binding lab-worker \
  --member="serviceAccount:$SA" --role=roles/pubsub.subscriber

3. Publish messages, including attributes for filtering practice.

gcloud pubsub topics publish lab-orders --message='{"orderId":"A1","amount":50}' \
  --attribute=eventType=order.created,tier=gold
gcloud pubsub topics publish lab-orders --message='{"orderId":"A2","amount":10}' \
  --attribute=eventType=order.created,tier=silver

4. Pull and acknowledge. Pull without --auto-ack first to see the message and its ackId, then pull with auto-ack to drain.

gcloud pubsub subscriptions pull lab-worker --limit=5 --format=json
gcloud pubsub subscriptions pull lab-worker --limit=5 --auto-ack

Expected output: JSON entries with base64-encoded data, your attributes, a messageId, and (first command) an ackId. After the auto-ack pull, the backlog drains.

5. Validate state. Inspect the subscription and check the backlog metric.

gcloud pubsub subscriptions describe lab-worker
gcloud pubsub topics list-subscriptions lab-orders

You should see deadLetterPolicy, retryPolicy, ackDeadlineSeconds: 20, and retainAckedMessages: true.

6. Exercise dead-lettering (conceptual within the CLI): publish a “poison” message and pull it repeatedly without acking (omit --auto-ack) more than five times — Pub/Sub will route it to lab-orders-dlq. Confirm by pulling the DLQ subscription:

gcloud pubsub topics publish lab-orders --message='{"poison":true}'
# pull a few times WITHOUT --auto-ack to burn delivery attempts, then:
gcloud pubsub subscriptions pull lab-dlq-sub --limit=5 --auto-ack

7. Snapshot and seek (replay). Because we set --retain-acked-messages, we can rewind.

gcloud pubsub snapshots create lab-snap --subscription=lab-worker
# Replay everything from the last 10 minutes:
gcloud pubsub subscriptions seek lab-worker \
  --time=$(date -u -v-10M +%Y-%m-%dT%H:%M:%SZ)
gcloud pubsub subscriptions pull lab-worker --limit=5 --auto-ack   # acked messages re-appear

Cleanup (do this — idle subscriptions and retained messages can accrue tiny charges):

gcloud pubsub snapshots delete lab-snap
gcloud pubsub subscriptions delete lab-worker lab-dlq-sub
gcloud pubsub topics delete lab-orders lab-orders-dlq

Cost note: Pub/Sub’s first 10 GiB/month of throughput is free, and this lab moves a handful of tiny messages, so it costs effectively nothing. The only things that could accrue charges are retained messages (storage) and snapshots (they hold messages alive) — which is exactly why the cleanup deletes them. The Free Tier / $300 credit absorbs all of it regardless.

Common mistakes & troubleshooting

Best practices

• Always make subscribers idempotent. At-least-once is the default and exactly-once only narrows the window; design every consumer to tolerate a repeat.
• Set ack deadlines to reality and use the high-level client libraries so the deadline is auto-extended for long messages — do not hand-roll pull loops unless you must.
• Attach a dead-letter topic (with a retry policy) to every important subscription, and alert on DLQ depth — it converts silent infinite retries into a visible, handleable failure.
• Set subscription expiration-period to never for anything in production to avoid the default auto-deletion.
• Use one topic per event type and filter at the subscription, rather than one topic per consumer; it keeps publishers simple and lets new consumers subscribe without touching producers.
• Attach schemas to topics that more than one team touches; reject bad data at the edge rather than debugging it downstream.
• Use ordering keys only where you truly need order, and choose high-cardinality keys to avoid serial hot-spots.
• Prefer the managed sink subscriptions (BigQuery / Cloud Storage) over writing and operating glue code when you just need events landed for analytics or archival.
• Snapshot before risky deploys so you can replay from a known-good point.

Security notes

• IAM is the access boundary. Grant roles/pubsub.publisher to publishers and roles/pubsub.subscriber to subscribers — and nothing broader. Avoid roles/pubsub.editor/admin on workload identities; reserve those for operators/IaC.
• Use per-service service accounts for publishers and subscribers so you can attribute and revoke access precisely; never embed long-lived keys — use Workload Identity (GKE) or attached service accounts (Cloud Run/Functions/GCE).
• Secure push subscriptions with OIDC authentication and verify the token (issuer, audience, signature) in your endpoint; an unauthenticated push endpoint is an open door for spoofed messages.
• Encrypt with CMEK where compliance requires customer-managed keys; remember the Pub/Sub service agent needs encrypt/decrypt on the key, and key loss makes messages unrecoverable.
• Use message storage policies to keep message data within required regions for data residency.
• Audit Logs (Admin Activity is always on; enable Data Access logs for publish/subscribe visibility) record who did what — wire them into your central log sink.
• Mind the DLQ service-agent grants: they are deliberately scoped to one topic and one subscription; do not widen them.

Interview & exam questions

1. A topic has three subscriptions. A message is published and one subscription’s subscriber acks it. What happens to the other two subscriptions’ copies? Nothing — each subscription is an independent queue with its own copy and its own ack state. The other two still hold (and must independently ack) the message. This fan-out independence is fundamental.

2. What is the default delivery guarantee, and what does it require of your code? At-least-once: every message is delivered one or more times, so subscribers must be idempotent (dedupe on message ID or a business key; use upserts).

3. What does enabling exactly-once delivery actually guarantee, and which subscription types support it? On pull subscriptions only: an acknowledged message is not redelivered, and acks are confirmed (you know whether the ack succeeded, and a lapsed ack ID fails rather than silently no-ops). It does not eliminate duplicates across unrelated systems — within the subscription it removes redelivery of acked messages.

4. Explain ordering keys. Is there global ordering? An ordering key scopes order: messages with the same key (published to the same region) are delivered in publish order if the subscription has ordering enabled. Different keys are unordered/parallel. There is no global ordering; design keys for high cardinality to avoid a serial bottleneck.

5. How do you stop a malformed message from being retried forever, and what IAM does that require? Configure a dead-letter topic with --max-delivery-attempts; after N failed deliveries the message is forwarded there. The Pub/Sub service agent needs roles/pubsub.publisher on the DLQ topic and roles/pubsub.subscriber on the source subscription.

6. Difference between the acknowledgement deadline and the retry policy? The ack deadline is how long a subscriber has to ack one delivery before it is considered failed and redelivered. The retry policy controls the delay between redeliveries (immediate vs exponential backoff). Orthogonal knobs.

7. Push vs pull — when each, and what flow-control model does each use? Pull: you run a subscriber, control the rate with client-side flow control (max outstanding messages/bytes); best for high/bursty throughput, long processing, exactly-once, ordering. Push: Pub/Sub POSTs to your HTTPS endpoint, rate governed by a server-side slow-start on your success rate; best for serverless/event-driven targets and webhooks. Push is at-least-once only.

8. You need every event landed in BigQuery for analytics with no transformation. What is the simplest design? A BigQuery subscription writing straight to a table (--use-topic-schema, --write-metadata) — no subscriber code or Dataflow. Grant the service agent bigquery.dataEditor; dedupe on message_id since it is at-least-once.

9. Seek vs snapshot — what is the difference and when do you use each? Seek to a timestamp rewinds/fast-forwards a subscription’s ack state within the retention window, no prior action needed. A snapshot must be taken in advance but captures the exact ack state at that instant; you seek to it later. Use a snapshot to mark a known-good point before a risky deploy; use seek-to-time for ad-hoc replay or to purge a backlog (seek to now).

10. Pub/Sub vs Pub/Sub Lite? Pub/Sub is serverless, global, pay-per-volume, feature-rich (push, exactly-once, filtering, sink subscriptions). Pub/Sub Lite was zonal/regional with pre-provisioned capacity, much cheaper for predictable high volume but far fewer features — and is now being deprecated, so prefer Pub/Sub (or Managed Kafka) for new work.

11. How do you ensure a publisher can only send well-formed messages? Attach a schema (Avro/Protobuf) to the topic with a JSON or BINARY encoding; Pub/Sub validates and rejects non-conforming publishes. Use schema revisions for safe evolution.

12. A subscription disappeared after a quiet weekend. Why, and how do you prevent it? The default subscription expiration period (31 days of inactivity) deleted it. Set --expiration-period=never for production subscriptions.

Quick check

1. Where do messages physically sit waiting for delivery — on the topic or on the subscription?
2. Which two subscription types require no subscriber code at all?
3. Which subscription type(s) support exactly-once delivery?
4. What two IAM grants does dead-lettering need, and on which resources?
5. What must you set on a subscription so that seek --time can replay messages you have already acked?

Answers

1. On the subscription — each subscription holds its own copy of the backlog; the topic only routes/fans out.
2. BigQuery and Cloud Storage subscriptions (Pub/Sub writes to the sink for you).
3. Pull subscriptions only (opt-in with --enable-exactly-once-delivery).
4. roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the source subscription, both for the Pub/Sub service agent.
5. --retain-acked-messages (and the target time must fall within the retention duration).

Exercise

Design and build a small “order processing” pipeline that demonstrates fan-out and reliability. Create a topic shop-orders with an attached Avro schema (orderId: string, amount: double, region: string). Attach three subscriptions: (a) a filtered pull subscription eu-orders that only receives messages where attributes.region = "eu"; (b) a BigQuery subscription orders-warehouse writing to a table you create, with metadata; and © a Cloud Storage subscription orders-archive writing newline-delimited JSON, rolling files every 2 minutes. Add a dead-letter topic to the pull subscription with 5 max attempts and the required IAM grants, plus an exponential retry policy. Publish ten messages (mix of eu/us regions, one deliberately malformed for the schema), and verify: the filtered subscription only sees EU messages, the malformed publish is rejected at publish time, valid messages appear in BigQuery and as objects in the bucket, and (by repeatedly failing to ack) a poison message lands in the DLQ. Finally, snapshot the pull subscription, seek back 5 minutes, and confirm replay. Tear everything down. Write down which settings were immutable and would have forced a recreate.

Certification mapping

• Associate Cloud Engineer (ACE): expect to set up topics and subscriptions, pick pull vs push, and reason about basic delivery and IAM for Pub/Sub. Know that subscriptions store messages, the default at-least-once guarantee, and how to grant publisher/subscriber roles.
• Professional Data Engineer (PDE): the heavier exam here. Expect scenario questions on streaming ingestion (Pub/Sub → Dataflow → BigQuery, or BigQuery subscriptions directly), exactly-once vs at-least-once and idempotency, ordering keys, dead-letter handling, replay with seek/snapshots, schemas, and Pub/Sub vs Pub/Sub Lite trade-offs. Filtering and the four subscription types are common distractor-rich questions.
• Professional Cloud Architect (PCA): Pub/Sub as the decoupling/fan-out backbone in event-driven architectures, choosing it versus Cloud Tasks (targeted HTTP task dispatch with per-task scheduling) and versus a self-managed Kafka, and data-residency via message storage policies / CMEK.

Glossary

• Topic — a named publish target that fans messages out to all attached subscriptions; does not itself store messages for delivery.
• Subscription — an independent queue bound to one topic; holds the message backlog and ack state for one consumer.
• Message — payload (data, ≤10 MB) + attributes (string map) + optional ordering key + Pub/Sub-assigned ID and publish time.
• Pull / StreamingPull — subscription type where the subscriber fetches messages (synchronously, or over a long-lived gRPC stream).
• Push — subscription type where Pub/Sub POSTs each message to your HTTPS endpoint; ack via 2xx.
• BigQuery / Cloud Storage subscription — managed subscription types that write messages directly into a BigQuery table or GCS objects, no subscriber code.
• Acknowledgement (ack) / nack — telling Pub/Sub a message is processed (don’t redeliver) / failed (redeliver).
• Acknowledgement deadline — time a subscriber has to ack before Pub/Sub redelivers; auto-extended by the client libraries.
• At-least-once / exactly-once — default guarantee (possible duplicates, be idempotent) / opt-in pull-only guarantee (no redelivery of acked messages, confirmed acks).
• Ordering key — a string that makes same-key messages deliver in publish order (per region, ordering enabled).
• Dead-letter topic (DLQ) — where messages go after exceeding max delivery attempts, instead of retrying forever.
• Retry policy — backoff (immediate or exponential) between redeliveries.
• Flow control — bounding outstanding (unacked) messages/bytes so a subscriber isn’t overwhelmed.
• Retention — how long messages are kept (subscription-level for backlog/replay; topic-level for future subscriptions).
• Seek — move a subscription’s ack state to a timestamp or snapshot (replay or purge).
• Snapshot — a captured ack state you can seek back to later (safe deploys, point-in-time replay).
• Schema — an Avro/Protobuf contract attached to a topic that validates published messages.
• Filter — a server-side, attribute-based expression that delivers only matching messages to a subscription (immutable, set at creation).
• Service agent — the Google-managed service-NUM@gcp-sa-pubsub account Pub/Sub uses to write to DLQ topics, BigQuery, GCS, and KMS.
• Pub/Sub Lite — a cheaper, capacity-provisioned, zonal/regional messaging service (being deprecated in 2026).

Next steps

• Go deep on the reliability mechanics with Pub/Sub Delivery Guarantees: Exactly-Once, Ordering Keys, Dead-Letter, and Flow Control — production subscriber code, idempotent ack handling, and tuning.
• Continue the course with Google Cloud KMS & Secret Manager, In Depth to secure the keys and secrets your publishers and subscribers depend on, including the CMEK keys referenced above.
• Pair this with the BigQuery deep dive to build the full Pub/Sub → BigQuery streaming-analytics path end to end.