Pub/Sub Delivery Guarantees: Exactly-Once, Ordering Keys, Dead-Letter, and Flow Control

In a nutshell

Think of Pub/Sub as a postal sorting office sitting between the people who send letters and the people who read them. A publisher drops a letter addressed to a topic (a named channel). The office makes a copy for every reader who signed up — each reader’s copy lives in their own subscription (their personal PO box). The office keeps re-delivering your letter until you sign the receipt (ack). By default the service is at-least-once: reliable, but every so often you get the same letter twice. That is not a bug — it is the deal, and the fix is to make your reader shrug at a duplicate (an idempotent consumer).

The four features in this lesson are the office’s premium options, and beginners routinely confuse them:

• Exactly-once delivery is registered mail with a tracked receipt — the office promises never to hand you the same letter twice once you have signed for it. It is slower and pricier, and it only covers the office’s own re-delivery — it cannot un-send a letter a nervous publisher mailed twice.
• Ordering keys are the instruction “deliver this one customer’s letters strictly in the order I posted them.” Order is guaranteed per key, not globally — and each such tracked lane moves only so fast.
• Dead-letter topics are the undeliverable-mail bin: after N failed delivery attempts, the office sets the poison letter aside so the rest of the mail keeps moving.
• Flow control is you telling the courier “don’t dump more than 100 parcels on my porch at once” — a limit that lives in your mail-room, not in the post office.

The single most important distinction to carry through the whole lesson: exactly-once is about how many times a message is delivered; ordering is about what sequence messages arrive in. They are independent knobs. You can have either, both, or neither. Confusing the two is the number-one Pub/Sub mistake.

Level: Advanced · Time: ~27 min

Read the diagram left to right: a publisher sends a message to a topic, which durably fans a copy into every attached subscription; the subscriber leases each message over one of four delivery mechanisms and acks it, and the six numbered points — flow control, ack deadline, delivery type, exactly-once, ordering, dead-letter — are the reliability knobs this lesson tunes.

Prerequisites & what you’ll be able to do

This is the advanced reliability companion to the broad foundation. Before this lesson you should be comfortable with:

• The Pub/Sub data model — topics, subscriptions as independent per-consumer queues, messages, publish/ack — from the Pub/Sub deep dive. This lesson assumes you already know what a subscription is; here we make it reliable.
• Running gcloud and a client library (the code here is Python, google-cloud-pubsub).
• Basic IAM — service accounts, predefined roles, policy bindings — from IAM fundamentals. The dead-letter section hinges on granting a Google-managed service account two roles.
• Optionally, how events flow into consumers, from Cloud Functions 2nd gen & Eventarc and the event-driven reference architecture.

After working through it you will be able to:

• Explain, precisely, what exactly-once delivery does and does not guarantee, and decide when it is worth its cost versus plain idempotency.
• Write a subscriber that only commits “processed” state after the ack future resolves, so an ack failure never leaves stale dedupe state.
• Choose ordering keys at the right granularity, avoid the single-hot-key throughput trap, and recover a wedged key with resume_publish.
• Attach retry policies and dead-letter topics correctly — including the IAM grants everyone forgets — and design a triage-then-replay reprocessing path.
• Size subscriber flow control to your container and handler latency, and pick pull, push, StreamingPull, or a managed export subscription per consumer.
• Operate the whole thing from Cloud Monitoring: backlog, oldest unacked age, and expired ack deadlines.

Pub/Sub is easy to start with and easy to get wrong. The defaults give you a horizontally scalable, at-least-once bus that will happily redeliver messages, reorder them across partitions, and silently retry a poison message forever while your subscriber CPU melts. Every one of those behaviors is configurable, but the configuration is subtle: exactly-once is region-scoped and pull-only, ordering keys cap your throughput, dead-letter topics need IAM you have to grant by hand, and flow control lives in the subscriber client rather than the subscription resource.

This is a working guide to wiring all of it correctly. Commands are gcloud and Python client library. Replace PROJECT_ID and PROJECT_NUMBER with your own throughout.

1. Delivery semantics: at-least-once vs exactly-once

Pub/Sub’s default is at-least-once delivery. A message is delivered to a subscriber at least once; under normal operation usually exactly once, but duplicates are expected and legal. Duplicates arise from three sources you cannot fully eliminate at the at-least-once tier:

• Ack deadline expiry. The subscriber held the message past its ack deadline (network blip, slow handler, GC pause), so Pub/Sub redelivers.
• Lost acks. The subscriber acked, but the ack didn’t reach the service in time, so the message is redelivered.
• Publisher retries. A publish RPC times out, the client retries, and the same logical event lands twice with different message IDs.

The correct baseline posture is idempotent consumers. Design every handler so that processing the same business event twice is a no-op: dedupe on a stable business key, use conditional writes, or fold into idempotent upserts. If your consumer is idempotent, at-least-once is almost always the right and cheapest choice.

Exactly-once delivery is a stronger, opt-in guarantee that is frequently misunderstood. It does not mean a message is processed exactly once across your whole system; it means that within a single subscription, once a message is successfully acknowledged, it will not be redelivered, and while a message is outstanding (lease not expired) it will not be redelivered to another subscriber. It removes the ack-deadline and lost-ack duplicate classes, not publisher-side duplicates. It costs more, has higher latency, and lower throughput. Reach for it only when idempotency is genuinely impractical.

Rule of thumb: make consumers idempotent first. Add exactly-once only for the small set of subscriptions where dedupe is impossible (e.g. non-idempotent financial side effects that can’t carry a business key).

Side by side, so the boundary is unambiguous:

The takeaway the table encodes: exactly-once shrinks the duplicate surface, it does not erase it. A publisher that double-publishes, or a second subscription reading the same topic, still sees the event twice. Idempotency is the property that survives every one of those cases; exactly-once is a targeted assist on top of it.

2. Enabling exactly-once delivery and idempotent ack handling

Exactly-once is a subscription property. Two hard constraints: it is pull-only (not supported on push subscriptions, because the push receiver can’t confirm the service received its response), and it only holds when subscribers connect in a single region. From outside Google Cloud, use a locational endpoint (for example us-east1-pubsub.googleapis.com:443) rather than the global one so all subscriber connections pin to one region.

gcloud pubsub subscriptions create orders-eo-sub \
  --topic=orders \
  --enable-exactly-once-delivery \
  --ack-deadline=60 \
  --message-retention-duration=7d

A 60s ack deadline is the recommended default for exactly-once subscriptions: longer deadlines absorb transient network events that would otherwise cause redelivery. The deadline range is 10 to 600 seconds.

The behavioral change you must code for is on the ack side. With exactly-once, an ack/nack/modAck returns a status the client can observe, and only the most recent ack ID for a message is valid — an ack ID expires when the deadline passes or when the lease is extended, and a stale ack ID returns INVALID_ARGUMENT. The client libraries surface this through a future on the ack call. You must wait for the ack to confirm before treating the message as durably done:

from concurrent.futures import TimeoutError
from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
sub_path = subscriber.subscription_path("PROJECT_ID", "orders-eo-sub")

def callback(message: pubsub_v1.subscriber.message.Message) -> None:
    try:
        process(message.data)  # your idempotent-ish side effect
    except Exception:
        # nack: let the retry policy decide redelivery timing
        nack_future = message.nack_with_response()
        nack_future.result()
        return

    # With exactly-once, ack() returns a future. Only treat the message
    # as done once the service confirms the ack succeeded.
    ack_future = message.ack_with_response()
    try:
        ack_future.result()  # raises if the ack was not accepted
    except Exception:
        # Ack failed (e.g. lease expired). The message WILL be redelivered;
        # do not commit any "already processed" marker here.
        return

flow = pubsub_v1.types.FlowControl(max_messages=100, max_bytes=50 * 1024 * 1024)
future = subscriber.subscribe(sub_path, callback=callback, flow_control=flow)
try:
    future.result()
except TimeoutError:
    future.cancel()
    future.result()

The critical discipline: do not record “I processed this” until the ack future resolves successfully. If the ack fails, the message is coming back, and your dedupe state must reflect that.

3. Ordering keys: guarantees, trade-offs, and resume-on-failure

Pub/Sub does not order messages globally. With ordering keys, messages that share the same key, published to the same region, are delivered to a given subscription in publish order. Messages with an empty ordering key are not ordered. Enable ordering on the subscription:

gcloud pubsub subscriptions create accounts-ordered-sub \
  --topic=accounts \
  --enable-message-ordering \
  --ack-deadline=30

On the publisher, you must set enable_message_ordering=True and stamp each message with an ordering key (an account ID, an aggregate ID — never a high-cardinality random value):

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient(
    publisher_options=pubsub_v1.types.PublisherOptions(enable_message_ordering=True)
)
topic_path = publisher.topic_path("PROJECT_ID", "accounts")

key = "acct-42"
future = publisher.publish(topic_path, b'{"event":"debit"}', ordering_key=key)
future.result()  # block to preserve order; a failure here matters (see below)

Two trade-offs you must design around:

• Throughput cap. Publishing throughput per ordering key is limited to 1 MB/s. Ordering serializes a key, so a hot key is a hard bottleneck. Choose keys that spread load: per-customer, per-device, per-aggregate — not a single global key.
• Redelivery cascades. If a message for a key is redelivered, all subsequent messages for that key are redelivered too, including already-acked ones, to preserve order. A single slow or failing message stalls its entire key, and unacked messages for one key can delay other keys during server restarts or rebalancing.

Resume-on-failure is the publisher-side gotcha. If a publish for an ordering key fails, the client library pauses all further publishes for that key and fails them until you explicitly resume. After handling the failure, call resume_publish for that key, otherwise that key is stuck:

key = "acct-42"
future = publisher.publish(topic_path, b'{"event":"credit"}', ordering_key=key)
try:
    future.result()
except Exception:
    # All subsequent publishes for this key are now rejected until resumed.
    publisher.resume_publish(topic_path, key)

You can combine ordering with exactly-once (--enable-message-ordering --enable-exactly-once-delivery); the subscriber must then ack in order.

4. Retry policies, exponential backoff, and redelivery

By default, when an ack deadline expires or a subscriber nacks, Pub/Sub redelivers immediately. A handler failing on a transient downstream dependency will hot-loop, hammering both the dependency and your error budget. Attach an exponential backoff retry policy so redelivery spreads out:

gcloud pubsub subscriptions update orders-eo-sub \
  --min-retry-delay=10s \
  --max-retry-delay=300s

Both bounds range from 10 to 600 seconds. Pub/Sub starts near the minimum and grows the delay toward the maximum on repeated failures for the same message. Note the interaction with ordering: backoff on a key holds up that key’s later messages, which is usually what you want (don’t skip ahead past a failed event) but is worth stating in your design docs.

To revert to immediate retry, clear the policy:

gcloud pubsub subscriptions update orders-eo-sub --clear-retry-policy

5. Dead-letter topics: configuration, IAM, and reprocessing

A retry policy delays poison messages but never stops them. A dead-letter topic caps delivery attempts and offloads the failures so the main subscription keeps flowing. Create a dedicated DLT plus a subscription on it (so messages are retained and inspectable), then attach the policy:

# 1. Dead-letter topic and a subscription to hold failures
gcloud pubsub topics create orders-dlq
gcloud pubsub subscriptions create orders-dlq-sub --topic=orders-dlq \
  --message-retention-duration=7d

# 2. Attach the dead-letter policy to the live subscription
gcloud pubsub subscriptions update orders-eo-sub \
  --dead-letter-topic=orders-dlq \
  --max-delivery-attempts=10

--max-delivery-attempts accepts 5 to 100 (default 5). It is approximate — Pub/Sub forwards on a best-effort basis — so don’t treat it as an exact counter.

The IAM step everyone forgets. Forwarding to the DLT and acking the original message are performed by the Pub/Sub service agent, not your identity. That agent needs two grants, and if you skip them the policy silently fails to forward. The service agent is service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com:

PUBSUB_SA="service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com"

# Publish forwarded messages into the dead-letter topic
gcloud pubsub topics add-iam-policy-binding orders-dlq \
  --member="serviceAccount:${PUBSUB_SA}" \
  --role="roles/pubsub.publisher"

# Acknowledge the undeliverable message on the source subscription
gcloud pubsub subscriptions add-iam-policy-binding orders-eo-sub \
  --member="serviceAccount:${PUBSUB_SA}" \
  --role="roles/pubsub.subscriber"

Reprocessing pattern. Don’t point a consumer directly at the DLT in a loop — you’ll recreate the hot-loop. Treat the DLQ as a quarantine: alert on it, triage, fix the bug or bad data, then replay. A clean replay path is a small job that pulls from orders-dlq-sub and republishes to the original orders topic once the root cause is resolved. Pub/Sub stamps delivery_attempt on dead-lettered messages, so your triage tooling can read it directly off the message attributes.

6. Subscriber flow control and outstanding-message tuning

Flow control is client-side, not a subscription property. StreamingPull will deliver as fast as it can; without limits, a subscriber pulls thousands of outstanding messages, blows its memory, and starts missing ack deadlines (which, with exactly-once or ordering, triggers exactly the redelivery storm you were trying to avoid). You bound concurrency with FlowControl:

from google.cloud import pubsub_v1

flow = pubsub_v1.types.FlowControl(
    max_messages=200,               # max outstanding (unacked) messages
    max_bytes=200 * 1024 * 1024,    # max outstanding bytes (200 MiB)
    max_lease_duration=600,         # cap total time the client extends a lease (s)
)
future = subscriber.subscribe(sub_path, callback=callback, flow_control=flow)

Tuning guidance:

• Size max_messages to roughly handler_throughput_per_sec * p99_handler_latency_sec, then bound max_bytes to a safe fraction of container memory. Whichever limit is hit first pauses delivery.
• The client auto-extends leases up to max_lease_duration. If a handler legitimately runs long, raise this so the lease isn’t lost mid-processing — but with a ceiling, so a wedged handler doesn’t pin a message forever.
• StreamingPull pauses cleanly under backpressure: when flow-control limits are reached the server stops sending without breaking the connection, and resumes when capacity frees up. This is why StreamingPull, not unary Pull, is the default for throughput-sensitive workloads.

Scale horizontally for throughput (more subscriber instances on the same subscription), and use flow control to keep each instance stable.

7. Push vs pull vs StreamingPull and managed export subscriptions

Pick the delivery mechanism to match the consumer:

For sink-style ingestion, prefer the managed export subscriptions over hand-rolled consumers. A BigQuery subscription writes messages directly to a table with no subscriber to operate:

gcloud pubsub subscriptions create events-to-bq \
  --topic=events \
  --bigquery-table=PROJECT_ID:analytics.events \
  --use-topic-schema \
  --write-metadata

A Cloud Storage subscription batches messages to objects, flushing on a size or duration threshold:

gcloud pubsub subscriptions create events-to-gcs \
  --topic=events \
  --cloud-storage-bucket=my-events-bucket \
  --cloud-storage-file-prefix=events/ \
  --cloud-storage-max-duration=300s \
  --cloud-storage-max-bytes=100MB

Note: exactly-once and ordering are pull-tier guarantees. Push and the managed export subscriptions are at-least-once, so the destination must tolerate duplicates (dedupe in BigQuery on a message key; idempotent object naming in GCS).

8. Monitoring backlog, oldest unacked age, and expired acks

You operate Pub/Sub by watching a few subscription metrics in Cloud Monitoring. The three that matter most:

• subscription/num_undelivered_messages — backlog size. A rising, non-draining backlog means consumers can’t keep up: scale out or speed up the handler.
• subscription/oldest_unacked_message_age — age of the oldest unacked message, in seconds. This is your true freshness SLO. If it climbs toward your message-retention-duration, you are about to lose data.
• subscription/expired_ack_deadlines_count — acks that missed their deadline. Sustained nonzero values mean handlers are too slow for the ack deadline, or flow control is letting too many messages outstanding. This directly causes redelivery (and, under ordering/exactly-once, cascades).

Watch dead-lettered volume via subscription/dead_letter_message_count, and on the publisher side keep an eye on topic/send_request_count error ratios.

A practical alerting policy in MQL — page when the oldest unacked message exceeds 10 minutes:

fetch pubsub_subscription
| metric 'pubsub.googleapis.com/subscription/oldest_unacked_message_age'
| filter (resource.subscription_id == 'orders-eo-sub')
| group_by 1m, [value_age_max: max(value.oldest_unacked_message_age)]
| condition value_age_max > 600 's'

Quick CLI sanity check on backlog and DLT depth during an incident:

gcloud pubsub subscriptions describe orders-eo-sub \
  --format="yaml(ackDeadlineSeconds, retryPolicy, deadLetterPolicy)"

Enterprise scenario

A payments platform team ran an orders topic feeding a ledger-posting service. Their first design used a single global ordering key to “guarantee” strict global order. In load testing they hit a wall at roughly 1 MB/s of publish throughput and could not push past it no matter how many subscriber instances they added. The cause was the per-ordering-key throughput cap: one key serializes everything through a single 1 MB/s lane, and subscriber scale-out cannot help a single-key bottleneck.

The constraint was real: posting two events for the same account out of order would corrupt a balance. But events for different accounts had no ordering relationship. The fix was to make the ordering key the account ID instead of a constant, turning one hot lane into thousands of independent ones, each with its own 1 MB/s budget. They paired it with a dead-letter topic (--max-delivery-attempts=10) so a single malformed event for one account couldn’t permanently stall that account’s lane, and added a resume_publish call on the publisher’s error path so a transient publish failure didn’t wedge a key. Aggregate throughput scaled with subscriber count, per-account ordering held, and the redelivery-cascade blast radius shrank from “the whole stream” to “one account.”

gcloud pubsub subscriptions create ledger-postings \
  --topic=orders \
  --enable-message-ordering \
  --enable-exactly-once-delivery \
  --ack-deadline=60 \
  --dead-letter-topic=orders-dlq \
  --max-delivery-attempts=10 \
  --min-retry-delay=10s \
  --max-retry-delay=300s

The lesson generalizes: ordering keys are a partitioning decision, not a correctness toggle. Choose the key at the granularity where order actually matters, and no coarser.

Going deeper

What exactly-once actually costs, and why

Exactly-once is not free magic — it is bookkeeping. To promise “never redeliver after ack,” the service has to durably persist per-message ack state and dedupe redelivery attempts within the region, and it has to make each ack a synchronous, confirmable operation rather than fire-and-forget. That is the whole reason ack_with_response() returns a future: the client is waiting for the service to durably record the ack. The consequences are concrete — higher per-message latency, lower peak throughput, and higher cost than a plain at-least-once subscription. This is why the guidance is always “idempotency first”: if your sink already tolerates duplicates, you are paying for a guarantee you don’t need.

The ack ID lifecycle is the subtle part. An ack ID identifies one delivery of a message, not the message itself. Every time the client extends the lease (a modifyAckDeadline / modAck call, which the library does automatically), the previous ack ID is invalidated and a new one issued. Under exactly-once the service enforces this strictly: acking with a stale ack ID fails rather than silently succeeding. The failure modes your ack future can surface:

• Transient (transient errors from the ack RPC) — the library retries automatically; you usually don’t see these.
• Permanent (INVALID_ARGUMENT on a stale/expired ack ID, PERMISSION_DENIED, FAILED_PRECONDITION) — the ack did not take; the message is or will be redelivered. Your code must treat “ack future raised” as “not done yet,” never as “done.”

The trap that follows: if you write a “processed” marker to your dedupe store before awaiting the ack future and the ack then fails, the message comes back but your store says “already handled,” so you silently drop it. Order of operations is: do the side effect → await the ack → then commit the processed marker (or make the marker itself idempotent and keyed so a later real duplicate is caught).

Region scope and message storage

Exactly-once holding “within a single region” is not arbitrary — Pub/Sub replicates a subscription’s backlog and ack state across zones inside a region, and the dedupe window is regional. Subscribers connecting through the global endpoint can be routed to different regional frontends, and the guarantee does not span them; that is why off-GCP subscribers must use a locational endpoint (REGION-pubsub.googleapis.com:443). This dovetails with message storage policies on the topic (--message-storage-policy-allowed-regions), which you set for data residency — pin storage to the region your subscribers connect to and you get residency and a clean exactly-once boundary at once.

Ordering internals and the cascade

“Publish order” means the order in which the service accepted publishes for a key in a region — which is why the publisher blocks on future.result() before sending the next message for the key, and why a failed publish pauses the key (sending the next one would create an ambiguous order). On the delivery side, preserving order after a failure is what forces the redelivery cascade: if message 5 for a key must be retried, messages 6, 7, 8… cannot be delivered ahead of it, and already-delivered-but-unacked ones may be re-sent to keep the sequence intact. Ordering therefore couples the fate of every message under a key. Two design rules fall out: keep keys fine-grained so a cascade is small, and keep per-message handlers fast so one slow message doesn’t stall its lane.

An idempotency pattern that actually works

Because exactly-once never covers publisher-side duplicates, production systems still carry a dedupe key. Have the publisher stamp a stable business idempotency key as a message attribute (an order ID, a payment intent ID — not the Pub/Sub message_id, which differs across publish retries):

publisher.publish(
    topic_path,
    b'{"amount":500}',
    ordering_key="acct-42",
    idempotency_key="pay-7f3a-2026-0001",  # stable across publish retries
)

The consumer does a conditional write keyed on idempotency_key — INSERT ... ON CONFLICT DO NOTHING in Postgres/Cloud SQL, a create-if-absent in Firestore/Spanner, or SETNX with a TTL in Memorystore. Set the TTL longer than your message retention so a late redelivery still finds the marker. This one pattern survives ack-deadline duplicates, publisher retries, and fan-out to a second subscription — which is why it is the durable foundation and exactly-once is the optional assist.

The same subscription as Infrastructure-as-Code

Everything above is one Terraform resource plus the two IAM bindings. Encoding the reliable subscription in IaC is how you stop the “someone forgot the service-agent grant” incident from recurring:

resource "google_pubsub_topic" "orders" { name = "orders" }
resource "google_pubsub_topic" "dlq"    { name = "orders-dlq" }

resource "google_pubsub_subscription" "ledger" {
  name  = "ledger-postings"
  topic = google_pubsub_topic.orders.id

  ack_deadline_seconds         = 60
  enable_exactly_once_delivery = true
  enable_message_ordering      = true
  message_retention_duration   = "604800s" # 7 days

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "300s"
  }

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.dlq.id
    max_delivery_attempts = 10
  }
}

# The Pub/Sub service agent needs both grants or dead-lettering silently fails.
data "google_project" "current" {}

locals {
  pubsub_sa = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

resource "google_pubsub_topic_iam_member" "dlq_publisher" {
  topic  = google_pubsub_topic.dlq.id
  role   = "roles/pubsub.publisher"
  member = local.pubsub_sa
}

resource "google_pubsub_subscription_iam_member" "src_subscriber" {
  subscription = google_pubsub_subscription.ledger.id
  role         = "roles/pubsub.subscriber"
  member       = local.pubsub_sa
}

Limits and quotas worth memorizing

Two of these interact with cost: retention is billed storage, so a 31-day window on a fat backlog is a real line item; and exactly-once/ordering both lower effective throughput, so a subscription that turns both on needs more subscriber instances (and more ack/modAck RPC volume) to hit the same drain rate as a plain one.

Practice challenges

Work these in a scratch project. They escalate from a single flag to a full diagnostic. Replace PROJECT_ID / PROJECT_NUMBER.

1. (Beginner) Create an exactly-once pull subscription. On an existing orders topic, create a subscription eo-lab with exactly-once delivery, a 60-second ack deadline, and 3-day retention.

<details> <summary>Solution</summary>

gcloud pubsub subscriptions create eo-lab \
  --topic=orders \
  --enable-exactly-once-delivery \
  --ack-deadline=60 \
  --message-retention-duration=3d

Why: exactly-once is a subscription-creation property (pull-only); the 60 s deadline is the recommended default because it absorbs transient blips that would otherwise cause redelivery. </details>

2. (Beginner→Intermediate) Add, then remove, an exponential backoff retry policy. Give eo-lab a backoff from 10 s to 300 s, confirm it, then revert to immediate redelivery.

<details> <summary>Solution</summary>

gcloud pubsub subscriptions update eo-lab \
  --min-retry-delay=10s --max-retry-delay=300s

gcloud pubsub subscriptions describe eo-lab --format="yaml(retryPolicy)"

gcloud pubsub subscriptions update eo-lab --clear-retry-policy

Why: without a retry policy a nacked or deadline-expired message redelivers immediately and hot-loops the downstream; backoff spaces the retries between the two bounds (each 10–600 s). </details>

3. (Intermediate) Wire a dead-letter topic with the required IAM. Create orders-dlq + a holding subscription, attach it to eo-lab with max 10 attempts, and grant the Pub/Sub service agent the two roles it needs.

<details> <summary>Solution</summary>

gcloud pubsub topics create orders-dlq
gcloud pubsub subscriptions create orders-dlq-sub --topic=orders-dlq \
  --message-retention-duration=7d

gcloud pubsub subscriptions update eo-lab \
  --dead-letter-topic=orders-dlq --max-delivery-attempts=10

PUBSUB_SA="service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud pubsub topics add-iam-policy-binding orders-dlq \
  --member="serviceAccount:${PUBSUB_SA}" --role="roles/pubsub.publisher"
gcloud pubsub subscriptions add-iam-policy-binding eo-lab \
  --member="serviceAccount:${PUBSUB_SA}" --role="roles/pubsub.subscriber"

Why: forwarding and acking the dead-lettered message are done by the service agent, not you — miss either grant and the policy silently never forwards, and poison messages just keep retrying. </details>

4. (Intermediate) Publish ordered messages and survive a publish failure. On an ordering-enabled publisher, publish two events for acct-42 in order, and on the error path resume the key so it isn’t left wedged.

<details> <summary>Solution</summary>

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient(
    publisher_options=pubsub_v1.types.PublisherOptions(enable_message_ordering=True)
)
topic_path = publisher.topic_path("PROJECT_ID", "accounts")

for body in (b'{"seq":1}', b'{"seq":2}'):
    future = publisher.publish(topic_path, body, ordering_key="acct-42")
    try:
        future.result()  # block so order is preserved
    except Exception:
        publisher.resume_publish(topic_path, "acct-42")  # unwedge the key
        raise

Why: after any publish failure for a key, the client rejects all further publishes for that key until resume_publish is called — forgetting it silently stalls that account’s entire lane. </details>

5. (Advanced) Write a correct exactly-once ack loop with flow control. Consume eo-lab, only mark a message processed after the ack future resolves, and cap the client at 200 outstanding messages / 200 MiB.

<details> <summary>Solution</summary>

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
sub_path = subscriber.subscription_path("PROJECT_ID", "eo-lab")

def callback(message):
    try:
        do_side_effect(message.data)          # 1. act
    except Exception:
        message.nack_with_response().result()
        return
    try:
        message.ack_with_response().result()  # 2. confirm the ack landed
    except Exception:
        return                                # ack failed → will redeliver; do NOT commit
    commit_processed(message.message_id)      # 3. only now record "done"

flow = pubsub_v1.types.FlowControl(
    max_messages=200, max_bytes=200 * 1024 * 1024, max_lease_duration=600
)
future = subscriber.subscribe(sub_path, callback=callback, flow_control=flow)
future.result()

Why: under exactly-once the ack is confirmable — committing “processed” before the ack resolves means a failed ack redelivers the message but your store drops it as a duplicate. Flow control keeps the client from over-pulling and blowing ack deadlines. </details>

6. (Advanced) Diagnose a redelivery storm. An exactly-once, ordering-enabled subscription shows steady backlog but a climbing expired_ack_deadlines_count and rising duplicate processing. Name the root cause and the two knobs that fix it.

<details> <summary>Solution</summary>

Root cause: the client holds more outstanding messages than it can process within the ack deadline. Leases expire → Pub/Sub redelivers → because ordering is on, redelivery cascades to later messages of the same key → duplicate work and more expiries, a self-reinforcing storm.

Fix (two knobs):

1. Lower max_messages (and/or max_bytes) in FlowControl so the client only leases what it can finish inside the deadline.
2. Raise the ack deadline and max_lease_duration if handlers legitimately run long, so leases aren’t lost mid-processing.

flow = pubsub_v1.types.FlowControl(max_messages=50, max_bytes=100*1024*1024,
                                   max_lease_duration=600)
# and: gcloud pubsub subscriptions update eo-lab --ack-deadline=120

Why: expired ack deadlines are the direct cause of redelivery; under ordering they cascade, so you attack them from both sides — pull less at once, and give each message more time. </details>

Common beginner mistakes

• “Exactly-once means each event is processed once, end to end.” No — it means no redelivery after a successful ack, within one subscription. Publisher retries (distinct message IDs) and a second subscription on the same topic still produce duplicates. Right model: idempotency is the end-to-end guarantee; exactly-once is a local assist.
• “Exactly-once also gives me ordering.” No — they are orthogonal. Exactly-once controls how many times; ordering controls in what sequence. Turn ordering on separately with --enable-message-ordering (and ordering keys on the publisher).
• “Turn on exactly-once everywhere to be safe.” It lowers throughput, raises latency, and costs more. Right model: make consumers idempotent first; reserve exactly-once for the few subscriptions where dedupe is genuinely impossible.
• “Ordering keys give me global order.” They give order per key, per region only. Messages with an empty ordering key are unordered. A single global/constant key does serialize everything — straight into the 1 MB/s per-key wall.
• “A dead-letter policy works as soon as I set --max-delivery-attempts.” Not without the IAM. The Pub/Sub service agent needs roles/pubsub.publisher on the DLT and roles/pubsub.subscriber on the source subscription, or forwarding silently fails and messages just keep retrying.
• “Flow control is a subscription setting I configure with gcloud.” No — it lives in the subscriber client (FlowControl(...)). There is no subscription field for it; two clients on the same subscription can have different limits.
• “Push subscriptions can be exactly-once.” No — exactly-once is pull-only, because a push receiver cannot confirm to the service that its ack was recorded.
• “I’ll just point a worker at the dead-letter subscription in a loop to retry.” That recreates the hot-loop you built the DLT to escape. Right model: the DLQ is a quarantine — alert, triage, fix the root cause, then replay to the original topic.
• “A bigger ack deadline is always safer.” Only up to a point. Too long and a genuinely dead handler pins its message (and, under ordering, its whole key) for that entire window before redelivery. Balance it against handler p99, and pair it with flow control.

Glossary

• Ack / nack — Acknowledge (I have processed this, drop it) or negative-acknowledge (I failed, redeliver per policy) a message.
• Ack deadline — The lease window (10–600 s) a subscriber has to ack after delivery. Miss it and Pub/Sub redelivers.
• Ack ID — An identifier for one delivery of a message. Invalidated when the lease is extended; under exactly-once a stale ack ID fails.
• At-least-once — The default guarantee: every message is delivered one or more times. Duplicates are legal; design idempotent consumers.
• Exactly-once delivery — Opt-in, pull-only, region-scoped guarantee that a message is not redelivered once acked, and not delivered to two subscribers at once. Does not cover publisher-side duplicates.
• Idempotent consumer — A handler for which processing the same event twice has the same effect as once (via a stable key, conditional write, or upsert).
• Ordering key — A string stamped on a message; messages sharing a key, published in one region, are delivered in publish order. Empty key = unordered.
• Redelivery cascade — Under ordering, retrying one message forces later (and some already-delivered) messages of the same key to be re-sent to preserve sequence.
• resume_publish — Publisher call that un-pauses an ordering key after a publish failure; without it the key is stuck.
• Dead-letter topic (DLT / DLQ) — A separate topic that poison messages are forwarded to after max-delivery-attempts, so the main subscription keeps flowing.
• Max delivery attempts — 5–100 (default 5); the approximate attempt count after which a message is dead-lettered. Best-effort, not exact.
• Retry policy / backoff — Exponential min/max delay (10–600 s each) between redeliveries, instead of the default immediate retry.
• Flow control — Client-side limits (max_messages, max_bytes, max_lease_duration) on how many unacked messages one subscriber holds.
• Outstanding message — A delivered-but-not-yet-acked message that counts against flow-control limits and holds a lease.
• Lease / modAck — The claim a subscriber holds on a message until its ack deadline; the client auto-extends it (a modifyAckDeadline call) up to max_lease_duration.
• StreamingPull — The default high-throughput bidirectional pull mechanism that pauses cleanly under backpressure.
• Unary Pull — Request/response pull (one batch per call); simpler cadence control, higher latency at volume.
• Push subscription — Pub/Sub POSTs each message to your HTTPS endpoint; at-least-once, server-controlled rate, no exactly-once.
• BigQuery / Cloud Storage subscription — Managed export subscriptions that write straight to a table or bucket with no subscriber code; at-least-once.
• Service agent — The Google-managed account service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com that performs dead-letter forwarding; needs explicit IAM grants.
• oldest_unacked_message_age — Metric: seconds since the oldest unacked message arrived; your true backlog-freshness SLO.
• expired_ack_deadlines_count — Metric: acks that missed their deadline; sustained values mean slow handlers or too-loose flow control, and cause redelivery.
• Seek / snapshot — Replay mechanisms: seek moves a subscription’s ack cursor to a timestamp or snapshot; a snapshot captures a subscription’s ack state for later replay.
• Locational endpoint — A regional API endpoint (REGION-pubsub.googleapis.com:443) that pins connections to one region — required for exactly-once from outside GCP.

Verify

Confirm the configuration and behavior end to end:

# Subscription has exactly-once, ordering, retry, and dead-letter set
gcloud pubsub subscriptions describe ledger-postings \
  --format="yaml(enableExactlyOnceDelivery, enableMessageOrdering, retryPolicy, deadLetterPolicy)"

# Service agent has both required IAM grants
gcloud pubsub topics get-iam-policy orders-dlq \
  --format="table(bindings.role, bindings.members)"
gcloud pubsub subscriptions get-iam-policy ledger-postings \
  --format="table(bindings.role, bindings.members)"

# Publish a couple of ordered messages and confirm in-order receipt
gcloud pubsub topics publish orders --message='{"seq":1}' --ordering-key=acct-42
gcloud pubsub topics publish orders --message='{"seq":2}' --ordering-key=acct-42

# Inspect backlog freshness during/after a load test
gcloud pubsub subscriptions describe ledger-postings \
  --format="value(name)"

Then check Monitoring: oldest_unacked_message_age should stay low under steady load, expired_ack_deadlines_count should be near zero, and num_undelivered_messages should drain rather than grow. Force a handler error path to confirm messages land in orders-dlq after the configured attempts, and confirm delivery_attempt is present on the dead-lettered message.

Checklist

• Consumers are idempotent; exactly-once is reserved for subscriptions where dedupe is impossible.
• Exactly-once subscriptions are pull-only and subscribers pin to a single region (locational endpoint off-GCP).
• Ack code waits on the ack future and does not mark “processed” until the ack is confirmed.
• Ordering keys are chosen for the granularity where order matters and spread load (no global hot key).
• Publisher handles publish failures with resume_publish so an ordering key cannot wedge.
• Retry policy uses exponential backoff (--min-retry-delay / --max-retry-delay) instead of immediate redelivery.
• Dead-letter topic attached with --max-delivery-attempts, plus a subscription to retain failures.
• Pub/Sub service agent granted roles/pubsub.publisher on the DLT and roles/pubsub.subscriber on the source subscription.
• Subscriber FlowControl (max_messages / max_bytes / max_lease_duration) sized to the container, scaling out horizontally for throughput.
• Alerts on oldest_unacked_message_age, num_undelivered_messages, expired_ack_deadlines_count, and dead-letter volume.
• Reprocessing is a triage-then-replay job, not a tight loop on the DLT.