Event-Driven Architecture on GCP: Pub/Sub, Eventarc, Cloud Tasks and Workflows

Quick take: Google Cloud gives you four asynchronous primitives — Pub/Sub for events, Eventarc for routing Google-generated events, Cloud Tasks for rate-controlled commands, Workflows for orchestrated sagas — plus Cloud Scheduler for time. Most production failures trace back to using one where another belonged, or to skipping the disciplines none of them can skip: idempotency, schema contracts, and observability of flows no stack trace will ever show you.

Every team that moves to Google Cloud eventually draws the same whiteboard picture: boxes for services, arrows labelled “event” between them. Six months later the picture is real and the pain starts — a duplicate payment.captured event double-charges a customer, a poison message wedges an ordered subscription, nobody can say where a stuck refund actually is across five services, and the notification service hammers a legacy SMTP relay into the ground because Pub/Sub fan-out has no per-consumer rate limit. None of these are bugs in Pub/Sub. They are architecture decisions that were never consciously made: event or command? Choreography or orchestration? Who owns the schema, and who owns the dead-letter queue?

This article is the pattern piece that sits above the plumbing. The deep mechanics of topics, subscriptions, ordering keys and exactly-once delivery live in GCP Pub/Sub and Event-Driven Architecture: Decouple and Scale — here we recap them only far enough to build on, then spend our time on the decisions that shape the whole system: events vs commands (and why Cloud Tasks exists at all when Pub/Sub already exists), choreography vs orchestration (and when a saga must move from emergent event chains into an explicit Workflows definition with compensation steps), Eventarc as the router that turns Google Cloud itself into an event producer, Cloud Scheduler as the time-based producer, and the cross-cutting disciplines — idempotency, the transactional outbox, schema management, trace propagation through async hops, backpressure — that decide whether your event mesh is a nervous system or a haunted house.

By the end you will be able to look at any async requirement — “when an invoice lands in the bucket, process it”; “charge the card, reserve the seat, and undo both if issuing the ticket fails”; “send 100,000 emails but never more than 50 per second”; “re-run reconciliation at 02:00 IST” — and name the right primitive, delivery semantics, failure path and observability hooks, with the gcloud, YAML and Terraform to build it. This assumes you have shipped at least one Pub/Sub consumer and are now designing the system around it.

What problem this solves

Synchronous architectures fail loudly and locally: a call times out, a stack trace points at the line. Event-driven architectures fail quietly and globally: everything returns 200, and three hours later finance asks why 4% of orders have no invoice. This article closes the gap between “we use Pub/Sub” and “we run a correct, observable, cost-sane event-driven system.” Concretely, these are the production failures you prevent by making its decisions deliberately:

If you have hit three or more rows of that table, you are the audience. The rest of the article works through the toolbox in order: the concepts, the backbone (Pub/Sub), the router (Eventarc), the command channel (Cloud Tasks + Scheduler), the orchestrator (Workflows), then the cross-cutting disciplines.

Learning objectives

By the end of this article you can:

• Classify any async interaction as an event or a command, and route it to Pub/Sub or Cloud Tasks with a defensible reason.
• Choose choreography or orchestration per flow, arguing coupling, visibility and failure handling rather than fashion.
• State the architecture-shaping Pub/Sub settings — push vs pull, ordering keys, exactly-once delivery, dead-letter topics, retry policy — with each one’s limit and cost.
• Build Eventarc triggers from all three source classes (direct, Cloud Audit Logs, custom Pub/Sub), explain the CloudEvents envelope, and wire destinations to Cloud Run, GKE, Workflows and internal HTTP endpoints.
• Write a Workflows saga with retries, except blocks and compensation steps, call Google APIs through connectors, and defend Workflows vs Cloud Composer.
• Implement idempotent consumers and a transactional outbox, and explain why exactly-once delivery alone does not make a flow exactly-once.
• Manage message schemas (Avro/Protobuf revisions, compatibility) so producers evolve without breaking consumers.
• Instrument an async flow end to end: trace context through Pub/Sub, the four metrics that page you, and deliberate backpressure at every layer.

Prerequisites & where this fits

You should be comfortable with core GCP mechanics: projects and IAM (members, roles, bindings), service accounts and OIDC tokens (service accounts & least privilege), deploying a container to Cloud Run (Cloud Run explained), and the Pub/Sub fundamentals from the Pub/Sub deep dive. The gcloud cheat sheet covers the CLI conventions used throughout. All examples use asia-south1 (Mumbai); substitute your region.

The division of labour, one line each: Pub/Sub is the event backbone (fan-out, backlog, DLQ); Eventarc routes Google Cloud and custom events to your services as CloudEvents; Cloud Tasks carries rate-controlled commands; Cloud Scheduler produces time; Workflows orchestrates sagas; Cloud Run hosts most handlers; BigQuery is the analytical sink; and Cloud Monitoring & Trace watch the whole mesh. If you come from AWS, map mentally: Pub/Sub ≈ SNS+SQS fused, Eventarc ≈ EventBridge, Cloud Tasks ≈ SQS with a managed dispatcher, Workflows ≈ Step Functions, Scheduler ≈ EventBridge Scheduler — the same patterns as AWS Lambda event-driven patterns, with different physics where this article flags them.

Core concepts

Four ideas carry everything else. Get these straight and every later decision becomes mechanical.

A message is not an event, and an event is not a command. A message is the transport unit — bytes plus attributes moving through a broker. An event is a statement of fact about the past: order.placed, object.finalized. The producer does not know or care who reacts; zero, one or ten consumers is a choice made after publish, by adding subscriptions. A command is an instruction aimed at a specific handler: send-this-email. It has exactly one intended executor, the sender cares that it happens, and duplicating it is harmful. GCP embodies the split in hardware: events ride Pub/Sub (fan-out, no per-consumer rate control), commands ride Cloud Tasks (one target, explicit rate and concurrency caps, scheduled delivery). Teams that push commands through Pub/Sub end up reimplementing Cloud Tasks badly in application code.

Delivery semantics are a contract you pay for. Every async system chooses between at-most-once (fast, lossy — never acceptable for business events), at-least-once (the Pub/Sub default: nothing is lost, retries create duplicates), and exactly-once (Pub/Sub can enforce non-redelivery of acked messages on regional pull subscriptions — but it cannot stop your publisher retrying, or your handler failing after the side effect). The consequence: end-to-end exactly-once is an application property, built from at-least-once delivery plus idempotent handlers plus, where the DB and the event must agree, a transactional outbox.

Choreography and orchestration are visibility trade-offs, not religions. In choreography, services react to each other’s events; the flow is emergent, coupling minimal, adding a consumer free — but nobody owns the end-to-end outcome, and partial failure leaves silent debris. In orchestration, a central definition (a Workflows program) calls each step, holds state, retries, and runs compensation on failure — at the cost of a component that knows every participant. Mature systems use both: choreography between bounded contexts (“order placed” is everyone’s business), orchestration inside a money-critical flow (reserve → charge → issue is one owner’s business).

The envelope matters as much as the payload. GCP’s routing layer, Eventarc, standardises every event — a Storage upload, an audit-log entry, your own Pub/Sub message — into a CloudEvents 1.0 envelope: id, source, type, subject, time as ce-* HTTP headers, payload in the body. One Cloud Run service can consume storage events today and Firebase events tomorrow without new parsing code, and the envelope is where trace context and idempotency keys ride.

The vocabulary in one table

And the delivery-semantics contract, since every later section leans on it:

Events vs commands, choreography vs orchestration

These are the two axes of every event-driven design review I run. Every arrow on your whiteboard should be classifiable in ten seconds on both axes; arrows that resist classification are where incidents live.

Axis 1 — event or command?

The classic smell: a topic named send-email-requests — a command wearing an event costume. No second consumer will ever subscribe, the SMTP relay needs a rate cap, and queued sends should be cancellable on unsubscribe: all three are Cloud Tasks features and Pub/Sub non-features.

Axis 2 — choreography or orchestration?

The decision table

Rule of thumb I hold teams to: facts fan out, money gets orchestrated, fragile things get queued, time goes through Scheduler. Print it on the whiteboard; it settles 80% of design arguments.

Pub/Sub: the backbone, recapped for architects

The Pub/Sub deep dive covers mechanics message by message; here is the architect’s working summary — the settings that change your system design, with their limits and costs. If any row surprises you, read the deep dive before building.

First, delivery models, because the choice cascades into scaling, ack deadlines and observability:

Push nuance that bites architects: a push endpoint acks with 102, 200, 201, 202 or 204; anything else is a nack that schedules a retry. Pub/Sub adapts its push rate to your endpoint’s success rate, so a failing consumer automatically gets fewer requests — good for the consumer, invisible-backlog-shaped for you. Alert on backlog, not on consumer error rate alone.

The design-shaping settings in one matrix:

And the minimal production-grade pair, CLI and Terraform — note the DLQ, backoff and non-expiring subscription, which I consider the floor for anything carrying business events:

gcloud pubsub topics create order-events --message-retention-duration=7d
gcloud pubsub topics create order-events-dlq

gcloud pubsub subscriptions create order-events-notify \
  --topic=order-events \
  --push-endpoint="https://notify-svc-xyz-el.a.run.app/pubsub" \
  --push-auth-service-account=push-invoker@$PROJECT_ID.iam.gserviceaccount.com \
  --ack-deadline=60 \
  --min-retry-delay=10s --max-retry-delay=600s \
  --dead-letter-topic=order-events-dlq --max-delivery-attempts=10 \
  --expiration-period=never

resource "google_pubsub_subscription" "notify" {
  name  = "order-events-notify"
  topic = google_pubsub_topic.order_events.id

  ack_deadline_seconds = 60

  push_config {
    push_endpoint = google_cloud_run_v2_service.notify.uri
    oidc_token {
      service_account_email = google_service_account.push_invoker.email
    }
  }
  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "600s"
  }
  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.order_events_dlq.id
    max_delivery_attempts = 10
  }
  expiration_policy { ttl = "" } # never expire
}

Consequences to carry forward: fan-out is free at the producer; each subscription is an independent failure domain (own backlog, retry, DLQ); there is no per-subscription rate limit (that is Cloud Tasks’ job); and at-least-once is the baseline contract everything downstream must survive.

Eventarc: the router that makes GCP itself a producer

Without Eventarc, reacting to platform activity means polling or hand-wiring service-specific notification channels (GCS Pub/Sub notifications, BQ log sinks), each with a different payload shape. Eventarc replaces that zoo with one model: a trigger matches events from a provider using filters and delivers them to a destination as a CloudEvents 1.0 HTTP POST, over a Pub/Sub transport it manages for you. Delivery is at-least-once, retried with backoff for up to 24 hours via the underlying subscription.

Source classes — where events come from

The direct-vs-audit-log decision deserves its own comparison because teams habitually reach for audit logs when a direct event exists (slower, noisier) or assume a direct event exists when it doesn’t:

The CloudEvents envelope

Every Eventarc delivery is an HTTP POST in binary content mode: attributes as ce-* headers, payload as the body. This is what your Cloud Run service actually receives for a GCS upload:

POST /events HTTP/1.1
Host: invoice-processor-xyz-el.a.run.app
Authorization: Bearer <OIDC token for the trigger's service account>
Content-Type: application/json
ce-id: 8632110139364665
ce-source: //storage.googleapis.com/projects/_/buckets/invoices-raw
ce-specversion: 1.0
ce-type: google.cloud.storage.object.v1.finalized
ce-subject: objects/2026/07/inv-88123.pdf
ce-time: 2026-07-07T05:41:22.719Z

{ "bucket": "invoices-raw", "name": "2026/07/inv-88123.pdf", "size": "482113", ... }

The attributes you will actually use:

Destinations and the IAM triangle

Three IAM grants cover 90% of Eventarc setups — the triangle: the trigger’s service account needs roles/eventarc.eventReceiver plus invoker rights on the destination; and for GCS direct events, the Cloud Storage service agent needs roles/pubsub.publisher on the project. Miss the third and trigger creation fails with a quotable error; miss the second and events flow into a 403 black hole — the permission-denied decision tree applies verbatim.

# The GCS service agent must be able to publish (once per project)
GCS_SA=$(gcloud storage service-agent --project=$PROJECT_ID)
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:${GCS_SA}" --role=roles/pubsub.publisher

# Trigger: invoice PDFs landing in a bucket → Cloud Run
gcloud eventarc triggers create invoice-uploaded \
  --location=asia-south1 \
  --destination-run-service=invoice-processor \
  --destination-run-region=asia-south1 \
  --destination-run-path=/events \
  --event-filters="type=google.cloud.storage.object.v1.finalized" \
  --event-filters="bucket=invoices-raw" \
  --service-account=eventarc-sa@$PROJECT_ID.iam.gserviceaccount.com

resource "google_eventarc_trigger" "invoice_uploaded" {
  name     = "invoice-uploaded"
  location = "asia-south1"

  matching_criteria {
    attribute = "type"
    value     = "google.cloud.storage.object.v1.finalized"
  }
  matching_criteria {
    attribute = "bucket"
    value     = google_storage_bucket.invoices_raw.name
  }
  destination {
    cloud_run_service {
      service = google_cloud_run_v2_service.invoice_processor.name
      region  = "asia-south1"
      path    = "/events"
    }
  }
  service_account = google_service_account.eventarc.email
}

Trigger options that shape behaviour

Eventarc Standard vs Advanced: Standard is trigger-per-route, project-scoped, free apart from the Pub/Sub transport. Advanced adds a central message bus, enrollments that filter with CEL, and pipelines that transform payloads and deliver across projects — an event mesh rather than point wiring, with per-event pricing. Single team/project → Standard; a platform team offering events-as-a-service across many teams → evaluate Advanced.

Cloud Tasks vs Pub/Sub: commands need a different tool

This is the most common architecture-review correction I make on GCP, so it gets the full matrix. Both services move messages and retry until acked — the resemblance ends there.

The queue configuration is the whole point of Cloud Tasks, so enumerate it properly:

gcloud tasks queues create email-commands \
  --location=asia-south1 \
  --max-dispatches-per-second=50 \
  --max-concurrent-dispatches=100 \
  --max-attempts=8 --min-backoff=2s --max-backoff=300s --max-doublings=5

# Enqueue: send one email, no earlier than 09:00 IST, dedup by task name
gcloud tasks create-http-task email-o-88123-confirmation \
  --queue=email-commands --location=asia-south1 \
  --url="https://email-svc-xyz-el.a.run.app/send" \
  --method=POST \
  --header="Content-Type: application/json" \
  --body-content='{"orderId":"o-88123","template":"confirmation"}' \
  --schedule-time="2026-07-08T03:30:00Z" \
  --oidc-service-account-email=tasks-invoker@$PROJECT_ID.iam.gserviceaccount.com

resource "google_cloud_tasks_queue" "email_commands" {
  name     = "email-commands"
  location = "asia-south1"

  rate_limits {
    max_dispatches_per_second = 50
    max_concurrent_dispatches = 100
  }
  retry_config {
    max_attempts  = 8
    min_backoff   = "2s"
    max_backoff   = "300s"
    max_doublings = 5
  }
}

Two design notes engineers miss. First, Cloud Tasks has no dead-letter queue: when maxAttempts exhausts, the task disappears. For work you cannot lose, have the handler detect the final attempt (the X-CloudTasks-TaskRetryCount header) and persist the failure — a Firestore row or a “failed-commands” Pub/Sub topic you triage like a DLQ. Second, scheduled tasks are a state-machine primitive: “hold this seat for 10 minutes, release unless confirmed” is one named task with scheduleTime = now+10m, deleted on confirmation — no cron sweep, no polling, no race.

Cloud Scheduler: the time producer

Cloud Scheduler is deliberately tiny: a managed unix-cron firing at-least-once per tick into one of three target types. Its architectural role is to be the only source of time in the system — replacing the “leader-elected cron container” every team eventually builds and breaks.

gcloud scheduler jobs create pubsub nightly-reconciliation \
  --location=asia-south1 \
  --schedule="0 2 * * *" --time-zone="Asia/Kolkata" \
  --topic=reconciliation-runs \
  --message-body='{"window":"daily"}'

Pattern guidance: Scheduler → Pub/Sub → workers when the tick fans out (per-tenant reconciliation: one tick, N tenant messages); Scheduler → Workflows execution when the tick starts an orchestrated job; never Scheduler → long synchronous handler (the 30-minute attempt deadline becomes your job’s invisible timeout). Ticks are at-least-once, so the job itself must be idempotent — the recurring theme.

Workflows: orchestration, sagas and compensation

Workflows is GCP’s serverless orchestrator: you declare steps in YAML, Google runs the state machine — durable across failures, scale-to-zero, priced per step (~$0.01 per 1,000 internal steps, 5,000/month free; external calls ~$0.025 per 1,000 after 2,000 free), with executions allowed to run up to one year. That last number is the quiet superpower: a workflow can sys.sleep for days or park on a callback awaiting human approval with no infrastructure existing in the meantime.

The language surface you actually use

Error handling and the saga pattern

Error handling composes from four pieces. Predefined retry policies — retry: ${http.default_retry} (retries 429/502/503/504 and connection errors; safe for idempotent calls) and retry: ${http.default_retry_non_idempotent} (when a POST must not re-fire) — or a custom policy with predicate, max_retries and backoff: {initial_delay, max_delay, multiplier}. Catches use except: as: e, where e is a map carrying code, message, tags (HttpError, TimeoutError, ConnectionError) and the HTTP body/headers where applicable — branch compensation on tags, never on message strings — and end with raise: ${e} so the execution fails honestly.

The saga pattern in Workflows is exactly this: each forward step is a try with a bounded retry; the except ladder runs compensations for every already-completed step in reverse order, then re-raises. Here is a complete, deployable order saga — reserve inventory, charge payment, issue ticket; any failure unwinds cleanly:

main:
  params: [event]
  steps:
    - init:
        assign:
          - order: ${json.decode(base64.decode(event.data.message.data))}
          - orderId: ${order.orderId}
          - project: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
    - reserveInventory:
        try:
          call: http.post
          args:
            url: https://inventory-svc-xyz-el.a.run.app/v1/reserve
            auth: { type: OIDC }
            headers: { X-Idempotency-Key: '${orderId + "-reserve"}' }
            body: { orderId: '${orderId}', items: '${order.items}' }
          result: reservation
        retry:
          predicate: ${http.default_retry_predicate}
          max_retries: 4
          backoff: { initial_delay: 2, max_delay: 30, multiplier: 2 }
    - chargePayment:
        try:
          call: http.post
          args:
            url: https://payments-svc-xyz-el.a.run.app/v1/charge
            auth: { type: OIDC }
            body:
              orderId: ${orderId}
              amountMinor: ${order.amountMinor}
              idempotencyKey: '${orderId + "-charge"}'
          result: charge
        retry: ${http.default_retry_non_idempotent}
        except:
          as: e
          steps:
            - compensateReserve:            # undo step 1
                call: http.post
                args:
                  url: https://inventory-svc-xyz-el.a.run.app/v1/release
                  auth: { type: OIDC }
                  body: { orderId: '${orderId}' }
            - publishFailed:
                call: googleapis.pubsub.v1.projects.topics.publish
                args:
                  topic: ${"projects/" + project + "/topics/order-failed"}
                  body:
                    messages:
                      - data: ${base64.encode(json.encode(e))}
            - failSaga:
                raise: ${e}
    - publishConfirmed:
        call: googleapis.pubsub.v1.projects.topics.publish
        args:
          topic: ${"projects/" + project + "/topics/order-confirmed"}
          body:
            messages:
              - data: ${base64.encode(json.encode(charge.body))}
              - attributes: { orderingKey: '${orderId}' }
    - done:
        return: ${charge.body}

gcloud workflows deploy order-saga --location=asia-south1 \
  --source=order-saga.yaml \
  --service-account=saga-runner@$PROJECT_ID.iam.gserviceaccount.com

# Wire events to the saga: Eventarc trigger with a Workflows destination
gcloud eventarc triggers create order-placed-to-saga \
  --location=asia-south1 \
  --destination-workflow=order-saga --destination-workflow-location=asia-south1 \
  --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \
  --transport-topic=projects/$PROJECT_ID/topics/order-events \
  --service-account=eventarc-sa@$PROJECT_ID.iam.gserviceaccount.com

Map every forward step to its compensation before you write YAML — if a step has no possible undo (an email cannot be unsent), it must move to the end of the saga, after every failure-prone step:

Connectors: the underrated half of Workflows

Connectors (googleapis.pubsub.v1, googleapis.tasks.v2, googleapis.bigquery.v2, googleapis.firestore.v1, and dozens more) are not just typed HTTP wrappers — they authenticate as the workflow’s service account, apply sane retries, and block on long-running operations (a BigQuery job, a Cloud Build) until completion, turning “submit + poll loop” into one step. A nightly “run BQ load job → verify row count → publish dataset.refreshed” pipeline is ~5 steps and costs fractions of a paisa per run.

Workflows vs the other orchestrators

The one-line rule: Workflows orchestrates services; Composer orchestrates data engineering; Dataflow transforms data; none of them replace Pub/Sub — they sit beside it.

Idempotency and the outbox pattern on GCP

Everything upstream — Pub/Sub, Eventarc, Tasks, Scheduler, Workflows retries — is at-least-once. Idempotency is therefore not a nice-to-have; it is the load-bearing wall. Two separate problems hide under the word:

Problem 1 — consuming twice. The same logical event arrives twice (ack-deadline miss, publisher retry, replay); the handler must make the second arrival a no-op. Strategies, weakest to strongest:

The transactional-marker shape on Firestore (the same idea works as a unique-constraint insert on Cloud SQL):

from google.cloud import firestore

db = firestore.Client()

@firestore.transactional
def apply_event(tx: firestore.Transaction, event_key: str, order: dict):
    marker_ref = db.collection("processed_events").document(event_key)
    if marker_ref.get(transaction=tx).exists:
        return "duplicate"                       # second delivery: no-op
    tx.set(db.collection("orders").document(order["orderId"]),
           {"status": "confirmed"}, merge=True)  # business write
    tx.set(marker_ref, {"at": firestore.SERVER_TIMESTAMP})
    return "applied"

# event_key = business key, NOT messageId: survives publisher duplicates
result = apply_event(db.transaction(), f'{order["orderId"]}-confirmed', order)

Give the marker collection a TTL policy (say 14 days — longer than max retention + replay window) so it doesn’t grow forever.

Problem 2 — publishing and committing must agree. The classic dual-write bug: the service commits the order row, then crashes before publishing order.placed (event lost) — or publishes first and the commit fails (phantom event). The fix is the transactional outbox: write the event into an outbox table in the same database transaction as the state change; a relay publishes outbox rows to Pub/Sub and marks them done. The relay is at-least-once — fine, because consumers are idempotent by Problem 1.

CREATE TABLE outbox (
  id            BIGSERIAL PRIMARY KEY,
  aggregate_id  TEXT        NOT NULL,        -- becomes the ordering key
  event_type    TEXT        NOT NULL,        -- e.g. order.placed.v1
  payload       JSONB       NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at  TIMESTAMPTZ                  -- NULL = pending
);

BEGIN;
INSERT INTO orders (id, status, amount_minor) VALUES ('o-88123', 'placed', 249900);
INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES ('o-88123', 'order.placed.v1',
        '{"orderId":"o-88123","amountMinor":249900}');
COMMIT;   -- state and event now succeed or fail together

Relay options on GCP, in ascending order of sophistication:

Schema management: the contract between strangers

An event-driven system is a set of teams communicating through payloads with no compiler between them. Pub/Sub schemas put the compiler back: attach an Avro or Protocol Buffer schema to a topic and Pub/Sub rejects non-conforming publishes with INVALID_ARGUMENT — at the source, protecting every subscriber at once. Validation happens on publish only (delivery is unvalidated bytes), and schemas support up to 20 revisions with commit-time compatibility ranges on the topic (--first-revision-id / --last-revision-id pinning).

gcloud pubsub schemas create order-events-schema --type=avro \
  --definition-file=order-placed-v1.avsc
gcloud pubsub topics create order-events \
  --schema=order-events-schema --message-encoding=json

The rules that keep evolution boring — encode them in CI as a lint job diffing .avsc/.proto before deploy:

1. Version the type, not the topic — order.placed.v1 → .v2 as an attribute/ce-type; a new topic only for a new domain concept.
2. Additive changes only within a major version: new optional fields with defaults. Rename = remove + add = major bump.
3. Consumers ignore unknown fields — mandatory reader behaviour, test it.
4. Producers own the schema; consumers own contract tests pinning the fields they read, run against the producer’s schema in CI.
5. Attributes are never schema-validated — document attribute conventions in the same repo as the schema.

Observability of async flows

A synchronous request carries its own story in the stack. An async flow’s story is scattered across a publisher, a broker, three consumers and a workflow — you must stitch it. Three instruments, in priority order.

1. The metrics that page you — alert on the first four per production subscription (Terraform for one shown, clone for the rest):

resource "google_monitoring_alert_policy" "stale_backlog" {
  display_name = "order-events-notify: oldest unacked > 10 min"
  combiner     = "OR"
  conditions {
    display_name = "oldest_unacked_message_age"
    condition_threshold {
      filter          = "resource.type = \"pubsub_subscription\" AND resource.labels.subscription_id = \"order-events-notify\" AND metric.type = \"pubsub.googleapis.com/subscription/oldest_unacked_message_age\""
      comparison      = "COMPARISON_GT"
      threshold_value = 600
      duration        = "300s"
      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_MAX"
      }
    }
  }
  notification_channels = [google_monitoring_notification_channel.oncall.id]
}

2. Trace propagation through the broker. Cloud Trace stitches HTTP hops automatically; a Pub/Sub hop breaks the chain unless context rides in the message. Modern Pub/Sub client libraries ship OpenTelemetry integration: enabled on publisher and subscriber, they emit publish/deliver spans and propagate W3C context in a googclient_traceparent message attribute automatically. Where you can’t use it (raw REST publishes, legacy libs), the manual pattern is one attribute:

3. Log correlation as the floor. Even with zero tracing, a correlation ID discipline — the business key (orderId) plus the event id logged as structured fields at every hop — turns “where is o-88123’s refund” into one Logs Explorer query: jsonPayload.orderId="o-88123". Put the field names in the schema repo and the code-review checklist; the Cloud Monitoring deep dive covers the dashboards and SLOs to hang off these signals.

Throttling, backpressure and flow control

Every event system eventually meets a consumer that cannot keep up. The design question is where the pressure goes. GCP gives you a knob at every layer — the failure mode is not knowing which one is actually in charge:

The architecture rule: let the backlog be the buffer. Consumers don’t need sizing for peak — they need backlog drain time after peak to meet the SLO (drain ≈ backlog ÷ (consumer throughput − arrival rate)), with the observability section’s alerts telling you when that math breaks. And never simulate throttling with sleeps inside Pub/Sub handlers — route that leg through Cloud Tasks and give the number to the queue.

Architecture at a glance

The diagram reads left to right as one event’s life. Producers: a Cloud Storage upload emits object.finalized, and the checkout API publishes order-placed v1 — both become traffic in the routing column, where the Pub/Sub topic (carrying an orderId ordering key, badge 1) fans out to subscriptions and the Eventarc trigger wraps platform events into CloudEvents pushed with OIDC auth. Consumers — a Cloud Run service and a gen2 function — are deliberately boring: idempotent handlers (badge 2) that dedupe on a business key before side effects, transform, and ack. Anything transactional hands off to the orchestration column: the Workflows saga owns the money path with retries and compensation (badge 3), while Cloud Tasks rate-caps commands aimed at fragile targets (badge 4). Every flow terminates in the sinks column: BigQuery ingests the analytical copy through a native subscription, and the dead-letter topic (badge 5) catches what ten delivery attempts could not process — alarmed, triaged, replayed.

Notice what the picture does not contain: no synchronous call crosses a column boundary, every hop is a place where a backlog can absorb a burst, and both failure paths — compensation and DLQ — are drawn as first-class citizens. The whole doctrine in one image.

Real-world scenario

UtsavSeats, a Mumbai-based event-ticketing platform (14 engineers, ~₹1.8 lakh/month GCP spend), sells concert and festival tickets with brutal traffic shape: near-zero baseline, then 15,000 checkout attempts per minute for the first ten minutes of a big on-sale. Their v1 was pure choreography: checkout published seat.held, payment.captured, ticket.issued events; every service reacted to every other service’s events; email rode the same Pub/Sub topics as everything else.

Two incidents in one on-sale exposed the design. First, the payment provider’s webhook fired twice for ~0.7% of captures (documented at-least-once behaviour); the ticket service had no idempotency, so 214 customers received two tickets for one seat — found via support tickets, fixed by manual audit. Second, the email service — fan-out from the same events — pushed 300 requests/second at a legacy SMTP relay contractually capped at 50/s; the relay IP-banned them mid-on-sale, and transactional receipts queued behind 40,000 marketing notifications with no way to cancel any of it. Meanwhile “where is order X?” meant three engineers grepping three services’ logs in parallel: choreography had no owner for the end-to-end flow.

The redesign took one sprint plus a two-week burn-in, and it is essentially this article. The money path moved to orchestration: an Eventarc trigger starts an order-saga Workflows execution per checkout; reserve-seat → charge → issue-ticket became explicit steps with try/retry, and compensation (release-seat, refund) in except blocks. The duplicate-webhook problem died twice over: the payment callback dedupes on providerEventId with a Firestore transactional marker, and the charge step carries an idempotencyKey the provider honours. Email became commands: a Cloud Tasks queue at maxDispatchesPerSecond=45 (5/s headroom under the contract), OIDC dispatch, marketing sends in a second queue paused during on-sales — a one-command incident lever. Seat holds became scheduled tasks: hold creation enqueues a named task release-{holdId} at now+10m, confirmation deletes it; the cron sweep and its races were deleted. Analytics stayed choreographed — a BigQuery subscription on order-events, zero consumer code.

The next comparable on-sale: checkout p99 held at 180 ms while the saga ran behind it; the SMTP relay saw a flat 45/s for six hours (queue depth peaked at 31,000 — invisible to customers, drained by 02:00); zero duplicate tickets. When one poison message (malformed UTF-8 venue name) failed ten deliveries, it parked in the DLQ, paged via dead_letter_message_count > 0, and was fixed and replayed the same evening — 26 minutes, one engineer, no customer impact. Monthly cost of the new machinery: Workflows ≈ ₹1,400, Cloud Tasks ≈ ₹350, Eventarc/Pub/Sub delta ≈ ₹2,100 — noise against one prevented incident.

Advantages and disadvantages

The disadvantages dominate in small systems with no burst profile, hard synchronous UX requirements, or teams without the maturity for distributed debugging. An honest monolith with a job queue beats a cargo-culted event mesh — the compute decision guide pairs well with that conversation.

Hands-on lab

Build the full toolbox in ~30 minutes in Cloud Shell: a GCS→Eventarc→Cloud Run event path, a Pub/Sub topic with DLQ, a Workflows saga with visible compensation, and a rate-limited Cloud Tasks queue. Everything fits comfortably in the free tier if torn down after.

Step 1 — project, APIs, variables.

export PROJECT_ID=$(gcloud config get-value project)
export REGION=asia-south1
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
gcloud services enable pubsub.googleapis.com eventarc.googleapis.com \
  run.googleapis.com workflows.googleapis.com cloudtasks.googleapis.com \
  cloudscheduler.googleapis.com

Step 2 — a consumer that shows you its events. Deploy the echo-friendly hello container; it logs every request body.

gcloud run deploy event-sink --region=$REGION \
  --image=us-docker.pkg.dev/cloudrun/container/hello --no-allow-unauthenticated

Step 3 — service account + the IAM triangle.

gcloud iam service-accounts create eventarc-lab
gcloud run services add-iam-policy-binding event-sink --region=$REGION \
  --member="serviceAccount:eventarc-lab@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/run.invoker
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:eventarc-lab@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/eventarc.eventReceiver
GCS_SA=$(gcloud storage service-agent --project=$PROJECT_ID)
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:${GCS_SA}" --role=roles/pubsub.publisher

Step 4 — Eventarc trigger on a bucket, then fire it.

gcloud storage buckets create gs://$PROJECT_ID-eda-lab --location=$REGION
gcloud eventarc triggers create lab-object-finalized --location=$REGION \
  --destination-run-service=event-sink --destination-run-region=$REGION \
  --event-filters="type=google.cloud.storage.object.v1.finalized" \
  --event-filters="bucket=$PROJECT_ID-eda-lab" \
  --service-account=eventarc-lab@$PROJECT_ID.iam.gserviceaccount.com
echo "hello eda" > sample.txt && gcloud storage cp sample.txt gs://$PROJECT_ID-eda-lab/
# Wait ~1–2 min for the first delivery, then:
gcloud run services logs read event-sink --region=$REGION --limit=20
# Expected: a POST with ce-type: google.cloud.storage.object.v1.finalized

Step 5 — topic + DLQ + production-grade subscription.

gcloud pubsub topics create lab-orders
gcloud pubsub topics create lab-orders-dlq
gcloud pubsub subscriptions create lab-orders-sub --topic=lab-orders \
  --ack-deadline=30 --min-retry-delay=10s --max-retry-delay=120s \
  --dead-letter-topic=lab-orders-dlq --max-delivery-attempts=5 \
  --expiration-period=never
gcloud pubsub subscriptions create lab-orders-dlq-sub --topic=lab-orders-dlq
# Grant the Pub/Sub service agent its DLQ roles (forwarder identity):
gcloud pubsub topics add-iam-policy-binding lab-orders-dlq \
  --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" \
  --role=roles/pubsub.publisher
gcloud pubsub subscriptions add-iam-policy-binding lab-orders-sub \
  --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" \
  --role=roles/pubsub.subscriber
gcloud pubsub topics publish lab-orders --message='{"orderId":"o-1"}' \
  --attribute=type=order.placed.v1
gcloud pubsub subscriptions pull lab-orders-sub --auto-ack --limit=1

Step 6 — a saga that compensates before your eyes. Save as lab-saga.yaml — amountMinor over 50,000 triggers a simulated payment failure, and you watch the compensation run:

main:
  params: [input]
  steps:
    - init:
        assign:
          - orderId: ${input.orderId}
          - compensated: false
    - reserve:
        assign:
          - reservation: ${"seat-held-" + orderId}
    - charge:
        try:
          steps:
            - maybeFail:
                switch:
                  - condition: ${input.amountMinor > 50000}
                    raise:
                      code: 402
                      message: "card declined"
            - ok:
                assign:
                  - charge: ${"charged-" + orderId}
        except:
          as: e
          steps:
            - compensateReserve:
                assign:
                  - compensated: true
            - failSaga:
                raise: ${e}
    - done:
        return: { orderId: '${orderId}', charge: '${charge}', compensated: '${compensated}' }

gcloud workflows deploy lab-saga --location=$REGION --source=lab-saga.yaml
gcloud workflows run lab-saga --location=$REGION \
  --data='{"orderId":"o-1","amountMinor":19900}'      # succeeds
gcloud workflows run lab-saga --location=$REGION \
  --data='{"orderId":"o-2","amountMinor":99900}'      # fails AFTER compensating
gcloud workflows executions list lab-saga --location=$REGION --limit=2
# Expected: one SUCCEEDED, one FAILED — describe the failed one and find
# "card declined" plus the compensateReserve step in its history.

Step 7 — a throttled command queue.

gcloud tasks queues create lab-commands --location=$REGION \
  --max-dispatches-per-second=1 --max-concurrent-dispatches=1 --max-attempts=3
SINK_URL=$(gcloud run services describe event-sink --region=$REGION --format='value(status.url)')
for i in 1 2 3 4 5; do
  gcloud tasks create-http-task lab-cmd-$i --queue=lab-commands --location=$REGION \
    --url="$SINK_URL/cmd" --method=POST --body-content="{\"n\":$i}" \
    --oidc-service-account-email=eventarc-lab@$PROJECT_ID.iam.gserviceaccount.com
done
gcloud run services logs read event-sink --region=$REGION --limit=20
# Expected: the five POSTs arrive ~1 second apart — the queue, not the sender, sets the pace.

Step 8 — teardown.

gcloud eventarc triggers delete lab-object-finalized --location=$REGION --quiet
gcloud storage rm -r gs://$PROJECT_ID-eda-lab
gcloud run services delete event-sink --region=$REGION --quiet
gcloud workflows delete lab-saga --location=$REGION --quiet
gcloud tasks queues delete lab-commands --location=$REGION --quiet
for s in lab-orders-sub lab-orders-dlq-sub; do gcloud pubsub subscriptions delete $s --quiet; done
for t in lab-orders lab-orders-dlq; do gcloud pubsub topics delete $t --quiet; done
gcloud iam service-accounts delete eventarc-lab@$PROJECT_ID.iam.gserviceaccount.com --quiet

Common mistakes & troubleshooting

The playbook — symptom first, because that is what you have at 02:00:

And the error-string reference for the family — what the API actually returns and what it means in this context:

Best practices

1. Classify every arrow: event → Pub/Sub, command → Cloud Tasks, time → Scheduler, saga → Workflows. Write the classification into the design doc; review it like an API.
2. DLQ everything in production — every subscription gets a dead-letter topic, max-delivery-attempts 5–10, a monitored DLQ subscription, and a written replay runbook. Alert at dead_letter_message_count > 0.
3. Idempotency is a code-review gate: no consumer merges without a dedup strategy named in the PR description (natural, marker, or transactional).
4. Outbox for anything where DB state and events must agree — payments, inventory, entitlements. Dual writes are a standing incident invitation.
5. Version event types (order.placed.v1) and enforce schemas at the topic. Additive evolution within a version; contract tests in consumer CI.
6. Give every fragile downstream its own Tasks queue with a rate 10% under its real capacity — and use pause as an incident lever (practise it).
7. Compensation is designed with the step, not after the outage: each saga step lands with its undo; irreversible steps go last.
8. Propagate trace context in message attributes (enable the client libraries’ OTel support) and log the business key at every hop.
9. Set --expiration-period=never and create subscriptions in Terraform alongside topics — the 31-day idle reaper and the “subscribed too late” gap both vanish under IaC. While you’re there: ack deadlines ≥ 2× handler p99.
10. One event mesh doctrine per org: naming (<domain>.<entity>.<verb>.v<N>), attribute conventions, DLQ policy, replay procedure — documented once, linted in CI.

Security notes

Event meshes multiply the identities that can invoke things, so least privilege does the heavy lifting. Per-leg service accounts: each trigger, push subscription, queue and workflow runs as its own SA with exactly the invoker/publisher roles that leg needs — never the default compute SA (see service accounts & least privilege). Authenticated push everywhere: Cloud Run consumers stay --no-allow-unauthenticated; Pub/Sub push, Eventarc, Tasks and Scheduler all sign OIDC tokens, and Cloud Run’s IAM check does the rest — there is no reason for an unauthenticated event endpoint on GCP.

Data protection: messages are encrypted at rest and in transit by default; use CMEK on topics where compliance demands key custody, and keep PII out of attributes (they surface in logs, filters and metric labels far more readily than payloads). Regulated perimeters: all five services respect VPC Service Controls, and Eventarc’s internal-HTTP destinations reach private receivers without public ingress. Finally, treat replay as privileged: seek/replay re-triggers side effects at scale, so gate pubsub.subscriptions.seek behind a break-glass role.

Cost & sizing

The family is cheap at mid-scale and its costs are load-shaped, not idle-shaped — the expensive mistakes are architectural (chatty events, payloads that should be GCS references, Composer where Workflows would do).

A worked month for a mid-size platform — 50 M events, 2 KB average payload, three consumers per event, 5 M commands, 1 M saga executions of ~12 steps:

Sizing heuristics: keep payloads ≤ ~10 KB (larger → GCS object + reference event); size Cloud Run as instances × concurrency × (1/handler-seconds) ≥ arrival rate + drain margin; and multiply your hottest saga’s step count by monthly volume before committing — a 40-step saga at 10 M executions/month is $4,000; restructure or batch. Baseline for perspective: one always-on Composer environment costs more than this entire worked example.

Interview & exam questions

Relevant to the Professional Cloud Architect and Professional Cloud Developer certs; also standard system-design interview territory.

1. Q: When would you choose Cloud Tasks over Pub/Sub? A: When the message is a command with one intended executor and I need controls Pub/Sub lacks: per-queue rate limiting, concurrency caps, scheduled delivery up to 30 days, named-task deduplication, and cancellation of pending work. Pub/Sub wins for facts that fan out to unknown consumers.

2. Q: Explain choreography vs orchestration and when each fits. A: Choreography lets services react to each other’s events — minimal coupling, easy consumer addition, but no owner of the end-to-end outcome. Orchestration (Workflows) centralises the flow: explicit steps, retries, compensation and one execution record. Broadcast facts choreograph; money paths and SLA-bound flows orchestrate.

3. Q: What is a saga and how does Workflows implement compensation? A: A saga replaces a distributed transaction with local transactions, each paired with an undo. In Workflows each forward step is a try with a retry policy; the except block runs compensations for already-completed steps (release reservation, refund charge) and re-raises so the execution fails honestly.

4. Q: Pub/Sub exactly-once delivery is enabled — can duplicates still occur end to end? A: Yes. Exactly-once applies to redelivery of acked messages on a regional pull subscription. A publisher retry creates a second message with a new messageId, and a handler crash after its side effect but before ack still repeats the effect. End-to-end exactly-once needs idempotent handlers and, where DB state and events must agree, an outbox.

5. Q: What problem does the transactional outbox solve? A: The dual-write inconsistency: committing state and publishing an event are two systems that can’t share a transaction. Writing the event to an outbox table in the same DB transaction, with an at-least-once relay publishing it, guarantees state and event agree; consumer idempotency absorbs relay duplicates.

6. Q: Three ways Eventarc gets events, and when do you use audit-log triggers? A: Direct events from integrated sources (fastest, typed — e.g. GCS object.finalized), Cloud Audit Logs events (near-universal API coverage, higher latency, Data Access logs required), and custom Pub/Sub topics. Audit-log triggers are the fallback when no direct event exists — e.g. reacting to a BigQuery job completion.

7. Q: How do you stop a burst of events from crushing a rate-limited legacy system? A: Don’t fan Pub/Sub directly into it. Route that leg through a Cloud Tasks queue whose maxDispatchesPerSecond/maxConcurrentDispatches sit under the target’s real capacity; the burst becomes queue depth. Pub/Sub can feed an enqueuer, keeping fan-out semantics upstream.

8. Q: An ordered subscription has stopped delivering for one customer but others flow. Why? A: Head-of-line blocking on that customer’s ordering key — a message that keeps failing blocks everything behind it on the same key (ordering also caps publishes ~1 MiB/s per key). Attach a dead-letter policy so the poison message parks after max attempts, unblocking the key, then fix and replay.

9. Q: Workflows vs Cloud Composer? A: Workflows is serverless, per-step billed, millisecond-start, year-long executions with callbacks — built for service orchestration and sagas. Composer is managed Airflow: an always-on environment (hundreds of dollars monthly) with the operator/backfill ecosystem data engineering expects. Orchestrate microservices with Workflows; orchestrate data pipelines with Composer.

10. Q: How does trace context survive a Pub/Sub hop? A: It must travel in the message, not the connection — the client libraries’ OpenTelemetry integration propagates W3C context via a googclient_traceparent attribute and emits publish/deliver spans; consumers extract it and start child spans. Logs correlate via the logging.googleapis.com/trace field.

11. Q: Design “release unsold seat holds after exactly 10 minutes” without cron sweeps. A: On hold creation, enqueue a named Cloud Tasks task release-{holdId} with scheduleTime = now+10m targeting the release endpoint; on purchase, delete the task. Exact-time dispatch, per-hold cancellation, named-task dedup — no polling, no sweep races.

12. Q: What belongs in message attributes vs payload? A: Attributes carry routing and infrastructure metadata — event type/version, tenant, trace context, idempotency key — filterable without deserialising. The payload carries the schema-governed domain fact. Never PII in attributes.

Quick check

1. A team publishes generate-invoice-pdf messages to a Pub/Sub topic consumed by one service that keeps overloading a PDF API. What two properties do they need that Pub/Sub can’t give them?
2. Your saga’s email step cannot be undone. Where does it go in the workflow and why?
3. dead_letter_message_count just went above zero. Is that an error in the DLQ setup?
4. Exactly-once delivery is on, yet finance sees duplicate ledger entries. Name two remaining duplicate sources.
5. Which Eventarc source class reacts to any GCP API call, and what must be enabled first?

Answers

1. Per-consumer rate limiting and concurrency caps (plus cancellation) — it’s a command, so it belongs in a Cloud Tasks queue with maxDispatchesPerSecond tuned to the PDF API.
2. Last, after every step that can fail — irreversible actions run only once all compensable steps have succeeded, so no compensation ever needs to “unsend” it.
3. No — it means the system worked: a poison message parked after max delivery attempts instead of wedging the subscription. The action is triage, fix, replay.
4. Publisher retries (new messageId, same fact) and handler crash after the side effect but before ack; exactly-once only prevents redelivery of already-acked messages on that pull subscription.
5. Cloud Audit Logs triggers (google.cloud.audit.log.v1.written filtered by serviceName/methodName) — Data Access audit logs must be enabled for the service, with their logging cost accepted.

Glossary

• Event: immutable statement that something happened, published without knowledge of consumers.
• Command: targeted instruction to one executor, typically rate-controlled and cancellable.
• Choreography: flow emerging from services reacting to each other’s events; no central owner.
• Orchestration: a central definition (Workflows) driving steps, state, retries and compensation.
• Saga: multi-step business transaction built from local transactions plus compensating undo steps.
• Compensation: the paired undo of a saga step (release, refund, void), run on downstream failure.
• CloudEvents: the CNCF envelope standard (ce-id, ce-source, ce-type…) Eventarc delivers in.
• Eventarc trigger: filter + destination binding routing Google Cloud or custom events to a service.
• Ordering key: per-message key giving in-order delivery per key at ~1 MiB/s publish per key.
• Exactly-once delivery: Pub/Sub regional pull feature preventing redelivery of acked messages — not end-to-end effect-exactly-once.
• Dead-letter topic (DLQ): topic receiving messages after maxDeliveryAttempts failures, for triage and replay.
• Idempotency key: business identifier letting a handler detect and no-op duplicate deliveries.
• Transactional outbox: event row committed atomically with state, published by an at-least-once relay.
• Connector: Workflows’ typed call to a Google API handling auth, retries and long-running operations.

Next steps

• Go deep on the backbone’s mechanics — subscription types, ordering, exactly-once, DLQ internals — in GCP Pub/Sub and Event-Driven Architecture: Decouple and Scale.
• Harden the consumers: Cloud Run explained — serverless containers that scale to zero, then the Cloud Run vs GKE vs Compute Engine decision for where handlers should live.
• Wire the identities correctly with GCP IAM service accounts & least privilege — the IAM triangle in this article is 90% of Eventarc tickets.
• Build the dashboards and SLOs from the observability section with Cloud Monitoring & operations, and land the analytical sink via BigQuery as your analytics warehouse.
• Porting patterns from AWS? Map EventBridge/SQS/Step Functions habits across with AWS Lambda event-driven patterns.