GCP Multi-Region Active-Active Architecture: Global Load Balancing, Spanner and Cross-Region Data

Every cloud region eventually has a bad day. Not a zone — a region: a networking control-plane fault, a capacity event, a bad rollout that takes your stack down no matter how many zones you spread across. When that day comes, single-region architecture has one answer — wait — and every minute is revenue, SLA credits and trust. Multi-region active-active removes the waiting: two or more regions serve production traffic simultaneously, so losing one is not a disaster to recover from but a capacity event to absorb.

Active-active is really two hard problems wearing one name: traffic (how users reach the closest healthy region, and how traffic moves in seconds when one dies — without trusting every DNS resolver on the internet) and state (containers run anywhere; data lives somewhere, and physics charges 100+ ms between continents). Google Cloud hands you a distinctive tool for each. For traffic, the global external Application Load Balancer is one worldwide load balancer with one anycast IP — failover is a routing decision inside Google’s network, not a DNS change you hope clients notice. For state, Cloud Spanner multi-region configurations deliver external consistency and a 99.999% SLA — synchronous-quorum writes across regions, RPO zero, no promote step.

This article is the full design: DR tiers and the RTO/RPO contract; the anycast front door and why it beats DNS failover (with an honest AWS/Azure contrast); Cloud Run and MIGs serving from two regions behind one backend service; the data-layer decision between Spanner, Cloud SQL cross-region replicas, AlloyDB and Firestore; GCS dual-region with turbo replication; session and cache design; failover drills; replication-lag realities; cross-region observability; and cost — because active-active is a spectrum, and the priciest mistake is buying five nines for data that needed three. Everything ships with real gcloud and Terraform.

What problem this solves

A well-built single-region deployment — multi-zone MIGs or Cloud Run, regional Cloud SQL HA — already survives zone loss. It cannot survive the failure domain above the zone: the region itself and its control planes and quotas. Active-active caps the blast radius of exactly five failure classes:

Only class 1 is “disaster recovery” in the classic sense; classes 2–5 are everyday value — spillover, latency, safer deploys. That is the honest financial argument: you are not paying double to insure a once-in-three-years event, you are buying a permanently better serving platform that also survives region loss. Who needs it: payments and trading platforms with contractual RPO/RTO, global SaaS, retail that cannot be down during a sale, and any workload whose 99.99%+ target mathematically exceeds a single region’s dependency chain.

What breaks without it: DNS-failover plans that were never tested and quietly depend on a 60-second TTL mobile carriers ignore; “restore from backups” plans with a real RTO of a working day; and read-replica DR where the promote runbook lives in one engineer’s head and replication lag at the moment of disaster decides — at random — how many transactions you lose.

Learning objectives

By the end of this article you can:

• Place any workload on the four-tier DR spectrum (backup-restore → pilot light → warm standby → active-active) and defend each tier’s RTO/RPO and cost with real GCP mappings.
• Explain how anycast and the global external ALB turn regional failover into a sub-minute routing event — and contrast with Route 53 DNS failover and AWS Global Accelerator.
• Build a two-region serving layer: Cloud Run behind serverless NEGs with outlier detection, and regional MIGs with balancing modes, capacity scalers and health-check failover.
• Choose the multi-region data engine per dataset — Spanner (TrueTime, external consistency, 99.999%), Cloud SQL replicas + promote, AlloyDB secondaries, Firestore — and state the RPO/RTO each delivers.
• Configure GCS dual-region with turbo replication and reason about the 15-minute RPO SLA vs default async replication.
• Design session and cache state so a region evacuation neither logs out nor corrupts users (JWTs, affinity limits, Memorystore’s regional boundary).
• Run failover as a drill: drain a region with --capacity-scaler=0, promote a replica, measure real RTO/RPO, alert on lag before it becomes loss.
• Model the cost delta honestly: Spanner premium vs replica pairs, cross-region egress, idle capacity, and where scale-to-zero makes region B nearly free.

Prerequisites & where this fits

You should be comfortable with GCP’s resource model, have deployed something real on Cloud Run or Compute Engine, and know basic Cloud SQL administration. The failure-domain mental model is upstream reading: GCP Regions and Zones: resiliency and what regions and zones really mean. Because GCP’s VPC is global by design, skim why the GCP network spans the planet — two regions sharing one VPC, one firewall policy and one address space quietly removes half the plumbing AWS multi-region designs need.

This article sits at the top of the GCP architecture track, composing services covered elsewhere (Cloud Run, compute choices, Cloud Monitoring) into a cross-region system. Running Azure too? The companion design is Azure multi-region active-active & disaster recovery — reading both shows what is cloud philosophy and what is physics.

Every layer crosses regions differently; knowing which layer you are debugging is half the incident:

The table whispers the core truth: everything above the database line fails over in seconds, by Google; everything at and below it fails over as fast as your data architecture allows — which is why the data-layer section below is the longest.

Core concepts

Active-active vs active-passive. In active-passive, region B holds capacity and replicas but serves nothing until a failover event changes state — DNS, a promote, a scale-up — and every untested state change fails at 2 a.m. In active-active, both regions serve all day; “failover” is just removing one region from a set the load balancer already balances across. No cliff-edge state change on the serving path is precisely why active-active RTOs are seconds. The price moves to the data layer, where “both regions accept writes” forces a choice between synchronous quorum (latency), async replication (RPO > 0), or partitioning writes by geography.

RTO and RPO are the contract. RTO (Recovery Time Objective): how long until service is restored. RPO (Recovery Point Objective): how much recently-written data you may lose, in time. They are business numbers: “checkout back in 5 minutes, zero lost payments” prices the ledger onto Spanner or an equivalent quorum store, because no asynchronous replica can promise RPO 0. Write the numbers per dataset, not per company; a catalog tolerating 15 minutes of loss should not pay the ledger’s bill.

The consistency-latency trade is physics. A synchronous cross-region commit cannot return faster than the round trip to a quorum of replicas — ~10–20 ms between nearby regions in a Spanner config, far more across continents. Async replication returns instantly and moves the cost to failover time, where lag becomes loss. Every data product below is one point on that line; the skill is matching each dataset to the right point.

GCP’s shape is unusually kind to multi-region. The VPC is global (one network, regional subnets, backbone transit — no peering mesh between your regions). The external Application Load Balancer is a global control plane — one URL map, one certificate set, one IP, backends in many regions. And several data services are natively multi-region (Spanner, Firestore, GCS dual/multi-region, Pub/Sub), so replication is a checkbox with an SLA, not a pipeline you operate. Know which services are regional and which are global before drawing a single box:

The availability math. Availability compounds down through serial dependencies and up through parallel paths. Single-region is serial: LB × Cloud Run × Cloud SQL ≈ 99.95%³ ≈ 99.85% by SLA composition — ~13 hours/year of allowed pain before your own bugs. Two independent 99.9% regions in parallel approach 1 − (0.001)² = 99.9999% for serving. Reality is less generous — regional failures share control planes, deploys and a data layer, which usually caps the result — but the direction is right:

Read the last row carefully: Google sells no composite “your app is now 99.999%” SLA. Active-active is how you engineer toward five nines; the SLAs floor each part, and lining the parts up is the rest of this article.

Vocabulary you need fluent before the deep dive:

DR tiers and the RTO/RPO contract

Active-active is the top rung of a ladder, and honest architecture starts by admitting most workloads belong on a lower rung. The four canonical tiers, priced and mapped to GCP:

Three rules make the table useful. Tier per dataset, not per company: the ledger runs tier 4 on Spanner while the recommendation cache sits at tier 1. RPO 0 is a quorum property: if “asynchronous replica” appears anywhere in a dataset’s path, its true RPO is the lag at the worst moment, whatever the slide says. An untested tier is the tier below it: a warm standby never promoted is a pilot light; a pilot light never lit is backup-and-restore with extra billing.

And because “what RTO/RPO does each service actually give me” is what every review board asks — the reference, SLA-backed figures marked:

The rows design reviews fight about are the Cloud SQL ones — dissected with commands below. First, the part that makes GCP active-active feel unfairly easy: the front door.

The global front door: one anycast IP, no DNS failover

The most under-appreciated fact in GCP networking: the global external Application Load Balancer is not a regional appliance with global aspirations — it is Google’s edge. Creating one gives you a single anycast IP address advertised via BGP from 100+ edge points of presence. A user in Mumbai and a user in Frankfurt hit the same IP and each lands on the nearest GFE (Google Front End), where TLS terminates milliseconds from the client; the request then rides Google’s private backbone (cold-potato routing, the defining behavior of Premium Network Tier) to whichever of your backend regions the LB chooses. One IP. One certificate set. One URL map. Backends in as many regions as you like.

Why anycast failover beats DNS failover

Contrast this with the DNS-failover pattern you inherit on clouds whose L7 load balancers are regional. There, “global” means: two regional ALBs, two IPs, and a DNS layer (Route 53 on AWS, Traffic Manager on Azure) handing out one of them per lookup, with health checks deciding which. Every link in that chain is a place failover goes to die:

The DNS column deserves the scar tissue: TTL is a request, not a command. Enterprise resolvers, mobile-carrier DNS, JVMs with networkaddress.cache.ttl=-1 and long-lived connection pools happily pin the dead region’s IP for minutes to hours after your failover “completed”. Anycast removes the client from the failover path: the IP stays correct; the route underneath it changed. (Azure Front Door plays the same anycast trick — see the Azure companion article; AWS Global Accelerator buys anycast entry, but the L7 brain — listeners, rules, WAF — stays duplicated per region behind it.)

Two prerequisites make or break this on GCP: the one-VIP global ALB exists only in Premium Network Tier (Standard forces regional forwarding rules — back to DNS games), and you want the EXTERNAL_MANAGED scheme (the Envoy-based global external ALB) — outlier detection and advanced traffic management live there, not in the classic ALB.

How the LB picks a region: waterfall by region

Knowing the routing algorithm lets you predict failover instead of hoping. Per request at an edge PoP, the LB (1) applies the URL map to select a backend service, (2) sends the request to the closest region with healthy capacity among that service’s backends, and (3) at capacity or unhealthy, spills to the next-closest region — the “waterfall”. Health state comes from centralized probes originating from 35.191.0.0/16 and 130.211.0.0/22 — firewall rules must allow them (the #1 rookie blocker). Consequences you design around:

• Failover is granular and automatic: a region failing health checks just stops receiving the waterfall, per request, no operator action.
• Capacity is part of routing: an alive-but-full region spills its excess — and if region B is sized for 50% when region A dies at peak, spillover becomes your rate limiter.
• Proximity is per-user: EU users never normally touch us-central1; after a europe-west1 drain they transparently pay the transatlantic RTT instead of erroring.

Session affinity across regions

Affinity is where active-active quietly breaks stateful assumptions — every mode is best-effort, legally broken by capacity changes, backend churn and failover:

The rule: affinity may optimize (cache locality), never guarantee. State that matters lives where both regions can reach it — the session-design section below.

Build it: gcloud

The full two-region front door, Cloud Run edition (MIG variant in the next section):

PROJECT=kv-pay-prod
REGIONS=(us-central1 europe-west1)

# 1. One global anycast IP — this is THE address, forever
gcloud compute addresses create pay-vip --global --ip-version=IPV4

# 2. A serverless NEG per region, each pointing at that region's Cloud Run service
for R in "${REGIONS[@]}"; do
  gcloud compute network-endpoint-groups create neg-checkout-$R \
    --region=$R --network-endpoint-type=serverless \
    --cloud-run-service=checkout
done

# 3. ONE global backend service holding BOTH regional NEGs
gcloud compute backend-services create bes-checkout \
  --global --load-balancing-scheme=EXTERNAL_MANAGED --protocol=HTTPS
for R in "${REGIONS[@]}"; do
  gcloud compute backend-services add-backend bes-checkout --global \
    --network-endpoint-group=neg-checkout-$R \
    --network-endpoint-group-region=$R
done

# 4. URL map, managed cert, HTTPS proxy, forwarding rule on the VIP
gcloud compute url-maps create um-pay --default-service=bes-checkout
gcloud compute ssl-certificates create cert-pay \
  --domains=pay.example.com --global
gcloud compute ssl-policies create tls-modern \
  --profile=MODERN --min-tls-version=1.2
gcloud compute target-https-proxies create thp-pay \
  --url-map=um-pay --ssl-certificates=cert-pay --ssl-policy=tls-modern
gcloud compute forwarding-rules create fr-pay --global \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --address=pay-vip --target-https-proxy=thp-pay --ports=443

Point pay.example.com at the VIP once, and DNS never participates in an incident again.

Build it: Terraform

resource "google_compute_global_address" "vip" {
  name = "pay-vip"
}

resource "google_compute_region_network_endpoint_group" "run" {
  for_each              = toset(["us-central1", "europe-west1"])
  name                  = "neg-checkout-${each.key}"
  region                = each.key
  network_endpoint_type = "SERVERLESS"
  cloud_run { service = "checkout" }
}

resource "google_compute_backend_service" "checkout" {
  name                  = "bes-checkout"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  protocol              = "HTTPS"
  security_policy       = google_compute_security_policy.edge.id

  dynamic "backend" {
    for_each = google_compute_region_network_endpoint_group.run
    content { group = backend.value.id }
  }

  outlier_detection {                # the serverless failover mechanism
    consecutive_errors    = 5
    interval            { seconds = 10 }
    base_ejection_time  { seconds = 30 }
    max_ejection_percent  = 50
  }

  log_config {
    enable      = true
    sample_rate = 1.0
  }
}

resource "google_compute_url_map" "pay" {
  name            = "um-pay"
  default_service = google_compute_backend_service.checkout.id
}

resource "google_compute_managed_ssl_certificate" "pay" {
  name    = "cert-pay"
  managed { domains = ["pay.example.com"] }
}

resource "google_compute_target_https_proxy" "pay" {
  name             = "thp-pay"
  url_map          = google_compute_url_map.pay.id
  ssl_certificates = [google_compute_managed_ssl_certificate.pay.id]
}

resource "google_compute_global_forwarding_rule" "pay" {
  name                  = "fr-pay"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  ip_address            = google_compute_global_address.vip.id
  target                = google_compute_target_https_proxy.pay.id
  port_range            = "443"
}

The serving layer: Cloud Run and MIGs in two regions

With the front door global, the serving layer’s job is: identical, stateless capacity in each region, sized so survivors absorb a dead region’s load. Two mainstream shapes behind the same backend-service abstraction:

Cloud Run multi-region: the serverless NEG contract

A Cloud Run service is regional; multi-region means deploying the same service into each region and stitching them with one serverless NEG per region on a single global backend service — exactly what the front-door code builds. Deployment is a loop, and should be a pipeline stage, not a human:

IMAGE=us-docker.pkg.dev/$PROJECT/apps/checkout:v2026.07.07
for R in us-central1 europe-west1; do
  gcloud run deploy checkout --region=$R --image=$IMAGE \
    --min-instances=2 --max-instances=200 --concurrency=80 \
    --cpu=2 --memory=1Gi --port=8080 \
    --ingress=internal-and-cloud-load-balancing \
    --set-env-vars=REGION=$R \
    --no-allow-unauthenticated
done

The settings that specifically matter for active-active:

The catch unique to serverless: serverless NEGs cannot be health-checked — an erroring (but infrastructurally alive) region would keep its waterfall share forever. The fix is outlier detection: the LB watches real responses per backend NEG and temporarily ejects one that keeps failing, re-admitting it later to probe recovery. It is failure-driven — some real requests must fail to trigger it — so treat it as damage limitation and tune consciously:

Two operational notes: users see errors in the window before ejection kicks in — measure it in game days (typically seconds at production request rates) — and ejection is per backend service, so /checkout and /catalog eject independently, which is what you want.

Regional MIGs: capacity-aware failover, the explicit way

Where Cloud Run infers capacity, MIG backends declare it. A regional MIG spreads identical VMs across ≥3 zones (zone loss handled inside the region); the backend service then needs each group’s traffic ceiling:

The multiplier that makes drills trivial: capacity-scaler (0.0–1.0) scales declared capacity without touching VMs. --capacity-scaler=0 is the region-evacuation switch — connections drain gracefully, no new traffic admitted, 1.0 un-drains. The single most useful flag in this architecture:

# Regional MIG per region + health check + backend wiring
gcloud compute health-checks create http hc-web --port=8080 \
  --request-path=/healthz --check-interval=5s --timeout=5s \
  --healthy-threshold=2 --unhealthy-threshold=2

for R in us-central1 europe-west1; do
  gcloud compute instance-groups managed create mig-web-$R \
    --region=$R --size=6 --template=tpl-web-v42
  gcloud compute instance-groups managed set-autoscaling mig-web-$R \
    --region=$R --min-num-replicas=6 --max-num-replicas=40 \
    --target-load-balancing-utilization=0.7
  gcloud compute backend-services add-backend bes-web --global \
    --instance-group=mig-web-$R --instance-group-region=$R \
    --balancing-mode=RATE --max-rate-per-instance=800 --capacity-scaler=1.0
done

# THE regional drain / undrain (game-day favorite):
gcloud compute backend-services update-backend bes-web --global \
  --instance-group=mig-web-us-central1 --instance-group-region=us-central1 \
  --capacity-scaler=0.0

Health-check math sets your serving RTO: at check-interval=5s, unhealthy-threshold=2, a hard regional failure stops receiving traffic in ~10–15 seconds plus propagation. The autoscaler handles the second half — growing the survivor — in VM-boot minutes; your capacity floor decides whether users feel the gap.

The N+1 capacity rule

Active-active sizing is an availability decision dressed as a cost decision. Two regions at 50% of global peak each means the survivor must instantly run at 100% — so each region must be provisioned, or provisionable within your RTO (quota granted, max-instances raised, autoscaler caps lifted), for the whole load. Three regions at ~34% cut the idle overhead (150% total vs 200%) at the price of a third data-locality story. Do the arithmetic explicitly in autoscaler configs and quota requests — “we’ll scale when it happens” is how spillover becomes a second outage. And the quiet killer: regional quotas (CPUs, IPs, Cloud Run instances) are default-sized for single-region thinking; file region-B increases the week you adopt the architecture.

The data layer decision: Spanner vs Cloud SQL vs AlloyDB vs Firestore

Everything so far fails over in seconds because it is stateless. The data layer is where active-active is won or lost, and the question is not “which database is best” but “which consistency-latency-cost point does this dataset need”:

Senior teams converge on polyglot by RPO: money tables to Spanner (or Firestore if document-shaped), the long tail on Cloud SQL with an honest minutes-grade RPO, caches regional and disposable. Forcing the whole estate onto Spanner buys consistency nobody asked for at a price everybody notices; leaving the ledger on async replicas writes the “we lost 3 minutes of payments” post-mortem in advance.

Cloud Spanner multi-region: TrueTime and external consistency

Spanner multi-region is the only relational option where both regions accept writes with zero loss on region failure — because there is no “failover” in the replication sense at all. A config places replicas of every database split in a fixed, named topology:

A typical two-serving-region config runs five voters: 2 read-write in region A, 2 in region B, 1 witness in region C. A write commits when a majority (3 of 5) durably logs it — which by construction spans two regions, hence RPO 0. Lose the leader region and the survivors elect a new leader in seconds; clients retry and continue. Nobody promotes anything.

TrueTime makes this globally consistent rather than merely durable: GPS receivers and atomic clocks in Google’s datacenters expose time as a bounded interval, and Spanner timestamps every transaction inside that bound, waiting out the uncertainty (commit-wait, single-digit ms) before acknowledging. If T2 starts after T1 commits, T2’s timestamp is provably greater — globally. That is external consistency: reads at a timestamp never see a partially-applied or out-of-order world from any region, with zero application effort.

The named configurations you will actually choose between (verify layouts with gcloud spanner instance-configs describe <config> — they are fixed per config):

Read the last row twice: the intercontinental config does not give low-latency writes on three continents — voting stays close (physics); distant continents get fast reads via read-only replicas. Choosing a config is choosing where the leader sits (nearest your write-heavy services — leader-aware routing in the client libraries sends writes there automatically) and where reads serve locally.

Reads are where you claw back latency, because Spanner lets you choose staleness per read:

Provisioning is in processing units (100 PU = 0.1 node; 1000 PU = 1 node), with managed autoscaling available — floor it for quorum comfort, cap it for budget:

gcloud spanner instances create pay-ledger \
  --config=nam3 --description="payments ledger" \
  --autoscaling-min-processing-units=1000 \
  --autoscaling-max-processing-units=5000 \
  --autoscaling-high-priority-cpu-target=65 \
  --autoscaling-storage-target=90

gcloud spanner databases create ledger --instance=pay-ledger

resource "google_spanner_instance" "ledger" {
  name         = "pay-ledger"
  config       = "nam3"
  display_name = "payments ledger"
  autoscaling_config {
    autoscaling_limits {
      min_processing_units = 1000
      max_processing_units = 5000
    }
    autoscaling_targets {
      high_priority_cpu_utilization_percent = 65
      storage_utilization_percent           = 90
    }
  }
}

Two production warnings. Hotspots: sequential primary keys (timestamps, auto-increment) funnel writes to one split and cap the five-nines database at one machine’s throughput — use UUIDv4, hashed prefixes or interleaved child tables. CPU headroom: keep multi-region high-priority CPU under ~45–65% so failover surges and split rebalancing have room; a hot instance fails over slower.

Cloud SQL cross-region replicas: honest asynchrony

Most estates arrive with MySQL/PostgreSQL schemas that would cost quarters to port. Cloud SQL cross-region read replicas are the pragmatic tier: the primary in region A streams async replication to a replica in B, which serves reads today and becomes the DR primary tomorrow — if you operate it with clear eyes:

# Primary (regional HA within us-central1) — Enterprise Plus for the DR features
gcloud sql instances create catalog-db \
  --database-version=POSTGRES_16 --edition=enterprise-plus \
  --tier=db-perf-optimized-N-4 --region=us-central1 \
  --availability-type=REGIONAL

# Cross-region replica in europe-west1
gcloud sql instances create catalog-db-euw1 \
  --master-instance-name=catalog-db --region=europe-west1 \
  --tier=db-perf-optimized-N-4

# The DR moment (unplanned): one-way promotion
gcloud sql instances promote-replica catalog-db-euw1

# The drill / planned move (Enterprise Plus): coordinated, zero-loss role swap
gcloud sql instances switchover catalog-db-euw1

The three role-change operations get confused in every incident channel; keep the distinction on the wall:

After promotion the writer endpoint changes — applications must repoint. Bake that in day one: connect via DNS you control (private Cloud DNS record per role) or the Enterprise Plus write endpoint (a DNS name that follows the primary across switchovers), never hardcoded IPs. And monitor the number that is your RPO — database/replication/replica_lag: alert at 30 s, page at 5 min, knowing bulk imports, ALTER TABLEs and vacuum storms spike it exactly when you least want.

AlloyDB secondary clusters

AlloyDB for PostgreSQL upgrades the same single-writer pattern: a primary cluster in region A, a secondary cluster in B fed by storage-layer replication (seconds of lag), read pools in both, with promote (emergency, one-way) and switchover (planned, reversible):

gcloud alloydb clusters create-secondary catalog-sec \
  --primary-cluster=projects/$PROJECT/locations/us-central1/clusters/catalog \
  --region=europe-west1

# drill:
gcloud alloydb clusters switchover catalog-sec --region=europe-west1

Choose AlloyDB over Cloud SQL for the faster engine (columnar cache, vectorized reads) and cleaner DR ergonomics without leaving PostgreSQL. It remains async cross-region — RPO small but not zero.

Firestore multi-region

For document-shaped state — profiles, carts, presence, mobile sync — Firestore in a multi-region location (nam5 United States, eur3 Europe) gives the Spanner treatment for free operationally: two serving regions plus a witness, strong consistency, automatic failover, 99.999% SLA, no capacity planning. Its limits are structural: document model, ~1 sustained write/second per document (fan out counters), query semantics that punish relational thinking. When data is naturally documents it is the cheapest RPO-0 you will ever buy; when it is not, forcing it costs more than Spanner would have.

Storage, cache and session state across regions

Cloud Storage: multi-region, dual-region and turbo replication

GCS is strongly consistent for read-after-write globally — the multi-region questions are where copies live and how fast they get there:

Dual-region is the active-active workhorse: you pick the pair to match your compute regions — one bucket, one namespace, served in both, no failover step. Respect the replication fine print:

gcloud storage buckets create gs://pay-media-prod \
  --location=us --placement=us-central1,us-east1 \
  --rpo=ASYNC_TURBO --uniform-bucket-level-access

resource "google_storage_bucket" "media" {
  name     = "pay-media-prod"
  location = "US"
  rpo      = "ASYNC_TURBO"
  custom_placement_config {
    data_locations = ["US-CENTRAL1", "US-EAST1"]
  }
  uniform_bucket_level_access = true
}

Notes that bite later: region loss during the replication window can strand objects written in the last ≤15 minutes until the region returns (RPO ≠ 0 — plan idempotent re-upload); co-locate the bucket pair with your serving pair (requests use the nearer copy); and front static assets with Cloud CDN — an edge cache hit makes the bucket’s region irrelevant. Storage classes and lifecycle are orthogonal — see Cloud Storage classes explained.

Memorystore: the cache stays regional — design accordingly

Memorystore for Redis (classic) is regional with zero cross-region replication; Memorystore for Redis Cluster adds async cross-region replica clusters. The stance that never regrets itself: the cache is regional, disposable and rebuildable — cross-region cache replication ships every write across the ocean for data that was, by definition, reconstructible.

Engineer, don’t hope, around the thundering herd after failover — a cold cache letting 100% of reads through to the database. Mitigations: request coalescing (singleflight), cache warmers on drain events, TTL jitter, a DB tier sized for a cold-cache minute.

Sessions and user state

Session design decides whether a region evacuation is invisible or a mass logout. The options, from best to workaround:

Corollaries: make every write path idempotent (an evacuated user’s retried POST must not double-charge — idempotency keys in the replicated tier), and propagate a request-region tag end-to-end. For async work, prefer Pub/Sub — a global service where both regions share the topic, with a message storage policy to constrain persistence regions if residency demands (see Pub/Sub event-driven architecture).

Failover mechanics, testing and replication-lag realities

What actually happens when a region dies

The timeline for a hard us-central1 failure with the reference architecture (Cloud Run + MIGs behind the global ALB, Spanner nam3, Cloud SQL replica in europe-west1):

Only two moments are human: the SQL promote decision and the eventual un-drain. Everything else is machinery — if capacity, quotas and min-instances were pre-provisioned.

Drills: if you have not run it, you do not have it

Failover you have never executed is a rumor. The drill catalogue, in escalating order of nerve:

Record for each drill: detection time, traffic-shift time (LB logs), data role-change time, client-observed error window (synthetic probes per continent), and lag at cutover. Those five numbers are your real RTO/RPO — put them in the DR doc instead of the aspirational ones.

Replication lag: the honest table

The consequence: any promotion decision reads the lag first. A 4-minute lag means promoting now = losing 4 minutes of writes; waiting risks a longer outage. Decide the policy before the incident (“promote if lag < 60 s; else escalate to IC”) and write it in the runbook.

Observability across regions

Active-active turns “is it down?” into “is it down for whom, where?” — telemetry must answer by region or it answers nothing. Three practices carry the weight (deep dive: Cloud Monitoring & operations):

Probe from outside, per continent. Uptime checks run from Google probers in USA, Europe, Asia-Pacific and South America against the anycast VIP; pair them with per-region direct checks so “region A is sick” is distinguishable from “the front door is sick”. Alert on SLO burn rate, not raw errors.

Slice everything by region. LB request logs and metrics carry the serving region (backend_scope) and statusDetails; app logs carry the REGION env var you set at deploy. Every dashboard chart gets a region group-by; every SLO a per-region child. The one Logging filter to bookmark — every request the LB could not serve from a backend:

resource.type="http_load_balancer"
resource.labels.url_map_name="um-pay"
jsonPayload.statusDetails!="response_sent_by_backend"

Trace across the ocean. Propagate trace context end-to-end (Cloud Trace/OpenTelemetry) so a checkout that entered in Frankfurt, ran in europe-west1 and committed to a Spanner leader in us-east4 reads as one trace — that leader hop is the latency you will be asked to explain. Watch Spanner commit latency by region and the surviving region’s p95 during drills.

Architecture at a glance

Read the diagram left to right — the request’s actual journey. Users everywhere resolve one hostname to one anycast VIP; TLS terminates at the nearest edge PoP, Cloud Armor filters before your regions see a packet, and the global external ALB applies one URL map to pick the closest healthy region with capacity. Region A (us-central1) and Region B (europe-west1) are deliberate mirrors: Cloud Run behind serverless NEGs (badge 2 — no health checks, outlier detection does the ejecting), MIGs across three zones, and a Cloud SQL primary in A streaming an async replica to B (badge 3: that lag is the catalog RPO; badge 4: promotion is one-way). On the right sits shared state: Spanner multi-region (badge 5: TrueTime quorum, RPO 0, self-electing leader) and a dual-region GCS bucket with turbo replication (badge 6: 15-minute RPO SLA).

Badge 1 is the thesis: failover never touches DNS. Kill Region A and the same IP keeps working — the waterfall reroutes to Region B in seconds, autoscalers absorb the surge, Spanner keeps committing, and the only human decision left is whether the replica’s lag is small enough to promote.

Real-world scenario

NovaPay, a Bengaluru-based payments processor, ran everything in us-central1: Cloud Run APIs, Cloud SQL PostgreSQL (regional HA), Memorystore sessions. Zonal HA had worked twice. Then a regional networking degradation put them hard down 47 minutes on a Friday — ₹2.1 crore in failed transactions, one unpleasant board call, and a mandate: RTO 5 minutes, RPO 0 for the ledger, budget scrutiny on everything.

The redesign followed this article’s shape. Serving: Cloud Run in us-central1 + europe-west1 (EU merchants got faster — an unplanned win), one global external ALB on one anycast IP, Cloud Armor in front, min-instances=3 per region after load tests exposed cold-start pain at surge. Data: split by RPO instead of migrating wholesale. The ledger and idempotency keys went to Spanner nam3 at 2,000 PU — a real migration (UUID keys to kill a week-one hotspot, interleaved transaction-line tables). The catalog and back office — 340 tables of stock PostgreSQL — stayed on Cloud SQL Enterprise Plus with a europe-west1 DR replica and the write endpoint. Sessions became JWTs; Memorystore shrank to a per-region read cache.

The first game day found the bodies: draining us-central1 worked in 40 seconds, but europe-west1 Cloud Run slammed into its default max-instances cap (quota nobody had filed), and the catalog replica showed 6 minutes of lag — the drill coincided with the nightly reconciliation batch. Fixes: quotas doubled, reconciliation chunked and moved off-peak, lag alert at 30 s, a written promote policy (“promote if lag < 60 s, else IC decides”). Second drill, eight weeks later: 3 m 40 s evacuation end-to-end, zero ledger loss, 11 s of client-visible errors.

The board stopped asking about the +90% after seeing the last row.

Advantages and disadvantages

Honest summary: active-active pays for itself when downtime is expensive, users are global, or an SLA says 99.99%+. A regional business tolerant of an hour’s outage gets 80% of the protection at 40% of the cost from warm standby (tier 3) — and this same architecture scales down to that tier by dropping min-instances while keeping the LB wiring.

Hands-on lab

The smallest honest active-active: Cloud Run in two regions behind one anycast IP — prove nearest-region routing, drain a region, watch traffic move. Cost: pennies for an hour (Cloud Run scales to zero; the forwarding rule bills ~US$0.025/h — tear down at the end).

1. Setup.

gcloud config set project YOUR_PROJECT_ID
gcloud services enable run.googleapis.com compute.googleapis.com
REGIONS=(us-central1 europe-west1)

2. Deploy the same service to both regions (revision suffix marks the region so responses are distinguishable):

for R in "${REGIONS[@]}"; do
  gcloud run deploy hello-aa --region=$R \
    --image=us-docker.pkg.dev/cloudrun/container/hello \
    --revision-suffix=$R --allow-unauthenticated
done

3. Wire the global front door (HTTP :80 to skip cert wait in a lab):

gcloud compute addresses create aa-vip --global
for R in "${REGIONS[@]}"; do
  gcloud compute network-endpoint-groups create neg-hello-$R \
    --region=$R --network-endpoint-type=serverless --cloud-run-service=hello-aa
done
gcloud compute backend-services create bes-hello --global \
  --load-balancing-scheme=EXTERNAL_MANAGED --protocol=HTTP
for R in "${REGIONS[@]}"; do
  gcloud compute backend-services add-backend bes-hello --global \
    --network-endpoint-group=neg-hello-$R --network-endpoint-group-region=$R
done
gcloud compute url-maps create um-hello --default-service=bes-hello
gcloud compute target-http-proxies create thp-hello --url-map=um-hello
gcloud compute forwarding-rules create fr-hello --global \
  --load-balancing-scheme=EXTERNAL_MANAGED --address=aa-vip \
  --target-http-proxy=thp-hello --ports=80
VIP=$(gcloud compute addresses describe aa-vip --global --format='value(address)')
echo "http://$VIP"

4. Prove nearest-region routing (wait ~2–3 min for the LB to program):

curl -s http://$VIP | grep -o 'hello-aa-[a-z0-9-]*'
# From India/Europe expect: hello-aa-europe-west1 ; from the Americas: hello-aa-us-central1

Run the same curl from Cloud Shell (US) vs your laptop — same IP, different serving region: anycast + waterfall in one line of output.

5. Evacuate a region and watch the failover — remove the nearest region’s backend while curling in a loop:

while true; do curl -s http://$VIP | grep -o 'hello-aa-[a-z0-9-]*'; sleep 1; done &
gcloud compute backend-services remove-backend bes-hello --global \
  --network-endpoint-group=neg-hello-europe-west1 \
  --network-endpoint-group-region=europe-west1

Within seconds the loop flips from ...europe-west1 to ...us-central1 — no DNS change, the IP never moved. Re-add the backend (add-backend, same flags) and watch it flip back.

6. See it in the logs:

gcloud logging read 'resource.type="http_load_balancer"' \
  --limit=5 --format='value(jsonPayload.statusDetails, httpRequest.status)'

7. Teardown (order matters):

kill %1
gcloud compute forwarding-rules delete fr-hello --global -q
gcloud compute target-http-proxies delete thp-hello -q
gcloud compute url-maps delete um-hello -q
gcloud compute backend-services delete bes-hello --global -q
for R in "${REGIONS[@]}"; do
  gcloud compute network-endpoint-groups delete neg-hello-$R --region=$R -q
  gcloud run services delete hello-aa --region=$R -q
done
gcloud compute addresses delete aa-vip --global -q

Common mistakes & troubleshooting

The playbook — symptom first, because that is how incidents arrive:

Best practices

• Write RTO/RPO per dataset, signed by the business — architecture follows the numbers, never the reverse.
• One anycast VIP, everything behind it; regional endpoints locked to internal + LB ingress only.
• Capacity is part of failover: N+1 sizing, region-B quotas filed, min-instances floors, autoscaler ceilings ≥ 2× normal.
• Outlier detection on every serverless backend service; health checks on everything that supports them.
• RPO-0 engines (Spanner, Firestore) for money paths; accept and alert on async lag everywhere else.
• Caches regional and disposable; design the cold-cache surge (coalescing, warmers, DB headroom).
• Writes idempotent, keys in the replicated tier — cross-region retries must be safe.
• Deploy region-by-region with automated rollback; a bad rollout burns one region for minutes, not two.
• Drill quarterly via switchover; record measured RTO/RPO in the DR doc after every drill.
• Alert on replica_lag, LB statusDetails anomalies and per-region SLO burn — page on burn rate.
• Everything in Terraform so region B is a for_each, not a hand-built snowflake that drifts.
• Quarterly, re-read quotas, certificates and Cloud Armor rules asking “does this exist in both regions?”.

Security notes

Multi-region widens the attack surface only if entry points multiply — so don’t let them. Enforce all ingress through the global LB: Cloud Armor (WAF, rate limiting, adaptive protection) and the TLS 1.2+ SSL policy then apply identically to every region, with per-region Cloud Run URLs unreachable (LB-only ingress, --no-allow-unauthenticated) and MIGs holding no external IPs. Keep cross-region service-to-service traffic on the global VPC’s private backbone — internal IPs, Private Service Connect/private services access for Cloud SQL and AlloyDB, never public hops. If you use CMEK, plan key residency deliberately: a Spanner nam3 instance or dual-region bucket needs keys available to all its locations (multi-region key rings), and losing key access is a self-inflicted regional outage. IAM is global — one least-privilege service account set per service, not per region, so drift cannot open a weaker door in region B. And set residency constraints (resource-location org policies, Pub/Sub message storage policies) plus audit logging for both regions before data flows, not after the first compliance review.

Cost & sizing

The bill, in driver order: the second serving stack, the data layer’s consistency tier, cross-region/Premium-tier traffic, the always-on floors. A reference month for a mid-size platform (~500 req/s, ~2 TB objects, US–EU pair; list-price ballpark at ₹84/US$):

Sizing rules of thumb: the serving layer roughly doubles; Spanner multi-region compute runs ~3.3× its regional price (US$3.00 vs US$0.90 node-hour on nam3 vs us-central1) plus pricier storage (~US$0.50 vs US$0.30/GB-mo); Cloud SQL DR runs ~2.1× one instance. The comparison everyone asks for, at a 4-node-equivalent workload:

That 4–5× gap between “SQL with a good runbook” and “Spanner nam3” is the true price of RPO 0 + RTO 0 with no 2 a.m. decisions — pay it precisely for datasets whose loss costs more, and lean on cheap levers elsewhere: Cloud Run scale-to-zero, committed-use discounts on the floors, Standard-tier egress for non-user-facing bulk moves.

Interview & exam questions

Directly relevant to the Professional Cloud Architect exam (case studies love RTO/RPO) and any senior-SRE loop:

1. Why does GCP multi-region failover not require DNS changes? The global external ALB exposes one anycast IP from Google’s edge; failover is an internal routing decision (health checks/outlier detection remove a region from the waterfall). Clients keep the same IP — TTLs and resolver behavior are irrelevant.
2. How would you do this on AWS, and what changes? Regional ALBs need Route 53 (DNS failover: health-check delay + TTL + resolver disobedience) or Global Accelerator (anycast entry, L7 config still duplicated per region). GCP’s difference is the single global L7 control plane.
3. What is “waterfall by region”? Each request goes to the closest region whose backends have capacity per balancing-mode targets; at capacity or unhealthy, traffic spills to the next-closest region automatically.
4. Serverless NEGs can’t be health-checked — how does Cloud Run failover work? Outlier detection observes real responses and temporarily ejects an erroring regional NEG. Failure-driven, so a brief error window is inherent; tune consecutiveErrors/interval/ejection and keep maxEjectionPercent ≤ 50.
5. How does Spanner achieve RPO 0 across regions? Writes commit via Paxos to a majority of voters spanning two regions plus a witness; TrueTime assigns globally ordered timestamps (external consistency). Any surviving majority holds every acknowledged write; leaders re-elect in seconds.
6. What is a witness replica? A voting, non-serving replica in a third region that keeps quorum available when either serving region is lost, without paying to serve data there.
7. Cloud SQL cross-region DR: real RTO and RPO? RPO = replication lag at the failure instant (seconds normally, minutes under bursts); RTO = detect + decide + promote-replica + repoint, realistically 5–30 minutes. Enterprise Plus switchover is the zero-loss planned path for drills.
8. Why is promotion one-way? The promoted replica starts a new timeline; the old primary’s diverged history cannot rejoin — it must be deleted and re-created as a replica of the new primary.
9. When Spanner vs Cloud SQL replicas vs Firestore? Spanner: relational + RPO 0 + auto-failover (ledgers, orders). Cloud SQL: existing SQL estates tolerating minutes. Firestore: document-shaped state wanting RPO 0 with zero ops. Split by dataset, not company.
10. What breaks affinity-based sessions here? Affinity is best-effort — capacity shifts, churn and failover re-route users. State belongs in replicated stores (JWTs, Firestore/Spanner sessions); affinity is only a cache-locality optimization.
11. GCS dual-region vs multi-region? Dual-region pins the exact pair to your compute regions and unlocks turbo replication (15-min RPO SLA); multi-region is Google-chosen placement, best-effort replication. Same namespace either way — no read failover step.
12. How do you prove RTO/RPO? Drills: drain a region (capacity-scaler=0/remove-backend), Enterprise Plus switchover; measure with synthetic probes and LB logs; record detection, traffic-shift, role-change and client-error windows; alert on replica_lag as the live RPO.

Quick check

1. What single fact makes GCP failover independent of DNS?
2. A serverless NEG region is throwing 500s — what mechanism removes it, and what can’t remove it?
3. Your Cloud SQL replica lag is 240 s when the region dies. What is your RPO if you promote now?
4. Which Spanner replica type votes but never serves reads, and why does it exist?
5. What does --rpo=ASYNC_TURBO promise, and on which bucket type?

Answers

1. The global external ALB’s single anycast IP — reroute happens inside Google’s network, per request; the client-facing address never changes.
2. Outlier detection ejects it based on real error streaks; health checks can’t, because serverless NEGs don’t support them.
3. ~240 seconds of writes — replication lag at the failure instant is the RPO of an async replica.
4. The witness replica — a third-region tie-breaker that keeps Paxos quorum available when either serving region is lost, without storing readable data.
5. 100% of newly written objects replicated to the second region within 15 minutes, SLA-backed — dual-region buckets only.

Glossary

• Active-active: all regions serve simultaneously; failover = removing one from a live set.
• Anycast: one IP advertised from many locations; routing delivers each client to the nearest.
• GFE: Google Front End — the edge fleet terminating TLS, hosting the global LB data plane.
• Waterfall by region: fill the nearest region to capacity, spill to the next.
• Serverless NEG: regional pointer from a global backend service to a Cloud Run service.
• Outlier detection: response-driven ejection of failing backends where health checks don’t exist.
• Balancing mode / capacity scaler: per-backend capacity target (RATE/UTILIZATION) and its 0.0–1.0 multiplier; 0 = drain.
• TrueTime: Spanner’s bounded-uncertainty clock (GPS + atomic) enabling global transaction ordering.
• External consistency: transactions appear committed in real-time order, globally.
• Witness replica: voting, non-serving Spanner replica in a third region for quorum resilience.
• Replica promotion: one-way conversion of an async replica into a primary; lag becomes loss.
• Switchover: planned, zero-loss, reversible role swap (Cloud SQL Enterprise Plus, AlloyDB).
• Turbo replication: GCS dual-region SLA — 100% of new objects replicated within 15 minutes.
• RTO / RPO: how long down / how much data lost.
• N+1 capacity: each region provisioned to absorb a failed peer’s full load.

Next steps

• Ground the failure-domain model first: GCP regions and zones: resiliency — zones, regions and what each survives.
• Understand why two regions share one network with zero plumbing: the global VPC explained.
• Go deeper on the serving tier you’ll multiply: Cloud Run explained and the Cloud Run vs GKE vs Compute Engine decision.
• Build the per-region telemetry this design depends on: Cloud Monitoring & operations observability.
• Compare the philosophy across clouds: Azure multi-region active-active & disaster recovery — anycast vs Front Door, Spanner vs Cosmos DB.