GCP Enterprise Architecture: Multi-Region DR & Resilience

In a nutshell

Disaster recovery (DR) is the plan for one specific bad day: an entire cloud region — a whole geographic cluster of Google data centres — goes dark, and your application and its data are sitting inside it. Resilience is the broader habit of engineering so that day is a non-event rather than a headline. This lesson is about doing both on Google Cloud, and — more importantly — about proving they work before you ever need them.

Two numbers rule everything here. RTO (recovery time objective) is how long you can be down before it hurts — the stopwatch from “region died” to “service restored.” RPO (recovery point objective) is how much data you can afford to lose, measured in time — the gap between your last safe copy and the moment of failure. A back-office tool might happily accept an RTO of hours and an RPO of a day; a payment ledger accepts neither and demands RTO in seconds and RPO of exactly zero. Every design decision in this lesson is, underneath, a negotiation between those two numbers and the monthly bill.

Here is the analogy to hold onto. A cold-standby DR plan is a spare car locked in a garage you have never started: on the day your main car dies on the motorway you hope it turns over — but the battery is flat, the tank is empty, and you discover this at the worst possible moment. An active/active design is a twin-engine aircraft where both engines fly the plane on every single flight: lose one at altitude and the aircraft keeps flying — and, crucially, you already know the second engine works, because it has been running the whole way. This lesson builds the twin-engine version on GCP, and then insists you run fire drills (here called game-days) that actually shut an engine off in mid-air to prove it.

Level: Advanced · Time: ~43 min

Prerequisites. You will get the most from this if you have already met the Global External Application Load Balancer and understand the basics of a managed relational database such as Cloud SQL, plus the difference between a GCP zone and a region. None of this is strictly required — every term is defined in the glossary at the end — but it makes the deep sections land faster.

After this lesson you will be able to:

• Define RTO and RPO precisely and read them straight off an incident timeline.
• Place any workload on the DR pattern spectrum — backup & restore, pilot light, warm standby, hot/active-active — and pick the cheapest pattern that still meets its tier.
• Choose the right GCP data-replication primitive (Cloud SQL cross-region replica, Spanner multi-region, dual-region GCS with turbo) for a given RPO.
• Explain why a global load balancer — not DNS — is the failover mechanism, and put real numbers to the difference.
• Separate the infrastructure-loss axis (replication) from the corruption/deletion axis (PITR, versioning, immutable backups) and design for both.
• Plan and run a game-day that turns your RTO/RPO targets from hopeful documentation into proven evidence.

Most disaster-recovery plans are works of fiction discovered to be fiction at the worst possible moment. A team writes a runbook, draws a diagram with a reassuring arrow labelled “failover to DR region,” provisions a cold standby that nobody has booted in eight months, and files the document where auditors can find it. Then a region degrades during business hours, the on-call engineer opens the runbook, and discovers that the standby database is three hours behind, the DNS TTL is 3,600 seconds, the Terraform that built the DR environment has drifted, and nobody actually knows whether the application even works there because it has never served a real request. The outage is no longer a regional incident; it is a company incident, and the post-mortem is brutal.

This article is the antidote: a complete, reusable GCP reference architecture for multi-region disaster recovery and resilience that is exercised continuously rather than documented hopefully. It is built on four load-bearing ideas. First, Cloud Spanner multi-region carries the transactional system-of-record with synchronous Paxos replication, so the hardest part of DR — losing zero committed data — is solved by the data layer itself rather than by a fragile replication job. Second, dual-region (or multi-region) Cloud Storage holds the object estate with automatic cross-region replication so files, uploads, and exports survive a regional loss. Third, capacity-aware regional failover through a global load balancer reroutes traffic in seconds without a DNS change or a human decision. Fourth — and this is the part that separates a real program from a binder — every tier is mapped to an explicit RTO and RPO tier, and those numbers are proven by scheduled game-days that kill a region on purpose. The design scales down to a single-product company protecting its one critical workload and up to a regulated enterprise running dozens of tiered services.

The business scenario

Picture an operator whose business has quietly become dependent on a system that was never designed to survive losing a data centre. The shape is the same from a 60-person fintech to a 6,000-person logistics enterprise; only the number of protected workloads and the regulatory weight change.

Three pressures consistently force teams toward this exact architecture.

The first is that a single region is a single point of failure, and the business has outgrown pretending otherwise. Every cloud region, on every provider, has bad days — a control-plane degradation, a network partition, a zonal power event that cascades, a botched maintenance. GCP regions are highly available within themselves (multiple zones), and a good zonal design survives a zone loss transparently. But a regional event — rare, yet real and recurring across the industry — takes the whole region’s services with it. When your revenue-bearing application and its database live in asia-south1, the honest statement is “we are one regional incident away from a multi-hour outage and possible data loss.” For a back-office tool that is acceptable. For the system that takes payments, books shipments, or holds the ledger, it is not.

The second is that “DR” has become a board-level, contractual, and regulatory obligation with numbers attached. Enterprise customers’ procurement teams now ask for documented RTO (recovery time objective — how long until service is restored) and RPO (recovery point objective — how much data, measured in time, you can afford to lose) in the contract. Regulators in banking, healthcare, and increasingly in data-protection regimes require demonstrated, tested resilience — not a plan, but evidence the plan works. “We have a DR region” is no longer an answer; “Tier-1 services have RTO 0 and RPO 0, proven monthly, here are the game-day reports” is.

The third is that traditional DR is expensive, drift-prone, and rarely tested — so it fails when called. The classic pattern is a cold or warm standby in a second region: a duplicate environment, a replication pipeline you babysit, and a failover procedure executed by hand under pressure. It is costly (you pay for idle capacity), fragile (the standby drifts from production because it is never exercised), and slow (failover is manual, gated by DNS and human judgment). The cruel irony is that the thing you are paying for to handle your worst day is the thing least likely to work on that day, because it has never been under load.

This architecture dissolves all three pressures by making resilience intrinsic and continuous rather than bolted-on and occasional. The data layer (Spanner, dual-region GCS) is already multi-region and synchronously durable, so there is no replication job to fail and, for the top tier, RPO is structurally zero. The traffic layer fails over automatically in seconds with no DNS wait and no runbook for the common case. And because the “DR region” is active — serving real traffic every day, not sitting cold — it cannot drift, and its readiness is proven continuously by the fact that customers are using it right now. The same blueprint protects one Tier-1 workload for a startup and a tiered portfolio for an enterprise; only the instance sizing, region count, and number of tiers change.

DR patterns and the RTO/RPO spectrum

Before we walk the specific architecture above in detail, it pays to lay out the whole menu of DR patterns and the two numbers that decide which one you actually need. Everything the rest of this lesson recommends is a deliberate point on this spectrum — chosen, not defaulted into.

RTO and RPO, precisely. Picture a single timeline of one incident. Somewhere on the left is your last recoverable copy of the data — the last backup, the last replicated commit. Then comes the moment of failure. Then a stretch of scrambling. Then the moment service is restored:

        last safe copy          failure          service restored
  ──────────●────────────────────✗──────────────────●──────────▶ time
            └───────── RPO ───────┘└────── RTO ───────┘
             (data you lose)        (time you're down)

RPO is the distance backwards from the failure to the last recoverable copy — the data written in that window is what you lose. RTO is the distance forwards from the failure to restoration — the time your users are down. They are independent axes: you can have a tiny RTO and a terrible RPO (service back in seconds, but on data that is an hour stale) or the reverse. DR design is the craft of pushing both toward zero, but only as far as each workload’s value justifies — because the cost of closing that last gap rises steeply.

The pattern spectrum. From cheapest/slowest to priciest/fastest, there are four canonical patterns (plus “cold standby” as a degenerate case):

(A “cold standby” is the degenerate case of backup & restore where you also keep an un-booted environment definition. It is the pattern this lesson argues hardest against, because “un-booted” is a synonym for “untested.”)

As you move down that table, RTO and RPO shrink while cost climbs. Pilot light is the classic cost-saver — you pay for a running database replica but not for running compute, and you spin the compute up on failover. The architecture in this lesson is the bottom row (hot/active-active) for its Tier-0 workloads, and it deliberately borrows the warm-standby and pilot-light rows for Tier-2 and Tier-3, because paying for the bottom row everywhere is how DR budgets explode.

Multi-zone vs multi-region — two different failures. A zone is one or a few data centres with shared power, cooling, and network; a region is three or more zones far enough apart to fail independently but close enough for low-latency synchronous replication. Spreading across zones (multi-zone / “regional” resources) is cheap, low-latency, and protects against the common failure — a single zone outage — with no data loss and often no user-visible impact. That is not really “DR”; it is just high availability, and it should be your default for every production workload. Spreading across regions (multi-region) protects against the rare but total failure — a whole region degrading — which is what “DR” usually means. Multi-region costs more (cross-region latency and egress, pricier data primitives) and is reserved for the workloads whose region-loss you must survive. The shorthand: multi-zone is the seatbelt you always wear; multi-region is the airbag you buy for the cars that carry precious cargo.

Choosing the data-replication primitive. Compute is easy to make multi-region — it is stateless, so you just run it in two places. The hard, expensive, RPO-defining choice is always the data tier. GCP gives you a menu:

The rule that falls out of this table: put the data whose loss is intolerable in a synchronous multi-region store (Spanner, Firestore), and be deliberate about everything else. For a Tier-2 relational workload, a Cloud SQL cross-region replica with a tested promote-replica is the correct — and far cheaper — answer; do not reach for Spanner by reflex. (Spanner’s own key-design constraints, like avoiding hotspot primary keys, are a real adoption cost you weigh at exactly this point.)

The cost-vs-RTO curve. Plot cost against “how close to zero is my RTO/RPO” and you get a hockey stick: going from a day of RPO down to an hour is cheap; from an hour to a minute costs more; from a minute to zero costs the most of all, because zero means synchronous cross-region replication running every second of every day. This is why tiering is the whole game. Spend the steep part of the curve only on the workloads that earn it, and let everything else sit on the cheap, flat part.

Architecture overview

The clearest way to understand this design is to walk two paths through it: the request path on a normal day, and the failover path on a bad one. They are deliberately almost the same path — that is the whole point.

Read the diagram top-to-bottom: one global IP and one load balancer at the top fan traffic to two regions that are both live in the middle, and those regions share one Spanner database and one dual-region bucket at the bottom — while the backup vault and the dotted game-day loop sit off to the side, because they guard against a different class of failure (silent corruption and untested drift) than the live topology does.

The normal request path. A user resolves the application hostname to a single global anycast IP advertised from every Google point of presence. Anycast routes them to the nearest edge; the Global External Application Load Balancer terminates TLS there. Cloud Armor screens the request (WAF, rate limits, L7 DDoS) before any compute is touched. The load balancer’s backend service is the keystone: it is a single global backend fronted by a serverless NEG per region, pointing at Cloud Run services running in two or more regions — say asia-south1 (Mumbai) and asia-south2 (Delhi), or a primary plus a geographically separate secondary. The load balancer steers each request to the closest healthy region with spare capacity. The Cloud Run service executes stateless business logic, reads secrets from Secret Manager, serves and stores objects (user uploads, generated documents, media) in dual-region Cloud Storage, and — for anything transactional — reads and writes Cloud Spanner multi-region. Spanner presents one logical database with one endpoint while synchronously replicating every commit across regions via Paxos. Asynchronous work flows through Pub/Sub to worker jobs. Telemetry streams to the Cloud Operations suite throughout.

Crucially, on a normal day both regions are doing real work. This is an active/active (or active/active-capable) topology, not active/passive. The secondary region is not a cold building waiting for an emergency; it is serving its share of traffic, running the same image, talking to the same Spanner instance, reading the same GCS buckets. Its readiness is continuously, automatically validated by the fact that it is in production right now.

The failover path — what changes when a region dies. Suppose asia-south1 suffers a regional incident. Here is the elegance: almost nothing in the architecture has to “do” anything, and no human is in the critical path.

• Traffic: The global load balancer’s health checks detect that asia-south1’s Cloud Run backend is unhealthy within seconds and drain it automatically, rerouting all traffic to asia-south2. There is no DNS change, so no TTL to wait out; the anycast IP is unchanged. RTO for the traffic tier is health-check-interval-scale — seconds.
• Transactional data: Spanner transparently elects new Paxos leaders for any splits that were led from the failed region. Committed writes are already durably replicated across regions, so no data is lost (RPO = 0) and the database keeps serving reads and writes through the event. There is no database failover to execute.
• Objects: Dual-region GCS continues serving from the surviving region. Objects written before the incident are already replicated (with a defined freshness guarantee); the bucket’s single global name keeps working unchanged.
• Async: Pub/Sub is a regional-resilient managed service; workers in the surviving region drain the backlog. In-flight messages are retained and redelivered.

So the failover path is: health check fails → LB drains the dead region → traffic and data continue from the survivor, with zero committed-data loss and no DNS or database failover step. The “DR runbook” for the common regional event is, for the most part, “watch the dashboards confirm the automatic behavior worked, and communicate.” The runbook that does exist is for the rarer, harder cases — a logical-corruption event that replicated everywhere, or a full dual-region loss — and those are addressed by backups and point-in-time recovery, not by the live topology.

The diagram, described in words. At the top, globe-distributed users feed a single anycast VIP into one global front end (Cloud Armor + URL map). From there, one global backend service fans out to two (or more) regional Cloud Run boxes drawn side by side — both lit up, both serving, not one greyed-out as “standby.” Below the compute tier sit two horizontally-drawn data primitives that visually straddle both regions to signify single logical resources: a Spanner cylinder spanning the regions (with a small “Paxos quorum, RPO=0” annotation) and a dual-region GCS bucket spanning the regions (annotated “async cross-region replication”). Off to the side, Pub/Sub and worker jobs; underneath everything, a monitoring/observability plane and, in a corner, a backup vault (Spanner backups + PITR, GCS versioning + Backup-and-DR) drawn deliberately outside the live replication path to signal that it protects against corruption, not just region loss. A dotted “game-day” arrow loops back from the monitoring plane to the compute tier, labelled “scheduled regional kill — proves RTO/RPO.”

Component breakdown

Three component choices deserve emphasis because they are where DR designs most often quietly fail.

Spanner is the reason this DR program has a credible RPO=0, and it must be configured for it. The default mental model of DR — “replicate the database to a standby and fail over” — has a non-zero RPO baked in: anything not yet replicated when the primary dies is lost, and the failover itself takes minutes. Spanner multi-region inverts this: a write is not acknowledged to the application until a Paxos quorum across regions has durably committed it, so a committed write cannot be lost to a single-region failure, and there is no failover step because there is no single primary to fail. But this only holds for data in Spanner. The architectural discipline is to put the data whose loss is intolerable — the ledger, orders, balances, identity — in Spanner, and to be deliberate about everything else. (Note also: Spanner’s multi-region replication protects against region loss, not against a bad DELETE that replicates everywhere in milliseconds. That is what PITR and backups are for — a separate axis of protection.)

Dual-region GCS has two RPO modes, and choosing wrong is a silent gap. By default, dual-region (and multi-region) buckets replicate objects asynchronously with no hard time bound — typically minutes, but not contractually fast. For a workload where a freshly uploaded object must survive an immediate regional loss (a payment receipt, a signed contract, a compliance record written seconds before the incident), enable Turbo replication, which carries a 15-minute RPO SLO for dual-region buckets. The cost is higher; the value is a bounded object RPO you can put in a contract. Decide this per bucket based on the data’s recovery requirement — do not assume “it’s multi-region, so it’s safe” covers a tight RPO.

The load balancer is the failover mechanism; DNS is not — and this is the single biggest RTO win. The most common legacy DR design fails traffic over by repointing DNS, which makes your RTO floor the DNS TTL plus resolver caching plus client behavior — frequently minutes to tens of minutes, and unevenly distributed across users. A single global backend service with health-checked regional NEGs removes DNS from the failover path entirely: the anycast IP never changes, and the LB drains an unhealthy region within health-check intervals. Build the LB per-region-and-glue-with-DNS anti-pattern and you have re-imported the very latency you were trying to engineer away.

Implementation guidance

Infrastructure as code, with the DR region defined by the same code as production. Provision everything with Terraform (Google provider); the cardinal rule of this architecture is that there is no separate, hand-built “DR environment” to drift. The two active regions are two instances of the same module, varying only by region parameter. A clean dependency order for terraform apply:

1. Foundation — project(s), VPC, subnets per region, Cloud NAT, firewall rules, org/IAM scaffolding (the Cloud Foundation Toolkit modules give you Google’s security baseline). Put backups/DR vaults in a separate project for blast-radius isolation.
2. Data — the Spanner instance (multi-region config) and database; the dual-region GCS bucket(s) with versioning, turbo replication where required, and retention/lock. Manage Spanner DDL as versioned migrations (wrench, or Liquibase’s Spanner extension) in CI.
3. Compute — Cloud Run services per region (google_cloud_run_v2_service), each with its runtime SA, Direct VPC egress, and min/max instances. Generate the regional services with for_each over the region list so adding a third region is a one-line change.
4. Edge — serverless NEGs per region, the single global backend service binding all NEGs, the URL map, Cloud Armor policy, managed cert, target HTTPS proxy, global forwarding rule.
5. DNS, secrets, backups — Cloud DNS A/AAAA to the global VIP; Secret Manager entries (multi-region replicated, values injected out-of-band); backup schedules and PITR settings.

Keep state in a GCS backend with versioning and locking, and split layers into separate state files so an edge change cannot destroy the Spanner instance. Because the regions come from the same parameterized module, your “DR region” is provably identical to production by construction — the resilience equivalent of test coverage. (Pulumi or Config Connector express the same topology; Deployment Manager is legacy and not recommended for new builds.)

Networking wiring for failover. Cloud Run reaches private resources via Direct VPC egress (preferred over the legacy Serverless VPC Access connector). Internet egress for third-party calls goes through Cloud NAT per region so you present stable, allowlistable IPs from each region — a detail teams forget until a partner’s allowlist blocks the failover region. Spanner and GCS are reached over Google’s private network; wrap them in VPC Service Controls so data cannot be exfiltrated to a project outside your perimeter even with a leaked credential — and design the perimeter to span both active regions/projects so failover doesn’t trip your own security boundary. The only public surface is the global LB VIP; Cloud Run accepts ingress from “internal and Cloud Load Balancing” only, in every region.

Identity wiring. Every Cloud Run service runs as a dedicated least-privilege service account — never the default compute SA — and the same SA bindings exist in every region (least privilege must not be weaker in the DR region, a classic gap). Grant roles/spanner.databaseUser (not admin) on the specific database and roles/secretmanager.secretAccessor on specific secrets. CI deploys via Workload Identity Federation (no JSON keys). End-user auth lives in the app tier (Identity Platform or your OIDC IdP), which is itself a globally-available service, so authentication survives a regional loss alongside the app.

Deploy and release across regions. Build in Cloud Build, push to Artifact Registry (which is multi-region), and roll out with Cloud Deploy to all active regions from one pipeline so images never diverge. Use Cloud Run’s revision-based traffic splitting for canaries. Two release disciplines are non-negotiable in this topology: (1) schema migrations must be backward-compatible / expand-then-contract, because one Spanner database is shared by all regions and revisions — never ship a breaking DDL in lockstep with code; and (2) a bad deploy is itself a disaster scenario — a global rollout of a broken revision can take out both regions at once, defeating the entire DR design. Mitigate with progressive delivery (canary one region first), automated SLO-gated rollback in Cloud Deploy, and the knowledge that PITR/backups — not the live topology — are your recovery from a logical disaster you shipped yourself.

Enterprise considerations

Security and Zero Trust. Resilience and security reinforce each other here: incidents are a favourite moment for attackers, so the defended posture must hold during failover, not just before it. Cloud Armor provides WAF and DDoS at the edge globally; the LB terminates TLS with modern ciphers; Cloud Run accepts traffic only from the LB, in every region. VPC Service Controls draw a perimeter spanning both active regions around Spanner, GCS, and Secret Manager. Data at rest is encrypted, optionally with CMEK in Cloud KMS — and a subtle DR requirement: the KMS key must be available in the failover region, so use a key in a location that covers both regions (or a multi-region key), or the failover region cannot decrypt data. Administrative surfaces sit behind Identity-Aware Proxy with context-aware access. The Zero-Trust stance — never trust the network, always verify identity — applies identically in primary and secondary.

Cost optimization — the honest part of DR economics. Active/active resilience is not free, and pretending otherwise is how DR programs lose credibility. The premium has three sources, each with a lever:

• Spanner multi-region is the floor cost — meaningfully more than a regional database. Lever: start at a small processing-unit count with the autoscaler on, buy committed-use discounts once the baseline is known, and reserve multi-region Spanner for the data that genuinely needs RPO=0 — keep lower-tier data in cheaper stores.
• Idle failover headroom. Because traffic is active/active, you are not paying for a fully-idle standby — but you are sizing each region to absorb the other’s load on failover. Lever: min-instances set to the failover surge, max-instances as the ceiling; Cloud Run scales the rest on demand, so you pay for headroom, not a duplicate idle fleet.
• Inter-region egress and turbo replication. Cross-region replication and chatty traffic cost money. Lever: keep service-to-service traffic in-region; enable Turbo replication only on buckets whose RPO truly requires it.

The defensible framing for the business: this is the cost of a proven RPO=0 / seconds-RTO posture for Tier-1 systems, and it is cheaper than the legacy alternative (a duplicate idle environment plus a team to babysit replication) once you account for the avoided drift, the avoided manual failover risk, and the value of the second region actually serving traffic.

Scalability. Each tier scales independently and horizontally, which is also what makes failover absorbable. The global LB is effectively unbounded. Cloud Run scales out per region; the headroom that handles a traffic spike is the same headroom that handles a sibling region’s failover load. Spanner scales by adding processing units with no downtime and no re-sharding — so growth never reopens the DR design. Read-heavy paths can use stale reads (e.g. 10–15 s) to serve from the nearest replica, raising read throughput and, incidentally, reducing cross-region leader round-trips.

Reliability and DR — the RTO/RPO tiers, stated explicitly. This is the heart of the article. Do not assign one RTO/RPO to the whole system; assign them per tier, match each tier to a pattern, and prove them. A workable tiering:

The discipline is twofold. First, be honest about which tier each workload is in — over-classifying everything as Tier-0 explodes cost; under-classifying the ledger is negligence. Second, prove the targets:

• Spanner multi-region carries a 99.999% availability SLA with RPO=0 and effectively automatic regional recovery — but you still enable backups + PITR because availability ≠ corruption-proof.
• For corruption/deletion/ransomware (which replicate everywhere instantly and so are immune to the live topology), rely on PITR, GCS Object Versioning + soft delete + retention lock, and the Backup and DR Service with immutable vaults in an isolated project.
• Run game-days. Schedule a recurring exercise — quarterly at minimum, monthly for the most critical — that actually drains a region (in staging, then with confidence in production with a guardrail) and confirms: traffic rerouted within the health-check window, Spanner lost zero writes, GCS served from the survivor, dashboards told the true story, and the on-call learned of it from monitoring, not customers. A DR target that has not been tested in production-like conditions is a hypothesis, not a guarantee. Game-days are what convert the binder into evidence — for the board, for the auditor, and for the team’s own confidence at 3 a.m.

Observability — you cannot recover from what you cannot see. Instrument the full path. The LB exports per-region, per-backend request/latency/error metrics — the primary signal that a region is degrading and draining. Spanner exposes leader distribution, replication lag, CPU, and lock/abort stats. GCS replication metrics show object-replication freshness against your RPO. Define SLOs (e.g. 99.99% of Tier-1 requests succeed; checkout p95 < 300 ms served from the user’s region) and alert on burn rate, not raw thresholds. Run uptime checks from multiple geographies so you detect a regional problem from the outside the way a customer would. And critically, build the failover dashboard you will stare at during an incident before the incident — a single view showing per-region health, Spanner leader location, replication freshness, and error rates — so a game-day (and the real thing) is read from one screen.

Governance. Enforce the perimeter with Organization Policy — restrict resource locations to the approved regions for data residency (a hard requirement that often constrains which DR region you may use), block public IPs, require CMEK where mandated. Use folders and projects to separate prod/non-prod and to isolate the backup vault’s blast radius. Centralize findings in Security Command Center. Most importantly for DR specifically, govern the RTO/RPO commitments themselves: maintain a service catalog mapping each workload to its tier and targets, store game-day reports as the evidence trail, and gate infra changes through PR review with policy-as-code so nobody can, say, downgrade the Spanner config or disable versioning without scrutiny.

Reference enterprise example

Vahana Logistics is a fictional B2B freight and last-mile platform headquartered in Pune. It runs the dispatch, tracking, proof-of-delivery, and settlement systems that 4,000 enterprise shippers and 90,000 drivers depend on every day. By Series D it had a problem the board could no longer accept: the entire platform — application and PostgreSQL database — lived in asia-south1 (Mumbai). A four-hour regional networking degradation one monsoon afternoon froze dispatch nationwide; drivers couldn’t get assignments, shippers couldn’t track loads, and because the proof-of-delivery uploads were single-region, the team genuinely could not confirm whether the last 40 minutes of delivery confirmations had been persisted. The estimated commercial impact, including SLA penalties to enterprise shippers, was around ₹3.4 crore (~$410k), and two large customers invoked contractual DR clauses demanding documented RTO/RPO before renewal. Engineering (a team of 31) was given a mandate: a proven DR posture for the money-and-trust-bearing systems, on a defensible budget.

What they built. They tiered their workloads first, refusing to gold-plate everything. The settlement ledger, order/load capture, and driver identity became Tier-1 and moved onto a Cloud Spanner multi-region instance (asia1-spanning config), giving RPO=0 with no replication pipeline to babysit. Proof-of-delivery photos, signed PODs, and invoices moved to a dual-region Cloud Storage bucket (asia1 = Mumbai + Delhi) with Turbo replication for a 15-minute object RPO and Object Versioning + retention lock so a POD can be neither lost to a region failure nor tampered with. The application moved to Cloud Run in both asia-south1 and asia-south2, active/active behind one Global External Application Load Balancer on a single anycast IP, with min-instances in each region sized to absorb the other region’s full load. Live tracking and the shipper dashboard were classified Tier-2 (async GCS, Cloud SQL read-replica for some reporting). Internal admin tools stayed Tier-3, protected by backups and restore-on-demand. The backup vault — Spanner PITR plus GCS versioning plus Backup-and-DR — lives in a separate project in a different region.

Decisions and trade-offs they made.

• They explicitly rejected an active/passive cold-standby design (the option their old vendor pitched) because it was both expensive (idle duplicate) and untrustworthy (would drift, never tested). Active/active cost more in Spanner but eliminated the standby fleet and made the second region’s readiness self-proving.
• They sized Spanner conservatively at launch with the autoscaler on and bought a committed-use discount after two months of baseline — turning their single largest new line item from a fear into a planned, discounted cost.
• They enabled Turbo replication only on the POD/invoice bucket, not on bulk telematics history, because only the former needed a tight, contractual RPO — saving meaningfully on replication cost.
• They scheduled a monthly production game-day that drains asia-south1 for ten minutes behind a feature-flag guardrail, with a one-screen failover dashboard and an automatic abort. The first game-day found a real gap: the failover region’s Cloud NAT egress IP wasn’t on a payment partner’s allowlist, so settlement calls would have failed on failover. They fixed it in a drill instead of in a crisis — exactly the point.

The outcome. Eight months after cutover, GCP took a genuine regional incident in asia-south1 during business hours. The global LB drained Mumbai within the health-check window; Spanner re-elected leaders and lost zero ledger writes; the POD bucket served every photo from Delhi; drivers kept getting assignments and shippers kept tracking loads. Dispatch never stopped. Settlements reconciled to the rupee. PODs: all present. The on-call engineer learned of the regional event from the monitoring channel and the (already-green) failover dashboard, not from a flood of customer escalations, and spent the incident communicating rather than recovering. Vahana now hands enterprise customers a one-page DR attestation — Tier-1 RTO seconds / RPO 0, proven monthly, with game-day report IDs — which directly unblocked two renewals worth more than the entire multi-region premium. The 31-person team runs this without a dedicated DBA or a DR-operations function, because the topology is the DR plan and the game-days are the proof.

When to use it

Use this architecture when a regional outage of the workload would be a material business, contractual, or regulatory event — i.e. the system takes money, holds the ledger, captures orders, manages identity, or carries data whose loss you must measure in zero. It is the right answer when customers or regulators demand documented and tested RTO/RPO, when “we have a DR plan” must become “here is evidence it works,” and when you have outgrown the honesty problem of a single-region critical system. It fits fintech, logistics, healthcare, regulated SaaS, marketplaces, and any platform whose downtime is somebody else’s incident. Because the data tier is intrinsically multi-region and the compute scales to demand, a smaller company can adopt the shape for its one Tier-1 workload and grow the tiered portfolio later — which is exactly how a reference architecture should behave.

Be honest about the trade-offs. The dominant one is Spanner’s cost and model: multi-region Spanner has a real monthly floor a hobby project cannot justify, and it has its own idioms (key design, interleaving, the absence of some PostgreSQL niceties — though the PostgreSQL interface narrows the gap). The second is that active/active is not free: you pay for failover headroom and cross-region replication, and the architecture asks for the operational maturity to run game-days. The third is a subtle one teams miss: multi-region replication is not a backup. A logical-corruption or malicious-delete event replicates everywhere in milliseconds; the live topology offers no protection from it. PITR, versioning, and immutable backup vaults are a separate, mandatory axis — if you implement only the live replication and call it DR, you have a sophisticated way to lose your data instantly in every region at once.

Anti-patterns to avoid. Do not fail over with DNS — a global backend service removes DNS TTL from your RTO; reintroducing it caps your recovery at minutes. Do not keep a cold, never-exercised standby — it will have drifted and will fail when called; active/active (or at least a continuously-tested warm standby) is the resilient choice. Do not classify every workload as Tier-0 — it explodes cost and dilutes focus; tier honestly. Do not assume “it’s multi-region” means a tight RPO — default GCS replication is asynchronous and unbounded; enable Turbo where the RPO is contractual. Do not forget that a bad global deploy can take out both regions at once — progressive, SLO-gated delivery is part of the DR design, not separate from it. And do not skip game-days: an untested DR target is a hypothesis, and the place to discover the allowlist gap, the missing KMS key in the failover region, or the under-provisioned headroom is a drill, not a disaster.

Alternatives at the edges. If your data is genuinely single-region and you can tolerate an RPO of seconds and an RTO of minutes, a regional Cloud SQL with a cross-region read replica and a documented, tested promote procedure is far cheaper and is the correct Tier-2 answer for many systems — use it deliberately rather than defaulting to Spanner for everything. If the workload is read-mostly with no transactional writes, a CDN over multi-region GCS (or AlloyDB for Postgres-heavy hybrid analytical-transactional needs) may be the better data tier. For VM-based or third-party applications you cannot re-platform onto Cloud Run/Spanner, the Backup and DR Service plus cross-region disk/instance recovery gives a slower (hours-RTO) but legitimate DR path — and is the honest choice for legacy that doesn’t fit the cloud-native shape. The constant across all of these: pick the pattern that meets the tier’s RTO/RPO at the lowest defensible cost, and then prove it on a schedule — because in disaster recovery, the only number that counts is the one you have actually demonstrated under failure.

Going deeper

This section is for the reader who has to build and defend the design. It trades the overview’s altitude for the mechanisms, edge cases, and quotas that decide whether the architecture holds up on the real bad day.

How Spanner delivers RPO=0 without a “failover” (TrueTime, Paxos, and splits). A Spanner database is not one thing in one place; it is carved into splits (contiguous key ranges), and each split is replicated as a Paxos group across the regions of your instance config. Within each group, one replica is the leader; writes go to the leader, which proposes the change and does not acknowledge the client until a quorum (a majority of voting replicas) has durably logged it. In a multi-region config such as nam3, that quorum spans regions — so a committed write physically exists in more than one region before your application is told “done.” That is the mechanical reason region loss cannot lose a committed write: the survivor already has it. When the leader’s region dies, the remaining replicas run a Paxos election and a new leader emerges in seconds — there is no “promote the standby” step, because there was never a single primary to promote. TrueTime, Google’s globally-synchronised clock (backed by atomic clocks and GPS in every data centre), is what lets Spanner order these distributed commits into one consistent timeline without a chatty coordinator; it is why you get external consistency across regions at all. Two consequences worth internalising: (1) writes pay a cross-region round-trip in latency — multi-region Spanner is slower per write than a single-region database, and that is the price of RPO=0; (2) read-only and witness replicas let you add durability/quorum members without paying full read-write replica cost, and bounded-staleness reads (e.g. a 10–15 s staleness) can be served from the nearest replica without a leader round-trip — the main lever for read latency and cross-region cost.

A subtlety the pretty diagrams skip: Spanner multi-region configs are a fixed menu. You do not get to pick any two regions; Google curates a set of configs (nam3, nam6, nam-eur-asia1, eur3, eur5, eur6, and others), each with a defined placement of read-write, read-only, and witness replicas. This matters for DR because your data-residency requirement (often enforced by Org Policy) may not line up with any config that also gives you the regional separation you want — a real tension you resolve at design time, sometimes by accepting a nearby region, sometimes by dropping to a regional Spanner or a Cloud SQL cross-region pattern for that workload. Run gcloud spanner instance-configs list and read the replica placement before you promise a config to anyone.

GCS replication, precisely — and why “multi-region” is not automatically a tight RPO. A dual-region or multi-region bucket has one global name and serves reads strongly-consistently from either region, but writes replicate asynchronously. In the default mode there is no contractual time bound on that replication — usually minutes, occasionally longer under load. Turbo replication (dual-region buckets only) upgrades this to a 15-minute RPO SLO: Google commits that objects replicate within 15 minutes, and exposes replication-time metrics so you can prove it. If a payment receipt is written two seconds before its origin region fails, default replication may not have copied it yet — turbo bounds that risk. You choose per bucket, because turbo costs more and most buckets do not need it. Independently of region loss, protect objects on the corruption axis with Object Versioning, soft delete, and retention lock — replication faithfully copies a malicious delete to both regions in milliseconds, so only versioning/immutability saves you there. One more operational trap: a bucket’s location is immutable. You cannot convert a single-region bucket to dual-region in place; you create a new bucket and migrate with gcloud storage rsync, so decide location up front.

The load balancer failover, with the numbers — and the serverless-NEG caveat. For instance-group (MIG) or zonal-NEG backends, the global external ALB attaches a health check; worst-case detection is roughly check-interval × unhealthy-threshold (e.g. 5 s × 3 ≈ 15 s), after which the backend drains and traffic overflows to the next region — and here capacity matters: with balancing mode RATE and a maxRatePerEndpoint, the LB overflows to another region when a region is full, not only when it is down. For serverless-NEG (Cloud Run) backends the model is different and often misunderstood: serverless NEGs do not take a classic backend health check and do not expose balancing-mode capacity; the LB routes to the nearest region and relies on Cloud Run’s own regional availability, with outlier detection (eject a region after N consecutive 5xx) and retries doing the ejection work. The practical implication for an active/active Cloud Run design: your failover “capacity awareness” comes from per-region min-instances headroom sized for the surge plus outlier detection — not from LB balancing modes. If you need true LB-level capacity balancing, that is an argument for regional MIGs or GKE backends. Either way, the anycast VIP and Google’s Maglev load-balancing mean DNS is never in the failover path — which is the whole point of the next paragraph.

Why DNS failover is slow, quantified. A DNS-based failover repoints an A record from region A to region B. Your RTO floor then becomes: record TTL (say 60–300 s) + resolver caching (many resolvers round up or ignore very low TTLs) + negative caching + client stickiness (browsers, connection pools, and mobile apps that hold connections open). In practice this lands at minutes to tens of minutes, unevenly across your users — some fail over quickly, some hang for ages. Cloud DNS does offer routing policies (geo, weighted, failover), and they are useful for coarse traffic steering, but they are the wrong tool for fast regional failover when a global anycast LB can drain a region in seconds without touching DNS at all. Keep your DNS TTLs sane, and keep DNS out of your critical-path RTO.

The corruption axis (logical DR) in mechanism. Replication defends against losing infrastructure; it is powerless against losing correctness. A bad migration, a buggy batch job, or a malicious actor issues a DELETE or an encrypting overwrite, and every synchronous replica dutifully applies it everywhere in milliseconds. The defenses are a separate axis, and each has a mechanism: Spanner PITR keeps old versions for a configurable version_retention_period (up to 7 days) so you can read or restore the database as of a timestamp before the damage; Cloud SQL PITR replays transaction logs from a base backup to any second within the retention window; GCS Object Versioning + soft delete keep prior/deleted object generations, and retention lock / bucket lock make them immutable so even an admin (or ransomware wielding admin creds) cannot purge them; and the Backup and DR Service stores application-consistent backups in backup vaults with enforced immutability, ideally in a separate project and region so the blast radius of a compromised prod project does not reach the backups. The design rule: the backup path must be reachable from prod but not deletable by prod’s day-to-day identities.

Resilience engineering: making failure boring. DR handles the region-scale catastrophe; resilience engineering handles the thousand smaller failures so they never escalate into one. Three disciplines matter. Graceful degradation: design each service to shed non-essential work under stress rather than fall over — circuit breakers around slow dependencies, load shedding at the edge (Cloud Armor rate limits), serving bounded-stale reads when the Spanner leader is briefly unreachable, feature flags that drop non-critical features, and a read-only mode the app can enter when writes are impossible. A checkout that still takes orders while “recommended for you” is disabled is resilient; one that white-screens because the recommendation service is slow is not. Chaos engineering: GCP has no managed fault-injection service equivalent to AWS FIS, so you build it — form a steady-state hypothesis (“checkout error rate stays < 0.1% and Spanner loses zero writes”), inject a real fault (remove a region’s backend, deploy a failing revision, add latency, revoke a dependency), and verify the hypothesis holds; tools such as Chaos Toolkit (with its chaosgcp extension) or Gremlin script this. Game-days are chaos engineering with an audience and a calendar: a scheduled, blast-radius-controlled exercise that drains a region on purpose, in staging first and then in production behind a guardrail, precisely to convert your RTO/RPO targets from hypotheses into evidence. The cultural payoff matters as much as the technical one — the team that has drained a region twenty times is calm when GCP drains it for them.

Quotas, limits, and the caveats that bite on the day. A few that regularly surface in real DR events. Capacity quota in the surviving region: if asia-south1 fails, asia-south2 must serve both regions’ load, and Cloud Run max-instances, CPU, or in-use IP quotas there can throttle you exactly when you need headroom — raise and verify quotas in every active region, not just the primary. Committed-use discounts are region- and resource-scoped, so a CUD bought for the primary does not discount the failover region’s surge. Not every service is in every region — asia-south2 (Delhi) has a narrower service catalog than asia-south1 (Mumbai); confirm every dependency exists in your DR region before you commit to it. CMEK keys are location-bound: if you encrypt data with a Cloud KMS key in a single region, the failover region cannot decrypt it — use a multi-region key or a key in a location that covers both regions, or your beautifully-replicated data is unreadable after failover. And VPC Service Controls perimeters must be drawn to span both active regions/projects, or your own security boundary blocks the failover path. Each of these is a “found it in a game-day, not a disaster” item.

Practice challenges

Work these top to bottom — the first two are conceptual, the middle two are single-service gcloud, and the last two are production-grade. Try each before opening the solution.

1. Tier three workloads (beginner). You run three systems: (a) a payments ledger, (b) a product catalog read by the storefront, © an internal weekly reporting dashboard. Assign each an RTO/RPO tier and name the DR pattern that fits.

<details> <summary>Solution</summary>

• (a) Payments ledger → Tier-0. RTO seconds, RPO 0. Pattern: hot / active-active on Spanner multi-region + dual-region GCS + active/active Cloud Run.
• (b) Product catalog → Tier-2. RTO minutes, RPO seconds. Pattern: warm standby / pilot light — a Cloud SQL (or AlloyDB) cross-region read replica with a tested promote-replica.
• © Reporting dashboard → Tier-3. RTO hours, RPO hours. Pattern: backup & restore — restore on demand into the surviving region; no live standby.

Why: tiering honestly is the first DR skill — over-classifying the catalog as Tier-0 wastes money, and under-classifying the ledger is negligence. </details>

2. Read RTO/RPO off a timeline (beginner). A system uses backup & restore. Backups run every 6 hours; the last successful backup completed at 02:00. A region failure hits at 07:30, and the team finishes restoring into another region at 09:00. What RPO did this incident realise, and what RTO?

<details> <summary>Solution</summary>

• RPO realised = 07:30 − 02:00 = 5 h 30 m of data lost (the writes since the last backup). The worst case for this schedule is ~6 h — the full gap between backups.
• RTO realised = 09:00 − 07:30 = 1 h 30 m of downtime (failure → service restored).

Why: RPO is measured backwards from the failure to the last recoverable copy; RTO is measured forwards to restoration. Tighten RPO by backing up more often (or replicating); tighten RTO by pre-staging the recovery. </details>

3. Dual-region bucket with turbo + versioning (intermediate). Create a Cloud Storage bucket that survives a regional loss with a contractual object RPO, spanning asia-south1 (Mumbai) and asia-south2 (Delhi), and is protected against accidental deletes. Then state what RPO you just bought.

<details> <summary>Solution</summary>

# Configurable dual-region (Mumbai + Delhi) with turbo replication + hardened access
gcloud storage buckets create gs://acme-pods \
  --placement=asia-south1,asia-south2 \
  --recovery-point-objective=ASYNC_TURBO \
  --uniform-bucket-level-access \
  --public-access-prevention

# Versioning defends the corruption/delete axis (replication cannot)
gcloud storage buckets update gs://acme-pods --versioning

You bought a 15-minute object RPO SLO across Mumbai and Delhi. Turbo requires a dual-region bucket (predefined, or — as here — configurable with --placement). Versioning is a separate axis: it protects against overwrite/delete, which replication faithfully copies to both regions.

Why: --recovery-point-objective=ASYNC_TURBO is the only thing that turns “multi-region, eventually” into a bounded, provable RPO — and a bucket’s location is immutable, so you must set this at creation time. </details>

4. Cloud SQL cross-region read replica + promote (intermediate). Your Tier-2 catalog runs on Cloud SQL in asia-south1. Add a cross-region read replica in asia-south2 for warm-standby DR, then show the failover (promote) step.

<details> <summary>Solution</summary>

# Create a cross-region read replica (async) in Delhi, itself zonally HA
gcloud sql instances create catalog-replica-del \
  --master-instance-name=catalog-primary-mum \
  --region=asia-south2 \
  --availability-type=REGIONAL

# On a real regional loss, promote the replica to a standalone primary:
gcloud sql instances promote-replica catalog-replica-del

Why: the replica gives seconds-of-RPO async replication for a fraction of Spanner’s cost; promote-replica is the manual, one-way failover step (RTO minutes), so it must be game-day’d in staging — after promotion the old primary is no longer its master and the relationship cannot be un-done. </details>

5. Spanner multi-region with PITR + drop protection (advanced). Stand up a Tier-0 Spanner database that delivers RPO=0 across regions and protects against a bad DELETE. Include point-in-time recovery and guard against an accidental database drop.

<details> <summary>Solution</summary>

# Multi-region instance (nam3 = US multi-region; pick a config that covers YOUR regions)
gcloud spanner instances create ledger-mr \
  --config=nam3 \
  --description="Tier-0 ledger" \
  --processing-units=1000

# Database with 7-day PITR (the corruption axis) + drop protection
gcloud spanner databases create ledger \
  --instance=ledger-mr \
  --version-retention-period=7d \
  --enable-drop-protection

# Confirm the config really is multi-region before you promise RPO=0:
gcloud spanner instance-configs describe nam3 --format="value(replicas)"

Why: a multi-region config gives RPO=0 (Paxos quorum + automatic leader failover — no promote step), while --version-retention-period=7d adds PITR for the corruption axis that replication can’t cover, and drop-protection stops a fat-fingered databases delete. Configs are a fixed menu — verify one covers your required regions (and residency) before committing. </details>

6. Design and run a game-day (advanced). Design a production game-day that drains asia-south1 from an active/active Cloud Run + global-LB stack and proves Tier-1 RTO/RPO. Give the steady-state hypothesis, the drain action, the assertions, and the abort/rollback.

<details> <summary>Solution</summary>

Drain by removing the region’s serverless NEG from the global backend service — reversible, LB-level, and it touches neither Cloud Run nor DNS:

# DRAIN: stop the LB routing to Mumbai
gcloud compute backend-services remove-backend app-backend --global \
  --network-endpoint-group=run-neg-asia-south1 \
  --network-endpoint-group-region=asia-south1

# ...observe for ~10 minutes...

# ABORT / RESTORE: put Mumbai back
gcloud compute backend-services add-backend app-backend --global \
  --network-endpoint-group=run-neg-asia-south1 \
  --network-endpoint-group-region=asia-south1

Scripted as a Chaos Toolkit experiment, the guardrails become explicit and the rollback is automatic:

version: "1.0.0"
title: "Regional drain: asia-south1"
description: "Prove Tier-1 RTO/RPO when Mumbai leaves the load balancer"
steady-state-hypothesis:
  title: "Checkout SLO holds and Spanner loses no writes"
  probes:
    - name: checkout-error-rate-under-slo
      type: probe
      tolerance: 0.001                 # abort if > 0.1% errors
      provider:
        type: http
        url: "https://api.example.com/internal/checkout-error-rate"
    - name: spanner-writes-committing
      type: probe
      tolerance: true
      provider:
        type: python
        module: chaosgcp.spanner.probes
        func: instance_healthy
method:
  - type: action
    name: remove-mumbai-backend
    pauses:
      after: 600
    provider:
      type: process
      path: gcloud
      arguments: >-
        compute backend-services remove-backend app-backend --global
        --network-endpoint-group=run-neg-asia-south1
        --network-endpoint-group-region=asia-south1
rollbacks:
  - type: action
    name: restore-mumbai-backend
    provider:
      type: process
      path: gcloud
      arguments: >-
        compute backend-services add-backend app-backend --global
        --network-endpoint-group=run-neg-asia-south1
        --network-endpoint-group-region=asia-south1

Assert: the LB shifts ~100% of traffic to asia-south2 within seconds; checkout p95 and error rate stay inside SLO; Spanner shows zero unavailability and no lost writes; the POD bucket serves from Delhi; the on-call is paged by monitoring, not by customers. Abort the moment the steady-state hypothesis breaks — Chaos Toolkit runs the rollbacks automatically.

Why: a game-day converts an RTO/RPO target into evidence; remove-backend/add-backend at the LB is a clean, reversible drain, and the steady-state hypothesis is what makes it safe to run in production. This is exactly the drill that surfaces the missing NAT allowlist entry or the un-covered KMS key before a real region loss does. </details>

Common beginner mistakes

“Multi-region replication is our backup.” It is not. Replication copies every write — including a bad DELETE or an encrypting overwrite — to every region in milliseconds. Right model: two independent axes. Replication protects against infrastructure loss (a region dies); PITR, Object Versioning, and immutable backup vaults protect against logical loss (correctness dies). You need both, and one never substitutes for the other.

“We’ll just fail over with DNS.” DNS failover makes your RTO floor the TTL plus resolver caching plus client stickiness — minutes to tens of minutes, unevenly across users. Right model: a single global backend service on an anycast VIP drains an unhealthy/erroring region in seconds with no DNS change. DNS steers coarse traffic; it does not do fast failover.

“Our cold standby is ready.” An environment nobody has booted in months has drifted from prod and is a hypothesis, not a recovery plan. Right model: active/active (the DR region serves real traffic and therefore cannot drift) or, at minimum, a warm standby exercised by regular game-days. Untested DR is not DR.

“Make everything Tier-0.” Assigning every workload RTO-seconds / RPO-0 explodes cost (multi-region Spanner for a nightly report) and dilutes focus. Right model: tier honestly, and spend the steep part of the cost-vs-RTO curve only where the business value earns it.

“It’s a multi-region bucket, so the data is safe instantly.” Default dual/multi-region replication is asynchronous with no time bound — an object written seconds before a region loss may not have copied yet. Right model: enable Turbo replication for a 15-minute RPO SLO on the buckets whose RPO is contractual; accept default async where minutes are fine.

“RTO and RPO are basically the same number.” They measure different things and move independently. Right model: RTO = how long you are down (time-to-restore); RPO = how much data you lose (time-of-data). A cache can run RTO-seconds / RPO-hours; a ledger demands both at zero.

“The other region has plenty of spare capacity.” On failover the survivor must carry both regions’ load, and Cloud Run max-instances, CPU quota, or IP quota there can throttle you at the worst moment. Right model: size each region’s min-instances for the failover surge and raise/verify quotas in every active region — then prove it in a game-day.

“A bad deploy isn’t a disaster.” A global rollout of a broken revision can take down both regions at once — the one failure a multi-region topology does not stop. Right model: progressive, SLO-gated delivery (canary one region, automated rollback) is part of the DR design, and PITR/backups are your recovery from a logical disaster you shipped yourself.

Glossary

• Active/active — A topology where two or more regions serve real traffic at the same time; losing one is absorbed by the others, and each region’s readiness is continuously proven because it is always in production.
• Active/passive (standby) — One region serves while a second waits (cold/warm/hot) to take over. Cheaper, but the standby can drift and is only ever as good as your last test.
• Anycast — Advertising one IP address from many locations so users are routed to the nearest one. The global LB VIP is anycast, which is why regional failover needs no DNS change.
• Backup & restore — The simplest DR pattern: keep backups elsewhere and restore on demand. Cheapest; RTO and RPO measured in hours.
• Blast radius — How far the damage from one failure (or one compromised identity) can spread. You shrink it with separate projects, separate regions, and least-privilege.
• Burn-rate alert — An SLO alert that fires on how fast you are consuming your error budget, rather than on a raw metric threshold — fewer false pages, faster real ones.
• Chaos engineering — Deliberately injecting failure to verify a system behaves as hypothesised; the practice behind game-days.
• CMEK (customer-managed encryption key) — Encryption with a key you control in Cloud KMS. For DR the key’s location must cover the failover region, or that region cannot decrypt.
• Cloud Armor — GCP’s edge WAF and DDoS service, applied at the global load balancer, independent of which region serves.
• Cold / warm / hot standby — Increasing readiness of a standby region: cold = built but off; warm = running but scaled down; hot = fully running (effectively active/active).
• Dual-region (GCS) — A bucket whose data lives in exactly two chosen regions under one global name; the only bucket type that supports Turbo replication.
• Failover — Shifting work from a failed component or region to a healthy one. The best failover (LB health/outlier ejection) needs no human in the loop.
• Game-day — A scheduled, guard-railed exercise that induces a real failure (e.g. draining a region) to prove RTO/RPO and train the team.
• Global External Application Load Balancer — GCP’s anycast L7 load balancer that fronts multiple regions behind one IP and drains unhealthy ones automatically.
• Graceful degradation — Shedding non-essential features under stress (read-only mode, circuit breakers, stale reads) instead of failing completely.
• High availability (HA) — Surviving common failures (a zone, an instance) with little or no impact — usually via multi-zone. Distinct from DR, which targets regional loss.
• Leader / replica (Spanner) — Within each Paxos group one replica is the leader that coordinates writes; the others are voting, read-only, or witness replicas.
• Multi-region — Spanning regions to survive a whole-region loss; the usual meaning of “DR.” Costs more than multi-zone.
• Multi-zone (regional) — Spanning zones within one region to survive a zone loss; cheap, low-latency, and your production default.
• Outlier detection — An LB feature that ejects a backend region after N consecutive errors; the effective failover mechanism for serverless-NEG (Cloud Run) backends.
• Paxos — The consensus algorithm Spanner uses so that a write is durable across a quorum of replicas before it is acknowledged.
• Pilot light — A DR pattern where only a minimal core (e.g. a running database replica) stays warm; compute is spun up on failover. Mid-cost, tens-of-minutes RTO.
• PITR (point-in-time recovery) — Restoring data to a chosen past timestamp (before corruption). Spanner keeps up to 7 days; Cloud SQL replays transaction logs.
• Quorum — A majority of voting replicas. Spanner needs a quorum to commit, which is why a single region’s loss cannot lose a committed write.
• Region — A group of independent zones in one geography (e.g. asia-south1, Mumbai); close enough for synchronous replication, far enough to fail independently from other regions.
• RPO (recovery point objective) — How much data, measured in time, you can afford to lose (last safe copy → failure).
• RTO (recovery time objective) — How long you can be down (failure → service restored).
• Serverless NEG — A network endpoint group that points the load balancer at a serverless backend (Cloud Run/Functions/App Engine) in a region.
• SLO (service level objective) — A target for a reliability metric (e.g. 99.99% of requests succeed) that DR must uphold and game-days verify.
• TrueTime — Google’s globally-synchronised clock (atomic clocks + GPS) that lets Spanner order distributed commits into one consistent timeline.
• Turbo replication — A dual-region GCS mode with a 15-minute RPO SLO for object replication.
• VPC Service Controls — A security perimeter around GCP data services to prevent exfiltration; for DR it must span both active regions/projects.
• Zone — A single failure domain (one or a few data centres) inside a region.