Education Platform Scaling for a National Exam Day on GCP

In a nutshell

Imagine a national concert where 60,000 fans all arrive in the same five minutes because the doors open at exactly 7:00 pm. If you hire staff and open gates only after the queue forms, you get a dangerous crush. But you know the doors open at 7:00, so you do the opposite: every gate open, every steward already in position, and tickets people scan themselves — all before the first fan walks up. That is this entire lesson. A national exam is that concert, except 1.4 million students press “Start” inside the same ninety seconds, and the “crush” is a website showing a spinner while someone’s university place slips away.

The hard part is not how much traffic — plenty of sites serve more over a day — it is the shape of it: a near-vertical wall from almost nothing to peak, at a second you can predict on the calendar. That predictability is the gift the whole design is built around. Because you know the exact moment, you pre-warm (bring capacity up before the spike instead of reacting to it), cache the unchanging parts at the edge so they never touch your servers, stand up a layered shield against attackers who love a high-stakes moment, and make sure the one thing that must never fail — saving a candidate’s answer — is written somewhere durable the instant it happens.

Everything below is how you compose Google Cloud’s building blocks — GKE for compute, Cloud Spanner for never-lose-a-write storage, Memorystore for a fast in-memory buffer, a global load balancer with Cloud Armor and a CDN at the edge, and Pub/Sub for coordination — to survive that ninety seconds. The same playbook scales down to a flash sale, a concert ticket on-sale, results day, or a tax-filing deadline: any spike you can see coming, you can pre-stage for.

Level: Advanced · Time: ~38 min

Prerequisites. You’ll get the most from this if you’re comfortable with the basics of cloud compute (virtual machines and containers), what a load balancer and a database do, and the idea of horizontal scaling (running more copies of a service). It helps to know roughly what Kubernetes and GKE are and what “autoscaling” means — but every term is defined in the Glossary at the end, so a determined beginner can follow the whole thing.

After this lesson you’ll be able to:

• Explain why ordinary reactive autoscaling fails for a synchronized cold-start spike, and what pre-warming, min-instances, and scheduled autoscaling do instead.
• Split a workload into static / dynamic / admin traffic classes and scale each on its own curve.
• Design a layered edge defense — global load balancer + Cloud CDN + Cloud Armor — that absorbs a DDoS and throttles a scripted cheater without blocking real candidates.
• Choose deliberately between Cloud SQL (read replicas + connection pooling) and Cloud Spanner for a write-heavy, must-be-consistent workload.
• Use a cache (Memorystore) and queue-based load leveling (Pub/Sub / Cloud Tasks) to flatten a burst.
• Write a graceful-degradation ladder, a load test that reproduces the real curve, and a game-day runbook with a T-minus timeline.

---

A national testing authority that runs a country’s university-entrance exam gives its platform team a deadline with no slack in it: on a single Saturday in May, 1.4 million candidates will press “Start” within the same ninety-second window, because the exam is timed identically nationwide and the proctors release it on a synchronized clock. The board chair has lived through the alternative — two years ago the legacy provider’s load balancer melted under the thundering herd, candidates stared at spinners for eleven minutes, the press called it a “rigged exam,” and the resulting court case forced a full re-sit at a cost that dwarfs a decade of cloud bills. The new mandate is unambiguous: the platform must absorb a cold-start spike from near-zero to peak in under two minutes, stay up through a hostile internet, and never lose a single submitted answer — because a lost answer is not a glitch, it is a candidate’s university place. This is the reference architecture for building that platform on Google Cloud, where the engineering problem is not steady-state throughput but the brutal, unforgiving shape of the load curve.

The pressures here are unlike a normal SaaS app, and naming them sets up every decision that follows. The spike is synchronous and predictable to the minute — you know the exact Saturday and the exact second, which is a gift, because it means you can pre-warm instead of react. The load is bursty, not sustained — peak concurrency lasts three hours and is near-zero the rest of the quarter, so paying for peak capacity year-round is indefensible. Correctness is sacred — answer autosave must be durable and globally consistent, because a candidate who answers in one region and reconnects through another must see their work. And the event is a magnet for attack — a DDoS during the entrance exam, or a single cheater scripting the API, is worth real money to someone, so the platform is hostile-internet-facing by definition. GCP gives you the primitives — GKE, Cloud Spanner, Memorystore, Cloud Armor — but the architecture is in how you compose them for this curve.

Why the obvious approaches fail

Three shortcuts will be proposed in the first planning meeting, and each fails in a way worth naming before someone builds it.

“Just turn on the Horizontal Pod Autoscaler and let it react.” Reactive autoscaling watches CPU, notices the spike, and then asks for pods — but GKE has to schedule pods onto nodes, and if the nodes do not exist, the Cluster Autoscaler must call Compute Engine to provision VMs, pull container images, and pass readiness probes. That control loop takes minutes you do not have when the entire country starts in ninety seconds. By the time capacity arrives, the spike is over and the candidates have already seen the spinner. Reactive scaling is a strategy for gradual load; it is a guaranteed outage for a synchronized cold start.

“Run it on a single regional SQL database.” A national exam means national users, and a single Cloud SQL primary becomes the write bottleneck for 1.4 million autosaving sessions while also being a single point of failure and a single region of latency. Sharding it by hand reintroduces every distributed-systems problem you were trying to avoid, and a failover mid-exam loses in-flight transactions — the one thing you swore would never happen.

“Cache everything and hope.” Putting a CDN in front and calling it done handles the static exam shell, but the exam interaction — fetching the next question, saving an answer, syncing a timer — is dynamic, per-candidate, and write-heavy. You cannot cache a write. The dynamic path is the whole problem, and a CDN does not touch it.

The architecture below threads the needle: pre-provision against a known schedule instead of reacting, use a horizontally-scalable consistent database built for exactly this write pattern, absorb reads at the edge and in-memory, and put the dynamic write path behind autoscaling compute that was warmed before the gun fired.

Architecture overview

Trace the diagram left to right as a single request’s journey at T-0: a candidate’s browser hits Akamai at the edge, only genuine dynamic calls pass the global load balancer and Cloud Armor, the session authenticates, lands on the pre-scaled GKE exam-session service, reads and writes hot state in Memorystore, and commits every answer durably to Cloud Spanner — while the proctor path stays a separate, higher-privilege lane. The platform separates three traffic classes that scale on entirely different curves, and keeping them distinct is the first discipline of operating it well: the static delivery path (the exam shell, JS bundles, instructions — heavy on read, trivially cacheable), the dynamic exam path (fetch question, autosave answer, sync timer — the write-heavy core that everything else exists to protect), and the proctoring/admin path (invigilators monitoring sessions, releasing the exam, handling incidents — low volume, high privilege).

The defining property of the whole topology is the one the board cares about: capacity is provisioned ahead of a known clock, not discovered after the spike. This is a scheduled event, so the architecture treats the exam start like a rocket launch — everything is warm, primed, and load-tested before T-0.

Dynamic exam path, following the request flow at T-0:

1. A candidate’s browser, already showing the pre-loaded exam shell, hits Akamai at the edge. Akamai serves all static assets from cache (so 1.4 million shell loads never touch GCP), terminates TLS, and runs as the first DDoS and bot-mitigation layer with rate controls tuned for exam-flood patterns. Only genuine dynamic API calls are forwarded to the origin.
2. Forwarded traffic reaches Google Cloud Load Balancing (global external Application Load Balancer) with Cloud Armor attached as the second, GCP-native defense layer — WAF rules, per-IP and per-token rate limiting, and Adaptive Protection ML-driven L7 DDoS detection. Cloud Armor is also where a scripted cheater hammering the answer-submit endpoint gets throttled before it reaches the application.
3. The candidate’s session is authenticated by a token minted at login. Candidates authenticate through the exam platform’s own identity service backed by Identity Platform, while proctors and administrators sign in through Okta as the workforce IdP — Okta enforces MFA and conditional access for invigilators, and federates to GCP so an invigilator’s privileged session is a first-class, auditable identity separate from any candidate.
4. The request lands on the exam-session service running on GKE — a regional, multi-zone cluster that was pre-scaled to peak before the exam window opened (more on the predictive mechanism below). The service reads and writes the candidate’s live exam state.
5. Hot, per-session state — the current question pointer, the server-authoritative countdown timer, recent autosaves — lives in Memorystore for Redis, which absorbs the brutal read/write rate of timers ticking and answers saving without hammering the database on every keystroke.
6. The durable answer of record is written to Cloud Spanner. Every autosave is an upsert into Spanner, which gives horizontal write scalability and external (globally strong) consistency — so a candidate who drops Wi-Fi and reconnects through a different region sees exactly the answers they saved, with no lost write and no stale read. This is the property that lets the authority promise “we will never lose your answer.”
7. Secrets the services cannot derive from workload identity — third-party proctoring-vendor API keys, the Okta introspection secret, exam-content decryption keys — come from HashiCorp Vault via a sidecar with GCP-backed auth, so nothing sensitive sits in a Kubernetes Secret. Critically, the encrypted exam paper itself is decrypted only at the synchronized release time, with Vault holding the key until the clock says go.

Proctoring/admin path, lower volume and higher privilege: invigilators use a console (Okta-gated) to monitor live sessions, flag anomalies, grant time accommodations, and — the highest-stakes action — trigger the synchronized exam release. That release is a control-plane event that flips a flag in Spanner and pushes via the real-time channel to every candidate at once.

Component breakdown

A few of these choices carry the architecture and deserve the why, because they are the ones teams get wrong on a spike workload.

Why predictive (scheduled) autoscaling, not reactive. Because the exact start time is known, the platform does not wait for CPU to climb — a scheduled scale-up drives the Cluster Autoscaler and HPA to peak capacity before the exam window, holds it warm through the event, and scales back down after. The Horizontal Pod Autoscaler is still configured, but on a custom metric — active exam sessions — not CPU, because CPU lags the real signal and active-session count rises with the herd. A dedicated surge node pool is provisioned and warmed (images pre-pulled, a low-priority “balloon” Deployment holding the nodes) so that when real pods land they schedule in seconds, not minutes. The mental model: you are not autoscaling into the spike, you are pre-staging for it and using autoscaling only to trim.

# HPA on a business metric (active exam sessions), with a high floor
# pre-staged for the known start time — not reacting to CPU.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: exam-session-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: exam-session
  minReplicas: 600        # pre-warmed floor, raised by the scheduled job before T-0
  maxReplicas: 1200
  metrics:
    - type: Pods
      pods:
        metric:
          name: active_exam_sessions_per_pod
        target:
          type: AverageValue
          averageValue: "1200"
  behavior:
    scaleUp:
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30   # allow fast doubling if the floor is under-set

Why Cloud Spanner instead of Cloud SQL. The autosave workload is a high-volume, geographically distributed write stream that must be strongly consistent and must never lose a committed transaction. Spanner is purpose-built for exactly this: it scales writes horizontally by splitting across nodes, gives external consistency (the strongest guarantee, so reads always reflect all prior writes globally), and survives a zone or region failure without losing committed data. A single Cloud SQL primary cannot scale the writes, and its failover window is precisely when you would lose answers. The cost is real — Spanner is more expensive per hour and demands schema discipline — but for “never lose a candidate’s answer,” it is the only primitive that delivers the guarantee.

Why Memorystore sits in front of Spanner. A server-authoritative countdown timer for 1.4 million candidates, ticking and being read constantly, plus autosaves landing every few seconds, would generate a punishing read/write rate straight onto the database. Memorystore for Redis absorbs that: the live timer and current-question pointer live in Redis, autosaves write to Redis immediately for instant client acknowledgment and are flushed durably to Spanner, and reads are served from memory. Redis is the shock absorber that keeps Spanner doing durable writes rather than serving hot per-keystroke reads. The tradeoff is a consistency seam — Redis is fast but volatile — so Spanner remains the answer of record and the flush path is designed to never drop a write.

Implementation guidance

Provision with Terraform, gate it with Wiz Code, deploy with Argo CD. Infrastructure is code, and on a workload where a misconfiguration is a national incident, the pipeline is part of the safety case.

1. A regional, multi-zone GKE cluster (private nodes, Workload Identity on) with two node pools: a baseline pool and a surge pool sized to peak, kept warm by a balloon Deployment.
2. Cloud Spanner with the exam schema — interleave the per-answer rows under the session row so a candidate’s session and all their answers co-locate, keeping autosave writes to a single split where possible.
3. Memorystore for Redis Standard tier (HA with automatic failover) with read replicas sized to the timer/read load.
4. Cloud Armor security policy attached to the global ALB, Adaptive Protection enabled, with explicit per-token rate-limit rules on the autosave and submit endpoints.
5. Akamai in front, caching static assets and pointing at the GCLB private origin as shielded backend.

The pipeline runs in GitHub Actions, authenticating to GCP via OIDC Workload Identity Federation so there is no long-lived service-account key to leak — a lesson the platform team intends never to relearn. Wiz Code scans every Terraform and Kubernetes manifest PR for misconfigurations (a public Spanner endpoint, an over-broad firewall, a missing Pod Security control) and blocks the merge if it finds one. Argo CD then syncs the desired cluster state from Git, so the production cluster is always a known, reviewed commit — no out-of-band kubectl apply on exam morning.

The load test is the deliverable, not an afterthought. You cannot claim to survive 1.4 million synchronized starts; you have to demonstrate it. A required pipeline gate runs a distributed load test that reproduces the curve — near-zero to peak in ninety seconds — against a production-clone environment, asserting p99 autosave latency and zero dropped writes. A scale event that has not been load-tested at the real shape is an unvalidated assumption, and on this platform unvalidated assumptions become court cases.

Synchronized release without a thundering-herd self-DDoS. The exam content is encrypted at rest; Vault holds the decryption key and releases it only at the start time, so even an insider cannot read the paper early. At T-0 the release flips a flag in Spanner and fans out over Pub/Sub to a WebSocket gateway that pushes to already-connected candidates — candidates are connected and idle before the start, so the gun does not trigger 1.4 million simultaneous new connections, only a lightweight push over existing ones. Pre-connecting the herd is what converts a connection storm into a trivial fan-out.

Enterprise considerations

Security & Zero Trust. The platform is hostile-internet-facing on its biggest day, so defense is layered and identity is strict. Akamai and Cloud Armor form two independent DDoS/WAF tiers — Akamai absorbs volumetric and static-flood attacks at the edge, Cloud Armor’s Adaptive Protection catches application-layer (L7) attacks and rate-limits the autosave/submit endpoints so a scripted cheater is throttled before reaching the app. Privileged actions are gated hard: invigilators authenticate through Okta with MFA, and the highest-stakes action — releasing the exam — requires step-up authentication, so no single stolen session can leak the paper. HashiCorp Vault holds the exam-paper key and releases it only at start time, making “early access to questions” a non-event. Wiz runs continuous CSPM across GKE and Spanner, alerting on any drift to public exposure or an over-permissive IAM binding, while Wiz Code shifts that same checking left into the IaC pipeline. CrowdStrike Falcon sensors on the GKE node pools provide runtime threat detection feeding the testing authority’s SOC, and any security event auto-raises a ServiceNow incident so there is a ticket and a bridge, not just a log line. Least-privilege IAM scopes each service to exactly the resources it needs, and candidate and proctor identities are separate trust domains by construction.

Cost optimization. The economics of a once-a-quarter spike are the whole game: peak capacity for three hours, near-zero for months. Engineer for the curve, not the peak.

The discipline is to treat the surge as ephemeral: provision it the day before, tear it down the day after, and never let “we might need it” leave peak capacity running into the next quarter. Pipe the per-window cost to Datadog so finance sees exactly what each exam day costs.

Scalability and the spike, concretely. Each tier scales on its own axis. GKE scales pods on the active-sessions custom metric and nodes via the Cluster Autoscaler against the warmed surge pool, so capacity is present before load. Spanner scales by adding nodes/processing units, raised for the event window. Memorystore scales reads via read replicas. The real ceiling to plan against is not CPU but the regional quotas — Compute Engine instance quota, in-use IP addresses, Spanner node limits — which is why the load test runs against the real project with the real quotas, and why a quota-increase request goes in weeks early. A spike architecture that hits an unraised quota at T-0 fails exactly as badly as one with no autoscaling at all.

Failure modes, and what each one looks like on the day. Name them before they page the war room.

• Reactive scaling lag — if the floor is set too low and the surge pool is not warmed, the spike arrives before capacity does and candidates see spinners. Mitigation: a high pre-staged minReplicas, the balloon Deployment holding warm nodes, and a load test that proves the floor is right.
• Spanner hot-spotting — monotonically increasing keys (e.g., a timestamp or sequential session ID as the primary key) funnel all writes to one split and cap throughput. Mitigation: hash/UUID-prefixed keys and interleaved tables so writes spread across splits — the single most important schema decision.
• Redis-to-Spanner flush gap — an answer acknowledged from Redis but not yet flushed when a node dies could be lost. Mitigation: write-through to Spanner on the critical save path with Redis as the fast acknowledgment, never Redis-only for the answer of record.
• Connection-storm self-DDoS at T-0 — if candidates connect at the start rather than before, the release triggers a connection storm. Mitigation: pre-connect the herd to the WebSocket gateway during the lobby, so T-0 is a fan-out, not a connection flood.
• DDoS masking real load — a volumetric attack at exam start is indistinguishable from the legitimate herd if you are not ready. Mitigation: Akamai + Cloud Armor tuned and tested before the day, with the war room watching attack and legitimate traffic on separate Datadog panels.
• Regional outage — see DR below.

Reliability & DR (RTO/RPO). The numbers are decided per tier and the bar is exceptional because a failed exam is not retryable in the moment. Cloud Spanner multi-region gives synchronous replication and survives a full region loss with zero RPO for committed answers — the non-negotiable guarantee. GKE is regional and multi-zone, so a zone failure is transparent; a region failure fails over to a warm standby cluster the load balancer can route to. Memorystore Standard tier provides automatic cross-zone failover. A pragmatic target for the exam-day service: RTO under 5 minutes, RPO zero for submitted answers, with the explicit design principle that a submitted answer is durable the instant Spanner commits it, region failure included. The exam itself can tolerate a brief connectivity blip for a candidate (the client buffers and re-syncs) far more easily than it can tolerate a single lost commit — so the architecture spends its reliability budget on durability first.

Observability and the war room. Exam day is run from a live war room, and Datadog is the single pane: real-time active-concurrency (the metric that proves the herd arrived and capacity held), p99 autosave latency, Spanner CPU and commit latency, Redis hit rate, Cloud Armor blocked-request rate, and SLO burn-rate alerts that page before candidates feel pain. Real User Monitoring on the candidate client surfaces what actual browsers experience, not just server health. A blocked-request spike or an SLO breach auto-opens a ServiceNow major incident with the bridge details, so the org is in incident response within seconds. The principle: on a scheduled high-stakes event, you watch leading indicators (concurrency rising, latency creeping) and act before the lagging ones (errors, abandons) ever move.

Governance. A strict change freeze goes into effect days before the exam — enforced as a ServiceNow change gate that blocks any non-emergency deploy, with Argo CD ensuring production matches a reviewed Git commit and nothing else. Exam content keys, IAM bindings, and Cloud Armor policies are all version-controlled and reviewed. Every privileged proctor action (release, accommodation grant, session intervention) is logged immutably for audit, because the integrity of the exam — and its defensibility in the inevitable challenge — depends on a complete record of who did what when.

Explicit tradeoffs

Accept these or do not build it. This architecture optimizes for a brutal, scheduled spike, and that focus has costs. Cloud Spanner is more expensive and more demanding than a single SQL database — you pay for horizontal write scale and global consistency in dollars and in schema discipline (get the key design wrong and you hot-spot), and for a small, steady workload it is overkill. Predictive scaling trades simplicity for safety: you must know and trust the schedule and pre-provision against it, which means a balloon Deployment and a warmed surge pool burning some money before the event and the operational ritual of scaling up and tearing down each cycle. The layered Akamai + Cloud Armor defense is two products to license, tune, and test rather than one. The Redis-in-front-of-Spanner design adds a consistency seam you must engineer carefully so a fast acknowledgment never becomes a lost answer. And the whole thing demands a real load test at the real shape, which is itself a significant engineering investment — but skipping it is how the legacy provider ended up in court.

The alternatives, and when they win. If your exams are asynchronous — candidates start whenever they like across a window — the synchronized-spike problem evaporates and you can lean on ordinary reactive autoscaling and a regional database; this entire architecture is overkill. If you are running a small institutional exam (a few thousand students, one university), a single Cloud SQL instance and a modest GKE deployment are simpler, cheaper, and entirely sufficient — graduate to this design only when scale and simultaneity demand it. If the platform is a learning management system rather than a high-stakes exam — think Moodle delivering courseware and quizzes where a momentary blip is an annoyance, not a lawsuit — the correctness and DR bar drops dramatically and a managed Moodle on autoscaled GKE with Cloud SQL is the pragmatic fit. And if you genuinely cannot predict the spike timing, you fall back to reactive scaling with a generous floor and accept the cold-start risk this design is built to eliminate. The architecture here is the destination for a national, synchronized, high-stakes exam; the right starting point depends on which of those three words actually apply to you.

Going deeper

The architecture above is the what; this section is the how it actually behaves under the hood — the mechanisms you configure, the GCP-native options the reference design abstracts over, and the failure physics an experienced engineer needs to reason about before the day.

The three ways to pre-warm, and when to use each

“Pre-warming” is one idea with three concrete implementations on GCP, one per compute model:

• GKE (this design): a high pod floor + a warmed node pool. The Horizontal Pod Autoscaler’s minReplicas sets a pod floor, but pods only start fast if nodes already exist with the image pulled. So you provision a surge node pool ahead of time and hold it warm with a low-priority balloon (pause-container) Deployment that reserves the nodes and is evicted the instant real pods need them. Pre-pull the container image onto every node (a DaemonSet or pre-pull Job) so pod start is a container launch, not a registry download.
• Cloud Run: --min-instances. For any service on Cloud Run rather than GKE, the equivalent of a warm floor is a minimum-instance count, which keeps idle instances alive and eliminates cold starts. You raise the floor before the event and drop it after:

gcloud run deploy exam-api \
  --image=REGION-docker.pkg.dev/PROJECT_ID/exam/api:TAG \
  --min-instances=50 \
  --max-instances=1000 \
  --region=REGION

• Managed Instance Groups (VMs): scheduled autoscaling. For a VM-based tier, a MIG autoscaler supports scheduled scaling that raises the minimum on a timezone-aware cron — the cleanest native “raise the floor before the gun” primitive:

gcloud compute instance-groups managed update-autoscaling exam-mig \
  --set-schedule=exam-saturday \
  --schedule-cron="0 8 * * 6" \
  --schedule-duration-sec=16200 \
  --schedule-time-zone="Asia/Kolkata" \
  --schedule-min-required-replicas=1000

The mental model is identical across all three: raise a guaranteed floor on a schedule, and let reactive autoscaling only trim above it.

Scheduled autoscaling on GKE with KEDA

Vanilla Kubernetes has no time-aware HPA — the HPA reacts to metrics, not the clock. The idiomatic way to add a schedule is KEDA’s cron scaler, which raises minReplicaCount between a start and end cron expression and, crucially, takes the maximum across all its triggers — so you can combine a scheduled floor with a metric-driven trim in one object:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: exam-session-scaler
spec:
  scaleTargetRef:
    name: exam-session
  minReplicaCount: 50            # off-peak floor
  maxReplicaCount: 1200
  cooldownPeriod: 600
  triggers:
    - type: cron                 # the scheduled pre-warm floor for exam morning
      metadata:
        timezone: Asia/Kolkata
        start: "0 8 * * 6"       # 08:00 Saturday — raise the floor before T-0
        end: "30 12 * * 6"       # 12:30 Saturday — exam over, release the floor
        desiredReplicas: "1000"
    - type: prometheus           # trim above the floor on the real signal (representative)
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        query: sum(active_exam_sessions)
        threshold: "1200"

Because arrival is synchronized, the pre-warm floor must sit close to peak, not a fraction of it — set desiredReplicas near your computed peak pod count so capacity is present when the gun fires, and let the metric trigger add the last few percent if concurrency overshoots. This is the concrete implementation of the reference design’s “minReplicas: 600, raised by the scheduled job before T-0.”

Cloud CDN vs. a third-party edge

The reference design uses Akamai at the edge, but the GCP-native equivalent is Cloud CDN, enabled directly on a backend of the global external Application Load Balancer. For the static exam shell it is often all you need: flip --enable-cdn on the backend service, set explicit Cache-Control on the immutable bundles, and enable cache-fill and negative caching.

gcloud compute backend-services update exam-static-backend \
  --enable-cdn \
  --cache-mode=CACHE_ALL_STATIC \
  --global

The design keeps both because two independent DDoS tiers is a deliberate resilience choice on a hostile-internet day — but a smaller exam should start with Cloud CDN alone and add a third party only when the threat model demands independent absorption. Either way the principle stands: the static class must never reach origin.

Queue-based load leveling for everything that isn’t the hot path

The autosave hot path is write-through to Spanner for durability — you do not put a queue in front of the answer of record. But a surprising amount of exam-day work is not latency-critical: grading a submitted paper, generating a submission receipt, emitting analytics, emailing “your exam is submitted,” reconciling proctor flags. Pushing that onto the synchronous path makes every request fatter exactly when you can least afford it. Queue-based load leveling decouples it: the request does the minimum durable write, drops a message, and returns; workers drain the queue at their sustainable rate while the backlog absorbs the spike.

• Pub/Sub for fan-out and event work — durable, at-least-once, with subscriber flow control so consumers pull only what they can handle. (See the Pub/Sub deep dive for delivery and ordering semantics.)
• Cloud Tasks when you need an explicit dispatch-rate cap into a backend — it enforces a maximum requests-per-second and concurrency so a downstream service (or a rate-limited third-party grading API) is never overrun:

gcloud tasks queues create exam-postsubmit \
  --max-dispatches-per-second=500 \
  --max-concurrent-dispatches=1000 \
  --max-attempts=100

The rule of thumb: synchronous only for what the candidate must see succeed right now (the durable save); everything else goes on a queue and is leveled.

The database connection storm — pooling and read replicas

Even with compute scaled perfectly, a classic exam-day outage is the connection storm: 1,000+ pods each opening a handful of database connections multiplies into tens of thousands, and a Postgres/Cloud SQL instance has a hard max_connections ceiling far below that. The fix is a connection pooler — PgBouncer in transaction pooling mode multiplexes thousands of client connections onto a small pool of real backend connections — deployed as a sidecar or a small Deployment in front of the database. On the read side, read replicas offload SELECT traffic from the primary so the primary is reserved for writes. This is the scaling path for the Cloud SQL alternative (the smaller-exam design): see Cloud SQL HA, read replicas & private connectivity for the mechanics.

Spanner sidesteps the raw max_connections wall — clients use a session pool in the client library against a horizontally-scaled backend rather than one primary — which is a large part of why it is chosen here. But the discipline transfers: size the session pool deliberately, watch for pool exhaustion under the herd, and never let per-pod connection counts multiply unbounded.

Graceful degradation — protect the core by shedding the periphery

When demand still exceeds capacity despite every precaution, the platform must degrade, not collapse. Degradation is a deliberate, ordered ladder — decided in advance, not at 09:01 in the war room:

1. Sample down telemetry — drop RUM/analytics sampling first; it costs the candidate nothing.
2. Relax autosave frequency — widen the autosave interval under load (the client still buffers locally, so nothing is lost, it just syncs less often).
3. Serve question content from cache — render questions from the CDN/Memorystore copy rather than a fresh origin fetch.
4. Disable non-essential UI — rich media, live leaderboards, “students online” counters.
5. Load-shed non-critical endpoints — return 429 with Retry-After on anything that isn’t the exam, and let Cloud Armor’s throttle rules enforce it at the edge.

The one rung you never shed is the answer-save/submit path. The design principle is that the client is local-first: it writes answers to local storage immediately and re-syncs to the backend, so a backend blip degrades to “syncs a few seconds later,” never “lost the answer.” Degradation spends the periphery to keep the core alive.

The load test is the proof, at the real shape

A steady “2× average” load test proves nothing here, because the failure is the curve, not the volume. The load test must reproduce near-zero to peak in ninety seconds against a production clone with the real project’s real quotas, and assert two things: p99 autosave latency stays under target, and zero writes are dropped. Tooling: a distributed Locust or k6 run (many workers, because one box cannot generate 1.4M synchronized virtual users), the connection ramp modelled as a step function rather than a gentle ramp, and a separate synthetic DDoS-plus-legitimate run so the war room learns to tell attack from herd on the dashboards before the day. Treat the load test as a required pipeline gate: an untested scale assumption is, on this platform, a court case in waiting.

The game-day runbook (T-minus timeline)

Exam day runs to a written runbook, not improvisation. A representative timeline:

The runbook is the difference between a scheduled event and a gamble: every high-stakes moment is rehearsed, timed, and owned before it happens.

The shape of the win

For the testing authority, the payoff is not “a website that stayed up.” It is that on the Saturday in May, 1.4 million candidates pressed “Start” inside the same ninety seconds, the first question rendered in under a second, every autosave landed durably in Spanner the instant it was made, a volumetric attack at 09:01 was absorbed by Akamai and Cloud Armor without a candidate noticing, and at the end of the three hours not one answer was lost and not one candidate saw a spinner — so the result stood, unchallenged, and there was no re-sit. That last clause is what funds the platform. Everything upstream — the scheduled pre-warm, the surge node pool, Spanner’s external consistency, Memorystore’s shock absorption, the layered DDoS defense, Vault holding the paper key until the gun, the load test at the real shape, the Datadog war room — exists so that a candidate, a regulator, and a court each conclude the exam was fair. Start narrower if your problem is smaller, but for a country’s entrance exam on a single synchronized morning, this is where the architecture has to land.

Practice challenges

Work them in order — each builds a piece of the real design. Try before opening the solution.

1. Size the pre-warm floor (beginner). 1.4M candidates, each pod serves ~1,200 concurrent sessions. Because the start is synchronized, roughly how many pods must be already running at T-0, and what does that make your minReplicas?

<details> <summary>Solution</summary>

1,400,000 ÷ 1,200 ≈ 1,167 pods at peak. Because everyone arrives at once, the pre-warm floor must be near peak — set minReplicas to ~1,100–1,200 by T-0 (with maxReplicas a little higher for headroom), not a small fraction you plan to “scale up into.”

Why: a synchronized start gives you no time to scale, so the floor is the plan — under-set it and the herd beats your capacity. </details>

2. Sort the traffic (beginner). Label each as CDN-cacheable or must hit origin: (a) the exam shell HTML, (b) the JS/CSS bundle, © “fetch my next question,” (d) “autosave my answer,” (e) the printed candidate instructions PDF, (f) “sync my countdown timer.”

<details> <summary>Solution</summary>

Cacheable at the edge: a, b, e (identical for everyone, immutable). Must hit origin: c, d, f (per-candidate, dynamic, write-heavy).

Why: separating the static class from the dynamic class is the first scaling discipline — the CDN removes millions of reads so origin only sees the writes that actually need it. </details>

3. Throttle a scripted cheater (intermediate). Write a Cloud Armor rule that per-token (Authorization header) rate-limits the /api/v1/answer endpoint, banning a token that exceeds 100 requests/minute.

<details> <summary>Solution</summary>

gcloud compute security-policies rules create 1000 \
  --security-policy=exam-armor-policy \
  --expression="request.path.matches('/api/v1/answer')" \
  --action=rate-based-ban \
  --enforce-on-key=HTTP-HEADER \
  --enforce-on-key-name=Authorization \
  --rate-limit-threshold-count=100 \
  --rate-limit-threshold-interval-sec=60 \
  --ban-duration-sec=600 \
  --conform-action=allow \
  --exceed-action=deny-429

Why: keying on the token (not IP) throttles the account doing the scripting even behind shared/mobile NAT, while genuine candidates saving normally stay under the threshold. </details>

4. Pre-warm on a schedule (intermediate). On GKE, raise the exam-session floor to 1,000 replicas between 08:00 and 12:30 on Saturday, Asia/Kolkata, and let it fall back afterward. Sketch the mechanism.

<details> <summary>Solution</summary>

A KEDA cron ScaledObject (shown in Going deeper): start: "0 8 * * 6", end: "30 12 * * 6", timezone: Asia/Kolkata, desiredReplicas: "1000", with a low minReplicaCount off-peak. On a MIG instead: gcloud compute instance-groups managed update-autoscaling --set-schedule=... --schedule-cron="0 8 * * 6" --schedule-min-required-replicas=1000.

Why: the clock, not a metric, drives pre-warming — you raise a guaranteed floor before the spike and let reactive scaling only trim above it. </details>

5. Design the answers key to avoid hot-spotting (advanced). Your Spanner Answers table is written on every autosave. Propose a primary key that spreads writes across splits, and name the anti-pattern to avoid.

<details> <summary>Solution</summary>

Key on (SessionId, QuestionId) where SessionId is a UUID / hash-prefixed value, and interleave Answers under the Sessions table so a candidate’s rows co-locate on one split. Avoid a monotonically increasing key — a timestamp or sequential session id — which funnels every new write to the last split and caps throughput. See Spanner schema design & hot-spot avoidance.

Why: Spanner scales writes by splitting the key range; a monotonic key defeats splitting and serializes the whole write load onto one server. </details>

6. Build the degradation ladder (advanced). Capacity is 90% of the herd. List, in order, what you shed to stay up — and the one thing you never shed. Then say where the post-submit grading work goes.

<details> <summary>Solution</summary>

Shed in order: analytics/RUM sampling → widen autosave interval → serve questions from cache → disable non-essential UI → 429 + Retry-After on non-critical endpoints. Never shed the answer-save/submit path (the client is local-first and re-syncs). Post-submit grading goes on a queue — publish to Pub/Sub (or Cloud Tasks with a dispatch cap) and let workers drain it at a sustainable rate, so the durable save stays synchronous but the expensive work is leveled.

Why: graceful degradation protects the core (durable answer capture) by spending the periphery, and queue-based load leveling keeps the spike out of the latency-critical path. </details>

Common beginner mistakes

• “Autoscaling will catch the spike.” The misconception is that the HPA and Cluster Autoscaler react in seconds. They don’t — scheduling a pod, provisioning a node, pulling the image, and passing readiness is a minutes-long control loop, and a synchronized start is over in ninety seconds. Right model: pre-provision against the known clock; autoscaling only trims above a pre-warmed floor.

• “A CDN in front means we’re scaled.” A CDN caches static reads; the exam is dynamic per-candidate writes, and you cannot cache a write. Right model: separate traffic classes — the CDN handles the static shell so you can spend all your engineering on the dynamic write path, which is the actual problem.

• “One big managed database is fine.” A single primary is simultaneously the write bottleneck, the single point of failure, and a single region of latency — and its failover window is exactly when in-flight answers vanish. Right model: horizontally-scalable strongly-consistent writes (Spanner), or for smaller scale a primary with read replicas and connection pooling, tradeoffs accepted knowingly.

• “Cache the answers in Redis for speed.” Redis is fast because it is in-memory and volatile — a node death loses anything not yet flushed. Right model: Redis for hot reads and instant acknowledgment, but the answer of record is a write-through to Spanner; never make the volatile cache the source of truth.

• “We load-tested at 2× average, we’re good.” Average load never exercises the failure mode, which is the near-vertical curve and the connection storm at T-0. Right model: reproduce the real shape — zero to peak in ninety seconds — against a production clone with real quotas, asserting zero dropped writes.

• “We’ll get the quota when we need it.” Compute instance count, in-use IPs, and Spanner nodes are hard ceilings, and an increase needs justification and can take days. Hitting an unraised quota at T-0 fails exactly as badly as having no autoscaling. Right model: request increases weeks early and load-test against the real project’s real limits.

• “The start gun opens the connections.” Letting 1.4M browsers open a fresh TLS/WebSocket connection at T-0 is a self-inflicted DDoS. Right model: pre-connect the herd during the lobby so the start is a lightweight fan-out over existing connections, not a connection flood.

Glossary

• Thundering herd — a very large number of clients doing the same thing at the same instant (here, 1.4M candidates pressing “Start” together), overwhelming a system that could handle the same total spread over time.
• Cold start — bringing capacity up from near-zero; slow because new instances must be scheduled, booted, and made ready before they serve traffic.
• Pre-warming — provisioning and readying capacity before a known spike instead of reacting to it, so it is already serving when load arrives.
• Min-instances / minReplicas — a guaranteed floor of running instances/pods that never scales below; the concrete “warm floor” you raise before an event.
• Scheduled autoscaling — raising that floor on a clock (a cron/schedule) rather than in response to a metric — the mechanism for pre-warming against a known time.
• Horizontal Pod Autoscaler (HPA) — Kubernetes controller that adds/removes pod replicas based on a metric (CPU or, better here, active sessions).
• Cluster Autoscaler — GKE controller that adds/removes nodes (VMs) when pods can’t be scheduled or nodes sit idle.
• Node pool / surge pool — a group of identically-configured GKE nodes; a surge pool is extra capacity provisioned and warmed just for the spike.
• Balloon / pause Deployment — low-priority placeholder pods that reserve warm nodes and are instantly evicted when real pods need the room.
• KEDA — Kubernetes Event-Driven Autoscaling; adds scalers (including a time-aware cron scaler) beyond the built-in HPA.
• Global external Application Load Balancer — GCP’s single global entry point that spreads L7 (HTTP) traffic across regional backends behind one anycast IP.
• Cloud CDN — GCP’s content delivery network, enabled on a load-balancer backend, that caches static content at Google’s edge so it never reaches origin.
• Cloud Armor — GCP’s WAF and DDoS service on the load balancer: rule-based filtering, rate limiting, and ML-driven Adaptive Protection.
• WAF (Web Application Firewall) — filters HTTP requests against rules (e.g., OWASP) to block malicious patterns before they reach the app.
• Rate limiting / rate-based ban — capping how many requests a key (IP, token) may make per interval, then throttling or banning offenders.
• DDoS — Distributed Denial of Service: many sources flooding a target to exhaust its capacity.
• Cloud Spanner — GCP’s horizontally-scalable, strongly-consistent relational database; scales writes by splitting data across nodes.
• External consistency — Spanner’s strongest guarantee: any read reflects all writes committed before it, globally — so a reconnecting candidate always sees their saved answers.
• Split / hot-spotting — Spanner shards a table into key-range splits; a hot-spot is when a poor (monotonic) key funnels all writes to one split, capping throughput.
• Interleaved tables — physically co-locating child rows (answers) under their parent (session) so related data lives on one split.
• Memorystore for Redis — GCP’s managed in-memory store; here the fast, volatile buffer for live timers and recent autosaves.
• Write-through — writing to the cache and the durable store on the critical path so an acknowledged write is never only in volatile memory.
• Queue-based load leveling — placing a durable queue between a spiky producer and a downstream consumer so the consumer drains at its own steady rate.
• Pub/Sub — GCP’s durable publish/subscribe messaging for fan-out and event decoupling, with subscriber flow control.
• Cloud Tasks — GCP’s managed task queue with an enforceable dispatch-rate and concurrency cap into a backend.
• Connection pool / PgBouncer — a layer that multiplexes many client connections onto a few real database connections, preventing connection exhaustion.
• Read replica — a read-only copy of a database that offloads SELECT traffic from the primary.
• Graceful degradation — deliberately reducing non-essential functionality under overload to keep the critical path alive, instead of failing everything.
• Load shedding — dropping or rejecting (e.g., 429) lower-priority requests to protect capacity for high-priority ones.
• Identity Platform — GCP’s customer identity (CIAM) service; here it authenticates candidates and issues session tokens.
• Workload Identity Federation — lets external systems (e.g., GitHub Actions) get GCP access via short-lived tokens with no stored service-account key.
• RTO / RPO — Recovery Time Objective (how fast you recover) and Recovery Point Objective (how much data loss is tolerable — zero for submitted answers here).
• SLO / burn rate — Service Level Objective (a reliability target) and how fast you’re consuming its error budget; burn-rate alerts page before users feel pain.
• p99 latency — the 99th-percentile response time; the slow tail that the unluckiest 1% of candidates actually feel.
• Game day / runbook — a rehearsed, written plan (often a T-minus timeline) executed during a high-stakes event so nothing is improvised.