GCP Enterprise Architecture: Production Microservices on GKE

In a nutshell

If you have ever flown through a large, well-run airport, you already understand this architecture. Dozens of airlines (your teams and services) operate independently — different destinations, schedules, and crews — yet they all share one set of runways, one security model, and one operations manual. Nobody lets each airline pour its own runway or run its own passport control. The airport provides those as shared, standardised services so the airlines can focus on flying.

That is exactly what a production microservices platform on Google Kubernetes Engine (GKE) does. Each team ships its own service on its own cadence, but the platform provides the shared machinery that would otherwise be reinvented — badly — twenty times over. The runways and taxiways are the network; the security checkpoints between areas are the service mesh (who may talk to whom, and every conversation encrypted); passport control is Workload Identity (each service proves who it is to reach Google APIs, with no forgeable badges or shared keys); customs inspecting every bag before it boards is Binary Authorization (only scanned, signed container images are allowed to run); and the single master operations manual every terminal follows identically is GitOps (the cluster is always exactly what a Git repository says it should be).

The last piece is who staffs and maintains the airport itself. With GKE Autopilot you rent a fully-staffed, continuously-maintained airport: Google runs, patches, secures, and right-sizes the underlying machines, and you pay for the space your flights actually use. You get Kubernetes’ API and its whole ecosystem without the night-shift job of keeping the runways clear.

For a beginner, the one idea to hold onto: this article is not about the services — it is about everything around them. The services are the easy part. The architecture is the network, mesh, identity, supply chain, and GitOps that turn a pile of containers into a platform an auditor, an on-call engineer, and a CFO can all trust. Read the rest with that lens — every component below exists to remove one specific way a platform gets you paged at 3 a.m.

Level: Advanced · Time: ~35 min

Most “microservices on Kubernetes” diagrams are a lie of omission. They show a tidy box of services talking to each other and call it an architecture. The real work — the part that separates a platform you can run with a small team from a 3 a.m. pager that never stops — is everything around the services: who is allowed to call whom and over what (the mesh), what is actually allowed to run (the supply chain), which Google APIs each pod can touch (identity), and how a fleet of clusters across regions stays identical without a human ever running kubectl apply against production (GitOps). This article is a complete, reusable GCP reference architecture for production microservices built on GKE Autopilot, Anthos Service Mesh, Artifact Registry, Workload Identity, and Config Sync. It is sized to start with one cluster and a dozen services, and to grow to a multi-region fleet running hundreds — without changing shape.

The business scenario

Picture an engineering organisation that has outgrown a monolith. Maybe it is a 40-engineer fintech that started with one Rails app and now has a dozen teams each wanting to ship on their own cadence; maybe it is a 600-engineer retailer breaking a fifteen-year-old Java monolith into domains. The company name and stack differ, but the forces pushing them onto this exact architecture are the same three every time.

The first is deployment independence colliding with operational reality. The whole point of microservices is that the payments team can ship without waiting on the catalog team. But the moment you have twenty services, the platform questions arrive all at once: how do they discover each other, how is traffic between them encrypted, how do you roll out version N+1 to 5% of traffic, how do you stop a bad deploy from taking down checkout? Teams that answer these per-service, in application code, end up with twenty subtly different retry policies and zero consistent security. The org needs these to be platform properties, not per-team folklore.

The second is the cluster-ops tax. Self-managed Kubernetes is a part-time job that quietly becomes three full-time jobs: node pool sizing, OS patching, autoscaler tuning, security CVE chases, control-plane upgrades, and the perennial “why is this node NotReady.” A 40-person company cannot spare two SREs to babysit node pools, and a 600-person company would rather spend those SREs on reliability of the product, not the substrate. The org wants Kubernetes’ API and ecosystem without operating the machines underneath it.

The third is the audit-and-supply-chain squeeze. Once you are processing payments or holding customer data, someone will ask: prove that only code that passed CI, was scanned, and was signed is running in production. Prove that service A cannot read service B’s database. Prove that prod and the DR region are configured identically. “Trust me, we use Kubernetes” is not an answer a SOC 2 or PCI auditor accepts, and it is not an answer a board accepts after the first incident.

This architecture answers all three. GKE Autopilot removes the cluster-ops tax by running and securing the nodes for you and billing per pod resource request. Anthos Service Mesh makes mTLS, traffic-splitting, and resilience platform properties applied by sidecar, not by application code. Artifact Registry plus Binary Authorization turns “only trusted code runs” from a slogan into an admission-control gate. Workload Identity gives every service its own scoped Google identity with no static keys. And Config Sync makes Git the single source of truth so every cluster in the fleet is, provably, what the repo says it is. The same blueprint runs a single us-central1 cluster for a startup and a three-region fleet for an enterprise; only the cluster count and quotas change.

Architecture overview

Follow one request and one deploy, and the design explains itself.

The request path. A user hits the public hostname, which resolves to a Global External Application Load Balancer fronted by Cloud Armor (WAF + L7 DDoS). The load balancer does not target VMs; it targets the cluster directly through container-native load balancing — the GKE Gateway controller programs the LB’s backend service with standalone network endpoint groups (NEGs) whose endpoints are the actual pod IPs of the ingress gateway. Traffic therefore goes edge → pod with no extra NodePort hop. The ingress lands on the Anthos Service Mesh ingress gateway (an Envoy pod) running inside the cluster.

From the gateway inward, every hop is mesh-managed. Each application pod runs the workload container plus an Envoy sidecar injected automatically; pod-to-pod traffic is upgraded to mutual TLS with certificates issued by the mesh’s CA and rotated on a short clock — the application code sends plaintext HTTP to localhost and the sidecars do the encryption. The mesh’s control plane (managed ASM, running on Google’s infrastructure, not your nodes) distributes routing, retry, timeout, and authorization policy to every sidecar. So a request from frontend to checkout to payments is encrypted, authenticated by SPIFFE identity, authorised by L7 policy, retried on transient failure, and traced end-to-end — without any of those three services containing a line of code for it.

When a service needs a Google API — checkout writing to Cloud SQL, inventory publishing to Pub/Sub, a worker reading from a GCS bucket — it does not use a downloaded service-account key. Its Kubernetes service account is bound via Workload Identity Federation for GKE to a Google service account (or, in the newer model, granted IAM directly on the KSA principal). The pod’s metadata server hands out short-lived tokens scoped to exactly that identity. payments can reach the payments database; catalog cannot, because its identity has no such grant.

The deploy path. A developer merges to main. Cloud Build (or any CI) builds the image, runs tests and a vulnerability scan, pushes to Artifact Registry, and — critically — produces a cryptographic attestation that this digest passed the pipeline. A separate Git repo of Kubernetes manifests (the “config” repo) is updated with the new image digest. Config Sync (part of Config Management) is continuously watching that repo; it pulls the change and reconciles the cluster to match, so the rollout is a Git commit, not a kubectl command. As the new pod tries to start, Binary Authorization intercepts the admission request and checks: is this exact image digest signed by the required attestors? No attestation, no admission — a hand-pushed or tampered image is rejected by the cluster itself.

So the end-to-end picture is two intersecting loops. The runtime loop: user → Cloud Armor → Global LB → (container-native NEG) → ASM ingress gateway → mTLS mesh of Autopilot pods → Google services via Workload Identity, with telemetry flowing to Cloud Operations. The delivery loop: commit → Cloud Build → Artifact Registry (+ attestation) → config repo → Config Sync → cluster, gated at the door by Binary Authorization.

The diagram in words: at the top, users hitting a single anycast VIP guarded by Cloud Armor. Below it, a Global LB whose arrow lands inside a cluster boundary on an ingress-gateway pod. Inside the boundary, a lattice of service pods each drawn with a small sidecar square, every connecting line labelled “mTLS.” Each pod has a thin dotted line out to a Google service (Cloud SQL, Pub/Sub, GCS, Secret Manager) labelled “Workload Identity — no keys.” Off to the left, a CI/CD column: Git → Cloud Build → Artifact Registry, with a lock icon (“attestation”) feeding a gate icon (“Binary Authorization”) sitting on the cluster’s admission boundary. A second Git repo (“config”) feeds a Config Sync agent inside the cluster. The whole cluster box is duplicated faintly to the right to signify a second region in the fleet, both fed by the same two Git repos.

Component breakdown

Each component earns its place by removing a specific failure mode rather than adding a feature.

Three of these choices deserve emphasis because they are where teams most often go wrong.

Autopilot’s billing unit is the pod request, so right-sizing requests is cost control. On a Standard cluster you pay for nodes whether pods use them or not, and slack hides in node headroom. On Autopilot you pay for the sum of pod CPU/memory requests (rounded to Autopilot’s allowed shapes). An over-requested replicas: 10 deployment asking for 2 vCPU when it uses 0.3 is now a line item you can see and fix. This is a feature, but it changes the discipline: VPA in recommendation mode and a habit of setting requests close to real usage are not optional.

Mesh mTLS and authorization are two separate switches, and only turning on the first is a common, dangerous half-measure. PeerAuthentication: STRICT encrypts and authenticates traffic, but by itself it still lets any meshed service call any other meshed service — encrypted. The Zero-Trust property you actually want comes from a default-deny AuthorizationPolicy plus explicit allows (“frontend may call checkout on POST /cart; nothing may call payments except checkout”). Encryption without authorization is a locked door with no lock on the inner rooms.

Binary Authorization is only as strong as where the attestation is created. If your CI signs the image before the vulnerability scan and tests pass, you have a signature that proves nothing. The attestor must sign the digest at the end of a pipeline that has already gated on scan and test results — and the signing key must live somewhere CI can use but humans cannot exfiltrate (Cloud KMS with tight IAM). Done right, a developer literally cannot docker push something into prod, because the cluster will refuse it at admission.

Implementation guidance

The whole platform should be code: Terraform for the Google-side infrastructure, and Git-backed Kubernetes manifests reconciled by Config Sync for everything inside the cluster. Resist the urge to gcloud/kubectl your way to a running system — the entire value proposition here is reproducibility.

Project and IaC layout. Use a small set of projects under a folder: a platform project for shared infra (Artifact Registry, KMS, the config repo’s deploy identity), and one project per environment (dev, staging, prod) — or per region in the fleet. Terraform provisions the cluster and the Google primitives; it should not manage in-cluster app objects (that is Config Sync’s job). Keep state in a GCS backend with versioning and per-environment state isolation.

A representative Terraform skeleton for the cluster (HCL):

resource "google_container_cluster" "platform" {
  name             = "platform-prod"
  location         = "us-central1"        # regional = control plane in 3 zones
  enable_autopilot = true                 # Autopilot
  release_channel { channel = "REGULAR" }

  # Keyless access to Google APIs from pods
  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }

  # Private cluster: nodes have no public IPs
  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false       # control plane reachable from authorized nets
    master_ipv4_cidr_block  = "172.16.0.0/28"
  }
  master_authorized_networks_config {
    cidr_blocks { cidr_block = var.admin_cidr  display_name = "ci-and-bastion" }
  }

  # Only signed images may run
  binary_authorization { evaluation_mode = "PROJECT_SINGLETON_POLICY_ENFORCE" }
}

Managed ASM and Config Management are then enabled as fleet features on the cluster (via the gke_hub feature/membership resources or the gcloud container fleet equivalents), so the mesh and GitOps agents are installed and auto-upgraded by Google rather than pinned by you.

Networking and identity wiring. Use VPC-native (alias IP) clusters so pods get real VPC IPs — this is what makes container-native NEGs and direct DB connectivity work. Databases sit behind Private Service Connect or private IP; there are no public database endpoints anywhere in the design. North-south traffic uses the Gateway API (gke-l7-global-external-managed GatewayClass) to provision the Global LB; attach Cloud Armor via a GCPBackendPolicy. The Gateway routes to the ASM ingress gateway service, and from there HTTPRoute/VirtualService rules carry traffic into the mesh.

For identity, the chain is: KSA → IAM. Annotate (or bind) each service’s Kubernetes service account to the matching Google identity, then grant that identity only the roles it needs:

# config repo — payments service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments
  namespace: payments
  annotations:
    iam.gke.io/gcp-service-account: payments@PROJECT.iam.gserviceaccount.com

# Terraform — bind the KSA to the GSA, then grant least privilege
resource "google_service_account_iam_member" "payments_wi" {
  service_account_id = google_service_account.payments.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "serviceAccount:${var.project_id}.svc.id.goog[payments/payments]"
}
resource "google_project_iam_member" "payments_sql" {
  project = var.project_id
  role    = "roles/cloudsql.client"     # only payments gets this
  member  = "serviceAccount:${google_service_account.payments.email}"
}

Turn on the constraints/iam.disableServiceAccountKeyCreation org policy so the old escape hatch — downloading a JSON key — is closed entirely.

In-cluster config (the GitOps repo). The config repo holds namespaces, ResourceQuota, NetworkPolicy, the ASM PeerAuthentication/AuthorizationPolicy set, and each team’s Deployments/Services/HTTPRoutes. A RootSync reconciles the platform-wide objects; per-team RepoSync objects let teams own their namespace’s manifests without cluster-admin. The strict-mTLS and default-deny posture lives here, not in any pipeline:

# Mesh-wide: encrypt everything, then deny by default
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: istio-system }
spec: { mtls: { mode: STRICT } }
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: deny-all, namespace: payments }
spec: {}            # empty spec on a namespace = deny all by default
---
# Explicit allow: only checkout may POST to payments
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: allow-checkout, namespace: payments }
spec:
  selector: { matchLabels: { app: payments } }
  action: ALLOW
  rules:
  - from: [{ source: { principals: ["cluster.local/ns/checkout/sa/checkout"] }}]
    to:   [{ operation: { methods: ["POST"], paths: ["/charge"] }}]

A canary is then just a weighted VirtualService committed to Git and picked up by Config Sync — 95/5, watch the SLO dashboards, then 50/50, then 100 — with no privileged access required to shift traffic.

Enterprise considerations

Security and Zero Trust. This architecture is Zero Trust by construction, not by add-on. Every east-west call is mutually authenticated (mTLS) and explicitly authorised (default-deny + allowlist) by the mesh; every workload has its own least-privilege Google identity with no static keys; only attested images run; and the network is private (private nodes, private DB endpoints, authorized control-plane networks). Add Policy Controller (managed OPA Gatekeeper, reconciled by Config Sync) to enforce guardrails like “no :latest tags,” “all images from our Artifact Registry,” “every pod sets resource requests,” and Pod Security Admission at the restricted level. Secrets come from Secret Manager via the CSI driver or Workload-Identity-scoped reads — never baked into images. The result is defence in depth where each layer (image → identity → network → mesh authz) is an independent gate.

Cost optimisation. Autopilot’s per-pod billing flips the cost model: there is no node slack to waste, but there is no slack to hide in either, so over-requesting becomes visible spend. Run VPA in recommendation mode and tune requests to real usage; use Horizontal Pod Autoscaler on golden-signal or custom metrics so replica count tracks demand. Use Spot Pods for fault-tolerant batch/async work (large discount, can be evicted). Co-locate Artifact Registry with the cluster region to avoid cross-region image-pull egress. For an enterprise fleet, a single shared platform cluster per region with hard ResourceQuota per team namespace is dramatically cheaper than a cluster-per-team sprawl — and Autopilot makes that multi-tenancy safe because Google enforces node-level isolation. Commit to Committed Use Discounts once your steady-state request baseline is known.

Scalability. Three independent axes scale here. Pods scale via HPA. Services scale organisationally because the mesh and GitOps make adding a service a manifest change, not a platform project. Clusters/regions scale via the fleet: add a regional Autopilot cluster, register it to the same fleet, point Config Sync at the same repos, and it converges to an identical configuration — then add it as a backend to the global LB (or use Multi-Cluster Services to make cross-cluster service discovery transparent). The control planes (Autopilot’s, managed ASM’s) are Google’s problem to scale, not yours.

Reliability and DR (RTO/RPO). A single regional cluster already survives a zone failure with no action — control plane and pods span three zones. For region loss, run a second regional cluster in the fleet, fed by the same Git repos, so it is already running the identical configuration (RTO is bounded by how fast the global LB drains the failed region’s NEGs and your DB failover, typically minutes, not a rebuild). RPO is set by the data tier, never by Kubernetes: Cloud SQL cross-region replicas give an RPO of seconds; Spanner multi-region gives RPO = 0. The stateless mesh and the GitOps reconciliation mean the compute side of DR is “the cluster is already up and configured” — your DR drills become data-failover drills, which is exactly where the real risk lives.

Observability. ASM emits the four golden signals (latency, traffic, errors, saturation) per service automatically, so you get a service-topology and a per-edge latency/error view without instrumenting anything. Add Managed Service for Prometheus for app-specific metrics and Cloud Trace (sidecars propagate context) for distributed traces across the service graph. Define SLOs with multi-window burn-rate alerts on customer-facing services; alert on symptoms (SLO burn), not causes (CPU). Config Sync exposes its own sync status, so “is prod what the repo says” is a dashboard, not a guess.

Governance. Git is the audit log. Every production change — a new image digest, a policy, a quota, a traffic weight — is a reviewed, attributable commit, and Config Sync guarantees the cluster matches it (drift is auto-reverted). Binary Authorization’s enforcement decisions are written to Cloud Audit Logs, giving an auditor a literal record of every admission. Policy Controller makes organisational standards machine-enforced rather than wiki-documented. Together this is what turns “we use Kubernetes” into “here is the cryptographic and Git evidence that only reviewed, signed, scanned code runs, with least-privilege identity, in a configuration that matches the repo.”

Going deeper

The sections above are the architecture as you would present it to a team. This one is the architecture as you would defend it in a design review — the internals, the numbers, and the decisions that are expensive to reverse.

Autopilot vs Standard — where the line actually falls. Autopilot and Standard run the same Kubernetes and the same upper layers of this design: the mesh, Artifact Registry, Binary Authorization, Workload Identity, and Config Sync are byte-for-byte identical on both. The only thing that differs is who owns the nodes. Autopilot removes node pools entirely — you cannot SSH to a node, cannot run privileged pods or most host-level DaemonSets, hostPath and host networking are restricted, and pods must fit Autopilot’s allowed resource shapes (CPU in 0.25-vCPU increments, memory constrained to a per-vCPU band, a minimum request per pod). In exchange Google sizes, patches, scales, and secures the machines and bills you for the sum of your pod requests. Reach for Standard only when you have a concrete node-level requirement Autopilot cannot express: GPUs with custom drivers or specific topologies, Windows node pools, privileged agents (some security or storage DaemonSets), very large single pods, or per-node kernel tuning. If you cannot name that requirement, Autopilot is the default — and the hardening that still matters on it is covered in the companion GKE Autopilot production hardening guide.

VPC-native and IP planning — the decision you cannot cheaply undo. Every cluster here is VPC-native: pods get real, routable VPC IPs from alias-IP secondary ranges rather than an overlay. A cluster draws from three pools — the node subnet’s primary range (one IP per node), a pod secondary range, and a services secondary range. GKE hands each node a slice of the pod range sized for its max-pods-per-node; at the default of 110 pods that slice is a /24 (256 addresses, doubled for headroom). The trap is arithmetic: a /20 pod range is only about sixteen nodes’ worth of /24s, so a cluster you thought could scale to fifty nodes silently hits pod IP exhaustion at sixteen. Size the pod range for max_nodes × per-node_CIDR, size the services range for your maximum Service count, and give each cluster a unique /28 for the private control-plane endpoint. Do this for the whole fleet at once, from a reserved, non-overlapping RFC 1918 plan, because two clusters with overlapping pod ranges cannot be peered or meshed — and secondary ranges are painful to grow after the fact. Autopilot manages the node side for you but you still choose these ranges at creation:

gcloud container clusters create-auto platform-prod \
  --region us-central1 \
  --network prod-vpc --subnetwork prod-subnet \
  --cluster-ipv4-cidr /17 \          # pod range: ~32k pod IPs
  --services-ipv4-cidr /22 \         # ~1k Service IPs
  --enable-private-nodes \
  --master-ipv4-cidr 172.16.0.0/28   # unique per cluster

North-south and the mesh boundary — two control planes on one path. A request crosses two independently-managed routing layers, and knowing which owns what saves hours of debugging. The Global External Application Load Balancer, provisioned by the Gateway API, owns edge → gateway: anycast entry, TLS termination, the Cloud Armor WAF, and — because its backends are container-native NEGs pointing at pod IPs — cross-region failover by draining unhealthy NEGs. The mesh (Cloud Service Mesh, the current name for managed Anthos Service Mesh) owns gateway → service: mTLS, L7 authorization, retries, timeouts, and canary weights via VirtualService. For more than one cluster, Multi-Cluster Gateway plus Multi-Cluster Services (MCS) make a single global LB front NEGs in several regions and let a service in one cluster resolve a service in another as if local — the mechanics are the subject of the companion GKE Gateway API multi-cluster traffic management lesson. The direction of travel is ambient mode: a sidecar-less data plane that moves mTLS and L4 into a per-node proxy and only pays for an L7 proxy where you actually use L7 policy, which shrinks the per-pod cost discussed under trade-offs.

Workload Identity — the token exchange under the hood. No pod here holds a Google credential. When a pod calls a Google API, the GKE metadata server hands it a token minted for its Kubernetes service account; the cluster’s Workload Identity pool (PROJECT.svc.id.goog) federates that token through Google’s Security Token Service into a short-lived (minutes-long) access token scoped to exactly the bound identity. Two binding models exist: the classic one maps a KSA to a Google service account (roles/iam.workloadIdentityUser on the GSA for member serviceAccount:PROJECT.svc.id.goog[ns/ksa]), and the newer, flatter one grants IAM directly to the KSA principal (principal://…/subject/ns/…) with no intermediate GSA at all. Because the pool is federated, a KSA in the cluster’s project can be granted a role on a resource in a different project — cross-project access with zero exported keys. Turn on the iam.disableServiceAccountKeyCreation org policy and the old escape hatch (downloading a JSON key) is closed fleet-wide; the mechanics and migration are in the companion GKE Workload Identity deep dive.

Data and statefulness — keep state off the mesh’s critical path. The services in this design are stateless on purpose; the durable state lives in managed data services — Cloud SQL, Spanner, Memorystore, Pub/Sub, GCS — reached over private IP or Private Service Connect, never a public endpoint. That is not dogma; it is what lets the compute tier be cattle. When you genuinely must run stateful workloads in the cluster (an engine the managed menu does not cover), use StatefulSets backed by regional persistent disks or hyperdisk, because a zonal disk pins its pod to one zone and quietly defeats the regional cluster’s zone-survivability. Note too that database connections usually leave the mesh: mTLS applies to in-mesh pod-to-pod traffic, so egress to a Cloud SQL private IP is modelled with a ServiceEntry or excluded from sidecar capture, and connection pooling belongs in the app, not the sidecar.

Observability — golden signals for free, SLOs on symptoms. The mesh emits the four golden signals (latency, traffic, errors, saturation) and a live service topology per workload with zero app instrumentation; Managed Service for Prometheus scrapes your app metrics without you operating a Prometheus; Cloud Trace assembles distributed traces from the context the sidecars propagate — though intra-app spans still require the app to forward the trace headers. Build SLOs with multi-window, multi-burn-rate alerts and page on symptoms (SLO burn on a customer-facing path), never on causes (a CPU number). Two signals people forget are themselves first-class telemetry: Config Sync’s sync status answers “is prod what the repo says,” and Binary Authorization’s audit log records every blocked admission.

The landing zone — org, folder, project. This architecture sits inside a Google resource hierarchy, and the hierarchy is the security boundary. An Organization contains folders (by environment or business unit) containing projects — and the project is the unit of IAM, billing, quota, and blast radius. A typical shape: a shared platform project for Artifact Registry, Cloud KMS, and the config repo’s deploy identity; separate dev/staging/prod projects each hosting their cluster(s); a fleet tying the clusters together for mesh and Config Sync across project lines. Org policies — disable service-account keys, restrict public IPs, allow only images from your registry, pin resource locations — are set once at the org or folder level and inherit downward, so a new project is compliant the moment it exists. Networking is usually Shared VPC: the network lives in a host project and clusters attach from service projects, which keeps IP planning and firewall policy central. A per-team RepoSync lets a team own its namespace without cluster-admin:

apiVersion: configsync.gke.io/v1beta1
kind: RepoSync
metadata:
  name: repo-sync
  namespace: orders
spec:
  sourceFormat: unstructured
  git:
    repo: https://github.com/acme/config
    branch: main
    dir: teams/orders
    auth: gcpserviceaccount
    gcpServiceAccountEmail: reposync-orders@PROJECT.iam.gserviceaccount.com

Cost and resilience — the levers, quantified. Autopilot bills the sum of pod requests (rounded to allowed shapes) rather than provisioned nodes, so there is no node slack to waste — and none to hide in. The levers, roughly in order of impact: run VPA in recommendation mode and right-size requests to real usage (this alone routinely recovers 30–40% once historical over-requesting is stripped); use HPA on golden-signal or custom metrics so replicas track demand; move fault-tolerant batch/async tiers to Spot Pods (large discount, evictable); buy Committed Use Discounts once your steady-state request baseline is known; and prefer one shared platform cluster per region with per-team ResourceQuota over cluster-per-team, because Autopilot already enforces node-level workload isolation. On resilience, three axes stack: a regional cluster spreads control plane and nodes across three zones and survives a zone loss with no human action (spread replicas with topologySpreadConstraints so a single zone never holds a whole service); a second regional cluster in the fleet, already converged by the same Git repos, turns region loss into a load-balancer drain rather than a rebuild; and RPO is always set by the data tier — a Cloud SQL cross-region replica gives seconds, Spanner multi-region gives zero — never by Kubernetes. The compute side of DR is therefore “already up and converged,” so your real DR drills are data-failover drills.

Trade-offs, at depth. Sidecars are not free: expect a small added per-hop latency (sub-millisecond to low-single-digit milliseconds) and tens of megabytes of memory per pod for the Envoy proxy — real at a thousand pods, which is precisely what ambient mode targets. Autopilot’s guardrails (no privileged pods, restricted host access, fixed node shapes) are a feature for most and a wall for a few. And the honest cost is conceptual: a team must hold the mesh, GitOps reconciliation, Workload Identity, and Binary Authorization in their heads at once. If you have three stateless HTTP services and no mesh needs, this is overkill — the “When to use it” section below is the honest decision guide.

Reference enterprise example

Meridian Pay is a (fictional) B2B payments platform: 90 engineers across nine product teams, processing card and ACH transactions for mid-market merchants. They began on a single Heroku-style monolith, hit deployment contention (every team blocked on one release train) and a PCI audit that the monolith’s flat access model could not satisfy. They adopted this architecture over a quarter.

What they built. Two regional GKE Autopilot clusters — us-central1 (primary) and us-east4 (DR) — in a prod project, registered to one fleet, plus a smaller staging cluster. Managed ASM across all three. Twenty-three services at launch (gateway-api, merchant, ledger, card-auth, ach, risk, notifications, reporting, and so on), each in its own namespace owned by the responsible team via a RepoSync. Artifact Registry in us-central1 with on-push scanning and CMEK; Cloud Build pipelines that scan, test, push, and attest with a Cloud KMS key no human can use. Binary Authorization in ENFORCED_BLOCK_AND_AUDIT_LOG. Cloud Spanner (multi-region nam3) for the ledger — chosen specifically for RPO = 0 on money — and Cloud SQL for less critical service databases. Workload Identity for all twenty-three services; the iam.disableServiceAccountKeyCreation org policy on from day one.

Decisions and numbers. Their steady-state pod requests summed to roughly 180 vCPU and 360 GB across prod — about 40% less than the node capacity their old self-managed cluster ran, once VPA recommendations stripped years of accreted over-requesting (the monolith era had taught everyone to “ask for plenty”). Spot Pods ran the reporting and reconciliation batch tier at a steep discount. The mesh’s default-deny authz turned the PCI “segmentation” requirement into a reviewable set of AuthorizationPolicy files: the auditor could read, in Git, that nothing but card-auth could call the ledger write path, and that notifications had zero access to cardholder data services. The single most valuable artifact in the audit was Binary Authorization’s audit log — direct evidence that every running digest was signed by the post-scan attestor.

The incident that proved it. Four months in, a us-central1 zone had a networking degradation. The regional cluster shed the affected zone’s pods and rescheduled them in the other two zones automatically; the global LB drained the unhealthy NEGs. Customer-facing impact was a brief latency blip, no outage, no data loss — and no human ran a command. Separately, a contractor once tried to hot-patch a fix by pushing an unsigned image straight to the registry and applying it; the cluster refused admission, Binary Authorization logged the block, and the fix went through the pipeline an hour later as it should have. The platform team stayed at four people while the service count grew past forty in the following year, because adding a service was a pull request, not a project.

Outcome. Deployment contention vanished — teams shipped on their own cadence behind canary weights they controlled via Git. The PCI audit passed on the strength of machine-enforced segmentation and supply-chain evidence rather than narrative. And the org spent its SREs on product reliability and SLOs, not on patching nodes and chasing autoscaler bugs.

When to use it

Use this architecture when you have genuine microservices (multiple teams shipping independently), you want Kubernetes’ API and ecosystem without operating nodes, and you have real security/compliance pressure (payments, health, regulated data) that demands provable east-west isolation and a trusted supply chain. It shines precisely when the number of services and teams is the thing growing — the mesh and GitOps make that growth cheap, where ad-hoc Kubernetes makes it exponentially expensive.

Trade-offs to accept. There is real conceptual weight: a team must understand the mesh (mTLS, authz, VirtualService), GitOps reconciliation, Workload Identity, and Binary Authorization. Sidecars add a small per-request latency and per-pod memory cost (the managed data plane and ambient-style options narrow this, but it is non-zero). Autopilot trades flexibility for managed-ness — you cannot SSH to a node, run privileged DaemonSets freely, or pick arbitrary node shapes; workloads needing GPUs with exotic drivers, host-level access, or very large single pods may chafe. And the per-pod billing punishes sloppy requests, which is good discipline but a behaviour change.

Anti-patterns. Turning on mTLS but not authorization — encryption without a default-deny policy is not Zero Trust. Signing images before the scan/test gate — an attestation that proves nothing. Running Config Sync but still hand-applying kubectl to prod — you have reintroduced drift and destroyed the audit story. One cluster per team — multiplies cost and ops for no isolation benefit Autopilot doesn’t already give you via namespaces and quotas. A shared, over-privileged node service account instead of per-service Workload Identity — collapses your blast radius back to “any pod can do anything.”

Alternatives, honestly. If you have only a handful of stateless HTTP services and no mesh requirements, Cloud Run (see the companion GCP Global Web Application reference) is simpler and cheaper — no cluster to reason about at all; reach for GKE only when you need sidecars, gRPC streaming, stateful workloads, DaemonSets, or fine-grained mesh policy. If you want the same patterns across clouds and on-prem, the Anthos / GKE Enterprise fleet model extends this exact design to attached and bare-metal clusters. If you are firmly on GKE Standard for cost or node-control reasons, the mesh/registry/identity/GitOps layers of this article still apply unchanged — only the node-management story differs, and you take back the cluster-ops tax Autopilot was removing. The decision is rarely “Kubernetes vs not”; it is “how much of the platform do I want Google to run,” and this architecture is the answer for teams who want to run services, not machines.

Practice challenges

Work these top to bottom; they escalate from “can you read the architecture” to “can you design its hard edges.” No live cluster is needed — reason each through, then check.

1 (beginner) — Find the gate. A contractor pushes an unsigned image straight to Artifact Registry and tries to run it in prod. Name the layer that stops it and the two policy fields that make the block enforced.

<details> <summary>Solution</summary>

Binary Authorization, at admission. evaluationMode: REQUIRE_ATTESTATION (the image digest must carry a valid attestation) and enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG (unattested images are blocked and logged). Why: the cluster itself refuses the pod at admission, so no CI change or kubectl trick can smuggle it in. </details>

2 (beginner → intermediate) — mTLS is not Zero Trust. Write the mesh-wide manifest that forces STRICT mTLS, then say in one sentence why it alone is not Zero Trust.

<details> <summary>Solution</summary>

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

Why: STRICT encrypts and authenticates every hop, but any meshed service can still call any other — you need a default-deny AuthorizationPolicy plus explicit allows to get Zero Trust. Encryption answers “is this traffic private,” authorization answers “is this call allowed.” </details>

3 (intermediate) — Budget the pod IPs. You expect up to 60 nodes at the default 110 max-pods-per-node. What is the minimum size of the pod secondary range, and what breaks if you under-size it?

<details> <summary>Solution</summary>

GKE gives each node a /24 (256 pod IPs) at 110 max-pods. 60 nodes × 256 = 15,360 IPs, so you need at least a /18 (16,384 addresses). Under-size it — say a /20 (~16 nodes’ worth) — and the cluster hits pod IP exhaustion: new nodes cannot get a pod CIDR and the cluster silently stops scaling. Why: the pod range is fixed at creation and hard to grow, so plan for peak node count up front. </details>

4 (intermediate) — Least-privilege identity. The orders service (KSA orders in namespace orders) must read exactly one Pub/Sub subscription and nothing else. Sketch the Workload Identity wiring.

<details> <summary>Solution</summary>

Annotate the KSA to a dedicated GSA, bind the KSA → GSA trust, then grant only the narrow role on the subscription, not the project:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders
  namespace: orders
  annotations:
    iam.gke.io/gcp-service-account: orders@PROJECT.iam.gserviceaccount.com

# trust: KSA may impersonate the GSA
gcloud iam service-accounts add-iam-policy-binding \
  orders@PROJECT.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT.svc.id.goog[orders/orders]"
# least privilege: subscriber on ONE subscription
gcloud pubsub subscriptions add-iam-policy-binding orders-events \
  --role roles/pubsub.subscriber \
  --member "serviceAccount:orders@PROJECT.iam.gserviceaccount.com"

Why: one identity per service, scoped to one resource — orders cannot read any other subscription, and there is no key to leak. </details>

5 (advanced) — Design the canary. Send 5% of payments traffic to v2 via Git, and describe the rollout sequence and what gates each step.

<details> <summary>Solution</summary>

A DestinationRule defines the subsets, a VirtualService weights them; both are committed to the config repo and applied by Config Sync:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payments
  namespace: payments
spec:
  host: payments
  subsets:
    - name: v1
      labels: { version: v1 }
    - name: v2
      labels: { version: v2 }
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payments
  namespace: payments
spec:
  hosts: [payments]
  http:
    - route:
        - { destination: { host: payments, subset: v1 }, weight: 95 }
        - { destination: { host: payments, subset: v2 }, weight: 5 }

Sequence: commit 95/5 → watch the SLO / error-rate / latency dashboards → commit 50/50 → watch → commit 100/0 (or roll back by reverting the commit). Why: every traffic shift is a reviewed Git change, needs no privileged cluster access, and is trivially reversible. </details>

6 (advanced) — Region-loss DR, in order. us-central1 is lost. List, in order, what must happen for us-east4 to take over, and identify which step sets your RPO.

<details> <summary>Solution</summary>

1. The global LB’s health checks fail for the us-central1 NEGs and drain them.
2. Traffic routes to the already-running us-east4 cluster — which is already converged to the identical config via Config Sync, so no deploy happens.
3. The data tier fails over (promote the Cloud SQL cross-region replica, or rely on Spanner multi-region already spanning both regions).

RPO is set by step 3, the data failover — not by Kubernetes. Why: the compute side of DR is “already up and configured,” so your real DR risk (and your real DR drill) lives entirely in the data tier. </details>

Common beginner mistakes

These are misconceptions, not typos — each is a wrong mental model that leads a beginner to build the platform wrong. (For architectural anti-patterns in the same spirit, see “When to use it” above.)

• “Autopilot is just Standard with less control, so pick whichever.” Wrong framing. Autopilot is the right default for the vast majority of stateless and stateful services, but it genuinely misfits workloads that need host access, privileged DaemonSets, exotic GPUs/drivers, Windows, or per-node tuning — force those onto it and you spend your days fighting the guardrails. Right model: default to Autopilot; switch to Standard only for a named node-level requirement. The mesh, identity, and supply-chain layers are identical either way, so the choice is purely about who runs the nodes.
• “I’ll sort out IP ranges later — take the defaults.” The most expensive shortcut here. Pod and Service secondary ranges are chosen at creation and are painful to grow; a too-small pod range silently caps how many nodes the cluster can ever run, and overlapping ranges across clusters block peering and mesh. Right model: plan a non-overlapping IP budget for the whole fleet before the first create — max nodes × per-node pod CIDR, plus the Service range, plus a unique /28 per control plane.
• “Turn the mesh on everywhere from day one.” Mesh-by-default taxes every pod with sidecar latency and memory and taxes the team with conceptual load — including services that never need L7 policy. Worse, teams flip on STRICT mTLS, see “encrypted,” and believe they are done. Right model: add services to the mesh deliberately, and always pair mTLS with a default-deny AuthorizationPolicy; consider ambient mode to cut the per-pod cost where you only need encryption.
• “One zone is fine to start; I’ll go regional later.” A zonal cluster puts its control plane and nodes in a single zone, so one zone incident is a total outage — and you cannot convert a zonal cluster to regional in place. Right model: use a regional cluster from the start (control plane + nodes across three zones, survives a zone with no action) and spread replicas with topologySpreadConstraints.
• “Kubernetes cost is just node count, so Autopilot hides my spend.” Backwards. Autopilot bills the sum of pod requests, so an over-requested replicas: 10 asking 2 vCPU while using 0.3 is now a visible, recurring line item — the model exposes waste rather than hiding it in node headroom. Right model: treat the pod request as the unit of cost: run VPA in recommendation mode, right-size, scale with HPA, and push tolerant tiers to Spot Pods.

Glossary