GCP Microservices on GKE: A Production Reference Architecture

Every team that moves microservices to Google Cloud asks the same question: “we know Kubernetes — what does production-grade look like on GKE (Google Kubernetes Engine) specifically?” Kubernetes is the easy 40%. The other 60% is a chain of GCP-specific decisions: cluster mode, ingress path, image registry, keyless API auth, private database access, async messaging, secrets delivery, rollout staging, observability. Get them right and GKE is boring in the best way. Leave them to defaults-plus-tutorials and the cluster that worked in the demo fails at the first traffic spike, the first poison message, or the first leaked service-account key.

This article is the reference architecture I deploy by default in 2026: a regional GKE Autopilot cluster fronted by the Gateway API on Google’s global external Application Load Balancer, images in Artifact Registry, keyless access via Workload Identity Federation for GKE, one Cloud SQL instance and (where justified) one Memorystore cache per service, Pub/Sub for everything asynchronous, Secret Manager through the CSI add-on, Cloud Deploy canary rollouts, and Managed Service for Prometheus plus Cloud Trace for observability. Every component gets the why, the rejected alternatives, the limits, the gotchas, and working gcloud, YAML and Terraform.

By the end you can draw this architecture from memory, defend each choice in a design review (including “why not Cloud Run?” and “why no service mesh yet?”), stand up a working slice of it in an afternoon, and troubleshoot the eight failure modes that account for most GKE microservices incidents.

What problem this solves

Microservices on Kubernetes rarely fail because “Kubernetes was the wrong choice.” They fail because the platform decisions around the cluster were never made explicitly — each team improvised, and the improvisations disagree. One service authenticates with a mounted JSON key (which leaks), another shares a database with its neighbour (which couples deploys), a third sits in a five-hop synchronous chain (so one slow pod becomes a sitewide outage). Nobody owns ingress, so four LoadBalancer Services each carry a public IP and no WAF. No dead-letter queue exists anywhere, so the first malformed message spins a consumer at 100% CPU for a weekend.

The fix is a reference architecture: one documented, defensible default per decision, deviated from only with a written reason. GCP’s managed pieces — Autopilot, the Gateway controller, Workload Identity, the Cloud SQL connectors, managed Prometheus — remove entire categories of toil, but only when wired together the intended way.

Here is what ships when each decision is left unmade, and what it costs you:

If you recognise three or more rows, this reference architecture pays for itself the first quarter.

Learning objectives

By the end of this article you can:

• Justify GKE Autopilot as the default cluster mode — and name the signals that push you to Standard instead.
• Publish services through the Gateway API (gke-l7-global-external-managed) with TLS, Cloud Armor, health checks and weighted canary routing.
• Wire the image path: Artifact Registry repo design, cleanup policies, scanning, and how Autopilot nodes pull.
• Configure Workload Identity Federation for GKE both ways so no pod ever mounts a JSON key.
• Give each service its own Cloud SQL (Auth Proxy sidecar or language connectors) and Memorystore cache, all on private IP.
• Decouple services with Pub/Sub — ack deadlines, retry policy, dead-letter topics, ordering, exactly-once, and the IAM gotchas.
• Deliver secrets via Secret Manager + the CSI add-on, and ship releases with Cloud Deploy canaries driven by Gateway traffic splits.
• Autoscale with HPA on CPU and Pub/Sub backlog, observe with Managed Service for Prometheus and Cloud Trace, enforce NetworkPolicy, and articulate when Cloud Service Mesh earns its cost — and when Cloud Run beats GKE outright.

Prerequisites & where this fits

You should know core Kubernetes objects (Deployment, Service, namespace, HPA), have basic gcloud fluency and a billed project, and know GCP networking at the level of “a VPC is global, subnets are regional.” If GKE modes are new, read GKE Autopilot vs Standard: Which Google Kubernetes Engine Mode Should You Actually Use? first; if you are still choosing a container platform at all, start with GCP Cloud Run vs GKE vs Compute Engine: Choose the Right Compute. This article sits at the composition layer — it assumes each ingredient and shows how they assemble into one defensible system; deeper dives live in GCP IAM and Service Accounts: Roles, Bindings and Least Privilege and GCP Pub/Sub and Event-Driven Architecture: Decouple and Scale.

A map of the layers this architecture covers, and the division of labour on each:

Core concepts

Five mental models carry the whole design.

1. The cluster is a regional substrate, not the architecture. An Autopilot cluster is always regional: control plane and pods spread across three zones (control plane SLA 99.95%, Autopilot pods 99.9%). Treat it as a utility — no SSH, no node pools, no node upgrade windows. Everything architectural lives above it (routes, identities, data contracts) or beside it (databases, topics).

2. A “service” is a bundle, not a Deployment. Each microservice is a repeatable kit: one namespace, one Deployment (+ HPA + PodDisruptionBudget), one Kubernetes ServiceAccount (KSA) bound to one IAM identity, one HTTPRoute on the shared Gateway, its own Cloud SQL database, optionally its own Memorystore, and the Pub/Sub topics it owns — stamped out by Terraform plus a manifest template, so service #14 is provisioned exactly like service #4.

3. Identity is the security perimeter. Pods do not authenticate to GCP with keys; they are identities. Workload Identity maps namespace/ksa-name to an IAM principal, and IAM grants that principal exactly the roles its service needs (roles/cloudsql.client, roles/pubsub.publisher on one topic). Blast radius of a compromised pod = that service’s grants, nothing more.

4. On Autopilot, requests are the bill. Autopilot charges for the resources your pods request (vCPU, memory, ephemeral storage), not for nodes — and pins limits equal to requests (no burst headroom), with defaults of 0.5 vCPU / 2 GiB if you omit requests (often 4× what a small service needs). Right-sizing requests is simultaneously your performance tuning and your cost control; it is the #1 Autopilot surprise.

5. Two planes of communication. Synchronous traffic enters once, through one global load balancer, and fans out via HTTP routes; service-to-service synchronous calls are rationed (shallow call graphs). Everything that tolerates seconds of delay — order events, notifications, projections, webhooks — rides Pub/Sub. That one discipline eliminates most cascade failures before any mesh or circuit breaker enters the conversation.

The vocabulary you need, in one pass:

Cluster foundation: why Autopilot is the default

Start with the mode decision, because it shapes everything downstream. Autopilot is the default here; Standard is the exception you argue into, not out of.

Microservices are exactly what Autopilot was built for — many small, stateless, replicated pods with no node-level requirements. In Standard mode you own node pool sizing, machine types, upgrades, bin-packing and the cluster autoscaler’s moods; in a 12-service estate that is a part-time job producing zero product value. Autopilot deletes the job: nodes are invisible, patched by Google, provisioned to fit pending pods, billed per pod request. It also hardens by default — Shielded nodes, no SSH, Workload Identity and Dataplane V2 always on, Container-Optimized OS only — and enrols in a release channel (rapid/regular/stable/extended; pick regular for production) so control-plane and node versions move together inside your maintenance windows.

Signals that justify Standard: a privileged node-level agent, kernel/sysctl tuning, local-SSD topology needs, or a measured >70–80% sustained utilisation profile where machine-type CUDs beat Autopilot pricing. Absent those, take Autopilot (full catalogue: GKE Autopilot vs Standard vs Enterprise: Choose Your Kubernetes Mode).

Autopilot’s guardrails, though, are contractual — design your manifests around them:

Create the cluster (region asia-south1 used throughout; substitute yours):

gcloud container clusters create-auto prod-apps \
  --region=asia-south1 \
  --release-channel=regular \
  --network=prod-vpc --subnetwork=gke-snet \
  --enable-private-nodes \
  --gateway-api=standard

The same in Terraform:

resource "google_container_cluster" "prod" {
  name             = "prod-apps"
  location         = "asia-south1"
  enable_autopilot = true
  network          = google_compute_network.prod.id
  subnetwork       = google_compute_subnetwork.gke.id

  release_channel { channel = "REGULAR" }
  gateway_api_config { channel = "CHANNEL_STANDARD" }

  private_cluster_config { enable_private_nodes = true }
  ip_allocation_policy {}   # VPC-native; required, and default on Autopilot
}

Private nodes mean pods have no public IPs; egress to the internet goes via Cloud NAT on the VPC (budget it — NAT gateway plus per-GiB processing). Put the cluster in a Shared VPC service project in larger orgs (see GCP VPC and Shared VPC: Networking Across Projects).

Traffic in: Gateway API on the global load balancer

One entry point for all synchronous traffic. Not a LoadBalancer Service per team (IP sprawl, no L7), not legacy Ingress (single-owner object, annotation soup). The Gateway API is the current answer — and on GKE the GKE Gateway controller is a Google-managed control plane that programs real Cloud Load Balancing infrastructure from your CRDs, not an ingress controller you install. The resource model is deliberately role-split:

Pick the GatewayClass deliberately:

Why this beats the alternatives you might reach for first:

The platform team’s Gateway, and one team’s route:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external-gw
  namespace: gateway-infra
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - name: api-tls          # K8s TLS Secret; or Certificate Manager via
                               # annotation networking.gke.io/certmap: prod-map
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels: { shared-gateway-access: "true" }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders-route
  namespace: orders
spec:
  parentRefs:
  - name: external-gw
    namespace: gateway-infra
  hostnames: ["api.example.com"]
  rules:
  - matches:
    - path: { type: PathPrefix, value: /orders }
    backendRefs:
    - name: orders
      port: 80

Two essentials tutorials skip. First, the LB’s default health check probes / — define health explicitly:

apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: orders-hc
  namespace: orders
spec:
  targetRef: { group: "", kind: Service, name: orders }
  default:
    config:
      type: HTTP
      httpHealthCheck: { requestPath: /healthz, port: 8080 }

Second, attach Cloud Armor (WAF, rate limits, geo rules) and a sane backend timeout:

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: orders-backend
  namespace: orders
spec:
  targetRef: { group: "", kind: Service, name: orders }
  default:
    timeoutSec: 30
    securityPolicy: edge-armor-policy

Under the hood the controller creates NEGs holding pod IPs, so the LB routes to pods directly (container-native load balancing) — no kube-proxy second hop, and health checks see pod health. Verify with kubectl get gateway external-gw -n gateway-infra (PROGRAMMED: True plus the address) and gcloud compute backend-services list. A global ALB runs on forwarding-rule hours plus per-GiB processing — roughly $20–35/month (₹1,700–3,000) before traffic.

The image path: Artifact Registry to node

Images live in Artifact Registry (AR) — regional, IAM-scoped, scannable — at REGION-docker.pkg.dev/PROJECT/REPO/IMAGE:TAG. Keep the registry in the same region as the cluster for fast, egress-free pulls. Container Registry (gcr.io) is deprecated; Docker Hub adds rate limits and an external dependency exactly when you least want one (node scale-up mid-incident).

gcloud artifacts repositories create apps \
  --repository-format=docker --location=asia-south1 \
  --immutable-tags

gcloud auth configure-docker asia-south1-docker.pkg.dev
docker build -t asia-south1-docker.pkg.dev/$PROJECT_ID/apps/orders:v1.4.2 .
docker push  asia-south1-docker.pkg.dev/$PROJECT_ID/apps/orders:v1.4.2

resource "google_artifact_registry_repository" "apps" {
  repository_id = "apps"
  format        = "DOCKER"
  location      = "asia-south1"
  docker_config { immutable_tags = true }

  cleanup_policies {
    id     = "keep-recent"
    action = "KEEP"
    most_recent_versions { keep_count = 10 }
  }
}

Pull mechanics on Autopilot: nodes authenticate as the cluster’s node identity, which reads same-project AR by default; cross-project registries need roles/artifactregistry.reader on that identity — the classic cause of ImagePullBackOff ... 403 Forbidden. Two upgrades as you mature: Image streaming (mounts image data on demand, so large images start in seconds) and Binary Authorization (admission policy allowing only signed/attested images from your repos — closing the “deployed from a personal registry” hole).

Workload Identity: keyless access to Google APIs

The single most important security decision in this architecture: no service-account JSON keys, anywhere. Workload Identity Federation for GKE (always on in Autopilot, pool PROJECT_ID.svc.id.goog) lets a pod’s KSA obtain short-lived Google tokens via the GKE metadata server. Client libraries pick them up through Application Default Credentials with zero code changes.

Two wiring styles:

Classic style, end to end for the orders service:

gcloud iam service-accounts create orders-sa
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:orders-sa@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/cloudsql.client"
gcloud iam service-accounts add-iam-policy-binding \
  orders-sa@$PROJECT_ID.iam.gserviceaccount.com \
  --member="serviceAccount:$PROJECT_ID.svc.id.goog[orders/orders-ksa]" \
  --role="roles/iam.workloadIdentityUser"

kubectl create serviceaccount orders-ksa -n orders
kubectl annotate serviceaccount orders-ksa -n orders \
  iam.gke.io/gcp-service-account=orders-sa@$PROJECT_ID.iam.gserviceaccount.com

Terraform for the identity kit (this block is stamped per service):

resource "google_service_account" "orders" { account_id = "orders-sa" }

resource "google_service_account_iam_member" "orders_wi" {
  service_account_id = google_service_account.orders.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "serviceAccount:${var.project_id}.svc.id.goog[orders/orders-ksa]"
}

resource "google_project_iam_member" "orders_sql" {
  project = var.project_id
  role    = "roles/cloudsql.client"
  member  = "serviceAccount:${google_service_account.orders.email}"
}

Maintain a per-service role map (e.g. orders-sa → roles/cloudsql.client + publisher on orders.events only; payments-sa → subscriber on payments-orders-sub only; frontend-sa → nothing) and review it quarterly — every service, every grant, nothing wildcarded. Grant topic- and secret-level bindings (gcloud pubsub topics add-iam-policy-binding, gcloud secrets add-iam-policy-binding) rather than project-wide roles wherever the resource supports it. Verify from inside a pod with gcloud auth list in a debug container — if you see the node identity instead of your service’s, the KSA annotation or serviceAccountName is missing.

Data per service: Cloud SQL and Memorystore

One database per service is the rule that keeps microservices micro. Sharing a database couples schemas, deploy windows and failure domains; the moment two services join each other’s tables you have a distributed monolith. The default relational store is Cloud SQL for PostgreSQL — regional (HA) instances for anything customer-facing, private IP only (Private Service Connect or private services access; never --assign-ip). How pods connect is where teams stumble:

The sidecar pattern inside the orders Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  namespace: orders
spec:
  replicas: 3
  selector: { matchLabels: { app: orders } }
  template:
    metadata:
      labels: { app: orders }
    spec:
      serviceAccountName: orders-ksa
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector: { matchLabels: { app: orders } }
      containers:
      - name: orders
        image: asia-south1-docker.pkg.dev/myproj/apps/orders:v1.4.2
        ports: [{ containerPort: 8080 }]
        resources:
          requests: { cpu: 500m, memory: 1Gi }
        readinessProbe:
          httpGet: { path: /healthz, port: 8080 }
          periodSeconds: 5
        env:
        - name: DB_HOST
          value: "127.0.0.1"       # via the proxy sidecar
      - name: cloud-sql-proxy
        image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.14.0
        args: ["--port=5432", "myproj:asia-south1:orders-pg"]
        securityContext: { runAsNonRoot: true }
        resources:
          requests: { cpu: 100m, memory: 128Mi }

Provision the instance (Enterprise edition, regional HA; Enterprise Plus buys 99.99% SLA and near-zero-downtime maintenance when a service justifies it):

gcloud sql instances create orders-pg \
  --database-version=POSTGRES_16 --edition=enterprise \
  --tier=db-custom-2-8192 --region=asia-south1 \
  --availability-type=REGIONAL --no-assign-ip \
  --network=projects/$PROJECT_ID/global/networks/prod-vpc

Watch connection limits: each tier caps max_connections (a db-custom-2-8192 Postgres lands in the low hundreds by default), and 30 pods × a 10-connection pool = 300 connections = incident. Keep per-pod pools at 5–10 and add PgBouncer when replica counts climb.

For caching and sessions, Memorystore provides Redis-protocol stores on private IP (port 6379):

gcloud redis instances create orders-cache \
  --region=asia-south1 --tier=standard --size=1 \
  --redis-version=redis_7_0

Treat the cache as disposable: services must start and serve (degraded) with it empty or absent. If losing Redis loses data, it stopped being a cache.

Async between services: Pub/Sub

Synchronous chains are how microservices architectures die: an orders → payments → inventory → notify HTTP chain needs four services up and fast simultaneously — availability multiplies, latency adds. The rule: commands may be synchronous; facts are asynchronous. When orders commits an order it publishes order.created to a Pub/Sub topic; payments, inventory and notify each consume through their own subscription at their own pace. Pub/Sub is serverless, global and capacity-planning-free — see GCP Pub/Sub and Event-Driven Architecture: Decouple and Scale for the full pattern language.

The subscription is where reliability is won or lost. Every knob, with production values:

gcloud pubsub topics create orders.events
gcloud pubsub topics create orders.events.dlq

gcloud pubsub subscriptions create payments-orders-sub \
  --topic=orders.events \
  --ack-deadline=60 \
  --min-retry-delay=10s --max-retry-delay=600s \
  --dead-letter-topic=orders.events.dlq \
  --max-delivery-attempts=5 \
  --expiration-period=never

resource "google_pubsub_subscription" "payments_orders" {
  name  = "payments-orders-sub"
  topic = google_pubsub_topic.orders_events.id

  ack_deadline_seconds = 60
  expiration_policy { ttl = "" }              # never expire
  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "600s"
  }
  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.orders_events_dlq.id
    max_delivery_attempts = 5
  }
}

The IAM gotcha that breaks dead-lettering silently: the Pub/Sub service agent (service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com) needs roles/pubsub.publisher on the DLQ topic and roles/pubsub.subscriber on the source subscription — without them, messages never dead-letter and retry forever. Two more rules from scar tissue: make every consumer idempotent (Pub/Sub is at-least-once; dedupe on a business key) and alert on num_undelivered_messages — backlog growth is your earliest warning a consumer is sick.

Config, secrets, and Secret Manager CSI

Non-secret config (flags, endpoints, tuning) belongs in ConfigMaps rendered per environment by the delivery pipeline. Secrets do not belong in Kubernetes Secret objects committed to git — base64 is not encryption, and etcd is not an audit system. The system of record is Secret Manager: versioned, IAM-gated, CMEK-capable, audit-logged, at $0.06 per secret-version per location per month plus $0.03 per 10,000 accesses (rounding error at this scale).

GKE ships a managed Secret Manager add-on (built on the Secrets Store CSI driver) — no Helm charts to babysit:

gcloud container clusters update prod-apps \
  --region=asia-south1 --enable-secret-manager

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: orders-secrets
  namespace: orders
spec:
  provider: gke
  parameters:
    secrets: |
      - resourceName: "projects/PROJECT_NUMBER/secrets/orders-db-password/versions/latest"
        path: "db-password"
---
# pod spec: mount via a csi volume with
# driver secrets-store-gke.csi.k8s.io and
# volumeAttributes.secretProviderClass: orders-secrets

The pod’s KSA needs roles/secretmanager.secretAccessor on each named secret (not project-wide) — granted in the same Terraform stamp as the identity kit. Pin production mounts to a numbered version (versions/3) rather than latest so deploys, not secret pushes, control change; rotate by adding a version then rolling the Deployment.

Shipping changes: Cloud Deploy and rollout strategies

kubectl apply from laptops does not survive the second team. Cloud Deploy provides a managed delivery pipeline: CI (Cloud Build or GitHub Actions) pushes the image to AR and creates a release; Cloud Deploy renders manifests with skaffold, applies to staging, and promotes to prod behind an approval — optionally as an automated canary that shifts Gateway API traffic weights for you.

The pipeline definition (clouddeploy.yaml):

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: orders-pipeline
serialPipeline:
  stages:
  - targetId: staging
    profiles: [staging]
  - targetId: prod
    profiles: [prod]
    strategy:
      canary:
        runtimeConfig:
          kubernetes:
            gatewayServiceMesh:          # drives HTTPRoute weights — no mesh required
              httpRoute: orders-route
              service: orders
              deployment: orders
        canaryDeployment:
          percentages: [10, 50]
          verify: true

gcloud deploy apply --file=clouddeploy.yaml --region=asia-south1
gcloud deploy releases create rel-$(git rev-parse --short HEAD) \
  --delivery-pipeline=orders-pipeline --region=asia-south1 \
  --images=orders=asia-south1-docker.pkg.dev/$PROJECT_ID/apps/orders:v1.4.2

The verify: true step runs a container you define (smoke tests against the canary); failure aborts and re-weights traffic back automatically. Promotion sits behind a required approval, so “who released this and when” has a first-class answer. Keep migrations decoupled from rollouts — expand-migrate-contract, never “the deploy runs the migration and 10% of pods expect the new column.”

Scaling: HPA on Autopilot

Autopilot removes node autoscaling from your job description — when the HPA asks for more replicas, capacity appears. Your entire scaling surface is therefore two things: honest per-pod requests (section above) and well-chosen HPA signals.

Web tier on CPU, worker tier on backlog — the canonical pair:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders
  namespace: orders
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: orders }
  minReplicas: 3          # one per zone — never 1 in prod
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 65 }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # don't flap on troughs
---
# payments worker: scale on Pub/Sub backlog (needs the
# Custom Metrics Stackdriver Adapter installed once per cluster)
  metrics:
  - type: External
    external:
      metric:
        name: pubsub.googleapis.com|subscription|num_undelivered_messages
        selector:
          matchLabels:
            resource.labels.subscription_id: payments-orders-sub
      target: { type: AverageValue, averageValue: "100" }

Three Autopilot-specific rules. First, because limits = requests, an undersized CPU request throttles the container at exactly p95 load — mystery latency; set requests from load tests. Second, scale-up needing fresh nodes takes tens of seconds to minutes: keep minReplicas covering baseline and, for instant headroom, run low-priority balloon pods (a PriorityClass below default — evicted the moment real pods need room, pre-warming capacity). Third, always ship a PodDisruptionBudget (minAvailable: 2); Autopilot honours it during continuous node upgrades, which otherwise will eventually drain both your replicas at 03:00.

The mesh question, network policy, and east-west security

Do you need a service mesh? For most estates under ~15 services: no — and on Autopilot the cost is unusually visible, because sidecar requests are billed. A default proxy footprint of 100m CPU / 128 MiB across 300 pods is ~30 vCPU and ~38 GiB of billed requests — roughly $1,100+/month (₹90k+) before any benefit, plus an upgrade and debugging surface. Adopt Cloud Service Mesh (Google’s managed Istio-based mesh — fleet-attached, managed control and data plane) when specific requirements arrive, not as a rite of passage:

Whether or not a mesh ever arrives, NetworkPolicy goes in on day one. Autopilot runs Dataplane V2 (eBPF), so enforcement is built in — no Calico install. Default-deny each namespace, then allow the specific edges of your call graph:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-ingress
  namespace: payments
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels: { kubernetes.io/metadata.name: orders }
    ports:
    - { protocol: TCP, port: 8080 }

Remember the LB: Gateway health checks and traffic arrive from Google’s proxy ranges via NEGs, so your default-deny must still admit the LB’s health-check source ranges (130.211.0.0/22, 35.191.0.0/16) to serving ports — the classic “everything worked until we added NetworkPolicy, now the Gateway says backends unhealthy” incident.

Observability: Managed Prometheus, Cloud Trace, Cloud Logging

You need all three signals wired before the first incident, not after. The GCP-native stack means zero self-hosted observability infrastructure:

Managed collection is enabled by default on current Autopilot clusters — you declare scrape targets with a CRD and Google runs the collectors:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: orders-metrics
  namespace: orders
spec:
  selector:
    matchLabels: { app: orders }
  endpoints:
  - port: metrics
    interval: 30s

Traces do not appear by magic: instrument with OpenTelemetry and export to Cloud Trace (direct exporter or an OTel collector Deployment). Propagate context on both planes — traceparent headers for sync calls, Pub/Sub message attributes for async hops — or every trace dies at the first topic. With that in place, Cloud Trace answers “which of the six hops added 900 ms” in one click. Wire the golden alerts on day one: per-route p99 and 5xx burn rates at the Gateway, num_undelivered_messages per subscription, DLQ depth > 0, Cloud SQL connections >80% of max, HPA maxReplicas saturation. The broader tooling story is in GCP Cloud Monitoring and Operations: Observability Built In.

When GKE is wrong: the Cloud Run decision

Honest architects run this table before every new system. Cloud Run deploys the same containers with far less platform to own — if your workload fits its model, GKE is over-engineering (see Cloud Run Explained: Serverless Containers That Scale to Zero).

The honest middle path many orgs land on: Cloud Run for spiky/simple edge services, one GKE estate for the interconnected core — both behind the same global load balancer, the same Artifact Registry, the same keyless IAM discipline. This reference architecture is for the day the core outgrows Cloud Run’s model.

Architecture at a glance

Read the diagram left to right, the direction a request travels. Clients resolve one anycast IP and hit the global external Application Load Balancer, declared as a Gateway (gke-l7-global-external-managed) with Cloud Armor attached; TLS terminates there (1). Each team’s HTTPRoute maps its path to its Service, and NEGs deliver straight to pod IPs inside the regional GKE Autopilot cluster, where every service runs with its own KSA — Workload Identity makes each GCP call keyless (2) — sized by honest requests and scaled by HPA (3). Each service owns its Cloud SQL database, reached privately through the Auth Proxy sidecar (4), and its Memorystore cache. Facts flow asynchronously through Pub/Sub with dead-letter queues guarding every subscription (5), while Managed Prometheus, Cloud Trace and Cloud Logging watch all of it — and the mesh remains a deliberate, deferred decision (6).

What the diagram deliberately does not show is also the point: no self-managed ingress controllers, no node pools, no Prometheus servers, no key files. Every box you don’t see is toil this architecture deleted.

Real-world scenario: Navikart’s flash-sale quarter

Navikart, a Bengaluru marketplace (fictional, assembled from three real engagements), ran a Django monolith on Compute Engine MIGs. Ahead of festive season they split the core into 12 services on this architecture: one Autopilot cluster in asia-south1, Gateway API on the global ALB, per-service Cloud SQL Postgres 16, Memorystore Standard for sessions, Pub/Sub for order events, Cloud Deploy canaries. Baseline: ~250 RPS; flash-sale target: 4,000 RPS for 90 minutes.

The first load test failed impressively: checkout p99 went from 180 ms to 4.2 s at barely 900 RPS — with CPU utilisation graphs looking healthy (~60%). The team had copied 250m CPU requests from a blog post; Autopilot pins limits to requests, and per-pod CPU throttling (visible in container_cpu_cfs_throttled_periods_total; averages hid it) was strangling bursts. Fix: load-test each service, re-request checkout at 1 vCPU / 1.5Gi, drop three over-provisioned internal services to 100m/256Mi. Net bill change: +4%. p99 back to 210 ms at 4,000 RPS.

The second failure was self-inflicted a week before the sale: a marketing service published a malformed order.created event (string total, not paise integer). The invoices consumer threw — and with the default immediate retry policy and no DLQ, Pub/Sub redelivered the message thousands of times an hour; HPA dutifully scaled the consumer to maxReplicas, burning ₹6,000 of compute overnight fighting one poisoned message. The fix is what this article makes mandatory: exponential backoff (10 s → 600 s), dead-letter topics with 5 attempts on all 14 subscriptions, a DLQ-depth alert, and schema validation that acks-and-parks unparseable messages.

Sale day was quiet. HPA took checkout from 6 to 44 pods over four minutes (balloon pods absorbed the first 30 seconds); Autopilot provisioned capacity with no human involvement; Cloud Armor rate rules blunted a scripted bot spike before it touched a pod. One canary that week — a payments release with a connection leak — was caught at the 10% stage by the verify step watching p99 and rolled back in 40 seconds; nobody paged.

The quarter’s platform bill: ~₹1.9L/month at peak season (cluster compute ~₹78k, Cloud SQL fleet ~₹66k, Memorystore ~₹12k, LB + NAT + Pub/Sub + observability the rest) — about 22% less than the over-provisioned MIG estate it replaced, with node ops eliminated. Their retro’s one-liner is the article’s thesis: “Kubernetes was fine; the defaults were the project.”

Advantages and disadvantages

The pattern in the right-hand column: this architecture trades operational burden for design discipline. Everything hard about it is a decision or a convention, not a 03:00 page — which is exactly the trade a platform should make.

Hands-on lab: a two-service slice in ~45 minutes

Free-tier-friendly: the $74.40/month GKE credit covers one cluster’s management fee, pod requests are tiny, images are Google’s samples, and teardown is complete. An hour of this costs a few rupees.

1. Project + APIs.

export PROJECT_ID=$(gcloud config get-value project)
gcloud services enable container.googleapis.com \
  artifactregistry.googleapis.com monitoring.googleapis.com

2. Create the Autopilot cluster (Gateway API on). Takes ~5–7 minutes.

gcloud container clusters create-auto lab-apps \
  --region=asia-south1 --release-channel=regular --gateway-api=standard
gcloud container clusters get-credentials lab-apps --region=asia-south1
kubectl get gatewayclass   # expect gke-l7-global-external-managed listed

3. Deploy two “services”.

kubectl create namespace store
kubectl -n store create deployment web --replicas=2 \
  --image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
kubectl -n store create deployment api --replicas=2 \
  --image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0
kubectl -n store expose deployment web --port=80 --target-port=8080
kubectl -n store expose deployment api --port=80 --target-port=8080

Watch Autopilot provision: kubectl get pods -n store -w shows Pending while nodes are created (~60–90 s), then Running.

4. One Gateway, two routes. Save and kubectl apply -f gateway.yaml:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: lab-gw, namespace: store }
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
  - { name: http, protocol: HTTP, port: 80 }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: web-route, namespace: store }
spec:
  parentRefs: [{ name: lab-gw }]
  rules:
  - matches: [{ path: { type: PathPrefix, value: / } }]
    backendRefs: [{ name: web, port: 80 }]
  - matches: [{ path: { type: PathPrefix, value: /api } }]
    backendRefs: [{ name: api, port: 80 }]

5. Test through the global LB. Programming takes 3–5 minutes the first time.

kubectl -n store get gateway lab-gw \
  -o jsonpath='{.status.addresses[0].value}'   # note the IP (e.g. 34.x.x.x)
curl http://34.x.x.x/        # → "Hello, world! Version: 1.0.0 ..."
curl http://34.x.x.x/api     # → "Hello, world! Version: 2.0.0 ..."

Early no healthy upstream or resets mean the LB/NEGs are still programming — wait; kubectl describe gateway lab-gw -n store shows conditions.

6. Add an HPA and watch Autopilot follow.

kubectl -n store autoscale deployment web --cpu-percent=60 --min=2 --max=10
# generate load from a throwaway pod:
kubectl -n store run hey --rm -it --image=busybox -- \
  /bin/sh -c 'while true; do wget -qO- http://web >/dev/null; done'
kubectl -n store get hpa,pods -w   # replicas climb; new nodes appear invisibly

7. A canary in one line. Edit web-route to split traffic — this is the primitive Cloud Deploy automates:

    backendRefs:
    - { name: web, port: 80, weight: 90 }
    - { name: api, port: 80, weight: 10 }

for i in $(seq 1 20); do curl -s http://34.x.x.x/ | head -1; done — roughly two of twenty responses now say Version 2.0.0.

8. Teardown (do not skip — the LB bills hourly).

kubectl delete namespace store            # deletes Gateway → deprovisions the LB
gcloud container clusters delete lab-apps --region=asia-south1 --quiet

Common mistakes & troubleshooting

The playbook for the failure modes this architecture actually produces:

Rows 1–4 account for most first-quarter pain; rows 9–10 for the most expensive nights. Print the table.

Best practices

• Stamp services from a template (Terraform module + manifest chart): namespace, KSA, IAM kit, HTTPRoute, HPA, PDB, PodMonitoring, DLQ’d subscriptions. Consistency is the architecture.
• Set requests from load tests, not folklore — on Autopilot they are your limits, bill and latency at once. Re-measure quarterly.
• Three replicas minimum, zone-spread, with a PDB on the request path; Autopilot upgrades nodes continuously and will eventually drain any two pods that let it.
• One Gateway, many HTTPRoutes: platform team owns Gateway/TLS/Cloud Armor; service teams own routes via allowedRoutes.
• Keyless or it doesn’t ship: CI rejects manifests mounting SA keys; Workload Identity is the only path to Google APIs.
• Every subscription gets exponential backoff + DLQ + a depth alert — the first poison message is a when.
• Per-service data, private IP only; IAM DB auth where supported, small per-pod pools, PgBouncer before you need it.
• Deploy through the pipeline, promote with approvals, canary with verify — laptops lose prod kubectl apply on day one.
• Immutable tags + cleanup policies + scanning in Artifact Registry; Binary Authorization once CI signs images.
• Instrument before incident: PodMonitoring per service, OTel context through Pub/Sub attributes, golden alerts as code.
• Write the mesh ADR now, adopt later — record trigger conditions (L7 authz, compliance mTLS, 30+ services) so nobody installs Istio out of boredom.

Security notes

Least privilege is structural: one KSA per service mapped to one IAM identity with resource-scoped grants (topic-level publisher, secret-level accessor) — audit the role map quarterly and alert on any roles/*.admin binding to a workload identity. Nodes are private (egress via Cloud NAT, optionally through a Secure Web Proxy allowlist); data services expose private IPs only; the sole public surface is the Gateway, where Cloud Armor enforces WAF, rate and geo policy in front of TLS 1.2+ (pinned via an SSL policy on GCPGatewayPolicy). East-west, default-deny NetworkPolicy encodes the call graph — a compromised pod reaches exactly its declared dependencies, and Dataplane V2 flow logs record the attempts. Supply chain: images only from your AR repos, scanned on push, tags immutable, Binary Authorization refusing anything unsigned. Secrets live in Secret Manager with per-secret IAM and Cloud Audit Logs, so “who read the DB password and when” is a query, not a shrug. Everything at rest is encrypted by default; add CMEK only where a regulator demands key custody — revoking a key is an outage with a compliance certificate.

Cost & sizing

What actually drives the bill, in the order it usually surprises people:

A worked “12 services, modest production” example — 36 always-on pods averaging 250m/512Mi (~10 vCPU / 20 GiB including HPA peaks, ≈$390), four Postgres instances of which two HA (≈$640), one Standard Memorystore GiB (≈$75), shared LB + NAT (≈$70), half a TiB of Pub/Sub (≈$20), curated observability (≈$60) and Artifact Registry (≈$12) — lands around $1,270/month ≈ ₹1.08L at ₹85/$, with the cluster management fee absorbed by the free credit.

Numbers are list-price, region-dependent and conservative — the point is the shape: databases usually out-cost the cluster, unset requests are the fastest way to double the compute line, and observability is cheap until cardinality isn’t. Review the requests-vs-usage gap (GMP has the data) monthly — the highest-yield FinOps ritual on Autopilot.

Interview & exam questions

Relevant to the Professional Cloud Architect and Professional Cloud DevOps Engineer exams, and to real design reviews:

1. Why choose GKE Autopilot over Standard for a microservices platform? Autopilot removes node management (provisioning, patching, bin-packing, upgrades), bills per pod request rather than per node, forces security best practice (Shielded nodes, Workload Identity, Dataplane V2) and carries a pod-level SLA. Standard remains for node-level agents, privileged workloads, kernel tuning, or proven high-utilisation fleets where machine-type CUDs win.

2. How does traffic reach a pod from the internet in this architecture? Client → anycast IP of the global external Application Load Balancer, provisioned by the GKE Gateway controller from a Gateway of class gke-l7-global-external-managed. TLS and Cloud Armor apply at the edge; an HTTPRoute matches host/path; NEGs route directly to pod IPs (container-native), skipping any node-port hop.

3. Explain Workload Identity Federation for GKE in two sentences. The cluster’s workload pool (PROJECT_ID.svc.id.goog) lets a pod exchange its Kubernetes ServiceAccount token for Google credentials — either as a directly-bound IAM principal or by impersonating a Google service account. No JSON keys exist, tokens are short-lived, and IAM scopes each service to its own resources.

4. A pod calls Pub/Sub and gets 403 despite the KSA annotation. What do you check? That the Deployment sets serviceAccountName (default KSA is the classic miss); that the GSA has roles/iam.workloadIdentityUser for the svc.id.goog[ns/ksa] member; and that the GSA holds the Pub/Sub role on the right resource. Confirm from inside the pod (gcloud auth list) which identity it actually got.

5. When do messages fail to dead-letter even though a DLQ is configured? When the Pub/Sub service agent lacks roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the source subscription — delivery attempts never convert into DLQ writes and the message retries indefinitely. Checkable via the subscription’s warning banner or IAM inspection.

6. How would you run a canary release without a service mesh? Gateway API HTTPRoute supports weighted backendRefs; Cloud Deploy’s canary strategy shifts weights (10% → 50% → 100%) between stable and canary Deployments, running a verify job at each step and re-weighting to zero on failure. A mesh only becomes necessary for per-user/header stickiness or traffic mirroring.

7. What does “limits equal requests” on Autopilot imply for performance tuning? No burstable headroom: a container hitting its CPU request is CFS-throttled even on an idle node. Requests must be set to measured p95 needs, and throttling metrics — not utilisation averages — are the signal for undersizing.

8. How do you autoscale a Pub/Sub consumer deployment? HPA with an External metric on pubsub.googleapis.com|subscription|num_undelivered_messages filtered to the subscription (via the Custom Metrics Stackdriver Adapter), targeting an average backlog per pod. Autopilot supplies node capacity automatically; pair with Spot pods for cost.

9. Cloud Run or GKE for a new 5-service product with spiky traffic and no platform team? Cloud Run: scale-to-zero matches spiky traffic, there is no cluster fee or pod sizing, and nothing needs Kubernetes-only primitives. GKE earns its keep with 10+ interconnected services, long-lived or stateful workloads, CRDs/operators, NetworkPolicy graphs and a team to own the platform.

10. What are the first six alerts you’d configure for this platform? Gateway 5xx-ratio and p99 burn rates per route; num_undelivered_messages growth; DLQ depth > 0; Cloud SQL connections vs max_connections (>80%); HPA at maxReplicas sustained; pod CPU-throttling rate. Together they catch edge failures, async rot, connection exhaustion and undersizing before users do.

Quick check

1. Which GatewayClass provisions the internet-facing global load balancer used here?
2. On Autopilot, what happens if a container specifies no resource requests?
3. Name the two IAM grants the Pub/Sub service agent needs for dead-lettering to work.
4. Why must Cloud SQL per-pod connection pools stay small?
5. What CRD tells Managed Service for Prometheus to scrape your pods?

Answers

1. gke-l7-global-external-managed.
2. Autopilot applies defaults (0.5 vCPU / 2 GiB) and bills them; limits are pinned to requests either way.
3. roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the source subscription.
4. Total connections = pods × pool size, and each Cloud SQL tier caps max_connections; HPA scale-outs otherwise exhaust the cap and new pods get connection refused.
5. PodMonitoring (namespace-scoped; ClusterPodMonitoring for cluster-wide).

Glossary

• Autopilot: GKE mode where Google operates nodes; you request pod resources and pay for them.
• Gateway API: Ingress successor (Gateway, HTTPRoute, policy CRDs) with role separation and traffic weights.
• NEG (network endpoint group): LB backend of pod IP:port pairs — container-native load balancing.
• Cloud Armor: Edge WAF/DDoS/rate-limit policy attached to LB backends.
• Artifact Registry: Regional artifact store at REGION-docker.pkg.dev; scanning, cleanup, per-repo IAM.
• Workload Identity Federation for GKE: KSA-to-IAM token exchange giving pods keyless Google credentials.
• KSA / GSA: Kubernetes ServiceAccount / Google (IAM) service account — WI links the two.
• Cloud SQL Auth Proxy: Sidecar tunnelling DB connections over mTLS with IAM checks (server port 3307).
• Memorystore: Managed Redis/Valkey (Basic, Standard-HA, Cluster) on private IP.
• Dead-letter topic (DLQ): Pub/Sub topic receiving messages after N failed deliveries — the poison-pill fuse.
• Secret Manager CSI add-on: GKE-managed driver mounting Secret Manager versions as pod files.
• Cloud Deploy: Managed delivery pipelines rendering with skaffold, promoting releases, canary-capable.
• Managed Service for Prometheus (GMP): Google-run Prometheus collection/storage queried with PromQL.
• Dataplane V2: GKE’s eBPF dataplane; native NetworkPolicy enforcement and flow logs.
• Cloud Service Mesh: Google’s managed Istio-based mesh — mTLS, L7 authz, traffic policy, telemetry.

Next steps

• Settle the cluster-mode decision in depth: GKE Autopilot vs Standard: Which Google Kubernetes Engine Mode Should You Actually Use? and the enterprise view in GKE Autopilot vs Standard vs Enterprise: Choose Your Kubernetes Mode.
• Pressure-test whether you need this platform at all: GCP Cloud Run vs GKE vs Compute Engine: Choose the Right Compute.
• Go deeper on the async backbone: GCP Pub/Sub and Event-Driven Architecture: Decouple and Scale.
• Harden the identity layer with GCP IAM and Service Accounts: Roles, Bindings and Least Privilege.
• Build out the day-2 view with GCP Cloud Monitoring and Operations: Observability Built In.