Capstone: A Production Multi-Cloud GitOps Platform Across AKS, EKS & GKE

Every lesson in this course built one part. You learned the pull model and what Argo CD is and is not, wrote a first Application, sourced it from Helm and Kustomize, registered a second cluster, templated dozens of apps with an ApplicationSet, gated releases with Argo Rollouts, locked the whole thing down with projects, RBAC and hardening, and learned to rebuild it all from Git after a disaster. This lesson is where every one of those parts snaps together into the single machine the course promised on day one: a production GitOps platform that runs one organization’s applications across AKS, EKS and GKE at the same time, from one Git source of truth, operated by one small platform team.

This is a capstone, so it is deliberately different from every lesson before it. We will not re-teach ExternalSecrets or the Rollout CRD field by field — you know those, and each phase links back to the lesson that owns it. Instead we synthesize: we take a concrete organization with real constraints, make the architectural decisions in the order you would actually make them, and stand the platform up in eight phases, showing the one critical manifest that anchors each phase and the reasoning that ties it to the rest. By the end you will have a mental model complete enough to walk into a design interview, sketch this platform on a whiteboard, and defend every decision — and a build runbook concrete enough to actually execute.

The honesty note this whole course runs on. There is no live three-cloud fleet attached to this lesson, so every command output below is labelled representative — it shows the shape you will see, not a transcript of a specific run. Every manifest is schema-correct for Argo CD 2.13+/3.x on Kubernetes 1.29+, with real fields, real ports and real API versions; every Terraform block uses real provider resources. Secrets are always placeholders. The one thing you must supply is three real clusters and the willingness to run the phases in order.

---

Why this matters

Most teams reach multi-cloud by accident, not by design. An acquisition brings an AWS estate into an Azure shop; a data-residency law forces one region onto a cloud you do not otherwise use; a big customer demands their workload run on their cloud. Suddenly you are operating three control planes, three ways to deploy, three on-call runbooks, and three copies of every mistake. The promise of GitOps — and the reason this course spent thirty-some lessons getting here — is that the deployment plane can be unified even when the infrastructure plane cannot. Argo CD does not care whether a target cluster is AKS, EKS or GKE; it reconciles Git into a Kubernetes API server, and all three expose the same API. The clouds differ only at the edges — identity, ingress, secrets, registries, load balancers — and this course has taught you exactly those edges, one at a time. The capstone is proving you can hold all of them in one design without the seams showing.

The stakes are concentration. A unified platform is a single pane of glass and a single point of leverage: one commit ships to three clouds, one platform team supports fifty app teams, one dashboard shows the health of the entire estate. That same concentration is a single blast radius — one bad ApplicationSet edit can break AKS, EKS and GKE in the same reconcile tick, and one compromised hub is cluster-admin on your entire fleet. Every serious decision in this lesson is really a decision about where to concentrate power and where to bound it: the topology that puts risk where you can live with it, the guardrails that keep self-service from becoming self-destruction, the DR plan that survives losing the hub itself.

The mental model to carry through all eight phases is a layer cake. Terraform owns the substrate (clusters, identities, networking) — the things that must exist before Argo CD can run. Argo CD owns everything on top (the platform layer, the apps, the policies) — the things that should be reconciled from Git, not clicked into existence. The line between them is the handoff, and getting that line in the right place is the difference between a platform you can rebuild in an afternoon and one nobody dares touch. We draw that line explicitly in Phase 1 and never cross it again.

---

The organization and its requirements

Meet Aurora Health, a fictional but plausible global digital-health SaaS. Aurora stores and processes patient data, which means the law decides where its data may live — and the law does not care about Aurora’s cloud preferences. Three regulatory regimes force three clouds:

Aurora cannot consolidate onto one cloud without breaking the law in two regions. Multi-cloud is not a preference here; it is a hard requirement, which is exactly the situation where a unified GitOps plane pays for itself. The organizational shape is the other half of the problem:

Four app teams today, a plan for twenty, and one platform team that must not become a ticket queue. That last constraint — self-service — shapes more of the design than any other. If onboarding a new app team means a platform engineer hand-writing Applications, the platform does not scale past its founders. The non-functional requirements make the constraints explicit and testable:

Hold these six requirements in mind — every architectural choice below is traceable to one of them, and the interview questions at the end are mostly “which requirement drove which decision.”

---

The architecture: hub-and-spoke across three clouds

The first and most consequential decision is topology: where does Argo CD run, and how many of it are there? This course covered the trade-off in depth in One Hub, Many Clouds; here we make the call for Aurora and move on.

Aurora runs hub-and-spoke: a small, dedicated management cluster hosts one highly-available Argo CD hub, and that hub manages three spokes — the AKS, EKS and GKE clusters that run the actual apps. We run the management cluster as an AKS cluster in a locked-down Azure management subscription, but note carefully: the hub is cloud-agnostic — it could equally be EKS or GKE. We put it on Azure because Aurora’s primary entity is there and it keeps the platform team’s blast-radius auditing in one cloud’s IAM; nothing about the design depends on that choice. The management cluster runs only platform tooling (Argo CD, its Redis HA, backups) and no Aurora application — a dedicated management cluster is the cleanest way to keep the control plane’s failure domain separate from the workloads it controls, and it is what makes the DR story in Phase 8 clean.

Here is the whole platform in one picture. Read it left to right: the platform monorepo and team repos on the left are the source of truth; the HA Argo CD hub and its ApplicationSets generate the fleet; an identity-and-policy gate decides who and what may deploy; the three cloud spokes each run the same platform stack behind their own cloud edge; and observability plus DR wrap the whole thing.

The five zones map exactly to how the platform is built and operated. Source is Git — the only place desired state lives. Mgmt Hub is the single control plane. Identity + Policy is the gate that turns “the hub can deploy anything anywhere” into “each team deploys its own thing, to its own place, only if it is signed.” Fleet Spokes are the three clouds, each running an identical platform stack behind a different edge. Operate is how you know it is working and how you get it back when it is not. The eight build phases below walk this diagram from left to right.

The phases build these layers bottom-up: substrate first (Phase 1), then the control plane and its identity (Phases 1–2), then the platform layer and its secrets (Phases 3–4), then the apps and their guardrails (Phases 5–6), then operations and survival (Phases 7–8).

---

The repository structure: platform monorepo + team repos

Before any cluster exists, decide where desired state lives — because everything else points at it. This course weighed monorepo against polyrepo at scale; Aurora uses a hybrid that gets the best of both, and it is the layout most mature platforms converge on.

platform-gitops/                     # ONE monorepo — Platform Engineering owns it
  bootstrap/
    root-app.yaml                    # the single hand-applied Application
    project-platform.yaml            # the AppProject the root app runs under
  appprojects/                       # one AppProject per app team
    team-records.yaml
    team-scheduling.yaml
    ...
  applicationsets/
    platform-ingress.yaml            # cluster generator → ingress on every spoke
    platform-observability.yaml      # Prometheus/Grafana on every spoke
    platform-eso.yaml                # External Secrets Operator on every spoke
    platform-rollouts.yaml           # Argo Rollouts controller on every spoke
    apps-records.yaml                # matrix: records' apps × its clusters
    ...
  platform/                          # the platform layer's charts/values
    ingress/{values.yaml,values-aks.yaml,values-eks.yaml,values-gke.yaml}
    observability/ ...
  policies/                          # Kyverno ClusterPolicies, NetworkPolicies

records-config/                      # ONE repo per app team — the team owns it
  base/                              # Helm chart ref or Kustomize base
  overlays/
    dev/       { values.yaml, rollout patch }
    staging/   { values.yaml }
    prod/      { values.yaml }       # prod = the correct cloud, chosen by label

The division of ownership is the whole point:

The rule that keeps this honest: the platform monorepo can reference a team’s config repo, but a team’s config repo can only ever affect that team. An app team merging a PR in records-config changes only records apps, on only the clusters its AppProject allows, because the app-delivery ApplicationSet and the team-records AppProject bound it. A platform engineer merging in platform-gitops can change the whole fleet — which is why that repo has the strictest review. Monorepo where blast radius is high and changes are rare; polyrepo where changes are frequent and must be isolated. This hybrid is not a compromise; it is the correct shape.

Never point targetRevision at a moving branch in prod. Team config repos promote by changing a pinned tag or SHA (see the promotion section); the platform monorepo’s ApplicationSets read main, but every app’s prod overlay pins an immutable image tag. A moving branch means any merge, anywhere, can roll to prod unreviewed — the opposite of auditability.

---

Phase 1 — Bootstrap: Terraform provisions the substrate

Goal: From nothing, end with four clusters (mgmt + three spokes), a per-cloud identity the hub can assume, HA Argo CD installed on the mgmt cluster, and the one root app applied. This phase is pure cluster bootstrapping and the Terraform → Argo CD handoff, done four times and then handed off exactly once.

The layer-cake line from Why this matters becomes concrete here. Terraform creates everything up to and including the Argo CD install and the root Application; the moment the root app is applied, Terraform stops and Argo CD takes over. Terraform must never manage an Argo CD Application other than the root — if it did, you would have two controllers fighting over the same resources.

The Terraform is ordinary; the two things that make it a platform bootstrap are that it (a) installs Argo CD via helm_release in HA mode and (b) applies the root app as its final act. Representative root-module shape (abbreviated — real modules are larger):

# platform-terraform/main.tf  (abbreviated, representative)
module "mgmt" {                              # the management cluster (AKS)
  source              = "./modules/aks"
  name                = "aurora-mgmt"
  location            = "westeurope"
  kubernetes_version  = "1.30"
  workload_identity   = true                 # hub authenticates to spokes via WI
}

module "spoke_aks" { source = "./modules/aks"; name = "aurora-prod-aks"; location = "westeurope" }
module "spoke_eks" { source = "./modules/eks"; name = "aurora-prod-eks"; region   = "us-east-1" }
module "spoke_gke" { source = "./modules/gke"; name = "aurora-prod-gke"; location = "asia-south1" }

# Install HA Argo CD on the mgmt cluster, then hand off.
resource "helm_release" "argocd" {
  name             = "argocd"
  namespace        = "argocd"
  create_namespace = true
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = "7.7.0"                  # chart version → Argo CD 2.13.x
  values           = [file("${path.module}/values/argocd-ha.yaml")]
  depends_on       = [module.mgmt]
}

# The ONE hand-off: apply the root app-of-apps. Terraform stops here.
resource "kubectl_manifest" "root_app" {
  yaml_body  = file("${path.module}/../platform-gitops/bootstrap/root-app.yaml")
  depends_on = [helm_release.argocd]
}

The argocd-ha.yaml values file is what makes the hub survive a node loss — the installation lesson covered it; the essentials are redundant application-controller shards, ≥2 replicas of server and repo-server, and Redis in HA mode:

# values/argocd-ha.yaml  (representative essentials)
redis-ha: { enabled: true }                  # 3-node Redis, not the single-pod cache
controller: { replicas: 1 }                  # shard count; scale with fleet size
server:      { replicas: 2 }
repoServer:  { replicas: 2, resources: { limits: { memory: 2Gi } } }
applicationSet: { replicas: 2 }
configs:
  cm:
    application.instanceLabelKey: argocd.argoproj.io/instance
    timeout.reconciliation: 180s

And the root Application — the single most important manifest in the whole platform, because it is the only thing applied by hand and everything else descends from it. It points at bootstrap/, a directory of child Applications (or ApplicationSets), giving you the app-of-apps pattern as the bootstrap mechanism:

# platform-gitops/bootstrap/root-app.yaml  — the one hand-applied manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # cascade-delete children on removal
spec:
  project: platform
  source:
    repoURL: https://github.com/aurora-health/platform-gitops.git
    targetRevision: main
    path: bootstrap                              # a dir of child apps/appsets
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc       # the mgmt cluster itself
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

What just happened. Terraform built four clusters and installed an HA Argo CD on the management cluster, then applied one Application named root. That root app now syncs everything else in bootstrap/ — the AppProjects, the ApplicationSets, the platform layer — with no further human kubectl apply. From this point on, the way you change the platform is to commit to platform-gitops and let the hub reconcile. Terraform’s job is done until you need a new cluster.

The bootstrap chicken-and-egg. The root app references repos and clusters that must be reachable before the first sync. Order matters: Terraform creates the spoke identities and writes the cluster Secrets (Phase 2) into the mgmt cluster in the same apply, so that when the root app fans out the platform layer, the hub can already authenticate to all three spokes. Apply the root app before the cluster Secrets exist and every spoke Application shows Unknown — the classic first-boot failure.

---

Phase 2 — Cluster registration and identity

Goal: The hub can authenticate to all three spokes, each spoke is labelled so ApplicationSets can target it, and humans log in through SSO with their groups mapped to Argo CD roles. This is multi-cluster registration, SSO and RBAC composed into one identity story.

A spoke is registered by creating a Secret in the hub’s argocd namespace, labelled argocd.argoproj.io/secret-type: cluster. The credential inside is where the clouds diverge: the hub must call three different API servers, each with that cloud’s own auth. The clean way is exec auth — the cluster Secret’s config carries an execProviderConfig that runs the cloud’s token binary at reconcile time, so the hub holds no long-lived kubeconfig.

The EKS spoke’s cluster Secret, showing the real execProviderConfig shape (AKS and GKE differ only in the command/args):

# platform-gitops/bootstrap/clusters/prod-eks.yaml  (written by Terraform; creds via ESO/Sealed Secrets in Git)
apiVersion: v1
kind: Secret
metadata:
  name: prod-eks-us
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    cloud: eks                      # ← the label that routes the fleet
    env: prod
    region: us-east-1
    residency: us
type: Opaque
stringData:
  name: prod-eks-us
  server: https://XXXXXXXX.gr7.us-east-1.eks.amazonaws.com
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "client.authentication.k8s.io/v1beta1",
        "command": "aws",
        "args": ["--region", "us-east-1", "eks", "get-token", "--cluster-name", "aurora-prod-eks"],
        "env": { "AWS_ROLE_ARN": "arn:aws:iam::111122223333:role/argocd-hub" }
      },
      "tlsClientConfig": { "insecure": false, "caData": "<base64 cluster CA>" }
    }

The labels are not decoration — they are the organizing principle of the entire fleet. Every ApplicationSet from Phase 3 onward selects on them. Aurora’s label schema:

Get one label wrong and an app either deploys to the wrong cloud (a residency violation) or nowhere at all (an empty generator, silently). Labelling at registration is the single cheapest investment that makes fleet-wide targeting declarative.

Humans authenticate through one SSO configured in argocd-cm, federating each cloud’s identity provider so an engineer uses their existing corporate login. The SSO providers differ per cloud but land in one Argo CD:

RBAC in argocd-rbac-cm maps those groups to project-scoped roles — the platform team is admin, every app team gets sync/get on only its own project, and nobody else can do anything:

# argocd-rbac-cm  (representative)
policy.default: role:readonly
policy.csv: |
  g, aurora-health:platform-engineers, role:admin
  p, role:records-dev, applications, get,  team-records/*, allow
  p, role:records-dev, applications, sync, team-records/*, allow
  g, aurora-health:records-engineers, role:records-dev
scopes: '[groups]'

What just happened. The hub can now reach all three spokes — each with its own cloud’s auth, no static kubeconfig — every spoke is labelled with the four dimensions the fleet routes on, and every human logs in once and sees exactly the projects their SSO group allows. Crucially, app teams get sync but not create/delete on Applications: those come from ApplicationSets the platform owns, so a team can promote its own app but cannot conjure a new one outside the guardrails.

---

Phase 3 — The platform layer via ApplicationSets

Goal: Every spoke runs the same baseline platform stack — ingress controller, cert-manager, External Secrets Operator, Prometheus/Grafana, and the Argo Rollouts controller — installed and kept in lockstep by ApplicationSets, with the per-cloud differences expressed as a single goTemplate branch, not three copies.

This is the phase where the fleet stops being three clusters and becomes one platform. Each component is deployed by a platform ApplicationSet whose clusters generator loops over every spoke labelled env=prod and stamps one Application per cluster. Add a spoke Secret with the right labels and the entire platform stack appears on it automatically.

The critical manifest is the platform ingress ApplicationSet, because it shows the whole per-cloud pattern in one place: a cluster generator, a RollingSync strategy that gates the blast radius cloud-by-cloud, and a goTemplate branch that selects a per-cloud values file. This is the manifest to internalize — every other platform ApplicationSet is a variation of it.

# platform-gitops/applicationsets/platform-ingress.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-ingress
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]        # a typo fails the render, not silently blanks it
  generators:
    - clusters:
        selector:
          matchLabels:
            env: prod                              # every prod spoke, any cloud
  strategy:
    type: RollingSync                              # ← gate the blast radius
    rollingSync:
      steps:
        - matchExpressions: [{ key: cloud, operator: In, values: [aks] }]   # AKS first
        - matchExpressions: [{ key: cloud, operator: In, values: [eks] }]   # then EKS
        - matchExpressions: [{ key: cloud, operator: In, values: [gke] }]   # then GKE
  template:
    metadata:
      name: 'ingress-{{.name}}'
      labels:
        cloud: '{{index .metadata.labels "cloud"}}'   # RollingSync steps match on THIS label
    spec:
      project: platform
      source:
        repoURL: https://github.com/aurora-health/platform-gitops.git
        targetRevision: main
        path: platform/ingress
        helm:
          valueFiles:
            - values.yaml                                        # shared defaults
            - 'values-{{index .metadata.labels "cloud"}}.yaml'   # ← the per-cloud branch
      destination:
        server: '{{.server}}'
        namespace: ingress
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

The per-cloud branch resolves to one of three values files, each setting the ingress class its cloud understands — one app definition, three correct edges:

# platform/ingress/values-eks.yaml  (the whole per-cloud delta — a few lines, not a copy)
ingress:
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip

The RollingSync strategy is the single most important safety feature in this phase and deserves a hard look. Without it, editing the platform ingress chart reconciles to AKS, EKS and GKE on the same tick — a mistake breaks all three clouds at once. With it, the change rolls to AKS first; only when those Applications are Healthy does EKS proceed, then GKE. A bad change is caught after the first cloud, not the last. The steps match on the cloud label the template stamps onto each generated Application — that wiring (template label ↔ rollingSync matchExpressions) is easy to get wrong and produces a rollout that never progresses if the labels do not line up.

What just happened. One ApplicationSet per platform component now maintains that component on every spoke, forever. Register cluster number four and the full stack lands on it with no new manifests. The per-cloud differences live in tiny values files keyed by the cloud label, and every fleet-wide platform change rolls out cloud-by-cloud instead of all at once.

The blast-radius rule for the whole platform layer. Every platform ApplicationSet gets RollingSync and missingkey=error, and every one sets applicationsSync: create-update (never create-delete) plus preserveResourcesOnDeletion: true on the destructive ones, so that narrowing a selector can never prune a running platform component off the fleet. The convenience of “one object drives everything” is exactly why one object must be handled with this much care.

---

Phase 4 — Secrets: External Secrets Operator per cloud

Goal: No plaintext secret ever touches Git or a manifest. Each spoke’s ESO (installed in Phase 3) pulls secrets from its own cloud’s secret store, using that cloud’s workload identity. This composes the secrets lesson with the three per-cloud identity models.

The pattern is uniform — a SecretStore per namespace declares the provider and how to authenticate; an ExternalSecret declares which remote keys to materialize into a native K8s Secret — but the provider block is the cloud edge. The identity is always the cloud’s workload identity, never a static credential, so nothing long-lived lives in the cluster.

The three SecretStore manifests, side by side — this is the one place worth showing all three, because the provider block is precisely the multi-cloud seam:

# AKS — Azure Key Vault
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: { name: cloud-secrets, namespace: records }
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://aurora-eu-kv.vault.azure.net"
      serviceAccountRef: { name: eso-records }
---
# EKS — AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: { name: cloud-secrets, namespace: records }
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef: { name: eso-records }   # IRSA-annotated SA
---
# GKE — Google Secret Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: { name: cloud-secrets, namespace: records }
spec:
  provider:
    gcpsm:
      projectID: aurora-prod-apac
      auth:
        workloadIdentity:
          clusterLocation: asia-south1
          clusterName: aurora-prod-gke
          serviceAccountRef: { name: eso-records }

Because the SecretStore is named identically (cloud-secrets) in every cluster, the ExternalSecret that consumes it is cloud-neutral — the app team writes it once and it works on all three clouds, since only the store’s provider block differs:

# records-config/base/external-secret.yaml  — identical on AKS, EKS and GKE
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: records-db, namespace: records }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: cloud-secrets, kind: SecretStore }
  target: { name: records-db, creationPolicy: Owner }
  data:
    - secretKey: password
      remoteRef: { key: records-db-password }

This is the payoff of naming the store consistently: the platform absorbs the per-cloud difference (in the SecretStore the platform ApplicationSet deploys), so the app team writes one portable ExternalSecret. The seam is hidden exactly where it should be.

What just happened. Every app on every cloud now gets its secrets injected from that cloud’s native store, authenticated by workload identity, with zero plaintext in Git. The app team’s manifest is identical across clouds; the platform hid the seam in the SecretStore. And because the cluster Secrets themselves (Phase 2) are the only credentials not derivable from a cloud store, they are the one thing the DR plan in Phase 8 must back up.

---

Phase 5 — Application delivery and progressive rollout

Goal: App teams ship to the fleet by merging a PR in their own config repo, and every production change goes out as a canary gated by Prometheus analysis — no platform ticket, no big-bang deploy. This is Argo Rollouts plus metric analysis and auto-rollback, delivered through a self-service ApplicationSet.

Each app team gets one app-delivery ApplicationSet in the platform monorepo, a matrix of the team’s app directories crossed with the clusters that team’s residency allows. The team never touches this ApplicationSet — they merely commit to their config repo, and apps appear:

# platform-gitops/applicationsets/apps-records.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: apps-records, namespace: argocd }
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:                                   # one entry per app dir the team commits
              repoURL: https://github.com/aurora-health/records-config.git
              revision: main
              directories: [{ path: apps/* }]
          - clusters:
              selector:
                matchLabels: { env: prod, residency: eu }   # records = EU data → AKS only
  template:
    metadata:
      name: 'records-{{.path.basename}}-{{.name}}'
    spec:
      project: team-records                         # ← the guardrail from Phase 6
      source:
        repoURL: https://github.com/aurora-health/records-config.git
        targetRevision: main
        path: '{{.path.path}}/overlays/prod'
      destination:
        server: '{{.server}}'
        namespace: records
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Note the residency selector: records handles EU patient data, so its clusters filter is residency: eu — the ApplicationSet cannot place a records app on the US or India cloud even if someone tried. Data residency is enforced by the generator, then again by the AppProject in Phase 6. Defense in depth, applied to a compliance requirement.

The app itself deploys as a Rollout, not a Deployment, so its production change is a canary. The steps shift traffic in stages with a metric gate between them:

# records-config/apps/api/base/rollout.yaml  (canary with a Prometheus gate)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: records-api, namespace: records }
spec:
  replicas: 6
  selector: { matchLabels: { app: records-api } }
  strategy:
    canary:
      canaryService: records-api-canary
      stableService: records-api-stable
      trafficRouting:
        nginx: { stableIngress: records-api }
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: success-rate }] }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
  template:
    metadata: { labels: { app: records-api } }
    spec:
      containers:
        - name: api
          image: aurora.azurecr.io/records-api:v2.4.1   # pinned tag — promotion bumps THIS
          ports: [{ containerPort: 8080 }]

The AnalysisTemplate is what makes the canary decide instead of just wait — it queries Prometheus and aborts (rolling back to stable in seconds) if the success rate drops below the SLO:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate, namespace: records }
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: "result[0] >= 0.99"          # ≥99% non-5xx
      failureLimit: 3                                  # 3 bad samples → abort
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="records-api",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{app="records-api"}[2m]))

The traffic router differs per cloud (NGINX here, but ALB or GCLB or a service mesh elsewhere) — that per-cloud wiring is the subject of the traffic-management lesson. The strategy is identical everywhere, which is why the app team writes it once.

What just happened. An app team ships by merging a PR: their app-delivery ApplicationSet notices the new/changed directory, Argo CD syncs the Rollout, and the new version rolls out as a Prometheus-gated canary that auto-aborts on a regression — all on the correct cloud for that team’s data residency, all without a platform engineer in the loop. That is the self-service NFR, delivered.

---

Phase 6 — Multi-tenancy and security

Goal: Bound what every actor can do, so self-service cannot become self-destruction. An AppProject per team is the tenancy wall; hardening — signed commits, image verification, default-deny networking — is the supply-chain and blast-radius wall.

The AppProject is the load-bearing guardrail. It pins the only repos, clusters/namespaces and resource kinds a team’s Applications may touch, and — critically for auditability — it can require signed commits via signatureKeys and forbid deploys outside a change window via syncWindows:

# platform-gitops/appprojects/team-records.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: team-records, namespace: argocd }
spec:
  description: "Records (EHR) — EU data only"
  sourceRepos:
    - https://github.com/aurora-health/records-config.git    # ONLY this repo
  destinations:
    - server: https://prod-aks-eu.example              # ONLY the EU (AKS) cluster
      namespace: records                                # ONLY this namespace
  clusterResourceWhitelist: []                          # deny ALL cluster-scoped resources
  namespaceResourceWhitelist:
    - { group: "*", kind: "*" }
  roles:
    - name: developer
      policies:
        - p, proj:team-records:developer, applications, sync, team-records/*, allow
      groups: [ aurora-health:records-engineers ]
  signatureKeys:
    - keyID: ABCDEF1234567890                           # commits MUST be GPG-signed by this key
  syncWindows:
    - kind: deny
      schedule: "0 22 * * 5"                            # no Friday-night prod deploys
      duration: 60h
      applications: ["*"]

Read what that project forbids: a records Application can only pull from the records repo, can only deploy to the EU AKS cluster’s records namespace, can create no cluster-scoped resource (no ClusterRole, no CRD, no escaping its namespace), must be built from a GPG-signed commit, and cannot sync during the Friday-night freeze. A fat-fingered or compromised app repo is contained by the project before anything is applied. That containment table across all four teams:

Above the projects sit two supply-chain controls, deployed to every spoke by a platform ApplicationSet. Kyverno admission verifies that every image is signed by Aurora’s cosign key and comes from an Aurora registry — a backstop that catches anything the project let through:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-aurora-images }
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: check-signature
      match: { any: [{ resources: { kinds: [Pod] } }] }
      verifyImages:
        - imageReferences:
            - "aurora.azurecr.io/*"
            - "111122223333.dkr.ecr.us-east-1.amazonaws.com/aurora/*"
            - "asia-south1-docker.pkg.dev/aurora/*"
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      <cosign public key placeholder>
                      -----END PUBLIC KEY-----

And a default-deny NetworkPolicy in every app namespace, so a compromised pod cannot talk to another team’s workloads — you then add back only the flows the app needs:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: records }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

What just happened. Every app team now operates inside a box it cannot see out of: its own repo, its own cluster and namespace, signed commits only, signed images only, no lateral network reach, and no ability to create Applications the platform did not template. Self-service is safe because the guardrails are declarative and enforced in depth — at the Argo CD gate, at admission, and at the network.

---

Phase 7 — Observability and notifications

Goal: Know the health of the whole fleet at a glance, and get paged when something drifts — without logging into three clouds. This is Argo CD observability and notifications applied at fleet scale.

Two things must be observable: Argo CD itself (is the platform reconciling?) and the fleet’s apps (are they healthy?). The hub’s application-controller exports Prometheus metrics — sync counts, reconciliation latency, app health — that a single Grafana dashboard turns into one view across AKS, EKS and GKE. The per-spoke Prometheus (from Phase 3) federates its app metrics up to the mgmt cluster’s Prometheus, so one query spans the fleet.

Notifications turn a breached SLO into a Slack page. Argo CD’s notifications controller subscribes to app events and routes them; Aurora pages the owning team’s channel on OutOfSync or Degraded and the platform channel on anything cluster-level:

# argocd-notifications-cm  (representative)
trigger.on-degraded: |
  - when: app.status.health.status == 'Degraded'
    send: [app-degraded]
template.app-degraded: |
  message: "🔴 {{.app.metadata.name}} is Degraded on {{.app.spec.destination.server}}"
subscriptions: |
  - recipients: [slack:aurora-platform]
    triggers: [on-degraded, on-sync-failed]

What just happened. One Grafana shows every app on every cloud; one set of SLOs defines “healthy fleet”; and a drift, a failed sync, or an unreachable spoke pages the right humans in Slack without anyone watching three consoles. Observability is what makes a one-team-fifty-app platform actually operable.

---

Phase 8 — Disaster recovery: the hub is gone

Goal: Survive the loss of the entire management cluster and rebuild the control plane from Git, with the fleet’s apps never going down. This is the HA & DR lesson as the platform’s final proof.

The reason DR here is survivable is the fact this whole course leaned on: Argo CD’s control plane is not in the data path. If the management cluster burns down, the records-api pods on the EKS spoke keep serving US patients — they do not route through the hub. You lose reconciliation (drift correction, self-heal, the ability to ship changes), not availability. DR is therefore rebuild-from-Git, not restore-a-database, and it is fast.

The recovery hinges on one distinction: almost everything is already in Git, and a tiny set of state is not.

The runbook, in order — the order matters because the hub must be able to reach the spokes before the root app fans out:

Step 6 is the safety check that separates a clean DR from a fleet-wide outage. When the rebuilt hub applies the root app, it compares Git to the already-running workloads on the spokes; because they match, it marks them Synced with nothing to change — it re-adopts them. If instead the diff shows everything wanting to redeploy, your restored state or a targetRevision is wrong; you stop and fix it rather than let recovery cause the outage you were recovering from.

What just happened. You proved the platform’s most important property: the control plane is rebuildable and the fleet outlives it. A dead hub is a degraded state (no new deploys, no drift correction) measured in minutes to recover, not an outage of Aurora’s applications. That is the difference between a platform and a house of cards.

---

Environment promotion across the fleet

A change to records-api should flow dev → staging → prod the same way on all three clouds, and every step must be a reviewed, attributable Git change — the auditability NFR. Aurora promotes by pinning an immutable image tag per environment overlay and moving that pin with a pull request; nothing auto-flows to prod. This is sync policies and the promotion patterns applied fleet-wide.

The promotion PR is tiny and auditable — one line:

# records-config/apps/api/overlays/prod/kustomization.yaml
images:
  - name: aurora.azurecr.io/records-api
    newTag: v2.4.1        # was v2.4.0 — THIS one-line PR is the prod promotion

Because prod overlays pin an exact tag (never a moving branch), the diff a reviewer approves is the thing that ships, and a rollback is a git revert of that one-line PR. An image updater can open the dev bump PR automatically, but the promotion to prod stays a human decision. The residency selector in each team’s ApplicationSet guarantees the promoted image lands only on the legally-correct cloud — promotion and residency are enforced by the same label machinery.

---

Cost and operations

Multi-cloud tempts a 3× bill; a disciplined platform avoids it. The unifying moves: one hub (not three), one observability stack, spot/preemptible nodes for platform components, and right-sized spokes.

Operationally, the platform’s leverage is that the number of things a human touches does not grow with the fleet. Adding cluster four is a labelled cluster Secret plus a Terraform module; the platform stack fans onto it automatically. Onboarding app team five is one AppProject and one app-delivery ApplicationSet in a PR. The platform team’s job becomes owning the guardrails and the templates, not the individual deployments — which is the only way five engineers support fifty teams.

The synthesis in one table — every phase, the lesson it draws on, and the one manifest that anchors it:

---

Hands-on lab

This lab is the capstone build runbook at design/config level. There is no live three-cloud fleet here, so you execute it against your own AKS/EKS/GKE clusters (or substitute three kind clusters to rehearse the shape for free — the ApplicationSets, projects and rollouts are cloud-neutral; only ingress/secrets/registry edges need the real clouds). Each phase gives the anchor manifest, the verify command with representative output, and a one-line “what just happened.” ⚠️ Real clouds bill: managed control planes, load balancers, NAT, and private endpoints all cost money — tear everything down at the end.

Phase 0 — repos. Create platform-gitops (structure from the repo section) and one *-config per team. Commit the bootstrap/, appprojects/, applicationsets/, platform/ and policies/ trees.

Phase 1 — bootstrap.

cd platform-terraform && terraform init && terraform apply    # 4 clusters + HA Argo CD + root app
argocd login <mgmt-argocd-host> --sso                          # first login via SSO
argocd app get root                                            # representative:
#   Name:            argocd/root
#   Project:         platform
#   Sync Status:     Synced to main (a1b2c3d)
#   Health Status:   Healthy

What just happened: Terraform built the substrate and handed off; the root app is now the source of the whole platform.

Phase 2 — register + identify. Apply the three cluster Secrets (via the bootstrap/clusters/ path the root app syncs), then:

argocd cluster list        # representative:
# SERVER                                          NAME          VERSION  STATUS   LABELS
# https://...eks.amazonaws.com                    prod-eks-us   1.30     Successful  cloud=eks,env=prod,residency=us
# https://prod-aks-eu.example                     prod-aks-eu   1.30     Successful  cloud=aks,env=prod,residency=eu
# https://...container.googleapis.com             prod-gke-in   1.30     Successful  cloud=gke,env=prod,residency=india

What just happened: the hub authenticates to all three clouds and every spoke carries its routing labels.

Phase 3 — platform layer. The root app has already synced the platform ApplicationSets. Verify the fan-out:

kubectl -n argocd get applicationset
kubectl -n argocd get applications -l app.kubernetes.io/part-of=platform -o wide
# representative: ingress-prod-aks-eu / -eks-us / -gke-in all Synced/Healthy

What just happened: one ApplicationSet produced one ingress Application per cloud, each with the correct class.

Phase 4 — secrets. Apply a team SecretStore + ExternalSecret, then confirm the native Secret materialized:

kubectl -n records get externalsecret records-db
# NAME         STORE          REFRESH  STATUS         READY
# records-db   cloud-secrets  1h       SecretSynced   True

What just happened: ESO pulled the secret from the cloud store via workload identity; no plaintext in Git.

Phase 5 — an app with a canary. Merge a tag bump in records-config, then watch the rollout:

kubectl argo rollouts get rollout records-api -n records --watch
# representative:  Status: ॥ Paused (setWeight 10)  →  Analysis: success-rate ✔  →  Healthy

What just happened: the app shipped as a Prometheus-gated canary, self-service, on the residency-correct cloud.

Phase 6 — guardrails. Apply the AppProjects and Kyverno policy, then prove the box holds:

# Try to sync an app to a namespace the project forbids — it must be REJECTED:
argocd app set records-api --dest-namespace billing
# representative: FATA[] application ... is not permitted in project team-records

What just happened: the AppProject rejected an out-of-bounds deploy before anything hit a cluster.

Phase 7 — observe. Import the fleet Grafana dashboard and confirm one view spans three clouds; send a test Slack alert. What just happened: one pane, three clouds, and drift now pages a human.

Phase 8 — DR game day. In a test environment, delete the mgmt cluster and run the runbook. Confirm the spokes kept serving throughout, and that argocd app diff is empty after re-adoption. What just happened: you proved the fleet survives losing the hub.

Verify the fleet is healthy (the one command that summarizes everything):

argocd app list -o wide | awk 'NR==1 || $0 !~ /Synced.*Healthy/'   # show only NOT-healthy apps
# representative healthy fleet: only the header prints — every app Synced/Healthy across all 3 clouds
kubectl --context mgmt -n argocd get applications \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status

Teardown (do this — real clouds bill):

argocd app delete root --cascade        # removes every generated app across the fleet
cd platform-terraform && terraform destroy   # removes all 4 clusters + identities + LBs
# Manually confirm no orphaned cloud LBs/NAT/private-endpoints remain in each cloud console.

---

Common mistakes and troubleshooting

The cross-cutting failures — the ones that only appear when the pieces are combined — are the ones that bite a multi-cloud platform. Single-lesson gotchas are covered in their own lessons; these are the seams.

The three nastiest deserve prose. Blast radius through ApplicationSets is the defining multi-cloud hazard: the very object that makes the fleet manageable is a loaded gun, because one edit fans to every matching cluster instantly. RollingSync is not optional at fleet scale — and its steps must match the labels your template stamps, a wiring people get subtly wrong so the rollout stalls at step one. Per-cloud identity on the hub is the failure that looks like an Argo CD bug but is a cloud IAM problem: the hub authenticates three different ways, and a single missing exec binary or ungranted role takes out exactly one cloud while the other two look perfect — always check whether the failure is isolated to one cloud label before you suspect Argo CD. DR that redeploys instead of re-adopts is the scariest, because it turns a recovery into the outage: the empty-diff check before enabling auto-sync is the one step you never skip.

---

Cheat-sheet

The platform, in one breath: Terraform builds a mgmt cluster + AKS/EKS/GKE spokes and hands off to one HA Argo CD hub; the root app-of-apps fans platform ApplicationSets (ingress, cert-manager, ESO, Prometheus, Rollouts) across every spoke with a per-cloud branch; app teams self-serve canary deploys through app-delivery ApplicationSets bounded by AppProjects + Kyverno; fleet SLOs + Slack watch it; a rebuild-from-Git runbook survives losing the hub.

Phased build runbook:

Key-manifest index:

Commands that summarize fleet state:

---

Interview and exam questions

Q: Design a GitOps platform for a company whose data-residency rules force workloads onto AKS, EKS and GKE simultaneously. What topology do you choose, and why? A: Hub-and-spoke: one HA Argo CD hub on a dedicated management cluster managing all three clouds as spokes. It gives one control plane, one ApplicationSet-driven fan-out, and one DR plan for a small platform team — the alternatives (one Argo per cluster, one hub per cloud) multiply operational surface without benefit at this scale. I bound the hub’s concentrated blast radius with HA, RBAC, and per-team AppProjects, and I keep the management cluster workload-free so the control plane’s failure domain is separate from the apps.

Q: One cloud’s apps all show Unknown while the other two are healthy. Walk me through the diagnosis. A: The failure is isolated to one cloud label, which almost always means the hub cannot authenticate to that one spoke — an Argo CD-shaped symptom with a cloud-IAM cause. Check three things in order: is that cloud’s exec plugin binary present in the hub image; is the hub’s identity actually granted on that spoke (access entry / RBAC / GSA binding); and is the cluster Secret’s execProviderConfig correct. It is not a general Argo CD outage, because the other two clouds work.

Q: Why is a single ApplicationSet edit the most dangerous operation on this platform, and how do you make it safe? A: A cluster-generator ApplicationSet reconciles a change to every matching spoke on the next tick, so a bad template breaks AKS, EKS and GKE at once — the fleet is one object. RollingSync makes it safe by rolling the change cloud-by-cloud (steps matching the cloud label the template stamps), so a mistake is caught after the first cluster. I also set create-update (never create-delete) and missingkey=error so a narrowed selector can’t prune the fleet and a typo fails the render instead of silently blanking a field.

Q: How does an app team ship to production without a platform engineer, and what stops them from doing something dangerous? A: They merge a one-line tag bump in their own config repo; their app-delivery ApplicationSet syncs it as a Prometheus-gated canary on the residency-correct cloud. What stops harm is layered: RBAC gives them sync not create; their AppProject pins the only repo/cluster/namespace they can touch, denies cluster-scoped resources, and requires signed commits; Kyverno rejects unsigned images at admission; and a default-deny NetworkPolicy prevents lateral movement. Self-service is safe because the guardrails are declarative and enforced in depth.

Q: The management cluster is destroyed. What happens to the running apps, and what is your recovery? A: Nothing happens to the running apps — the hub is not in the data path, so the spokes keep serving. I lose reconciliation, not availability. Recovery is rebuild-from-Git: Terraform a fresh mgmt cluster, install HA Argo CD, restore the cluster Secrets (the one thing not in app Git), re-apply config, apply the root app, and confirm argocd app diff is empty so the hub re-adopts rather than redeploys. RTO is under an hour and it is a degraded-reconciliation window, not an app outage.

Q: Where do you draw the line between Terraform and Argo CD, and why does it matter? A: Terraform owns the substrate — clusters, node pools, per-cloud identities, networking — up to and including installing Argo CD and applying the one root Application. Argo CD owns everything above: the platform layer, apps, projects, policies. The line matters because if Terraform managed Argo CD Applications too, two controllers would fight over the same resources; and because a clean handoff is exactly what makes DR a fast rebuild rather than an archaeology project.

Q: How do you enforce data residency technically, not just by policy? A: With cluster labels and two independent enforcement points. Each spoke is labelled residency: eu|us|india; a team’s app-delivery ApplicationSet selects only its residency (records → residency: eu), so it cannot generate an app on the wrong cloud. Then the team’s AppProject destinations pins the exact cluster and namespace, rejecting any out-of-zone deploy at the Argo CD gate. Two layers, both declarative, so a residency violation is impossible-by-construction rather than caught-in-review.

Q: Your monthly cloud bill jumped after onboarding a fourth cluster. What’s your first hypothesis and fix? A: The new spoke likely brought on-demand nodes and a public load balancer per app with no autoscaler — the classic multi-cloud cost multiplier. Fix: spot/preemptible pools for stateless platform pods, a shared ingress rather than an LB per service, cluster autoscaler with scale-to-zero for dev, and a check for orphaned cloud LBs/NAT. The platform’s cost discipline is one hub, one observability stack, and keeping data (and thus egress) in-cloud.

Q: Why name the ESO SecretStore identically in every cluster? A: So the app team’s ExternalSecret is cloud-neutral. The per-cloud difference (azurekv vs aws vs gcpsm, and the workload-identity binding) lives in the SecretStore the platform ApplicationSet deploys; because the store’s name is the same everywhere, the ExternalSecret referencing it is written once and works on all three clouds. The platform absorbs the seam so the app doesn’t have to.

Q: How do you promote a change across the fleet while keeping it auditable, and how do you roll back? A: Pin an immutable image tag per environment overlay and move the pin with a reviewed PR — dev auto-syncs on merge, staging and prod are tag-bump PRs with review gates, and prod also passes canary analysis. Because prod overlays never point at a moving branch, the diff a reviewer approves is exactly what ships, and rollback is a git revert of that one-line PR. The residency selector guarantees the promoted image lands only on the legal cloud.

---

Key takeaways

• The deployment plane unifies even when infrastructure can’t. Argo CD reconciles Git into any Kubernetes API server, so one hub runs AKS, EKS and GKE from one source of truth; the clouds differ only at the edges — identity, ingress, secrets, registries — which is exactly what the rest of this course taught you to handle.
• A capstone is synthesis, not a topic list. The platform is eight phases building five layers bottom-up: substrate (Terraform) → control plane + identity → platform layer + secrets → apps + guardrails → operations + DR. Each phase is one prior lesson, composed — not re-taught.
• Concentration is the core trade-off. One hub is one pane of glass and one blast radius. Every safety decision — HA, RollingSync, AppProjects, Kyverno, default-deny networking, the empty-diff DR check — is about bounding a risk you deliberately concentrated for the operational win.
• Labels are the fleet’s organizing principle. cloud, env, region, residency on each cluster Secret turn fleet-wide targeting, per-cloud branching, and data-residency enforcement into declarative selectors instead of hand-written per-cluster manifests.
• Self-service is safe only because the guardrails are declarative. App teams ship by merging a PR because AppProjects, RBAC, Kyverno and NetworkPolicy bound what a merge can do — enforced at the Argo CD gate, at admission, and at the network. That is what lets five platform engineers support fifty app teams.
• The control plane is rebuildable and off the data path. Losing the hub is a degraded-reconciliation window measured in minutes, not an app outage — because Git is the backup and the spokes keep serving. Prove it with a DR game day; the diff-before-sync re-adoption check is the step you never skip.
• You can now design and defend a real multi-cloud GitOps platform end to end — the topology, the handoff, the fan-out, the guardrails, the promotion flow, and the recovery — which is the whole thing this course set out to teach.