GitHub Actions to Argo CD: Progressive Delivery with Policy Gates

In a nutshell

Progressive delivery is the practice of shipping a new version to a few users first, watching real health signals, and only then letting it own all the traffic — with an automatic reversal if those signals go bad. GitOps is the practice of describing what should be running as files in a Git repository and having a controller pull those files into the cluster, so Git is the single source of truth and the audit log. This lesson wires the two together on Kubernetes: GitHub Actions builds and signs the image, a Git commit records the desired state, Argo CD reconciles the cluster to that commit, and Argo Rollouts does the careful, metric-gated traffic shift. The word that matters is promote, not deploy — anything can deploy; the hard part is deploying in a way that proves itself before it can hurt anyone.

Picture a restaurant chain rolling out a new recipe. You do not change the recipe in all 500 kitchens overnight. Head office writes the certified recipe into a master binder (that is Git), and a regional manager walks every kitchen to make sure each one matches the binder and fixes any that have drifted (that is Argo CD). The new dish is piloted at one location first; the manager watches the reviews and the plates coming back to the kitchen (that is the canary and its metric analysis), and if customers start sending it back, the dish is pulled immediately and the old one restored (that is the automatic rollback). And a health inspector at the kitchen door refuses any dish that did not come from the certified binder or lacks its provenance sticker (that is the admission policy gate — OPA Gatekeeper or Kyverno checking the image is signed and from an approved source).

So there are four moving parts, each with one job. GitHub Actions (CI) builds, tests, scans, and signs the artifact, then writes a one-line change to Git. Argo CD (GitOps) continuously makes the cluster match Git and heals any drift. Argo Rollouts (progressive delivery) shifts traffic in small steps and, between steps, runs an analysis that queries your metrics (Prometheus, Datadog) and aborts automatically on a breach. Policy gates at the cluster door reject anything unsigned, unapproved, or unsafe before a single pod runs. The heartbeat of the whole thing is one loop: pause → check real signals → promote or roll back.

Contrast that with the old way — a single deploy that flips everyone to the new version at once and hopes the smoke test caught the bug. Progressive delivery replaces hope with evidence, and replaces the 3 a.m. manual rollback with an automatic one.

Level: Intermediate · Time: ~38 min

Prerequisites & what you’ll be able to do

You will get the most from this lesson if you already know:

• What rolling, blue/green, and canary deployments are at a concept level — this lesson automates canary with real health gates, so if that vocabulary is fuzzy, read Deployment strategies: rolling, blue/green, canary, flags first.
• How a GitHub Actions workflow is put together — jobs, steps, permissions, and especially OIDC federation instead of stored keys. The GitHub Actions OIDC keyless deploys lesson is the companion for the auth half.
• Basic Kubernetes objects — Deployment, ReplicaSet, Service — and the idea of an admission webhook that can accept or reject a manifest before it runs.
• A little PromQL and the notion of metrics and SLOs — the canary judge is only as good as the query behind it, and the Prometheus + Grafana monitoring stack lesson covers the query language.
• (Helpful, not required) how container image signing works with Cosign/Sigstore, since admission verifies those signatures — see Sigstore keyless signing and policy-controller admission.

After working through it you will be able to:

• Explain precisely how progressive delivery differs from a plain rolling update, and pick the right one per workload.
• Draw the two-repo, pull-based GitOps topology and say exactly where CI’s authority ends and the cluster’s begins — and why they must never share credentials.
• Author an Argo Rollouts Rollout with canary steps, pauses, and an AnalysisTemplate that gates promotion on real metrics from Prometheus or Datadog.
• Wire an automated abort/rollback that fires on a single metric breach, and configure it so “no data” counts as inconclusive, never as “healthy.”
• Choose between a PR to the config repo and Argo CD Image Updater for promotion, and say why production usually wants the PR.
• Write an OPA Gatekeeper or Kyverno admission policy that admits only signed, approved-registry, resource-limited workloads — and roll it out in dry-run / audit mode first.
• Name the main failure modes (metrics gap, webhook down, stuck canary, Git drift) and the mitigation for each.

The mandate: no human pushes to production

A health-insurance company’s claims-platform team ships once a sprint, on a Thursday night, with the whole squad on a bridge call and a runbook open. It works until it doesn’t: last quarter a release to the member-portal API changed a benefits-eligibility response shape, the on-call missed it during the manual smoke test, and for ninety minutes members were told they had no coverage. In a regulated payer that is not an inconvenience — it is a reportable incident, a spike in call-center volume, and a compliance officer asking why a change touching protected health information went out with a person clicking through a checklist. The mandate that came down was specific: no human pushes to production, every change is auditable, and a bad release rolls itself back before most members ever see it. This article is the reference architecture for that mandate — a progressive-delivery pipeline that builds and signs artifacts in GitHub Actions, deploys them through Argo CD GitOps onto GKE, rolls them out as Argo Rollouts canaries, and refuses to promote anything that fails a policy gate or a live health signal.

The pressures are the ones every platform team eventually hits, just sharper in a payer. Safety: a regression in eligibility or claims logic has real-world consequences, so a release has to prove itself on a sliver of traffic before it owns all of it. Auditability: a HIPAA-regulated shop needs to answer “what was running at 14:32, who approved it, and what scanned clean” without spelunking through shell history. Velocity: the business still wants to ship daily, which is impossible if every deploy needs a bridge call. Blast radius: when something does break, it should degrade a percentage of traffic for two minutes, not the whole member base for ninety. Progressive delivery — ship to a few, watch real signals, promote or abort automatically — is the pattern that satisfies all four, and GitOps is what makes it auditable by construction.

Why not the obvious shortcuts

Three cheaper approaches will get proposed in the first planning meeting, and each fails in a way worth naming.

“Just kubectl apply from the CI job.” Push-based CD hands your CI runners cluster-admin-grade credentials, makes the pipeline the source of truth for what’s deployed (so cluster drift is invisible), and leaves you with no record of desired state other than a job log that rotates out in 30 days. The first time someone hotfixes the cluster by hand, your CI and reality silently diverge.

“Blue-green the whole service.” Standing up a full parallel copy and flipping a load balancer is a real strategy, but it’s all-or-nothing: the new version takes 100% of traffic the instant you cut over, so a subtle eligibility bug hits every member at once — exactly the ninety-minute outage we’re trying to kill. It also doubles capacity cost during the cutover window.

“Add more manual gates.” More checklists and more bridge calls slow velocity without improving safety, because the failure mode was a human missing a regression, and humans miss regressions. The fix is not more humans; it’s machine-evaluated gates on real signals.

Progressive delivery with GitOps threads the needle. Git is the single source of truth and the audit log. Argo CD continuously reconciles the cluster to Git, so drift self-heals and “what’s running” is always answerable. Argo Rollouts shifts traffic in small increments and consults health analysis between steps, so a bad version is caught at 5% and rolled back automatically. And policy gates — in CI and at the cluster admission boundary — stop a non-compliant artifact from ever reaching the canary in the first place.

Architecture overview

The diagram reads left to right in two halves that touch only at Git: GitHub Actions builds, scans, signs, and commits a new image digest (steps 1–4); then inside GKE, Argo CD pulls that commit, OPA Gatekeeper admits only the signed, policy-clean manifest, and Argo Rollouts shifts a traffic slice, consults Datadog, and promotes or auto-rolls-back (steps 5–8) — each numbered badge marks a gate where a bad change is stopped before it spreads.

The platform has two cleanly separated halves that meet at exactly one place — the Git repository — and never share credentials. The CI half lives in GitHub Actions: it builds, scans, signs, and then writes a desired-state change to Git. The CD half lives in the cluster: Argo CD watches Git and pulls changes in; nothing pushes to the cluster from outside. This separation is the whole game. CI never holds cluster credentials, and the cluster never reaches back into CI. The boundary between “build the thing” and “run the thing” is a Git commit, which is also the audit record.

The defining property the compliance team cares about is provenance you can prove: every image is built by a known workflow with no long-lived cloud keys, scanned by two independent tools, cryptographically signed, and admitted to the cluster only if its signature and policy checks pass. A push to production is no longer an action a person takes — it is a state that Git describes and the cluster converges to.

CI path, following the control flow:

1. A developer merges a PR to main. A GitHub Actions workflow starts and authenticates to Google Cloud via Workload Identity Federation (OIDC) — it exchanges its short-lived OIDC token for a GCP access token, so there is no stored service-account JSON key to leak. The same OIDC trust lets it pull from and push to Artifact Registry.
2. The workflow scans infrastructure-as-code before it builds anything: Checkov lints the Terraform and Kubernetes manifests for misconfigurations (public buckets, privileged pods, missing encryption), and Wiz Code runs in the pipeline to catch IaC and dependency risk with the context of how the resource is actually exposed in the running cloud — a finding Wiz flags as “internet-reachable + critical CVE” is a hard stop, where a buried lab finding is not.
3. The application image builds, and the workflow signs it with Cosign (keyless, using the same OIDC identity, recorded in a transparency log) and generates an SBOM and SLSA provenance attestation. Signature and attestation are pushed alongside the image in Artifact Registry.
4. The workflow then makes its only change to the deploy surface: it bumps the image digest in the GitOps config repository — a separate repo holding Kustomize/Helm manifests — via a commit (often a PR for prod, auto-merged for lower environments). CI’s job ends here. It has touched Git, not the cluster.

CD path, pull-based and continuous:

1. Argo CD, running inside GKE, detects the new commit in the config repo and reconciles. For the member-portal API it renders an Argo Rollouts Rollout resource (not a vanilla Deployment) pointing at the new image digest.
2. Before any pod schedules, the OPA Gatekeeper admission webhook evaluates the manifests against cluster policy — image must come from the approved Artifact Registry, must carry a valid Cosign signature, must set resource limits, must not run privileged. A manifest that violates policy is rejected at admission, so a bad change fails loudly in Argo CD’s sync status instead of quietly running.
3. Argo Rollouts begins the canary: it routes a small slice of traffic to the new version and pauses. During each pause it runs an AnalysisRun that queries Datadog — error rate, p95 latency, and a custom eligibility-success metric — against defined thresholds.
4. If the analysis stays green across the canary steps, Rollouts promotes the new version to 100% and the old ReplicaSet scales down. If any step breaches a threshold, Rollouts aborts and rolls back automatically to the last good version, and a ServiceNow incident is opened from the Datadog monitor so on-call has a ticket, not just a page.

Component breakdown

A few of these choices carry the why, because they’re where teams go wrong.

Why two IaC scanners, not one. Checkov and Wiz Code overlap but answer different questions. Checkov is a fast, free, deterministic policy linter — perfect as a cheap pre-build gate and easy to extend with payer-specific checks (e.g., “every storage class touching claims data must be encrypted with a CMEK key”). Wiz Code adds the context Checkov can’t have: it correlates an IaC finding with the live cloud, so it can tell you a misconfiguration is actually internet-reachable and tied to a critical CVE versus a theoretical lab risk. Running both means cheap deterministic gates plus prioritization by real exposure, and you only hard-fail the build on findings that are genuinely high-severity in context — otherwise you train developers to ignore the scanner.

Why keyless signing and admission verification together. Signing an image proves who built it; it does nothing if the cluster will run any image. The value comes from pairing Cosign signatures with an OPA Gatekeeper (or Kyverno/Binary Authorization) admission check that requires a valid signature from your build identity before a pod schedules. That closes the loop: an attacker who pushes a malicious image to the registry can’t get it admitted, because it isn’t signed by the trusted CI OIDC identity. Provenance is only worth the bytes if something enforces it at the door.

Implementation guidance

Wire OIDC first, and prove no static keys remain. The single biggest security win here is that GitHub Actions never holds a downloadable GCP key — the leaked-credentials lesson the platform team intends never to repeat. Configure a Workload Identity Pool scoped to your repo and branch, and the auth step needs no secret at all:

# .github/workflows/build.yaml
permissions:
  contents: read
  id-token: write          # required for OIDC
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: projects/4711/locations/global/workloadIdentityPools/gh/providers/repo
          service_account: ci-build@claims-prod.iam.gserviceaccount.com
      - name: IaC scan (Checkov)
        run: checkov -d ./infra --hard-fail-on HIGH
      - name: Build, sign, attest
        run: |
          docker build -t "$IMG" .
          docker push "$IMG"
          cosign sign --yes "$IMG"            # keyless, uses the OIDC identity
          cosign attest --yes --predicate sbom.json --type spdx "$IMG"

The provider is constrained so only this repository on a protected branch can assume the build service account; a fork or a feature branch gets nothing.

Separate the app repo from the config repo. Keep application source in one repository and the rendered deploy manifests (Kustomize bases/overlays or a Helm chart) in another. CI’s last step writes the new image digest into the config repo. This separation gives you environment-scoped review (a human approves the prod overlay change while dev/stage auto-merge), a clean per-environment audit trail, and an Argo CD that watches exactly one source of truth per environment.

Model the rollout, not a deployment. The member-portal API is an Argo Rollouts Rollout with explicit canary steps and a Datadog analysis between them. The steps below mean: take 5% of traffic, hold while analysis runs, then 25%, then 50%, then full — aborting the moment analysis fails.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: member-portal-api }
spec:
  strategy:
    canary:
      canaryService: member-portal-canary
      stableService: member-portal-stable
      trafficRouting: { istio: { virtualService: { name: member-portal-vs } } }
      steps:
        - setWeight: 5
        - pause: { duration: 3m }
        - analysis:                      # query Datadog; abort on breach
            templates: [{ templateName: datadog-health }]
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }

The AnalysisTemplate is where the safety lives — it queries Datadog for the metrics that actually matter to members, not just CPU:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: datadog-health }
spec:
  metrics:
    - name: error-rate
      interval: 1m
      failureLimit: 1                    # one breach aborts + rolls back
      provider:
        datadog:
          query: "sum:portal.http.5xx{service:member-portal,version:canary}.as_rate()"
      failureCondition: "result > 0.01"  # >1% 5xx fails the canary
    - name: eligibility-success
      provider:
        datadog:
          query: "sum:portal.eligibility.ok{version:canary} / sum:portal.eligibility.total{version:canary}"
      failureCondition: "result < 0.995" # the metric that caught nothing last quarter

That second metric — eligibility-success rate — is the one whose absence caused the original incident. Encoding it as a hard gate means the exact regression that took ninety minutes to notice now aborts the canary in three.

Enforce policy at admission, and roll it out in dry-run. OPA Gatekeeper ConstraintTemplates define the rules; start every new constraint in dryrun so you see what would be blocked without breaking deploys, then flip to enforce once the violations are clean. A representative set for this platform: images only from *-docker.pkg.dev/claims-prod/*, a valid Cosign signature present, CPU/memory limits set, no privileged: true, no latest tag. The signed-image constraint is what makes the whole supply-chain story enforceable rather than aspirational.

Enterprise considerations

Security & supply chain. The design is defense-in-depth across the lifecycle: no static cloud keys (OIDC federation), two-tool IaC scanning (Checkov for deterministic gates, Wiz Code for exposure-aware prioritization), keyless signing with SLSA provenance, and OPA Gatekeeper verifying signatures and constraints at admission so only known-good artifacts run. At runtime, CrowdStrike Falcon sensors run as a DaemonSet on the GKE node pools for threat detection on the workloads themselves, feeding the payer’s SOC. HashiCorp Vault holds both application secrets (delivered to pods via the Vault Secrets Operator with short leases, never as static Kubernetes Secrets) and any residual pipeline secrets. Access to Argo CD and GitHub federates through Okta as the workforce IdP, brokered to Entra ID where Azure-side RBAC is needed, so an engineer’s group membership maps directly to what they can sync, override, or approve — and a single source of identity means offboarding actually removes access everywhere.

Cost optimization. Progressive delivery is cheaper than blue-green precisely because it never runs a full second copy of the service. The levers that matter on GKE:

Datadog’s usage metrics feed the right-sizing, and because rollbacks are automatic and fast, a bad release burns a few minutes of canary capacity instead of a full deploy plus an emergency redeploy.

Scalability. Each half scales on its own axis. GitHub Actions parallelizes across repos and runners, so build throughput grows with concurrency, not a shared bottleneck. Argo CD scales to hundreds of applications with the app-of-apps pattern and sharded application controllers; for many clusters, point one Argo CD at all of them or run ApplicationSets to template environments. The canary mechanism is per-service, so adding services adds independent rollouts, not coordination overhead. The natural ceilings are the GKE control-plane and node quotas and the Datadog metrics query volume during many simultaneous analyses — both planned for as the service count grows.

Failure modes, and what each one looks like. Name them before they page you.

• Datadog metrics gap during a canary — if the query returns no data (an agent hiccup, a renamed metric), a naive analysis treats “no data” as “healthy” and promotes blind. Mitigation: set failureLimit/inconclusiveLimit so missing data is inconclusive, not pass, and the rollout pauses for a human rather than promoting.
• Argo CD / Git drift — someone hotfixes the cluster with kubectl edit and self-heal silently reverts it mid-incident, or auto-sync fights a manual change. Mitigation: self-heal on for prod, a documented break-glass that disables sync deliberately, and alerts on out-of-sync status.
• A stuck canary — analysis is inconclusive and the rollout sits paused at 25% indefinitely. Mitigation: progressDeadlineSeconds to bound the pause and a Datadog monitor that pages when a rollout exceeds its expected window.
• Gatekeeper webhook down — if the admission webhook is unreachable with failurePolicy: Fail, all deploys block; with Ignore, policy silently stops being enforced. Mitigation: run the webhook HA across zones, scope Fail to the namespaces that truly need it, and monitor webhook health as a first-class signal.
• Signature verification false-negative — a legitimately built image fails admission because the signing identity or trust root drifted. Mitigation: verify signatures in CI (fail fast) and keep the Gatekeeper trust config in the same GitOps repo so it changes reviewably.

Reliability & DR (RTO/RPO). GitOps gives you a strong recovery story almost for free: because Git is the desired state, rebuilding a cluster is “stand up GKE, install Argo CD, point it at the config repo, let it reconcile.” Decide the numbers per tier — for this platform, RTO 30 minutes to reconstitute a cluster from Git and RPO near zero for desired state (it’s all in Git, replicated by the Git host). Stateful dependencies (databases, Vault) have their own backup/replication SLAs that dominate the real RPO; the deploy layer itself is reproducible from source. Run Argo CD HA, and keep the config repo and signing trust roots backed up off the primary Git host.

Observability. The pipeline emits a deployment marker to Datadog at the start of every rollout, so a latency or error-rate change on a dashboard is visually correlated with the exact release that caused it. Instrument the canary’s AnalysisRun results as first-class events, track rollout success rate, mean time to rollback, canary abort rate, and lead time from merge to 100% — the DORA-style metrics the platform team reports up. Argo CD’s own UI and audit log answer “what’s running and who synced it” for any point in time, and a ServiceNow change record is opened on every prod promotion and incident on every automated rollback, giving compliance the documented trail the mandate demanded.

Governance. Pin everything that can drift: image digests not tags in the config repo (an immutable reference, never a moving latest), Argo CD app revisions to specific Git SHAs for prod, and Gatekeeper policies in version control so a rule change is a reviewed PR. Promotion through environments is a Git PR with Okta-backed approval and a ServiceNow change gate on prod, so “who approved this” is answerable by design. Log every signature verification and policy decision for audit. The combination — Git as the record, signatures as provenance, admission as enforcement, automated analysis as the safety net — is what lets a payer say “yes” to shipping daily on a system that touches PHI.

Explicit tradeoffs

Accept these or don’t build it. Progressive delivery adds real moving parts: a service mesh or Gateway API for weighted traffic, an analysis configuration you must tune (too-tight thresholds abort good releases and erode trust; too-loose ones let regressions through), and the discipline of two repositories and signed artifacts. Canaries also make every deploy slower by design — a release that used to flip instantly now takes fifteen-plus minutes of staged rollout and analysis. That is the point for a regulated, member-facing API, and it is pure overhead for an internal batch job that no member ever sees. The GitOps model means there is no kubectl apply shortcut when you’re firefighting; you change Git and wait for reconcile, which is safer and occasionally maddening, so the break-glass path has to be documented and practiced before you need it.

The alternatives, and when they win. If your service is stateless, idempotent, and has no meaningful “half-deployed” risk, rolling updates are simpler and need none of this machinery. If you genuinely need an instant, atomic switch with trivial rollback and can pay for double capacity during cutover, blue-green beats canary — it’s the right call for a database schema migration coordinated with a deploy. If you’re a small team optimizing for speed over control, push-based CD straight from GitHub Actions stands up in an afternoon; graduate to GitOps and admission policy when audit, supply-chain, or blast-radius requirements demand it. And if you’re already deep in a single cloud, the cloud-native equivalents (Cloud Deploy, Binary Authorization) cover much of this — the Argo stack earns its place when you want the same delivery model across clusters and clouds without re-platforming per provider.

Going deeper

The reference architecture above is the what. This section is the how it actually works — the mechanics an experienced engineer needs to operate it, tune it, and debug it at 2 a.m.

Progressive delivery vs a plain rolling update

A Kubernetes Deployment already does a rolling update: it creates pods of the new version and deletes old ones a few at a time, bounded by maxSurge and maxUnavailable, until every pod is new.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 0

That is genuinely safer than a big-bang replace — but its only notion of “healthy” is the pod’s readiness probe. A readiness probe answers “is this process accepting connections?”, not “is this version returning correct benefits-eligibility responses?” The original incident passed every readiness probe and still told members they had no coverage. A rolling update also has no decision point: once it starts it marches to 100% unless a human notices and runs kubectl rollout undo.

Progressive delivery — what Argo Rollouts adds — keeps the incremental idea but inserts two things a Deployment lacks: weighted traffic control (send exactly 5%, then 25%, to the new version, independent of pod counts) and analysis between steps (query real business and latency metrics and decide whether to proceed). The rollout pauses, judges, and either promotes or aborts on evidence.

The trade is real: rolling updates need nothing extra, and for a stateless internal job with no meaningful half-deployed risk they are the correct, simpler choice. You reach for progressive delivery when a bad version has real blast radius and you want a machine, not a human, to catch it at 5%.

How Argo Rollouts actually shifts traffic

A Rollout is a drop-in replacement for a Deployment’s spec (same pod template) that the Argo Rollouts controller manages as two ReplicaSets at once — a stable one and a canary one — fronted by two Services (stableService and canaryService). How the 5% actually reaches the canary depends on the trafficRouting provider:

• A service mesh or ingress that supports weighting — Istio (VirtualService), SMI (TrafficSplit), the Gateway API (HTTPRoute), NGINX, AWS ALB, Traefik, or a plugin — lets Rollouts set a precise percentage regardless of how many pods exist. This is what the lesson’s Rollout uses via Istio.
• No traffic router at all — Rollouts falls back to a basic canary: it approximates the weight with pod counts (5% weight ≈ 1 canary pod out of 20). Cheap, but the granularity is your replica count, and every request still round-robins across both versions, so it is coarser than mesh-based weighting.

The controller never mutates the stable ReplicaSet while the canary is under test, which is why an abort is instant: it just sets the canary weight back to 0 and scales the canary ReplicaSet down — the stable version was serving the other 95% the whole time.

The analysis engine: AnalysisTemplate, AnalysisRun, and honest verdicts

An AnalysisTemplate is a reusable metric query with pass/fail logic; when a rollout reaches an analysis step it instantiates one as an AnalysisRun. The most common provider is Prometheus (Datadog — used above — plus New Relic, CloudWatch, Wavefront, and a raw web/job provider are all supported):

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      count: 5                       # sample 5 times, one minute apart
      successCondition: result[0] >= 0.99
      failureLimit: 1                # one failed sample fails the run
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

An AnalysisRun ends in one of four states — Successful, Failed, Inconclusive, or Error — and the knobs that make it honest are:

• successCondition / failureCondition — expressions over result. Give one or the other; anything not matching flips to the opposite verdict.
• failureLimit — how many measurement failures are tolerated before the run fails (and the rollout aborts). 0 means a single breach aborts.
• inconclusiveLimit — measurements that are neither clearly pass nor fail; enough of them pauses the rollout for a human.
• count / interval — inline analysis samples a fixed number of times between steps; background analysis (spec.strategy.canary.analysis) runs for the whole rollout in parallel, so a regression that appears at minute 12 still aborts.

The subtle failure is missing data. If Prometheus returns an empty vector — a renamed label, a scrape gap, a crashed canary that stopped emitting — result[0] does not exist, which surfaces as Error/Inconclusive, not a silent pass. Keep it that way: set failureLimit/inconclusiveLimit so no-data pauses for a human instead of promoting blind. Treating “no error metric” as “no errors” is exactly how a broken release promotes green. Also set consecutiveErrorLimit so a flaky metrics endpoint does not abort a healthy rollout on the first timeout.

The pause / promote / rollback loop

The rollout is a small state machine you can drive by hand or leave to analysis. There are two kinds of pause:

• Timed — pause: { duration: 5m } — hold for a fixed window (often just long enough for background analysis to gather data), then continue automatically.
• Indefinite — pause: {} with no duration — hold forever until someone explicitly promotes. This is the manual approval gate; a change-management sign-off lives here.

You drive and observe the loop with the kubectl argo rollouts plugin:

kubectl argo rollouts get rollout member-portal-api --watch   # live step / weight / analysis view
kubectl argo rollouts promote member-portal-api               # advance past the current pause
kubectl argo rollouts promote member-portal-api --full        # skip all remaining steps + analysis
kubectl argo rollouts abort  member-portal-api                # send 100% back to stable now
kubectl argo rollouts undo   member-portal-api                # revert to a previous revision
kubectl argo rollouts retry  member-portal-api                # resume an aborted rollout

Automatic aborts come from analysis: a Failed AnalysisRun sets the rollout Degraded and shifts traffic back to stable with no human in the loop. Bound the whole thing with progressDeadlineSeconds so a rollout stuck “inconclusive” at 25% does not sit paused forever — it fails and pages instead. In GitOps, remember the loop lives below Git: abort is a live cluster action for a firefight, but the durable fix is reverting the digest in the config repo — otherwise Argo CD will re-sync the bad version straight back.

Two ways to promote: a PR to the config repo, or Argo CD Image Updater

CI has to get the new image digest into Git somehow, and the choice of mechanism is really a choice about who approves prod:

• PR to the config repo (what the lesson uses for prod). CI opens a pull request that bumps the digest in the environment overlay. CODEOWNERS + branch protection force a human (or a ServiceNow change gate) to approve the prod overlay, while dev/stage auto-merge. The audit trail is the merge, with an approver’s name on it.
• Argo CD Image Updater. A controller watches Artifact Registry and, on a new tag/digest matching your rules, writes the change back to Git itself — no CI commit step:

metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=us-docker.pkg.dev/claims-prod/member-portal-api
    argocd-image-updater.argoproj.io/api.update-strategy: digest
    argocd-image-updater.argoproj.io/write-back-method: git   # commit to Git, not just live state

Image Updater is fast and hands-off — perfect for dev and stage — but it removes the human approval a regulated prod deploy needs, so most payer-style shops use it below prod and PRs at prod. Whichever you pick, write-back-method: git keeps Git authoritative; the alternative (argocd, writing to live app state) reintroduces the drift GitOps exists to kill.

At scale this all sits under app-of-apps or an ApplicationSet that templates one Argo CD Application per service or per cluster, with sync waves (argocd.argoproj.io/sync-wave annotations) ordering dependent resources. One useful subtlety: Argo CD ships a built-in health check for the Rollout CRD, so an app stays Progressing (not falsely Healthy) while a canary is mid-analysis — the CD tool and the delivery controller agree on what “done” means.

Admission gates: OPA Gatekeeper, Kyverno, and image-signature verification

Signing an image proves who built it; it is worthless unless the cluster refuses anything unsigned. That enforcement is an admission controller — a webhook the API server calls before persisting an object. Three current options:

• OPA Gatekeeper — policies are Rego inside ConstraintTemplates, instantiated as Constraints. Roll every new rule out in dryrun (audit-only) before flipping to deny:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata: { name: only-claims-registry }
spec:
  enforcementAction: dryrun          # flip to deny once violations are clean
  match:
    kinds: [{ apiGroups: [""], kinds: ["Pod"] }]
  parameters:
    repos: ["us-docker.pkg.dev/claims-prod/"]

• Kyverno — policies are plain YAML, which most teams find easier, and it verifies Cosign signatures natively with verifyImages (no extra component). Set validationFailureAction: Audit first, then Enforce:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-signed-images }
spec:
  validationFailureAction: Enforce   # Audit first to roll out safely
  rules:
    - name: verify-cosign-keyless
      match: { any: [{ resources: { kinds: [Pod] } }] }
      verifyImages:
        - imageReferences: ["*-docker.pkg.dev/claims-prod/*"]
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/claims-org/*"
                    rekor: { url: "https://rekor.sigstore.dev" }

That policy is the enforcement half of the lesson’s Cosign story: it admits an image only if it carries a keyless signature from your GitHub Actions OIDC identity, recorded in Rekor. An attacker who pushes a malicious image to the registry cannot get it admitted, because they cannot produce that signature.

• Sigstore policy-controller and Kubernetes-native ValidatingAdmissionPolicy (CEL, in-tree and GA since 1.30) are the two other paths — the first is signing-specialized, the second removes the external webhook (and its failure mode) for policies expressible in CEL. Gatekeeper can also verify signatures via an external-data provider such as Ratify.

Whichever engine you run, the operational rule is the same: a Fail failure policy on an unreachable webhook blocks all deploys, while Ignore silently stops enforcing — so run the webhook HA and monitor its health as a first-class signal.

Canary is not the only Rollouts strategy: blue-green

The same Rollout object does blue-green when an instant, atomic switch matters more than gradual exposure — a schema-coupled release, say. Instead of canary you specify blueGreen, with an active and a preview Service and analysis around the cutover:

spec:
  strategy:
    blueGreen:
      activeService: member-portal-active
      previewService: member-portal-preview
      autoPromotionEnabled: false          # hold for approval before the switch
      prePromotionAnalysis:  { templates: [{ templateName: success-rate }] }
      postPromotionAnalysis: { templates: [{ templateName: success-rate }] }
      scaleDownDelaySeconds: 300           # keep the old version warm for fast rollback

prePromotionAnalysis vets the new version on the preview Service before it takes live traffic; postPromotionAnalysis watches after the switch; scaleDownDelaySeconds keeps the old ReplicaSet alive so rollback is re-pointing a Service, not a redeploy. Blue-green costs ~2× capacity during the window (as the lesson’s tradeoffs note) but buys the atomic switch that canary deliberately gives up.

Practice challenges

Work these in order — they escalate from beginner to advanced. Try each before you expand the solution.

Challenge 1 — Progressive delivery or a rolling update? (beginner)

You have two services: (a) an internal nightly report generator — stateless, no live users; and (b) the member-facing eligibility API. For each, choose a plain rolling update or an Argo Rollouts canary, and justify it in one line.

<details> <summary>Solution</summary>

• (a) report generator → plain rolling update (a normal Deployment). There is no live traffic to shift and no blast radius worth the extra machinery; readiness probes are enough.
• (b) eligibility API → Rollouts canary with metric analysis. A bad version has real member impact, so it must prove itself on 5% of traffic and roll back on evidence.

Why: progressive delivery buys blast-radius control and metric-based abort — worth it only when a bad release actually hurts someone; otherwise the simpler rolling update wins. </details>

Challenge 2 — Find the pipeline boundary (beginner)

Name the single place where the CI half and the CD half of this architecture meet, and state one credential CI must not hold.

<details> <summary>Solution</summary>

They meet only at the Git config repository: CI writes a commit, and Argo CD pulls it. CI must not hold cluster credentials (a kubeconfig / cluster-admin token) — the cluster pulls, and nothing pushes into it from outside.

Why: keeping the boundary at a Git commit is what makes deploys auditable and keeps CI runners off the cluster’s credential surface — the entire point of pull-based GitOps. </details>

Challenge 3 — Write the canary steps (intermediate)

Write the steps: block for a canary that goes 10% → hold 5 min → run analysis → 50% → hold 5 min → 100%.

<details> <summary>Solution</summary>

steps:
  - setWeight: 10
  - pause: { duration: 5m }
  - analysis:
      templates: [{ templateName: success-rate }]
  - setWeight: 50
  - pause: { duration: 5m }

There is deliberately no final setWeight: 100 — reaching the end of steps promotes to 100% automatically.

Why: setWeight sets the canary traffic percentage and pause holds between shifts; Rollouts implicitly finishes at 100% when the step list ends, so an explicit setWeight: 100 is redundant. </details>

Challenge 4 — Make “no data” safe (intermediate)

An AnalysisTemplate uses successCondition: result[0] >= 0.995. During a canary the metric endpoint returns an empty result for three minutes. A colleague says “no data means no errors, so it should pass.” Why are they wrong, and which two fields make the behavior explicit?

<details> <summary>Solution</summary>

An empty vector means result[0] does not exist, so the condition cannot be satisfied — the measurement is an error/inconclusive, not a pass. Set failureLimit and inconclusiveLimit (and consecutiveErrorLimit for a flaky endpoint) so missing data pauses for a human instead of promoting.

Why: “no signal” is not “good signal”; a canary that treats missing metrics as success will happily promote a version that crashed so hard it stopped emitting. </details>

Challenge 5 — Enforce signed images, safely (advanced)

You must require that only images signed by your GitHub Actions OIDC identity can run, but you cannot risk blocking existing prod workloads on day one. Outline the rollout in two steps, and name the field that makes it non-blocking first.

<details> <summary>Solution</summary>

1. Deploy a Kyverno ClusterPolicy with verifyImages (keyless attestor: your token.actions.githubusercontent.com issuer + repo subject) set to validationFailureAction: Audit — it reports violations without blocking.
2. Watch the policy reports until existing workloads are clean/signed, then flip to validationFailureAction: Enforce.

The Gatekeeper equivalent is enforcementAction: dryrun → deny.

Why: audit/dry-run first surfaces what would break without causing an outage, so you fix unsigned workloads before the gate starts rejecting them — the same discipline as shipping any admission policy. </details>

Challenge 6 — Choose the promotion path (advanced)

A regulated prod environment needs a named human approval on every deploy; the dev environment wants zero-touch promotion the instant CI publishes an image. Pick the promotion mechanism for each, and name the one Argo CD Image Updater setting that keeps Git authoritative.

<details> <summary>Solution</summary>

• Dev → Argo CD Image Updater, watching the registry and committing digest bumps automatically (update-strategy: digest).
• Prod → PR to the config repo with CODEOWNERS / branch protection (and a ServiceNow change gate) so a human approves the overlay change.
• Keep Git authoritative with write-back-method: git (not argocd), so Image Updater commits to the repo rather than writing to live app state.

Why: prod needs an attributable approval that only a Git PR gives you, while dev optimizes for speed; write-back-method: git ensures even the automated path leaves the drift-free, auditable source of truth GitOps depends on. </details>

Common beginner mistakes

• “GitOps just means my CI runs kubectl apply.” That is push-based CD, and it is the opposite of GitOps. GitOps is pull-based: a controller inside the cluster (Argo CD) reads desired state from Git and converges to it, so drift self-heals and Git is the audit log. If your pipeline holds a kubeconfig and applies manifests, you have automation, not GitOps.
• “A canary is just a slow rolling update.” A rolling update has no decision point — it marches to 100% on readiness probes alone. A canary pauses and judges on real metrics between steps and can abort automatically. The pause-and-analyze loop, not the gradualness, is the whole value.
• “A green analysis means the release is safe.” It means your queries did not breach their thresholds. A query that returns no data (renamed label, crashed canary) or one scoped too broadly can score green on a broken release. Make no-data inconclusive, scope metrics to the canary, and gate on the business metric that actually matters (here, eligibility-success), not just CPU.
• “Signing the image makes the supply chain secure.” Signing proves provenance; it changes nothing until an admission policy requires that signature at the cluster door. Cosign without a Gatekeeper/Kyverno verify rule is a lock you bought but never installed.
• “I’ll flip the new admission policy straight to enforce.” A policy that starts in deny / Enforce can block every existing non-conforming workload the moment it lands. Always roll out in dryrun / Audit, read the violation reports, fix them, then enforce.
• “When prod breaks, I’ll just kubectl edit the fix.” In GitOps, Argo CD’s self-heal reverts your live edit back to what Git says — mid-incident. The fix is a commit (or a documented, deliberate break-glass that pauses sync), not a hand-edit the controller will fight.
• “kubectl argo rollouts abort fixed it.” Abort is a live action; the config repo still points at the bad digest, so the next sync re-deploys it. Durable rollback is reverting the digest in Git.
• “Progressive delivery is strictly better, so use it everywhere.” It adds a mesh, analysis tuning, and slower deploys. For a stateless internal job with no blast radius it is pure overhead — a plain rolling update is the right call. Match the ceremony to the risk.

Glossary

• Progressive delivery — releasing a new version to a small slice of traffic first, judging real health signals, then promoting or auto-rolling-back; the umbrella over canary and blue-green with analysis.
• GitOps — describing desired cluster state as files in Git and having an in-cluster controller pull and reconcile them, so Git is the single source of truth and audit log.
• CI (Continuous Integration) — here, GitHub Actions building, testing, scanning, and signing the artifact, then committing the new desired state to Git.
• CD (Continuous Delivery/Deployment) — getting that artifact into the running environment; in GitOps it is pull-based reconciliation, not a push from CI.
• Argo CD — the GitOps controller that continuously reconciles a Kubernetes cluster to a Git config repo and self-heals drift.
• Argo Rollouts — a Kubernetes controller providing Rollout, a Deployment replacement with canary/blue-green strategies, traffic weighting, and metric analysis.
• Rollout (CRD) — the argoproj.io/v1alpha1 object that replaces a Deployment and carries the progressive-delivery strategy.
• Canary — shifting a small percentage of traffic to the new version and judging it before wider promotion.
• Blue-green — running new (green) alongside old (blue) and switching traffic atomically once the new version is verified; instant rollback, ~2× capacity.
• Rolling update — the built-in Deployment strategy that replaces pods a few at a time (maxSurge/maxUnavailable), judged only by readiness probes.
• stableService / canaryService — the two Kubernetes Services Rollouts uses to address the stable and canary ReplicaSets during a canary.
• trafficRouting — the mesh/ingress integration (Istio, SMI, Gateway API, NGINX, ALB, …) that lets Rollouts set a precise traffic weight independent of pod counts.
• AnalysisTemplate — a reusable metric-query-plus-pass/fail definition that a rollout step instantiates to judge a version.
• AnalysisRun — a running instance of an AnalysisTemplate; ends Successful, Failed, Inconclusive, or Error.
• successCondition / failureCondition — expressions over the metric result that decide whether a measurement passes or fails.
• failureLimit / inconclusiveLimit — how many failed / inconclusive measurements are tolerated before the run (and the rollout) fails or pauses for a human.
• setWeight / pause — canary step primitives: set the canary traffic percentage; hold for a duration or indefinitely (the manual gate).
• promote / abort / undo — kubectl argo rollouts verbs to advance past a pause, send traffic back to stable, or revert to a prior revision.
• progressDeadlineSeconds — a bound on how long a rollout may make no progress before it is marked failed, so a stuck canary pages instead of hanging.
• Admission controller / webhook — a component the Kubernetes API server calls to accept or reject an object before it is persisted; where policy is enforced.
• OPA Gatekeeper — a policy admission controller using Rego ConstraintTemplates / Constraints; rules roll out in dryrun, then deny.
• Kyverno — a policy admission controller using YAML policies; verifies Cosign signatures natively via verifyImages; Audit, then Enforce.
• ValidatingAdmissionPolicy — Kubernetes-native, in-tree admission policies written in CEL (GA since 1.30); no external webhook to run.
• Cosign — the Sigstore tool that signs container images; keyless signing uses a short-lived OIDC identity and records to a transparency log.
• Keyless signing — signing with an ephemeral, OIDC-derived certificate instead of a stored private key; the proof lives in Rekor.
• Rekor — Sigstore’s public transparency log, recording signatures and attestations so they can be verified later.
• SLSA provenance / SBOM — a signed statement of how and where an artifact was built (SLSA) and a bill of its components (SBOM).
• Workload Identity Federation (OIDC) — exchanging a CI job’s short-lived OIDC token for cloud credentials, so no downloadable service-account key exists.
• Argo CD Image Updater — a controller that watches a registry and writes new image digests back to Git (or app state); write-back-method: git keeps Git authoritative.
• app-of-apps / ApplicationSet — Argo CD patterns for templating many Application objects (per service or per cluster) from one source.
• sync wave — an Argo CD annotation that orders resource application within a single sync.
• Config repo — the separate Git repository holding rendered deploy manifests (Kustomize/Helm) that Argo CD watches, distinct from the application source repo.
• Digest (vs tag) — an immutable content-hash reference to an image (@sha256:…); pinned in the config repo so deploys are reproducible, unlike a moving tag.

The shape of the win

For the claims-platform team, the payoff is not “fancier CI.” It is that a developer merges a PR on a Tuesday afternoon, the image builds keylessly, scans clean on Checkov and Wiz Code, gets signed, and lands in Git — and then nobody touches the cluster. Argo CD reconciles, Gatekeeper admits only the signed image, Argo Rollouts shifts 5% of member traffic to the new version, Datadog watches the eligibility-success rate, and when that metric dips the canary aborts and rolls back in three minutes with a ServiceNow ticket already filed — before the call center notices anything. The ninety-minute outage that started this project becomes a three-minute blip on a dashboard that auto-recovered. Everything upstream — the OIDC federation, the dual IaC scans, the Cosign signatures, the Gatekeeper admission, the Datadog gates — exists so that “ship daily on a system touching PHI” and “no human pushes to production” are the same sentence. Start narrower if you must, but for a regulated, member-facing service at velocity, this is where progressive delivery has to land.