Compliance & Audit in GitOps: Who Deployed What, Drift Evidence & Policy with OPA/Kyverno

Ask an auditor what they spend their week doing and the answer is nearly always the same: chasing evidence. Who approved this change? When did it actually go to production? Who could have changed it, and how do you know they didn’t? In a click-ops world those answers live in someone’s memory, a ticketing system nobody fills in consistently, and a cluster whose current state is a mystery until you kubectl get it. Reconstructing the trail after the fact is slow, incomplete, and — worst of all — not tamper-evident.

GitOps quietly solves the hardest part of that problem, almost as a side effect. When the only way a change reaches production is a commit merged through a pull request and reconciled by Argo CD, your delivery pipeline is your audit trail. Every change is a signed, timestamped, attributable, reviewed record in Git; every deployment is recorded by Argo CD; and every out-of-band change lights up as drift. You do not build a compliance system alongside your deployment system — the deployment system already emits the evidence, because it cannot operate without producing it.

This lesson is about turning that latent evidence into a real, defensible compliance posture. We will map GitOps artifacts to the actual controls auditors test (SOC 2 CC8.1, ISO 27001, PCI DSS, NIST 800-53, SOX ITGCs), reconstruct “who deployed what” from Git history plus Argo CD’s own history and events, treat drift as the unauthorized-change control it is, lock provenance down with GPG-signed commits, and enforce compliance with policy-as-code using both OPA Gatekeeper and Kyverno — at the pull request and at admission. Everything targets Argo CD 2.13+/3.x on Kubernetes 1.29+.

There is no cluster attached to this lesson, so every command output below is labelled representative — it shows the shape to expect, not a live run. Every manifest, ClusterPolicy, ConstraintTemplate and AppProject is schema-correct with real fields; secrets and key IDs are placeholders you replace. Policy patterns are the upstream Kyverno/Gatekeeper canonical forms, not invented ones.

Why this matters

Change management is the control auditors care about most, because uncontrolled change is where breaches and outages come from. Every framework has a version of it: changes to production must be authorized, tested, approved, documented, and traceable to a person. The traditional way to satisfy that is a change-advisory board, tickets, and screenshots — a paper trail bolted onto a process that can be bypassed by anyone with cluster credentials. The paper says one thing; the cluster may say another; and proving they match is manual.

GitOps inverts this. The desired state lives in Git, and a controller — Argo CD — continuously makes the cluster match Git and nothing else. That single architectural choice hands you four properties auditors love, for free: attribution (every change is a commit by a known author), authorization (it merged through a reviewed PR), traceability (the history is immutable and diffable), and detection (anything the cluster does that Git didn’t say is drift). The job of this lesson is to make each of those properties legible to an auditor and to enforce the parts that are otherwise just convention.

The mental model to carry through: the pipeline is the evidence, and policy-as-code is the enforcement. Git gives you the tamper-evident record of intent; Argo CD gives you the record of execution and the drift signal; and OPA/Kyverno give you the machine-checked guarantees that what got deployed actually met the rules — checked twice, once at the PR (fast feedback) and once at admission (an un-bypassable backstop). Miss any one leg and you have a story with a hole an auditor will find.

The contrast with the old way is stark, and it is worth having this table ready because it is the “before/after” a compliance lead will immediately grasp:

Here is the whole chain, end to end. Read it left to right: a proposed change enters only as a signed, reviewed pull request; policy runs at PR time and again as an admission gate; Argo CD verifies the commit signature, syncs, and records it; drift detection watches continuously; and every stage drops an artifact into the evidence bundle.

The six badges mark the load-bearing controls: separation of duties is structural, not a policy document (1); policy runs twice on purpose (2); signed commits make provenance forgery-resistant (3); admission is the backstop that catches what shift-left misses (4); drift is the continuous unauthorized-change control (5); and the evidence assembles itself as a by-product (6). The rest of the lesson is those six ideas, in depth, with real manifests.

Git as a tamper-evident audit log

Start with the artifact every other control leans on: the Git commit. A commit is not just a diff. It is a record that binds what changed (the patch), when (the author and committer timestamps), who (author identity, and with signing, cryptographic proof of it), and — through the commit message and the linked pull request — why. Stack commits into a history and you have an append-only, content-addressed ledger: every commit’s hash includes its parent’s hash, so you cannot alter a historical commit without changing every hash after it. That is what “tamper-evident” means precisely — you can rewrite history, but you cannot do it invisibly, and branch protection plus mirrored remotes make even that detectable.

Map the anatomy of a commit-plus-PR onto the questions an auditor asks, and the fit is exact:

The strategic payoff is that this one artifact satisfies the change management control across every framework at once. You do not implement SOC 2 change management and separately implement ISO change management — you implement “all production change flows through a reviewed, signed PR reconciled by Argo CD,” and it maps onto all of them. Here is that mapping explicitly; this table is the one to show an auditor first, because it turns your pipeline into a controls narrative:

One honest caveat so you do not oversell it: Git is tamper-evident, not tamper-proof. Someone with force-push rights on a protected branch could rewrite history, and the fix is procedural — protected branches, required signatures, disallowed force-push, and a second mirrored remote (or the SIEM copy of the events) so any rewrite is detectable. Auditors accept “tamper-evident with detection” as a strong control; they do not accept “trust me.” The rest of this lesson is about making the evidence both complete and hard to forge.

Who deployed what: reconstructing the deployment history

“Who deployed what, when?” is the single most common audit question, and in GitOps it has a precise, multi-source answer. Git tells you what the intent was and who authored it; Argo CD tells you when that intent was actually applied to a cluster and by whom it was triggered. You need both, because a commit merging is not the same event as the change reaching a cluster — auto-sync may lag, a sync window may hold it, or a human may have triggered it manually.

Start with Git. The manifest path is the history of that workload:

# What is running in prod for checkout, and who last touched each file?
git log --oneline --decorate -- apps/checkout/prod/
# a1b9f3c (HEAD -> main, tag: prod-2026.07.14) chore: bump checkout to v3.4.1 (#812)
# 7d2e004 fix: raise checkout memory limit to 512Mi (#805)
# c40188a feat: add owner label to checkout workloads (#799)

# Who changed THIS line, in which commit, when?
git blame -L 18,24 apps/checkout/prod/deployment.yaml
# a1b9f3c2 (Priya Nair 2026-07-14 09:12:41 +0000 19)         image: acme.azurecr.io/checkout:v3.4.1

What this proves: every line of desired state is attributable to a commit, an author, and a timestamp, and the commit links to PR #812 with its approval. That is attribution and authorization in two commands. Now cross to Argo CD for the execution record — when that revision actually synced to the cluster:

# Argo CD's own record of what synced, when
argocd app history checkout-prod
# ID  DATE                           REVISION
# 6   2026-07-14 09:14:02 +0000 UTC  main (a1b9f3c)
# 5   2026-07-12 15:30:11 +0000 UTC  main (7d2e004)
# 4   2026-07-09 08:02:55 +0000 UTC  main (c40188a)

The history is stored on the Application itself in .status.history (bounded by spec.revisionHistoryLimit, default 10), so it is declarative and backup-able like everything else. Rolling back is argocd app rollback checkout-prod 5 — and note that the rollback is itself a recorded event. These sources answer different questions, and an auditor will want the right one for each:

Argo CD’s audit events

Be honest about what Argo CD is here: it is not a dedicated audit-log product with an immutable store. Its audit trail is two streams — Kubernetes Events it emits into the argocd namespace for Application activity, and the structured request logs the argocd-server writes for API calls (logins, syncs, RBAC decisions, terminate-op, rollbacks). Every meaningful action produces one or both. The event reasons you care about:

# Representative: Argo CD events for one application
kubectl get events -n argocd --field-selector involvedObject.name=checkout-prod \
  --sort-by=.lastTimestamp
# LAST SEEN   TYPE     REASON               OBJECT                    MESSAGE
# 4m          Normal   OperationStarted     application/checkout-prod  Initiated sync to main (a1b9f3c) by priya@acme.io
# 3m          Normal   OperationCompleted   application/checkout-prod  Sync operation to a1b9f3c succeeded

There is one operational trap that is really a compliance trap: Kubernetes Events are ephemeral. The API server garbage-collects them after roughly an hour (--event-ttl, default 1h). If your only copy of “who synced what” is kubectl get events, your audit trail evaporates every hour. The control is to ship events and server logs off-cluster to a SIEM with retention — and because logging destinations are cloud-specific, this is where the multi-cloud edge appears:

The immutable-store column matters more than it looks: an auditor’s first challenge to any log-based control is “could someone delete or edit these logs to hide their tracks?” WORM storage — Azure immutable blobs, S3 Object Lock in compliance mode, GCS bucket retention lock — answers that with “no, not even an administrator, until the retention period expires.” That converts your shipped Argo CD events from a convenience into defensible evidence.

Drift is evidence: continuous unauthorized-change detection

Here is the idea most teams miss: drift is not a nuisance, it is a control. Argo CD compares Git (desired) to the live cluster (actual) on every reconcile, a few minutes apart by default. When they diverge, the Application goes OutOfSync. In a properly run GitOps estate the only legitimate cause of OutOfSync is a commit that has not been synced yet — because humans are not supposed to touch the cluster directly. So any OutOfSync that is not explained by a pending commit is, by definition, an out-of-band change: someone ran kubectl edit, a controller mutated something, or an attacker altered a workload. That is exactly the “detect unauthorized changes to production” control (SOC 2 CC7.2, NIST SI-7, PCI Req 10 / 11.5) — and Argo CD runs it continuously, for free, on everything it manages.

Read the states through a compliance lens:

The distinguishing move is argocd app diff, which shows exactly what diverged. If the diff is empty but the app is OutOfSync, the cause is a pending revision; if the diff shows a changed field that no commit introduced, that is your out-of-band change, and it needs to become an alert and a ticket:

# What exactly drifted? (empty output = live matches the target revision)
argocd app diff checkout-prod
# ===== apps/Deployment checkout ======
# 27c27
# <       replicas: 3        # Git says 3
# ---
# >       replicas: 8        # someone scaled it live — out-of-band change

Alerting on drift (tie it to notifications)

Detection is only a control if someone is told. Argo CD Notifications turns a state change into a message to Slack/Teams and — crucially for audit — to a webhook you can point at your SIEM. There is a built-in trigger for degraded health and sync failures; for pure OutOfSync drift you write a small custom trigger. The config lives in argocd-notifications-cm:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token          # resolved from argocd-notifications-secret
  # Custom trigger: fire when an app drifts out of sync
  trigger.on-out-of-sync: |
    - when: app.status.sync.status == 'OutOfSync'
      oncePer: app.status.sync.revision
      send: [drift-audit]
  # Built-in-style triggers you also want for audit
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [drift-audit]
  trigger.on-sync-status-unknown: |
    - when: app.status.sync.status == 'Unknown'
      send: [drift-audit]
  template.drift-audit: |
    message: |
      :rotating_light: {{.app.metadata.name}} is {{.app.status.sync.status}}/{{.app.status.health.status}}.
      Possible out-of-band change on {{.app.spec.destination.server}}.
      Target revision: {{.app.status.sync.revision}}.
    webhook:
      siem:
        method: POST
        body: |
          {"event":"argocd_drift","app":"{{.app.metadata.name}}",
           "sync":"{{.app.status.sync.status}}","health":"{{.app.status.health.status}}",
           "revision":"{{.app.status.sync.revision}}"}

You subscribe an application (or a whole project by default subscription) with annotations, routing the audit copy to a security channel and the SIEM webhook:

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-out-of-sync.slack: sec-gitops-audit
    notifications.argoproj.io/subscribe.on-health-degraded.slack: sec-gitops-audit
    notifications.argoproj.io/subscribe.on-out-of-sync.siem: ""   # the webhook service

oncePer: app.status.sync.revision is the detail that keeps this usable — it fires once per drifting revision instead of every reconcile, so the SIEM sees one clean event per incident rather than a storm. Argo CD’s notifications engine is covered in depth in its own lesson elsewhere in the course; the compliance-specific point here is that the drift trigger is what makes drift auditable — a timestamped, retained event — rather than merely visible in the UI until the next reconcile.

selfHeal as enforcement — and when it is the wrong control

syncPolicy.automated.selfHeal: true upgrades drift from detected to auto-remediated: when live diverges from Git, Argo CD reverts the cluster back to Git on the next reconcile. As a control this is powerful — it means an out-of-band change to a self-healing app is not just alerted, it is undone within minutes, so the window of unauthorized state is tiny. For a regulated prod workload, selfHeal is usually the right posture.

But selfHeal has a sharp edge that is itself a compliance concern: it will happily revert a change you meant to keep. During a break-glass emergency, if an on-call engineer patches a live Deployment to stop an incident and selfHeal is on, Argo CD reverts the fix and the incident resumes — and now your emergency action left no durable state and fought your tooling. The rule: selfHeal enforces the invariant “the cluster equals Git,” so during any authorized deviation you must either land the change in Git first (so there is nothing to revert) or disable selfHeal on that one app for the window. We will make that explicit in the break-glass runbook below.

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true       # revert out-of-band changes automatically = enforcement
    syncOptions:
      - RespectIgnoreDifferences=true   # don't fight controllers you deliberately ignore

Think of drift handling as a ladder of increasing control strength — pick the rung each app’s risk warrants, and be able to name which rung you are on when an auditor asks how unauthorized change is controlled:

Signed commits and verified provenance

Attribution is only as strong as the identity behind it, and plain Git identity is trivially forged — git config user.email ceo@acme.io and every commit you push claims to be from the CEO. For a control that says “changes are attributable to a person,” that is a real gap. The fix is cryptographically signed commits: the author signs each commit with a GPG (or SSH/Sigstore) key, and Argo CD refuses to deploy a revision unless its top commit is signed by a key you have explicitly trusted. Now write access to Git is no longer sufficient to deploy — an attacker would also need a trusted private signing key, which lives on a hardware token or in a developer’s keyring, not in the repo.

Argo CD enforces this at the project level with AppProject.spec.signatureKeys. List one or more trusted GPG key IDs; any Application in the project must resolve a targetRevision whose tip commit is signed by one of them, or the sync is refused:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  description: Regulated production — signed commits required
  sourceRepos:
    - https://github.com/acme/platform-gitops.git
  destinations:
    - server: https://prod-eu.example.com
      namespace: 'checkout-*'
  clusterResourceWhitelist: []        # deny cluster-scoped resources by default
  signatureKeys:
    - keyID: 4AEE18F83AFDEB23          # a GPG key ID you have registered & trusted

The trusted public keys are managed cluster-wide (they live in the argocd-gpg-keys-cm ConfigMap) via the CLI, and the project only references key IDs:

What actually happens on sync, depending on the commit’s signature:

# Representative: an unsigned commit blocked by a signed-commits project
argocd app get checkout-prod
# ...
# Health Status:   Healthy
# Sync Status:     OutOfSync
# Conditions:      ComparisonError: target revision a1b9f3c in Git is not signed, but a signature is required

Two honest caveats. First, signature verification checks the tip commit of the resolved revision — pin targetRevision to a signed tag or a branch whose merges are all signed, and require signed commits in branch protection so an unsigned commit can never become the tip. Second, this is one leg of provenance; the other is signed images (cosign) verified at admission, covered in the policy section. Commit signing proves who authored the desired state; image signing proves what artifact runs. Together they make provenance end to end — and both tie back to the hardening lesson Hardening Argo CD: Least Privilege, Network Policy & Admission Control, which treats the signing keys themselves as high-value secrets.

Policy-as-code gates: OPA Gatekeeper and Kyverno

Signed, reviewed commits prove who and what, but they do not prove the change was compliant — a signed, approved PR can still deploy a privileged pod from an untrusted registry with no owner label if nothing checks. That is the job of policy-as-code: machine-checked rules that pass or fail a manifest against your compliance requirements. Two engines dominate the Kubernetes world, and it is worth knowing both because auditors and platform teams encounter each.

Kyverno writes policies as Kubernetes-native YAML — no new language — and can validate, mutate, generate resources, and verify image signatures. OPA Gatekeeper writes policies in Rego (a purpose-built policy language) via ConstraintTemplate + Constraint, and is the Kubernetes front end for the broader OPA ecosystem. The comparison that actually drives a choice:

Whichever you pick, the same principle applies as with drift and signing: enforce at admission, but check first at the PR. Admission control is the un-bypassable backstop — the API server itself rejects a non-compliant resource no matter who applies it (Argo CD, a Helm release, or a human with kubectl). But admission gives feedback late, only when the change tries to land. Running the same rules at PR time with a CLI (conftest, kyverno apply, gator test) gives the developer feedback in seconds, in the PR, before merge. You want both, and they are genuinely complementary:

Kyverno: the compliance policy set

A Kyverno ClusterPolicy is a set of rules; each rule matches resources and then validates, mutates, generates, or verifies images. The rule types map cleanly onto compliance needs:

Here is a real, schema-correct policy bundling the compliance requirements the brief calls for — require an owner label, forbid the mutable :latest tag, disallow privileged containers, and restrict images to your three cloud registries. These are the upstream Kyverno canonical patterns, not invented ones:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: baseline-compliance
  annotations:
    policies.kyverno.io/title: Baseline compliance for regulated workloads
spec:
  background: true                 # also scan existing resources → PolicyReports
  rules:
    # 1) Attribution: every workload must carry an owner label
    - name: require-owner-label
      match:
        any:
          - resources:
              kinds: [Pod]        # Kyverno autogen extends this to Deployments etc.
      validate:
        failureAction: Enforce     # 1.10+ per-rule form; older: spec.validationFailureAction
        message: "The 'owner' label is required for audit attribution."
        pattern:
          metadata:
            labels:
              owner: "?*"          # ?* = must be present and non-empty

    # 2) No mutable :latest tag — require a tag, then forbid :latest
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "An explicit image tag is required."
        pattern:
          spec:
            containers:
              - image: "*:*"
    - name: forbid-latest-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "The mutable ':latest' tag is not allowed."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

    # 3) No privileged containers
    - name: disallow-privileged
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "Privileged mode is disallowed."
        pattern:
          spec:
            =(ephemeralContainers):
              - =(securityContext):
                  =(privileged): "false"
            =(initContainers):
              - =(securityContext):
                  =(privileged): "false"
            containers:
              - =(securityContext):
                  =(privileged): "false"

    # 4) Allowed registries only (ACR / ECR / Artifact Registry)
    - name: restrict-registries
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "Images must come from an approved registry."
        pattern:
          spec:
            containers:
              - image: "acme.azurecr.io/* | *.dkr.ecr.*.amazonaws.com/* | *-docker.pkg.dev/*"

A few load-bearing details. The =() in the privileged rule is Kyverno’s conditional anchor — “if this field exists, it must match” — which is how you say “privileged, if set, must be false” without forcing every pod to declare it. The ?* anchor means “must exist and be non-empty,” which is exactly what an attribution label needs. And you do not repeat the rule for Deployment, StatefulSet, DaemonSet, Job, CronJob — Kyverno’s auto-gen feature (pod-policies.kyverno.io/autogen-controllers) synthesizes the equivalent rules against the pod template of those controllers automatically. That single behavior is why a Pod-level policy actually governs your whole estate.

Read the policy back as a compliance catalog — requirement, the rule that enforces it, and the anchor that does the work — because this is the map you keep next to the framework controls:

The action is controlled per rule by validate.failureAction (Kyverno 1.10+); the older cluster-wide equivalent is spec.validationFailureAction, and per-namespace exceptions use spec.validationFailureActionOverrides:

The rollout discipline that keeps you from an outage: ship every new policy as Audit first. Let it run for a sprint, read the PolicyReports to see what would have been blocked, fix the offenders, then flip to Enforce. Going straight to Enforce on a cluster full of pre-existing violations blocks the next redeploy of half your workloads.

The verifyImages rule closes the supply-chain leg — only run images a trusted key signed:

    - name: verify-signed-images
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "acme.azurecr.io/*"
          failureAction: Enforce
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...   # your cosign public key
                      -----END PUBLIC KEY-----

OPA Gatekeeper: the same control in Rego

The equivalent in Gatekeeper is two objects: a ConstraintTemplate that carries the Rego and defines a new Constraint kind, and a Constraint that instantiates it with parameters and a match. The canonical “required labels” template (used verbatim from the upstream docs so you can trust it):

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels          # the kind the template defined
metadata:
  name: require-owner-label
spec:
  enforcementAction: deny        # deny | dryrun | warn
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet", "DaemonSet"]
  parameters:
    labels: ["owner"]

The component model and where the audit evidence comes from:

enforcementAction: dryrun is Gatekeeper’s equivalent of Kyverno’s Audit — same “observe before you enforce” discipline. And Gatekeeper’s audit controller periodically re-evaluates existing resources against every constraint and writes the failures to status.violations, so you can hand an auditor “here is every current violation of every policy” without waiting for the next deploy.

Testing policy at the PR (shift-left)

Both engines have offline CLIs so the exact same rules run in CI, blocking the merge before a bad manifest can even reach admission:

# Kyverno: run the cluster policy against rendered manifests in CI
kyverno apply baseline-compliance.yaml --resource ./rendered/ --audit-warn
# Applying 5 policy rule(s) to 12 resource(s)...
# pass: 10, fail: 2, warn: 0, error: 0, skip: 0
# policy baseline-compliance -> rule forbid-latest-tag FAILED: checkout uses acme.azurecr.io/checkout:latest

# Gatekeeper: gator tests constraints against files offline
gator test --filename ./rendered/ --filename ./policy/
# FAIL: require-owner-label  Deployment/checkout  you must provide labels: {"owner"}

# conftest: run raw Rego against config in CI
conftest test ./rendered/deployment.yaml -p ./policy/
# FAIL - deployment.yaml - main - image uses a disallowed registry: docker.io/library/nginx

A non-zero exit fails the CI job, the PR check goes red, and the change never merges — fast feedback, and a CI artifact you attach to the PR as evidence that policy passed at the time of the change.

Separation of duties: no human writes to prod

Separation of duties (SoD) is the control auditors probe hardest, because it is where insider risk and fraud live: no single person should be able to author a change and push it to production unchecked. In click-ops this is hard — anyone with cluster-admin can do both, and you enforce SoD with policy documents and hope. GitOps makes it structural: the only path to prod is a PR that a different person approves, merged into a branch that only Argo CD reads, and applied by Argo CD’s service account — which no human logs in as.

Lay out who can do what, and the separation is visible:

The keystone is the last row: Argo CD is the only writer to production, and it only ever applies what Git says. No human holds prod write credentials, so there is no one to bypass the review. That single fact maps directly onto the SoD controls:

To make it real you must remove human prod-write access — SoD is meaningless if a developer still has cluster-admin on the side. That means scoping Argo’s own RBAC (covered in RBAC: Local Users & Policies) so app teams get project-scoped sync/get at most, and scoping the reconcile boundary with AppProject (see AppProjects & Multi-Tenancy Boundaries). Branch protection does the Git half: require a review, require the author to be different from the approver, require signed commits, and forbid force-push. The two halves together are the control — and note it is the pull-based model that makes it possible, which is exactly why GitOps Principles: Push vs Pull matters here: a push pipeline needs prod credentials sitting in CI, re-opening the very hole SoD is meant to close.

Regulated delivery: change windows, break-glass, and data residency

Regulated environments add three requirements on top of “every change is reviewed”: deploys must respect change windows, genuine emergencies need a break-glass path that is still auditable, and data must stay in its jurisdiction.

Change windows are handled by AppProject.spec.syncWindows. A window is allow or deny, on a cron schedule, for a set of apps/namespaces/clusters. A deny window during business hours is a classic change-freeze:

spec:
  syncWindows:
    - kind: deny
      schedule: '0 9 * * 1-5'    # 09:00 Mon–Fri...
      duration: 8h               # ...for 8h = a 09:00–17:00 business-hours freeze
      applications:
        - '*'
      manualSync: false          # even manual syncs are blocked during the freeze
      timeZone: Europe/London

# Representative: confirm the freeze is active
argocd proj windows list prod
# ID  STATUS  KIND  SCHEDULE     DURATION  APPLICATIONS  MANUALSYNC
# 0   Active  deny  0 9 * * 1-5  8h        *             Disabled

Break-glass is the escape hatch for a real incident, and its whole point is that it stays auditable — the emergency change must leave the same trail a normal one does. The runbook:

The failure mode to design against is step 3 done as bare kubectl edit with no ticket and no backfill — that is an unauthorized change that also fights selfHeal. A good break-glass keeps the PR path even under pressure (a pre-approved emergency reviewer, a hotfix branch), so the trail is intact and Git stays the source of truth.

Data residency is where the cloud edge returns: keep both the workloads and the evidence in the required jurisdiction. Argo CD routes workloads to region-pinned clusters via AppProject destinations and cluster labels; the evidence stays in-region through the logging store you chose earlier:

Assembling the audit evidence (and reporting)

Everything above produces artifacts; the final skill is assembling them into the bundle an auditor accepts, and standing up the dashboards that show the posture continuously. The bundle is short because the pipeline did the work:

For continuous reporting rather than point-in-time export, the surfaces are:

# Representative: standing compliance posture from Kyverno PolicyReports
kubectl get clusterpolicyreport -o wide
# NAME                          KIND      PASS   FAIL   WARN   ERROR   SKIP   AGE
# cpol-baseline-compliance      Cluster   214    3      0      0       0      21d

# Drill into the failures — this is what the auditor wants to see closing over time
kubectl get polr -A -o jsonpath='{range .items[*].results[?(@.result=="fail")]}{.policy}{" "}{.rule}{" "}{.resources[0].name}{"\n"}{end}'
# baseline-compliance forbid-latest-tag legacy-cron
# baseline-compliance require-owner-label sandbox-debug

The point of the report surfaces is that compliance stops being a quarterly fire drill and becomes a number you watch: fail count on the ClusterPolicyReport trending to zero, drift alerts closing within an SLA, and every prod sync in argocd app history traceable to a signed, approved PR.

Hands-on lab

This lab builds the compliance stack at the config level — every manifest is real and schema-correct, and every output is labelled representative because there is no cluster attached. You can apply these against any Argo CD 2.13+/3.x install with Kyverno installed to make them live. We will: (1) require signed commits on a prod project, (2) add the Kyverno compliance policy, (3) wire drift alerting, and (4) assemble the evidence bundle.

Step 1 — Require signed commits on the prod AppProject.

# Trust the release signing key cluster-wide, then confirm
argocd gpg add --from ./release-signing.pub
argocd gpg list
# KEYID             TYPE  IDENTITY
# 4AEE18F83AFDEB23  rsa   Acme Release Signing <release@acme.io>

# prod-appproject.yaml — signatureKeys makes unsigned commits un-deployable
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  sourceRepos: ["https://github.com/acme/platform-gitops.git"]
  destinations:
    - server: https://prod-eu.example.com
      namespace: 'checkout-*'
  clusterResourceWhitelist: []
  signatureKeys:
    - keyID: 4AEE18F83AFDEB23
  syncWindows:
    - kind: deny
      schedule: '0 9 * * 1-5'
      duration: 8h
      applications: ['*']
      manualSync: false
      timeZone: Europe/London

What just happened: any Application in prod now refuses to sync a revision whose tip commit is not signed by the trusted key — provenance enforced by the platform, not by convention. The deny sync window adds a business-hours change freeze.

Step 2 — Apply the Kyverno compliance policy in Audit, then Enforce.

# Apply the policy (start in Audit — see the earlier baseline-compliance.yaml)
kubectl apply -f baseline-compliance.yaml
# clusterpolicy.kyverno.io/baseline-compliance created

# After a background scan, read what WOULD be blocked
kubectl get clusterpolicyreport cpol-baseline-compliance -o wide
# NAME                        PASS  FAIL  WARN  ERROR  SKIP  AGE
# cpol-baseline-compliance    118   6     0     0      0     3m

What just happened: background: true scanned existing workloads and produced a ClusterPolicyReport showing six pre-existing violations — the list you fix before flipping failureAction to Enforce, so enforcement never breaks a legitimate redeploy.

Step 3 — Prove enforcement blocks a non-compliant deploy.

# Representative: try to run a privileged, :latest, no-owner pod under Enforce
kubectl run bad --image=docker.io/nginx:latest --privileged
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
# baseline-compliance:
#   require-owner-label: 'The ''owner'' label is required for audit attribution.'
#   forbid-latest-tag: 'The mutable '':latest'' tag is not allowed.'
#   disallow-privileged: 'Privileged mode is disallowed.'
#   restrict-registries: 'Images must come from an approved registry.'

What just happened: four controls fired at admission in one shot — attribution, no mutable tag, no privilege, approved registry — and the request was rejected regardless of who sent it. That is the un-bypassable backstop.

Step 4 — Wire drift alerting to Slack and the SIEM.

# Apply the notifications config (see argocd-notifications-cm above) and subscribe the app
kubectl apply -f argocd-notifications-cm.yaml
kubectl annotate app checkout-prod -n argocd \
  notifications.argoproj.io/subscribe.on-out-of-sync.slack=sec-gitops-audit
# application.argoproj.io/checkout-prod annotated

# Simulate an out-of-band change and watch drift appear
kubectl scale deploy/checkout -n checkout-eu --replicas=8   # someone bypasses Git
argocd app get checkout-prod
# Sync Status:  OutOfSync   (drift detected → notification fired to sec-gitops-audit + SIEM)

What just happened: an out-of-band scale produced OutOfSync, which fired the on-out-of-sync trigger once (thanks to oncePer) to the security channel and the SIEM webhook — drift became a timestamped, retained audit event.

Step 5 — Assemble the evidence bundle.

# The four artifacts an auditor asks for, produced in four commands
git log --format=fuller -- apps/checkout/prod/ > evidence/01-git-history.txt
argocd app history checkout-prod            > evidence/02-sync-history.txt
kubectl get polr,cpolr -A -o yaml           > evidence/03-policy-reports.yaml
kubectl get appproject prod -n argocd -o yaml \
  cm argocd-rbac-cm -n argocd -o yaml       > evidence/04-access-controls.yaml
# Bundle: change history + authorship, what synced when, policy compliance, access controls

What just happened: the entire change-management, monitoring, and access-control story exported in four commands — because the pipeline emitted the evidence as it ran.

Teardown.

# Remove lab objects (config-only; no billable cloud resources were created)
kubectl delete clusterpolicy baseline-compliance
kubectl delete -f prod-appproject.yaml
kubectl delete cm argocd-notifications-cm -n argocd
argocd gpg rm 4AEE18F83AFDEB23
rm -rf evidence/

Common mistakes and troubleshooting

The failure modes here are subtle because a misconfigured compliance control looks the same as a working one until an auditor — or an attacker — finds the gap. Use real Argo CD/Kyverno states to diagnose.

Three gotchas cost the most audit credibility:

1. Audit mode mistaken for enforcement. A Kyverno policy in Audit (or a Gatekeeper Constraint in dryrun) records violations but allows them. Teams ship the policy, see it in the report, and believe production is protected — while non-compliant workloads keep deploying. Audit is a discovery stage, not a control. The control is Enforce/deny, and an auditor will ask which mode you are in. Roll out in Audit, then flip, and keep the flip in Git so the change to Enforce is itself an approved, dated commit.

2. Ephemeral evidence. The most common real hole is that the audit trail exists only in places that expire: Kubernetes Events (GC’d hourly), a local terminal’s scrollback, or a Slack channel with a 90-day retention on a free plan. If the evidence is not in a retained, ideally immutable store, you do not have a control — you have a demo. Ship events and logs to a SIEM with WORM retention before you claim the monitoring control, because the auditor’s question is always “show me last quarter,” not “show me now.”

3. SoD with a side door. GitOps SoD is only real if humans genuinely cannot write to prod. The classic finding is a beautifully reviewed PR pipeline sitting next to a cluster-admin kubeconfig that three engineers still use “for emergencies.” That side door voids the entire separation-of-duties narrative. Enforcing SoD means removing the human prod-write path (scope Argo RBAC and AppProject, delete standing admin credentials) and alerting on any manual change — not just building the nice path and leaving the old one open.

Cheat-sheet

Control → GitOps evidence (the auditor’s first table):

Policy-as-code quick reference:

Audit-evidence checklist (run before an audit):

Interview and exam questions

Q: Why do auditors consider Git a good audit log, and where is that claim weak? A: Git binds what/when/who/why into an append-only, content-addressed history where each commit hash includes its parent’s, so historical changes can’t be altered invisibly — it is tamper-evident. The weakness is that it is not tamper-proof: someone with force-push on a protected branch could rewrite history. You close that with protected branches, required signatures, no force-push, and an off-cluster (SIEM/mirror) copy so any rewrite is detectable.

Q: Map SOC 2 CC8.1 onto specific GitOps artifacts. A: CC8.1 (change management) requires changes to be authorized, tested, approved, documented, and implemented. The PR is the change (documented in the diff and message), the required review is the authorization/approval, CI checks are the testing, the merge is the record of approval, and the Argo CD sync (in argocd app history) is the implementation record. One artifact chain satisfies the whole control.

Q: In GitOps, what does OutOfSync mean from a compliance standpoint, and how is it a control? A: It means live cluster state diverges from Git. Since humans shouldn’t write to prod directly, an OutOfSync not explained by a pending commit is an out-of-band (unauthorized) change — which is exactly the “detect unauthorized change” control (SOC 2 CC7.2, NIST SI-7). It becomes a real control when you alert on it (notifications → SIEM) and optionally auto-revert with selfHeal. Use argocd app diff to tell a pending-commit OutOfSync from a genuine live edit.

Q: How do you enforce that only signed commits can be deployed, and what does it protect against? A: Set signatureKeys (a list of trusted GPG key IDs) on the AppProject; Argo CD then refuses to sync any revision whose tip commit isn’t signed by a trusted key. Trusted public keys are managed with argocd gpg add. It protects against forged authorship and against an attacker with mere Git write access — deploying now also requires a trusted private signing key.

Q: Kyverno vs OPA Gatekeeper — when would you choose each? A: Kyverno writes policies in Kubernetes-native YAML and can validate, mutate, generate, and verify images, with native PolicyReport CRDs — lower learning curve, and it can generate resources (NetworkPolicies, quotas). Gatekeeper uses Rego via ConstraintTemplate/Constraint — steeper, but a fit if you have Rego expertise or share policy with non-Kubernetes OPA. Choose Kyverno for fast K8s-native policy and generation; Gatekeeper when Rego/OPA is already your standard.

Q: Why run policy at both the PR and admission if admission is un-bypassable? A: Admission is the backstop but gives feedback late — only when a change tries to land, and after review effort is spent. PR-time checks (conftest, kyverno apply, gator test) give the developer feedback in seconds and block the merge, and produce a CI artifact proving policy passed at change time. Admission catches anything that skipped CI (force-merge, direct apply). Together = fast feedback plus an un-bypassable guarantee.

Q: What’s the difference between Kyverno Audit and Enforce, and how do you roll a new policy out safely? A: Audit records violations in PolicyReports but allows the request; Enforce rejects it at admission. Roll out as Audit with background: true first, read the reports to find pre-existing violations, fix them, then flip to Enforce. Going straight to Enforce blocks the next redeploy of every already-violating workload.

Q: How does GitOps enforce separation of duties structurally rather than by policy document? A: The only path to prod is a PR approved by someone other than the author (branch protection), merged into a branch only Argo CD reads, and applied by Argo CD’s service account — which no human logs in as. Author ≠ approver ≠ applier (NIST AC-5). It’s only real if you also remove human prod-write access; otherwise a side-door kubeconfig voids it.

Q: A developer patches a live prod Deployment during an incident and it keeps reverting. What’s happening and what’s the correct process? A: selfHeal: true is reverting the out-of-band change back to Git every reconcile. The correct break-glass is: open an incident ticket, either land the fix as a fast-tracked signed PR (so Git already matches) or disable selfHeal on that one app for the window, apply, backfill to Git, then re-enable selfHeal — keeping the audit trail intact throughout.

Q: Kubernetes Events show who synced, but the auditor wants last quarter’s. What went wrong and how do you fix it? A: Events are garbage-collected after ~1 hour (--event-ttl), so kubectl get events only shows the recent window. The fix is to ship Argo CD events and api-server logs to a SIEM with retention — ideally immutable (WORM) storage (Azure immutable blob, S3 Object Lock, GCS retention lock) — before relying on it as a monitoring control.

Q: What exactly would you hand an auditor to prove change management for one production service? A: git log/git blame on the manifest path (change history + authorship + timestamps), the PR records with approvals (authorization + SoD), argocd app history (what synced when), the SIEM records of who triggered syncs, the CI policy-run logs and PolicyReports (policy compliance), and the drift-alert history (continuous monitoring). The pipeline produced all of it as a by-product.

Q: How do you keep both workloads and audit evidence within a data-residency boundary? A: Pin workloads to region-labeled clusters via AppProject destinations and cluster labels so Argo only deploys them in-region, and keep the evidence in an in-region logging store (in-region Log Analytics/CloudWatch/Cloud Logging plus in-region immutable object storage). For strict sovereignty use the cloud’s boundary — Azure sovereign cloud, AWS GovCloud, Google Assured Workloads.

Key takeaways

• The pipeline is the evidence. In GitOps every production change is a signed, reviewed, timestamped commit reconciled by Argo CD, so change management, attribution, and authorization are emitted as by-products — map them to SOC 2 CC8.1, ISO 27001 A.8.32, PCI 6.5, and NIST CM-3/AC-5 directly.
• “Who deployed what” is a two-source answer: Git (git log/git blame) for authored intent, Argo CD (argocd app history, events) for actual execution and who triggered it. Ship the events to a SIEM because Kubernetes Events expire in ~1 hour.
• Drift is a control, not a nuisance. OutOfSync unexplained by a pending commit is an unauthorized change; alert on it (notifications → SIEM) and optionally enforce with selfHeal. That is continuous unauthorized-change detection for free.
• Sign your provenance. AppProject.signatureKeys makes unsigned commits un-deployable, so Git write access alone can’t ship a change — pair it with cosign image verification for end-to-end provenance.
• Policy-as-code, checked twice. Run the same compliance rules (owner label, no :latest, no privileged, approved registries) at the PR with conftest/kyverno apply/gator for fast feedback and at admission with Kyverno/Gatekeeper as an un-bypassable backstop. Roll out in Audit/dryrun, then Enforce/deny.
• Separation of duties is structural: author ≠ approver ≠ the Argo CD service account that applies — but only if you remove human prod-write access, or a side-door kubeconfig voids the whole control.
• Assemble evidence continuously, not quarterly. PolicyReport/ClusterPolicyReport CRDs, Constraint status.violations, argocd app history, and SIEM dashboards turn compliance into a number you watch trend to zero — and the audit bundle exports in a handful of commands.