Jenkins-to-Argo CD GitOps Delivery for a Regulated Enterprise

In a nutshell

This lesson builds one delivery pipeline out of two tools that do deliberately different jobs. Jenkins is the builder: it compiles the code, runs the tests, scans for vulnerabilities, signs the resulting container image, and then — instead of deploying anything — writes a small change into a Git repository proposing the new version. Argo CD is the deployer: it lives inside the Kubernetes cluster, watches that Git repository, and continuously makes the cluster match whatever Git says it should be. Nobody runs kubectl apply by hand. You change production by changing Git, and a robot makes it so.

The mental model is a thermostat. You set a target temperature; the thermostat never stops comparing the room to the target and quietly acts to close the gap. GitOps runs production the same way: Git holds the target (the “desired state”), and Argo CD is the thermostat that continuously compares the live cluster to Git and corrects any difference. Jenkins is the workshop upstream that builds the part and submits a written request to change the target — it never reaches over and fiddles with the thermostat itself. And because every change to the target is a signed, reviewed Git commit, the question every auditor eventually asks — who approved this, what exactly shipped, and does the cluster still match? — is answered by scrolling one logbook.

Why should a beginner care about a pattern this elaborate? Because it is how serious, regulated organizations — banks, hospitals, government contractors — actually ship to production, and the ideas generalize far below that scale. “Desired state in Git, an agent reconciles it, drift is corrected, rollback is git revert” is the spine of modern platform engineering. Learn it here in its full regulated form and the simpler versions look easy.

Level: Intermediate · Time: ~35 min

Before this lesson, you’ll get the most from it if you’re comfortable with the ideas in Jenkins shared libraries, Vault dynamic secrets in CI/CD, and keyless signing with Sigstore — this lesson stitches all three into one governed pipeline. If Argo CD’s reconciliation engine itself is new to you, GitHub Actions + Argo CD progressive delivery introduces it at a gentler pace.

After this lesson you’ll be able to:

• Explain the CI/CD split — why Jenkins proposes and Argo CD disposes — and why that split is what an auditor accepts as separation of duties.
• Draw the two-repository model (app repo vs config repo) and say what governs each.
• Write an Argo CD Application with self-heal and prune, and read a Synced / Healthy (or OutOfSync) status correctly.
• Add change-control guardrails: sync windows, project RBAC, PR approval, and signed-commit enforcement on the config repo.
• Promote a build from dev → stage → prod through Git, and roll back a bad release with a single git revert.
• Map each moving part to a compliance control family (SOX, PCI-DSS, FedRAMP, HITRUST/SOC 2).

A mid-sized health-insurance payer is two weeks out from its annual HITRUST and SOC 2 Type II audit, and the auditor has asked a question the platform team cannot currently answer cleanly: show me, for any production deployment in the last quarter, who approved it, what exactly was deployed, and proof the running cluster matches what was approved. Today the answer is a mess of Jenkins console logs, a kubectl apply someone ran from a laptop during an incident, and a ServiceNow change ticket that says “deploy v3.4” with no link to an artifact. The deployment that fixed a claims-processing outage at 2 a.m. has no change record at all. The CISO knows this is the finding that turns a clean audit into a qualified one — and in a regulated payer, a qualified SOC 2 is something the sales team has to explain to every prospect for a year.

The mandate that comes down is precise: every change to production must be reviewed, approved, recorded, reproducible, and continuously verified against drift — without slowing the forty-engineer delivery org to a crawl. This article is the reference architecture for that mandate: a Jenkins-to-Argo CD GitOps pipeline where Jenkins owns build, test, and signing; Git is the single source of truth for what production should look like; Argo CD continuously reconciles the cluster to Git; ServiceNow holds the auditable change record; and HashiCorp Vault means no long-lived secret ever sits in a pipeline. It is the design that lets the platform team answer the auditor’s question with a single Git history and a single Argo CD screen.

Why GitOps, and why the split between Jenkins and Argo CD

The instinct in a regulated shop is to bolt approval gates onto the existing push-based Jenkins pipeline — Jenkins builds, then Jenkins runs kubectl apply after a manual approval step. That fails the audit for a structural reason: a push pipeline gives Jenkins standing credentials to production, the approval is a moment in time that nothing re-verifies, and the moment someone runs a manual kubectl out-of-band, the cluster silently diverges from any record. You cannot prove the running state matches the approved state because nothing is continuously checking.

GitOps inverts the model. The desired state of production lives in Git as declarative manifests. An in-cluster agent — Argo CD — pulls that state and reconciles the cluster to match it, continuously. Three properties fall out of this, and each maps directly to a control an auditor cares about:

• Git is the audit log. Every change to production is a commit: authored, reviewed via pull request, timestamped, signed, immutable. “Who approved this and what changed” is git log.
• Drift is detected and corrected. If someone hand-edits a Deployment at 2 a.m., Argo CD flags the cluster as OutOfSync and can auto-revert it. The 2 a.m. fix now has to go through Git, or it gets undone — which is exactly the control the payer was missing.
• No CI system holds production credentials. Argo CD pulls from inside the cluster; Jenkins never touches the EKS API. The blast radius of a compromised Jenkins shrinks from “owns production” to “can write a manifest PR that still needs approval.”

The division of labor is the heart of the design. Jenkins is the continuous-integration plane: it builds the image, runs tests, scans, signs the artifact, and proposes a change by writing to Git. Argo CD is the continuous-delivery plane: it takes approved Git state and makes the cluster real. CI proposes; CD disposes. Keeping these planes separate — separate credentials, separate identities, separate audit trails — is what produces clean separation of duties.

Architecture overview

Trace the flow: Jenkins (the CI plane) builds, tests, scans with Wiz, signs with Cosign, and opens a pull request to the config repo — then stops; a human approval plus a ServiceNow change record gate the merge; and Argo CD (the CD plane) pulls the approved manifests, verifies signatures at admission, reconciles the EKS cluster, and from then on never stops comparing the cluster to Git, raising any drift as an incident.

The platform is built around two Git repositories with deliberately different governance, because conflating them is the most common GitOps mistake and the one that quietly breaks separation of duties.

The application repository holds source code and the Jenkinsfile. Developers own it; they push features and open PRs here all day. The config repository (the GitOps repo) holds the Kubernetes manifests that describe what runs in each environment — and it is governed like production, because it is production’s source of truth. A change to prod/ in the config repo requires a reviewed, approved PR; that PR is the change approval the auditor wants.

The build-and-propose flow (Jenkins, the CI plane):

1. A developer merges to main in the application repo. A webhook triggers the Jenkins pipeline, which runs on ephemeral Kubernetes agents (the Jenkins Kubernetes plugin spins a fresh pod per build, so no build state lingers and no agent accumulates secrets).
2. The pipeline — defined almost entirely in a Jenkins shared library so all forty services get identical, centrally-governed behavior rather than forty hand-copied Jenkinsfiles — runs unit and integration tests, builds the container image, and pushes it to Amazon ECR tagged with the immutable Git SHA, never latest.
3. The pipeline pulls every credential it needs — the ECR push token, the signing key reference, the config-repo deploy key — from HashiCorp Vault at runtime via the Vault Jenkins integration, authenticated with the Kubernetes auth method tied to the build pod’s service account. Nothing sensitive is stored in Jenkins itself.
4. Wiz Code scans the Terraform/Helm IaC and the application dependencies on the pull request, and the image is scanned for CVEs; a critical finding fails the build before anything is signed.
5. The pipeline signs the image with Sigstore Cosign (keyless, using an OIDC identity, or a Vault-held key), producing a cryptographic attestation that this artifact came from this pipeline. An SBOM is generated and attached.
6. The pipeline opens a pull request against the config repository that bumps the image tag for the target environment — typically by patching a Kustomize overlay or Helm values file. This PR is the proposed change. Jenkins’ job ends here: it has proposed, not deployed.

The approve-and-reconcile flow (Argo CD, the CD plane):

1. The config-repo PR triggers a ServiceNow change record automatically (via the ServiceNow DevOps integration or a webhook), linking the change to the artifact, the SBOM, the signer, and the Wiz scan result. For lower environments, a normal change auto-approves on green checks; for production, a standard or normal change requires CAB or delegated approval, which gates the PR merge.
2. On approval and merge to the config repo, Argo CD — watching that repo — detects the new desired state. Before applying, an Argo CD admission/sync hook verifies the Cosign signature (via a Kyverno or admission policy), so an unsigned or tampered image never reaches the cluster even if the manifest somehow references it.
3. Argo CD reconciles the EKS cluster to match Git: it applies the manifests, waits for the rollout to become healthy, and reports Synced / Healthy. If reconciliation fails, Argo CD surfaces it and (optionally) auto-rolls-back to the last-known-good Git revision.
4. From this point on, Argo CD continuously compares cluster state to Git. Any manual drift is flagged OutOfSync and, where self-heal is enabled, reverted — and the drift event is pushed to Datadog and raised as a ServiceNow incident, because in a regulated environment an unexplained production change is a security event, not a curiosity.

The two planes never share credentials. Jenkins can write a PR to the config repo; only Argo CD can talk to the cluster; only an approved human can merge to prod/. That triangle is separation of duties expressed in infrastructure.

Component breakdown

A few choices are load-bearing and worth the why, because teams routinely get them wrong.

Why a Jenkins shared library, not forty Jenkinsfiles. In a forty-service org, copy-pasted pipelines drift instantly — one team skips the scan, another signs with a stale key, a third logs a secret. A shared library centralizes the entire build-sign-propose flow into versioned Groovy that every repo references with one line. Want to add an SBOM step or tighten the Wiz gate across the whole estate? Change the library, bump the version, done. It also means the pipeline itself is code-reviewed and audited, which is its own control: an auditor can read one library instead of forty files.

// Jenkinsfile in every application repo — thin by design
@Library('platform-pipeline@v2.4') _

regulatedDelivery(
  service:      'claims-adjudication',
  env:          'prod',
  configRepo:   'git@github.com:payer/gitops-config.git',
  sign:         true,
  wizPolicy:    'block-critical'
)

Why sign images and verify at admission, not just at build. Signing in CI proves provenance, but proof is worthless if nothing checks it at the point of deployment. The pair is the control: Cosign signs in Jenkins, and Kyverno’s verifyImages policy rejects any image at the EKS admission webhook whose signature does not validate against the expected identity. An attacker who slips a malicious image into ECR cannot get it scheduled, because the cluster refuses to run anything this pipeline did not sign. Provenance becomes enforceable, not aspirational.

Why ServiceNow is the change record and not Jenkins’ own approval step. Jenkins can pause for a manual approval, but that approval lives in a console log no auditor trusts and no CAB process governs. Routing the change through ServiceNow ties the deployment to the organization’s real change-management workflow — CAB review for production, an approver who is not the author (separation of duties), and a permanent record linked to the artifact, SBOM, and scan. The 2 a.m. emergency fix becomes an emergency change in ServiceNow with retroactive review, instead of an untracked kubectl apply. The change ticket and the Git PR reference each other, so the audit trail is closed on both ends.

Implementation guidance

Stand up the GitOps repo and Argo CD with the app-of-apps pattern. Argo CD’s ApplicationSet / app-of-apps lets one root application manage every service’s deployment declaratively, so onboarding a new service is a Git commit, not a console click. Enable selfHeal and prune for production so drift is actively corrected and orphaned resources are removed:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: claims-adjudication-prod
  namespace: argocd
spec:
  project: payer-prod
  source:
    repoURL: git@github.com:payer/gitops-config.git
    targetRevision: main
    path: apps/claims-adjudication/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: claims
  syncPolicy:
    automated:
      selfHeal: true      # revert manual drift back to Git
      prune: true         # delete resources removed from Git
    syncOptions:
      - CreateNamespace=false

Wire Vault into Jenkins with the Kubernetes auth method, and keep TTLs short. The build pod’s Kubernetes service account is the Vault identity; the pipeline exchanges its projected SA token for a short-lived Vault token, reads exactly the secrets it needs, and the lease expires when the build pod dies. No secret is ever written to a Jenkins credential store, an environment file, or a job config. This is the control that turns “a compromised Jenkins leaks every production secret” into “a compromised build pod has a few minutes of narrowly-scoped access.” For the config-repo write, prefer a deploy key scoped to a single repo (or a short-lived GitHub App token), never a personal access token with org-wide reach.

Separate the two repos’ branch protection deliberately. The application repo can be relatively permissive — developers iterate fast. The config repo’s prod path must require: a reviewed PR, an approver who is not the author, a passing status check that the ServiceNow change is approved, and signed commits. This is where separation of duties is enforced in practice. A developer can ship code freely; they cannot put it into production without an independent approval recorded in both Git and ServiceNow.

Keep the per-deploy diff to one line. The only thing Jenkins changes in the config repo on a routine deploy is the image tag (a Kustomize images: patch or a Helm image.tag value). Everything else about the environment — replicas, resources, network policy — changes through deliberate, separately-reviewed PRs. This makes the routine path trivially auditable: a deploy PR that touches more than the image tag is a red flag a reviewer can spot in seconds.

Enterprise considerations

Security and separation of duties. The architecture is least-privilege by construction. Jenkins never holds an EKS credential — it can build, sign, and open a PR, nothing more. Argo CD is the only system with cluster write access, and it only applies what is in approved Git. Okta provides SSO and group-mapped RBAC across Jenkins, Argo CD, and ServiceNow, with MFA and automatic deprovisioning when an engineer offboards — so access reviews are a single source, not three. Layer on Wiz Code as the shift-left gate that blocks risky IaC and vulnerable dependencies before merge, and Wiz CSPM watching the running EKS estate for posture drift and exposure. CrowdStrike Falcon runs as a DaemonSet on the nodes for runtime threat detection, streaming to the SOC. Sigstore Cosign + Kyverno close the supply-chain loop: only signed, scanned artifacts run. The threat model worth naming explicitly is the insider or compromised-CI push — and the answer is that even a fully-owned Jenkins cannot deploy to production, because the merge to prod/ needs an independent human and the cluster rejects unsigned images.

Cost optimization. A GitOps pipeline’s cost is mostly compute for builds and the control planes; the levers are straightforward.

Spot interruptions on CI are low-stakes — a build simply retries — which makes CI an ideal Spot workload and an easy FinOps win to point at.

Scalability. Each plane scales independently. Jenkins scales horizontally on ephemeral Kubernetes agents, so build concurrency is bounded by cluster capacity, not a fixed agent pool. Argo CD scales with ApplicationSets to manage hundreds of applications, and shards its controller across replicas for very large estates or multiple clusters (a hub Argo CD managing many spoke EKS clusters via cluster registrations). The shared library means adding the forty-first service costs one thin Jenkinsfile and one config-repo directory — onboarding is O(1), which is the whole point of investing in the platform.

Failure modes, and what each one looks like. Name them before they page you.

• Config-repo PR merged but Argo CD won’t sync — usually an invalid manifest or a failing health check. Argo CD shows OutOfSync/Degraded and (with auto-rollback) reverts to last-good Git. Mitigation: render-and-validate manifests in the Jenkins PR check so a broken manifest never merges.
• Image signed but Kyverno rejects it — the signing identity doesn’t match the verify policy (common after a key or OIDC issuer rotation). Pods stay Pending with an admission error. Mitigation: test the verify policy in a lower environment on every signing-config change; alert on admission denials in Datadog.
• Vault unavailable mid-build — the pipeline can’t fetch the ECR or deploy-key secret and fails closed. This is correct behavior (no secret, no build) but it makes Vault a build-time dependency. Mitigation: run Vault HA (Raft) with auto-unseal; cache nothing, retry with backoff.
• Drift storm — a controller or operator outside Argo CD keeps re-writing a field, and self-heal fights it forever. Mitigation: ignoreDifferences for fields legitimately owned by another controller; alert if an app flaps sync state.
• ServiceNow integration down — change records don’t open, blocking the prod merge gate. Mitigation: a documented break-glass emergency change path with mandatory retroactive review, so a Sev-1 fix is never stuck behind a ticketing outage.

Reliability and rollback. Rollback in GitOps is a git revert — re-point the config repo at the previous known-good image tag, and Argo CD reconciles back. Because Git is immutable history, every prior production state is recoverable by commit, which makes “roll back to exactly what ran last Tuesday” a one-line operation with a full audit trail. For progressive safety on critical services, pair Argo CD with Argo Rollouts for canary or blue-green deploys so a bad version is caught at small blast radius before full rollout. Argo CD’s reconciliation is itself the reliability mechanism: even if the cluster is partially clobbered, it converges back to the declared state.

Observability and DORA. Instrument the full lead time in Datadog: pipeline duration, queue time, scan results, and — critically — deployment markers on the Argo CD sync so a latency or error regression in production can be correlated to the exact deploy that caused it. Emit the four DORA metrics the engineering leadership reports on — deployment frequency, lead time for changes, change-failure rate, and time-to-restore — straight from Git and Argo CD events, since GitOps makes all four directly measurable from commit and sync history. Monitor sync failures and drift events as first-class alerts: in a regulated payer, an unexplained OutOfSync is a potential security incident and auto-raises a ServiceNow ticket, not just a Slack message.

Explicit tradeoffs

Accept these or do not build it. GitOps adds a second repository and a mental model shift — engineers must internalize that you change production by changing Git, never by touching the cluster, and that hand-running kubectl will be reverted by the platform. That is a feature for compliance and an adjustment for muscle memory; expect a few weeks of “why did my manual change disappear” before the team trusts it. The Jenkins-and-Argo-CD split means two systems to operate instead of one all-in-one pipeline — more moving parts, two upgrade cadences, two RBAC surfaces — justified only when separation of duties and continuous drift-correction actually matter to you. They emphatically do for a regulated payer; they may not for a three-person startup. The signing-and-admission machinery (Cosign, Kyverno, key rotation) is real overhead you can skip for a hobby cluster and cannot skip when an auditor will ask whether unsigned images can run. And routing every production change through ServiceNow CAB adds latency to the deploy — deliberately, because the whole point is that production changes are reviewed and recorded.

The alternatives, and when they win. If you do not need a separate CI plane and Argo CD’s ecosystem, Flux is a lighter pull-based GitOps engine that some teams prefer for its Kubernetes-native, controller-only footprint. If your build tooling already lives in GitHub, GitHub Actions feeding Argo CD removes Jenkins entirely and is simpler for GitHub-centric shops — Jenkins earns its place here because the payer has years of shared-library investment and pipelines for non-containerized legacy systems too. If you genuinely have one small cluster and one team, a single push pipeline with a manual gate is less machinery — but it cannot answer the auditor’s drift question, which is the line that decides it for a regulated org. And if you want the change-management rigor without GitOps, Spinnaker offers managed-delivery pipelines with built-in approval stages, at the cost of a heavier platform to run.

The shape of the win

For the payer’s platform team, the payoff is not “faster deploys” — though lead time does drop once the routine path is one-line PRs. The payoff is the audit conversation. When the auditor asks who approved this production change, what exactly shipped, and prove the cluster matches it, the answer is now a single Git history (authored, reviewed, signed), a single ServiceNow change record (CAB-approved, linked to a signed artifact and an SBOM), and a single Argo CD screen showing Synced / Healthy with drift actively corrected. The 2 a.m. emergency fix is an emergency change with retroactive review, not a ghost. Everything upstream — the Jenkins shared library, the Vault-injected secrets, the Cosign signatures, the Wiz Code gate, the Okta-governed access — exists to make that one conversation end with a clean opinion instead of a qualified one. In a regulated payer, a clean SOC 2 and HITRUST report is not a compliance checkbox; it is something the sales team puts in front of every prospect, and that is what funds the platform.

Going deeper

Everything above is the shape of the system. This section is the machinery underneath it — the parts you reach for when the pattern meets a real change-management process, a multi-environment promotion, and an auditor’s control matrix.

Sync windows: freezing production on a schedule

Auto-sync is wonderful until a change-freeze — the last two weeks of a quarter, a claims-cutover weekend, an active incident where you want nothing moving. Argo CD’s sync windows, configured on the AppProject, are the declarative freeze switch. The rule that surprises people: if you configure any allow window, syncs are denied whenever no allow window is active. So a single allow window turns into “prod deploys only during this slot, frozen otherwise” — no separate deny needed. deny windows, when you use them, always win over allow.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payer-prod
  namespace: argocd
spec:
  description: Production apps for the payer platform
  sourceRepos:
    - git@github.com:payer/gitops-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: 'claims'
  syncWindows:
    - kind: allow
      schedule: '0 14 * * 2,4'      # Tue & Thu 14:00 the window opens
      duration: 4h                  # ...and stays open four hours
      timeZone: America/New_York
      applications:
        - '*'
      manualSync: true              # break-glass: a human may still sync in a Sev-1
  roles:
    - name: prod-syncer
      description: May trigger a sync but not edit project or app config
      policies:
        - 'p, proj:payer-prod:prod-syncer, applications, sync, payer-prod/*, allow'
      groups:
        - 'payer:sre-oncall'

manualSync: true is the compliance-friendly escape hatch: automated sync is blocked outside the window, but an on-call engineer can still push a manual sync during an incident — an intentional, attributable action rather than a silent bypass. Pair that with the ServiceNow emergency-change path and even the break-glass is on the record.

Project RBAC: who is allowed to sync what

The roles block above is the other half of the AppProject. Argo CD RBAC is a list of p, <subject>, <resource>, <action>, <object>, <effect> policy lines, and AppProject roles let you scope them tightly: the prod-syncer role can sync applications in payer-prod/* but cannot edit the project, change a repo URL, or delete an app. Map the role to an Okta group (payer:sre-oncall) and access is granted by group membership managed in one place, with MFA and offboarding for free. This is the separation-of-duties boundary inside Argo CD: the people who can approve config-repo PRs, the people who can trigger a sync, and the people who administer Argo CD are three different Okta groups, not one shared admin login.

Promotion: moving one artifact across dev → stage → prod

The cardinal rule of promotion is build once, promote the same artifact. You do not rebuild the image per environment — that would produce a different binary in prod than the one you tested, which defeats the entire provenance story. Jenkins builds and signs exactly one image, identified by its immutable digest, and promotion is nothing more than pointing successive environment overlays at that same digest through PRs:

• dev auto-updates on merge to the app repo (a normal, auto-approving change).
• stage is a PR that copies the dev overlay’s digest into the stage overlay, gated on integration tests.
• prod is a PR that copies the stage-verified digest into the prod overlay, gated on CAB approval and signed commits.

An ApplicationSet fans one template across all three, so a new service is one list entry, not three hand-written Applications:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: claims-adjudication
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - env: dev
            project: payer-nonprod
          - env: stage
            project: payer-nonprod
          - env: prod
            project: payer-prod
  template:
    metadata:
      name: 'claims-adjudication-{{env}}'
    spec:
      project: '{{project}}'
      source:
        repoURL: git@github.com:payer/gitops-config.git
        targetRevision: main
        path: 'apps/claims-adjudication/overlays/{{env}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'claims-{{env}}'
      syncPolicy:
        automated:
          selfHeal: true
          prune: true

For real reproducibility, prefer pinning by digest over a SHA tag in the overlay — a tag is a mutable pointer, a sha256: digest is the content itself, so “what ran in prod” is provable byte-for-byte. Kustomize supports both a newTag and a digest field in its images: transformer.

Signed commits, attestations, and the reconciliation loop

Three integrity layers stack here and it’s worth separating them, because auditors ask about each:

• Signed commits on the config repo prove who authored the change (git commit -S, or keyless gitsign). Enforce with branch protection requiring signed commits on prod/.
• Image signatures (Cosign) prove this artifact came from this pipeline and are enforced at admission by Kyverno.
• Attestations are signed statements about the artifact — the SBOM, the build provenance, the scan result — recorded in Rekor. SLSA grades how tamper-resistant that provenance is; a hermetic, signed, non-falsifiable build reaches roughly SLSA Build L3, which is the level FedRAMP-track programs increasingly ask for.

Under the hood, Argo CD’s reconciliation is a three-way diff: it compares the desired state (rendered from Git) against the live state (what the API server reports) using the last-applied state as the tiebreaker, then computes the minimal patch. argocd app diff <app> shows you exactly that delta before a sync; sync waves (argoproj.io/sync-wave annotations) order resources so a database migration Job runs before the Deployment that needs it; and resource hooks (PreSync, PostSync) hang jobs off the sync lifecycle. Self-heal is just this loop firing on detected drift rather than on a Git change — which is why it fights any other controller that also writes the same field.

Finally, the framing an auditor actually wants — each mechanism mapped to the control family it satisfies. Exact control IDs depend on your org’s control matrix, but the mapping is indicative:

Practice challenges

Work these top to bottom; they escalate from “read the system” to “harden the system.” Each has a worked solution — try it before you open it.

1. (Beginner) Reconstruct the audit trail. For a single production deploy of claims-adjudication, list — in order — every place the answer to “who approved this and what shipped” is recorded, and name the one Git command that reconstructs the signed change history for the prod overlay.

<details> <summary>Solution</summary>

The trail, end to end: config-repo PR (non-author approver + signed commit) → ServiceNow change record (CAB approver, linked to the SBOM, scan, and artifact) → Argo CD sync history (argocd app history claims-adjudication-prod) → Git itself:

git log --show-signature -- apps/claims-adjudication/overlays/prod

Why: the record is distributed across Git, ServiceNow, and Argo CD, but every entry cross-references the same immutable commit — so any one of them reconstructs the whole story. </details>

2. (Beginner) Make the GitOps deploy edit. Given the prod Kustomize overlay, write the exact change Jenkins commits to roll out image sha-9f3c2ab, and explain in one sentence why Jenkins opens a pull request instead of pushing straight to main.

<details> <summary>Solution</summary>

# apps/claims-adjudication/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: 111122223333.dkr.ecr.us-east-1.amazonaws.com/claims-adjudication
    newTag: sha-9f3c2ab      # the ONLY line that changes on a routine deploy

Why: the prod/ path requires a reviewed, non-author-approved merge, so Jenkins can only propose the one-line change — the approval is what turns a proposal into production. </details>

3. (Intermediate) Harden the Argo CD Application. Extend a prod Application so a transient sync failure retries with exponential backoff instead of flapping, and so you keep ten rollback points in Argo CD’s history. Which two fields do you add?

<details> <summary>Solution</summary>

spec:
  revisionHistoryLimit: 10        # keep 10 rollback points
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    retry:
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m

Why: retry.backoff absorbs transient API/health blips without hammering the cluster, and revisionHistoryLimit bounds how far back a one-click rollback can reach. </details>

4. (Intermediate) Freeze prod, keep the emergency door. Write the AppProject sync window that permits automated prod syncs only Tuesday and Thursday 14:00–18:00 America/New_York, yet still lets an on-call engineer sync manually during a Sev-1.

<details> <summary>Solution</summary>

syncWindows:
  - kind: allow
    schedule: '0 14 * * 2,4'     # opens Tue & Thu 14:00...
    duration: 4h                 # ...closes at 18:00
    timeZone: America/New_York
    applications:
      - '*'
    manualSync: true             # on-call can break-glass sync anytime

Why: with an allow window present, Argo CD denies syncs whenever it isn’t active — and manualSync: true preserves an attributable emergency path instead of a silent bypass. </details>

5. (Advanced) Enforce provenance at admission. Write a Kyverno ClusterPolicy in enforce mode that rejects any ECR image not keylessly signed by the payer’s GitHub OIDC identity. Then state what a developer sees if a manifest references an unsigned image.

<details> <summary>Solution</summary>

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-payer-signatures
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: verify-cosign-keyless
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "111122223333.dkr.ecr.us-east-1.amazonaws.com/*"
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/payer/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

The developer sees pods stuck Pending with an admission-webhook denial (“image verification failed”). Why: signing in CI is inert on its own — the deploy-time gate is what makes provenance enforceable rather than aspirational. </details>

6. (Advanced) Kill a drift storm without disabling self-heal. An HPA scales claims-adjudication, but Argo CD keeps reverting replicas back to the value in Git, so the two controllers fight forever. Fix it so Git still owns everything except replica count.

<details> <summary>Solution</summary>

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: claims-adjudication
      jqPathExpressions:
        - .spec.replicas

(and remove replicas from the base manifest so Git isn’t asserting a value at all). Why: self-heal should ignore fields another controller legitimately owns; scoping ignoreDifferences to .spec.replicas ends the fight without giving up drift correction on everything else. </details>

Common beginner mistakes

“Just put the manifests in the same repo as the app code — one repo is simpler.” It is simpler, and it quietly destroys separation of duties: anyone who can merge code can now merge a production change, and every feature PR churns the deploy history. The right model is two repos with different governance — a permissive, dev-owned application repo and a config repo governed like production, where the prod/ path is the approval gate.

“Argo CD builds and deploys my app.” Argo CD never builds anything. It only reconciles declared state — it makes the cluster match Git. Jenkins builds, tests, scans, and signs; Argo CD takes the already-built, already-approved manifest and applies it. Confusing the two leads people to hunt for build logs in Argo CD and to hand it registry credentials it should never possess.

“The approval is the Jenkins ‘Proceed’ button.” A manual approval step inside the pipeline feels like a control, but it lives in a console log no CAB governs, and the same engineer who wrote the change can click it. Real change approval is a non-author PR review on the config repo plus a ServiceNow change record — two independent systems, one of them a human who is not the author.

“Tag it latest and let Argo CD pull the newest image.” Mutable tags break the two properties this whole design rests on. The manifest never changes, so Git can no longer tell you what’s actually running, and two clusters both pulling latest can silently diverge. Use an immutable SHA tag, ideally pinned by digest — the tag change is the deploy event and the record of it.

“Turn on self-heal everywhere on day one.” Self-heal feels strictly safer, but it will revert fields other controllers legitimately own — HPA replica counts, cert-manager annotations, injected sidecars — and start a fight-loop. Enable it deliberately, and add ignoreDifferences for any field a different controller writes.

“During the incident I’ll just kubectl apply the fix.” In this architecture that is the worst move: Argo CD detects the out-of-band edit as drift and reverts it, and you’ve created exactly the untracked production change the whole system exists to prevent. Ship the fix through the break-glass emergency-change path — a fast-tracked signed PR plus retroactive CAB review, with a manual sync if a window is closed.

Glossary

• GitOps — An operating model where the desired state of your systems lives in Git and an automated agent continuously makes reality match it. Git is both the source of truth and the audit log.
• Desired state — The declarative description (Kubernetes manifests) of what should be running. You describe the destination, not the steps to get there.
• Reconciliation — The continuous loop of comparing live state to desired state and acting to close any gap. Argo CD’s core job.
• Drift — Any difference between what’s running in the cluster and what Git says should be running (for example, someone hand-edited a Deployment).
• Synced / OutOfSync — Argo CD’s verdict on whether the cluster matches Git. Synced means it matches; OutOfSync means there’s drift or an undeployed change.
• Healthy / Degraded — A separate Argo CD verdict on whether the deployed resources actually work (pods ready, rollout complete), independent of whether they match Git.
• Self-heal — An Argo CD option that automatically reverts drift back to the Git state instead of merely flagging it.
• Prune — An Argo CD option that deletes cluster resources which have been removed from Git, keeping Git authoritative.
• Pull-based deployment — The cluster’s own agent pulls approved state from Git. Contrast push-based, where an external CI system pushes changes in using standing credentials.
• Argo CD — The GitOps engine: an in-cluster controller that reconciles Kubernetes to Git.
• Application (Argo CD) — Argo CD’s unit of deployment: a mapping from a Git path to a cluster/namespace, with a sync policy.
• ApplicationSet — A generator that stamps out many Applications from one template (for example, one per environment), making multi-env and multi-cluster fan-out declarative.
• App-of-apps — A pattern where one root Application manages a tree of child Applications, so onboarding a service is a Git commit.
• AppProject — An Argo CD grouping that scopes which repos, clusters, and namespaces its Applications may touch, plus RBAC and sync windows. The separation-of-duties boundary inside Argo CD.
• Sync window — A schedule that allows or blocks syncs at specific times (for example, freeze prod except a Tue/Thu change window).
• Jenkins — The CI server here: it builds, tests, scans, signs, and opens the config-repo PR, running each build on an ephemeral Kubernetes pod.
• Jenkins shared library — Centrally-governed pipeline code every service references in one line, so all pipelines behave identically and are audited once.
• Jenkinsfile — The pipeline definition committed to an app repo; here it’s a thin wrapper that calls the shared library.
• Ephemeral agent — A build worker (a Kubernetes pod) created for one build and destroyed after, so no secrets or state accumulate.
• Application repository — The dev-owned repo of source code and the Jenkinsfile. Permissive; iterated fast.
• Config repository (GitOps repo) — The governed repo of Kubernetes manifests that describe production. Its prod/ path is the change-approval gate.
• Kustomize — A tool that layers environment-specific patches (overlays) over a shared base manifest, so the per-deploy diff is one image reference.
• Helm — A Kubernetes package manager using templated charts and a values file; an alternative or companion to Kustomize for rendering manifests.
• Overlay — A Kustomize layer (such as overlays/prod) that patches the base for one environment.
• Immutable tag / digest — A container image reference that never moves. A SHA tag — or better, a sha256: digest — guarantees you know exactly what ran.
• Cosign / Sigstore — Tooling that cryptographically signs container images (keylessly, via a short-lived OIDC identity) to prove provenance.
• Rekor — Sigstore’s public transparency log that records signatures so they can be verified and cannot be quietly deleted.
• SBOM — Software Bill of Materials: a machine-readable inventory of everything inside an artifact, attached as an attestation.
• Attestation — A signed statement about an artifact (its build provenance, SBOM, or scan result), verifiable independently of the artifact itself.
• SLSA — A framework of graded levels describing how tamper-resistant your build and provenance are (higher levels = stronger guarantees).
• Kyverno — A Kubernetes policy engine; its verifyImages rule rejects unsigned or unverified images at admission.
• Admission controller — A cluster gate that inspects (and can reject) resources before they’re created — where signature verification is enforced.
• HashiCorp Vault — A secrets broker that issues short-lived, tightly-scoped credentials to the build pod at runtime, so nothing long-lived sits in Jenkins.
• Kubernetes auth method — Vault’s way of trusting a pod’s Kubernetes service-account token as its identity, exchanged for a short Vault token.
• IRSA — IAM Roles for Service Accounts: AWS’s mechanism for giving a pod scoped AWS permissions without static keys.
• ECR / EKS — Amazon’s container registry (ECR, where images live) and managed Kubernetes (EKS, where workloads run).
• ServiceNow / CAB — The enterprise change-management system (ServiceNow) and its Change Advisory Board (CAB) that reviews and approves production changes.
• Separation of duties (SoD) — The control that no single person can both author and approve/deploy a production change. Enforced here by non-author PR review, independent CAB approval, and Argo-only cluster access.
• RBAC — Role-Based Access Control: mapping identities (via Okta groups) to permitted actions across Jenkins, Argo CD, and ServiceNow.
• Signed commit — A Git commit cryptographically signed by its author (GPG, SSH, or gitsign), so the config-repo history proves who changed what.
• Rollback (git revert) — Undo a deploy by committing the reverse of the change; Argo CD reconciles the cluster back to the prior state, with a full audit trail.
• DORA metrics — Four delivery-health measures (deployment frequency, lead time for changes, change-failure rate, time-to-restore), all directly measurable from Git and Argo CD events here.
• Argo Rollouts — A companion controller adding canary and blue-green strategies so a bad version is caught at small blast radius.
• Break-glass / emergency change — A pre-approved fast path for Sev-1 fixes: a fast-tracked signed PR plus retroactive CAB review, keeping emergencies auditable instead of untracked.