GitOps with Argo CD and Flux: Deliver from Git

Quick take: GitOps makes Git the single source of truth for both application and infrastructure state. A controller running inside the cluster continuously watches a repo and reconciles the live cluster toward the committed manifests. If someone kubectl edits a resource by hand, the controller detects the drift and (when self-heal is on) reverts it. Deployment becomes a git push; rollback becomes a git revert; the audit log is your commit history.

A platform team of six ran eleven Kubernetes clusters across three regions and two cloud accounts. Manifests lived in a wiki, in kubectl apply muscle memory, and in three people’s heads. Two clusters had quietly diverged — different ingress-nginx versions, a replicas: 1 someone set during an incident and never reverted, a ConfigMap patched live and lost on the next deploy. Nobody could answer “what is actually running in prod-eu?” without reading the cluster. A bad Friday deploy took ninety minutes to roll back because the rollback was re-running the pipeline with last week’s image tag, and the build cache had expired.

They adopted GitOps. Every manifest moved into Git. Argo CD (and later Flux on the edge clusters) ran inside each cluster, watched a repository, and made the live state match the committed state. The cluster became a follower of Git. Drift — including that stray replicas: 1 — was reverted within minutes, automatically. Rollback became git revert <sha> && git push, reconciled in under a minute. “What’s running in prod-eu?” became “read clusters/prod-eu/ on main.” This is the model this article teaches end to end, at the depth a senior platform engineer needs to run it in production — not the marketing version.

You will learn the four GitOps principles and why each one matters operationally, how the reconciliation loop works (observe → diff → act, every few minutes and on every webhook), how drift detection and self-heal differ and when to enable each, how Argo CD and Flux differ and where each wins, the app-of-apps and ApplicationSet patterns for managing dozens of apps and clusters, sync waves and hooks for ordering and migrations, multi-cluster topologies, the three real answers to secrets in Git (SOPS, Sealed Secrets, External Secrets Operator), and how GitOps composes with progressive delivery (Argo Rollouts / Flagger). Everything is grounded in real CRDs, real CLI, and real failure modes — with the tables you keep open while you operate.

What problem this solves

Traditional CI/CD is push-based imperative delivery: a pipeline runner outside the cluster holds cluster credentials, runs kubectl apply or helm upgrade, and hopes the result matches intent. Three structural problems fall out of that model, and GitOps is the direct answer to all three.

First, there is no continuously-enforced source of truth. The pipeline applies a manifest once, at deploy time. Five minutes later someone runs kubectl scale during an incident, an operator mutates a resource, or a helm rollback half-completes — and the live cluster no longer matches anything in version control. The divergence is invisible until it bites. GitOps closes the loop: a controller re-asserts the committed state on a schedule, so the cluster is always converging back to Git, not just at deploy time.

Second, credentials point the wrong way. Push CD requires your CI runner (often a SaaS outside your network) to hold cluster admin credentials and reach the API server — frequently meaning a publicly-exposed control plane or a long-lived kubeconfig in a secrets store. GitOps inverts this to pull-based: the agent runs inside the cluster and reaches out to Git and the registry. No external system needs cluster credentials; the API server can be fully private. This is the single biggest security argument for GitOps.

Third, rollback and audit are afterthoughts. “Roll back” in push CD means re-running an old pipeline, which may no longer be reproducible (expired caches, mutable :latest tags, drifted dependencies). “Who changed this and why?” means correlating pipeline logs, tickets, and Slack. In GitOps, the desired state is the Git history: rollback is git revert, the audit trail is git log and PR reviews, and every production change went through a reviewable, signed, attributable commit.

Who hits these problems: any team running Kubernetes at more than toy scale — multiple environments, multiple clusters, more than a couple of engineers with kubectl access, or a compliance requirement that every production change be reviewed and attributable. It bites hardest on multi-cluster fleets (drift multiplies per cluster), regulated workloads (you must prove what ran and who approved it), and platform teams offering self-service (you need a guardrail that isn’t “trust everyone with cluster-admin”).

To frame the whole field before the deep dive, here is the contrast that drives every later decision:

Learning objectives

By the end of this article you can:

• State the four GitOps principles (declarative, versioned & immutable, pulled automatically, continuously reconciled) and explain the production consequence of each.
• Trace the reconciliation loop for both Argo CD and Flux — observe, diff, act — and reason about reconcile interval, webhooks, and what triggers a sync.
• Distinguish drift detection from self-heal and auto-prune, and decide which to enable per environment without causing fights with HPAs and admission webhooks.
• Choose between Argo CD and Flux (or run both) based on UI needs, multi-tenancy model, Helm strategy, and team operating style — backed by a feature matrix.
• Implement app-of-apps and ApplicationSets to manage dozens of applications and clusters from a single declarative root.
• Order deployments with sync waves, and run migrations/backups/tests with resource hooks (PreSync/Sync/PostSync/SyncFail), or Flux dependsOn + Kustomization ordering.
• Architect multi-cluster GitOps (hub-and-spoke vs standalone agents) and generate per-cluster app definitions safely.
• Keep secrets out of plaintext Git correctly using SOPS, Sealed Secrets, or the External Secrets Operator, and pick the right one per constraint.
• Compose GitOps with progressive delivery (Argo Rollouts / Flagger) so canaries and automated rollback live in Git too.
• Diagnose the real failure modes: OutOfSync that won’t heal, stuck Progressing, drift fights, hook deadlocks, and ComparisonError.

Prerequisites & where this fits

You should be comfortable with Kubernetes fundamentals: Deployments, Services, ConfigMaps, Namespaces, RBAC (ServiceAccounts, Roles, ClusterRoles), and reading/writing YAML manifests. You should know how kubectl apply works (server-side vs client-side apply, the last-applied-configuration annotation) and what a CRD and a controller/operator are — GitOps tools are controllers reconciling CRDs. Hands-on experience with Kustomize (bases and overlays) and Helm (charts, values, releases) is assumed; GitOps tools render manifests through exactly those two engines. You should be able to run kubectl, helm, and git from a shell.

This article sits in the delivery & platform-engineering track. It is the deployment-mechanism layer underneath your release strategy: it assumes you understand CI/CD Pipelines Explained: From Code Commit to Production (GitOps replaces the CD half, not the CI half — you still build and test images in a pipeline), and it pairs tightly with Deployment Strategies: Blue-Green, Canary and Rolling Updates and Progressive Delivery and Feature Flags: Release Without Fear, because GitOps is how you store and enforce those strategies declaratively. It leans on Git Branching Strategies: Trunk-Based, GitFlow and Feature Branches (your repo layout is a branching decision) and on Infrastructure as Code: Terraform, Pulumi, CDK and Cloud-Native Options (GitOps is “IaC for the cluster’s runtime state”). Secrets connect to CI/CD Secrets and Credential Management: Secure Your Pipelines, and the operational signals connect to DevOps Observability: Logs, Metrics, Traces and SLOs.

A quick map of who owns what during a GitOps incident, so you call the right layer fast:

Core concepts

Five mental models make every later section obvious. Read them once; the rest of the article is consequences.

Git is the desired state; the cluster is the actual state; reconciliation is the function that minimises their difference. This is the entire idea expressed as control theory. You declare what you want in Git (the setpoint). A controller continuously measures what is (the cluster’s live objects) and applies the delta to drive actual toward desired. Everything else — drift, self-heal, sync waves, multi-cluster — is a refinement of this one loop. Crucially, the controller never “does a deploy” in the imperative sense; it converges. A deploy is just “the setpoint moved, so the next reconcile moves the cluster.”

The controller runs inside the cluster and pulls. Argo CD’s application-controller and Flux’s source-controller/kustomize-controller/helm-controller are Pods in the cluster. They reach out to Git (clone/fetch) and the registry, render manifests, and apply them to the local (or a registered remote) API server. No external CI system needs cluster credentials. This pull model is why the API server can be private and why a compromised CI runner can’t directly touch prod — it can only open a pull request.

Drift is any difference between live and desired; how you respond to it is a policy, not a fact. Detection is always on (the diff is computed every reconcile). The response is configurable: report only (show OutOfSync, alert, let a human act), self-heal (revert the live object back to Git automatically), and prune (delete live objects that no longer exist in Git). These are independent knobs. Aggressive self-heal on the wrong resource starts a fight with another controller (an HPA managing replicas, a mutating webhook injecting a sidecar) — so you must know which fields you own and which the platform owns.

Rendering is Kustomize or Helm; GitOps tools don’t invent a new templating language. Argo CD and Flux both call kustomize build or helm template (Flux’s Helm controller does a real Helm release; Argo CD by default renders templates and applies them, with a flag to use real Helm). So “GitOps YAML” is just your normal Kustomize overlays and Helm charts, plus a thin Application (Argo CD) or Kustomization/HelmRelease (Flux) CRD that says which repo, which path, which revision, which cluster/namespace, and which sync policy. Master Kustomize and Helm first; the GitOps layer is a wrapper.

Sync is not atomic, and ordering is explicit. A sync applies many objects. By default they go in a sensible order (namespaces and CRDs first), but cross-resource dependencies (run the DB migration before the new app version; create the namespace before the resources in it) are your responsibility, expressed with sync waves (Argo CD annotations) or dependsOn (Flux). A hook is a special object that runs at a phase boundary (before sync, after sync, on failure) — typically a Job for migrations, smoke tests, or notifications. Get ordering wrong and you get a half-applied state that looks like a flaky deploy but is a missing wave annotation.

The vocabulary in one table

Pin down every moving part before the deep sections. The glossary repeats these for lookup; this is the mental model side by side:

The four GitOps principles

The CNCF OpenGitOps working group distilled GitOps into four principles. They’re not slogans — each one has a concrete operational meaning and a failure mode if you violate it.

The deep implication of principle 4 is the one people miss: GitOps is not “deploy on merge.” “Deploy on merge” is just push CD triggered by Git. True GitOps continuously reconciles whether or not anything merged — so a manual kubectl edit made at 3 a.m. is reverted at the next reconcile even though no commit happened. If your “GitOps” only acts on a webhook and never re-checks the cluster, you have automated push CD, not GitOps, and you don’t get drift correction.

A second subtlety on principle 2: immutable means you reference a specific revision (a commit SHA or a semver tag), not a moving branch for the artifact you actually run in prod. Pointing prod at HEAD of main is convenient but means “what’s deployed” changes the instant someone merges — you lose the ability to say “prod is at exactly this SHA” and to promote a specific tested revision. Mature setups pin environments to tags or SHAs and promote by changing the pin in a PR.

How the reconciliation loop actually works

The reconciliation loop is the heart of GitOps. Both tools implement the same conceptual cycle — observe, diff, act — differing in component layout and defaults.

The loop, step by step, for a single application:

What triggers a reconcile

A reconcile happens on three triggers, and understanding them prevents the “why didn’t my change deploy?” confusion:

The polling interval is the floor on how fast drift is corrected when self-heal is on, and the floor on deploy latency if you don’t wire webhooks. Setting Flux interval: 1m on every Kustomization across a large fleet hammers the API server and Git; setting it to 10m means drift can persist for ten minutes. The pattern is short interval + webhooks: the webhook handles “deploy fast on merge,” the interval handles “correct drift and catch missed webhooks.” Argo CD additionally watches live objects via Kubernetes informers, so it reacts to drift faster than its poll interval suggests — a kubectl edit is often reverted within seconds, not at the next 3-minute poll.

Server-side apply and three-way diff

The act phase is not a naive overwrite. Both tools use server-side apply (SSA) with field management, computing a three-way diff between (a) the desired manifest, (b) the live object, and © the last-applied state. This is why GitOps doesn’t clobber fields it doesn’t manage — an HPA-managed replicas or a webhook-injected sidecar container is owned by a different field manager and is left alone, unless you also declare that field, which is exactly how drift fights start.

A field-ownership example you’ll hit within a week: you set replicas: 3 in Git, but an HPA also manages replicas. With self-heal on, the controller reverts to 3 every reconcile, the HPA scales back up, and the two ping-pong. The fix is to not declare replicas (let the HPA own it) and add ignoreDifferences for /spec/replicas. Knowing which fields you own is the difference between a stable GitOps setup and a flapping one.

Drift detection, self-heal, and prune

These three are independent policies, and conflating them causes most GitOps accidents. Detection is free and always on; the response is what you configure.

Detection: what counts as drift

Drift is computed by the three-way diff, so it only flags fields the controller manages. The drift you should heal is a manual kubectl edit/scale or a half-applied previous sync — re-assert Git (and capture the lesson in a commit). The drift you should not heal is another controller doing its job: an HPA owning replicas, a mutating webhook injecting a sidecar, cert-manager or a service mesh patching annotations — for those, ignoreDifferences or exclude the resource. And an out-of-band hotfix that should be permanent isn’t really drift: promote it to Git first, and it becomes desired state. The operational rule: self-heal the fields you own; ignore the fields other controllers own. Most “GitOps keeps reverting my change” complaints are someone fighting an HPA or a mutating admission webhook over a field they shouldn’t have declared.

Self-heal and prune in real YAML (Argo CD)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-api
  namespace: argocd
spec:
  project: shop
  source:
    repoURL: https://github.com/acme/shop-deploy.git
    targetRevision: v2.4.1          # pin to a tag, not HEAD, for prod
    path: apps/shop-api/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true                    # delete objects removed from Git
      selfHeal: true                 # revert manual drift back to Git
    syncOptions:
      - CreateNamespace=true         # create the target namespace if missing
      - PruneLast=true               # prune only after healthy sync (safer)
      - ApplyOutOfSyncOnly=true      # only apply resources that actually differ
    retry:
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 5m
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas            # HPA owns this — don't fight it

The equivalent in Flux (Kustomization)

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: shop-api
  namespace: flux-system
spec:
  interval: 5m                       # reconcile cadence (drift floor)
  retryInterval: 1m
  sourceRef:
    kind: GitRepository
    name: shop-deploy
  path: ./apps/shop-api/overlays/prod
  prune: true                        # delete orphaned objects
  wait: true                         # block until resources are healthy
  timeout: 5m
  force: false                       # true => replace immutable resources
  patches:
    - patch: |
        - op: remove
          path: /spec/replicas       # let HPA own replicas
      target:
        kind: Deployment
        name: shop-api

A hard-won safety note on prune: enabling prune: true plus a bad render (a Kustomize error that produces an empty manifest set, or a path typo that points at an empty directory) can cause the controller to conclude “all these objects were removed from Git” and delete your entire app. Guardrails: keep prune off until you trust the pipeline, use PruneLast, set prune: false on the bootstrap/root app, and (Argo CD) use Validate=true and resource-exclusion. Treat prune like rm -rf with a loaded source.

Argo CD vs Flux: how they differ and how to choose

Both are CNCF-graduated GitOps controllers. They solve the same problem with different philosophies: Argo CD is a single, batteries-included application delivery platform with a strong UI and an application-centric model; Flux is a set of composable single-responsibility controllers (the GitOps Toolkit) with a CLI-and-CRD-first model and tight OCI/Helm integration. Neither is “better” — they fit different teams.

Where each one wins

Many mature platforms run both: Argo CD for the application teams who want the UI and self-service, Flux for the platform layer (cluster add-ons, image automation) where everything should be CRD-driven and headless. They coexist fine as long as they don’t both manage the same objects.

Component reference

The controllers you’ll actually watch, restart, and read logs from:

App-of-apps and ApplicationSets: managing the fleet

One Application per microservice per environment per cluster doesn’t scale by hand — fifty services across four clusters is two hundred objects to create and keep consistent. Two patterns solve this: app-of-apps (a root app that points at a directory of child app definitions) and ApplicationSets (a controller that generates Applications from a template + a generator).

App-of-apps

The app-of-apps pattern bootstraps and manages many apps from one root. You create a single Argo CD Application whose source is a directory of more Application manifests. Argo CD syncs the root, which creates the children, which sync their real workloads. It’s GitOps managing GitOps.

# root-app.yaml — the single bootstrap app you create by hand (or via Terraform)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/gitops.git
    targetRevision: main
    path: bootstrap/apps          # a folder full of Application manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }

gitops/
├── bootstrap/
│   └── apps/                     # the "app-of-apps" — one Application per child
│       ├── shop-api.yaml         # → apps/shop-api/overlays/prod
│       ├── shop-web.yaml
│       ├── ingress-nginx.yaml    # platform add-on
│       └── cert-manager.yaml
└── apps/
    ├── shop-api/{base,overlays/{dev,prod}}
    └── shop-web/{base,overlays/{dev,prod}}

The trade-off: app-of-apps is simple and explicit (every child is a file you can read), but it’s static — adding a cluster means hand-editing child files, and there’s no templating across them. That’s where ApplicationSets come in.

ApplicationSets and generators

The ApplicationSet controller renders many Applications from one template, driven by a generator that produces a list of parameter sets. This is the scalable answer to “the same app on every cluster” or “an app per directory in the repo.”

A real Cluster generator that deploys ingress-nginx to every cluster labelled env=prod:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ingress-nginx
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - clusters:
        selector:
          matchLabels:
            env: prod              # only clusters labelled env=prod
  template:
    metadata:
      name: 'ingress-nginx-{{.name}}'   # one app per cluster
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/gitops.git
        targetRevision: main
        path: addons/ingress-nginx
      destination:
        server: '{{.server}}'      # the cluster's API server URL
        namespace: ingress-nginx
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

The power and the danger are the same property: one change to the generator or template fans out to every generated app. A typo in the template can create — or, with prune on, delete — dozens of apps across clusters at once. Treat ApplicationSet changes with the review rigor of an infrastructure change, and consider applicationsSync: create-only or preserveResourcesOnDeletion while you build confidence.

Sync waves and hooks: ordering and lifecycle

A sync applies many objects, but real apps have ordering constraints: the namespace must exist before its resources; a CRD must be installed before a custom resource that uses it; a database migration must run before the new app version starts; a smoke test should run after. GitOps gives you two ordering mechanisms — sync waves (intra-sync ordering of resources) and resource hooks (run a Job at a phase boundary).

Sync waves (Argo CD)

A sync wave is an integer annotation on a resource. Argo CD applies resources in ascending wave order, waiting for each wave’s resources to be healthy before starting the next. Default wave is 0; CRDs/namespaces get applied early automatically, but anything else you order yourself.

# Wave -1: install the CRD first
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.acme.io
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
---
# Wave 0 (default): the app's config and deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Wave 1: only after the app is healthy, apply the ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-api
  annotations:
    argocd.argoproj.io/sync-wave: "1"

Resource hooks (Argo CD)

A hook is a resource (almost always a Job) annotated to run at a sync phase. The phases are PreSync (before applying the manifests — run DB migrations here), Sync (alongside the main apply), PostSync (after everything is applied and healthy — smoke tests, cache warm), and SyncFail (only if the sync failed — alert, rollback side-effects). A hook delete policy controls cleanup.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync                 # run before the app updates
    argocd.argoproj.io/hook-delete-policy: HookSucceeded  # clean up on success
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/acme/shop-api:v2.4.1
          command: ["./migrate", "up"]

A migration deadlock to avoid: a PreSync migration Job that itself needs the new app’s config (a ConfigMap applied in the main Sync) will hang — the config isn’t applied yet. Keep PreSync jobs self-contained (own image, own env, own DB connection) and don’t make them depend on objects that only land during Sync.

Flux ordering: dependsOn and health gates

Flux expresses ordering differently — between Kustomizations via dependsOn and health checks (wait: true + healthChecks), and within Helm via real Helm hooks (helm-controller honors pre-install/post-upgrade hooks natively, an advantage over Argo CD’s helm template default).

# infra must be healthy before apps reconcile
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: apps, namespace: flux-system }
spec:
  interval: 5m
  dependsOn:
    - name: infra                  # wait for the 'infra' Kustomization first
  sourceRef: { kind: GitRepository, name: gitops }
  path: ./apps
  wait: true                       # block on health
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: shop-api
      namespace: shop

Multi-cluster GitOps

At fleet scale the question becomes topology: one control plane managing many clusters (hub-and-spoke) versus each cluster running its own agent (standalone). The choice drives blast radius, credential flow, and how you generate per-cluster config.

The credential-direction trade-off is the crux. Hub-and-spoke means the hub stores a kubeconfig/ServiceAccount token for every spoke — convenient and centrally visible, but a compromised hub can touch every cluster. Standalone agents store no cross-cluster credentials; each cluster pulls only its own config, so the blast radius of any one compromise is one cluster. Regulated and high-isolation environments usually choose standalone (or hub-with-agent) for exactly this reason; teams optimizing for a single operator view choose hub-and-spoke.

Registering a remote cluster with the Argo CD hub:

# Add a spoke cluster to the Argo CD hub (creates a ServiceAccount + stores creds)
argocd cluster add prod-eu-context --name prod-eu --label env=prod --label region=eu

# Now an ApplicationSet cluster generator (env=prod) targets it automatically
argocd cluster list

Per-cluster differences (region, replica counts, hostnames, feature flags) are the real multi-cluster challenge. Don’t fork the whole manifest per cluster; layer them:

A clean fleet repo layout that scales:

gitops/
├── clusters/
│   ├── prod-eu/                  # what this cluster runs (Flux Kustomizations / Argo apps)
│   ├── prod-us/
│   └── staging/
├── infrastructure/              # add-ons: ingress, cert-manager, monitoring
│   ├── base/
│   └── overlays/{prod,staging}/
└── apps/                        # the actual workloads
    ├── base/
    └── overlays/{prod-eu,prod-us,staging}/

Secrets in GitOps: the three real answers

GitOps has one notorious gap: you must not commit plaintext secrets to Git, yet Git is your source of truth. There are exactly three production-grade patterns, and choosing among them is a real decision with real trade-offs. (A base64-encoded Secret is not encryption — base64 is encoding; anyone with repo read access trivially decodes it. Never do this.)

SOPS

SOPS (Secrets OPerationS) encrypts the values of a YAML/JSON file with a key from age, PGP, or a cloud KMS, leaving the keys readable so diffs are meaningful. You commit the encrypted file; the controller decrypts it in-cluster before applying. Flux’s kustomize-controller has native SOPS decryption; Argo CD does it via a config-management plugin (e.g. ksops).

# secret.enc.yaml — committed to Git; only values are ciphertext
apiVersion: v1
kind: Secret
metadata:
  name: shop-db
  namespace: shop
stringData:
  DB_PASSWORD: ENC[AES256_GCM,data:9f3a...,iv:...,tag:...,type:str]
sops:
  age:
    - recipient: age1q9...        # the cluster's age public key
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        ...
        -----END AGE ENCRYPTED FILE-----

# Tell Flux to decrypt with the cluster's age key (mounted as a Secret)
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: apps, namespace: flux-system }
spec:
  interval: 5m
  sourceRef: { kind: GitRepository, name: gitops }
  path: ./apps
  decryption:
    provider: sops
    secretRef:
      name: sops-age            # holds the age private key

Sealed Secrets

Sealed Secrets (Bitnami) gives each cluster a controller with a private key. You encrypt a normal Secret into a SealedSecret CRD using the cluster’s public key (kubeseal); only that cluster’s controller can decrypt it. The SealedSecret is safe in Git; the controller unseals it into a real Secret at runtime.

# Encrypt a Secret to a SealedSecret using the cluster's public cert
kubectl create secret generic shop-db --namespace shop \
  --from-literal=DB_PASSWORD='s3cr3t' --dry-run=client -o yaml \
  | kubeseal --controller-namespace kube-system --format yaml > sealed-shop-db.yaml
# Commit sealed-shop-db.yaml — it's encrypted to THIS cluster only

External Secrets Operator

ESO keeps the secret outside Git entirely — in Azure Key Vault, AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault — and commits only a reference. The operator fetches the value and projects it into a Kubernetes Secret, refreshing on an interval. Git never contains the secret in any form, encrypted or not.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: shop-db
  namespace: shop
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-kv               # points at a SecretStore (Key Vault + workload identity)
    kind: SecretStore
  target:
    name: shop-db                # the K8s Secret ESO will create
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: shop-db-password    # the secret name in Key Vault

Choosing among them

The deeper point for all three: GitOps stores the reference or ciphertext, never the plaintext, and the cluster (or the operator) holds the decryption capability. This preserves the GitOps property (everything is in Git) without the catastrophe of plaintext credentials in version history — which, once pushed, are effectively leaked forever even after deletion.

Progressive delivery on top of GitOps

GitOps gets the desired manifests onto the cluster; it does not, by itself, do canary or blue-green rollouts with metric-based automated rollback. That’s progressive delivery, and it composes on top of GitOps via two sibling projects: Argo Rollouts (pairs with Argo CD) and Flagger (pairs with Flux). The key insight: the Rollout/Canary definition itself lives in Git — so your release strategy is also versioned, reviewed, and reconciled.

Argo CD manages a Rollout (a drop-in Deployment replacement) the same way it manages any resource — but the Rollout controller then drives the canary steps and consults an AnalysisTemplate (Prometheus/Datadog queries) to decide promote-or-abort:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop-api
  namespace: shop
spec:
  replicas: 6
  strategy:
    canary:
      steps:
        - setWeight: 10          # send 10% of traffic to the new version
        - pause: { duration: 5m }
        - analysis:              # query metrics; abort if they breach
            templates: [{ templateName: success-rate }]
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100
  selector: { matchLabels: { app: shop-api } }
  template:
    metadata: { labels: { app: shop-api } }
    spec:
      containers:
        - name: api
          image: ghcr.io/acme/shop-api:v2.4.1

For Flux, Flagger watches a normal Deployment and a Canary CRD; on each image change it spins up a canary, ramps traffic via your service mesh/ingress, runs metric checks, and promotes or rolls back — all reconciled by Flux.

The full pattern: CI builds and pushes the image and writes the new tag to Git; GitOps reconciles the new Rollout/Deployment; progressive delivery ramps traffic with metric gates and auto-rolls-back on breach — and that rollback is also just the controller reverting to the previous healthy state, which you can capture as a git revert. For the strategy mechanics, see Progressive Delivery and Feature Flags: Release Without Fear and the GitOps-specific deep dive Progressive Delivery: Canary, Blue-Green and Automated Rollback with GitOps.

Architecture at a glance

The first diagram shows the GitOps machine end to end. Read it left to right. A developer opens a pull request against the deployment repo (changing an image tag, a replica count, a config value); reviewers approve and merge to main. That commit is now the desired state in Git. The GitOps controller running inside the cluster — Argo CD’s application-controller (via repo-server) or Flux’s source- and kustomize-controller — pulls the repo, renders it through Kustomize or Helm, and diffs the result against the live cluster state. Where they differ, the controller server-side-applies the delta to the Kubernetes API server, which rolls out the change to workloads (Deployments, Services, Ingress). Notice the arrow directions: every credential-bearing connection points outward from the cluster (to Git and the registry) or inward to the local API server — nothing external holds cluster admin. The CI pipeline is off to the side: it builds the image, pushes it to the registry, and writes the new tag back to Git — it never touches the cluster directly.

The second diagram zooms into the reconciliation loop and the drift case specifically. The controller runs a continuous cycle: observe the desired state (latest revision from Git) and the actual state (live objects via informers/cluster reads), diff them, and if they differ, act (apply) — then re-assess health and loop. The drift path is the interesting one: when an operator runs a manual kubectl edit or kubectl scale against the cluster, the actual state diverges from desired. The very next observe-diff detects the divergence and marks the app OutOfSync; with self-heal enabled, the act phase reverts the live object back to the Git-defined state, returning the loop to Synced. With self-heal off, the loop simply reports OutOfSync and waits for a human to sync — the divergence is visible but not auto-corrected. This is the difference between “GitOps that enforces” and “GitOps that only notifies,” and the diagram makes the branch explicit.

Real-world scenario

Northwind Logistics runs a freight-tracking platform on Kubernetes: 38 microservices across four AKS clusters — prod-eu, prod-us, staging, and dev — operated by an eight-person platform team. Before GitOps, each cluster was deployed by a per-cluster Azure Pipelines stage running helm upgrade with cluster admin credentials stored as pipeline secrets. The control planes were public (the SaaS runner needed to reach them). Drift was rampant: prod-eu had ingress-nginx 1.9 while prod-us had 1.7 because a chart bump only ran in one region; three services had replicas hand-bumped during past incidents and never reverted; and “what’s in prod?” required kubectl get against each cluster.

The trigger to change was an audit finding plus an outage. A junior engineer, paging at 2 a.m., ran kubectl set image directly on prod-us to ship a hotfix faster than the pipeline. It worked — and then the next scheduled pipeline run, an hour later, redeployed the old image from Git, silently reverting the fix and re-breaking the service. Forty-five minutes of confusion followed because nobody connected the redeploy to the regression. The lesson was stark: with no continuous reconciliation, the cluster and the repo were two separate truths, and whichever wrote last won.

They adopted Argo CD in a hub-and-spoke topology (one Argo CD instance managing all four clusters) for the application teams who wanted the UI, plus Flux on the cluster-add-on layer (ingress, cert-manager, monitoring) where the platform team wanted everything headless and CRD-driven. The deployment repo was restructured: apps/ with Kustomize bases and per-cluster overlays, infrastructure/ for add-ons, and a bootstrap/ app-of-apps. An ApplicationSet with a Cluster generator deployed the platform add-ons to every cluster labelled env=prod, so a chart bump now hit prod-eu and prod-us from one PR — ending the version-skew problem permanently. Secrets moved to SOPS with an Azure KMS key per environment (Flux decrypts in-cluster), so the database passwords lived in Git as ciphertext, rotatable by PR.

The migration ran in waves over six weeks: dev first (self-heal off, observe drift for two weeks to build trust and find which fields fought controllers — they discovered three HPAs and the Istio sidecar injector needed ignoreDifferences), then staging, then the two prod clusters with self-heal and prune on but PruneLast=true. The AKS API servers were switched to private clusters — possible only because the SaaS runner no longer needed inbound access; Argo CD pulled from inside.

The payoff was concrete. The 2-a.m. hotfix scenario reversed: a manual kubectl edit on prod-us was now reverted by Argo CD within ~20 seconds (informer-driven self-heal), forcing the right behavior — open a PR. Rollback of a bad release dropped from a 45-minute pipeline re-run to a git revert that reconciled in under a minute. Drift across the fleet went to zero and stayed there; “what’s running in prod-eu?” became “read clusters/prod-eu on main.” An auditor’s “prove every production change was reviewed and attributable” was satisfied by git log and the PR history. The platform spend was unchanged — GitOps added two small controller deployments per cluster — but the mean time to recover for deployment incidents fell roughly 70%, and the audit risk of public control planes was eliminated. The sentence the team painted on the wall: “The cluster is a follower of Git. If you want to change prod, change the repo — the cluster will catch up, and if you don’t, it’ll undo you.”

The migration as a timeline, because the order is the lesson:

Advantages and disadvantages

GitOps is a strong default for Kubernetes delivery, but it is not free of cost or sharp edges. Weigh it honestly:

GitOps is right for teams running Kubernetes at real scale who want auditability, drift safety, and a private control plane — which is most production Kubernetes. It is overkill for a single small cluster with one or two engineers, where a simple kubectl apply in CI is plenty and the CRD/operator overhead isn’t justified. It is a poor fit as your only delivery model if much of your estate is non-Kubernetes (VMs, serverless, managed PaaS) — there, GitOps covers the K8s slice and you use the platform’s own IaC/CD for the rest. And it is not a substitute for testing: GitOps faithfully deploys whatever you merged, including a broken manifest — the gate is your PR review and CI, not the reconciler.

Hands-on lab

Stand up Argo CD on a local cluster, deploy an app from Git, watch self-heal revert a manual change, then roll back via Git. Free-tier-friendly (a local kind or minikube cluster; no cloud spend). Run in a Bash shell with kubectl, helm/kustomize, and git available.

Step 1 — Create a local cluster and install Argo CD.

kind create cluster --name gitops-lab
kubectl create namespace argocd
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server --timeout=180s

Expected: the argocd-* pods reach Running; argocd-server rollout completes.

Step 2 — Get the admin password and the CLI logged in.

# Initial admin password is stored in a secret
ARGOCD_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d)
echo "admin password: $ARGOCD_PWD"

# Port-forward the UI/API and log in with the CLI
kubectl -n argocd port-forward svc/argocd-server 8080:443 >/tmp/pf.log 2>&1 &
sleep 3
argocd login localhost:8080 --username admin --password "$ARGOCD_PWD" --insecure

Expected: 'admin:login' logged in successfully. The UI is at https://localhost:8080.

Step 3 — Create an Application pointing at the public guestbook sample.

argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default \
  --sync-policy automated --self-heal --auto-prune
argocd app get guestbook

Expected: the app appears with Sync Status: OutOfSync then Synced, Health: Healthy once pods start.

Step 4 — Confirm the deployed state.

kubectl get deploy,svc -l app.kubernetes.io/instance=guestbook
argocd app get guestbook --refresh

Expected: a guestbook-ui Deployment with the replica count defined in Git (1), Synced/Healthy.

Step 5 — Introduce drift and watch self-heal revert it.

# Manually scale the deployment OUT of band — this is "drift"
kubectl scale deploy guestbook-ui --replicas=5
# Watch Argo CD detect OutOfSync and self-heal back to the Git value (1)
watch -n 2 'kubectl get deploy guestbook-ui -o jsonpath="{.spec.replicas}"; echo'

Expected: replicas briefly show 5, then Argo CD reverts them to 1 within seconds (informer-driven self-heal). The app momentarily goes OutOfSync then back to Synced. This is the core GitOps property in action — the cluster follows Git, not your manual edit.

Step 6 — Roll back via Git semantics (history + sync).

# Argo CD tracks sync history; view it
argocd app history guestbook
# Roll back to a previous synced revision by ID (simulates 'git revert' + reconcile)
argocd app rollback guestbook <REVISION_ID>
argocd app get guestbook

Expected: the app re-syncs to the chosen historical revision. In a real repo this is git revert <sha> && git push, after which Argo CD reconciles automatically.

Validation checklist. You installed an in-cluster controller, deployed an app whose source of truth is a Git repo, proved self-heal by reverting a manual kubectl scale, and rolled back via history. No kubectl apply of the app itself ever happened from your shell — the controller did it. The lab steps mapped to what each proves:

Cleanup.

argocd app delete guestbook --yes
kind delete cluster --name gitops-lab
# stop the port-forward
kill %1 2>/dev/null || true

Cost note. Entirely local (kind/minikube) — zero cloud spend. On a managed cluster, Argo CD itself is a handful of small pods; the cost is the cluster you already run.

Common mistakes & troubleshooting

This is the playbook you bookmark. First a scannable table, then the worst offenders expanded.

The expanded form for the two that bite hardest:

Prune deleted production objects (#4). With prune: true, a render that produces an empty or reduced manifest set — a Kustomize error, a deleted file, a path typo pointing at an empty directory — makes the controller conclude those objects were removed from Git and delete them from the cluster. Confirm: application events show Pruned/Deleted for resources you didn’t intend to remove, and the Git diff emptied a path or broke the render. Fix: re-sync the corrected manifests to restore. Prevent recurrence with PruneLast=true, prune: false on the bootstrap/root app, CI render validation (kustomize build must produce the expected object count), and resource-exclusion for critical objects. Treat prune as rm -rf with a loaded source.

Secret is empty and the app crash-loops (#6). The secrets layer failed silently — SOPS couldn’t decrypt (missing age/KMS key, or KMS IAM denied), Sealed Secrets can’t unseal (wrong cluster’s key), or ESO can’t fetch (bad SecretStore auth / Key Vault RBAC). The Secret ends up empty or missing, the app reads a blank value, and it crash-loops — looking like an app bug. Confirm: kubectl get secret <name> -o yaml shows no data or a missing resource; the relevant controller logs show a decrypt/fetch error (kustomize-controller for SOPS, the sealed-secrets controller, or external-secrets). Fix: restore the controller’s access to the key/store (KMS IAM, age key Secret, Key Vault role assignment), or re-seal/re-encrypt with the correct key. Never “fix” it by committing a plaintext secret — that leaks it into history permanently.

Best practices

• Pin production to a tag or SHA, not a moving branch. “Prod is at exactly v2.4.1” is a property worth keeping; promote by changing the pin in a reviewed PR, so you always know — and can reproduce — what’s live.
• Separate the app source repo from the deployment (config) repo. CI builds and tests the app in its repo and writes the new image tag to the deployment repo. This keeps the CD trigger reviewable and stops app-code churn from spamming the reconciler.
• Own the fields you set; ignore the rest. Don’t declare replicas if an HPA owns it; add ignoreDifferences for webhook-injected and server-defaulted fields. This single discipline prevents most self-heal flapping.
• Roll out self-heal and prune in stages. Start observe-only (detect drift, don’t act), find the fields that fight controllers, then enable self-heal, then prune with PruneLast. Never enable prune on a root/bootstrap app you don’t fully trust.
• Validate renders in CI before merge. Run kustomize build / helm template (and a policy check like kubeconform/conftest) in the PR pipeline so a broken manifest never reaches the reconciler — a broken render plus prune is a deletion event.
• Use sync waves for real ordering only. CRDs/namespaces before their consumers; migrations (PreSync) before deploy; ingress after the app is healthy. Don’t over-engineer waves where Kubernetes’ own dependency handling suffices.
• Keep PreSync hooks self-contained. A migration Job must carry its own image and config; if it depends on objects that only land during Sync, it deadlocks.
• Treat ApplicationSet/generator changes as infrastructure changes. They fan out to many apps/clusters; review them with that blast radius in mind, and use preserveResourcesOnDeletion while building confidence.
• Never commit plaintext secrets; pick one of SOPS / Sealed Secrets / ESO. Base64 is not encryption. Decide per constraint (in-Git ciphertext vs external store) and apply it consistently.
• Wire short interval + webhooks. The webhook deploys fast on merge; the interval corrects drift and catches missed webhooks. Tune Flux interval per resource so you don’t hammer the API server fleet-wide.
• Make the API server private once you’re pull-based. The biggest security win of GitOps is that no external CI needs inbound cluster access — capture it by locking down the control plane.
• Alert on GitOps signals, not just app health. App OutOfSync duration, sync failures, ComparisonError, controller reconcile latency, and prune events are leading indicators of a delivery problem.

The signals worth alerting on before the next incident:

Security notes

• The pull model is the headline control — use it fully. Because the controller pulls from inside the cluster, no external system needs cluster admin credentials. Make the API server private, scope CI to only writing to Git/registry, and you’ve removed the largest attack surface of push CD (a compromised SaaS runner with a prod kubeconfig).
• Scope the controller’s RBAC tightly. Argo CD’s AppProject and Flux’s namespaced reconcilers + RBAC limit which repos, clusters, and namespaces a tenant can deploy to. Don’t run everything as cluster-admin; bound each team to its own destinations.
• Sign your commits and verify them. Argo CD and Flux can require GPG-signed commits (Flux verify on GitRepository) so only signed, attributable revisions are reconciled — closing the “someone pushed an unsigned change” gap.
• Never put plaintext secrets in Git. Once pushed, a secret is in history forever (and on every clone), so deletion doesn’t help — you must rotate it. Use SOPS/Sealed/ESO so Git only ever holds ciphertext or a reference, and the cluster/operator holds the decryption capability.
• Lock down the GitOps UI/API. Argo CD’s server is powerful (sync, rollback, exec). Put it behind SSO (OIDC), enforce RBAC, restrict network access, and don’t expose it to the internet without authentication.
• Protect the deployment repo like production. Branch protection, required reviews, and required status checks on the deployment repo are your production change control — a merge there changes prod. Treat repo write access as prod access.
• Pin and scan images; pin chart/CRD sources. Reference images by digest where possible, scan them in CI, and pin Helm chart and CRD versions so a moved tag can’t silently change what reconciles.
• Use workload identity for secret backends. With ESO/SOPS-KMS, the controller should authenticate to Key Vault/KMS via workload identity / IRSA / managed identity, not a long-lived static credential.

The security controls mapped to what they prevent:

Cost & sizing

GitOps adds little to the bill — its cost is operational, not infrastructural. The components are a few small controller Pods per cluster; the spend you already have is the cluster.

• Controller footprint. Argo CD is a handful of pods (server, repo-server, application-controller, redis); Flux is a set of small controllers. On a modest cluster they fit comfortably in a few hundred millicores and a few hundred MB to a couple of GB total — negligible against the workloads they manage. The exception is very large apps/repos, where repo-server (render) and application-controller (diff) memory grows; split big apps and raise limits rather than running one giant Application.
• Hub vs per-cluster. A hub-and-spoke Argo CD is one control plane to run (cheaper to operate) but a single point of failure; per-cluster agents multiply the (small) footprint but isolate failure. The cost difference is operational overhead, not cloud spend.
• Webhook vs interval. This is a load decision, not a billing one: a too-short Flux interval across hundreds of Kustomizations hammers the API server and Git provider (risking rate limits), while a too-long interval delays drift correction. Short interval + webhooks is the efficient middle.
• Secret backends carry their own small cost. ESO with a cloud secret manager incurs the manager’s per-secret/per-operation charge; SOPS with KMS incurs KMS key + decrypt-call charges; Sealed Secrets is free (in-cluster key). All are tiny next to the value of not leaking credentials.
• The real saving is MTTR and toil. GitOps doesn’t lower your cloud bill; it lowers the cost of operating — faster rollback, no drift firefighting, reproducible clusters, and audit you don’t assemble by hand. Northwind’s spend was flat but deployment-incident MTTR fell ~70%.

A rough sizing picture and what drives each cost:

Interview & exam questions

1. What are the four GitOps principles, and which one distinguishes GitOps from “deploy on merge”? Declarative, versioned & immutable, pulled automatically, and continuously reconciled. The fourth — continuous reconciliation — is what distinguishes true GitOps: it re-asserts desired state on a schedule regardless of whether anything merged, so a manual kubectl edit is reverted even with no new commit. “Deploy on merge” only acts on a webhook and never re-checks the cluster, so it doesn’t correct drift.

2. Why is the pull model more secure than push-based CD? In pull-based GitOps the controller runs inside the cluster and reaches out to Git/registry, so no external system needs cluster admin credentials and the API server can be private. Push CD requires the (often SaaS) runner to hold a powerful kubeconfig and reach the control plane, expanding the attack surface to “compromise the runner, own the cluster.”

3. Explain drift detection vs self-heal vs prune. Drift detection computes and reports the diff between live and desired (always on, read-only). Self-heal automatically reverts live drift back to Git. Prune deletes live objects that no longer exist in Git. They’re independent policies — you can detect without healing, and heal without pruning — and prune is the dangerous one because a bad render can delete production objects.

4. An app flaps OutOfSync and replicas keep changing. Diagnose. Self-heal is fighting another controller (almost always an HPA managing replicas, or a mutating webhook) over a field you declared in Git. Fix by not declaring the field (let the HPA own replicas) and adding ignoreDifferences for that path so the diff stays clean. The rule: own the fields you set, ignore the fields other controllers set.

5. Compare Argo CD and Flux at a high level. Argo CD is an all-in-one, application-centric platform with a first-class UI, the Application/ApplicationSet model, and a hub multi-cluster pattern. Flux is the GitOps Toolkit — composable single-responsibility controllers (source/kustomize/helm/image), CLI-and-CRD-first with no built-in UI, native SOPS decryption, real Helm releases, and idiomatic agent-per-cluster. Choose Argo CD for UI/self-service and central multi-cluster; Flux for headless platform automation, native Helm, and in-cluster image automation.

6. What is the app-of-apps pattern and when do you outgrow it? A single root Application points at a directory of more Application manifests, so syncing the root bootstraps all children — GitOps managing GitOps. You outgrow it when you need the same app across many clusters or per-directory generation, because app-of-apps is static (per-cluster edits are manual). That’s when you move to ApplicationSets with generators.

7. What problem do sync waves and resource hooks solve? Ordering. A sync applies many objects but has dependencies: CRDs/namespaces before their consumers, a DB migration before the new app (PreSync hook), ingress after the app is healthy (later wave), a smoke test after deploy (PostSync hook). Sync waves order resources within a sync; hooks run Jobs at phase boundaries (PreSync/Sync/PostSync/SyncFail).

8. How do you handle secrets in GitOps? Never commit plaintext (base64 is not encryption). Three patterns: SOPS (encrypt values in Git, controller decrypts in-cluster via age/KMS — native in Flux), Sealed Secrets (encrypt to a per-cluster controller key; only that cluster can unseal), and External Secrets Operator (keep secrets in Key Vault/Secrets Manager/Vault, commit only a reference). Git holds ciphertext or a reference; the cluster/operator holds the decryption capability.

9. How does GitOps relate to progressive delivery? GitOps gets the desired manifests onto the cluster but doesn’t itself do canary/blue-green with metric-based rollback. Argo Rollouts (with Argo CD) and Flagger (with Flux) add that — and crucially the Rollout/Canary definition lives in Git, so the release strategy is versioned and reconciled too. CI builds the image, GitOps reconciles it, progressive delivery ramps traffic with metric gates and auto-rolls-back.

10. Why can enabling prune delete your production app, and how do you prevent it? If a render produces an empty or reduced manifest set (a Kustomize error, a deleted file, a path typo), the controller concludes those objects were removed from Git and deletes them. Prevent it with PruneLast=true, prune: false on root/bootstrap apps, CI render validation (assert the expected object count), and resource-exclusion for critical objects. Treat prune like rm -rf with a loaded source.

11. What’s the difference between hub-and-spoke and standalone-agent multi-cluster, and what’s the security trade-off? Hub-and-spoke runs one control plane (e.g. Argo CD) that holds credentials to many registered clusters — single pane of glass, but the hub is a SPOF and its compromise reaches every cluster. Standalone agents (idiomatic Flux) run one controller per cluster watching only its own config, holding no cross-cluster credentials — strong isolation, contained blast radius, but no single UI. Regulated/high-isolation estates lean standalone.

12. A new image was built but isn’t running in the cluster. What do you check? Whether the new tag was written to the deployment repo (CI write-back), and whether a webhook fired or you’re waiting on the poll interval. Confirm with argocd app get (deployed revision vs latest commit) or Flux’s reconcile status. Fix by ensuring CI commits the tag, wiring a Git webhook, or forcing argocd app sync / flux reconcile. GitOps only deploys what’s in Git — if CI didn’t commit the tag, nothing changed.

These map to CKA/CKAD (Kubernetes objects, controllers, RBAC), the Argo CD / Flux project knowledge expected in platform-engineering interviews, and cloud certs where the deployment target lives — AZ-204 (deploy to AKS), AZ-400 (DevOps, release strategies, secrets), and the secrets angle touches AZ-500. A compact cert mapping:

Quick check

1. Your “GitOps” pipeline only acts on a webhook when something merges and never re-checks the cluster. Which GitOps principle is it violating, and what capability do you lose?
2. An app is flapping Synced↔OutOfSync and the deployment’s replicas keeps changing. What is almost certainly happening, and what are the two fixes?
3. True or false: committing a base64-encoded Secret to Git is an acceptable way to store secrets in GitOps.
4. You enabled prune: true and a chunk of production disappeared after a merge. What likely happened, and name two guardrails that prevent it.
5. You build a new image in CI, but it never deploys. Name two things to check.

Answers

1. It violates continuous reconciliation (principle 4) — it’s automated push CD triggered by Git, not GitOps. You lose drift correction: a manual kubectl edit made when nothing merged is never detected or reverted, so the cluster silently diverges from intent between deploys.
2. Self-heal is fighting another controller — almost always an HPA (or a mutating webhook) that owns replicas, which you also declared in Git. The two fixes are: (a) stop declaring replicas in the manifest so the HPA owns it, and (b) add ignoreDifferences for /spec/replicas so the diff stays clean.
3. False. Base64 is encoding, not encryption — anyone with repo read access decodes it instantly, and once pushed it’s in history forever. Use SOPS, Sealed Secrets, or External Secrets Operator so Git holds ciphertext or a reference only.
4. A render produced an empty or reduced manifest set (Kustomize error, deleted file, or a path typo pointing at an empty directory), so the controller concluded those objects were removed from Git and deleted them. Guardrails: PruneLast=true, keep prune: false on root/bootstrap apps, and validate renders in CI (assert the expected object count) before merge.
5. Check (a) whether CI wrote the new image tag to the deployment repo (the desired state must actually change in Git) and (b) whether a webhook fired or you’re waiting on the poll interval — confirm the deployed revision vs the latest commit with argocd app get / flux status, and force a sync if needed.

Glossary

• GitOps — an operating model where Git is the single source of truth for declarative infrastructure and apps, and an in-cluster agent continuously reconciles the cluster toward the committed state.
• Reconciliation loop — the continuous observe → diff → act cycle a GitOps controller runs to drive actual cluster state toward the desired state in Git.
• Desired state / actual state — what you declared in Git versus what is live in the cluster; reconciliation minimizes their difference.
• Drift — any difference between the live cluster and the Git-declared state; detected by the three-way diff.
• Self-heal — automatically reverting live drift back to the Git-defined state (Argo CD selfHeal; Flux continuous apply).
• Prune — deleting live objects that no longer exist in Git; powerful and dangerous (a bad render can delete production).
• Argo CD — a CNCF GitOps controller with a first-class UI and an application-centric model (Application, ApplicationSet, AppProject).
• Flux — the CNCF GitOps Toolkit: composable single-responsibility controllers (source/kustomize/helm/image), CRD-and-CLI-first, native SOPS and real Helm.
• Application (Argo CD) — the CRD declaring a source (repo/path/revision), a destination (cluster/namespace), and a sync policy.
• Kustomization / HelmRelease (Flux) — the CRDs that build a Kustomize path or run a Helm release and reconcile them.
• App-of-apps — a root Argo CD Application pointing at a directory of more Application manifests, bootstrapping many apps from one.
• ApplicationSet — an Argo CD controller that generates many Applications from a template plus a generator (List/Cluster/Git/Matrix/PR/SCM).
• Sync wave — an integer annotation ordering resource application within a sync; Argo CD waits for each wave to be healthy before the next.
• Resource hook — a Job run at a sync phase (PreSync/Sync/PostSync/SyncFail), e.g. a DB migration before deploy or a smoke test after.
• dependsOn (Flux) — ordering between Kustomizations, with wait/healthChecks to gate on health before the next reconciles.
• Three-way diff / server-side apply — comparing desired vs live vs last-applied and merging by field ownership, so controllers co-own objects without clobbering.
• ignoreDifferences — telling the diff to skip certain fields/paths (e.g. HPA-owned replicas, webhook mutations) to stop self-heal fights.
• SOPS — encrypts secret values in Git via age/PGP/KMS; the controller decrypts in-cluster (native in Flux).
• Sealed Secrets — encrypts a Secret into a SealedSecret only a per-cluster controller can unseal.
• External Secrets Operator (ESO) — syncs secrets from an external manager (Key Vault/Secrets Manager/Vault) into Kubernetes Secrets; Git holds only a reference.
• Argo Rollouts / Flagger — progressive-delivery controllers (canary/blue-green with metric-based rollback) that compose on top of Argo CD / Flux, with the strategy defined in Git.
• Hub-and-spoke / standalone agent — multi-cluster topologies: one control plane managing many registered clusters, versus one agent per cluster holding no cross-cluster credentials.

Next steps

You can now run GitOps in production — reconcile from Git, correct drift, manage a fleet, handle secrets, and layer on canaries. Build outward:

• Next: Progressive Delivery: Canary, Blue-Green and Automated Rollback with GitOps — put Argo Rollouts/Flagger on top of what you just built, with the strategy in Git.
• Related: CI/CD Pipelines Explained: From Code Commit to Production — the build-and-test half that feeds image tags into your GitOps repo.
• Related: Deployment Strategies: Blue-Green, Canary and Rolling Updates — the release-strategy mechanics GitOps stores declaratively.
• Related: Infrastructure as Code: Terraform, Pulumi, CDK and Cloud-Native Options — how GitOps relates to (and complements) IaC for cluster and cloud resources.
• Related: CI/CD Secrets and Credential Management: Secure Your Pipelines — the secrets discipline behind SOPS/Sealed/ESO.
• Related: Deploy Argo CD on Kubernetes with OIDC SSO, RBAC, and ApplicationSets for Multi-Cluster GitOps — a hands-on build of the multi-cluster Argo CD setup described here.