Diffing & Drift: ignoreDifferences, Server-Side Diff, Mutating Webhooks & Perpetual OutOfSync

In a nutshell

Argo CD has one job, and it does it over and over: answer “does the cluster still match Git?” When the answer is “no,” it says OutOfSync. Usually that is exactly what you want. But sometimes an app goes OutOfSync and stays there no matter how many times you sync it — because a different piece of software in the cluster (an autoscaler, a service-mesh injector, a defaulting webhook) keeps editing a field you never wrote. Argo faithfully reports that edit as drift, and if self-heal is on, the two controllers fight forever. This lesson is about teaching Argo which fields it does not own, so the diff stops flagging other people’s edits and only real drift trips the verdict.

Think of Argo as a proofreader comparing your published manuscript (Git) against the copy on the shelf (the live cluster). The catch: other editors are allowed to scribble in the shelf copy — the autoscaler changes the print run, the mesh stamps a watermark. The proofreader keeps circling those scribbles as “errors” until you tell it: those margins belong to the other editors — don’t compare them. ignoreDifferences, managedFieldsManagers and Server-Side Diff are three ways of drawing that boundary.

Level: Intermediate · Time: ~42 min

Prerequisites: you can create an Application and read its Sync vs Health status (see Sync Status & Health Assessment). After this you can: read an argocd app diff, name the controller that owns a drifting field, and choose between jsonPointers, jqPathExpressions, managedFieldsManagers and Server-Side Diff with confidence.

---

There is one Argo CD problem that shows up on every platform team’s Slack sooner or later, and it always sounds the same: “This app is stuck OutOfSync. I’ve synced it five times. It goes green for a second, then flips straight back. Nothing in Git changed. What is going on?” This is perpetual OutOfSync, and it is not a bug in Argo CD. It is Argo CD doing exactly its job — reporting that the live cluster no longer matches Git — while something else in the cluster keeps quietly editing a field Argo never set.

That “something else” is almost always another controller doing legitimate work: a HorizontalPodAutoscaler adjusting replicas, a mutating webhook injecting a service-mesh sidecar, an admission controller stamping a default, cert-manager rewriting a CA bundle inside a Secret. Argo renders your manifest, sees a field on the live object that isn’t in your manifest, and honestly calls it drift. If you have automated self-heal on, Argo now reverts that field, the other controller re-applies it, and the two of them war forever — a flap you can watch in real time.

The fix is never to disable self-heal and paper over the whole thing. The fix is to teach Argo which fields it does not own, so the diff ignores them and only real drift trips the verdict. This lesson is about the three tools for that job — ignoreDifferences, managedFieldsManagers, and Server-Side Diff — and, just as importantly, about how the diff actually works so you can tell which tool a given problem needs. Every manifest here is schema-correct for Argo CD 2.13+/3.x on Kubernetes 1.29+, and the lab reproduces the HPA flap end-to-end and fixes it three different ways so you can compare them.

---

Why this matters

Argo CD’s entire value proposition rests on one comparison: does the live cluster match the desired state in Git? That comparison produces the Synced / OutOfSync verdict you see on every app. When the comparison is right, Argo is a truth machine — it catches every out-of-band kubectl edit, every manual scale, every drift, and (with self-heal) drags the cluster back. When the comparison produces false positives, the whole thing inverts: your dashboard is a wall of yellow, real drift hides in the noise, and engineers learn to ignore Argo — which defeats the point of running it.

Perpetual OutOfSync is the number-one cause of that false-positive noise, and it comes from a genuine tension in Kubernetes: many controllers legitimately share ownership of a single object. You write a Deployment with replicas: 2 and commit it. Then an HPA takes over replicas because scaling is its job. Then a mesh webhook adds a sidecar container because injection is its job. The object in the cluster is now a blend of your fields and their fields — but Git only has your fields. A naive field-by-field diff flags every field the others added.

The mental model to hold for the whole lesson is this: the diff is only as good as its notion of “what Argo owns.” A diff that assumes Argo owns every field on the object will fight every other controller in the cluster. A diff that correctly scopes to only the fields Argo actually set will sit quietly Synced while HPAs, webhooks and defaulters do their thing. Everything below — ignoreDifferences, managed fields, Server-Side Diff — is a different way of drawing that ownership boundary. Getting it right is the difference between a GitOps platform people trust and one they route around.

This lesson assumes you already know the two status axes from Sync Status & Health Assessment — that OutOfSync is about matching Git, not about whether pods are healthy — and it pairs tightly with Sync Policies: Automated Sync, Self-Heal & Prune, because self-heal is what turns a harmless false diff into an active flap.

---

How Argo CD actually computes the diff

Before you can fix a bad diff you have to know how the diff is produced. The Sync Status lesson introduced the five-step pipeline; here we go deeper on the two steps that produce false positives — normalize and compare — because that is where every fix in this lesson plugs in.

The application-controller runs this loop for every managed resource:

Two properties of this pipeline cause nearly every surprise:

The two operands are not symmetric. Desired is the rendered output of your Git source — a small, tidy set of fields you actually declared. If you build with Kustomize or Helm, that means the diff runs against the kustomize build / helm template output, not your raw overlay or values files, so a change that alters the rendered result flips you OutOfSync while a comment does not (the Kustomize Integration lesson covers what that rendering step emits). Live is the full object as the API server returns it — your fields plus every default the server filled in plus every field other controllers mutated plus system metadata. The diff’s whole job is to decide which of those extra live fields count.

Normalization is where the false positives live and die. Argo automatically removes a set of fields that are never meaningful to compare — this built-in normalization is why your apps aren’t permanently OutOfSync on metadata.resourceVersion:

The last row is the entire problem. Argo can strip fields it knows are noise, but it cannot guess that your HPA owns replicas or that your mesh injects istio-proxy. Those are cluster-specific facts only you know — so those are the fields you have to declare. The diagram below is the whole engine in one picture: desired and live flow into the controller, the diff runs field-by-field, and the three filters remove the fields Argo doesn’t own before the verdict is struck.

The badges mark the load-bearing ideas: another controller owns a field Argo never set (1); the naive diff flags it as drift and the app goes perpetually OutOfSync (2); a scoped ignoreDifferences drops that exact field (3); managedFieldsManagers drops everything a named controller owns (4); Server-Side Diff makes the API server account for defaults and webhooks automatically (5); and with any filter in place only real drift trips the verdict, so you never resort to disabling self-heal (6).

---

Perpetual OutOfSync: who keeps changing your fields

When an app is stuck OutOfSync and re-syncing doesn’t help, the diagnosis is always the same shape: a field exists on the live object that isn’t in your rendered manifest, and something keeps putting it back. The skill is identifying which field and which actor. Here are the offenders you will actually meet, ranked by how often they generate tickets:

Two of these deserve extra attention because they trip up almost everyone.

The HPA case is the canonical one. You declare replicas: 2 in Git. The HPA’s job is to own the replica count, so it writes replicas: 5 (or whatever the metrics demand) directly onto the Deployment through the scale subresource. Now desired.spec.replicas = 2 and live.spec.replicas = 5 — a permanent, legitimate disagreement. Without a filter, the app is OutOfSync forever. With self-heal on, it’s worse: Argo sets it back to 2, the HPA immediately scales it back to 5, and you get a visible flap every reconciliation interval. This is the single most common perpetual-OutOfSync ticket in existence, and it’s the one we reproduce in the lab.

Sidecar injection is the second. A service mesh installs a mutating admission webhook that intercepts pod creation and injects an istio-proxy (or linkerd-proxy) container, an init container, and volumes — none of which are in your Deployment manifest. Argo renders your two-container Deployment, reads back a three-container Deployment, and flags the whole injected container as drift. You cannot fix this by editing your manifest (you don’t want the sidecar in Git — the mesh owns it), so you must tell the diff to ignore what the webhook owns.

The cloud edge: which webhooks mutate your objects

The diff engine itself is completely cloud-neutral — the same normalization runs identically on AKS, EKS and GKE. But the set of controllers mutating your objects is very much cloud-specific, because each managed Kubernetes platform ships its own admission webhooks for identity, load balancing, and (on Autopilot) resource sizing. If you run the same manifest on all three clouds, you will hit different false diffs on each:

The practical takeaway is the reason Server-Side Diff exists: on managed clouds you cannot enumerate every webhook by hand, and the set changes as you enable features. GKE Autopilot silently rewriting your resource requests is a textbook example — you never wrote those values, Autopilot did, and no amount of editing your manifest makes the diff go away. The server-side approach lets the API server tell Argo what the object would look like after all these webhooks run, so the injected fields simply match. We’ll get there; first, the surgical tool.

---

ignoreDifferences, field by field

ignoreDifferences is a list on the Application spec — the same spec whose source and destination anatomy the first Application lesson walks through — or a global rule in argocd-cm, covered later. Each entry names a resource and then names the fields on it to drop from the diff. It has exactly these fields:

You supply at least one of the three selection mechanisms (jsonPointers, jqPathExpressions, managedFieldsManagers). They can be combined. Here is the canonical HPA fix — the smallest possible change that stops the flap:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/web.git
    targetRevision: main
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: web
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: web              # scope to just this Deployment
      jsonPointers:
        - /spec/replicas     # the HPA owns this — stop diffing it

jsonPointers, jqPathExpressions, managedFieldsManagers — which to reach for

These three are not interchangeable; each is best at a different shape of problem.

jsonPointers use JSON Pointer syntax, which is worth knowing precisely because a wrong pointer silently does nothing — it matches no field, so the diff is unchanged and you think the feature is broken:

Note the escaping: ~1 means a literal / inside a key, and ~0 means a literal ~. Array elements are addressed by ordinal index (/0, /1), which is exactly why pointers are fragile for lists whose order can change — that’s the case for jqPathExpressions.

jqPathExpressions let you select by content, which is what you need for an injected sidecar (you don’t know its index, but you know its name):

  ignoreDifferences:
    - group: apps
      kind: Deployment
      jqPathExpressions:
        # ignore the whole injected sidecar container, wherever it lands in the list
        - '.spec.template.spec.containers[] | select(.name == "istio-proxy")'
        - '.spec.template.spec.initContainers[] | select(.name == "istio-init")'

managedFieldsManagers is the Server-Side-Apply-era answer, and it is usually the cleanest of the three because it matches the way Kubernetes already tracks ownership. Every field on an object records which manager last set it in metadata.managedFields. When the HPA writes replicas through the scale subresource, the manager recorded is kube-controller-manager. So instead of naming the field, you name the manager:

  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: web
      managedFieldsManagers:
        - kube-controller-manager   # ignore every field the HPA controller owns

The advantage is that you don’t have to enumerate fields at all — if the controller starts writing a second field tomorrow, this rule already covers it. For a webhook-injected sidecar, you point it at the webhook’s manager name and the entire injection (container, volumes, annotations) is ignored in one line. The catch is that field ownership is only reliably tracked when writes go through Server-Side Apply, which modern controllers and the API server do — but it’s why managedFieldsManagers pairs naturally with ServerSideApply=true.

Here is the same three-way decision as a worked table you can copy from:

That last row is the honest guidance the rest of the lesson builds toward: when the list of things to ignore is long or unknowable, stop maintaining a list and let the server compute the diff.

---

Ignoring a diff is not pruning, and not self-heal

A subtle and important point that catches people: ignoreDifferences only affects the diff. It changes whether the app is reported OutOfSync. It does not, by default, change what a sync does, and it has nothing to do with prune. These three controls are orthogonal, and conflating them is a classic mistake:

The consequence worth internalizing: with a plain ignoreDifferences on /spec/replicas, your app shows Synced (good) — but the desired manifest still contains replicas: 2. So the next time a sync runs for any reason — you bump the image tag, someone clicks Sync, self-heal fires on a different field — Argo applies the full manifest including replicas: 2, momentarily stomping the HPA’s value before the HPA scales it back. The diff was ignored; the write was not.

If you need the sync itself to leave the field alone, add the RespectIgnoreDifferences=true sync option:

  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true
    automated:
      prune: true
      selfHeal: true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: web
      jsonPointers:
        - /spec/replicas

Now both the diff and the sync skip /spec/replicas, and the HPA is never disturbed. One caveat to know: RespectIgnoreDifferences reliably honours jsonPointers and managedFieldsManagers; its support for jqPathExpressions has historically been limited, so when you need the sync (not just the diff) to leave a field alone, prefer jsonPointers or a managed-fields rule. And note what none of this changes: self-heal on every other field still works. Ignoring replicas does not disable drift correction for the image, the env vars, or anything else — which is exactly why “just set selfHeal: false” is the wrong fix. That switch disables drift correction for the entire app, blinding you to the real changes GitOps exists to catch.

---

Server-Side Diff: let the API server do the math

Everything above draws the ownership boundary by hand. Server-Side Diff draws it automatically, and it is the modern answer that removes whole categories of ignoreDifferences hacks.

The idea: instead of Argo rendering the desired manifest and diffing it against live itself (a “client-side” diff that has no idea what defaults or webhooks would do), Argo asks the API server to perform a dry-run Server-Side Apply of the desired manifest and return the object that would result. Because that dry-run runs the full admission chain — defaulting, mutating webhooks, and the SSA merge that respects other managers’ fields — the returned object already contains the server defaults, the injected sidecar, the Autopilot-adjusted resources, everything. Argo then diffs that against live. The injected and defaulted fields are present on both sides, so they simply match. No list to maintain.

Enable it per-app with the ServerSideDiff=true sync option (it pairs naturally with ServerSideApply=true):

  syncPolicy:
    syncOptions:
      - ServerSideApply=true
      - ServerSideDiff=true
    automated:
      prune: true
      selfHeal: true

Or turn it on fleet-wide via the controller parameter controller.diff.server.side: "true" in the argocd-cmd-params-cm ConfigMap (restart the application-controller to pick it up). When it’s on globally, you can opt a single resource out with the annotation argocd.argoproj.io/compare-options: ServerSideDiff=false.

Client-side vs server-side, head to head:

The honest ordering for a real platform is: reach for Server-Side Diff first. It fixes the largest class of problems (webhooks, server defaults, Autopilot resource mutation) with one flag and no per-field maintenance. Then, for the specific fields it doesn’t cover — chiefly the HPA replicas case, because the HPA writes through the scale subresource rather than something the dry-run reproduces — reach for managedFieldsManagers (kube-controller-manager), which is self-maintaining. Only drop to targeted jsonPointers when you need to name one exact field, or when you also need RespectIgnoreDifferences on the sync. And never reach for selfHeal: false to silence a flap — that’s not a fix, it’s turning off the smoke detector.

---

Diff normalization and global rules

Everything so far was per-Application. When the same false diff hits many apps — say every Deployment in the fleet is behind the same defaulting webhook — you don’t want to copy ignoreDifferences into every Application. Put it once in the argocd-cm ConfigMap and it applies globally.

Global ignoreDifferences uses resource.customizations.ignoreDifferences.<group>_<kind> (note the _ separator between group and kind; core group is empty, so it’s just _Kind):

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  # ignore HPA-owned replicas on EVERY Deployment, fleet-wide
  resource.customizations.ignoreDifferences.apps_Deployment: |
    managedFieldsManagers:
      - kube-controller-manager
    jsonPointers:
      - /spec/replicas
  # a rule that applies to ALL resource kinds
  resource.customizations.ignoreDifferences.all: |
    jsonPointers:
      - /metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration

Two related argocd-cm keys round out normalization, and it’s worth knowing which does what because their names are similar:

knownTypeFields deserves a word because it fixes a specific and confusing false diff: quantity formatting. If a CRD stores a value that is semantically a resource quantity but Argo doesn’t know its type, 500m in Git and 0.5 in live look different. Declaring the field’s known type makes the normalizer parse both as quantities and compare them equal:

data:
  resource.customizations.knownTypeFields.example.com_MyResource: |
    - field: spec.cpuRequest
      type: core/v1/ResourceList

Finally, the RespectIgnoreDifferences=true sync option discussed earlier is the bridge between normalization and sync: normalization/ignoreDifferences decide what the diff ignores; RespectIgnoreDifferences extends that same ignore-set to the sync write. Keep the two ideas distinct and you’ll never be surprised by a sync stomping a field you thought you’d protected.

---

Debugging a diff

When an app is OutOfSync and you can’t see why from the UI, drive the diff from the CLI. argocd app diff prints the exact field-level difference the controller sees — it is the fastest way to identify which field and therefore which filter you need.

# Show the live-vs-desired diff for one app (exit code 1 if there IS a diff)
argocd app diff web

Representative output for the HPA case — labelled, since the shape is what matters:

# representative argocd app diff output
===== apps/Deployment web/web ======
5c5
<   replicas: 5
---
>   replicas: 2

Read it as: the < line is the live cluster value (replicas: 5, written by the HPA) and the > line is the desired value from Git (replicas: 2). That one field is the entire reason the app is OutOfSync — which immediately tells you to ignore /spec/replicas (or the HPA’s field-manager). The most useful argocd app diff flags:

That last technique is the one that tells you which field-manager to name. To find out who owns replicas:

# Who last wrote the fields on this Deployment? Look for the scale subresource.
kubectl -n web get deploy web --show-managed-fields -o yaml | grep -A6 managedFields

# representative — the HPA writes replicas via kube-controller-manager
  managedFields:
  - manager: kube-controller-manager
    operation: Update
    subresource: scale
    fieldsV1:
      f:spec:
        f:replicas: {}

The manager: kube-controller-manager and subresource: scale lines are your proof that managedFieldsManagers: [kube-controller-manager] is the correct, self-maintaining fix. In the UI, the same information lives in the app’s Diff tab (toggle “Compact diff” off to see full context) and the per-resource Manifest view, which shows Desired and Live side by side.

---

Going deeper

The core lesson gives you the three filters and Server-Side Diff. This section covers the parts that bite after you have deployed them — secret redaction, the mutation-webhook caveat hiding inside Server-Side Diff, the reconcile-path cousin of ignoreDifferences, the diff cache, and how the diff actually merges lists.

Secrets are redacted in the diff — that is not a bug

The first time a Secret drifts, engineers panic: argocd app diff reports it OutOfSync but every value is masked. Argo CD intentionally redacts the data and stringData values of Secret resources in both the CLI diff and the UI — replacing each value with a fixed token (++++++++) — so credentials never leak into a terminal, a screenshot, or the application-controller logs. You still see which keys were added, removed, or changed; you just never see the value.

# representative — a Secret diff shows changed KEYS, redacted VALUES
===== /Secret example/db-creds ======
2c2
<   password: ++++++++   # live (redacted)
---
>   password: ++++++++   # desired (redacted)

The practical consequences:

• A Secret can read OutOfSync with every value masked, which tells you something changed but not what. To find out, inspect the source of the value — the SealedSecret, ExternalSecret, or SOPS file — not the rendered diff (the secrets lesson covers those controllers).
• Redaction is why a Secret an operator keeps rewriting (cert-manager re-injecting ca.crt, say) is a perpetual-OutOfSync case you cannot debug by reading values — you fix it with ignoreDifferences on the exact key (/data/ca.crt), exactly as for any other owned field.
• Redaction applies to the diff view only — it changes nothing about what is stored, synced, or applied.

Server-Side Diff and mutation webhooks: the IncludeMutationWebhook refinement

Server-Side Diff’s dry-run Server-Side Apply always accounts for API-server defaulting and the SSA merge that respects other field-managers. But there is one precise caveat worth committing to memory: by default the dry-run does not execute mutating admission webhooks. So a mesh sidecar injected by a webhook can still show as drift under a plain ServerSideDiff=true, because the predicted object never gained the injected container.

To fold mutation webhooks into the dry-run, add the compare-option annotation — on the individual resource, or on the Application to cover all of its resources:

metadata:
  annotations:
    argocd.argoproj.io/compare-options: IncludeMutationWebhook=true

It is opt-in for two reasons: cost and determinism. Running every mutating webhook on every diff adds latency, and it assumes the webhook is deterministic — if it returns a slightly different mutation each call you would trade a false diff for a flapping one. So the honest rule is: ServerSideDiff=true alone kills the server-defaulting class of false diffs; add IncludeMutationWebhook=true when a webhook-injected sidecar or field still shows. That is the missing half of the “reach for Server-Side Diff first” advice.

ignoreResourceUpdates: the reconcile-path cousin (a scale/performance control)

ignoreDifferences changes the diff. Its easily-confused sibling ignoreResourceUpdates changes when the controller wakes up at all — a performance control, not a correctness one, and one that matters enormously on large fleets.

Here is the problem it solves. The application-controller watches every managed resource. Each time the live object changes — including changes to noisy, meaningless fields like status, metadata.generation, resourceVersion, or an operator’s “last-reconciled” annotation — the controller is nudged to re-run its reconcile. On a cluster with thousands of objects churning their status every few seconds, that is a firehose of wake-ups that produce no diff change yet still burn CPU.

resource.customizations.ignoreResourceUpdates.<group>_<kind> (and .all) tells the controller: a live update that only touches these fields is not worth reconciling for. It uses the same jsonPointers / jqPathExpressions shape as ignoreDifferences, but it feeds the watch/reconcile path, not the diff:

data:
  # don't wake the controller just because a Rollout rewrote its status
  resource.customizations.ignoreResourceUpdates.argoproj.io_Rollout: |
    jsonPointers:
      - /status
  resource.ignoreResourceUpdatesEnabled: "true"

Argo ships sensible built-in ignoreResourceUpdates defaults for well-known noisy types, gated by the master switch resource.ignoreResourceUpdatesEnabled. Keep the mental split clean: ignoreDifferences → does this field count as OutOfSync? ignoreResourceUpdates → is this live change even worth a reconcile? The scaling & sharding lesson treats this as a first-class knob for large installations.

Normal refresh vs hard refresh, and the diff cache

argocd app diff and the UI both read from a cache: the application-controller stores the repo-server’s rendered manifests keyed by the Git commit SHA and normally re-compares live against that cached render. Two refresh levels drive it:

The distinction matters because a normal refresh will happily keep serving a stale render. If you changed a Helm subchart pulled at build time, or a config-management plugin’s output depends on something outside the tracked path, the commit SHA is unchanged and only a hard refresh re-runs the templating. A surprising share of “I fixed it in Git but Argo won’t see it” tickets are just a missing hard refresh.

How the diff merges lists — and why index pointers are fragile

Argo’s compare is a structured merge, the same family as kubectl apply and Server-Side Apply — not a dumb line diff. For built-in types it knows the merge keys: containers merge by name, ports by containerPort+protocol, volumes by name. That is why a jqPathExpressions selector like containers[] | select(.name == "istio-proxy") is robust — it matches the element, wherever it sits in the list.

The fragility appears with jsonPointers by ordinal index (/spec/template/spec/containers/0/...). Index 0 is only “the app container” until a webhook injects a sidecar at position 0 and shifts yours to 1 — now your pointer silently ignores the wrong container. For CRDs, whether a list merges by key at all depends on the CRD’s OpenAPI schema (x-kubernetes-list-type: map with x-kubernetes-list-map-keys); a list without that marker is compared atomically — the whole list is replaced, so any per-element change flips the app OutOfSync and there is no stable index to pin. The rule that falls out: select list elements by name (jqPathExpressions), never by index, whenever another controller can add or reorder elements.

---

Hands-on lab

You will reproduce the perpetual-OutOfSync flap with a real HPA, then fix it three different ways and watch the app go stably Synced each time. This runs on a free local kind cluster — nothing here bills. Every command is copy-pasteable; the outputs shown are representative (labelled as such), not captured from a live run.

Prerequisites: kind, kubectl, argocd CLI, and a Git repo you can push to (any host). Replace https://github.com/YOU/diff-demo.git with yours throughout.

Step 1 — Cluster and Argo CD.

kind create cluster --name diff-lab
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server --timeout=180s
# log in (port-forward in another terminal: kubectl -n argocd port-forward svc/argocd-server 8080:443)
argocd login localhost:8080 --username admin \
  --password "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)" \
  --insecure

What just happened: a throwaway cluster with Argo CD installed and the CLI logged in.

Step 2 — metrics-server (so the HPA can actually scale).

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# kind's kubelet uses a self-signed cert; let metrics-server accept it (LAB ONLY — never in prod)
kubectl -n kube-system patch deploy metrics-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deploy/metrics-server --timeout=120s

⚠️ --kubelet-insecure-tls disables kubelet certificate verification. It is fine for a local kind lab and never acceptable on a real cluster. If you’d rather skip metrics-server entirely, you can simulate the HPA in Step 5 with a manual kubectl scale — the diff Argo sees is byte-for-byte identical.

Step 3 — commit the desired state (replicas: 2). In your repo, create manifests/deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: diff-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.k8s.io/hpa-example   # burns CPU on demand
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 100m
            limits:
              cpu: 200m

Commit and push. Then create the Argo app (manual sync policy for now, so we can watch each step deliberately):

argocd app create diff-demo \
  --repo https://github.com/YOU/diff-demo.git \
  --path manifests --revision main \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace diff-demo \
  --sync-option CreateNamespace=true
argocd app sync diff-demo
argocd app get diff-demo
# representative: Sync Status: Synced   Health Status: Healthy

What just happened: the app is Synced/Healthy with two replicas — the clean starting point.

Step 4 — add the HPA and create the drift. Apply an HPA directly (it’s the “other controller” the platform owns, not part of your app’s Git):

kubectl apply -f - <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: diff-demo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
EOF

# drive CPU up so the HPA scales toward 5 (Ctrl-C after ~1 min; or just: kubectl -n diff-demo scale deploy/web --replicas=5)
kubectl -n diff-demo run load --image=busybox --restart=Never -- \
  /bin/sh -c "while true; do wget -q -O- http://web.diff-demo; done"

kubectl -n diff-demo get hpa web        # REPLICAS eventually shows 5
argocd app get diff-demo
# representative: Sync Status: OutOfSync   Health Status: Healthy
argocd app diff diff-demo

# representative
===== apps/Deployment diff-demo/web ======
5c5
<   replicas: 5      # live: the HPA scaled it up
---
>   replicas: 2      # desired: what Git says

What just happened: the HPA now owns replicas, so live (5) and Git (2) permanently disagree. The app is OutOfSync forever — and note it’s still Healthy, because the pods are fine; this is purely a diff problem. If you now enabled selfHeal, you’d watch it flap: Argo sets 2, the HPA scales back to 5, repeat.

Step 5 — Fix #1: ignoreDifferences with jsonPointers.

argocd app set diff-demo \
  --ignore-differences-jsonpointers /spec/replicas \
  --ignore-differences-group apps \
  --ignore-differences-kind Deployment
# (equivalently, add the ignoreDifferences block to the Application YAML and commit it)
argocd app get diff-demo
# representative: Sync Status: Synced   Health Status: Healthy   (stable — no flap)
argocd app diff diff-demo    # exits 0, no output

What just happened: the diff now drops /spec/replicas, so the app is stably Synced. Remember the nuance: the desired manifest still says replicas: 2, so a future sync would momentarily write 2 unless you also add RespectIgnoreDifferences=true.

Step 6 — Fix #2: managedFieldsManagers (the self-maintaining fix). First confirm who owns the field, then ignore that manager:

kubectl -n diff-demo get deploy web --show-managed-fields -o yaml | grep -A6 managedFields
# representative: manager: kube-controller-manager, subresource: scale, f:replicas: {}

# swap the field-level ignore for a manager-level one (edit the Application):
#   ignoreDifferences:
#     - group: apps
#       kind: Deployment
#       name: web
#       managedFieldsManagers: [kube-controller-manager]
argocd app get diff-demo
# representative: Sync Status: Synced   Health Status: Healthy

What just happened: instead of naming the field, you named the controller. If the HPA controller starts writing another field tomorrow, this rule already covers it — no maintenance.

Step 7 — Fix #3: ServerSideDiff=true.

argocd app set diff-demo \
  --sync-option ServerSideApply=true \
  --sync-option ServerSideDiff=true
argocd app diff diff-demo --server-side-generate
argocd app get diff-demo
# representative: Sync Status: Synced   Health Status: Healthy

What just happened: Argo now asks the API server to dry-run apply your manifest and diffs the result against live. For the HPA specifically you’ll typically still keep the managedFieldsManagers rule (the scale subresource is special), but Server-Side Diff is what you’d rely on for the webhook and defaulting cases — turn it on and a whole class of false diffs from injected sidecars and server defaults disappears with no per-field rules at all.

Step 8 — compare, then tear down.

kubectl -n diff-demo delete pod load --ignore-not-found
argocd app delete diff-demo --yes
kind delete cluster --name diff-lab

What just happened: full teardown, zero lingering cost. You’ve now reproduced the most common Argo CD diff problem and fixed it three ways — and you can articulate the trade-offs between them.

---

Common mistakes and troubleshooting

Three gotchas cost the most hours:

1. A wrong jsonPointers path fails silently. This is the cruelest one, because there’s no error — Argo simply finds nothing at that path and the diff is unchanged, so it looks like ignoreDifferences doesn’t work. Ninety percent of the time the path is misspelled, points at the wrong nesting level, or uses an array index that shifted. Always confirm the field name against argocd app diff output first, mind the RFC 6901 escaping (~1 for /), and remember that array elements are addressed by ordinal — which is exactly why jqPathExpressions (select-by-name) is safer for lists.

2. Ignoring a real drift you needed to see. ignoreDifferences is a loaded footgun when scoped too broadly. Ignoring all of /spec on a Deployment to silence the HPA also hides an accidental image downgrade, a removed env var, a broken resource limit — genuine drift GitOps exists to catch. Scope every rule to the narrowest thing that fixes the false positive: exact group/kind/name, and the single field (or single manager), never a whole subtree. If you find yourself ignoring big chunks of an object, the right tool is Server-Side Diff, not a wider ignore.

3. selfHeal: false as a “fix.” When an app flaps, the tempting one-liner is to disable self-heal. Don’t. That doesn’t fix the false diff — the app is still OutOfSync — and it disables drift correction for every field on the app, so a real kubectl edit in prod now goes uncorrected and unnoticed. The flap is a signal that the diff is wrong about ownership; fix the diff (Server-Side Diff, managed fields, or a scoped pointer) and leave self-heal doing its job on everything else.

---

Common beginner mistakes

These are the mental-model errors behind the tickets — distinct from the symptom→cause→fix table above. Each is a belief that feels right and quietly leads you wrong.

• “OutOfSync means the app is broken.” No — OutOfSync is a statement about Git vs live, nothing more. A perfectly healthy app serving traffic can be OutOfSync (an HPA scaled it); a completely broken app can be Synced (it matches Git, and Git is wrong). Sync and Health are independent axes — never read one as the other.

• “ignoreDifferences stops Argo from changing the field.” It does not. It only removes the field from the diff, so the app reports Synced — but the desired manifest still contains the field, and the next sync will write it. If the sync itself must leave the field alone, you need RespectIgnoreDifferences=true. Ignoring in the diff and ignoring in the sync are two different switches.

• “The app is flapping, so I’ll turn off self-heal.” This treats the symptom and creates a bigger problem. selfHeal: false does not make the app Synced — it is still OutOfSync — and it disables drift correction for every field, so a real out-of-band kubectl edit in production now goes uncorrected and unnoticed. Fix the diff’s notion of ownership; leave self-heal protecting everything else.

• “I added a jsonPointers rule and nothing happened, so the feature is broken.” Far more likely: the pointer matches no field and silently no-ops. A misspelling, the wrong nesting level, a shifted array index, or missing RFC 6901 escaping (~1 for /) all fail without an error. Confirm the exact field against argocd app diff first, and prefer jqPathExpressions for list elements.

• “Server-Side Diff handles everything a webhook does.” Almost — but a plain ServerSideDiff=true accounts for server defaulting and the SSA merge, not mutating admission webhooks. A webhook-injected sidecar can still show as drift until you add IncludeMutationWebhook=true. Knowing that one caveat saves an afternoon.

• “The diff runs against my values.yaml / kustomize overlay.” It runs against the rendered output — helm template / kustomize build — not your inputs. A change that does not alter the rendered manifest (a comment, a reordered value) will not flip you OutOfSync, and a change that does alter it will, even if the input edit looked cosmetic.

• “Ignoring all of /spec is fine, it’s just the HPA.” An ignore rule is a loaded footgun when scoped too broadly. Silence /spec and you also hide an accidental image downgrade, a deleted env var, a broken resource limit — exactly the drift GitOps exists to catch. Scope to the narrowest field or manager; if the list keeps growing, that is the signal to switch to Server-Side Diff, not to widen the ignore.

---

Practice challenges

Six exercises, easy to hard. Try each before opening the solution. All manifests are schema-correct for Argo CD 2.13+/3.x; use placeholders (123456789012, https://kubernetes.example.com) for anything environment-specific.

1. (Beginner) Read the diff. Given this output, which line is the live cluster and which is Git, and which field is drifting?

===== apps/Deployment web/web ======
5c5
<   replicas: 4
---
>   replicas: 2

<details> <summary>Solution</summary>

The < line is live (replicas: 4, what the cluster has now) and the > line is desired from Git (replicas: 2). The drifting field is /spec/replicas. Why: argocd app diff always prints live on the < side and desired on the > side — memorising that direction is half of every diff diagnosis. </details>

2. (Beginner) Smallest fix for an HPA flap. An HPA owns replicas on a Deployment named api in namespace shop. Write the minimal ignoreDifferences entry that stops the OutOfSync.

<details> <summary>Solution</summary>

  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: api
      namespace: shop
      jsonPointers:
        - /spec/replicas

Why: scope to the one Deployment and the one field the HPA owns — the narrowest rule that fixes the false positive without blinding you to real drift on any other field. </details>

3. (Intermediate) Ignore an injected sidecar by name. Linkerd injects a linkerd-proxy container and a linkerd-init init container. Write a filter that ignores both regardless of their position in the list.

<details> <summary>Solution</summary>

  ignoreDifferences:
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - '.spec.template.spec.containers[] | select(.name == "linkerd-proxy")'
        - '.spec.template.spec.initContainers[] | select(.name == "linkerd-init")'

Why: jqPathExpressions selects the element by name, so it still matches after the injector shifts list indices — an ordinal jsonPointers path would silently target the wrong container. </details>

4. (Intermediate) The sync still stomps the field. With jsonPointers: [/spec/replicas] the app is Synced, but every image-tag sync briefly resets replicas to the Git value before the HPA scales back. Fix it so the sync leaves the field alone.

<details> <summary>Solution</summary>

  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true
    automated:
      selfHeal: true
      prune: true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: api
      jsonPointers:
        - /spec/replicas

Why: ignoreDifferences only affects the diff; RespectIgnoreDifferences=true extends the ignore-set to the write. Prefer jsonPointers/managedFieldsManagers here — jq support for RespectIgnoreDifferences has historically been limited. </details>

5. (Advanced) Fix it fleet-wide, two ways. Every Deployment across 200 apps is OutOfSync because a defaulting webhook stamps fields you never wrote. Give (a) a global ignoreDifferences and (b) the one-flag alternative.

<details> <summary>Solution</summary>

(a) Global rule in argocd-cm:

data:
  resource.customizations.ignoreDifferences.apps_Deployment: |
    managedFieldsManagers:
      - my-defaulting-webhook

(b) The better default — enable Server-Side Diff fleet-wide in argocd-cmd-params-cm:

data:
  controller.diff.server.side: "true"

(then restart argocd-application-controller). Why: server-side defaulting is exactly what SSD’s dry-run reproduces, so the injected defaults appear on both sides with no per-field list to maintain — reach for it before enumerating managers. </details>

6. (Advanced) SSD is on but the sidecar still shows. ServerSideDiff=true is enabled, yet a Deployment behind a mutating webhook that injects an istio-proxy sidecar is still OutOfSync. Explain why, fix it, then name the command that proves which manager owns the injected fields.

<details> <summary>Solution</summary>

Plain Server-Side Diff does not run mutating admission webhooks in its dry-run, so the predicted object lacks the injected sidecar. Turn them on:

metadata:
  annotations:
    argocd.argoproj.io/compare-options: IncludeMutationWebhook=true

Confirm ownership with:

kubectl -n shop get deploy api --show-managed-fields -o yaml | grep -A6 managedFields

Why: IncludeMutationWebhook=true folds the webhook’s mutation into the dry-run so the sidecar matches on both sides; --show-managed-fields reveals the field-manager if you would rather ignore it by manager instead. </details>

---

Cheat-sheet

ignoreDifferences shapes (per-Application spec.ignoreDifferences[]):

Which selector to pick:

Server-Side Diff & options:

Global normalization in argocd-cm:

Debugging commands:

---

Interview and exam questions

Q: An app is stuck OutOfSync and re-syncing doesn’t help. Walk me through your diagnosis. A: Run argocd app diff <app> to see the exact field(s) that differ. If a field is present in live but not desired, some controller owns it. Identify the actor (HPA → /spec/replicas; mesh webhook → a sidecar container; Autopilot → resources) and confirm the owner with kubectl get ... --show-managed-fields. Then apply the narrowest fix: Server-Side Diff for webhooks/defaults, managedFieldsManagers for a whole controller, or a scoped jsonPointers for one field. Never selfHeal: false.

Q: Why does an HPA cause perpetual OutOfSync, and what exactly happens if self-heal is on? A: The HPA writes spec.replicas on the Deployment, but Git still says the original count, so live and desired permanently disagree on that field — OutOfSync forever. With self-heal on, Argo reverts replicas to the Git value, the HPA immediately re-scales to its computed value, and the two controllers flap every reconcile interval. The fix is to remove replicas from the diff (ignore the field or the kube-controller-manager field-manager).

Q: What’s the difference between jsonPointers, jqPathExpressions, and managedFieldsManagers? A: jsonPointers selects an exact field by RFC 6901 path — best for a single stable field like /spec/replicas. jqPathExpressions selects by a jq query, so you can pick a list element by value (the container named istio-proxy) rather than by fragile index. managedFieldsManagers ignores every field owned by a named field-manager — the SSA-era approach, best when a whole controller’s writes should be ignored and self-maintaining as that controller evolves.

Q: Does ignoreDifferences stop a sync from writing the ignored field? A: No. By default ignoreDifferences only affects the diff (whether the app is OutOfSync). The desired manifest still contains the field, so a sync will write it. To make the sync also leave the field alone, add the RespectIgnoreDifferences=true sync option — and prefer jsonPointers/managedFieldsManagers, since jq support for it has been limited.

Q: What is Server-Side Diff and why is it usually the better first move than ignoreDifferences? A: With ServerSideDiff=true, Argo asks the API server to perform a dry-run Server-Side Apply of the desired manifest and diffs the result against live. Because the dry-run runs defaulting and mutating webhooks and respects other field-managers, injected sidecars and server defaults appear on both sides and simply match — removing whole categories of false diffs with one flag and no per-field list to maintain. You reach for it first because it’s self-maintaining; ignoreDifferences is the surgical fallback for what SSD doesn’t cover (notably HPA replicas).

Q: You add ignoreDifferences on /spec/replicas but the diff is unchanged. What went wrong? A: Almost certainly the JSON Pointer doesn’t match — a typo, the wrong nesting, or a bad array index. A wrong pointer fails silently (it matches no field). Confirm the field name against argocd app diff, check RFC 6901 escaping (~1 = /), and if it’s a list element selected by name, switch to jqPathExpressions.

Q: How do you fix the same false diff across 200 apps without editing each Application? A: Put a global rule in argocd-cm under resource.customizations.ignoreDifferences.<group>_<kind> (e.g. apps_Deployment) — same shape as the per-app list. Or enable Server-Side Diff fleet-wide via controller.diff.server.side: "true" in argocd-cmd-params-cm. Both avoid per-app duplication.

Q: A CRD is always OutOfSync because 500m in Git shows as 0.5 in live. Fix it. A: Argo doesn’t know the field is a resource quantity, so it compares the strings. Declare the field’s known type via resource.customizations.knownTypeFields.<group>_<kind> in argocd-cm (type core/v1/ResourceList), and the normalizer will parse both as quantities and compare them equal.

Q: Why is selfHeal: false the wrong way to stop a flap? A: It doesn’t fix the diff — the app is still OutOfSync — and it disables drift correction for the entire app, so a real out-of-band change in prod goes silently uncorrected. The flap means the diff is wrong about which fields Argo owns; fix the diff (Server-Side Diff / managed fields / scoped pointer) and keep self-heal protecting every other field.

Q: How do you find which field-manager owns a drifting field? A: kubectl get <kind> <name> --show-managed-fields -o yaml and read metadata.managedFields. Each entry lists the manager, the operation, and (for the HPA) subresource: scale with f:spec: f:replicas: {}. The manager name is exactly what you put in managedFieldsManagers.

Q: On EKS, a Deployment with an IRSA/Pod-Identity service account is OutOfSync on env vars you never set. Why, and the cleanest fix? A: The EKS pod-identity-webhook mutating admission webhook injects AWS_* env vars and a projected token volume into pods using that service account. Those live fields aren’t in Git, so the naive diff flags them. Cleanest fix: ServerSideDiff=true, so the dry-run runs the same webhook and the injected fields match on both sides. (The AKS Workload Identity webhook and GKE Autopilot resource-defaulting cause the identical class of problem, fixed the same way.)

---

Glossary

---

Key takeaways

• The diff compares rendered desired (your Git fields only) against normalized live (your fields plus server defaults, other controllers’ fields, and metadata). False positives come from live-only fields Argo can’t guess it should ignore.
• Perpetual OutOfSync means another actor owns a field you never set — an HPA on replicas, a webhook injecting a sidecar, a defaulting controller, cert-manager. Identify the field and the actor with argocd app diff and --show-managed-fields.
• ignoreDifferences affects the diff, not the sync. Add RespectIgnoreDifferences=true when the sync itself must leave a field alone — and it’s separate from prune and selfHeal.
• Pick the selector to fit the problem: jsonPointers for one exact field, jqPathExpressions to select a list element by name, managedFieldsManagers to ignore a whole controller’s fields (self-maintaining).
• Reach for Server-Side Diff first. ServerSideDiff=true makes the API server account for defaults and webhooks via a dry-run apply, erasing whole categories of ignoreDifferences hacks — especially injected sidecars, server defaults, and GKE Autopilot resource mutation.
• The cloud edge is the set of mutating webhooks, not the diff itself: AKS Workload Identity, EKS Pod Identity/IRSA, and GKE Autopilot each inject fields — and Server-Side Diff handles all three the same way.
• Never use selfHeal: false to silence a flap. It hides all drift; it doesn’t fix the ownership boundary. Fix the diff and keep self-heal working on every other field.