Sync Status & Health Assessment: Synced/OutOfSync, Healthy/Degraded/Progressing

In a nutshell

Every Argo CD application carries two badges, and telling them apart is the single most useful skill in the whole tool. Sync status answers one question — does the live cluster match what Git says? Health status answers a completely different one — do the live resources actually work? They move independently, so an app can be “Synced but Degraded” (Argo CD deployed your manifest perfectly and the app still crash-loops) or “OutOfSync but Healthy” (the app runs fine, but someone hand-edited the cluster).

Think of a building inspector running two separate checks. First: does the finished building match the approved blueprints? — that is sync. Second: do the water and power actually come on when you flip the switch? — that is health. A house can be built exactly to a bad blueprint (matches the plan, nothing turns on), or work perfectly after an unpermitted extra room (differs from the plan, still livable). One check is about the paperwork; the other is about reality. Read them as one signal and you will spend hours fixing the wrong half.

Level: Beginner · Time: ~34 min

You should already be comfortable with: the idea of a Git-defined desired state that Argo CD pulls and reconciles, and roughly what the repo-server and application-controller each do. After this lesson you will be able to: read both badges in argocd app get, explain all four Synced/OutOfSync × Healthy/Degraded combinations, say why a Synced app can be Degraded, spot the “invisible CRD” health blind spot, and add a Lua health check so Argo CD understands your own custom resources.

---

Open the Argo CD UI for any application and you see two badges side by side: one says something like Synced or OutOfSync, the other says Healthy, Progressing, or Degraded. Almost every beginner reads them as one blurry signal — “green good, yellow bad” — and that single misconception is the root of more wasted Argo CD debugging than anything else in the tool.

They are two completely independent questions. One is about Git. The other is about your workload. You can be Synced and Degraded (Argo CD deployed exactly what you asked, and your app is crash-looping). You can be OutOfSync and Healthy (the app is running fine, but someone hand-edited the cluster). Reading one as if it were the other sends you debugging the wrong half of the system every time.

This lesson pulls those two axes apart and keeps them apart. Once you can look at any app and instantly say “sync is the diff against Git, health is whether the resources work,” the rest of Argo CD — and every troubleshooting lesson after this one — falls into place. It is the single most important conceptual distinction in the whole tool, so we go slowly and prove every claim on a real cluster in the lab.

---

Why this matters

Argo CD is a reconciliation engine. It takes a desired state you wrote in Git, looks at the live state in a Kubernetes cluster, and works to make them equal. To do that job it has to continuously answer one obvious question: are they equal right now? That question — and only that question — is what the sync status reports. Synced means the live cluster matches what Git says; OutOfSync means it doesn’t. Nothing more.

But “the cluster matches Git” is not the same as “the application works.” Git can contain a manifest that references an image tag that doesn’t exist. Argo CD will faithfully apply it — cluster now matches Git, so it reports Synced — and the pods will sit in ImagePullBackOff forever. To catch that, Argo CD asks a second, separate question: are the live resources actually healthy? That is the health status, and it is computed by inspecting each resource’s own live state, not by comparing anything to Git.

Here is where the confusion costs real hours. A beginner sees a red or yellow badge, assumes “Argo CD is broken,” and starts poking at repository credentials and sync policies — when the actual problem is a typo in their own image tag that has nothing to do with Argo CD. Or the reverse: they see Synced in big green letters, declare victory, and never notice the app has been Degraded the whole time. The fix in both cases is to read the two axes separately, which is exactly what everything below teaches.

Two axes. Say it out loud before every debugging session: sync = does live match Git? · health = do the live resources work? If you can name which axis is red before you touch anything, you have already cut the problem in half.

---

Two axes, not one: the mental model

Picture a 2×2 grid. Along one axis runs sync status — a comparison between the desired state rendered from Git and the observed state in the cluster. Along the other axis runs health status — an assessment of whether the live resources are functioning. They are orthogonal: knowing the value of one tells you nothing about the value of the other. Every one of the four corners is a real, common situation you will meet, and each one demands a different response from you.

The first table to burn into memory is simply the contrast between the two axes:

The crucial reading of that last row: when health is bad, Argo CD is usually working perfectly. Its job on the health axis is to report, honestly, that your resources are unhappy. It will not rewrite your Deployment to make it healthy — that’s your code, your config, your problem. On the sync axis it can act (re-applying Git), but on the health axis it is a thermometer, not a doctor.

Now the four combinations — the teaching payload of the entire lesson:

Read the second row again, because it is the one that saves careers: Synced + Degraded means the problem is yours, not Argo CD’s. Argo CD did precisely what you told it — made the cluster match Git. If the result is a broken app, the bug is in what you committed, and the answer is in kubectl logs, not in your repo credentials.

Here is the whole model as one picture. Read it left to right: Git (desired) and the cluster (live) are the two operands; the application-controller compares them to emit the sync verdict and, separately, runs per-kind health checks on the live resources to emit the health verdict; both roll up into the single Application status where the four combinations live.

The badges mark the ideas worth tattooing on your brain: the diff is desired-vs-live (1); health is a per-kind assessment of the live resource (2); OutOfSync is drift, not failure (3); Degraded is your app’s bug (4); Synced+Healthy is the only rest state you want (5); and one bad child drags the whole app’s rollup down while a resource with no health check is invisible to it (6). If you understand only this diagram from the lesson, you already read Argo CD better than most people who have run it for a year.

This mental model builds directly on the GitOps principles of a declarative desired state that Argo CD continuously pulls and reconciles, and on the Argo CD architecture where the repo-server renders manifests and the application-controller runs the reconcile loop. If those two ideas are fuzzy, skim them first — the two axes are just the two outputs of that loop.

---

Sync status: does live match Git?

Sync status is the simpler axis, so we start here. It has exactly three values:

How the diff is actually computed

OutOfSync is not a vague feeling — it is the result of a precise, repeatable pipeline. Understanding the steps tells you exactly where to look when the diff surprises you.

Two things follow immediately. First, the diff is computed against rendered manifests, not your raw Git files. If you use Helm or Kustomize, Argo CD compares the output of the template engine to the cluster — so a values change that alters the rendered output flips you to OutOfSync, while a comment in a chart does not. Second, the comparison is between the desired state and the live state, so it catches drift in both directions: a change you pushed to Git that hasn’t synced yet, and a change someone made to the live cluster that isn’t in Git.

You can see the exact diff without changing anything:

# Show precisely which fields differ between Git and live (no changes made)
argocd app diff my-app

# (representative output)
===== apps/Deployment my-app/web ======
23c23
<     image: registry.example.com/web:1.4.2   # desired (Git)
---
>     image: registry.example.com/web:1.4.1   # live (cluster)

That is the whole meaning of OutOfSync made concrete: one field, two values, Git on top, cluster below.

Normalization: why “identical” YAML can still be OutOfSync (and vice versa)

Kubernetes rewrites objects the moment you submit them. It adds defaults (imagePullPolicy, terminationGracePeriodSeconds), stamps metadata (creationTimestamp, resourceVersion, uid), and controllers mutate specs (an HPA rewrites replicas, a webhook injects a sidecar, a CA injects a bundle). If Argo CD compared raw, it would scream OutOfSync over fields you never wrote. Normalization is the step that strips this noise so the diff reflects your intent.

When the diff surprises you, it is nearly always a normalization gap: a controller keeps rewriting a field Argo CD isn’t told to ignore, so the app is perpetually OutOfSync no matter how many times you sync. The fix is a scoped ignoreDifferences for that exact field — covered in depth in a later lesson on diffing and drift; for now, just recognize the pattern: sync succeeds, then instantly flips back to OutOfSync = a controller is mutating a field you need to ignore.

Unknown sync status and ComparisonError

If Argo CD cannot even render or reach your source — the repo is down, the revision doesn’t exist, the Helm chart has a template error, the path is wrong — it can’t compute a diff, so the sync status is Unknown and the app carries a ComparisonError condition with the reason:

argocd app get my-app

# (representative output)
Sync Status:   Unknown
Conditions:    ComparisonError: rpc error: code = Unknown desc = failed to
               generate manifests: helm template . failed: <chart error>

Unknown on the sync axis is never “the app is broken” — it is “Argo CD couldn’t do the comparison.” Read the ComparisonError message; it points at the repo, the revision, or the render, essentially never at your running pods.

---

Health status: are the resources actually working?

Now the second axis. Health status has six values, and each is computed by looking at the resource’s own live state — never at Git.

Health is computed by per-resource-kind checks

The key insight: health is not one algorithm — it is a different check per resource kind. “Healthy” means something specific to each kind, and Argo CD ships built-in checks (native Go for core workloads, bundled Lua scripts for many popular CRDs) that encode exactly what “working” means for that kind.

Two rows in that table earn special attention because they are the most common source of “why is my sync stuck?”: Service and Ingress of load-balancer type report Healthy only once the cloud has assigned an external address. Until then they are Progressing — which is completely normal for the first minute or two after creation, and is a genuine cloud edge we cover below.

The progressing deadline: when Progressing becomes Degraded

Progressing should be temporary. What stops it from being temporary forever is the workload’s own deadline. A Deployment has spec.progressDeadlineSeconds (default 600s). If it can’t make progress within that window — pods never become ready because the image is bad or the probe never passes — Kubernetes sets the Deployment’s Progressing condition to reason ProgressDeadlineExceeded, and Argo CD’s Deployment health check reads that and reports Degraded.

So the lifecycle of a bad rollout is: Progressing (pods trying to come up) → after progressDeadlineSeconds → Degraded (Kubernetes gave up). This matters for sync ordering: Argo CD’s sync waves apply resources in numbered groups and wait for each wave to become Healthy before starting the next. A resource stuck Progressing past its deadline flips to Degraded and stalls the sync — which is exactly the behavior you want (don’t run the migration Job if the database never came up). Sync waves and hooks get their own later lesson; the takeaway here is that health directly gates how a sync proceeds, so a health problem can masquerade as a “stuck sync.”

Missing and Suspended: the two states beginners misread

Missing is not the same as Degraded. Missing means a resource is declared in Git but does not exist live — Argo CD expected to find it and didn’t. You see it in the seconds after adding a new resource (before it’s created), or when a resource failed to apply, or when it’s queued in a later sync wave that hasn’t run yet. It is often transient; if it persists, something is blocking creation (RBAC, a failed admission webhook, a namespace that doesn’t exist).

Suspended is not an error at all — it is Argo CD correctly reporting that you intentionally paused something. A CronJob with spec.suspend: true and a deliberately paused Argo Rollout both report Suspended. Newcomers see the non-green badge and panic; the right reaction is “yes, I paused that on purpose.”

---

The app-level rollup: from many resources to one status

An Application usually manages many resources — a Deployment, a Service, an Ingress, a ConfigMap, maybe a Job. Yet the app shows a single sync status and a single health status. How do dozens of resource statuses collapse into one? Two different rollup rules, one per axis.

Sync rollup is simple: if every managed resource is Synced, the app is Synced; if any is OutOfSync, the app is OutOfSync. One drifted ConfigMap makes the whole app OutOfSync.

Health rollup is “worst status wins.” Argo CD ranks the health states from best to worst and reports the worst one present among the resources:

This is why one bad child sinks the whole app: a single Degraded pod turns the Application Degraded even if forty other resources are perfectly Healthy. When an app shows Degraded, the app-level badge doesn’t tell you which resource — you have to open the resource tree (next section) to find the one dragging it down.

The most dangerous rollup subtlety: a resource with no health check is skipped entirely. If Argo CD has no way to assess a kind (no built-in check, no Lua script), it returns no health opinion for that resource and excludes it from the rollup — it is not counted as Healthy, Degraded, or anything. The app can therefore read Healthy while a critical custom resource is completely broken, because that resource is simply invisible to the health calculation. This is not a bug; it’s the reason the next section exists.

Reconnecting to the four combinations: the rollup is exactly what produces them. Synced + Degraded means all resources match Git (sync rollup all-Synced) but one is failing (health rollup worst-wins landed on Degraded). OutOfSync + Healthy means everything’s working (health all-Healthy) but one resource drifted from Git (sync rollup any-OutOfSync). The two rollups run independently — which is the orthogonality, made mechanical.

---

Custom health checks: teaching Argo CD about your CRDs

Argo CD knows how to assess a Deployment because someone wrote that check into it. For your CustomResourceDefinition — a Widget, a Database, a vendor operator’s CRD — Argo CD has no idea what “working” means unless you tell it. Out of the box such a resource has no health opinion, so (per the rollup rule above) it’s invisible: the app can report Healthy while your Widget is on fire.

The fix is a custom health check: a small Lua script, configured in the argocd-cm ConfigMap, that reads the resource’s live .status and returns a verdict. Argo CD runs it exactly like a built-in check.

The configuration key follows a strict shape:

The Lua script receives the live object as obj and must return a table with a status string and a message:

Here is a complete, real example for a Widget CRD whose controller writes .status.phase:

# argocd-cm ConfigMap (namespace argocd) — health check for a custom resource
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  resource.customizations.health.demo.kloudvin.io_Widget: |
    hs = {}
    if obj.status ~= nil and obj.status.phase ~= nil then
      if obj.status.phase == "Ready" then
        hs.status = "Healthy"
        hs.message = "Widget is ready"
        return hs
      end
      if obj.status.phase == "Failed" then
        hs.status = "Degraded"
        hs.message = "Widget failed: " .. (obj.status.reason or "unknown")
        return hs
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for widget to become ready"
    return hs

Read the logic top to bottom, because it mirrors how every health check thinks: check for a definitive good state (Ready → Healthy), check for a definitive bad state (Failed → Degraded), and if neither, assume it’s still on its way (Progressing). Never default to Healthy when you can’t tell — default to Progressing, so an unfinished or stuck resource shows honestly instead of falsely green.

A few things that keep this correct in practice:

• The obj tree can be nil at any level. Always guard (obj.status ~= nil and obj.status.phase ~= nil) before indexing, or the script errors and the resource goes Unknown — which is worse than Progressing because Unknown is the worst rollup rank.
• Argo CD already bundles checks for many popular CRDs (cert-manager, Argo Rollouts, and more) in its resource_customizations directory, so you only write Lua for kinds it doesn’t already cover. Check before you write.
• After editing argocd-cm, Argo CD picks up the change on the next reconcile; a hard refresh (argocd app get <app> --hard-refresh) forces re-evaluation immediately.

We wire exactly this up, end to end, in the lab — including deliberately breaking the Lua to see the Unknown state.

---

Reading it all: argocd app get, the resource tree & UI colours

Everything above surfaces in one command. argocd app get prints a header (the app-level rollup) followed by the resource tree — every managed resource with its own per-resource SYNC and HEALTH columns. This is where you go from “the app is Degraded” to “this pod is the reason.”

# The single most useful diagnostic command in Argo CD
argocd app get my-app

# (representative output)
Name:               argocd/my-app
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          demo
Source:
- Repo:             https://github.com/acme/demo.git
  Target:           main
  Path:             manifests
Sync Policy:        Automated
Sync Status:        Synced to main (a1b2c3d)
Health Status:      Degraded

GROUP  KIND        NAMESPACE  NAME      STATUS   HEALTH      MESSAGE
       Service     demo       web       Synced   Healthy     service/web created
apps   Deployment  demo       web       Synced   Degraded    Deployment "web" exceeded its progress deadline
       Pod         demo       web-xyz   Synced   Degraded    Back-off pulling image "web:does-not-exist"

Read it as a drill-down. The header says the app is Synced / Degraded. The tree tells you why: the Service is fine, but the Deployment hit ProgressDeadlineExceeded and the Pod can’t pull its image, so the worst-child rollup landed on Degraded — and the MESSAGE column hands you the exact cause. Yet the whole app is Synced: Git and live match perfectly. This is the Synced + Degraded corner, and the tree proves the fix is your image tag, not Argo CD.

The header fields worth knowing:

The UI encodes the same two axes as two separate icons, and knowing the colours lets you triage a whole app grid at a glance:

Two icons, two axes — always. If the green check (sync) is present but the heart is red (health), that’s Synced+Degraded on sight.

The one genuine cloud edge: Service/Ingress health depends on your cloud’s load balancer

Sync and health computation are otherwise cloud-neutral — the diff and the health rules are identical whether the target is AKS, EKS, or GKE. There is exactly one place a cloud edge leaks into the health axis, and it’s worth covering across all three: a LoadBalancer Service or an Ingress is only Healthy once the cloud has assigned it an external address, and who assigns that address — and how long it takes — differs per cloud.

The practical consequence is identical on all three clouds: right after a sync, a load-balancer-backed Service or Ingress sits at Progressing for a few seconds to a couple of minutes while the cloud provisions the address. That is not a bug and not a sync failure — it’s the health check honestly reporting “no address yet.” What differs per cloud is the failure case: if it never resolves, the cause is a missing or misconfigured controller or its cloud identity — AGIC/ingress-nginx and Azure Workload Identity on AKS, the AWS Load Balancer Controller and its IRSA/Pod Identity on EKS, or the GKE Ingress controller and its IAM on GKE. Those install details live in the dedicated AKS/EKS/GKE lessons; here, just recognize the pattern: a Service or Ingress stuck Progressing long after a sync is a cloud load-balancer problem, not an Argo CD problem.

---

Going deeper

The two-axis model is the load-bearing idea, and for daily work it is enough. Three things sit just underneath it that separate reading Argo CD from operating it: a hidden third axis, the diff mode that decides what even counts as OutOfSync, and the internals of the health engine.

The hidden third axis: operationState

Sync status is a diff (“does live match Git?”). It says nothing about whether the last attempt to close that diff actually worked. That lives in a separate field — status.operationState — and beginners never notice it until a sync mysteriously “does nothing.”

operationState.phase records the outcome of the most recent sync operation:

Sync status and operationState can disagree. You click Sync, an admission webhook rejects one manifest, and the app sits at Sync Status: OutOfSync (the diff never closed) with Operation: Failed and a SyncError condition explaining why. Read only the sync badge and it looks like “nothing happened”; the truth is in the operation result. argocd app get prints it on the Operation: line and the reason lands in Conditions:. So the honest picture is three axes — the diff (sync status), the last apply’s result (operationState), and whether the resources work (health) — and a failed sync wave or hook surfaces here, not on the health badge.

What actually counts as OutOfSync: client-side vs server-side diff

The diff runs field-by-field over normalized manifests — but two engines can do that comparison, and which one you use changes how many false OutOfSyncs you see.

Server-side diff (available and stable in Argo CD 3.0) is the cure for the classic “perpetually OutOfSync over a field I never set.” Enable it cluster-wide with controller.diff.server.side: "true" in the argocd-cmd-params-cm ConfigMap, or per-app with an annotation:

# Enable server-side diff for a single Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true

This is not the same as ServerSideApply=true, a sync option that changes how manifests are applied (field ownership). One changes how the diff is computed; the other how the sync is performed. The full treatment of drift and ignoreDifferences is the diffing & drift lesson.

Inside the health engine: Go for core kinds, Lua for the rest

“Health is a different check per kind” has a concrete implementation. For a fixed set of core kinds (Deployment, StatefulSet, DaemonSet, ReplicaSet, Pod, Service, Ingress, PVC, Job, HPA, PodDisruptionBudget, and more) Argo CD runs a native Go check compiled into the controller. For every CRD it looks for a Lua script. Two facts follow that you can rely on:

• Precedence. A check you define in argocd-cm under resource.customizations.health.<group>_<kind> wins. If you define none, Argo CD falls back to a Lua script it bundles for many popular CRDs (cert-manager, Argo Rollouts, Sealed Secrets, and others); only if that is also absent does the resource get no opinion — the blind spot.
• Lua returns only four states. A health script may return Healthy, Progressing, Degraded, or Suspended. It can not return Missing or Unknown — those two are produced by the engine: Missing when a declared resource isn’t live yet, Unknown when a check ran and errored (a Lua script indexing a nil field). That is why guarding every obj.status access matters — an unguarded index doesn’t fall back to Progressing, it throws the resource to the worst rollup rank.

Unknown is a plumbing signal, not an app signal

Both axes have an Unknown, and both mean “Argo CD couldn’t answer,” never “your app is broken” — but the cause differs by axis:

The tell is blast radius: one resource Unknown on health is a Lua bug; every app on a cluster going Unknown on sync is infrastructure — the repo-server pod, or connectivity to that cluster’s API server. The dedicated OutOfSync/Degraded/Unknown troubleshooting lesson drills each case.

Gating CI on both axes: argocd app wait

Because sync, operation, and health are separate, a CI job that “waits for the deploy” must say which it means. argocd app wait blocks until the conditions you name are all true:

# Block until the app is BOTH reconciled to Git AND its resources are Healthy,
# failing the pipeline if it doesn't converge within 5 minutes.
argocd app wait my-app --sync --health --timeout 300

Companions worth knowing: --operation waits for an in-flight sync to finish, --suspended waits for a Suspended state, and --degraded returns when the app goes Degraded — a fast-fail for canary pipelines. Waiting on --sync alone is a classic mistake: it returns the instant the diff closes, long before pods are actually up, so a smoke test fired right after can hit a Progressing app. Pair --sync --health when “deployed” is supposed to mean “working.”

---

Hands-on lab

Time to watch both axes move independently on a real cluster. This lab assumes a working Argo CD on a local kind/minikube cluster (nothing here bills) and the small demo app from Your First Application. Every step shows the command, a representative result, and a one-line “what just happened.” We deliberately create each state — Synced/Healthy, then OutOfSync, then Degraded, then a CRD’s blind spot — and finish with teardown.

These outputs are representative shapes, not a transcript from your exact cluster — revisions, pod suffixes, and timings will differ. The states and commands are what matter.

Step 0 — Start from a known-good app. Point an Application at a simple manifest directory (a Deployment + Service) and sync it.

# Create and sync a minimal app (adjust repo/path to your own)
argocd app create demo \
  --repo https://github.com/acme/argocd-demo.git \
  --path manifests --dest-server https://kubernetes.default.svc \
  --dest-namespace demo --sync-policy manual
argocd app sync demo
argocd app get demo

# (representative)
Sync Status:    Synced to main (a1b2c3d)
Health Status:  Healthy

What just happened: the baseline. Live matches Git (Synced) and the Deployment’s replicas are available (Healthy). This is the corner you want at rest.

Step 1 — Create drift: OutOfSync while staying Healthy. Hand-edit the live cluster, bypassing Git entirely.

# Change a live label directly — the app keeps running fine
kubectl -n demo label deployment web team=rogue --overwrite
argocd app get demo

# (representative)
Sync Status:    OutOfSync from main (a1b2c3d)
Health Status:  Healthy

GROUP  KIND        NAMESPACE  NAME  STATUS      HEALTH
apps   Deployment  demo       web   OutOfSync   Healthy

What just happened: you changed live state that Git doesn’t know about, so the diff is non-empty → OutOfSync. But the pods are still running perfectly → Healthy. This is the OutOfSync + Healthy corner: drift, not failure. Run argocd app diff demo to see the exact label that differs. Re-sync (argocd app sync demo) to erase the drift and return to Synced.

Step 2 — Ship a broken image: Synced but Progressing → Degraded. Now push a bad change through Git (the GitOps way) and watch health fail while sync succeeds.

# In your Git repo: set the Deployment image to a tag that doesn't exist,
# e.g. image: web:does-not-exist  — commit and push, then:
argocd app sync demo
argocd app get demo

# (representative, moments after sync)
Sync Status:    Synced to main (e5f6g7h)
Health Status:  Progressing

# (representative, after progressDeadlineSeconds ~10 min, or immediately once pods back off)
Sync Status:    Synced to main (e5f6g7h)
Health Status:  Degraded

GROUP  KIND        NAMESPACE  NAME     STATUS   HEALTH      MESSAGE
apps   Deployment  demo       web      Synced   Degraded    exceeded its progress deadline
       Pod         demo       web-...  Synced   Degraded    Back-off pulling image "web:does-not-exist"

What just happened: Argo CD applied exactly what you committed, so Sync = Synced. But the new pods can’t pull the image, so they sit Progressing, then hit the progress deadline and flip to Degraded. This is the career-saving Synced + Degraded corner: Argo CD did its job; the bug is in what you committed. The MESSAGE column names it. Fix the tag in Git, commit, sync — back to Synced/Healthy.

Step 3 — A CRD with no health check: the invisible resource, then a Lua check. Add a custom resource Argo CD doesn’t understand and watch it contribute nothing to health, then teach Argo CD to assess it.

# Add to your Git repo: a tiny CRD (status subresource enabled) + one instance
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.demo.kloudvin.io
spec:
  group: demo.kloudvin.io
  names: { kind: Widget, listKind: WidgetList, plural: widgets, singular: widget }
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      subresources: { status: {} }
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:   { type: object, properties: { size: { type: string } } }
            status: { type: object, properties: { phase: { type: string }, reason: { type: string } } }
---
apiVersion: demo.kloudvin.io/v1
kind: Widget
metadata:
  name: my-widget
  namespace: demo
spec:
  size: small

argocd app sync demo && argocd app get demo

# (representative) — the Widget row has a blank HEALTH column
GROUP              KIND     NAMESPACE  NAME       STATUS   HEALTH
demo.kloudvin.io   Widget   demo       my-widget  Synced

Health Status:  Healthy        # <-- the app is "Healthy" despite the Widget being unassessed!

What just happened: the Widget is Synced (it exists as Git declares) but has no HEALTH value — Argo CD has no check for this kind, so it forms no opinion and excludes the Widget from the rollup. The app proudly reports Healthy even though you have no idea whether the Widget works. That is the blind spot.

Now add the Lua health check from the section above to argocd-cm and force a re-evaluation:

# Apply the argocd-cm with the Widget health check, then hard-refresh
kubectl apply -f argocd-cm.yaml
argocd app get demo --hard-refresh

# (representative) — the Widget now gets assessed; no controller set .status, so:
GROUP              KIND     NAMESPACE  NAME       STATUS   HEALTH        MESSAGE
demo.kloudvin.io   Widget   demo       my-widget  Synced   Progressing   Waiting for widget to become ready

Health Status:  Progressing    # <-- the Widget now participates in the rollup

What just happened: Argo CD now runs your Lua check. Because nothing has written .status.phase, the script returns Progressing — honest: “I can’t confirm it’s ready.” The Widget is finally visible to health. Simulate its controller by writing the status yourself:

# Pretend the Widget's operator marked it Ready
kubectl -n demo patch widget my-widget --type=merge --subresource=status \
  -p '{"status":{"phase":"Ready"}}'
argocd app get demo --hard-refresh

# (representative)
demo.kloudvin.io  Widget  demo  my-widget  Synced  Healthy   Widget is ready
Health Status:  Healthy

What just happened: the Lua saw phase: Ready and returned Healthy — now a real Healthy, one you can trust. Patch phase to Failed and re-refresh to watch it go Degraded; deliberately break the Lua (index a nil field) and re-refresh to see the genuine Unknown health state. You have now driven every value on both axes.

Teardown.

# Remove the app (and its resources), and revert argocd-cm
argocd app delete demo --cascade
kubectl -n argocd apply -f argocd-cm-original.yaml   # or edit out the Widget key

What just happened: --cascade deletes the managed resources (Deployment, Service, CRD, Widget) along with the Application; reverting argocd-cm removes the custom health check. Your cluster is back to a clean Argo CD.

---

Practice challenges

Work these top to bottom — they escalate from “read the badges” to “reason about three axes at once.” Try each before opening the solution.

1. (Beginner) Name the axis. An app shows Synced and Degraded. Which half of the system is broken, and what is the first command you run?

<details> <summary>Show solution</summary>

The health half — your workload — not Argo CD. Synced proves the cluster already matches Git, so the bug is in what you committed. Run argocd app get my-app, read down the resource tree to the Degraded child and its MESSAGE, then kubectl -n <ns> logs/describe that resource.

Why: Synced+Degraded always points at the manifest or image, never at repo credentials or the controller. </details>

2. (Beginner) Predict both badges. You run kubectl -n demo edit deployment web and change a label by hand. What do the two axes read, and how do you see exactly what differs?

<details> <summary>Show solution</summary>

OutOfSync (live now differs from Git) and Healthy (the pods never stopped running). See the exact field with argocd app diff demo. Re-sync to erase the drift, or commit the change to Git if you meant it.

Why: a hand-edit is a sync-axis event; it doesn’t touch health unless the resource’s actual condition changes. </details>

3. (Intermediate) Gate a pipeline correctly. After argocd app sync, write the one argocd app wait line a CI job should run so it blocks until the app is reconciled and its resources are up, failing after five minutes.

<details> <summary>Show solution</summary>

argocd app wait my-app --sync --health --timeout 300

Why: --sync alone returns the moment the diff closes — before pods are ready; adding --health makes “deployed” mean “working,” and --timeout fails the job instead of hanging. </details>

4. (Intermediate) Silence a false diff. An app is perpetually OutOfSync on spec.replicas because a HorizontalPodAutoscaler owns that field. Write the ignoreDifferences block for the Application.

<details> <summary>Show solution</summary>

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Why: the HPA rewrites replicas on the live object, so every sync instantly re-drifts; telling Argo CD not to diff that one field ends the loop. (Server-side diff is the other fix.) </details>

5. (Advanced) Teach Argo CD a CRD. A Database in group acme.example.com shows a blank HEALTH column. Give the argocd-cm key name and a minimal, nil-safe Lua check: Healthy when .status.phase == "Running", Degraded on "Error", otherwise Progressing.

<details> <summary>Show solution</summary>

Key: resource.customizations.health.acme.example.com_Database (group and kind joined by an underscore).

hs = {}
if obj.status ~= nil and obj.status.phase ~= nil then
  if obj.status.phase == "Running" then
    hs.status = "Healthy"; hs.message = "Database running"; return hs
  end
  if obj.status.phase == "Error" then
    hs.status = "Degraded"; hs.message = "Database error"; return hs
  end
end
hs.status = "Progressing"; hs.message = "Waiting for database"
return hs

Why: without a check the CRD is excluded from the rollup (the blind spot); the nil guard keeps a missing .status from throwing the resource to Unknown instead of an honest Progressing. </details>

6. (Advanced) Three axes at once. After a sync, argocd app get shows Sync Status: OutOfSync, Operation: Failed, and a SyncError about a rejected apply. Explain why the diff is still open, and where you look — not at health.

<details> <summary>Show solution</summary>

The operation failed — an admission webhook (or RBAC, or an invalid field) rejected the apply — so the manifests never landed, which is exactly why the diff is still OutOfSync. Health is irrelevant here: read the SyncError in Conditions: and the Operation: message, then the application-controller logs.

Why: sync status (the diff) and operationState (did the last apply succeed) are different axes; a failed apply leaves the diff open without ever becoming a health problem. </details>

---

Common mistakes and troubleshooting

Every row here traces back to conflating the two axes or misreading a rollup. Keep it close during real incidents.

Three gotchas deserve extra words because they cost the most hours:

1. “The badge is yellow/red, so Argo CD is broken.” Almost never. OutOfSync means you have drift or an un-synced commit; Degraded means your workload is failing. In both cases Argo CD is doing its job — detecting a diff, or honestly reporting an unhealthy resource. Before you touch the application-controller, ask which axis is unhappy and whether that axis is even about Argo CD: the sync axis can implicate it (a ComparisonError is Argo CD’s side), but a Degraded health almost never does.

2. The Synced+Degraded trap. The single most misdiagnosed state. Because the big green Synced is so reassuring, people hunt for the problem anywhere but their own manifest. The discipline: the instant you see Degraded, ignore the sync badge and go straight to the resource tree and kubectl logs/describe. Synced tells you the cause is in Git; the health tree tells you which resource and why.

3. The invisible CRD. A Healthy app is only as trustworthy as its health checks. Custom resources without a check aren’t counted in the rollup, so their app’s Healthy is a partial truth. Wherever you depend on a CRD, write or confirm a health check for it — otherwise you’re watching a green light that can’t see half the room.

---

Common beginner mistakes

These are mental-model errors — the wrong picture in your head — as opposed to the symptom→fix table above. Each one quietly sends beginners down the wrong path.

“Refreshing will fix an OutOfSync app.” It won’t. Refresh (and even --hard-refresh) only re-reads the world: it recomputes the diff and re-evaluates health. It changes the reading, never the cluster. Only sync applies Git and moves an app toward Synced. Right model: refresh updates the report; sync changes the state.

“OutOfSync means Argo CD will fix it automatically.” Only if the app has an automated sync policy with self-heal on. A manual-sync app will sit at OutOfSync indefinitely, perfectly content, until a human or CI runs a sync. Right model: OutOfSync is a report, not a promise; whether it auto-heals depends entirely on the app’s sync policy (automated + self-heal).

“The health check reads my Git manifest.” It never touches Git. Health is computed only from the live resource’s own .status; Git is the desired side of the sync comparison and is invisible to health. Right model: sync compares Git↔live; health inspects live alone.

“A failed sync shows up as OutOfSync.” Not necessarily. A failed sync operation surfaces in operationState as Failed/Error with a SyncError — while the sync status often still reads OutOfSync precisely because the apply didn’t land. Watch only the sync badge and a failed deploy looks like “nothing happened.” Right model: three axes — the diff, the last apply’s result, and health — and they can disagree.

“Progressing is an error I need to fix.” Progressing is the normal converging state: pods rolling, a LoadBalancer waiting on its address. It only becomes a problem if it never resolves — past progressDeadlineSeconds a Deployment flips to Degraded on its own. Right model: Progressing means “wait”; Degraded means “act.”

“Suspended or Missing means something failed.” Both are routinely benign. Suspended is Argo CD faithfully reporting a deliberate pause (a suspend: true CronJob, a paused Rollout). Missing is usually the split second before a just-declared resource is created, or one waiting in a later sync wave. Right model: only Degraded and a check-errored Unknown are inherently “go look now”; the rest are states, not alarms.

---

Cheat-sheet

Bookmark this. It answers “which axis is which, and how do I read both fast?”

Sync status values:

Health status values:

The four combinations (the whole lesson in four rows):

Custom health check shape (argocd-cm data key): resource.customizations.health.<group>_<kind> → Lua returning hs.status (Healthy/Progressing/Degraded/Suspended) and hs.message.

---

Interview and exam questions

Q: In one sentence each, what do Argo CD’s sync status and health status mean? A: Sync status answers “does the live cluster match the desired state rendered from Git?” (Synced/OutOfSync/Unknown); health status answers “are the live resources actually working?” (Healthy/Progressing/Degraded/Suspended/Missing/Unknown). They are independent axes.

Q: An app is Synced but Degraded. Whose problem is it, and where do you look? A: It’s your problem, not Argo CD’s. Synced means Argo CD applied exactly what Git contains; Degraded means a live resource is failing. Open the resource tree in argocd app get, find the Degraded child, read its MESSAGE, and go to kubectl logs/describe. Common causes: bad image tag, crash loop, failing readiness probe.

Q: How can an app be OutOfSync yet Healthy at the same time? A: The resources are running fine (Healthy) but live differs from Git (OutOfSync) — typically because someone ran kubectl edit (drift) or because a new commit hasn’t been synced yet (in manual-sync apps). Sync and health measure different things, so this combination is normal.

Q: How does Argo CD compute OutOfSync? A: The repo-server renders the desired manifests at targetRevision (helm/kustomize/plain YAML); the application-controller fetches live objects, normalizes both sides (stripping system defaults and anything under ignoreDifferences), and compares field by field. If every managed resource matches → Synced; if any differ → OutOfSync; if it can’t render/compare → Unknown.

Q: What does “health is computed by per-resource-kind checks” mean? Give three examples. A: “Healthy” is defined differently per kind. A Deployment is Healthy when its available replicas meet the desired count; a LoadBalancer Service is Healthy only once .status.loadBalancer.ingress has an address; a Job is Healthy when its Complete condition is true. The verdict comes from each resource’s own live status, never from Git.

Q: Explain the app-level health rollup. Why can one resource turn a 40-resource app Degraded? A: Health rolls up “worst status wins,” ranked Healthy → Suspended → Progressing → Missing → Degraded → Unknown. The app reports the worst status among its resources, so a single Degraded pod makes the whole app Degraded regardless of how many resources are Healthy. Sync rolls up differently: any OutOfSync resource makes the app OutOfSync.

Q: You add a custom resource and the app still shows Healthy, but you’re not sure the CR works. Why, and what do you do? A: Argo CD has no health check for that kind, so it forms no opinion and excludes the resource from the rollup — the app looks Healthy because the CR is invisible to the calculation, not because it’s confirmed working. Add a Lua health check under resource.customizations.health.<group>_<kind> in argocd-cm so the CR is actually assessed.

Q: What’s the difference between Missing and Degraded health? A: Missing means a resource is declared in Git but doesn’t exist live (often transient right after creation, or blocked by RBAC/admission/a pending sync wave). Degraded means a resource exists but has failed (crash loop, exceeded progress deadline, failed Job). Missing = “not there”; Degraded = “there and broken.”

Q: An app sits at Progressing and never becomes Healthy. How do you diagnose it? A: Open the resource tree and find the Progressing child. Common causes: pods failing readiness (bad image/probe) or a LoadBalancer Service/Ingress waiting on an external address. For Deployments, after spec.progressDeadlineSeconds (default 600s) Kubernetes flags ProgressDeadlineExceeded and Argo CD flips it to Degraded, which surfaces the real reason.

Q: Sync status is Unknown. What happened and where’s the fix? A: Argo CD couldn’t run the comparison — a ComparisonError. The source is unreachable, the revision doesn’t exist, the path is wrong, or the Helm/Kustomize render failed. Read the ComparisonError in argocd app get; the fix is on the repo/render side, never in your running pods.

Q (scenario): A junior engineer says “Argo CD broke our deploy — it’s showing red.” The app is Synced/Degraded. How do you coach them? A: Reframe the two axes: Synced proves Argo CD did its job (cluster matches Git), so it didn’t “break” anything. The red is on the health axis — a live resource is failing because of what’s in Git. Walk them to the resource tree, read the MESSAGE, then the pod logs. The fix is in their manifest or image; Argo CD is correctly reporting reality.

Q: A LoadBalancer Service shows Progressing for two minutes after a sync on EKS. Bug or normal? Does it differ on AKS/GKE? A: Normal. A LoadBalancer Service is Healthy only once the cloud assigns an external address; until then it’s Progressing. The wait exists on all three clouds — the address is provisioned by the Azure cloud-controller/LB on AKS, the NLB via the in-tree CCM or AWS Load Balancer Controller on EKS, and the Google Cloud LB on GKE. It’s only a problem if it never resolves, which points at the LB controller or its cloud identity.

---

Glossary

---

Key takeaways

• Sync status and health status are two orthogonal axes. Sync asks “does live match Git?” (Synced/OutOfSync/Unknown); health asks “do the live resources work?” (Healthy/Progressing/Degraded/Suspended/Missing/Unknown). Never read one as the other.
• OutOfSync is not failure. It means the cluster drifted from Git or a commit hasn’t synced — the resources can be perfectly Healthy. It implicates Git or a drift, not your running app.
• Synced + Degraded is your bug, not Argo CD’s. Argo CD applied exactly what Git contains; a failing app after a clean sync means the problem is in your manifest or image. Go straight to the resource tree and kubectl logs.
• Health is per-resource-kind. “Healthy” means available replicas for a Deployment, an assigned address for a LoadBalancer Service/Ingress, completion for a Job. Built-in checks cover core kinds and many popular CRDs.
• The app rolls up “worst status wins” for health, so one bad child turns the whole app Degraded — and a resource with no health check is skipped entirely, letting an app read Healthy while that resource is silently broken.
• Teach Argo CD your CRDs with a Lua health check under resource.customizations.health.<group>_<kind> in argocd-cm: read the resource’s .status, return Healthy/Progressing/Degraded, and default to Progressing when you can’t tell.
• Read both axes in argocd app get: the header is the rollup, the resource tree is the drill-down with per-resource STATUS, HEALTH, and MESSAGE. The MESSAGE column usually names the exact cause.
• The computation is cloud-neutral except one edge: a LoadBalancer Service or Ingress is only Healthy once the cloud assigns an address — AKS, EKS, and GKE each provision it differently, so a brief Progressing after a sync is normal, and a permanent one is a cloud LB problem.