Argo CD Lesson 2 of 45

Argo CD Architecture: API Server, Repo Server, Application Controller, Redis & Dex

Most people meet Argo CD as a web page: a graph of green hexagons, a big blue SYNC button, an app that says Synced and Healthy. That UI is honest, but it hides the machine. Behind the graph sit five cooperating processes, each with one job, talking to each other over gRPC and to a Redis cache in the middle. Learn those five and Argo CD stops being magic. You will know why an app is stuck, which pod’s logs to open, and what a given error message is actually complaining about — before you have read a single line of the docs.

This lesson opens the box. We will name every component, say exactly what each does, and follow one change all the way through: from argocd app sync on your laptop, through the repo-server turning your Git repo into Kubernetes YAML, into the application-controller’s reconcile loop where desired state meets live state, and out to the cluster. Then we will do the thing that makes this knowledge pay: build the debugging map — the table that tells you, for any symptom, which component is to blame.

Everything here is cloud-neutral. Argo CD’s internals are byte-for-byte identical on AKS, EKS, GKE, a laptop kind cluster, or bare metal — the same seven pods, the same reconcile loop. Only the edges differ (how you expose the UI to the internet, which identity provider Dex federates to), and where those edges appear we cover all three clouds in one line each. The engine is universal.


Why this matters

Here is the trap. You install Argo CD, point it at a repo, and for a week everything is green. Then one morning an app is OutOfSync and won’t move, or the UI throws ComparisonError, or a login that worked yesterday returns rpc error: code = Unauthenticated. If Argo CD is a single black box in your head, you are now stuck — you restart pods at random and hope. If instead you hold the component model, the same symptoms name their own culprit: ComparisonError is the repo-server failing to render; a stuck sync is the application-controller; a broken login is the API server or Dex. The map turns panic into a two-minute diagnosis.

The second reason is that every later lesson assumes this picture. When we talk about sync waves, we mean the order the controller applies resources. When we talk about private repos, we mean credentials the repo-server uses to clone. When we talk about SSO, we mean the API server delegating to Dex. When we talk about scaling to hundreds of clusters, we mean sharding the controller. Without the architecture, those lessons are vocabulary you memorise; with it, they are obvious consequences of who-does-what.

The mental model to anchor everything: Argo CD is a control loop, not a pipeline. A CI pipeline runs once, top to bottom, and stops. Argo CD’s application-controller runs forever, in a loop, continuously asking one question — “does the cluster match Git?” — and closing the gap when the answer is no. Git is the desired state; the cluster is the actual state; the controller is the thermostat that drives one toward the other. Hold that and the rest is detail. (If the pull-based, continuously-reconciled model itself is new, the sibling lesson GitOps From First Principles is the place to start; this lesson assumes you accept the model and asks how it is built.)


The component map at a glance

Argo CD is not one program. A default install drops seven workloads into one namespace, of which five are the core you must know and two are important-but-optional controllers. Every one is an ordinary Kubernetes Deployment or StatefulSet — there is no hidden daemon, no host agent. Here is the whole cast in one table; the rest of the lesson is just this table, expanded.

Component Kubernetes kind One-line job Talks to If it’s down, you see…
API server (argocd-server) Deployment Serves the UI + CLI; authentication, RBAC, app CRUD Clients, repo-server, Redis, Dex, k8s API UI won’t load, argocd login fails
Repository server (argocd-repo-server) Deployment Clones Git, renders manifests (kustomize/helm/plain/plugins) Git repos, Redis ComparisonError, apps can’t render
Application controller (argocd-application-controller) StatefulSet The reconcile loop: diff desired vs live, sync, run hooks, report status repo-server, Redis, target clusters Nothing syncs; status never updates
Redis (argocd-redis) Deployment Ephemeral cache of rendered manifests + cluster state server, repo-server, controller Slow UI, stale views, cache errors
Dex (argocd-dex-server) Deployment Optional bundled OIDC provider for SSO federation API server, your IdP SSO login broken (local login still works)
ApplicationSet controller (argocd-applicationset-controller) Deployment Generates many Applications from generators (templating) Git, k8s API New generated apps don’t appear
Notifications controller (argocd-notifications-controller) Deployment Sends alerts on app state changes (Slack, email, webhooks) k8s API, notification services No notifications fire

Three facts to notice immediately, because they drive everything below:

  1. Only the application-controller is a StatefulSet. The rest are stateless Deployments. That single design choice is why its pod is named argocd-application-controller-0 (a StatefulSet ordinal) and why “scaling the controller” means sharding across ordinals -0, -1, -2 — each owning a slice of your clusters — rather than plain replicas. (Sharding is a Tier-4 scaling topic; just file the shape away.)
  2. Redis sits in the middle. The server, repo-server, and controller do not cache in their own memory; they share Redis. That is why Redis being unhealthy degrades all three at once, and why — because it is only a cache — losing it costs performance, never data.
  3. The two extra controllers are genuinely separable. ApplicationSet and Notifications are their own binaries with their own jobs; you can uninstall either and core sync still works. We forward-reference them here and give each its own lesson later.

Now, each component in turn.


The API server: the front door

The argocd-server process is the only component you talk to directly. The web UI is served by it; the argocd CLI is a gRPC/REST client of it; every automation token authenticates against it. Think of it as the front door and the reception desk combined — it does not do the deployment work itself, but nothing reaches the work without passing through it first.

It handles a specific, bounded set of responsibilities:

Responsibility What it means in practice
Serve the UI + API One process serves the web app, a REST API, and a gRPC API — the CLI and UI use the same endpoints
Authentication Validates the session/JWT on every call: local admin user, API tokens, or SSO (via Dex or a direct OIDC issuer)
Authorization (RBAC) Enforces argocd-rbac-cm policy — can this subject sync/get/create this app/project?
Application management Create/read/update/delete Application, AppProject, repo and cluster records (writing the CRDs the controller then acts on)
Repo/manifest proxy Calls the repo-server on your behalf for the UI’s live diff and manifest views
Webhook receiver Accepts Git provider webhooks at /api/webhook to trigger an immediate refresh instead of waiting for the poll

One protocol detail worth internalising because it explains a whole class of connection errors: argocd-server multiplexes gRPC and HTTP on the same port (container port 8080, exposed by the argocd-server Service as 80/443). The CLI speaks gRPC; the browser speaks HTTP; both hit the same endpoint. When you see the CLI hang or throw a TLS/gRPC handshake error, it is almost always about how you reached this port (plain vs TLS, a proxy stripping gRPC, --grpc-web needed behind an L7 load balancer) — not about the controller or your manifests.

Because the API server is the internet-facing piece, exposing it is the one genuinely cloud-specific edge in this lesson. Out of the box its Service is ClusterIP (reachable only inside the cluster); to reach the UI from your laptop you either kubectl port-forward, or put a cloud load balancer / ingress in front. The mechanics differ per cloud:

Cloud Managed cluster Typical way to expose argocd-server Notes
Azure AKS Application Gateway via AGIC, or a Service type=LoadBalancer (Azure LB) Front with Entra ID later for SSO; ⚠️ a public LB/App Gateway bills hourly
AWS EKS AWS Load Balancer Controller — ALB (Ingress) or NLB (Service) ALB needs --grpc-web or a gRPC-aware listener for the CLI; ⚠️ ALB/NLB bills
Google GKE GKE Ingress (external GCLB) or a Service type=LoadBalancer Container-native LB via NEGs is the clean path; ⚠️ forwarding rules bill

The install lesson walks through each of these end to end (including HA and first login); here the point is only that the API server is the component that owns “can I reach the UI?” If login or the UI is the problem, this pod — or the load balancer in front of it — is where you look. If apps are the problem, the API server is almost never the cause, because — and this is the fact that surprises people — you can delete the API server entirely and your apps keep reconciling. The controller does not need it.


The repository server: turning Git into manifests

The argocd-repo-server has the most underestimated job in the system: it is the one that clones your Git repositories and turns whatever is in them into plain Kubernetes manifests. Argo CD does not deploy “your repo”; it deploys the rendered output of your repo. The repo-server is the renderer.

Give it a source (a repo URL, a revision, a path) and it:

  1. Clones (or reuses a cached clone of) the repo at the requested targetRevision.
  2. Looks at the path and decides which tool to run — plain YAML, Kustomize, Helm, or a config-management plugin.
  3. Runs that tool to produce a flat stream of Kubernetes manifests.
  4. Returns those manifests to the caller (the controller during reconcile, or the API server for the UI diff), and caches them in Redis keyed by repo + revision + parameters.

The tool it picks is automatic, based on what the path contains:

If the path contains… Repo-server runs… Produces
Plain .yaml/.json manifests (nothing — reads them directly) The manifests as-is
A kustomization.yaml kustomize build Kustomize’s rendered output
A Chart.yaml helm template (server-side render, not helm install) The chart rendered with your values
A .jsonnet file the Jsonnet VM Generated manifests
A configured plugin match your Config Management Plugin (CMP) Whatever the plugin prints to stdout

Two things here routinely bite newcomers, so say them plainly. First, Helm under Argo CD is helm template, not helm install. The repo-server renders the chart to YAML locally and hands the YAML to the controller; Tiller does not exist, and there is no Helm release stored in the cluster. Your “release” is the Application. That is why helm list shows nothing and why Helm hooks behave differently — the manifests are just applied by Argo CD like any other. Second, the repo-server is where the words “render” and “generate” always point. Any error that mentions manifest generation, a kustomize build failure, a missing Helm value, or ComparisonError originates here, because rendering is exclusively this component’s job.

Caching is the repo-server’s other half, and it is why big installs stay fast:

Cache (in Redis) Keyed by Why it matters
Rendered manifests repo + revision + path + parameters The expensive helm template/kustomize build runs once per unique input, not once per reconcile
Git revision metadata repo + ref Resolving a branch/tag to a commit SHA is cached briefly
Repo clones (on local disk) repo The pod keeps working copies so it re-fetches, not re-clones, on each refresh

The cache is also the source of a classic gotcha: after a Git change the repo-server may briefly serve a cached render, so a --hard-refresh (which bypasses the manifest cache and forces a fresh helm template/kustomize build) is the fix when an app looks stale even though you know Git moved. And because the repo-server runs your tooling with your credentials, it is also the component that needs repo access — SSH keys, HTTPS tokens, Helm/OCI registry creds all live as Secrets the repo-server consumes (covered in the repositories lesson).


The application controller: the reconcile loop

This is the heart of Argo CD, so we go slowly. The argocd-application-controller is the process that actually makes your cluster match Git. It is a control loop: for every Application, over and over, it compares desired state (the manifests the repo-server renders from Git) against live state (what is actually running in the target cluster), decides whether they match, judges whether the running app is healthy, and — if you have asked it to — applies the difference. Everything the UI shows you (Synced/OutOfSync, Healthy/Degraded) is this loop reporting its findings.

One full pass of the loop, for one Application, looks like this:

Step What the controller does Result it records
1. Trigger A reconcile is kicked off (see triggers below)
2. Get desired state Asks the repo-server to render the app’s source at targetRevision The target manifests
3. Get live state Reads the actual objects from the target cluster (served from its watched cluster cache) The live manifests
4. Diff Compares desired vs live field-by-field, honouring ignoreDifferences Synced or OutOfSync
5. Assess health Runs per-resource health checks (built-in + custom Lua) and rolls them up Healthy / Progressing / Degraded / Suspended / Missing
6. Act (if automated) If OutOfSync and syncPolicy.automated is set, apply manifests in sync-wave order, run hooks, prune An OperationState (Succeeded/Failed)
7. Report Writes sync status, health, revision, and conditions back to the Application status What the UI/CLI display

Steps 4 and 5 produce two independent answers, and conflating them is the single most common beginner confusion. Sync status answers “does live match Git?” Health status answers “is the running thing actually working?” They are orthogonal — an app can be any combination:

Healthy Degraded
Synced The happy path: cluster matches Git and the app works Matches Git, but the app is broken (bad image, failing probe) — Git itself is wrong
OutOfSync Works, but drifted from Git (someone ran kubectl edit, or a new commit hasn’t been applied) Drifted and broken — usually mid-failed-deploy

(The full state machine — including Progressing, Missing, and Unknown — is the whole subject of the sibling lesson Sync Status & Health Assessment. Here the load-bearing idea is just that the controller computes both, separately.)

What actually triggers a reconcile

The loop does not run continuously at full tilt; it is triggered. Knowing the triggers is what lets you answer “why is Argo CD slow to notice my commit?”

Trigger When it fires Latency to apply
App resync timer Every timeout.reconciliation (default 180s = 3 min) Up to 3 minutes
Git webhook Instantly, when your provider POSTs to /api/webhook on push Near-instant
Spec change You edit the Application (via UI/CLI/Git) — the controller watches its own CRDs Immediate
Watched resource change A managed live object changes (the controller watches the target cluster) Immediate
Manual refresh argocd app get <app> --refresh (soft) or --hard-refresh (bypass manifest cache) On demand

The default 3-minute poll is the number to remember. Out of the box, a commit to Git can take up to three minutes to be noticed, because the controller only re-examines each app every timeout.reconciliation. That interval lives in the argocd-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  # How often the controller re-checks every app even with no event. Default 180s.
  timeout.reconciliation: 180s

You could lower it, but the right fix for “GitOps feels slow” is webhooks, not a tighter poll. A webhook makes a push near-instant while the 3-minute poll stays as a safety net for missed webhooks. Cranking the poll down to, say, 10s instead just multiplies load on the repo-server and every Git host you talk to — a self-inflicted rate-limit. Reach for a webhook first.

A subtle but important point about soft vs hard refresh: a normal (soft) refresh re-runs the diff using cached manifests — it re-reads live cluster state but trusts the repo-server’s cache for desired state. A hard refresh additionally throws away the manifest cache and forces the repo-server to re-clone and re-render. So when an app is stuck showing an old revision even though Git moved, a hard refresh — not a pod restart — is the targeted fix, because it invalidates exactly the cache that is stale.


Redis, Dex, and the two extra controllers

Redis — the cache in the middle

argocd-redis is a plain Redis instance, and its role is easy to state and easy to get wrong: it is an ephemeral cache, never a database. The server, repo-server, and controller all read and write it instead of caching in their own memory, which is what lets them stay fast and (in HA) stay stateless.

What Redis holds Written by Purpose
Rendered manifests repo-server Skip re-running helm template/kustomize build on every reconcile
Cluster resource state application-controller Fast diffs without re-listing the whole cluster each pass
App/UI response data API server Snappy UI and API reads

The crucial property: because it is only a cache, Redis is safe to lose. The source of truth for desired state is Git; the source of truth for live state is the cluster itself. If Redis is wiped or restarted, Argo CD simply rebuilds the cache — it re-renders manifests and re-lists cluster resources. The cost is performance, not correctness: while the cache is cold, reconciles are slower and the UI may momentarily show stale or errored panels. If Redis is fully down (not just wiped), you will see degraded behaviour across the UI and slow or failing reconciles, with cache-connection errors in the logs — but the instant Redis is back, everything recovers with no data lost. This is exactly why production installs run argocd-redis-ha (Redis with Sentinel) — not to protect data, but to keep the cache continuously available so performance never craters.

Dex — optional SSO federation

argocd-dex-server is an embedded Dex instance: a small OIDC provider that Argo CD bundles so you can wire up single sign-on without standing up your own identity broker. It is optional in two senses — you can run Argo CD with just the local admin user and no Dex at all, and even for SSO you can bypass Dex by pointing Argo CD at an OIDC issuer directly. Dex exists for the cases where your identity provider isn’t a clean OIDC issuer, or you want SAML/LDAP/GitHub-org federation normalised into OIDC for Argo CD.

The flow is: the API server delegates login to Dex; Dex federates to your real IdP; the token comes back and the API server enforces RBAC on the groups inside it. Which IdP you federate to is a cloud/identity edge:

Cloud / platform Typical IdP Dex federates to Connector
Azure / Entra Microsoft Entra ID OIDC / SAML connector
AWS IAM Identity Center or Amazon Cognito SAML / OIDC connector
Google / GKE Google Workspace (Cloud Identity) OIDC / SAML connector
Any GitHub / GitLab org, LDAP, Okta Dedicated connectors

Dex’s config lives under dex.config in argocd-cm, with client secrets in argocd-secret. The one thing to know here is the blast radius: misconfigure Dex and only SSO login breaks — the local admin login and already-running apps are untouched. The full setup (per cloud) is the subject of the SSO lesson; for architecture, Dex is “the optional auth sidecar the API server talks to.”

ApplicationSet and Notifications controllers

Two more controllers ship by default and each earns its own lesson later, but you should recognise them in get pods:

Controller Its job Reads / writes Failure looks like
ApplicationSet Templating engine: turns generators (list, cluster, Git, matrix, PR) into many Applications automatically Watches ApplicationSet CRDs; creates/updates Applications New apps don’t get generated; ApplicationSet status shows a generator error
Notifications Watches app state and fires alerts (Slack, Teams, email, webhooks) on triggers you define Reads Application status; calls external services Deploys succeed/fail silently — no message arrives

Both are strictly additive: core sync works without either. The ApplicationSet controller is how you go from hand-writing one Application per app-per-cluster to generating hundreds — the scaling story the App-of-Apps and ApplicationSets lessons pick up in Tier 2.


How a change flows end to end

Now assemble the pieces into one motion. Say you run argocd app sync guestbook from your laptop. Read the diagram left to right, then the walkthrough — the numbered badges mark exactly where each component’s job (and its failure mode) lives.

Argo CD internal architecture showing clients and a Git repo on the left feeding the API server and optional Dex SSO, then the repo-server rendering manifests and the application-controller running its reconcile loop against Redis, and finally applying and diffing against a target Kubernetes cluster on the right

The path, step by step:

  1. CLI → API server. The argocd CLI opens a gRPC call to argocd-server. The server validates your token (local, API-key, or SSO via Dex) and checks RBAC: may you sync this app? If login fails, you never leave this box — the error is Unauthenticated, and the culprit is the API server or Dex, never the controller.
  2. API server records intent. The server writes a sync operation onto the Application resource. It does not deploy anything itself — it just records that a sync was requested. (This is why declarative and imperative management converge: the CLI and a Git edit both end up mutating the same CRD.)
  3. Controller wakes. The application-controller, watching its Application CRDs, sees the requested operation and begins a reconcile.
  4. Controller → repo-server. It asks the repo-server for the desired manifests. The repo-server clones the repo at targetRevision, runs kustomize build/helm template/plain YAML, and returns rendered manifests — caching them in Redis. A broken chart or kustomization surfaces here as ComparisonError.
  5. Controller diffs and applies. It compares desired against the live cluster state (kept warm in its Redis-backed cache), computes the diff, then applies the changes to the target cluster in sync-wave order, running any hooks.
  6. Cluster → status → you. The controller watches the applied objects, assesses health, writes Synced/Healthy back to the Application status, and the API server serves that back to your CLI and the UI.

Notice what depends on what. The API server is only needed at the ends (to accept your request and to show you the result); the actual reconcile — steps 3 through 6 — runs whether or not any human is watching. That is the control-loop nature made concrete: kill the UI and your apps still self-heal; kill the controller and everything freezes.


Namespaces, CRDs, and the pods you’ll see

One namespace, by convention

By default the whole control plane lives in the argocd namespace. Nothing forces that name — you can install elsewhere — but argocd is the universal convention, and every command in this course uses -n argocd. The workloads there are the seven above plus their Services, ConfigMaps (argocd-cm, argocd-cmd-params-cm, argocd-rbac-cm), and Secrets (argocd-secret, plus repo/cluster Secrets).

Where your apps run is a separate question. Argo CD can deploy into the same cluster it runs in, into other namespaces, or into entirely different clusters — the Application’s destination decides. The control plane and the workloads it manages are decoupled.

Three CRDs are the whole API surface

Argo CD extends Kubernetes with a small set of Custom Resource Definitions, all in the argoproj.io/v1alpha1 API group. These are Argo CD’s declarative interface — the UI and CLI are just ergonomic ways to edit them:

CRD Short name What it declares Managed by
Application app / apps One deployable unit: a source (repo/path/revision) → a destination (cluster/namespace), plus sync policy You / API server / ApplicationSet
ApplicationSet appset A generator + template that produces many Applications You; the ApplicationSet controller acts on it
AppProject appproj A guardrail: which repos, clusters, namespaces and resource kinds a group of apps may use Platform team

A minimal Application is worth seeing now, because it is the object every other lesson manipulates and the thing the controller reconciles:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd                     # the Application lives in the control-plane namespace
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook                     # the repo-server renders whatever is here
  destination:
    server: https://kubernetes.default.svc   # deploy into THIS cluster
    namespace: guestbook
  syncPolicy:
    automated:                          # let the controller sync without a human click
      prune: true
      selfHeal: true

You do not need to understand every field yet — the first-Application lesson is entirely about this manifest. What matters architecturally is the mapping: source is the repo-server’s input, destination is the controller’s target, syncPolicy tells the controller whether to act automatically, and the whole object is what the API server writes and the controller reads.

The pods you’ll actually see

On a fresh non-HA install, kubectl -n argocd get pods shows the cast in the flesh (representative output — names have random suffixes, and an HA install adds Redis-HA and multiple server/repo replicas):

kubectl -n argocd get pods
# NAME                                                READY   STATUS    RESTARTS   AGE
# argocd-application-controller-0                     1/1     Running   0          6m
# argocd-applicationset-controller-6d8f4b5c9d-abcde   1/1     Running   0          6m
# argocd-dex-server-7b9c8f6d5b-fghij                  1/1     Running   0          6m
# argocd-notifications-controller-5c7d9f8b6c-klmno    1/1     Running   0          6m
# argocd-redis-6fd9c8b7f5-pqrst                       1/1     Running   0          6m
# argocd-repo-server-84b7d9c6f8-uvwxy                 1/1     Running   0          6m
# argocd-server-6c9f7d8b5c-z1234                      1/1     Running   0          6m

The -0 on the controller (versus the ReplicaSet-hash suffixes on the others) is the StatefulSet tell from earlier. And the matching Services expose the internal ports the components use to talk to each other:

kubectl -n argocd get svc
# NAME                               TYPE        PORT(S)                     (what it is)
# argocd-server                      ClusterIP   80/TCP,443/TCP              UI + API (→ 8080)
# argocd-repo-server                 ClusterIP   8081/TCP,8084/TCP           render gRPC + metrics
# argocd-redis                       ClusterIP   6379/TCP                    the cache
# argocd-dex-server                  ClusterIP   5556/TCP,5557/TCP,5558/TCP  OIDC http/grpc/metrics
# argocd-applicationset-controller   ClusterIP   7000/TCP,8080/TCP           webhook + metrics
# argocd-metrics                     ClusterIP   8082/TCP                    controller metrics
# argocd-server-metrics              ClusterIP   8083/TCP                    server metrics

Two reading tips: the application-controller has no serving Service other than argocd-metrics (it is a controller, not a server — it dials out to clusters and the repo-server, nothing dials in), and argocd-server is the only one whose Service you would ever expose beyond the cluster.


The debugging map: which component owns which symptom

This is the table the whole lesson exists to give you. When Argo CD misbehaves, do not guess — read the symptom, find the component, open that pod’s logs. Memorise the shape of it and you have turned every future incident into a lookup.

Symptom you observe Component to blame Why First thing to check
UI won’t load; argocd login fails API server (or its LB/ingress) It serves UI+API and does auth kubectl -n argocd logs deploy/argocd-server; the load balancer in front
rpc error: code = Unauthenticated API server / Dex Token/SSO validation failed Token expiry; argocd-dex-server logs if using SSO
ComparisonError; app can’t render Repo server Rendering is exclusively its job kubectl -n argocd logs deploy/argocd-repo-server for the helm/kustomize error
helm template/kustomize build error Repo server It runs those tools repo-server logs; try the same command locally on the path
App stuck OutOfSync / never syncs Application controller It runs the reconcile + apply kubectl -n argocd logs statefulset/argocd-application-controller
App stuck Progressing forever Application controller (+ the workload) It runs the health assessment controller logs; then the app’s own pods/events
Status never updates; everything Unknown Application controller (down/wedged) No reconcile means no status Is argocd-application-controller-0 Running?
Sync latency; commits take minutes Reconcile trigger (poll vs webhook) Default 3-min timeout.reconciliation Add a Git webhook; check webhook delivery
UI slow, panels stale or errored Redis Shared cache is down/cold kubectl -n argocd logs deploy/argocd-redis; is the pod up?
SSO login broken; local login works Dex (misconfig) Only the SSO path uses Dex argocd-dex-server logs; dex.config in argocd-cm
Controller OOMing with many apps/clusters Application controller (needs sharding) One shard can’t hold all cluster caches Controller memory; shard across replicas (Tier-4)
New generated apps don’t appear ApplicationSet controller It generates Applications kubectl -n argocd logs deploy/argocd-applicationset-controller
Application/AppProject kind not found CRDs missing (incomplete install) The API types were never installed kubectl get crd | grep argoproj.io

The pattern to internalise: rendering problems are the repo-server, sync/status problems are the controller, login/UI problems are the API server, “slow/stale” is Redis, and “SSO only” is Dex. Five rules cover the vast majority of real incidents.


Hands-on lab

This lab is pure exploration — you will not change anything, just see the architecture running and practise the mapping. It assumes you already have Argo CD installed (the install lesson in this tier covers that on AKS, EKS, GKE, or a free local cluster); if you don’t yet, do that first — a kind or minikube cluster is perfect and none of this bills. Everything below is read-only kubectl/argocd.

Step 1 — Meet the whole cast.

kubectl -n argocd get pods,svc
# You should see the 7 workloads and their Services from the section above.

What just happened: you are looking at the entire Argo CD control plane. Find each of the five core components by name. Confirm the controller ends in -0 (StatefulSet) while the rest carry ReplicaSet hashes (Deployments).

Step 2 — Confirm the CRDs are installed.

kubectl get crd | grep argoproj.io
# applications.argoproj.io
# applicationsets.argoproj.io
# appprojects.argoproj.io

What just happened: these three CRDs are Argo CD’s entire declarative API. If this command comes back empty, your install is incomplete — that is the “kind not found” row in the debug map, seen from the other side.

Step 3 — Watch the application-controller think.

kubectl -n argocd logs statefulset/argocd-application-controller --tail=40 -f
# time="..." level=info msg="Refreshing app status (controller refresh requested)" application=argocd/guestbook
# time="..." level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: guestbook)" application=argocd/guestbook
# time="..." level=info msg="Reconciliation completed" application=argocd/guestbook time_ms=137

What just happened (representative logs): you are watching the reconcile loop from the last section, live. Each Comparing app state line is step 4 (the diff); Reconciliation completed closes one pass. Leave this streaming and move to Step 4.

Step 4 — Trigger a reconcile and watch it land. In a second terminal:

argocd app get guestbook --refresh    # force a soft refresh now, don't wait for the 3-min poll

What just happened: the --refresh flag made the controller reconcile immediately instead of at the next timeout.reconciliation tick. Flip back to your Step 3 terminal — you should see a fresh Refreshing app statusComparing app stateReconciliation completed burst appear the moment you ran it. You just caused a trip around the loop and observed it in the logs. That connection — command in one terminal, loop reaction in the other — is the whole mental model made tangible.

Step 5 — Watch the repo-server render.

kubectl -n argocd logs deploy/argocd-repo-server --tail=30
# time="..." level=info msg="manifest cache miss" ...
# time="..." level=info msg="Generating manifests with no cache" ...

What just happened (representative): a cache miss followed by Generating manifests is the repo-server doing its one job — running your kustomize/helm/plain-YAML render. A cache hit on a repeat means Redis served the previous render. This is exactly where ComparisonError would show up if a chart were broken.

Step 6 — Prove the API server is separable (optional, conceptual).

# Read-only observation: note the controller reconciles on its own timer.
kubectl -n argocd logs statefulset/argocd-application-controller --tail=5
# Even with nobody logged into the UI, reconciles keep happening.

What just happened: you confirmed the claim from earlier — the reconcile loop runs without any client attached. The API server is the door you use, not the engine that does the work.

Step 7 — Map a symptom yourself. Without scrolling up, answer: an app shows ComparisonError. Which pod’s logs do you open? (Answer: argocd-repo-server — rendering is its job.) Now: argocd login returns Unauthenticated. Which pod? (argocd-server, and argocd-dex-server if you use SSO.) If both came instantly, you have the map.

Teardown: nothing to tear down — this lab created no resources. If you spun up a local cluster only for this, delete it when done (kind delete cluster / minikube delete) so it isn’t left running.


Common mistakes and troubleshooting

The debugging map above is your symptom→component index. This section adds the causes and fixes for the mistakes beginners actually make, plus prose on the three nastiest.

Symptom Cause Fix
ComparisonError with a helm/kustomize message The repo-server can’t render — bad chart, missing value, wrong path Read argocd-repo-server logs; run the same helm template/kustomize build locally on that path
App shows an old commit after you pushed repo-server served a cached render argocd app get <app> --hard-refresh to bypass the manifest cache
Commit takes ~3 minutes to deploy Default timeout.reconciliation: 180s poll; no webhook Add a Git webhook to /api/webhook; don’t just shrink the poll
argocd loginrpc error: code = Unauthenticated Expired token, or SSO/Dex misconfigured Re-login; if SSO, check argocd-dex-server logs and dex.config
CLI hangs / TLS or gRPC handshake error behind an LB L7 load balancer not gRPC-aware Use argocd login --grpc-web, or an ALB/ingress configured for gRPC
Everything shows Unknown, status frozen application-controller is down or wedged Check argocd-application-controller-0 is Running; read its logs
UI slow, panels intermittently error Redis down or cold Check argocd-redis; on prod run argocd-redis-ha for cache availability
error: failed to get repo / clone fails repo-server lacks credentials or network to Git Check the repo Secret and repo-server egress; test the URL from the pod
SSO login fails but admin login works Dex misconfiguration only Fix dex.config; local/API-token auth is unaffected
no matches for kind "Application" CRDs not installed (incomplete install) Re-apply the install manifests; kubectl get crd | grep argoproj
Controller pod OOMKilled as fleet grows One controller shard holding too many cluster caches Increase memory now; shard the controller across replicas (Tier-4)

Three gotchas cost the most hours, so spell them out:

1. Blaming the cluster for a render error. ComparisonError feels like a deployment failure, so people go poke at their pods and Deployments. But ComparisonError fires in step 4 of the reconcile — the controller could not even get the desired manifests, because the repo-server failed to render them. Nothing was ever applied to the cluster. The fix is always in the repo-server’s world: a bad Helm value, a kustomization.yaml that doesn’t build, a path pointing at the wrong directory. Reproduce it by running helm template or kustomize build on that path yourself; the error will be identical, and you never touched the cluster.

2. Fighting the 3-minute poll the wrong way. When “GitOps feels slow,” the instinct is to shrink timeout.reconciliation. That is a trap: a low poll multiplies load on the repo-server and hammers your Git host (GitHub/GitLab will rate-limit you), and it still isn’t instant. The correct fix is a webhook, which makes a push near-instant while the 3-minute poll remains a cheap safety net for the rare missed webhook. Latency is a trigger problem, not a poll-interval problem.

3. Treating Redis like it holds your data. Because “database-shaped things” usually hold state, people panic when Redis restarts and assume apps are lost. They are not. Redis is only a cache; desired state is in Git and live state is in the cluster. A Redis blip degrades performance and can throw transient UI errors, and then everything recovers on its own when the cache warms back up. Run argocd-redis-ha in production not to protect data but to keep the cache continuously available so you never even notice the blip.


Cheat-sheet

The commands and mappings you will reach for constantly.

Command What it does
kubectl -n argocd get pods See all Argo CD components and their status
kubectl -n argocd get svc See the internal ports each component exposes
kubectl -n argocd logs statefulset/argocd-application-controller The reconcile loop’s logs (sync/diff/health)
kubectl -n argocd logs deploy/argocd-repo-server Manifest rendering logs (ComparisonError origin)
kubectl -n argocd logs deploy/argocd-server API/UI/auth logs (login problems)
kubectl -n argocd logs deploy/argocd-dex-server SSO federation logs
kubectl -n argocd logs deploy/argocd-redis Cache health
kubectl get crd | grep argoproj.io Confirm the 3 CRDs are installed
argocd app get <app> --refresh Force an immediate soft reconcile
argocd app get <app> --hard-refresh Reconcile and bust the manifest cache
kubectl -n argocd get cm argocd-cm -o yaml Read core config (incl. timeout.reconciliation)
kubectl -n argocd get application <app> -o yaml The raw CRD the controller reconciles

Component → job → failure signature, in one glance:

Component Its one job Failure signature
API server UI/CLI, auth, RBAC, app CRUD Login/UI fails; Unauthenticated
Repo server Clone Git + render manifests ComparisonError; helm/kustomize errors
Application controller The reconcile loop (diff, sync, health) Stuck OutOfSync/Progressing; frozen status
Redis Ephemeral cache Slow/stale UI; cache errors (no data loss)
Dex Optional OIDC SSO SSO login broken; local login fine
ApplicationSet controller Generate many Applications Generated apps don’t appear
Notifications controller Alert on state changes No notifications fire

Ports worth knowing: server 443/808080, repo-server 8081, Redis 6379, Dex 5556, controller metrics 8082.


Interview and exam questions

Q: Name the five core components of Argo CD and give each a one-line job. A: API server — serves the UI/CLI and handles auth, RBAC, and app management. Repository server — clones Git and renders manifests (kustomize/helm/plain/plugins). Application controller — runs the reconcile loop: diff desired vs live, assess health, sync. Redis — an ephemeral cache of rendered manifests and cluster state. Dex — an optional bundled OIDC provider for SSO federation.

Q: Which component renders your Helm chart, and does Argo CD run helm install? A: The repo-server, and no — it runs helm template to render the chart to plain manifests locally, then hands them to the controller to apply. There is no Tiller and no Helm release stored in the cluster; the Application is the release. That’s why helm list shows nothing.

Q: An app shows ComparisonError. Which component is at fault and why? A: The repo-server. ComparisonError means the controller couldn’t obtain the desired manifests because rendering failed — a broken chart/kustomization, a missing value, or a wrong path. Nothing was applied to the cluster; the fix is in the repo-server’s render, reproducible with helm template/kustomize build on that path.

Q: What is the reconcile loop, and what two independent answers does it produce? A: It’s the application-controller’s continuous cycle: get desired state (repo-server), get live state (cluster), diff, assess health, and — if automated — apply. It produces sync status (Synced/OutOfSync: does live match Git?) and health status (Healthy/Progressing/Degraded: is the app actually working?). They’re orthogonal — an app can be Synced but Degraded, or OutOfSync but Healthy.

Q: What is timeout.reconciliation, its default, and the right way to make syncs faster? A: It’s the interval in argocd-cm at which the controller re-checks every app even with no event; default 180s (3 minutes). The right fix for slow syncs is a Git webhook to /api/webhook (near-instant on push), keeping the poll as a safety net — not shrinking the poll, which just loads the repo-server and rate-limits your Git host.

Q: If Redis goes down, do you lose data? What breaks? A: No data loss — Redis is only a cache; desired state lives in Git, live state in the cluster. What breaks is performance: slow reconciles, stale or errored UI panels, cache-connection errors in the logs. When Redis returns, Argo CD rebuilds the cache and recovers. Prod runs argocd-redis-ha for cache availability, not data safety.

Q: You delete the argocd-server pod. Do your applications stop syncing? A: No. The application-controller reconciles independently of the API server. You lose the UI, the CLI, and SSO/webhook reception, but existing apps keep diffing and self-healing. Kill the controller, though, and all reconciliation freezes — that’s the component whose outage actually stops deployments.

Q: Why is the application-controller a StatefulSet while the others are Deployments, and what does that imply for scale? A: It holds per-cluster state (watched-resource caches) and is scaled by sharding — each ordinal replica (-0, -1, …) owns a subset of clusters — rather than by interchangeable replicas. That’s why its pod is ...-controller-0 and why “scale the controller” means configure sharding, not just bump replicas.

Q: What are Argo CD’s three CRDs and what does each declare? A: Application (one deployable unit: sourcedestination + sync policy), ApplicationSet (a generator + template that produces many Applications), and AppProject (a guardrail restricting which repos, clusters, namespaces, and kinds a group of apps may use). All are in argoproj.io/v1alpha1.

Q: Is Dex required? What breaks if it’s misconfigured? A: No — you can run with just the local admin user, or use a direct OIDC issuer without Dex. Dex is the bundled OIDC broker for SSO federation (Entra ID, Google, IAM Identity Center, GitHub org, LDAP…). Misconfigure it and only SSO login breaks; local and API-token auth, and all running apps, are unaffected.

Q (scenario): The UI is down but kubectl get applications -n argocd shows everything Synced/Healthy. What’s going on and is it urgent? A: The API server (or its load balancer/ingress) is down, but the controller is fine — apps are still reconciling, which is why the CRDs report healthy. It’s a visibility/access outage, not a delivery outage: fix argocd-server or the LB in front of it, but deployments are not at risk in the meantime.

Q (scenario): A teammate lowered timeout.reconciliation to 10s and now Git pushes sometimes fail with 403. Why? A: The tight poll makes the repo-server re-fetch/re-render far more often, hammering the Git host until it rate-limits (403 secondary rate limit on GitHub). Restore the default 180s and add a webhook for latency instead — you get near-instant syncs without the load.


Key takeaways

argocdgitopskubernetesrepo-serverapplication-controllerreconciliationredisdexssocrdakseksgketroubleshooting
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments