Every Argo CD platform eventually collides with one decision that nothing else can paper over: how do you lay out the Git repositories that Argo CD watches? Get it right and 40 teams ship independently while a single platform group keeps the fleet coherent. Get it wrong and you spend your quarters untangling merge conflicts, chasing broken ownership, and explaining why a one-line change to a shared label required 40 pull requests — or why one team’s typo just took down another team’s production namespace.
This lesson is deliberately not about where your application source code lives. Whether checkout and payments share a code monorepo or live in separate repos is a build-and-CI question, and Argo CD never looks at those repos. This lesson is about the config repos — the Git repositories that hold your Kubernetes manifests, Helm values, Kustomize overlays, Application/ApplicationSet/AppProject definitions — the repos Argo CD actually reconciles against live clusters. That distinction is the whole game, and half of all “monorepo vs polyrepo” arguments are two people talking about two different repos.
We will build the mental model from first principles, compare the three real patterns (single config monorepo, config polyrepo, and the hybrid), tie each to the exact ApplicationSet generator that makes it work, and then walk a concrete “100 apps, 40 teams” design end to end — including the CODEOWNERS files, the AppProject-per-team boundaries, and the blast-radius math. The hands-on lab scaffolds the hybrid layout on your laptop with no cluster required, so you can hold the structure in your hands before you commit an org to it.
Why this matters
Repo architecture is the decision with the longest half-life on a GitOps platform. You choose your CI trigger strategy in an afternoon and change it whenever; you choose your repo topology once, and by the time it hurts you have 100 apps, hundreds of Application objects, and 40 teams with muscle memory built around the current layout. Migrating a live fleet from one topology to another is a multi-quarter project with a change-freeze in the middle. This is the decision to make deliberately.
It matters because four things you care about are all downstream of it, and they pull in different directions:
| What you want | Monorepo pushes you toward | Polyrepo pushes you toward |
|---|---|---|
| Simple discovery — one place to see everything | Strong: one git clone, grep the fleet |
Weak: state is scattered across N repos |
| Clean ownership — each team owns its stuff, hard boundary | Weak: everyone shares one repo + one history | Strong: one repo per team, per-repo RBAC |
| Small blast radius — a bad change hurts little | Weak: one commit can touch everything | Strong: a bad change is trapped in one repo |
| Independent velocity — teams don’t block each other | Weak: shared branch protection, shared CI queue | Strong: each team’s cadence is its own |
There is no free lunch in that table. The entire craft of platform engineering at scale is buying back the column you sacrificed with a mechanism — recovering polyrepo-grade ownership inside a monorepo with path-scoped CODEOWNERS, or recovering monorepo-grade discovery across a polyrepo with an ApplicationSet SCM generator. The patterns below are named combinations of those buy-backs.
The mental model to anchor everything: Argo CD reconciles (source repo, path, target cluster, namespace) tuples. A repo topology is just a policy for how you slice those tuples across Git repositories, and — crucially — who is allowed to change which slice. Monorepo puts every slice in one repo; polyrepo gives each owner their own repo; hybrid splits by responsibility. Everything else in this lesson is detail on that one idea.
The two axes: app-code repos vs config repos
Before comparing patterns, pin down what we are even structuring. There are two independent repo decisions in any organization, and conflating them is the number-one source of confusion.
| App-code repo | Config / GitOps repo | |
|---|---|---|
| Holds | Source code, Dockerfile, unit tests, the app’s own Helm chart or Kustomize base | Rendered/overlay manifests, values files, Application/ApplicationSet/AppProject YAML |
| Consumed by | CI (build, test, push image) | Argo CD (repo-server renders it; application-controller applies it) |
| Changes on | Every feature commit | Every deploy/promotion (an image tag bump, a replica change) |
| Argo CD watches it? | No — Argo CD never builds images | Yes — this is the source of truth Argo CD reconciles |
| Mono-vs-poly here is about | Build tooling, code review, test isolation | Deploy blast radius, sync ownership, render cost |
The clean handoff between them is the boundary this whole course keeps returning to: CI builds and pushes an image, then writes a new image tag into the config repo; Argo CD notices the config repo changed and rolls it out. CI ends where the config repo begins. (This lesson lives entirely on the right-hand column; the app-code column is a CI concern.)
A quick vocabulary table, because “the repo” means different things to different roles and you will be in rooms with all of them:
| Term | What it precisely means here |
|---|---|
| Config monorepo | One Git repo holding the config for many/all apps, envs, and clusters |
| Config polyrepo | Many Git repos, each holding config for one team or one app |
| Hybrid | A platform/bootstrap monorepo plus per-team config repos, glued by ApplicationSet |
| Source of truth | The (repoURL, path, targetRevision) in an Application.spec.source |
| Blast radius | The set of live resources a single Git change can affect |
| Discovery | How an engineer (or the platform) finds “all apps and their desired state” |
With those fixed, the mono-vs-poly question at the config layer has exactly three serious answers. We take them one at a time, then compare them head to head.
Pattern 1: the single config monorepo
Everything Argo CD watches lives in one repository. All apps, all environments, all clusters, the ApplicationSets that generate them, and the AppProjects that fence them — one git clone, one history, one place.
A representative layout for a monorepo that Argo CD reconciles:
gitops/ # ← the ONE repo Argo CD watches
├── bootstrap/
│ └── root-app.yaml # app-of-apps root Application
├── applicationsets/
│ ├── apps.yaml # Git directory generator over apps/*
│ └── addons.yaml # cluster generator for shared add-ons
├── projects/
│ ├── team-payments.yaml # AppProject per team
│ └── team-checkout.yaml
├── apps/
│ ├── payments/
│ │ ├── base/ # Kustomize base (or a Helm chart ref)
│ │ └── overlays/
│ │ ├── dev/
│ │ ├── staging/
│ │ └── prod/
│ └── checkout/
│ ├── base/
│ └── overlays/{dev,staging,prod}/
├── addons/ # shared platform add-ons
│ ├── ingress-nginx/
│ ├── cert-manager/
│ └── kube-prometheus-stack/
└── .github/
└── CODEOWNERS # path-scoped ownership
Argo CD discovers apps by walking apps/* with a Git directory generator — add a directory, get an Application for free, no manual Application manifest:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: monorepo-apps
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: https://github.com/acme/gitops.git
revision: main
directories:
- path: apps/*
template:
metadata:
name: '{{.path.basename}}' # e.g. payments, checkout
annotations:
# scope webhook-triggered re-render to this app's own files
argocd.argoproj.io/manifest-generate-paths: '.'
spec:
project: '{{.path.basename}}'
source:
repoURL: https://github.com/acme/gitops.git
targetRevision: main
path: 'apps/{{.path.basename}}/overlays/prod'
destination:
server: https://kubernetes.default.svc
namespace: '{{.path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
The appeal is real and should not be undersold:
| Monorepo strength | Why it happens |
|---|---|
| Trivial discovery | Everything is one git clone; grep -r image: . shows every pinned tag in the fleet |
| Atomic cross-cutting changes | Bump a shared label, a common securityContext, or a Kustomize component across all apps in one PR, one commit, one revert |
| One source of truth for structure | Directory conventions, projects, and ApplicationSets are co-located and self-documenting |
| Simple onboarding of a repo | Nothing to provision — a new app is a new directory, not a new repo + webhook + credential |
| Consistent tooling | One CI config, one linter config, one policy set applies uniformly |
And the costs are equally real — they are exactly the things this lesson exists to mitigate:
| Monorepo cost | What it looks like in practice |
|---|---|
| Large blast radius | A bad edit to a shared base, or a fat-fingered kustomization.yaml, can render or sync-break many apps at once |
| CODEOWNERS complexity | One repo must encode 40 teams’ boundaries in one .github/CODEOWNERS, and “last match wins” bites the careless |
| Repo-server render cost | On a naive setup, a push anywhere can invalidate cache and re-render everything; big repos are slow to clone and manifest-generate |
| Shared history / merge pressure | 40 teams committing to main means more conflicts on shared files and a noisier history |
| Shared CI queue | Without path filtering, every PR runs the whole validation suite, and one team’s flaky check blocks the queue |
Three of those costs — render churn, blast radius, and CI — are performance and safety problems with well-known fixes (manifest-generate-paths, path-scoped CI, CODEOWNERS, sync waves), which we cover in their own sections. The monorepo is not a trap; it is a pattern that demands discipline in exchange for simplicity. For a small-to-medium platform it is very often the right first answer, and the honest guidance at the end of this lesson is: start here.
Pattern 2: the config polyrepo
Flip the model: every team (or, in the extreme, every app) gets its own config repository. Argo CD watches many repos, one owner per repo.
A per-team polyrepo layout, replicated across N repos:
team-payments-config/ # ← one repo, owned by the payments team
├── gitops/
│ ├── base/
│ └── overlays/{dev,staging,prod}/
├── .github/
│ └── CODEOWNERS # entire repo → @acme/payments-team
└── README.md
team-checkout-config/ # ← a second repo, owned by checkout
├── gitops/
│ └── ...
└── ...
The natural generator here is the SCM Provider generator: point it at the Git organization, and it creates exactly one Application per repo that matches your filters. A new team repo appears in the org and — with no platform edit — an Application materializes:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: team-config-repos
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- scmProvider:
github:
organization: acme-teams
allBranches: false
tokenRef:
secretName: github-scm-token
key: token
filters:
- repositoryMatch: '^team-.*-config$' # only *-config repos
pathsExist: ['gitops/kustomization.yaml'] # only if it looks like ours
template:
metadata:
name: '{{.repository}}'
spec:
project: '{{.repository}}' # project per repo
source:
repoURL: '{{.url}}'
targetRevision: '{{.branch}}'
path: gitops
destination:
server: https://kubernetes.default.svc
namespace: '{{.repository}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncPolicy:
applicationsSync: create-update # controller may create/update, NOT delete
preserveResourcesOnDeletion: false
Notice applicationsSync: create-update at the ApplicationSet level — that is a deliberate safety valve we return to under troubleshooting: it lets auto-onboarding create and update Applications but forbids the controller from deleting an app if a repo temporarily vanishes or stops matching.
Polyrepo’s strengths are the mirror image of the monorepo’s costs:
| Polyrepo strength | Why it happens |
|---|---|
| Hard ownership boundary | One repo per team means per-repo RBAC, per-repo CODEOWNERS, per-repo branch protection — no shared surface to fight over |
| Small blast radius | A bad change is physically confined to one repo; it cannot render-break another team’s app |
| Independent velocity | Each team’s CI, review cadence, and merge policy are entirely its own |
| Clean audit | “Who changed payments’ prod?” is answered by one repo’s history, not filtered out of a 40-team firehose |
| Least-privilege credentials | Argo CD can hold a narrowly-scoped deploy key per repo instead of one key to everything |
And its costs are the monorepo’s strengths, inverted:
| Polyrepo cost | What it looks like in practice |
|---|---|
| Discovery overhead | “Show me every app and its state” now means querying Argo CD or scanning N repos, not one grep |
| Consistency drift | 40 repos drift toward 40 slightly-different structures, README styles, and CI configs unless enforced |
| Cross-cutting change is N PRs | A fleet-wide label or security fix becomes 40 pull requests across 40 repos with 40 review cycles |
| Onboarding friction | Each new app/team needs a repo provisioned, credentials wired, webhook configured, conventions applied |
| Generator/credential sprawl | More repos to authenticate to, more webhooks, more places a secret can rot |
There is a sub-decision inside polyrepo worth calling out, because “one repo per app” and “one repo per team” behave very differently at 100 apps:
| Repo per team | Repo per app | |
|---|---|---|
| Repo count at 100 apps / 40 teams | ~40 repos | ~100+ repos |
| Ownership granularity | Team owns a repo of several apps | One owner per app, maximum isolation |
| Cross-app change within a team | One PR (apps co-located) | N PRs even inside one team |
| Discovery / credential sprawl | Moderate | High |
| Best when | Teams own coherent app groups | Apps are truly independent products |
Repo-per-team is the pragmatic middle of polyrepo; repo-per-app is maximal isolation you pay for in sprawl. For 100 apps and 40 teams, repo-per-team is almost always the polyrepo variant that survives contact with reality.
Pattern 3: the hybrid
The hybrid is not a compromise so much as a division by responsibility, and it is where most enterprises actually land. There are two kinds of content on a GitOps platform with fundamentally different owners and change patterns:
- Platform content — the app-of-apps root, the ApplicationSets, the AppProjects, shared add-ons (ingress controller, cert-manager, the metrics stack), org-wide policy. Owned by one platform team. Changes rarely, must be coherent, benefits from being in one place.
- App content — each team’s own workload manifests, values, overlays. Owned by each team. Changes constantly, must be isolated, benefits from per-team boundaries.
Hybrid puts platform content in a monorepo (best-in-class discovery and atomic changes for the small platform team) and app content in per-team polyrepos (best-in-class ownership and blast radius for the many app teams). The glue is app-of-apps plus an ApplicationSet SCM generator.
platform-gitops/ # ← MONOREPO, owned by @acme/platform-team
├── bootstrap/
│ └── root-app.yaml # app-of-apps: syncs everything below
├── applicationsets/
│ ├── team-repos.yaml # SCM generator → one app per team repo
│ └── addons.yaml # shared add-ons across clusters
├── projects/
│ ├── team-payments.yaml # AppProject per team (the fence)
│ └── team-checkout.yaml
├── addons/
│ ├── ingress-nginx/
│ └── cert-manager/
└── .github/CODEOWNERS # whole repo → platform team
team-payments-config/ # ← POLYREPO, owned by @acme/payments-team
└── gitops/{base,overlays/*}/
team-checkout-config/ # ← POLYREPO, owned by @acme/checkout-team
└── gitops/{base,overlays/*}/
The app-of-apps root bootstraps the whole platform layer in one apply:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: bootstrap
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: platform
source:
repoURL: https://github.com/acme/platform-gitops.git
targetRevision: main
path: bootstrap
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
That root points at bootstrap/, which contains child Applications for applicationsets/, projects/, and addons/. The ApplicationSets then fan out: the SCM generator (shown in Pattern 2) discovers every team-*-config repo and renders one Application each, every app pinned to its team’s AppProject. This is exactly the structure the diagram below walks.
| Hybrid property | Where it comes from |
|---|---|
| Small platform team gets a monorepo | Platform content is co-located, atomic, easy to grep and reason about |
| Many app teams get isolation | Each app team owns one repo, one CODEOWNERS, one blast radius |
| Zero-touch onboarding | SCM generator auto-onboards a new team repo; platform team does nothing |
| Central policy, distributed apps | AppProjects, add-ons, and org policy stay platform-owned; workloads stay team-owned |
| Ownership cuts cleanly | Repos, CODEOWNERS, and Argo CD RBAC all cut along the same platform-vs-team line |
The trade-off hybrid does not escape: a truly fleet-wide change to app content (say, a new required label on every workload) is still N PRs across N team repos — because that is app content, and app content is distributed on purpose. Hybrid buys you a small, coherent platform core; it does not abolish the polyrepo tax on cross-cutting app changes. What it does is make sure that tax only applies to genuine app-level changes, while everything platform-level stays a single atomic PR.
Here is the hybrid structure end to end — read it left to right, from who owns what, through the repos and generators, to the AppProject fence and the clusters where workloads land:
The badges mark where each pattern is won or lost: the monorepo’s atomic-but-broad blast radius (1), the polyrepo’s hard per-repo ownership line (2), the SCM generator auto-onboarding a repo with zero platform edits (3), the AppProject as the real sync-time fence (4), the blast radius held at the project boundary so a bad change never crosses into another team (5), and the org chart driving repos plus RBAC along one line (6). If you internalize only the diagram, internalize this: CODEOWNERS gates the pull request; the AppProject gates the sync. You need both, and they are different mechanisms.
The big comparison
Everything above, in one place. This is the table to bring to the architecture review.
| Dimension | Single config monorepo | Config polyrepo | Hybrid |
|---|---|---|---|
| Discovery | Best — one clone, grep the fleet | Worst — scattered across N repos | Good — platform central, apps via Argo CD |
| Ownership boundary | Weak — path-scoped CODEOWNERS only | Strong — one repo per owner | Strong for apps, central for platform |
| Blast radius | Large — one commit can touch all | Small — trapped in one repo | Small for apps, contained for platform |
| Cross-cutting platform change | 1 PR | N PRs | 1 PR (platform is a monorepo) |
| Cross-cutting app change | 1 PR | N PRs | N PRs (apps are distributed) |
| Review / RBAC granularity | Path-level in one repo | Repo-level, cleanest | Repo-level for apps, central for platform |
| repo-server render cost | Highest — needs manifest-generate-paths |
Lowest — small repos | Low — small app repos, small platform repo |
| CI complexity | Needs path filtering to scale | Simple per repo, N pipelines to maintain | Simple per app repo + one platform pipeline |
| Onboarding a new app | Add a directory | Provision a repo + creds + webhook | Add a repo; SCM generator does the rest |
| Consistency enforcement | Easy — one place | Hard — N repos drift | Medium — platform enforces conventions |
| Merge-conflict pressure | Highest on shared files | Lowest | Low |
| Best team size | 1 small team, or few teams | Many independent teams | 1 platform team + many app teams |
The same information as a decision aid — pick by the strongest signal in your org, not by fashion:
| If your org looks like this… | Lean toward |
|---|---|
| One platform team, <15 apps, everyone trusts everyone | Monorepo — simplest thing that works |
| Strict compliance isolation between teams; a shared repo is a non-starter | Polyrepo (repo per team) |
| Teams must not even see each other’s config (regulated / multi-tenant SaaS) | Polyrepo (repo per app or per team) |
| A platform team plus many product teams, growing fast | Hybrid — the enterprise default |
| You genuinely can’t decide | Start monorepo, plan the seam — split to hybrid when ownership pain arrives |
And the blast-radius scenarios spelled out, because “blast radius” is abstract until it is your outage:
| A change to… | Monorepo can affect | Polyrepo can affect | Hybrid can affect |
|---|---|---|---|
| One app’s overlay | Just that app (if CI/CODEOWNERS scoped) | Just that app’s repo | Just that app’s repo |
| A shared Kustomize base | Every app using the base | N/A (bases aren’t shared) | N/A (apps own their bases) |
| An ApplicationSet template | Every app it generates | Every app it generates | Every app it generates |
| An AppProject | Every app in that project | Every app in that project | Every app in that project |
| A shared add-on (ingress) | Every cluster running it | Every cluster running it | Every cluster running it |
Read that table twice: ApplicationSets, AppProjects, and shared add-ons are wide-blast-radius objects in every topology. No repo split saves you there — those are guarded by RBAC, CODEOWNERS on the platform repo, and progressive rollout, not by which repo they live in. Polyrepo shrinks the blast radius of app config; it does nothing for the shared control-plane objects, which is precisely why the hybrid keeps those in a tightly-governed platform monorepo.
ApplicationSet and app-of-apps as the glue
Each repo pattern maps to a specific generator. Choosing the topology is choosing the generator; they are two names for one decision. (For the generator mechanics themselves, see ApplicationSets and generators and, for the bootstrap parent, the app-of-apps pattern.)
| Repo pattern | Primary generator | How discovery works | Onboarding a new app |
|---|---|---|---|
| Monorepo | Git directory generator over apps/* |
Controller lists directories in the one repo | Create a directory; app appears |
| Polyrepo | SCM Provider generator over the org | Controller queries the Git provider API for repos | Create a matching repo; app appears |
| Hybrid | app-of-apps → SCM Provider (+ Git dir for add-ons) | Root app syncs ApplicationSets, which fan out | Create a team repo; SCM generator onboards it |
The Git directory generator (monorepo) keys off filesystem structure; the SCM Provider generator (polyrepo/hybrid) keys off the existence of repos. Their template parameters differ, and mixing them up is a common first-day error:
| Generator | Key template params (goTemplate) | Selection mechanism |
|---|---|---|
| Git directories | .path.path, .path.basename, .path.basenameNormalized |
directories: [{path: apps/*}], optional exclude |
| Git files | keys from each matched JSON/YAML file’s contents | files: [{path: apps/**/config.json}] |
| SCM Provider | .organization, .repository, .url, .branch, .sha, .labels |
filters: — repositoryMatch, pathsExist, branchMatch, labelMatch |
| Pull request | .number, .branch, .head_sha |
per-provider PR query + labels |
The SCM Provider generator’s filters are your onboarding gate, and getting them right is the difference between “a new team repo just works” and “a random fork in the org spawned a rogue Application.” The filters that matter:
| Filter field | What it does | Why you want it |
|---|---|---|
repositoryMatch |
Regex the repo name must match | Restrict to ^team-.*-config$, not every repo in the org |
pathsExist |
Only match repos that contain these path(s) | Require gitops/kustomization.yaml — proves it’s really one of ours |
pathsDoNotExist |
Exclude repos containing a path | Skip archived repos carrying a .archived marker |
branchMatch |
Regex the branch must match | Only track main, not every feature branch |
labelMatch |
Match a repo topic/label | Onboard only repos tagged argocd-managed |
Because the SCM Provider generator talks to a Git provider API, this is one of the few genuinely provider-specific edges in an otherwise cloud-neutral lesson. Repo architecture itself is the same on any Kubernetes — AKS, EKS, GKE, on-prem — because Argo CD reconciles Git against clusters identically everywhere. But which SCM block you write depends on where the config repos are hosted:
| Git host | Generator block | Auth (tokenRef secret) |
Note |
|---|---|---|---|
| GitHub / GHES | github: (organization, optional api: for Enterprise) |
PAT or GitHub App token | allBranches: false unless you need every branch |
| GitLab | gitlab: (group, includeSubgroups) |
Group/project access token | includeSubgroups: true to walk nested groups |
| Gitea | gitea: (owner, api) |
Gitea token | Self-hosted; set api to your instance URL |
| Azure DevOps | azureDevOps: (organization, teamProject) |
Azure DevOps PAT | Scoped per project |
| Bitbucket Server | bitbucketServer: (project, api) |
HTTP access token | Data Center / Server |
| Bitbucket Cloud | bitbucketCloud: (owner) |
App password | SaaS Bitbucket |
The cluster the workloads land on can be AKS, EKS, GKE, or anything else, selected by the destination — a hybrid fleet routinely spans all three, and a cluster generator (or a matrix of SCM × clusters) fans one team’s app across every matching cluster. The repo topology does not change when the clouds do; only the scmProvider block and the destinations do.
For the hybrid, you often combine generators. A matrix of the SCM Provider generator crossed with a cluster generator gives you “every team repo, on every matching cluster,” while a separate Git directory generator over the platform monorepo’s addons/ handles shared platform components. Keep the two concerns in two ApplicationSets — do not try to make one generator do both jobs.
Team ownership: CODEOWNERS, branch protection, AppProject-per-team
Ownership is enforced at two independent gates, and a mature platform uses both because they catch different failures:
- The pull-request gate — CODEOWNERS + branch protection. Stops an unreviewed change from merging. Lives in the Git provider.
- The sync gate — the AppProject. Stops a merged change from syncing outside its allow-list. Lives in Argo CD.
CODEOWNERS without an AppProject means a compromised or mis-scoped repo can still push resources into another team’s namespace once it’s merged. An AppProject without CODEOWNERS means anyone who can merge can change the desired state unreviewed. You need both.
CODEOWNERS: path-scoped in a monorepo, whole-repo in a polyrepo
In a monorepo, one CODEOWNERS file encodes every team’s boundary by path. The single most important rule, and a frequent outage cause: in GitHub CODEOWNERS the last matching pattern wins, so order from general to specific.
# platform-gitops/.github/CODEOWNERS
# Last match wins — general rules first, specific paths override below.
# Default: the platform team owns anything not otherwise claimed.
* @acme/platform-team
# Shared control-plane objects stay platform-owned even if teams touch nearby.
/applicationsets/ @acme/platform-team
/projects/ @acme/platform-team
/addons/ @acme/platform-team
# Per-team app directories: the team owns their app; platform co-reviews.
/apps/payments/ @acme/payments-team @acme/platform-team
/apps/checkout/ @acme/checkout-team @acme/platform-team
In a polyrepo, CODEOWNERS is trivial because the repo is the boundary — the whole repo belongs to one team:
# team-payments-config/.github/CODEOWNERS
* @acme/payments-team
That single-line polyrepo CODEOWNERS is precisely the ownership win, made concrete: the repo is the boundary, so there is one trivial file, no ordering to get wrong, and “who owns X?” is always “the repo owner.” The monorepo buys back the same isolation with path rules, but path rules are fiddly, onboarding a team means editing shared rules, and the “last match wins” ordering trips people up — a real failure mode we return to under troubleshooting.
Branch protection
CODEOWNERS only requests review; branch protection makes it required. Without it, CODEOWNERS is advisory and anyone with write access can merge unreviewed. The settings that matter for a GitOps config repo:
| Branch-protection setting | Why it matters for config repos |
|---|---|
| Require pull request before merging | No direct pushes to the branch Argo CD tracks |
| Require review from Code Owners | Makes CODEOWNERS enforced, not advisory |
| Require status checks (CI) to pass | Lint/kubeconform/policy gates run before merge |
| Dismiss stale approvals on new commits | A post-approval push can’t sneak in unreviewed |
| Restrict who can push / include administrators | Even admins go through review; no bypass |
| Require linear history / signed commits | Cleaner audit; provenance on the source of truth |
The AppProject: the sync-time fence
The AppProject is Argo CD’s multi-tenancy boundary and the second gate. It restricts which repos an app may deploy from, which clusters/namespaces it may deploy to, and which resource kinds it may touch — enforced at sync time by the controller, independent of Git. (This lesson uses AppProjects as a boundary; for the full treatment of project-based multi-tenancy, that is its own dedicated topic in the course.)
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-payments
namespace: argocd
spec:
description: Payments squad — apps, staging + prod-eu only
sourceRepos:
- https://github.com/acme-teams/team-payments-config.git # ONLY this repo
destinations:
- name: prod-eu
namespace: 'payments-*' # ONLY these namespaces
- name: staging
namespace: 'payments-*'
clusterResourceWhitelist: [] # deny ALL cluster-scoped kinds
namespaceResourceBlacklist:
- group: ""
kind: ResourceQuota # can't rewrite their own quota
- group: ""
kind: LimitRange
roles:
- name: deployer
policies:
- p, proj:team-payments:deployer, applications, sync, team-payments/*, allow
- p, proj:team-payments:deployer, applications, get, team-payments/*, allow
groups:
- acme:payments-engineers
The four fields that do the fencing:
| AppProject field | What it constrains | Set it to… |
|---|---|---|
sourceRepos |
Which Git repos apps in this project may use | Exactly the team’s repo(s); never '*' in prod |
destinations |
Which (cluster, namespace) pairs are allowed |
The team’s clusters + a team-* namespace glob |
clusterResourceWhitelist |
Which cluster-scoped kinds are allowed | [] (empty = none) unless the team truly needs them |
namespaceResourceBlacklist |
Namespaced kinds to forbid | ResourceQuota, LimitRange — platform owns those |
An empty clusterResourceWhitelist plus a namespace-scoped destinations glob is the single highest-leverage guardrail on the whole platform: a compromised or fat-fingered app repo cannot create a ClusterRole, escape its namespace, or deploy to another team’s cluster — the project rejects the sync before anything is applied. The RBAC layer on top maps SSO groups to project-scoped actions:
| Role | Applications create/delete | Applications sync/get | Scope |
|---|---|---|---|
| Platform admin | Yes | Yes | All projects |
App team (deployer) |
No (apps come from the ApplicationSet) | Yes | Their project only |
| Everyone else | No | Read-only | All (read) |
App teams deliberately get sync/get but not create/delete on Applications: the Applications are generated by the platform-owned ApplicationSet, so a team changes desired state in Git, not the Application objects themselves. That keeps the generator authoritative and prevents a team from hand-editing its way out of the project boundary.
Blast radius and how to contain it
Blast radius is the set of live resources one Git change can affect. The monorepo’s central weakness is a wide default blast radius; the mitigations below shrink it back toward polyrepo levels without giving up the monorepo’s discovery. Each control contains a different class of failure — layer them.
| Control | Contains which failure | Where it lives |
|---|---|---|
| Path-scoped CI | A malformed manifest in app A failing/altering app B’s pipeline | Git provider CI (path filters) |
| CODEOWNERS | An unreviewed change to a shared file or another team’s app | Git provider PR gate |
| AppProject | A merged change trying to sync outside its repo/cluster/namespace allow-list | Argo CD sync gate |
manifest-generate-paths |
A push to app A needlessly re-rendering (and re-syncing risk) app B | Argo CD repo-server |
| Sync waves | Applying a dependent resource before its prerequisite within one app | Argo CD sync ordering |
ApplicationSet RollingSync |
A generator change flipping all generated apps at once | ApplicationSet controller |
applicationsSync: create-update |
The controller deleting apps when a repo vanishes/stops matching | ApplicationSet syncPolicy |
Path-scoped CI is the first line and the cheapest. In a monorepo, run each app’s validation only when its files change, so team A’s flaky test never blocks team B and a broken overlay is caught before merge without dragging the whole repo through CI:
# .github/workflows/validate.yml — only lint the app whose files changed
on:
pull_request:
paths:
- 'apps/payments/**' # this job runs only for payments changes
jobs:
validate-payments:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Render + validate
run: |
kustomize build apps/payments/overlays/prod | \
kubeconform -strict -summary
For fleets with dozens of apps, a dynamic matrix (compute the changed app directories, then fan out one validation job per changed app) scales this without hand-writing 100 path filters. The principle is the same: CI blast radius should equal the change’s blast radius.
The ApplicationSet RollingSync strategy contains the widest monorepo failure — a change to a generator template or a shared value that would otherwise flip every generated Application simultaneously. It rolls the change through in labeled waves and waits between them (for cross-cluster fleet rollout mechanics, see scaling and sharding):
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: team-config-repos
namespace: argocd
spec:
strategy:
type: RollingSync
rollingSync:
steps:
- matchExpressions:
- key: envLabel
operator: In
values: [canary] # a handful of apps first
- matchExpressions:
- key: envLabel
operator: In
values: [prod] # then the rest, only if canary is healthy
# ... generators + template as before ...
RollingSync (progressive syncs) must be enabled on the ApplicationSet controller — it has historically been behind the --enable-progressive-syncs flag / a feature-flag env var, and moved toward default-on in the 2.13+/3.x line. Confirm it’s enabled in your version before relying on it; if it isn’t, the strategy block is silently ignored and everything syncs at once. That “silently ignored” failure mode is exactly why you verify the flag, not just the manifest.
The promotion of a version through environments (dev → staging → prod) is a related but distinct concern — that ordering is about which revision each environment tracks, and is covered in environment promotion. Blast-radius containment is about limiting how much a single change can touch; promotion is about when a known-good change advances.
Performance at scale
The monorepo’s other tax is performance, and it is entirely a repo-server story. The repo-server clones the repo and renders manifests; the application-controller reconciles the rendered result against clusters. A big monorepo makes the repo-server work harder, and a naive setup makes it re-render far more than necessary. (This section is a summary tuned to repo topology; the full scaling treatment — controller sharding, repo-server sizing, cache internals — is in scaling, sharding, and monorepo performance.)
What actually costs you in a monorepo:
| Cost driver | Why the monorepo makes it worse | Lever |
|---|---|---|
| Clone time | One large repo, deep history, many files | Shallow clones (default), fewer large binaries in-repo |
| Manifest generation | A naive push re-renders every app | manifest-generate-paths scopes re-render to changed paths |
| Cache invalidation | One commit SHA covers the whole repo | manifest-generate-paths keeps unchanged apps on cached render |
| Reconcile fan-out | Many Applications share one repoURL | Webhooks (not polling) + repo-server replicas |
| Polling load | Default ~3-minute poll × hundreds of apps | Configure a Git webhook so refresh is event-driven |
The highest-leverage monorepo-specific lever is the manifest-generate-paths annotation. It tells Argo CD which paths in the repo actually affect an Application’s rendered output, so on a webhook push the repo-server only re-renders apps whose relevant paths changed — turning “a commit to app A re-renders all 100 apps” into “a commit to app A re-renders app A”:
metadata:
annotations:
# semicolon-separated; '.' = this app's own source path,
# add shared paths this app depends on (leading '/' = repo root)
argocd.argoproj.io/manifest-generate-paths: '.;/shared/common-labels'
Include every path the render truly depends on (the app’s own directory plus any shared base or component it references), or a change to that shared path won’t trigger a re-render and the app will silently miss it. This annotation only takes effect with webhooks; under plain polling Argo CD re-checks everything anyway.
The scaling knobs, summarized for the topology decision:
| Knob | What it scales | When the monorepo needs it |
|---|---|---|
| Webhook (vs polling) | Refresh latency + repo-server load | Always, past a handful of apps |
manifest-generate-paths |
Re-render scope per push | Any monorepo with many apps |
| repo-server replicas | Parallel render throughput | Many concurrent syncs / large renders |
--parallelismlimit |
Concurrent manifest jobs per repo-server | Render CPU saturation |
Controller sharding (--replicas) |
Reconcile load across clusters | Many clusters (shards by cluster, not app) |
One nuance that trips up monorepo operators: application-controller sharding shards by cluster, not by app. Splitting one giant monorepo across more controller shards does nothing if all its apps target one cluster — that load is on the repo-server (render) and is scaled with repo-server replicas and manifest-generate-paths, not controller shards. Match the lever to the bottleneck: render cost → repo-server; reconcile cost across many clusters → controller shards.
Polyrepo and hybrid sidestep much of this by construction: small repos clone fast, render cheap, and a push touches one small repo. That is a genuine performance argument for splitting — but it is usually not the deciding factor, because manifest-generate-paths plus webhooks tame the monorepo well past 100 apps. Split for ownership and blast radius; treat the performance improvement as a bonus, not the reason.
The “100 apps, 40 teams” worked design
Now the concrete recommendation, with the reasoning, for the canonical scale in this lesson’s title. The answer is hybrid, and here is exactly why and how.
Why hybrid, not monorepo: 40 teams sharing one repo means 40 teams’ CODEOWNERS rules in one file, 40 teams’ CI in one queue, and one shared history — the ownership and merge pressure alone justify splitting the app content out. Why hybrid, not full polyrepo: the platform content (ApplicationSets, AppProjects, shared add-ons) must stay coherent and change atomically; scattering it across 40 repos would make a simple org-wide policy change a 40-PR nightmare and destroy discoverability of the control plane. Hybrid puts each kind of content where it belongs.
The repo inventory:
| Repo | Count | Owner | Holds | Generator |
|---|---|---|---|---|
platform-gitops |
1 | Platform team | app-of-apps root, ApplicationSets, AppProjects, shared add-ons | app-of-apps + Git dir (add-ons) |
team-<name>-config |
~40 | Each app team | That team’s app overlays/values | SCM Provider (auto-onboarded) |
| App source repos | ~100 | Service teams | Code + chart/base (CI territory) | none (Argo CD ignores these) |
The onboarding flow that makes 40 teams scale to a 1-person-day platform effort:
| Step | Who | Action |
|---|---|---|
| 1 | App team | Create team-<name>-config from a template repo (structure + CODEOWNERS baked in) |
| 2 | Platform (once) | Commit an AppProject for the team to platform-gitops/projects/ |
| 3 | SCM generator | Auto-detects the new repo (name + pathsExist match) → renders one Application |
| 4 | AppProject | Fences that Application to the team’s repo, clusters, and namespaces |
| 5 | App team | Commits overlays; Argo CD syncs; team ships independently |
Steps 3–4 are the payoff: after the one-time project commit, the platform team does nothing to onboard each new app the team adds — the SCM generator and the project boundary do the work. The template repo in step 1 is what keeps 40 repos from drifting into 40 structures.
The sizing/scaling plan for this fleet:
| Concern | Setting for ~100 apps / 40 teams |
|---|---|
| Refresh | Git webhooks on all repos (no polling at this scale) |
| Render scope | manifest-generate-paths on every generated Application |
| repo-server | 3+ replicas behind the service; raise --parallelismlimit if render-bound |
| Controller shards | Shard by cluster if the fleet spans many clusters (AKS + EKS + GKE) |
| Onboarding safety | SCM filters (name regex + pathsExist) + applicationsSync: create-update |
| Rollout safety | RollingSync on any ApplicationSet whose template many apps share |
| Boundaries | One AppProject per team; empty clusterResourceWhitelist; namespace glob |
This design gives the platform team a single coherent repo they can reason about, gives 40 app teams hard isolation and independent velocity, auto-onboards new work with zero platform toil, and holds every team’s blast radius at the project boundary. It is the shape most enterprises converge on, and it is worth building deliberately rather than discovering by accident.
Repo hygiene and conventions
Whatever topology you pick, consistency is what keeps it legible at 100 apps. These conventions are cheap to adopt early and painful to retrofit late.
| Convention | Recommendation |
|---|---|
| Repo naming | Predictable and matchable: team-<name>-config, platform-gitops. The SCM generator’s regex depends on it |
| Top-level layout | Stable apps/, addons/, projects/, applicationsets/, bootstrap/ — never improvise per repo |
| Env layout | base/ + overlays/{dev,staging,prod}/; overlays hold only the delta, never a full copy |
| One engine per app | Helm or Kustomize per app, never both; keep the override surface small and reviewable |
Pin targetRevision |
Prod tracks a tag or SHA, never a moving branch — a merge elsewhere shouldn’t auto-ship |
README.md per repo |
What it deploys, which clusters, who owns it, how to add an app |
| CODEOWNERS present | Every config repo has one; monorepo path-scoped, polyrepo whole-repo |
| Template repo | New team repos are created from a template so structure/CODEOWNERS/CI are uniform from day one |
The /apps /envs /clusters question comes up constantly, so make it explicit. There are two common axes for organizing overlays, and picking one and sticking to it matters more than which you pick:
| Layout axis | Structure | Best when |
|---|---|---|
| App-major | apps/<app>/overlays/<env>/ |
Teams own apps; most changes are per-app (most common) |
| Env-major | envs/<env>/<app>/ |
You promote whole environments together and want per-env review |
| Cluster-major | clusters/<cluster>/<app>/ |
Cluster topology dominates (edge fleets, per-cluster divergence) |
App-major is the default for the hybrid because it aligns the directory tree with the ownership tree — a team’s stuff is contiguous, which makes CODEOWNERS, path-scoped CI, and manifest-generate-paths all trivially clean. Reach for env-major or cluster-major only when your dominant change pattern genuinely runs along that axis.
Hands-on lab
You will design and scaffold the hybrid layout on your laptop — no cluster required. We build a platform monorepo (app-of-apps + a shared add-on), a per-team config repo, wire an ApplicationSet SCM generator to auto-onboard team repos, add CODEOWNERS and an AppProject per team, then prove how a team’s change is scoped. Because there is no cluster here, we validate the structure and manifests (which is where topology bugs actually live) and label all command output as representative. Everything is copy-pasteable.
Prerequisites: bash, python3. We optionally use PyYAML to lint manifests:
python3 -m pip install --quiet pyyaml # optional; the lint step uses it
Step 1 — Scaffold the platform monorepo.
mkdir -p ~/argocd-repos-lab && cd ~/argocd-repos-lab
mkdir -p platform-gitops/{bootstrap,applicationsets,projects,addons/ingress-nginx,.github}
What just happened: you created the platform-owned monorepo skeleton — bootstrap/ for the app-of-apps root, applicationsets/ for the generators, projects/ for the per-team fences, and addons/ for shared components. This is the coherent core the platform team owns.
Step 2 — Write the app-of-apps root.
cat > platform-gitops/bootstrap/root-app.yaml <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: bootstrap
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: platform
source:
repoURL: https://github.com/acme/platform-gitops.git
targetRevision: main
path: bootstrap
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated: { prune: true, selfHeal: true }
YAML
What just happened: one root Application that, once applied to a cluster, syncs everything else in the platform repo. This is the single bootstrap seam — apply it once and the platform layer materializes.
Step 3 — Write the SCM Provider generator (the auto-onboarding glue).
cat > platform-gitops/applicationsets/team-repos.yaml <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: team-config-repos
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- scmProvider:
github:
organization: acme-teams
allBranches: false
tokenRef:
secretName: github-scm-token
key: token
filters:
- repositoryMatch: '^team-.*-config$'
pathsExist: ['gitops/kustomization.yaml']
template:
metadata:
name: '{{.repository}}'
spec:
project: '{{.repository}}'
source:
repoURL: '{{.url}}'
targetRevision: '{{.branch}}'
path: gitops
destination:
server: https://kubernetes.default.svc
namespace: '{{.repository}}'
syncPolicy:
automated: { prune: true, selfHeal: true }
syncPolicy:
applicationsSync: create-update
preserveResourcesOnDeletion: false
YAML
What just happened: this ApplicationSet will discover any team-*-config repo in the acme-teams org that contains gitops/kustomization.yaml, and render one Application per repo — each pinned to a project named after the repo. applicationsSync: create-update means a repo disappearing can’t make the controller delete a live app. This is the zero-touch onboarding mechanism.
Step 4 — Add a shared add-on and the platform CODEOWNERS.
cat > platform-gitops/addons/ingress-nginx/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/kubernetes/ingress-nginx//deploy/static/provider/cloud?ref=controller-v1.11.2
YAML
cat > platform-gitops/.github/CODEOWNERS <<'EOF'
# Last match wins — general first, specific overrides below.
* @acme/platform-team
/applicationsets/ @acme/platform-team
/projects/ @acme/platform-team
/addons/ @acme/platform-team
EOF
What just happened: the platform repo now owns a shared ingress add-on and declares that the whole repo — especially the control-plane directories — is platform-owned. In a real repo, branch protection with “require review from Code Owners” makes this enforced rather than advisory.
Step 5 — Write the AppProject that fences a team.
cat > platform-gitops/projects/team-payments.yaml <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-payments-config
namespace: argocd
spec:
description: Payments squad
sourceRepos:
- https://github.com/acme-teams/team-payments-config.git
destinations:
- name: staging
namespace: 'payments-*'
- name: prod-eu
namespace: 'payments-*'
clusterResourceWhitelist: []
namespaceResourceBlacklist:
- group: ""
kind: ResourceQuota
roles:
- name: deployer
policies:
- p, proj:team-payments-config:deployer, applications, sync, team-payments-config/*, allow
- p, proj:team-payments-config:deployer, applications, get, team-payments-config/*, allow
groups:
- acme:payments-engineers
YAML
What just happened: the payments team’s Application (named team-payments-config by the generator, matching project: '{{.repository}}') is now fenced: it may only source from the payments repo, only deploy into payments-* namespaces on staging/prod-eu, and may not create any cluster-scoped resource. This is the sync-time boundary.
Step 6 — Scaffold a team’s polyrepo.
mkdir -p team-payments-config/{gitops/base,gitops/overlays/prod,.github}
cat > team-payments-config/gitops/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- overlays/prod
YAML
cat > team-payments-config/gitops/base/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
YAML
cat > team-payments-config/gitops/base/deployment.yaml <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
spec:
replicas: 2
selector: { matchLabels: { app: payments-api } }
template:
metadata: { labels: { app: payments-api } }
spec:
containers:
- name: api
image: ghcr.io/acme/payments-api:1.4.0
YAML
cat > team-payments-config/gitops/overlays/prod/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: ghcr.io/acme/payments-api
newTag: 1.4.0
YAML
cat > team-payments-config/.github/CODEOWNERS <<'EOF'
* @acme/payments-team
EOF
What just happened: the team’s repo has the gitops/kustomization.yaml the SCM filter requires, a base + prod overlay, and a whole-repo CODEOWNERS. The moment this repo lands in the org, the generator from Step 3 onboards it — no platform edit.
Step 7 — Lint every manifest (representative validation).
python3 - <<'PY'
import glob, sys
try:
import yaml
except ImportError:
print("PyYAML not installed; skipping (pip install pyyaml to enable)"); sys.exit(0)
ok, bad = 0, 0
for f in glob.glob("**/*.yaml", recursive=True):
try:
docs = [d for d in yaml.safe_load_all(open(f)) if d]
for d in docs:
assert "apiVersion" in d and "kind" in d, f"missing apiVersion/kind in {f}"
ok += len(docs)
except Exception as e:
print(f"FAIL {f}: {e}"); bad += 1
print(f"\nvalidated {ok} manifests across the hybrid layout — {bad} file(s) failed")
PY
Representative output:
validated 8 manifests across the hybrid layout — 0 file(s) failed
What just happened: every YAML in both repos parses and carries a real apiVersion/kind. This is the check that catches the topology bugs that actually bite — a malformed kustomization.yaml, a missing field — before a cluster ever sees them. (On a real cluster you would additionally run kustomize build ... | kubeconform -strict and argocd app diff.)
Step 8 — Prove a team’s change is scoped.
# Simulate the payments team bumping their image tag.
sed -i.bak 's/newTag: 1.4.0/newTag: 1.4.1/' \
team-payments-config/gitops/overlays/prod/kustomization.yaml
rm -f team-payments-config/gitops/overlays/prod/kustomization.yaml.bak
# What did the change touch? Only the payments repo.
echo "Files changed by this deploy:"
echo " team-payments-config/gitops/overlays/prod/kustomization.yaml"
echo
echo "Gates this change must pass:"
echo " 1. CODEOWNERS: team-payments-config/* -> @acme/payments-team (PR review)"
echo " 2. AppProject: sourceRepos allows ONLY team-payments-config.git (sync)"
echo " 3. Destination: only payments-* namespaces on staging/prod-eu (sync)"
Representative output:
Files changed by this deploy:
team-payments-config/gitops/overlays/prod/kustomization.yaml
Gates this change must pass:
1. CODEOWNERS: team-payments-config/* -> @acme/payments-team (PR review)
2. AppProject: sourceRepos allows ONLY team-payments-config.git (sync)
3. Destination: only payments-* namespaces on staging/prod-eu (sync)
What just happened: a payments deploy touched exactly one file in exactly one repo, gated by the payments CODEOWNERS at merge time and by the payments AppProject at sync time. There is no path by which this change reaches checkout’s namespace or another team’s cluster — the blast radius is the payments repo, full stop. That is the whole design working: CODEOWNERS gated the PR, the AppProject gated the sync.
Teardown.
cd ~ && rm -rf ~/argocd-repos-lab
What just happened: everything was local files, so cleanup is one rm -rf — nothing billed, nothing left in a cluster.
Common mistakes and troubleshooting
The failures below are the ones that actually cost teams time when they scale a repo topology. The nastiest are discussed in prose after the table.
| Symptom | Likely cause | Fix |
|---|---|---|
| One bad monorepo commit breaks/re-syncs many apps | No path-scoped CI and wide render blast radius | Add path-filtered CI per app; add manifest-generate-paths; scope CODEOWNERS |
| Monorepo syncs are slow; repo-server CPU pinned | Every push re-renders every app | Set argocd.argoproj.io/manifest-generate-paths; enable webhooks; add repo-server replicas |
| A fleet-wide app change means 40 PRs | Working as designed for polyrepo app content | Accept it, or move the shared piece into a platform-owned component/base in the monorepo |
| New team repo never becomes an Application (polyrepo drift) | SCM generator filters don’t match, or no webhook | Check repositoryMatch regex + pathsExist; verify org token scope; confirm requeue/webhook |
| Unreviewed change merged to another team’s path | CODEOWNERS present but not enforced | Turn on branch protection “require review from Code Owners”; check “last match wins” ordering |
| A team deployed into another team’s namespace | AppProject destinations/sourceRepos too loose (e.g. '*') |
Tighten to the team’s repo + team-* namespace glob; empty clusterResourceWhitelist |
Application references a repo the project forbids |
Generated app’s project doesn’t permit its repoURL |
Fix sourceRepos on the AppProject, or template the correct project |
| SCM generator onboarded a rogue/broken repo | Filters too broad; applicationsSync allows delete/update freely |
Add pathsExist + tighter repositoryMatch; set applicationsSync: create-update |
| Two teams’ PRs conflict on a shared monorepo file | Shared surface + no ownership split on that file | Split the shared file, or make it platform-owned via CODEOWNERS + a component |
| 40 polyrepos have 40 different structures | No template repo / no enforced conventions | Create team repos from a template; lint structure in CI; document the layout |
RollingSync steps ignored; all apps flip at once |
Progressive syncs not enabled on the controller | Enable --enable-progressive-syncs (or the feature-flag env) and redeploy the controller |
ComparisonError / rpc error: failed to get repo after split |
Argo CD lacks credentials for a new polyrepo | Add the repo credential/secret; confirm the deploy key/token has read scope |
The three that hurt most:
1. CODEOWNERS that isn’t enforced (or is mis-ordered). CODEOWNERS by itself only requests review — without branch protection requiring Code Owner review, anyone with write access merges unreviewed and your carefully-drawn boundaries are decorative. And even when enforced, GitHub’s last-match-wins rule silently hands ownership to the wrong team if you order specific rules before the general *. The failure is invisible until the wrong (or no) reviewer is required on a sensitive path. Always: general rules first, specific paths last, branch protection on, and test it by opening a throwaway PR touching a scoped path and confirming the expected owner is requested.
2. An AppProject boundary that’s too loose. The most common real breach isn’t malice; it’s an AppProject with sourceRepos: ['*'] or a destinations namespace of '*' copied from a tutorial. That project fences nothing — a compromised or fat-fingered repo can sync a ClusterRole or land in another team’s namespace, and the split-repo blast-radius benefit evaporates because the sync gate was left open. The fix is boring and absolute: per-team sourceRepos listing exactly the team’s repo, destinations scoped to the team’s clusters and a team-* namespace glob, and an empty clusterResourceWhitelist. Audit every project for '*' — those are your open doors.
3. The SCM generator onboarding a repo it shouldn’t. Auto-onboarding is the hybrid’s superpower and its sharpest edge. If repositoryMatch is loose (or absent) and there’s no pathsExist, any repo appearing in the org — a fork, a scratch repo, a compromised one — can spawn a live Application. Worse, with an aggressive applicationsSync policy, a repo that stops matching can cause the controller to delete apps. Guard the front door with a specific name regex and a pathsExist that proves the repo is genuinely one of yours, and set applicationsSync: create-update so the controller can never delete an app out from under a team. Treat the generator’s filters as a security boundary, because they are one.
Cheat-sheet
The decision, the layout, and the enforcement pattern in one place.
The mono/poly/hybrid decision:
| You have… | Pick | Because |
|---|---|---|
| Few apps, one team, high trust | Monorepo | Simplest; discovery + atomic changes for free |
| Many teams needing hard isolation | Polyrepo (repo per team) | Clean ownership, small blast radius |
| A platform team + many app teams | Hybrid | Central platform, distributed apps, auto-onboarding |
| Can’t decide | Start monorepo | Split to hybrid when ownership pain arrives |
The recommended 100-app layout (hybrid):
| Layer | Repo | Generator |
|---|---|---|
| Platform | platform-gitops (monorepo) |
app-of-apps + Git dir (add-ons) |
| Apps | team-<name>-config × ~40 |
SCM Provider (auto-onboard) |
| Boundary | one AppProject per team |
fences repo + clusters + namespaces |
The CODEOWNERS + AppProject pattern (two gates):
| Gate | Mechanism | Stops |
|---|---|---|
| PR (merge) | CODEOWNERS + branch protection | Unreviewed change from merging |
| Sync | AppProject sourceRepos/destinations/clusterResourceWhitelist: [] |
Merged change from syncing outside its lane |
Key fields and commands:
| Item | What it does |
|---|---|
git: directories: [{path: apps/*}] |
Monorepo: one Application per directory |
scmProvider: filters: [{repositoryMatch, pathsExist}] |
Polyrepo/hybrid: one Application per matching repo |
argocd.argoproj.io/manifest-generate-paths: '.' |
Scope webhook re-render to changed paths (monorepo perf) |
syncPolicy.applicationsSync: create-update |
ApplicationSet may create/update apps, never delete |
spec.strategy.type: RollingSync |
Roll a generator change through apps in waves |
AppProject clusterResourceWhitelist: [] |
Deny all cluster-scoped kinds (highest-leverage guardrail) |
resources-finalizer.argocd.argoproj.io |
Cascade delete of app-of-apps children |
argocd app get <app> / argocd app diff <app> |
Inspect an app’s state / preview a change (on a real cluster) |
kubectl get applicationset -n argocd |
See generators and how many apps they render |
Interview and exam questions
Q: This lesson is about “config repos,” not “app-code repos.” What’s the difference, and why does it matter for the monorepo-vs-polyrepo debate?
A: App-code repos hold source, Dockerfiles, and tests, and are consumed by CI to build images — Argo CD never looks at them. Config repos hold the manifests, values, and Application/ApplicationSet/AppProject YAML that Argo CD reconciles against clusters. The monorepo-vs-polyrepo decision that affects Argo CD is entirely about the config repos — blast radius, sync ownership, and render cost. Half of all confused arguments are two people debating two different repos.
Q: Give the one-line trade-off between a config monorepo and a config polyrepo. A: A monorepo gives you the best discovery and atomic cross-cutting changes at the cost of a large blast radius and CODEOWNERS complexity; a polyrepo gives you hard per-team ownership and a small blast radius at the cost of discovery drift and cross-cutting changes becoming N pull requests.
Q: What is the hybrid pattern, and which generator glues it together? A: Hybrid keeps platform content (app-of-apps root, ApplicationSets, AppProjects, shared add-ons) in one platform-owned monorepo, and each team’s app config in its own repo. An SCM Provider generator watches the Git org and renders one Application per matching team repo, so new teams auto-onboard with no platform edit; a Git directory generator handles the shared add-ons. It’s the enterprise default because it puts each kind of content where its owner and change pattern want it.
Q: A change to a shared Kustomize base in a monorepo — what’s its blast radius, and how do you contain it?
A: Its blast radius is every app that references the base. Contain it with path-scoped CI (validate all dependents when the base changes), CODEOWNERS making the base platform-owned, manifest-generate-paths so only dependents re-render, and — if a generator template is involved — RollingSync so the change rolls through apps in waves instead of flipping all at once.
Q: Why do you need both CODEOWNERS and an AppProject? Isn’t one enough? A: They gate different moments. CODEOWNERS (plus branch protection) is the pull-request gate — it stops an unreviewed change from merging. The AppProject is the sync gate — it stops a merged change from syncing outside its allow-list of repos, clusters, and namespaces. CODEOWNERS without an AppProject lets a merged mistake reach another team’s namespace; an AppProject without CODEOWNERS lets anyone who can merge change desired state unreviewed. You need both.
Q: What’s the single highest-leverage field on an AppProject for multi-tenancy, and why?
A: An empty clusterResourceWhitelist: [] (paired with a namespace-scoped destinations glob). It means a compromised or fat-fingered app repo cannot create any cluster-scoped resource — no ClusterRole, no escaping its namespace — and the project rejects the sync before anything is applied. It converts a shared control plane into safe multi-tenancy.
Q: How does an SCM Provider generator decide which repos become Applications, and how do you stop it onboarding a rogue repo?
A: It queries the Git provider API for repos in an org/group and applies filters. Restrict it with repositoryMatch (a name regex like ^team-.*-config$) and pathsExist (require a path like gitops/kustomization.yaml that proves the repo is genuinely yours). Add applicationsSync: create-update so the controller can never delete an app if a repo vanishes. Treat those filters as a security boundary.
Q: Your monorepo has 100 apps and syncs are slow — repo-server CPU is pinned. Walk through the fix.
A: The bottleneck is render, which lives in the repo-server. Enable Git webhooks so refresh is event-driven instead of a 3-minute poll of every app; add argocd.argoproj.io/manifest-generate-paths to every Application so a push only re-renders apps whose paths changed; scale repo-server replicas and raise --parallelismlimit if still render-bound. Note that application-controller sharding shards by cluster, not app, so it won’t help a single-cluster monorepo’s render cost.
Q: In a monorepo CODEOWNERS, why does rule ordering matter, and what’s the safe convention?
A: GitHub CODEOWNERS uses last match wins, so if a specific path rule appears before the general *, the general rule can override it and require the wrong (or no) owner. The safe convention is general-to-specific: put * first, then progressively more specific paths, so the most specific rule is last and wins. Always verify with a throwaway PR that the expected owner is requested.
Q: A team says “we need to change a label on every workload in the fleet.” How does the answer differ across the three topologies? A: In a monorepo it’s one PR (all app config is co-located). In a polyrepo it’s N PRs, one per team repo. In a hybrid it depends on what kind of change: a platform-level change (a shared component, an ApplicationSet template) is one PR in the platform monorepo, but a change to app content is still N PRs because app content is distributed on purpose. If it’s truly fleet-wide app content, consider moving the shared piece into a platform-owned Kustomize component so it becomes a single platform PR.
Q: When would you deliberately choose full polyrepo over hybrid? A: When teams must not even see each other’s config — hard regulatory or multi-tenant-SaaS isolation — so a shared platform monorepo (which every team can read) is unacceptable, and you accept the discovery and cross-cutting-change costs in exchange for maximal separation. Otherwise, hybrid gives you most of polyrepo’s isolation for app content while keeping the control plane coherent.
Q: You’re advising a 5-engineer startup with 12 apps and one team. Monorepo or hybrid?
A: Monorepo. With one team and few apps, the ownership and blast-radius problems hybrid solves don’t exist yet, and the monorepo’s discovery and atomic-change benefits are pure upside. The right move is to design the seam — app-major apps/<app>/overlays/<env>/, per-path CODEOWNERS, an ApplicationSet already generating apps — so that splitting to hybrid later is mechanical, not a rewrite. Start simple; split when ownership pain actually arrives.
Key takeaways
- This is a decision about config repos, not app-code repos. Argo CD reconciles the config repos; where your source code lives is a separate CI concern. Conflating the two is the root of most monorepo-vs-polyrepo confusion.
- Three real patterns, one trade-off each. Monorepo buys discovery and atomic changes at the cost of blast radius; polyrepo buys ownership and small blast radius at the cost of discovery and N-PR cross-cutting changes; hybrid splits by responsibility and is the enterprise default.
- The topology is the generator. Monorepo → Git directory generator; polyrepo → SCM Provider generator (one app per repo); hybrid → app-of-apps + SCM Provider that auto-onboards team repos.
- Two gates, not one. CODEOWNERS + branch protection gate the pull request; the AppProject gates the sync. You need both, and an empty
clusterResourceWhitelistwith a namespace-scopeddestinationsglob is the highest-leverage guardrail you can set. - Contain the monorepo’s blast radius with mechanism, not hope. Path-scoped CI, CODEOWNERS,
manifest-generate-paths, sync waves, and ApplicationSetRollingSynceach contain a different class of failure — layer them. - Monorepo performance is a repo-server story. Webhooks +
manifest-generate-pathstame render cost well past 100 apps; controller sharding shards by cluster, not app, so it won’t fix a single-cluster monorepo’s render load. - For 100 apps and 40 teams, build the hybrid deliberately. One platform monorepo, ~40 per-team config repos auto-onboarded by an SCM generator, one AppProject per team — coherent platform, isolated teams, zero-touch onboarding.
- Start monorepo, plan the seam, split when it hurts. Most orgs end up hybrid; the ones that get there cleanly designed the split-friendly structure (app-major overlays, generated apps, scoped CODEOWNERS) from day one.