Argo CD Lesson 41 of 45

Monorepo vs Polyrepo GitOps: Structuring Your Repos for 100 Apps and 40 Teams

Every Argo CD platform eventually collides with one decision that nothing else can paper over: how do you lay out the Git repositories that Argo CD watches? Get it right and 40 teams ship independently while a single platform group keeps the fleet coherent. Get it wrong and you spend your quarters untangling merge conflicts, chasing broken ownership, and explaining why a one-line change to a shared label required 40 pull requests — or why one team’s typo just took down another team’s production namespace.

This lesson is deliberately not about where your application source code lives. Whether checkout and payments share a code monorepo or live in separate repos is a build-and-CI question, and Argo CD never looks at those repos. This lesson is about the config repos — the Git repositories that hold your Kubernetes manifests, Helm values, Kustomize overlays, Application/ApplicationSet/AppProject definitions — the repos Argo CD actually reconciles against live clusters. That distinction is the whole game, and half of all “monorepo vs polyrepo” arguments are two people talking about two different repos.

We will build the mental model from first principles, compare the three real patterns (single config monorepo, config polyrepo, and the hybrid), tie each to the exact ApplicationSet generator that makes it work, and then walk a concrete “100 apps, 40 teams” design end to end — including the CODEOWNERS files, the AppProject-per-team boundaries, and the blast-radius math. The hands-on lab scaffolds the hybrid layout on your laptop with no cluster required, so you can hold the structure in your hands before you commit an org to it.


Why this matters

Repo architecture is the decision with the longest half-life on a GitOps platform. You choose your CI trigger strategy in an afternoon and change it whenever; you choose your repo topology once, and by the time it hurts you have 100 apps, hundreds of Application objects, and 40 teams with muscle memory built around the current layout. Migrating a live fleet from one topology to another is a multi-quarter project with a change-freeze in the middle. This is the decision to make deliberately.

It matters because four things you care about are all downstream of it, and they pull in different directions:

What you want Monorepo pushes you toward Polyrepo pushes you toward
Simple discovery — one place to see everything Strong: one git clone, grep the fleet Weak: state is scattered across N repos
Clean ownership — each team owns its stuff, hard boundary Weak: everyone shares one repo + one history Strong: one repo per team, per-repo RBAC
Small blast radius — a bad change hurts little Weak: one commit can touch everything Strong: a bad change is trapped in one repo
Independent velocity — teams don’t block each other Weak: shared branch protection, shared CI queue Strong: each team’s cadence is its own

There is no free lunch in that table. The entire craft of platform engineering at scale is buying back the column you sacrificed with a mechanism — recovering polyrepo-grade ownership inside a monorepo with path-scoped CODEOWNERS, or recovering monorepo-grade discovery across a polyrepo with an ApplicationSet SCM generator. The patterns below are named combinations of those buy-backs.

The mental model to anchor everything: Argo CD reconciles (source repo, path, target cluster, namespace) tuples. A repo topology is just a policy for how you slice those tuples across Git repositories, and — crucially — who is allowed to change which slice. Monorepo puts every slice in one repo; polyrepo gives each owner their own repo; hybrid splits by responsibility. Everything else in this lesson is detail on that one idea.


The two axes: app-code repos vs config repos

Before comparing patterns, pin down what we are even structuring. There are two independent repo decisions in any organization, and conflating them is the number-one source of confusion.

App-code repo Config / GitOps repo
Holds Source code, Dockerfile, unit tests, the app’s own Helm chart or Kustomize base Rendered/overlay manifests, values files, Application/ApplicationSet/AppProject YAML
Consumed by CI (build, test, push image) Argo CD (repo-server renders it; application-controller applies it)
Changes on Every feature commit Every deploy/promotion (an image tag bump, a replica change)
Argo CD watches it? No — Argo CD never builds images Yes — this is the source of truth Argo CD reconciles
Mono-vs-poly here is about Build tooling, code review, test isolation Deploy blast radius, sync ownership, render cost

The clean handoff between them is the boundary this whole course keeps returning to: CI builds and pushes an image, then writes a new image tag into the config repo; Argo CD notices the config repo changed and rolls it out. CI ends where the config repo begins. (This lesson lives entirely on the right-hand column; the app-code column is a CI concern.)

A quick vocabulary table, because “the repo” means different things to different roles and you will be in rooms with all of them:

Term What it precisely means here
Config monorepo One Git repo holding the config for many/all apps, envs, and clusters
Config polyrepo Many Git repos, each holding config for one team or one app
Hybrid A platform/bootstrap monorepo plus per-team config repos, glued by ApplicationSet
Source of truth The (repoURL, path, targetRevision) in an Application.spec.source
Blast radius The set of live resources a single Git change can affect
Discovery How an engineer (or the platform) finds “all apps and their desired state”

With those fixed, the mono-vs-poly question at the config layer has exactly three serious answers. We take them one at a time, then compare them head to head.


Pattern 1: the single config monorepo

Everything Argo CD watches lives in one repository. All apps, all environments, all clusters, the ApplicationSets that generate them, and the AppProjects that fence them — one git clone, one history, one place.

A representative layout for a monorepo that Argo CD reconciles:

gitops/                              # ← the ONE repo Argo CD watches
├── bootstrap/
│   └── root-app.yaml                # app-of-apps root Application
├── applicationsets/
│   ├── apps.yaml                    # Git directory generator over apps/*
│   └── addons.yaml                  # cluster generator for shared add-ons
├── projects/
│   ├── team-payments.yaml           # AppProject per team
│   └── team-checkout.yaml
├── apps/
│   ├── payments/
│   │   ├── base/                    # Kustomize base (or a Helm chart ref)
│   │   └── overlays/
│   │       ├── dev/
│   │       ├── staging/
│   │       └── prod/
│   └── checkout/
│       ├── base/
│       └── overlays/{dev,staging,prod}/
├── addons/                          # shared platform add-ons
│   ├── ingress-nginx/
│   ├── cert-manager/
│   └── kube-prometheus-stack/
└── .github/
    └── CODEOWNERS                   # path-scoped ownership

Argo CD discovers apps by walking apps/* with a Git directory generator — add a directory, get an Application for free, no manual Application manifest:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monorepo-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/acme/gitops.git
        revision: main
        directories:
          - path: apps/*
  template:
    metadata:
      name: '{{.path.basename}}'          # e.g. payments, checkout
      annotations:
        # scope webhook-triggered re-render to this app's own files
        argocd.argoproj.io/manifest-generate-paths: '.'
    spec:
      project: '{{.path.basename}}'
      source:
        repoURL: https://github.com/acme/gitops.git
        targetRevision: main
        path: 'apps/{{.path.basename}}/overlays/prod'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

The appeal is real and should not be undersold:

Monorepo strength Why it happens
Trivial discovery Everything is one git clone; grep -r image: . shows every pinned tag in the fleet
Atomic cross-cutting changes Bump a shared label, a common securityContext, or a Kustomize component across all apps in one PR, one commit, one revert
One source of truth for structure Directory conventions, projects, and ApplicationSets are co-located and self-documenting
Simple onboarding of a repo Nothing to provision — a new app is a new directory, not a new repo + webhook + credential
Consistent tooling One CI config, one linter config, one policy set applies uniformly

And the costs are equally real — they are exactly the things this lesson exists to mitigate:

Monorepo cost What it looks like in practice
Large blast radius A bad edit to a shared base, or a fat-fingered kustomization.yaml, can render or sync-break many apps at once
CODEOWNERS complexity One repo must encode 40 teams’ boundaries in one .github/CODEOWNERS, and “last match wins” bites the careless
Repo-server render cost On a naive setup, a push anywhere can invalidate cache and re-render everything; big repos are slow to clone and manifest-generate
Shared history / merge pressure 40 teams committing to main means more conflicts on shared files and a noisier history
Shared CI queue Without path filtering, every PR runs the whole validation suite, and one team’s flaky check blocks the queue

Three of those costs — render churn, blast radius, and CI — are performance and safety problems with well-known fixes (manifest-generate-paths, path-scoped CI, CODEOWNERS, sync waves), which we cover in their own sections. The monorepo is not a trap; it is a pattern that demands discipline in exchange for simplicity. For a small-to-medium platform it is very often the right first answer, and the honest guidance at the end of this lesson is: start here.


Pattern 2: the config polyrepo

Flip the model: every team (or, in the extreme, every app) gets its own config repository. Argo CD watches many repos, one owner per repo.

A per-team polyrepo layout, replicated across N repos:

team-payments-config/                # ← one repo, owned by the payments team
├── gitops/
│   ├── base/
│   └── overlays/{dev,staging,prod}/
├── .github/
│   └── CODEOWNERS                    # entire repo → @acme/payments-team
└── README.md

team-checkout-config/                # ← a second repo, owned by checkout
├── gitops/
│   └── ...
└── ...

The natural generator here is the SCM Provider generator: point it at the Git organization, and it creates exactly one Application per repo that matches your filters. A new team repo appears in the org and — with no platform edit — an Application materializes:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-config-repos
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - scmProvider:
        github:
          organization: acme-teams
          allBranches: false
          tokenRef:
            secretName: github-scm-token
            key: token
        filters:
          - repositoryMatch: '^team-.*-config$'      # only *-config repos
            pathsExist: ['gitops/kustomization.yaml'] # only if it looks like ours
  template:
    metadata:
      name: '{{.repository}}'
    spec:
      project: '{{.repository}}'                       # project per repo
      source:
        repoURL: '{{.url}}'
        targetRevision: '{{.branch}}'
        path: gitops
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.repository}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
  syncPolicy:
    applicationsSync: create-update        # controller may create/update, NOT delete
    preserveResourcesOnDeletion: false

Notice applicationsSync: create-update at the ApplicationSet level — that is a deliberate safety valve we return to under troubleshooting: it lets auto-onboarding create and update Applications but forbids the controller from deleting an app if a repo temporarily vanishes or stops matching.

Polyrepo’s strengths are the mirror image of the monorepo’s costs:

Polyrepo strength Why it happens
Hard ownership boundary One repo per team means per-repo RBAC, per-repo CODEOWNERS, per-repo branch protection — no shared surface to fight over
Small blast radius A bad change is physically confined to one repo; it cannot render-break another team’s app
Independent velocity Each team’s CI, review cadence, and merge policy are entirely its own
Clean audit “Who changed payments’ prod?” is answered by one repo’s history, not filtered out of a 40-team firehose
Least-privilege credentials Argo CD can hold a narrowly-scoped deploy key per repo instead of one key to everything

And its costs are the monorepo’s strengths, inverted:

Polyrepo cost What it looks like in practice
Discovery overhead “Show me every app and its state” now means querying Argo CD or scanning N repos, not one grep
Consistency drift 40 repos drift toward 40 slightly-different structures, README styles, and CI configs unless enforced
Cross-cutting change is N PRs A fleet-wide label or security fix becomes 40 pull requests across 40 repos with 40 review cycles
Onboarding friction Each new app/team needs a repo provisioned, credentials wired, webhook configured, conventions applied
Generator/credential sprawl More repos to authenticate to, more webhooks, more places a secret can rot

There is a sub-decision inside polyrepo worth calling out, because “one repo per app” and “one repo per team” behave very differently at 100 apps:

Repo per team Repo per app
Repo count at 100 apps / 40 teams ~40 repos ~100+ repos
Ownership granularity Team owns a repo of several apps One owner per app, maximum isolation
Cross-app change within a team One PR (apps co-located) N PRs even inside one team
Discovery / credential sprawl Moderate High
Best when Teams own coherent app groups Apps are truly independent products

Repo-per-team is the pragmatic middle of polyrepo; repo-per-app is maximal isolation you pay for in sprawl. For 100 apps and 40 teams, repo-per-team is almost always the polyrepo variant that survives contact with reality.


Pattern 3: the hybrid

The hybrid is not a compromise so much as a division by responsibility, and it is where most enterprises actually land. There are two kinds of content on a GitOps platform with fundamentally different owners and change patterns:

  1. Platform content — the app-of-apps root, the ApplicationSets, the AppProjects, shared add-ons (ingress controller, cert-manager, the metrics stack), org-wide policy. Owned by one platform team. Changes rarely, must be coherent, benefits from being in one place.
  2. App content — each team’s own workload manifests, values, overlays. Owned by each team. Changes constantly, must be isolated, benefits from per-team boundaries.

Hybrid puts platform content in a monorepo (best-in-class discovery and atomic changes for the small platform team) and app content in per-team polyrepos (best-in-class ownership and blast radius for the many app teams). The glue is app-of-apps plus an ApplicationSet SCM generator.

platform-gitops/                     # ← MONOREPO, owned by @acme/platform-team
├── bootstrap/
│   └── root-app.yaml                # app-of-apps: syncs everything below
├── applicationsets/
│   ├── team-repos.yaml              # SCM generator → one app per team repo
│   └── addons.yaml                  # shared add-ons across clusters
├── projects/
│   ├── team-payments.yaml           # AppProject per team (the fence)
│   └── team-checkout.yaml
├── addons/
│   ├── ingress-nginx/
│   └── cert-manager/
└── .github/CODEOWNERS               # whole repo → platform team

team-payments-config/                # ← POLYREPO, owned by @acme/payments-team
└── gitops/{base,overlays/*}/

team-checkout-config/                # ← POLYREPO, owned by @acme/checkout-team
└── gitops/{base,overlays/*}/

The app-of-apps root bootstraps the whole platform layer in one apply:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/platform-gitops.git
    targetRevision: main
    path: bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

That root points at bootstrap/, which contains child Applications for applicationsets/, projects/, and addons/. The ApplicationSets then fan out: the SCM generator (shown in Pattern 2) discovers every team-*-config repo and renders one Application each, every app pinned to its team’s AppProject. This is exactly the structure the diagram below walks.

Hybrid property Where it comes from
Small platform team gets a monorepo Platform content is co-located, atomic, easy to grep and reason about
Many app teams get isolation Each app team owns one repo, one CODEOWNERS, one blast radius
Zero-touch onboarding SCM generator auto-onboards a new team repo; platform team does nothing
Central policy, distributed apps AppProjects, add-ons, and org policy stay platform-owned; workloads stay team-owned
Ownership cuts cleanly Repos, CODEOWNERS, and Argo CD RBAC all cut along the same platform-vs-team line

The trade-off hybrid does not escape: a truly fleet-wide change to app content (say, a new required label on every workload) is still N PRs across N team repos — because that is app content, and app content is distributed on purpose. Hybrid buys you a small, coherent platform core; it does not abolish the polyrepo tax on cross-cutting app changes. What it does is make sure that tax only applies to genuine app-level changes, while everything platform-level stays a single atomic PR.

Here is the hybrid structure end to end — read it left to right, from who owns what, through the repos and generators, to the AppProject fence and the clusters where workloads land:

Left-to-right Argo CD hybrid GitOps repo architecture: a platform team owning a bootstrap monorepo and 40 app teams owning their own config repos, feeding a Git directory generator and an SCM Provider generator, rendered by the ApplicationSet controller into Applications each pinned to a per-team AppProject that allow-lists repos and namespaces, landing scoped workloads on the target clusters with blast radius contained

The badges mark where each pattern is won or lost: the monorepo’s atomic-but-broad blast radius (1), the polyrepo’s hard per-repo ownership line (2), the SCM generator auto-onboarding a repo with zero platform edits (3), the AppProject as the real sync-time fence (4), the blast radius held at the project boundary so a bad change never crosses into another team (5), and the org chart driving repos plus RBAC along one line (6). If you internalize only the diagram, internalize this: CODEOWNERS gates the pull request; the AppProject gates the sync. You need both, and they are different mechanisms.


The big comparison

Everything above, in one place. This is the table to bring to the architecture review.

Dimension Single config monorepo Config polyrepo Hybrid
Discovery Best — one clone, grep the fleet Worst — scattered across N repos Good — platform central, apps via Argo CD
Ownership boundary Weak — path-scoped CODEOWNERS only Strong — one repo per owner Strong for apps, central for platform
Blast radius Large — one commit can touch all Small — trapped in one repo Small for apps, contained for platform
Cross-cutting platform change 1 PR N PRs 1 PR (platform is a monorepo)
Cross-cutting app change 1 PR N PRs N PRs (apps are distributed)
Review / RBAC granularity Path-level in one repo Repo-level, cleanest Repo-level for apps, central for platform
repo-server render cost Highest — needs manifest-generate-paths Lowest — small repos Low — small app repos, small platform repo
CI complexity Needs path filtering to scale Simple per repo, N pipelines to maintain Simple per app repo + one platform pipeline
Onboarding a new app Add a directory Provision a repo + creds + webhook Add a repo; SCM generator does the rest
Consistency enforcement Easy — one place Hard — N repos drift Medium — platform enforces conventions
Merge-conflict pressure Highest on shared files Lowest Low
Best team size 1 small team, or few teams Many independent teams 1 platform team + many app teams

The same information as a decision aid — pick by the strongest signal in your org, not by fashion:

If your org looks like this… Lean toward
One platform team, <15 apps, everyone trusts everyone Monorepo — simplest thing that works
Strict compliance isolation between teams; a shared repo is a non-starter Polyrepo (repo per team)
Teams must not even see each other’s config (regulated / multi-tenant SaaS) Polyrepo (repo per app or per team)
A platform team plus many product teams, growing fast Hybrid — the enterprise default
You genuinely can’t decide Start monorepo, plan the seam — split to hybrid when ownership pain arrives

And the blast-radius scenarios spelled out, because “blast radius” is abstract until it is your outage:

A change to… Monorepo can affect Polyrepo can affect Hybrid can affect
One app’s overlay Just that app (if CI/CODEOWNERS scoped) Just that app’s repo Just that app’s repo
A shared Kustomize base Every app using the base N/A (bases aren’t shared) N/A (apps own their bases)
An ApplicationSet template Every app it generates Every app it generates Every app it generates
An AppProject Every app in that project Every app in that project Every app in that project
A shared add-on (ingress) Every cluster running it Every cluster running it Every cluster running it

Read that table twice: ApplicationSets, AppProjects, and shared add-ons are wide-blast-radius objects in every topology. No repo split saves you there — those are guarded by RBAC, CODEOWNERS on the platform repo, and progressive rollout, not by which repo they live in. Polyrepo shrinks the blast radius of app config; it does nothing for the shared control-plane objects, which is precisely why the hybrid keeps those in a tightly-governed platform monorepo.


ApplicationSet and app-of-apps as the glue

Each repo pattern maps to a specific generator. Choosing the topology is choosing the generator; they are two names for one decision. (For the generator mechanics themselves, see ApplicationSets and generators and, for the bootstrap parent, the app-of-apps pattern.)

Repo pattern Primary generator How discovery works Onboarding a new app
Monorepo Git directory generator over apps/* Controller lists directories in the one repo Create a directory; app appears
Polyrepo SCM Provider generator over the org Controller queries the Git provider API for repos Create a matching repo; app appears
Hybrid app-of-appsSCM Provider (+ Git dir for add-ons) Root app syncs ApplicationSets, which fan out Create a team repo; SCM generator onboards it

The Git directory generator (monorepo) keys off filesystem structure; the SCM Provider generator (polyrepo/hybrid) keys off the existence of repos. Their template parameters differ, and mixing them up is a common first-day error:

Generator Key template params (goTemplate) Selection mechanism
Git directories .path.path, .path.basename, .path.basenameNormalized directories: [{path: apps/*}], optional exclude
Git files keys from each matched JSON/YAML file’s contents files: [{path: apps/**/config.json}]
SCM Provider .organization, .repository, .url, .branch, .sha, .labels filters:repositoryMatch, pathsExist, branchMatch, labelMatch
Pull request .number, .branch, .head_sha per-provider PR query + labels

The SCM Provider generator’s filters are your onboarding gate, and getting them right is the difference between “a new team repo just works” and “a random fork in the org spawned a rogue Application.” The filters that matter:

Filter field What it does Why you want it
repositoryMatch Regex the repo name must match Restrict to ^team-.*-config$, not every repo in the org
pathsExist Only match repos that contain these path(s) Require gitops/kustomization.yaml — proves it’s really one of ours
pathsDoNotExist Exclude repos containing a path Skip archived repos carrying a .archived marker
branchMatch Regex the branch must match Only track main, not every feature branch
labelMatch Match a repo topic/label Onboard only repos tagged argocd-managed

Because the SCM Provider generator talks to a Git provider API, this is one of the few genuinely provider-specific edges in an otherwise cloud-neutral lesson. Repo architecture itself is the same on any Kubernetes — AKS, EKS, GKE, on-prem — because Argo CD reconciles Git against clusters identically everywhere. But which SCM block you write depends on where the config repos are hosted:

Git host Generator block Auth (tokenRef secret) Note
GitHub / GHES github: (organization, optional api: for Enterprise) PAT or GitHub App token allBranches: false unless you need every branch
GitLab gitlab: (group, includeSubgroups) Group/project access token includeSubgroups: true to walk nested groups
Gitea gitea: (owner, api) Gitea token Self-hosted; set api to your instance URL
Azure DevOps azureDevOps: (organization, teamProject) Azure DevOps PAT Scoped per project
Bitbucket Server bitbucketServer: (project, api) HTTP access token Data Center / Server
Bitbucket Cloud bitbucketCloud: (owner) App password SaaS Bitbucket

The cluster the workloads land on can be AKS, EKS, GKE, or anything else, selected by the destination — a hybrid fleet routinely spans all three, and a cluster generator (or a matrix of SCM × clusters) fans one team’s app across every matching cluster. The repo topology does not change when the clouds do; only the scmProvider block and the destinations do.

For the hybrid, you often combine generators. A matrix of the SCM Provider generator crossed with a cluster generator gives you “every team repo, on every matching cluster,” while a separate Git directory generator over the platform monorepo’s addons/ handles shared platform components. Keep the two concerns in two ApplicationSets — do not try to make one generator do both jobs.


Team ownership: CODEOWNERS, branch protection, AppProject-per-team

Ownership is enforced at two independent gates, and a mature platform uses both because they catch different failures:

  1. The pull-request gate — CODEOWNERS + branch protection. Stops an unreviewed change from merging. Lives in the Git provider.
  2. The sync gate — the AppProject. Stops a merged change from syncing outside its allow-list. Lives in Argo CD.

CODEOWNERS without an AppProject means a compromised or mis-scoped repo can still push resources into another team’s namespace once it’s merged. An AppProject without CODEOWNERS means anyone who can merge can change the desired state unreviewed. You need both.

CODEOWNERS: path-scoped in a monorepo, whole-repo in a polyrepo

In a monorepo, one CODEOWNERS file encodes every team’s boundary by path. The single most important rule, and a frequent outage cause: in GitHub CODEOWNERS the last matching pattern wins, so order from general to specific.

# platform-gitops/.github/CODEOWNERS
# Last match wins — general rules first, specific paths override below.

# Default: the platform team owns anything not otherwise claimed.
*                       @acme/platform-team

# Shared control-plane objects stay platform-owned even if teams touch nearby.
/applicationsets/       @acme/platform-team
/projects/              @acme/platform-team
/addons/                @acme/platform-team

# Per-team app directories: the team owns their app; platform co-reviews.
/apps/payments/         @acme/payments-team @acme/platform-team
/apps/checkout/         @acme/checkout-team @acme/platform-team

In a polyrepo, CODEOWNERS is trivial because the repo is the boundary — the whole repo belongs to one team:

# team-payments-config/.github/CODEOWNERS
*   @acme/payments-team

That single-line polyrepo CODEOWNERS is precisely the ownership win, made concrete: the repo is the boundary, so there is one trivial file, no ordering to get wrong, and “who owns X?” is always “the repo owner.” The monorepo buys back the same isolation with path rules, but path rules are fiddly, onboarding a team means editing shared rules, and the “last match wins” ordering trips people up — a real failure mode we return to under troubleshooting.

Branch protection

CODEOWNERS only requests review; branch protection makes it required. Without it, CODEOWNERS is advisory and anyone with write access can merge unreviewed. The settings that matter for a GitOps config repo:

Branch-protection setting Why it matters for config repos
Require pull request before merging No direct pushes to the branch Argo CD tracks
Require review from Code Owners Makes CODEOWNERS enforced, not advisory
Require status checks (CI) to pass Lint/kubeconform/policy gates run before merge
Dismiss stale approvals on new commits A post-approval push can’t sneak in unreviewed
Restrict who can push / include administrators Even admins go through review; no bypass
Require linear history / signed commits Cleaner audit; provenance on the source of truth

The AppProject: the sync-time fence

The AppProject is Argo CD’s multi-tenancy boundary and the second gate. It restricts which repos an app may deploy from, which clusters/namespaces it may deploy to, and which resource kinds it may touch — enforced at sync time by the controller, independent of Git. (This lesson uses AppProjects as a boundary; for the full treatment of project-based multi-tenancy, that is its own dedicated topic in the course.)

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments
  namespace: argocd
spec:
  description: Payments squad  apps, staging + prod-eu only
  sourceRepos:
    - https://github.com/acme-teams/team-payments-config.git   # ONLY this repo
  destinations:
    - name: prod-eu
      namespace: 'payments-*'                                  # ONLY these namespaces
    - name: staging
      namespace: 'payments-*'
  clusterResourceWhitelist: []                                 # deny ALL cluster-scoped kinds
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota                                      # can't rewrite their own quota
    - group: ""
      kind: LimitRange
  roles:
    - name: deployer
      policies:
        - p, proj:team-payments:deployer, applications, sync, team-payments/*, allow
        - p, proj:team-payments:deployer, applications, get,  team-payments/*, allow
      groups:
        - acme:payments-engineers

The four fields that do the fencing:

AppProject field What it constrains Set it to…
sourceRepos Which Git repos apps in this project may use Exactly the team’s repo(s); never '*' in prod
destinations Which (cluster, namespace) pairs are allowed The team’s clusters + a team-* namespace glob
clusterResourceWhitelist Which cluster-scoped kinds are allowed [] (empty = none) unless the team truly needs them
namespaceResourceBlacklist Namespaced kinds to forbid ResourceQuota, LimitRange — platform owns those

An empty clusterResourceWhitelist plus a namespace-scoped destinations glob is the single highest-leverage guardrail on the whole platform: a compromised or fat-fingered app repo cannot create a ClusterRole, escape its namespace, or deploy to another team’s cluster — the project rejects the sync before anything is applied. The RBAC layer on top maps SSO groups to project-scoped actions:

Role Applications create/delete Applications sync/get Scope
Platform admin Yes Yes All projects
App team (deployer) No (apps come from the ApplicationSet) Yes Their project only
Everyone else No Read-only All (read)

App teams deliberately get sync/get but not create/delete on Applications: the Applications are generated by the platform-owned ApplicationSet, so a team changes desired state in Git, not the Application objects themselves. That keeps the generator authoritative and prevents a team from hand-editing its way out of the project boundary.


Blast radius and how to contain it

Blast radius is the set of live resources one Git change can affect. The monorepo’s central weakness is a wide default blast radius; the mitigations below shrink it back toward polyrepo levels without giving up the monorepo’s discovery. Each control contains a different class of failure — layer them.

Control Contains which failure Where it lives
Path-scoped CI A malformed manifest in app A failing/altering app B’s pipeline Git provider CI (path filters)
CODEOWNERS An unreviewed change to a shared file or another team’s app Git provider PR gate
AppProject A merged change trying to sync outside its repo/cluster/namespace allow-list Argo CD sync gate
manifest-generate-paths A push to app A needlessly re-rendering (and re-syncing risk) app B Argo CD repo-server
Sync waves Applying a dependent resource before its prerequisite within one app Argo CD sync ordering
ApplicationSet RollingSync A generator change flipping all generated apps at once ApplicationSet controller
applicationsSync: create-update The controller deleting apps when a repo vanishes/stops matching ApplicationSet syncPolicy

Path-scoped CI is the first line and the cheapest. In a monorepo, run each app’s validation only when its files change, so team A’s flaky test never blocks team B and a broken overlay is caught before merge without dragging the whole repo through CI:

# .github/workflows/validate.yml — only lint the app whose files changed
on:
  pull_request:
    paths:
      - 'apps/payments/**'          # this job runs only for payments changes
jobs:
  validate-payments:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Render + validate
        run: |
          kustomize build apps/payments/overlays/prod | \
            kubeconform -strict -summary

For fleets with dozens of apps, a dynamic matrix (compute the changed app directories, then fan out one validation job per changed app) scales this without hand-writing 100 path filters. The principle is the same: CI blast radius should equal the change’s blast radius.

The ApplicationSet RollingSync strategy contains the widest monorepo failure — a change to a generator template or a shared value that would otherwise flip every generated Application simultaneously. It rolls the change through in labeled waves and waits between them (for cross-cluster fleet rollout mechanics, see scaling and sharding):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-config-repos
  namespace: argocd
spec:
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: envLabel
              operator: In
              values: [canary]        # a handful of apps first
        - matchExpressions:
            - key: envLabel
              operator: In
              values: [prod]          # then the rest, only if canary is healthy
  # ... generators + template as before ...

RollingSync (progressive syncs) must be enabled on the ApplicationSet controller — it has historically been behind the --enable-progressive-syncs flag / a feature-flag env var, and moved toward default-on in the 2.13+/3.x line. Confirm it’s enabled in your version before relying on it; if it isn’t, the strategy block is silently ignored and everything syncs at once. That “silently ignored” failure mode is exactly why you verify the flag, not just the manifest.

The promotion of a version through environments (dev → staging → prod) is a related but distinct concern — that ordering is about which revision each environment tracks, and is covered in environment promotion. Blast-radius containment is about limiting how much a single change can touch; promotion is about when a known-good change advances.


Performance at scale

The monorepo’s other tax is performance, and it is entirely a repo-server story. The repo-server clones the repo and renders manifests; the application-controller reconciles the rendered result against clusters. A big monorepo makes the repo-server work harder, and a naive setup makes it re-render far more than necessary. (This section is a summary tuned to repo topology; the full scaling treatment — controller sharding, repo-server sizing, cache internals — is in scaling, sharding, and monorepo performance.)

What actually costs you in a monorepo:

Cost driver Why the monorepo makes it worse Lever
Clone time One large repo, deep history, many files Shallow clones (default), fewer large binaries in-repo
Manifest generation A naive push re-renders every app manifest-generate-paths scopes re-render to changed paths
Cache invalidation One commit SHA covers the whole repo manifest-generate-paths keeps unchanged apps on cached render
Reconcile fan-out Many Applications share one repoURL Webhooks (not polling) + repo-server replicas
Polling load Default ~3-minute poll × hundreds of apps Configure a Git webhook so refresh is event-driven

The highest-leverage monorepo-specific lever is the manifest-generate-paths annotation. It tells Argo CD which paths in the repo actually affect an Application’s rendered output, so on a webhook push the repo-server only re-renders apps whose relevant paths changed — turning “a commit to app A re-renders all 100 apps” into “a commit to app A re-renders app A”:

metadata:
  annotations:
    # semicolon-separated; '.' = this app's own source path,
    # add shared paths this app depends on (leading '/' = repo root)
    argocd.argoproj.io/manifest-generate-paths: '.;/shared/common-labels'

Include every path the render truly depends on (the app’s own directory plus any shared base or component it references), or a change to that shared path won’t trigger a re-render and the app will silently miss it. This annotation only takes effect with webhooks; under plain polling Argo CD re-checks everything anyway.

The scaling knobs, summarized for the topology decision:

Knob What it scales When the monorepo needs it
Webhook (vs polling) Refresh latency + repo-server load Always, past a handful of apps
manifest-generate-paths Re-render scope per push Any monorepo with many apps
repo-server replicas Parallel render throughput Many concurrent syncs / large renders
--parallelismlimit Concurrent manifest jobs per repo-server Render CPU saturation
Controller sharding (--replicas) Reconcile load across clusters Many clusters (shards by cluster, not app)

One nuance that trips up monorepo operators: application-controller sharding shards by cluster, not by app. Splitting one giant monorepo across more controller shards does nothing if all its apps target one cluster — that load is on the repo-server (render) and is scaled with repo-server replicas and manifest-generate-paths, not controller shards. Match the lever to the bottleneck: render cost → repo-server; reconcile cost across many clusters → controller shards.

Polyrepo and hybrid sidestep much of this by construction: small repos clone fast, render cheap, and a push touches one small repo. That is a genuine performance argument for splitting — but it is usually not the deciding factor, because manifest-generate-paths plus webhooks tame the monorepo well past 100 apps. Split for ownership and blast radius; treat the performance improvement as a bonus, not the reason.


The “100 apps, 40 teams” worked design

Now the concrete recommendation, with the reasoning, for the canonical scale in this lesson’s title. The answer is hybrid, and here is exactly why and how.

Why hybrid, not monorepo: 40 teams sharing one repo means 40 teams’ CODEOWNERS rules in one file, 40 teams’ CI in one queue, and one shared history — the ownership and merge pressure alone justify splitting the app content out. Why hybrid, not full polyrepo: the platform content (ApplicationSets, AppProjects, shared add-ons) must stay coherent and change atomically; scattering it across 40 repos would make a simple org-wide policy change a 40-PR nightmare and destroy discoverability of the control plane. Hybrid puts each kind of content where it belongs.

The repo inventory:

Repo Count Owner Holds Generator
platform-gitops 1 Platform team app-of-apps root, ApplicationSets, AppProjects, shared add-ons app-of-apps + Git dir (add-ons)
team-<name>-config ~40 Each app team That team’s app overlays/values SCM Provider (auto-onboarded)
App source repos ~100 Service teams Code + chart/base (CI territory) none (Argo CD ignores these)

The onboarding flow that makes 40 teams scale to a 1-person-day platform effort:

Step Who Action
1 App team Create team-<name>-config from a template repo (structure + CODEOWNERS baked in)
2 Platform (once) Commit an AppProject for the team to platform-gitops/projects/
3 SCM generator Auto-detects the new repo (name + pathsExist match) → renders one Application
4 AppProject Fences that Application to the team’s repo, clusters, and namespaces
5 App team Commits overlays; Argo CD syncs; team ships independently

Steps 3–4 are the payoff: after the one-time project commit, the platform team does nothing to onboard each new app the team adds — the SCM generator and the project boundary do the work. The template repo in step 1 is what keeps 40 repos from drifting into 40 structures.

The sizing/scaling plan for this fleet:

Concern Setting for ~100 apps / 40 teams
Refresh Git webhooks on all repos (no polling at this scale)
Render scope manifest-generate-paths on every generated Application
repo-server 3+ replicas behind the service; raise --parallelismlimit if render-bound
Controller shards Shard by cluster if the fleet spans many clusters (AKS + EKS + GKE)
Onboarding safety SCM filters (name regex + pathsExist) + applicationsSync: create-update
Rollout safety RollingSync on any ApplicationSet whose template many apps share
Boundaries One AppProject per team; empty clusterResourceWhitelist; namespace glob

This design gives the platform team a single coherent repo they can reason about, gives 40 app teams hard isolation and independent velocity, auto-onboards new work with zero platform toil, and holds every team’s blast radius at the project boundary. It is the shape most enterprises converge on, and it is worth building deliberately rather than discovering by accident.


Repo hygiene and conventions

Whatever topology you pick, consistency is what keeps it legible at 100 apps. These conventions are cheap to adopt early and painful to retrofit late.

Convention Recommendation
Repo naming Predictable and matchable: team-<name>-config, platform-gitops. The SCM generator’s regex depends on it
Top-level layout Stable apps/, addons/, projects/, applicationsets/, bootstrap/ — never improvise per repo
Env layout base/ + overlays/{dev,staging,prod}/; overlays hold only the delta, never a full copy
One engine per app Helm or Kustomize per app, never both; keep the override surface small and reviewable
Pin targetRevision Prod tracks a tag or SHA, never a moving branch — a merge elsewhere shouldn’t auto-ship
README.md per repo What it deploys, which clusters, who owns it, how to add an app
CODEOWNERS present Every config repo has one; monorepo path-scoped, polyrepo whole-repo
Template repo New team repos are created from a template so structure/CODEOWNERS/CI are uniform from day one

The /apps /envs /clusters question comes up constantly, so make it explicit. There are two common axes for organizing overlays, and picking one and sticking to it matters more than which you pick:

Layout axis Structure Best when
App-major apps/<app>/overlays/<env>/ Teams own apps; most changes are per-app (most common)
Env-major envs/<env>/<app>/ You promote whole environments together and want per-env review
Cluster-major clusters/<cluster>/<app>/ Cluster topology dominates (edge fleets, per-cluster divergence)

App-major is the default for the hybrid because it aligns the directory tree with the ownership tree — a team’s stuff is contiguous, which makes CODEOWNERS, path-scoped CI, and manifest-generate-paths all trivially clean. Reach for env-major or cluster-major only when your dominant change pattern genuinely runs along that axis.


Hands-on lab

You will design and scaffold the hybrid layout on your laptop — no cluster required. We build a platform monorepo (app-of-apps + a shared add-on), a per-team config repo, wire an ApplicationSet SCM generator to auto-onboard team repos, add CODEOWNERS and an AppProject per team, then prove how a team’s change is scoped. Because there is no cluster here, we validate the structure and manifests (which is where topology bugs actually live) and label all command output as representative. Everything is copy-pasteable.

Prerequisites: bash, python3. We optionally use PyYAML to lint manifests:

python3 -m pip install --quiet pyyaml   # optional; the lint step uses it

Step 1 — Scaffold the platform monorepo.

mkdir -p ~/argocd-repos-lab && cd ~/argocd-repos-lab
mkdir -p platform-gitops/{bootstrap,applicationsets,projects,addons/ingress-nginx,.github}

What just happened: you created the platform-owned monorepo skeleton — bootstrap/ for the app-of-apps root, applicationsets/ for the generators, projects/ for the per-team fences, and addons/ for shared components. This is the coherent core the platform team owns.

Step 2 — Write the app-of-apps root.

cat > platform-gitops/bootstrap/root-app.yaml <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/platform-gitops.git
    targetRevision: main
    path: bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }
YAML

What just happened: one root Application that, once applied to a cluster, syncs everything else in the platform repo. This is the single bootstrap seam — apply it once and the platform layer materializes.

Step 3 — Write the SCM Provider generator (the auto-onboarding glue).

cat > platform-gitops/applicationsets/team-repos.yaml <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-config-repos
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - scmProvider:
        github:
          organization: acme-teams
          allBranches: false
          tokenRef:
            secretName: github-scm-token
            key: token
        filters:
          - repositoryMatch: '^team-.*-config$'
            pathsExist: ['gitops/kustomization.yaml']
  template:
    metadata:
      name: '{{.repository}}'
    spec:
      project: '{{.repository}}'
      source:
        repoURL: '{{.url}}'
        targetRevision: '{{.branch}}'
        path: gitops
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.repository}}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
  syncPolicy:
    applicationsSync: create-update
    preserveResourcesOnDeletion: false
YAML

What just happened: this ApplicationSet will discover any team-*-config repo in the acme-teams org that contains gitops/kustomization.yaml, and render one Application per repo — each pinned to a project named after the repo. applicationsSync: create-update means a repo disappearing can’t make the controller delete a live app. This is the zero-touch onboarding mechanism.

Step 4 — Add a shared add-on and the platform CODEOWNERS.

cat > platform-gitops/addons/ingress-nginx/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://github.com/kubernetes/ingress-nginx//deploy/static/provider/cloud?ref=controller-v1.11.2
YAML

cat > platform-gitops/.github/CODEOWNERS <<'EOF'
# Last match wins — general first, specific overrides below.
*                       @acme/platform-team
/applicationsets/       @acme/platform-team
/projects/              @acme/platform-team
/addons/                @acme/platform-team
EOF

What just happened: the platform repo now owns a shared ingress add-on and declares that the whole repo — especially the control-plane directories — is platform-owned. In a real repo, branch protection with “require review from Code Owners” makes this enforced rather than advisory.

Step 5 — Write the AppProject that fences a team.

cat > platform-gitops/projects/team-payments.yaml <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments-config
  namespace: argocd
spec:
  description: Payments squad
  sourceRepos:
    - https://github.com/acme-teams/team-payments-config.git
  destinations:
    - name: staging
      namespace: 'payments-*'
    - name: prod-eu
      namespace: 'payments-*'
  clusterResourceWhitelist: []
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota
  roles:
    - name: deployer
      policies:
        - p, proj:team-payments-config:deployer, applications, sync, team-payments-config/*, allow
        - p, proj:team-payments-config:deployer, applications, get,  team-payments-config/*, allow
      groups:
        - acme:payments-engineers
YAML

What just happened: the payments team’s Application (named team-payments-config by the generator, matching project: '{{.repository}}') is now fenced: it may only source from the payments repo, only deploy into payments-* namespaces on staging/prod-eu, and may not create any cluster-scoped resource. This is the sync-time boundary.

Step 6 — Scaffold a team’s polyrepo.

mkdir -p team-payments-config/{gitops/base,gitops/overlays/prod,.github}

cat > team-payments-config/gitops/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - overlays/prod
YAML

cat > team-payments-config/gitops/base/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
YAML

cat > team-payments-config/gitops/base/deployment.yaml <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 2
  selector: { matchLabels: { app: payments-api } }
  template:
    metadata: { labels: { app: payments-api } }
    spec:
      containers:
        - name: api
          image: ghcr.io/acme/payments-api:1.4.0
YAML

cat > team-payments-config/gitops/overlays/prod/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: ghcr.io/acme/payments-api
    newTag: 1.4.0
YAML

cat > team-payments-config/.github/CODEOWNERS <<'EOF'
*   @acme/payments-team
EOF

What just happened: the team’s repo has the gitops/kustomization.yaml the SCM filter requires, a base + prod overlay, and a whole-repo CODEOWNERS. The moment this repo lands in the org, the generator from Step 3 onboards it — no platform edit.

Step 7 — Lint every manifest (representative validation).

python3 - <<'PY'
import glob, sys
try:
    import yaml
except ImportError:
    print("PyYAML not installed; skipping (pip install pyyaml to enable)"); sys.exit(0)

ok, bad = 0, 0
for f in glob.glob("**/*.yaml", recursive=True):
    try:
        docs = [d for d in yaml.safe_load_all(open(f)) if d]
        for d in docs:
            assert "apiVersion" in d and "kind" in d, f"missing apiVersion/kind in {f}"
        ok += len(docs)
    except Exception as e:
        print(f"FAIL {f}: {e}"); bad += 1
print(f"\nvalidated {ok} manifests across the hybrid layout — {bad} file(s) failed")
PY

Representative output:

validated 8 manifests across the hybrid layout — 0 file(s) failed

What just happened: every YAML in both repos parses and carries a real apiVersion/kind. This is the check that catches the topology bugs that actually bite — a malformed kustomization.yaml, a missing field — before a cluster ever sees them. (On a real cluster you would additionally run kustomize build ... | kubeconform -strict and argocd app diff.)

Step 8 — Prove a team’s change is scoped.

# Simulate the payments team bumping their image tag.
sed -i.bak 's/newTag: 1.4.0/newTag: 1.4.1/' \
  team-payments-config/gitops/overlays/prod/kustomization.yaml
rm -f team-payments-config/gitops/overlays/prod/kustomization.yaml.bak

# What did the change touch? Only the payments repo.
echo "Files changed by this deploy:"
echo "  team-payments-config/gitops/overlays/prod/kustomization.yaml"
echo
echo "Gates this change must pass:"
echo "  1. CODEOWNERS: team-payments-config/* -> @acme/payments-team (PR review)"
echo "  2. AppProject: sourceRepos allows ONLY team-payments-config.git (sync)"
echo "  3. Destination: only payments-* namespaces on staging/prod-eu (sync)"

Representative output:

Files changed by this deploy:
  team-payments-config/gitops/overlays/prod/kustomization.yaml

Gates this change must pass:
  1. CODEOWNERS: team-payments-config/* -> @acme/payments-team (PR review)
  2. AppProject: sourceRepos allows ONLY team-payments-config.git (sync)
  3. Destination: only payments-* namespaces on staging/prod-eu (sync)

What just happened: a payments deploy touched exactly one file in exactly one repo, gated by the payments CODEOWNERS at merge time and by the payments AppProject at sync time. There is no path by which this change reaches checkout’s namespace or another team’s cluster — the blast radius is the payments repo, full stop. That is the whole design working: CODEOWNERS gated the PR, the AppProject gated the sync.

Teardown.

cd ~ && rm -rf ~/argocd-repos-lab

What just happened: everything was local files, so cleanup is one rm -rf — nothing billed, nothing left in a cluster.


Common mistakes and troubleshooting

The failures below are the ones that actually cost teams time when they scale a repo topology. The nastiest are discussed in prose after the table.

Symptom Likely cause Fix
One bad monorepo commit breaks/re-syncs many apps No path-scoped CI and wide render blast radius Add path-filtered CI per app; add manifest-generate-paths; scope CODEOWNERS
Monorepo syncs are slow; repo-server CPU pinned Every push re-renders every app Set argocd.argoproj.io/manifest-generate-paths; enable webhooks; add repo-server replicas
A fleet-wide app change means 40 PRs Working as designed for polyrepo app content Accept it, or move the shared piece into a platform-owned component/base in the monorepo
New team repo never becomes an Application (polyrepo drift) SCM generator filters don’t match, or no webhook Check repositoryMatch regex + pathsExist; verify org token scope; confirm requeue/webhook
Unreviewed change merged to another team’s path CODEOWNERS present but not enforced Turn on branch protection “require review from Code Owners”; check “last match wins” ordering
A team deployed into another team’s namespace AppProject destinations/sourceRepos too loose (e.g. '*') Tighten to the team’s repo + team-* namespace glob; empty clusterResourceWhitelist
Application references a repo the project forbids Generated app’s project doesn’t permit its repoURL Fix sourceRepos on the AppProject, or template the correct project
SCM generator onboarded a rogue/broken repo Filters too broad; applicationsSync allows delete/update freely Add pathsExist + tighter repositoryMatch; set applicationsSync: create-update
Two teams’ PRs conflict on a shared monorepo file Shared surface + no ownership split on that file Split the shared file, or make it platform-owned via CODEOWNERS + a component
40 polyrepos have 40 different structures No template repo / no enforced conventions Create team repos from a template; lint structure in CI; document the layout
RollingSync steps ignored; all apps flip at once Progressive syncs not enabled on the controller Enable --enable-progressive-syncs (or the feature-flag env) and redeploy the controller
ComparisonError / rpc error: failed to get repo after split Argo CD lacks credentials for a new polyrepo Add the repo credential/secret; confirm the deploy key/token has read scope

The three that hurt most:

1. CODEOWNERS that isn’t enforced (or is mis-ordered). CODEOWNERS by itself only requests review — without branch protection requiring Code Owner review, anyone with write access merges unreviewed and your carefully-drawn boundaries are decorative. And even when enforced, GitHub’s last-match-wins rule silently hands ownership to the wrong team if you order specific rules before the general *. The failure is invisible until the wrong (or no) reviewer is required on a sensitive path. Always: general rules first, specific paths last, branch protection on, and test it by opening a throwaway PR touching a scoped path and confirming the expected owner is requested.

2. An AppProject boundary that’s too loose. The most common real breach isn’t malice; it’s an AppProject with sourceRepos: ['*'] or a destinations namespace of '*' copied from a tutorial. That project fences nothing — a compromised or fat-fingered repo can sync a ClusterRole or land in another team’s namespace, and the split-repo blast-radius benefit evaporates because the sync gate was left open. The fix is boring and absolute: per-team sourceRepos listing exactly the team’s repo, destinations scoped to the team’s clusters and a team-* namespace glob, and an empty clusterResourceWhitelist. Audit every project for '*' — those are your open doors.

3. The SCM generator onboarding a repo it shouldn’t. Auto-onboarding is the hybrid’s superpower and its sharpest edge. If repositoryMatch is loose (or absent) and there’s no pathsExist, any repo appearing in the org — a fork, a scratch repo, a compromised one — can spawn a live Application. Worse, with an aggressive applicationsSync policy, a repo that stops matching can cause the controller to delete apps. Guard the front door with a specific name regex and a pathsExist that proves the repo is genuinely one of yours, and set applicationsSync: create-update so the controller can never delete an app out from under a team. Treat the generator’s filters as a security boundary, because they are one.


Cheat-sheet

The decision, the layout, and the enforcement pattern in one place.

The mono/poly/hybrid decision:

You have… Pick Because
Few apps, one team, high trust Monorepo Simplest; discovery + atomic changes for free
Many teams needing hard isolation Polyrepo (repo per team) Clean ownership, small blast radius
A platform team + many app teams Hybrid Central platform, distributed apps, auto-onboarding
Can’t decide Start monorepo Split to hybrid when ownership pain arrives

The recommended 100-app layout (hybrid):

Layer Repo Generator
Platform platform-gitops (monorepo) app-of-apps + Git dir (add-ons)
Apps team-<name>-config × ~40 SCM Provider (auto-onboard)
Boundary one AppProject per team fences repo + clusters + namespaces

The CODEOWNERS + AppProject pattern (two gates):

Gate Mechanism Stops
PR (merge) CODEOWNERS + branch protection Unreviewed change from merging
Sync AppProject sourceRepos/destinations/clusterResourceWhitelist: [] Merged change from syncing outside its lane

Key fields and commands:

Item What it does
git: directories: [{path: apps/*}] Monorepo: one Application per directory
scmProvider: filters: [{repositoryMatch, pathsExist}] Polyrepo/hybrid: one Application per matching repo
argocd.argoproj.io/manifest-generate-paths: '.' Scope webhook re-render to changed paths (monorepo perf)
syncPolicy.applicationsSync: create-update ApplicationSet may create/update apps, never delete
spec.strategy.type: RollingSync Roll a generator change through apps in waves
AppProject clusterResourceWhitelist: [] Deny all cluster-scoped kinds (highest-leverage guardrail)
resources-finalizer.argocd.argoproj.io Cascade delete of app-of-apps children
argocd app get <app> / argocd app diff <app> Inspect an app’s state / preview a change (on a real cluster)
kubectl get applicationset -n argocd See generators and how many apps they render

Interview and exam questions

Q: This lesson is about “config repos,” not “app-code repos.” What’s the difference, and why does it matter for the monorepo-vs-polyrepo debate? A: App-code repos hold source, Dockerfiles, and tests, and are consumed by CI to build images — Argo CD never looks at them. Config repos hold the manifests, values, and Application/ApplicationSet/AppProject YAML that Argo CD reconciles against clusters. The monorepo-vs-polyrepo decision that affects Argo CD is entirely about the config repos — blast radius, sync ownership, and render cost. Half of all confused arguments are two people debating two different repos.

Q: Give the one-line trade-off between a config monorepo and a config polyrepo. A: A monorepo gives you the best discovery and atomic cross-cutting changes at the cost of a large blast radius and CODEOWNERS complexity; a polyrepo gives you hard per-team ownership and a small blast radius at the cost of discovery drift and cross-cutting changes becoming N pull requests.

Q: What is the hybrid pattern, and which generator glues it together? A: Hybrid keeps platform content (app-of-apps root, ApplicationSets, AppProjects, shared add-ons) in one platform-owned monorepo, and each team’s app config in its own repo. An SCM Provider generator watches the Git org and renders one Application per matching team repo, so new teams auto-onboard with no platform edit; a Git directory generator handles the shared add-ons. It’s the enterprise default because it puts each kind of content where its owner and change pattern want it.

Q: A change to a shared Kustomize base in a monorepo — what’s its blast radius, and how do you contain it? A: Its blast radius is every app that references the base. Contain it with path-scoped CI (validate all dependents when the base changes), CODEOWNERS making the base platform-owned, manifest-generate-paths so only dependents re-render, and — if a generator template is involved — RollingSync so the change rolls through apps in waves instead of flipping all at once.

Q: Why do you need both CODEOWNERS and an AppProject? Isn’t one enough? A: They gate different moments. CODEOWNERS (plus branch protection) is the pull-request gate — it stops an unreviewed change from merging. The AppProject is the sync gate — it stops a merged change from syncing outside its allow-list of repos, clusters, and namespaces. CODEOWNERS without an AppProject lets a merged mistake reach another team’s namespace; an AppProject without CODEOWNERS lets anyone who can merge change desired state unreviewed. You need both.

Q: What’s the single highest-leverage field on an AppProject for multi-tenancy, and why? A: An empty clusterResourceWhitelist: [] (paired with a namespace-scoped destinations glob). It means a compromised or fat-fingered app repo cannot create any cluster-scoped resource — no ClusterRole, no escaping its namespace — and the project rejects the sync before anything is applied. It converts a shared control plane into safe multi-tenancy.

Q: How does an SCM Provider generator decide which repos become Applications, and how do you stop it onboarding a rogue repo? A: It queries the Git provider API for repos in an org/group and applies filters. Restrict it with repositoryMatch (a name regex like ^team-.*-config$) and pathsExist (require a path like gitops/kustomization.yaml that proves the repo is genuinely yours). Add applicationsSync: create-update so the controller can never delete an app if a repo vanishes. Treat those filters as a security boundary.

Q: Your monorepo has 100 apps and syncs are slow — repo-server CPU is pinned. Walk through the fix. A: The bottleneck is render, which lives in the repo-server. Enable Git webhooks so refresh is event-driven instead of a 3-minute poll of every app; add argocd.argoproj.io/manifest-generate-paths to every Application so a push only re-renders apps whose paths changed; scale repo-server replicas and raise --parallelismlimit if still render-bound. Note that application-controller sharding shards by cluster, not app, so it won’t help a single-cluster monorepo’s render cost.

Q: In a monorepo CODEOWNERS, why does rule ordering matter, and what’s the safe convention? A: GitHub CODEOWNERS uses last match wins, so if a specific path rule appears before the general *, the general rule can override it and require the wrong (or no) owner. The safe convention is general-to-specific: put * first, then progressively more specific paths, so the most specific rule is last and wins. Always verify with a throwaway PR that the expected owner is requested.

Q: A team says “we need to change a label on every workload in the fleet.” How does the answer differ across the three topologies? A: In a monorepo it’s one PR (all app config is co-located). In a polyrepo it’s N PRs, one per team repo. In a hybrid it depends on what kind of change: a platform-level change (a shared component, an ApplicationSet template) is one PR in the platform monorepo, but a change to app content is still N PRs because app content is distributed on purpose. If it’s truly fleet-wide app content, consider moving the shared piece into a platform-owned Kustomize component so it becomes a single platform PR.

Q: When would you deliberately choose full polyrepo over hybrid? A: When teams must not even see each other’s config — hard regulatory or multi-tenant-SaaS isolation — so a shared platform monorepo (which every team can read) is unacceptable, and you accept the discovery and cross-cutting-change costs in exchange for maximal separation. Otherwise, hybrid gives you most of polyrepo’s isolation for app content while keeping the control plane coherent.

Q: You’re advising a 5-engineer startup with 12 apps and one team. Monorepo or hybrid? A: Monorepo. With one team and few apps, the ownership and blast-radius problems hybrid solves don’t exist yet, and the monorepo’s discovery and atomic-change benefits are pure upside. The right move is to design the seam — app-major apps/<app>/overlays/<env>/, per-path CODEOWNERS, an ApplicationSet already generating apps — so that splitting to hybrid later is mechanical, not a rewrite. Start simple; split when ownership pain actually arrives.


Key takeaways

argocdgitopskubernetesmonorepopolyrepoapplicationsetapp-of-appsappprojectcodeownersrbacmulti-tenancyscm-generatorplatform-engineering
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments