Building Multi-Tenant Kubernetes: Virtual Clusters, Hierarchical Namespaces, Quotas, and Isolation Tiers

In a nutshell

Picture one Kubernetes cluster as a shared apartment building that several families — teams, customers, a partner’s software — live in at once. Multi-tenancy is the craft of letting them all share that building (its plumbing, foundations, and elevators — your nodes and control plane) without one family flooding another’s flat, reading their mail, or hogging the hot water. You are trading the cost of one shared building against the blast radius of a shared wall.

Kubernetes gives you that isolation as a slider with three practical settings, from cheapest-and-softest to strongest-short-of-a-separate-building:

• Soft — a namespace per tenant. Each family gets a labelled flat — a Namespace — fenced with four soft locks: RBAC (who may open which door), a ResourceQuota (a cap on how much water and power the flat may draw), a NetworkPolicy (which flats may phone which), and Pod Security Admission (no knocking out load-bearing walls). Cheap, instant, and correct for teams you already trust. But every flat shares the building’s front desk (one API server) and its foundations (one kernel), so a namespace is not a security wall.
• Structured — Hierarchical Namespaces. When one family needs several rooms — dev, stage, prod — the Hierarchical Namespace Controller (HNC) lets them own a parent namespace and self-create subnamespaces beneath it that automatically inherit the parent’s locks. It removes the copy-paste drift of running dozens of flat namespaces by hand. This is an organisational upgrade, not a stronger wall — the front desk and foundations are still shared.
• Virtual — vcluster. When a family needs their own front desk — to install their own building rules (CRDs), run their own concierge (operators), or follow a different building code (API version) — a virtual cluster (vcluster) gives them a real, private Kubernetes API server running inside a single host namespace. It is the strongest isolation short of handing them a separate building. The twist: their furniture (pods) is still carried down and placed in the shared building’s rooms — vcluster pods run on the same host nodes, so it isolates the control plane but not the kernel.

The one idea to carry through the whole lesson: pick the weakest tier that satisfies each tenant’s threat model, and make the boundary something the cluster enforces, not something a wiki asks people to respect. A namespace is a naming-and-policy scope; a vcluster is API autonomy; only a separate cluster removes the shared kernel.

Level: Advanced · builds on the fundamentals · Time: ~36 min read · You’ll need first: what a Pod, Namespace, and CPU/memory requests/limits are, plus the basics of RBAC and NetworkPolicy (all covered in earlier lessons, linked below).

After this lesson you can:

• Classify your “tenants” into isolation tiers (trusted-internal / API-autonomous / untrusted) and pick a model per class rather than one globally.
• Explain precisely what a namespace does and does not isolate, and why “namespace = security boundary” is the field’s most expensive myth.
• Stand up soft tenancy — namespace + RBAC + ResourceQuota + LimitRange + default-deny NetworkPolicy + Pod Security Admission — as an enforced baseline.
• Use HNC to give tenants safe self-service subnamespaces with inherited policy, and a HierarchicalResourceQuota to budget a whole subtree.
• Give a tenant a vcluster for CRD/API autonomy, and articulate exactly what it isolates (API server, CRDs, RBAC, version) versus shares (nodes, kernel, CNI).
• Reach for sandboxed runtimes + dedicated nodes or a separate cluster when “tenant” means untrusted code.

This lesson builds directly on three earlier ones: Namespaces, quotas & LimitRanges for the governance objects, Network policies & default-deny for the traffic boundary, and containerd, gVisor & RuntimeClass for the kernel-isolation tier.

Read the diagram left → right as increasing isolation, not a pipeline — the three middle tiers are alternatives you choose between per tenant class. On the left, everyone shares one host cluster. Soft tenancy (cyan) fences a namespace with RBAC, quota, NetworkPolicy and PSA but leaves the API server and kernel shared — badge ①, the “namespace ≠ security boundary” truth. HNC (purple) adds a subnamespace tree that propagates policy downward (badge ②): tidier operations, same security tier. vcluster (green) hands the tenant their own API server with private CRDs and RBAC (badge ③), isolating the control plane (badge ④) — yet its pods still sync down to the shared host nodes (badge ⑤, red), which is why even a vcluster shares the kernel and untrusted code needs a sandbox or a separate cluster.

“Multi-tenancy” on Kubernetes is not one decision; it is a slider. At one end you give every tenant a namespace and trust RBAC. At the other you give every tenant their own cluster and trust nothing shared. Every option in between trades blast radius against cost and operational toil. The mistake teams make is picking a point on that slider once, globally, and then discovering six months in that their “tenants” are actually three different populations — internal app teams, a partner integration, and an untrusted CI workload — each of which needs a different isolation tier. This guide treats tenancy as a tiered design: pick the weakest model that satisfies the threat model for each tenant class, and make the boundary enforceable rather than aspirational.

1. Pick a tenancy model per tenant class, not per cluster

Kubernetes gives you a namespace as the unit of naming and RBAC, but a namespace is emphatically not a security boundary on its own. Pods in different namespaces share the same kernel, the same nodes, the same CNI, and — critically — the same API server. The model you choose decides which of those you actually isolate.

The decisive questions, in order:

1. Does the tenant need to install CRDs or cluster-scoped resources? If yes, a shared namespace is out — CRDs are global, and two tenants wanting different versions of the same CRD will collide. This pushes you to vcluster or a dedicated cluster.
2. Is the tenant trusted not to attempt kernel-level escape? If no — untrusted code, partner workloads, anything internet-facing and high-risk — soft isolation is insufficient regardless of policy, because a container escape lands on a shared node. This pushes you toward sandboxed runtimes (Section 6) or dedicated nodes/clusters.
3. Do tenants need to see each other’s objects at all? Soft isolation leaks API-server-level metadata (node names, events, sometimes other namespaces depending on RBAC). vcluster gives each tenant a syntactically complete, separate Kubernetes API.

The principal-level framing: soft multi-tenancy is a cost optimization, not a security control. It is correct for tenants you would already trust on a shared cluster. The moment “tenant” means “someone I do not trust,” you are choosing between a sandbox runtime and a separate cluster, and you should price both before assuming soft tenancy is cheaper.

New to this? Read the table as a ladder, not a menu of equals. Each rung down isolates one more thing at more cost. “Isolates API server” means the tenant gets their own Kubernetes control plane — they cannot even see your other tenants’ objects. “Kernel/node isolation” means their code cannot escape onto a node shared with someone else. Most teams over-buy: they reach for cluster-per-tenant when a namespace with the soft-tenancy stack below would have been correct and nearly free.

The soft-tenancy stack — five layers on one namespace. When people say “soft multi-tenancy,” they mean a namespace hardened with five cooperating controls, each with its own lesson in this course: RBAC binds the tenant’s group to a namespaced Role so they can act only inside their namespace; a ResourceQuota + LimitRange cap and default their consumption (namespaces, quotas & LimitRanges, and Section 2 below); a default-deny NetworkPolicy fences their traffic (network policies & default-deny, and Section 5); and Pod Security Admission (PSA) — the built-in admission controller that labels each namespace baseline or restricted — stops a tenant from running privileged, host-mounting, or root pods. Miss any one layer and the “tenancy” leaks: no PSA and a tenant runs a privileged pod that mounts the host filesystem; no NetworkPolicy and they port-scan their neighbours; no quota and one buggy Deployment starves the cluster. Soft tenancy is all five together, enforced by admission — or it is theatre.

2. Resource governance: ResourceQuota, LimitRange, and priority fairness

Before isolation, solve sharing. The first failure mode in any shared cluster is not a breach — it is one tenant scheduling 400 pods and starving everyone else. Three objects fix this, and they are not interchangeable.

ResourceQuota caps the aggregate a namespace may consume. LimitRange constrains individual objects and supplies defaults so pods without explicit requests do not slip through the quota accounting.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: tenant-acme
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 64Gi
    limits.cpu: "40"
    limits.memory: 128Gi
    pods: "100"
    services.loadbalancers: "2"
    count/persistentvolumeclaims: "20"
    requests.storage: 500Gi

A ResourceQuota that constrains requests.cpu or limits.memory has a sharp edge: once it is in force, every pod in the namespace must declare the corresponding request/limit, or admission rejects the pod. Tenants will not do this reliably. LimitRange backstops them with defaults and floors:

apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-defaults
  namespace: tenant-acme
spec:
  limits:
    - type: Container
      default:           # applied as limit if pod omits one
        cpu: "500m"
        memory: 256Mi
      defaultRequest:    # applied as request if pod omits one
        cpu: "100m"
        memory: 128Mi
      max:
        cpu: "4"
        memory: 8Gi
      min:
        cpu: "50m"
        memory: 32Mi

Quota stops a tenant from consuming too much in total; it does not guarantee that under contention their pods win against another tenant’s. For that you need scheduling priority. Define a per-tier PriorityClass and pin tenant pods to it so a platinum tenant preempts a free-tier batch job rather than queueing behind it:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-platinum
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Production tenants; may preempt free-tier/batch."

Crucially, also quota the priority classes themselves with a scoped ResourceQuota, or a free-tier tenant will simply set priorityClassName: tenant-platinum and defeat the whole scheme:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: deny-high-priority
  namespace: tenant-free-001
spec:
  hard:
    pods: "0"                 # zero pods allowed at this priority...
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["tenant-platinum"]

API-server fairness is the third axis. A tenant hammering the API with list/watch loops can degrade the control plane for everyone. API Priority and Fairness (APF, GA since 1.29) lets you carve API concurrency into queues. Bound a noisy tenant’s service accounts to a low-share PriorityLevelConfiguration via a FlowSchema so their requests cannot crowd out control-plane traffic.

3. Hierarchical Namespace Controller for policy inheritance

Flat namespaces force you to copy RBAC, quotas, and NetworkPolicies into every tenant by hand, and they drift. The Hierarchical Namespace Controller (HNC) — a kubernetes-sigs project — adds parent/child relationships so policy propagates down a tree. A tenant gets a root namespace; their environments become subnamespaces that inherit the tenant’s RoleBindings automatically.

# Install the HNC CRDs and controller (pin a real release tag in production)
HNC_VERSION=v1.1.0
kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/${HNC_VERSION}/default.yaml

# Install the kubectl-hns plugin for ergonomics
kubectl krew install hns

Create a tenant root, then self-service subnamespaces beneath it. A SubnamespaceAnchor in the parent is the request; HNC creates the actual namespace and enforces that it cannot outlive or escape its parent:

kubectl create namespace tenant-acme
kubectl hns create acme-dev   -n tenant-acme    # creates subnamespace, parented to tenant-acme
kubectl hns create acme-stage -n tenant-acme
kubectl hns tree tenant-acme
# tenant-acme
# ├── acme-dev
# └── acme-stage

Now anything you place in tenant-acme — a RoleBinding granting the tenant’s group edit, a default NetworkPolicy, a LimitRange — propagates into every child. HNC marks propagated objects and prevents children from deleting them. You control what propagates per type:

# Propagate RoleBindings and LimitRanges down every tree; do NOT propagate ResourceQuotas
# (you usually want per-namespace quotas, not an inherited one)
kubectl hns config set-resource rolebindings --mode Propagate
kubectl hns config set-resource limitranges  --mode Propagate
kubectl hns config set-resource resourcequotas --mode Remove

HNC’s gain is operational, not security: it does not isolate the API server or the kernel. It eliminates the drift and copy-paste that make flat soft-tenancy unmaintainable past a few dozen namespaces, and it gives tenants a safe self-service primitive (create a subnamespace) without cluster-admin.

4. Virtual clusters with vcluster: API isolation and CRD freedom

When a tenant needs their own Kubernetes API — to install CRDs, run their own operators, define cluster-scoped RBAC, or pin a different API behavior — a shared namespace cannot deliver it. A virtual cluster (vcluster, by LoftLabs) runs a real, lightweight Kubernetes control plane (API server + controller-manager backed by an embedded datastore such as SQLite or an external one) inside a single host namespace. The tenant’s API server is genuinely separate; their pods, however, are synced down and scheduled on the host cluster’s nodes, so you keep one pool of compute.

# Install the vcluster CLI, then create a virtual cluster inside a host namespace
vcluster create acme --namespace tenant-acme-vc

# Connect: this opens a kubeconfig context pointed at the tenant's *own* API server
vcluster connect acme --namespace tenant-acme-vc
kubectl get namespaces            # tenant sees only THEIR namespaces, not the host's
kubectl apply -f some-crd.yaml    # tenant installs CRDs freely; isolated to their vcluster

What the tenant sees is a clean cluster. What actually happens: the vcluster syncer translates the tenant’s pods, services, secrets, and configmaps into the host namespace (rewriting names to avoid collisions) and schedules them on host nodes. CRDs, RBAC, and most cluster-scoped objects live only in the virtual control plane and never touch the host API.

Pin the host-side blast radius with a values.yaml that disables host node visibility and constrains what the syncer is allowed to do:

# vcluster.yaml — install with: vcluster create acme -n tenant-acme-vc -f vcluster.yaml
sync:
  toHost:
    pods:
      enabled: true
    ingresses:
      enabled: true
  fromHost:
    nodes:
      enabled: true
      selector:
        # tenant's vcluster only "sees" nodes in this pool, for scheduling
        labels:
          tenant-pool: "acme"
controlPlane:
  distro:
    k8s:
      enabled: true

vcluster’s isolation properties: strong at the API/control-plane layer (separate API server, separate etcd-equivalent, separate CRD and RBAC space), weak at the kernel layer (pods still land on shared host nodes). It is the right tier for tenants you trust at the kernel level but who need API autonomy. For untrusted tenants, combine vcluster with the runtime isolation in Section 6, or do not share nodes at all.

5. Network isolation: default-deny and per-tenant ingress

Tenancy is only as strong as the network boundary, and the default boundary is none — every pod can reach every other pod cluster-wide. Establish a baseline default-deny per tenant namespace (in the soft/HNC model) so a compromised pod in tenant-acme cannot reach tenant-globex:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: tenant-acme
spec:
  podSelector: {}                 # selects every pod in the namespace
  policyTypes: [Ingress, Egress]
  # no ingress/egress rules => deny both directions

Then add back only intended flows. The two flows every tenant needs: DNS egress to kube-dns, and ingress from the shared ingress controller. Allow them narrowly with a namespace selector rather than opening the namespace up:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-ingress
  namespace: tenant-acme
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx

Two operational notes that bite teams. First, NetworkPolicy is enforced by the CNI, not the API server; on a CNI that ignores it (e.g., a plain Flannel install) the YAML applies and does nothing — verify enforcement before trusting it. Second, in a soft-tenancy model you cannot let tenants self-define cross-namespace allows, or they will simply allow themselves into a neighbor; for cross-tenant rules, use an admission policy (Kyverno/Gatekeeper) that forbids namespaceSelector references outside the tenant’s own subtree. For per-tenant ingress, give each tenant a hostname/path on a shared controller and isolate TLS with a tenant-scoped Secret, or run a dedicated ingress controller per high-tier tenant if you need full data-plane separation.

6. Runtime and node isolation with gVisor/Kata and dedicated pools

For tenants you do not trust at the kernel level, no amount of NetworkPolicy or RBAC helps: a container escape is a host compromise, and the host is shared. Two mechanisms raise the floor.

Sandboxed runtimes put a barrier between the container and the host kernel. gVisor (runsc) intercepts syscalls in a userspace kernel; Kata Containers run each pod in a lightweight VM. Register the runtime as a RuntimeClass, then opt tenant pods into it:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc        # must match the containerd runtime handler configured on nodes

apiVersion: v1
kind: Pod
metadata:
  name: untrusted-job
  namespace: tenant-partner
spec:
  runtimeClassName: gvisor     # this pod runs under the sandbox
  containers:
    - name: app
      image: registry.example.com/partner/job:1.4.2

Enforce that every pod in an untrusted tenant’s namespace uses the sandbox with an admission policy, so a tenant cannot omit runtimeClassName and land on the bare runtime. Dedicated node pools complete the picture: taint a pool for the tenant and require their pods (often via the sandboxed RuntimeClass’s scheduling block) to tolerate it, so untrusted workloads never co-locate with platform or other-tenant pods:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor-isolated
handler: runsc
scheduling:
  nodeSelector:
    tenant-isolation: sandboxed
  tolerations:
    - key: tenant-isolation
      operator: Equal
      value: sandboxed
      effect: NoSchedule

The honest tradeoff: sandboxes add per-pod overhead and break some workloads (certain syscalls, some CSI drivers, GPU passthrough). They are the correct tier when “tenant” means untrusted code but a full cluster-per-tenant is too expensive. When in doubt for regulated or hostile workloads, separate clusters remain the only model with no shared kernel. The runtime mechanics — how runtimeClassName routes through the kubelet and containerd to runsc or Kata — are covered in depth in containerd, gVisor & RuntimeClass.

7. Tenant onboarding automation and self-service guardrails

None of the above scales if onboarding a tenant is a ticket. Encode a tenant as a single declarative object and let a controller fan it out into namespace + quota + RBAC + NetworkPolicy + (optionally) a vcluster. The Capsule project (clastix) models this as a Tenant CRD; many platforms build their own. The shape:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: acme
spec:
  owners:
    - name: acme-admins
      kind: Group
  namespaceOptions:
    quota: 5                     # tenant may self-create up to 5 namespaces
  resourceQuotas:
    scope: Tenant                # quota pooled across all the tenant's namespaces
    items:
      - hard:
          limits.cpu: "40"
          limits.memory: 128Gi
  networkPolicies:
    items:
      - policyTypes: [Ingress, Egress]
        podSelector: {}
        egress:
          - to:
              - namespaceSelector:
                  matchLabels:
                    kubernetes.io/metadata.name: kube-system
                ports: [{ protocol: UDP, port: 53 }]

The guardrail principle: tenants get self-service within a fence. They can create namespaces (up to a cap), deploy workloads, and define their own intra-tenant policy — but they cannot exceed quota, escape their network boundary, schedule onto other tenants’ nodes, or grant themselves cluster-admin. Anything they cannot be trusted to self-serve becomes an admission rule (Kyverno ClusterPolicy / Gatekeeper Constraint) that rejects the violating object at apply time, not a wiki page asking them not to.

Going deeper

The soft-vs-hard tenancy spectrum — and why a namespace is not a security boundary

“Hard” versus “soft” tenancy is not marketing; it is a precise question — what does the boundary actually stop? Line up everything two tenants share on a single cluster and cross each off as you climb tiers:

(* a vcluster runs its own CoreDNS inside the virtual cluster, but the pods backing it still land on host nodes.)

A namespace is a naming and RBAC scope, and almost nothing else. It does not get its own kernel (a container escape on a shared node reaches every namespace’s pods on that node); it does not get its own API server (the same kube-apiserver authenticates and authorises every tenant, so a bug or an over-broad ClusterRole leaks across all of them at once); it does not get its own network dataplane (without a NetworkPolicy, every pod routes to every other pod cluster-wide); and it does not get its own CRD registry (a CRD is global — two tenants cannot install different versions of the same one). That is why the whole discipline keeps repeating the point: soft tenancy is a cost optimisation among tenants you trust, and every “isolation” you layer on top (quota, NetworkPolicy, PSA) is a policy the shared control plane enforces on their behalf. Remove the policy and the isolation evaporates, because the substrate underneath was always shared. Hard tenancy is the tier where the substrate itself is no longer shared — a separate control plane, and ideally a separate kernel.

vcluster internals: the virtual control plane, the syncer, and the host namespace

A vcluster has three moving parts, and understanding them tells you exactly what it can and cannot isolate.

1. The virtual control plane. A real kube-apiserver (or k3s/k0s in some distros) plus a controller-manager, backed by an embedded datastore — SQLite by default, or etcd/PostgreSQL/MySQL via Kine for HA. This runs as ordinary pods inside one host namespace. When the tenant runs kubectl, they talk to this API server, not the host’s. Every object they create — Deployments, CRDs, RBAC, ConfigMaps — is written to their datastore first.
2. The syncer. A single component that watches the virtual API and copies a curated subset of objects down into the host namespace, rewriting names to avoid collisions (a pod web in vcluster acme becomes something like web-x1y2-acme on the host). Low-level resources that must exist on a real node — Pods, Services, Endpoints, ConfigMaps, Secrets, PVCs — are synced to the host. High-level and cluster-scoped resources — Deployments, ReplicaSets, CRDs, RBAC, ServiceAccounts, most operators’ custom resources — stay only in the virtual control plane and are reconciled there. A few host resources (Nodes, and optionally IngressClasses/StorageClasses) are synced from the host so the tenant can schedule and see capacity.
3. The host namespace. Everything the syncer pushes down lives in one namespace on the host cluster. That is the vcluster’s real footprint and its blast radius: pin it with a host-side ResourceQuota, a default-deny NetworkPolicy, and PSA — exactly as you would any soft tenant. Because from the host’s point of view, a vcluster is a soft tenant whose pods happen to be driven by a second API server.

So the isolation ledger is unambiguous. Isolated: the API server, the datastore, the CRD namespace, cluster-scoped RBAC, the API version and feature-gates, and the tenant’s view (they never see host or other-tenant objects). Shared: the nodes, the kernel, the CNI dataplane, the container runtime, and the host scheduler that ultimately places every synced pod. This is why vcluster is the correct tier for API autonomy among kernel-trusted tenants, and the wrong tier, on its own, for untrusted code — a truth badge ④ and ⑤ of the diagram exist to hammer home.

HNC propagation and tree quotas

HNC’s model is a labelled forest. A subnamespace is created by a SubnamespaceAnchor object placed in the parent — that is literally what kubectl hns create writes:

apiVersion: hnc.x-k8s.io/v1alpha2
kind: SubnamespaceAnchor
metadata:
  name: acme-dev
  namespace: tenant-acme

Objects marked for propagation flow parent → all descendants; HNC re-creates them and guards them, so a child can neither edit nor delete a propagated object, nor orphan itself from its parent. You choose per-type behaviour — Propagate, Remove, or Ignore. RoleBindings, NetworkPolicies, LimitRanges, ConfigMaps, and Secrets are common propagators; ResourceQuotas usually are not, because you want per-namespace budgets rather than one inherited cap applied verbatim to every child.

For a pooled budget across a whole subtree, HNC ships the HierarchicalResourceQuota (HRQ) — a quota enforced across a namespace and all its descendants combined, so a tenant with five subnamespaces cannot exceed the parent’s total no matter how they spread pods:

apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchicalResourceQuota
metadata:
  name: team-budget
  namespace: tenant-acme      # the PARENT; caps tenant-acme + every descendant combined
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 128Gi
    pods: "200"

The gotchas that surprise teams: propagation is eventually-consistent (a RoleBinding dropped in the parent appears in children a moment later, not instantly); it guards rather than overwrites (if a child already had a same-named object, HNC raises a condition you must read with kubectl hns tree, instead of silently clobbering); and it flows only downward, so deleting a parent namespace cascades to every subnamespace beneath it. HNC deletion is a subtree rm -rf, not a single-namespace delete — a fact worth a second look before you kubectl delete ns a tenant root.

Noisy neighbours: quota is not fairness, and there are two axes

A ResourceQuota caps how much a tenant may request over time; it says nothing about who wins when the cluster is full right now. Those are different failures with different tools. Scheduling contention is decided by PriorityClass + preemption — a high-priority pod can evict a lower-priority one to schedule — which is exactly why Section 2 insists you also quota the PriorityClass, or any tenant names the platinum class and defeats the scheme.

The subtler axis is the control plane itself: a tenant running tight list/watch loops against the shared API server can throttle everyone, no pod scheduling involved. That is what API Priority and Fairness (APF) governs — it shards API concurrency into priority levels fed by flow schemas, so one tenant’s request storm queues in their lane instead of starving control-plane traffic:

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: tenant-low
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10        # this level's slice of API concurrency
    limitResponse:
      type: Queue
      queuing:
        queues: 32
        queueLengthLimit: 50
        handSize: 6
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: tenant-acme-flows
spec:
  priorityLevelConfiguration:
    name: tenant-low
  matchingPrecedence: 1000
  distinguisherMethod:
    type: ByUser
  rules:
    - subjects:
        - kind: ServiceAccount
          serviceAccount:
            namespace: tenant-acme
            name: "*"
      resourceRules:
        - verbs: ["*"]
          apiGroups: ["*"]
          resources: ["*"]
          namespaces: ["*"]

APF is GA (flowcontrol.apiserver.k8s.io/v1, GA since 1.29). Watch apiserver_flowcontrol_rejected_requests_total and apiserver_flowcontrol_current_inqueue_requests to see a tenant being shaped (queued) rather than dropped — the goal is to protect the API server’s own liveness and control-plane traffic, not to punish the tenant.

Choosing: vcluster vs separate clusters vs Capsule/kiosk

Capsule (clastix) and the older kiosk both model a Tenant as an object that fans out into namespaces + quota + RBAC + NetworkPolicy — they are soft-tenancy orchestrators, giving you fleet management and self-service without a second API server. vcluster sits on a different axis: API autonomy. They compose cleanly — many platforms run Capsule for the trusted majority and hand a vcluster to the few tenants who genuinely need CRDs.

The cost dimension

Price the tiers before assuming soft is cheapest. Soft namespaces and HNC add near-zero marginal cost — they are objects in the control plane you already run. A vcluster costs one small always-on control-plane pod set per tenant (tens to low-hundreds of MiB of memory plus a little CPU) and its datastore — cheap enough to run hundreds, far cheaper than a real cluster, but not free the way a namespace is. A separate managed cluster costs the control-plane fee (on managed platforms), a baseline of system daemonsets and add-ons per cluster, and — the real bill — the human toil of patching, upgrading, and observing N clusters instead of one. The decision is rarely “which is technically strongest” (a separate cluster always is); it is “what is the weakest tier that meets this tenant’s threat model,” because every tier up multiplies operational cost. That framing — weakest sufficient isolation — is the whole discipline of platform multi-tenancy.

Verify

Prove each boundary holds before you onboard a real tenant. Treat green output as the contract.

# 1. Quota is enforced: this should be REJECTED once it exceeds requests.cpu
kubectl -n tenant-acme run quota-test --image=pause \
  --overrides='{"spec":{"containers":[{"name":"c","image":"pause",
  "resources":{"requests":{"cpu":"50"}}}]}}'
# Expected: error ... exceeded quota: tenant-quota

# 2. Default-deny holds: a pod in tenant-acme cannot reach a pod in tenant-globex
kubectl -n tenant-acme run probe --image=nicolaka/netshoot --rm -it --restart=Never -- \
  curl -m 4 http://svc.tenant-globex.svc.cluster.local
# Expected: timeout / connection refused (NOT a 200)

# 3. Priority-class quota holds: a free tenant cannot claim platinum priority
kubectl -n tenant-free-001 run sneaky --image=pause \
  --overrides='{"spec":{"priorityClassName":"tenant-platinum"}}'
# Expected: forbidden: exceeded quota: deny-high-priority

# 4. vcluster API isolation: tenant context sees only their namespaces
vcluster connect acme -n tenant-acme-vc -- kubectl get ns
# Expected: default, kube-system, etc. of the VIRTUAL cluster only — never host tenants

# 5. Sandbox is actually in use (run inside the pod scheduled with runtimeClassName: gvisor)
kubectl -n tenant-partner exec untrusted-job -- dmesg 2>&1 | head -1
# Expected under gVisor: "Operation not permitted" — runsc blocks dmesg; bare runc would succeed

Also confirm the CNI enforces policy at all (a no-op CNI silently passes test 2 for the wrong reason): apply a deny-all, then prove a flow that should break actually breaks, before trusting that flows you allowed are the only ones open.

Enterprise scenario

A fintech platform team ran a shared “internal-apps” EKS cluster with namespace-per-team and RBAC — soft tenancy, and fine for years. Then a new requirement landed: a regulated reconciliation product had to onboard a third-party vendor’s batch engine that shipped as a Helm chart bundling its own CRDs and a cluster-scoped operator. Two problems collided. The vendor’s operator wanted cluster-wide CRD installation, which would have been visible to and collidable with every other team. And compliance classified the vendor code as untrusted, forbidding it from sharing a kernel with workloads that touched cardholder data.

A full cluster-per-vendor was the obvious answer and got rejected on cost and lead time — provisioning, patching, and observability for a new cluster per vendor integration was weeks of platform toil they could not absorb per deal. The team split the requirement across two tiers instead. They gave the vendor a vcluster for API/CRD autonomy, so the vendor’s operator and CRDs lived entirely inside the virtual control plane and never touched the host API or other teams. Then they satisfied the kernel-isolation requirement by pinning that vcluster’s pods to a dedicated, tainted node pool running gVisor, enforced with a Kyverno policy that rejected any pod in the synced host namespace lacking the sandbox RuntimeClass:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-sandbox-for-vendor
spec:
  validationFailureAction: Enforce
  rules:
    - name: vendor-pods-must-be-sandboxed
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["tenant-vendor-vc"]   # the host namespace backing the vcluster
      validate:
        message: "Vendor pods must run under the gvisor-isolated RuntimeClass."
        pattern:
          spec:
            runtimeClassName: "gvisor-isolated"

The result: API autonomy without a new cluster, kernel isolation without trusting the vendor, and a boundary enforced by admission rather than convention. The deciding insight was that “untrusted tenant needing CRDs” is not one tier on the slider — it is two separate boundaries (control-plane and kernel) that you compose, and vcluster-plus-sandbox composed them at a fraction of cluster-per-tenant cost. Six months later they had a repeatable vendor-onboarding pattern: one Tenant object, one vcluster, one sandboxed pool.

Practice challenges

Work these on a free local cluster (kind create cluster or k3d cluster create) plus the vcluster and kubectl-hns CLIs where noted — no paid infrastructure is required. Each has a graded difficulty and a hidden solution; try it before you peek. Outputs shown are representative.

Challenge 1 (Beginner) — Prove a namespace is not a wall. Create two namespaces tenant-a and tenant-b, run an nginx pod and Service in tenant-b, then curl it from a pod in tenant-a. Does it connect? What single object would stop it?

<details> <summary>Solution</summary>

kubectl create ns tenant-a; kubectl create ns tenant-b
kubectl -n tenant-b run web --image=nginx:1.27 --port=80
kubectl -n tenant-b expose pod web --port=80
kubectl -n tenant-a run probe --image=nicolaka/netshoot --rm -it --restart=Never -- \
  curl -s -m 4 http://web.tenant-b.svc.cluster.local | head -1

Representative output:

<!DOCTYPE html>

Why: the pod network is flat by default — tenant-a reaches tenant-b with nothing in the way. A default-deny NetworkPolicy in tenant-b is the object that closes it. The namespace was never the boundary. </details>

Challenge 2 (Beginner) — Quota needs a LimitRange partner. Put requests.cpu: "2" on a namespace via ResourceQuota, then kubectl run a pod with no resources. Explain the rejection and fix it without editing the pod.

<details> <summary>Solution</summary>

kubectl create ns t2
kubectl -n t2 create quota q --hard=requests.cpu=2
kubectl -n t2 run p --image=nginx:1.27 --restart=Never
# Error from server (Forbidden): pods "p" is forbidden: failed quota: q:
#   must specify requests.cpu
kubectl -n t2 apply -f - <<'EOF'
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: t2 }
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: "100m" }
      default: { cpu: "200m" }
EOF
kubectl -n t2 run p --image=nginx:1.27 --restart=Never   # now admitted

Why: once a compute resource is under quota, every pod must declare it. A LimitRange supplies the default so the pod carries a value to count. Quota and LimitRange are partners, never substitutes. </details>

Challenge 3 (Intermediate) — HNC subnamespace inheritance. Install HNC, create parent tenant-acme, add a RoleBinding to it, self-create subnamespace acme-dev, and prove the RoleBinding propagated.

<details> <summary>Solution</summary>

kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/v1.1.0/default.yaml
kubectl krew install hns
kubectl create ns tenant-acme
kubectl -n tenant-acme create rolebinding acme-edit \
  --clusterrole=edit --group=acme-devs
kubectl hns create acme-dev -n tenant-acme
kubectl -n acme-dev get rolebinding

Representative output:

NAME        ROLE               AGE
acme-edit   ClusterRole/edit   3s     # propagated by HNC, marked as managed

Why: HNC copies propagatable objects (RoleBindings among them) from parent to every descendant and guards them — acme-dev cannot delete acme-edit. This is the anti-drift primitive that makes soft tenancy scale past a few dozen namespaces. </details>

Challenge 4 (Intermediate) — Close the PriorityClass escape. A tenant-platinum PriorityClass (value: 100000) exists. Prove a free-tier namespace can grab it, then write the quota that forbids it.

<details> <summary>Solution</summary>

apiVersion: v1
kind: ResourceQuota
metadata:
  name: deny-high-priority
  namespace: tenant-free-001
spec:
  hard:
    pods: "0"                       # zero pods allowed AT this priority
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["tenant-platinum"]

kubectl -n tenant-free-001 run sneaky --image=nginx:1.27 \
  --overrides='{"spec":{"priorityClassName":"tenant-platinum"}}'
# Error ... forbidden: exceeded quota: deny-high-priority,
#   requested: pods=1, used: pods=0, limited: pods=0

Why: a PriorityClass is cluster-scoped and nameable by anyone. A scoped ResourceQuota with scopeName: PriorityClass caps how many pods a namespace may run at that priority — set it to 0 for low tiers or your fairness scheme is decorative. </details>

Challenge 5 (Advanced) — vcluster CRD autonomy. Create a vcluster, connect to it, install a CRD inside it, and prove the CRD does not exist on the host cluster.

<details> <summary>Solution</summary>

vcluster create acme -n tenant-acme-vc
vcluster connect acme -n tenant-acme-vc      # kubectl now talks to the VIRTUAL cluster
kubectl apply -f - <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  scope: Namespaced
  names: { plural: widgets, singular: widget, kind: Widget }
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec: { type: object }
EOF
kubectl get crds | grep widgets                 # present INSIDE the vcluster
vcluster disconnect
kubectl get crds | grep widgets || echo "not on host — isolated"
#   not on host — isolated

Why: CRDs live only in the virtual control plane’s datastore; the syncer never pushes them to the host API. Two vclusters can install different versions of widgets.example.com without colliding — impossible in shared soft tenancy, where a CRD is global. </details>

Challenge 6 (Advanced) — Pool a budget across a subtree. Using HNC, cap tenant-acme and all its subnamespaces combined at 200 pods with a HierarchicalResourceQuota, and reason about what happens when acme-dev alone tries to launch the 201st.

<details> <summary>Solution</summary>

apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchicalResourceQuota
metadata:
  name: team-budget
  namespace: tenant-acme          # parent; caps the whole subtree
spec:
  hard:
    pods: "200"

kubectl -n tenant-acme apply -f team-budget.yaml
kubectl -n tenant-acme get hrq team-budget -o jsonpath='{.status.used}'; echo
# {"pods":"200"}  -> the 201st pod in ANY descendant is now rejected

Why: an HRQ is enforced across the parent and every descendant, so the tenant cannot dodge the cap by spreading pods across subnamespaces. Plain per-namespace ResourceQuotas would each get their own 200 — a very different, much looser budget. </details>

Common beginner mistakes

These are mental-model slips, not error messages — the misconceptions that send people to the wrong isolation tier. (For symptom → cause → fix, use the Pitfalls list below; this section is about why the model in your head is wrong.)

1. “A namespace isolates my tenant, so soft tenancy is secure.” Why it’s wrong: a namespace scopes names and RBAC and little else — the API server, the node kernels, the CNI dataplane, and CoreDNS are all shared. Every “isolation” you feel (quota, NetworkPolicy, PSA) is a policy the shared control plane applies for you; delete the policy and it is gone. Right model: soft tenancy is a cost optimisation among tenants you already trust. The instant “tenant” means “code I don’t trust,” you are choosing between a sandboxed runtime and a separate cluster — not between namespaces.

2. “I gave the tenant a quota, so noisy neighbours are solved.” Why it’s wrong: a ResourceQuota caps total requested capacity over time; it says nothing about who wins when the cluster is full right now, and nothing about a tenant hammering the API server with watch loops. Right model: three different problems, three different tools — ResourceQuota for aggregate consumption, PriorityClass + preemption for scheduling contention (with a scoped quota so low tiers cannot name the high class), and API Priority & Fairness for control-plane load.

3. “vcluster gives the tenant their own cluster, so untrusted code is fine in one.” Why it’s wrong: vcluster isolates the control plane — API server, CRDs, RBAC, version — but its pods are synced down and run as ordinary pods on the shared host nodes, under one kernel. A container escape from a vcluster pod is still a host compromise. Right model: vcluster = API autonomy for kernel-trusted tenants. For untrusted code, add gVisor/Kata on a tainted node pool, or use a separate cluster.

4. “CRDs are fine in soft tenancy — each team installs what they need.” Why it’s wrong: a CRD is a cluster-scoped, global object. Two teams that install different versions of widgets.example.com collide, and one team’s operator watching that CRD sees every namespace’s custom resources. Right model: if a tenant needs their own CRDs or operators, that requirement alone pushes them off shared namespaces onto a vcluster (or a separate cluster). Shared CRDs are a cross-tenant coupling, not a convenience.

5. “HNC propagation is automatic, so I can drop policy in the parent and forget it.” Why it’s wrong: propagation is real but has edges — it is eventually-consistent (children lag a moment), it guards propagated objects (a child that already had a same-named object gets a conflict condition, not a silent overwrite), and it flows only downward, so deleting a parent namespace cascades to the entire subtree. Right model: treat the parent namespace as the single source of truth for shared policy, watch kubectl hns tree for conflict conditions, and remember that deleting a parent is a subtree rm -rf, not a single-namespace delete.

6. “We’ll just give every tenant their own cluster to be safe.” Why it’s wrong: a separate cluster is the strongest boundary, but you pay for it per cluster, forever — control-plane fees, per-cluster add-ons and daemonsets, and the human cost of patching, upgrading, and observing N clusters. Teams that reflexively pick cluster-per-tenant drown in operational toil. Right model: pick the weakest tier that meets the threat model. Most “tenants” are trusted internal teams for whom a fenced namespace is correct and nearly free; reserve separate clusters for genuinely untrusted or compliance-bound workloads.

Checklist

• Classified tenants into tiers (trusted internal / API-autonomous / untrusted) before picking a model
• ResourceQuota per tenant on requests, limits, pods, PVCs, and loadbalancers
• LimitRange supplying defaults so quota-required requests/limits are never missing
• Per-tier PriorityClass, and a scoped quota preventing low tiers from claiming high priority
• Default-deny NetworkPolicy per tenant namespace, with DNS + ingress added back narrowly
• Verified the CNI actually enforces policy (not a silent no-op)
• Untrusted tenants pinned to a sandboxed RuntimeClass on a tainted, dedicated node pool
• Sandbox/quota/network requirements enforced by admission (Kyverno/Gatekeeper), not documentation
• HNC or a Tenant CRD propagating RBAC/policy so soft-tenancy does not drift
• vcluster for any tenant needing CRDs or their own API server
• API Priority and Fairness FlowSchema bounding noisy tenants’ API concurrency
• Tenant onboarding reduced to one declarative object, not a ticket

Pitfalls

• Treating a namespace as a security boundary. It isolates names and RBAC, nothing else. Shared kernel, nodes, CNI, and API server remain. Match the model to the threat, not to the org chart.
• Quota without LimitRange. The first pod a tenant submits without an explicit request gets rejected by quota admission, and they will blame the platform. Always pair them.
• Forgetting to quota the PriorityClass. A value: 100000 priority class is worthless if any tenant can name it. Cap high-priority pod counts per low tier or your fairness scheme is decorative.
• Letting tenants write cross-namespace NetworkPolicies. In soft tenancy this lets a tenant allow themselves into a neighbor. Constrain namespaceSelector via admission to the tenant’s own subtree.
• Assuming vcluster sandboxes the kernel. It isolates the API/control plane only; pods run on shared host nodes. Compose it with gVisor/Kata or dedicated nodes for untrusted tenants.
• Self-service without a fence. “Tenants can create namespaces” must be bounded (cap, quota, mandatory baseline policy) or you have handed out cluster-admin with extra steps.

Glossary

• Multi-tenancy — running several independent teams, environments, or customers (“tenants”) on one shared Kubernetes cluster without them interfering with each other.
• Tenant — a unit of isolation: a team, an environment, a customer, or a third-party workload that should not see or affect the others.
• Soft (multi-)tenancy — sharing a cluster with tenants you trust, isolated only by namespace-level policy (RBAC + quota + NetworkPolicy + PSA). Cheap; not a security boundary against untrusted code.
• Hard (multi-)tenancy — isolation strong enough for tenants you do not trust: at minimum a separate control plane (vcluster) and often a separate kernel (sandbox or separate cluster).
• Namespace — the Kubernetes unit of naming and RBAC scope. Groups objects and prevents name clashes; does not by itself isolate the kernel, nodes, network, or API server.
• ResourceQuota — a namespace-scoped cap on aggregate consumption (total CPU/memory requested, pod/PVC/loadbalancer counts).
• LimitRange — per-object min/max and default requests/limits in a namespace, so pods that omit sizes still carry values the quota can count.
• PriorityClass — a named scheduling priority; higher-value pods can preempt lower ones under contention. Must itself be quota-guarded so low tiers cannot claim it.
• API Priority and Fairness (APF) — the kube-apiserver mechanism that shards request concurrency into priority levels (via FlowSchema + PriorityLevelConfiguration) so one tenant’s API load cannot starve the control plane. GA since 1.29.
• NetworkPolicy — a namespaced, CNI-enforced allow-list for pod-to-pod traffic. A default-deny policy per tenant is the baseline network boundary; it is a no-op if the CNI does not enforce it.
• Pod Security Admission (PSA) — the built-in admission controller that enforces the Pod Security Standards (privileged/baseline/restricted) per namespace via labels, blocking privileged or host-mounting pods.
• RBAC — role-based access control: bindings that grant a subject (user, group, ServiceAccount) verbs on resources, scoped to a namespace or cluster-wide. The “who may act” layer of soft tenancy.
• Hierarchical Namespace Controller (HNC) — a kubernetes-sigs project adding parent/child relationships between namespaces so policy (RBAC, LimitRanges, NetworkPolicies) propagates down a tree.
• Subnamespace — a namespace created under a parent via a SubnamespaceAnchor; it inherits the parent’s propagated policy and cannot outlive or escape its parent.
• SubnamespaceAnchor — the small object placed in a parent namespace that requests HNC to create a subnamespace; kubectl hns create writes one.
• HierarchicalResourceQuota (HRQ) — an HNC quota enforced across a namespace and all its descendants combined, so a tenant cannot dodge the cap by spreading pods across subnamespaces.
• Virtual cluster (vcluster) — a real, lightweight Kubernetes control plane (API server + controller-manager + datastore) running inside a single host namespace, giving a tenant their own API, CRDs, and RBAC while sharing the host’s nodes. By Loft Labs.
• Virtual control plane — the API server + controller-manager + embedded datastore that make up a vcluster; where the tenant’s high-level and cluster-scoped objects actually live.
• Syncer — the vcluster component that copies low-level objects (pods, services, secrets) from the virtual control plane down into the host namespace (rewriting names) and schedules them on host nodes.
• Host cluster / host namespace — the real cluster a vcluster runs inside, and the single namespace on it that holds the vcluster’s synced pods and its control-plane pods.
• CRD (CustomResourceDefinition) — a cluster-scoped, global object that adds a new API type. Because it is global, competing tenant CRDs collide — a key reason to reach for vcluster.
• RuntimeClass — a Kubernetes object selecting the container runtime for a pod (runc, gVisor/runsc, Kata); the opt-in that routes untrusted pods to a sandbox.
• Sandboxed runtime — a runtime that puts a barrier between container and host kernel: gVisor (user-space kernel) or Kata (per-pod microVM). The kernel-isolation layer soft tenancy lacks.
• Capsule / kiosk — soft-tenancy orchestrators that model a Tenant as one object fanning out into namespaces + quota + RBAC + NetworkPolicy; fleet management without a second API server.
• Noisy neighbour — a tenant whose resource or API-server load degrades others on the shared cluster; countered with quota, PriorityClass, and APF.
• Blast radius — how much breaks when one tenant fails or is compromised; the thing each isolation tier shrinks, at rising cost.

Next steps: wire noisy-neighbor detection (per-namespace CPU-throttling and API-request-rate dashboards) and cost showback (label every tenant object with a tenant key and bill from kube-state-metrics usage) into the same pipeline that provisions tenants, so capacity governance and chargeback are byproducts of onboarding rather than a separate quarterly scramble.