Containerization Lesson 51 of 113

Kubernetes RBAC & Service Accounts, In Depth (Fundamentals)

Every request that reaches the Kubernetes API server — a kubectl get pods, a controller reconciling a Deployment, a pod calling the API from inside the cluster — is checked twice before anything happens: who are you? (authentication) and are you allowed to do this? (authorisation). RBAC — Role-Based Access Control — is how Kubernetes answers the second question. It is the layer that decides whether a request becomes an action or a Forbidden.

This lesson takes RBAC apart field by field. By the end you will be able to read any Role, ClusterRole, RoleBinding or ClusterRoleBinding and say exactly what it grants and to whom; you will understand ServiceAccounts and the short-lived tokens that pods now use; and you will be able to prove any permission with a single command. This is the foundational companion to the advanced Least-Privilege RBAC design lesson — here we build the mental model and cover every primitive; there we cover designing, aggregating and auditing RBAC at scale.

In a nutshell

Level: Beginner-friendly → Intermediate · Time: ~40 min (skim the tables first, then run the lab)

Picture your cluster as a large office building. Authentication is the security desk checking your ID at the door — it proves who you are. RBAC (Role-Based Access Control) is the set of rules that then decides which rooms your badge opens — who can do what. RBAC never checks your ID; it only reads the identity authentication already established and looks up what that identity is allowed to touch.

There are two kinds of “who” in this building. A User is a human (you, at your laptop, running kubectl). A ServiceAccount is an identity for a program — a pod, a controller, a CI job — something running inside the cluster that needs to call the Kubernetes API. A ServiceAccount is not a person and has no password: it is a robot badge that Kubernetes hands to a workload so the workload can prove who it is. Every pod runs as exactly one ServiceAccount, the way every employee carries exactly one badge.

The rest of RBAC is just four objects that spell out the badge rules. A Role (or ClusterRole) is a list of what may be done — “may read pods, may not touch secrets.” A RoleBinding (or ClusterRoleBinding) is what hands that list to a specific badge. Nothing is granted until a binding connects the two — and there is no “deny”: RBAC only ever adds permissions, so you tighten access by handing out smaller lists, never by writing a rule that says no.

If you remember one sentence: RBAC decides who (a subject) can do what (verbs) to which things (resources), and a ServiceAccount is the badge a pod wears so it can be a “who.”

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

You need a working kubectl and a cluster you can experiment in — a free local one is perfect (kind, minikube, or k3d). It helps to have met the core objects from earlier in the course — Pods, Deployments & Services — and to be comfortable with kubectl itself (get, describe, apply). You should also understand Namespaces, because the namespaced-vs-cluster-wide distinction is the heart of RBAC. This is the Security lesson of the Kubernetes Zero-to-Hero course; everything runs on free, local tooling with no cloud account required. Targets the current API (Kubernetes v1.30+), where RBAC has been stable (rbac.authorization.k8s.io/v1) for years.

Where RBAC sits: the request pipeline

When any client talks to the API server, the request passes through three gates in order. RBAC is only the middle one — getting this picture right prevents most confusion.

Stage Question What does it Failure shows as
Authentication Who are you? Validates a client certificate, bearer token, or OIDC token and produces a username + groups (or a ServiceAccount identity). Does not consult RBAC. 401 Unauthorized
Authorisation Are you allowed? One or more authorisers decide. RBAC is the usual one; others are Node, ABAC, and Webhook. The request is allowed if any authoriser says yes. 403 Forbidden
Admission control Is the object acceptable / should it be changed? Mutating and validating admission webhooks (and built-in controllers like ResourceQuota, Pod Security Admission) run after authorisation. admission webhook denied the request

Three consequences worth internalising:

  1. RBAC never authenticates. It receives an already-established identity (the username/groups string) and matches it. It never checks that a user “exists” — Users and Groups are not Kubernetes objects (more in Subjects below).
  2. Authorisation is a union of authorisers. On a managed cluster the Node authorizer grants kubelets exactly what they need for their own node; RBAC handles everyone else. A “yes” from any enabled authoriser wins, so RBAC can only grant, never deny over the top of another authoriser.
  3. RBAC is purely additive. Within RBAC there are no deny rules. A subject’s effective permission is the union of every binding that matches them. You reduce access by removing or narrowing bindings — never by adding a denial.

If you ever see Forbidden with a clear message naming the missing verb/resource, that is the authorisation stage (RBAC). If you see admission webhook ... denied, authorisation already said yes and a later policy said no. They are different problems with different fixes.

The four RBAC objects: scope is everything

RBAC has exactly four object kinds, all in the API group rbac.authorization.k8s.io/v1. They split along two axes: permissions (Role/ClusterRole — what may be done) and bindings (RoleBinding/ClusterRoleBinding — who gets it). The dimension people trip over is scope, not function.

Object Scope Holds Grants permissions in
Role namespaced a set of rules its own namespace only
ClusterRole cluster-wide (not namespaced) a set of rules depends on how it’s bound; can also grant cluster-scoped resources & non-resource URLs
RoleBinding namespaced links subjects → a Role or a ClusterRole one namespace (the binding’s own)
ClusterRoleBinding cluster-wide links subjects → a ClusterRole every namespace + cluster-scoped resources

A Role and a ClusterRole are inert on their own — they are just permission templates. Nothing happens until a binding ties a Role/ClusterRole to one or more subjects. A binding has two halves: a roleRef (which permission set) and a subjects list (who receives it).

Why ClusterRole exists when Role seems enough

A ClusterRole is needed for three things a namespaced Role cannot express:

The combination that surprises everyone

A RoleBinding can reference either a Role or a ClusterRole. When a RoleBinding references a ClusterRole, the subject gets those permissions only in the binding’s namespace — the cluster-wide reach of the ClusterRole is clipped to that one namespace.

roleRef ↓ / binding → RoleBinding (in ns team-a) ClusterRoleBinding
Role (in ns team-a) permissions in team-a invalid — a ClusterRoleBinding cannot reference a namespaced Role
ClusterRole permissions in team-a only (clipped) permissions in all namespaces + cluster-scoped resources

This table is the most exam-tested fact in RBAC. The workhorse is the bottom-left cell: one ClusterRole named app-developer, bound by a RoleBinding in each tenant namespace — reusable, yet scoped.

# One reusable permission set...
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: app-developer
rules:
  - apiGroups: ["", "apps"]
    resources: ["pods", "deployments", "services", "configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
# ...scoped to exactly one namespace by a RoleBinding referencing it
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-a-developers
  namespace: team-a            # grant applies ONLY here
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole            # reference the cluster role...
  name: app-developer
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: "eng-team-a"         # ...for this group, in this namespace only

roleRef is immutable. Once a binding is created you cannot change which Role/ClusterRole it points at — you must delete and recreate it. The subjects list, by contrast, can be edited freely.

The anatomy of a rule, field by field

The permission lives inside the rules list of a Role or ClusterRole. A rule is, conceptually, apiGroups × resources × verbs, optionally narrowed by resourceNames, with subresources addressed via resources. This is where exhaustiveness matters, so here is every field.

Field What it does Values Default When to set Gotcha
apiGroups Which API group(s) the resources belong to. [""] = the core group (pods, services, configmaps, secrets, nodes…); ["apps"], ["batch"], ["networking.k8s.io"], ["rbac.authorization.k8s.io"], etc. ["*"] = all groups. none (required for resource rules) Always, for resource rules. The core group is the empty string "", not "core" and not omitting the field. Forgetting this is the #1 beginner error.
resources Which resource type(s), by their plural, lowercase API name. ["pods"], ["deployments"], ["secrets"], ["pods/log"], ["deployments/scale"], ["*"]. none (required, unless nonResourceURLs is used) Always, for resource rules. Use the plural name as it appears in the API (pods, not Pod). Subresources are written resource/subresource and are separate grants.
verbs Which actions are permitted. See the verb table below; ["*"] = all verbs. none (required) Always. watch is separate from list; many tools need both. deletecollection is separate from delete.
resourceNames Restrict the rule to named instances of the resource. e.g. ["my-config", "tls-cert"]. empty = all instances When you want “read this secret” not “read all secrets”. Does not work with create, list, watch, or deletecollection (the name isn’t known at request time, or the verb is collection-wide). It works with get, update, patch, delete.
nonResourceURLs Grant access to non-resource endpoints (not REST objects). ["/healthz", "/metrics", "/version", "/api/*"]. none Monitoring/health probes against the API server. ClusterRole only; cannot be combined with resources in the same rule; pairs with non-resource verbs like get.

The verbs, exhaustively

Verbs map to HTTP methods against the API. Memorise these — they are the vocabulary of every rule.

Verb What it allows Maps to
get Read a single named object. GET /…/name
list Read a collection (all objects of a type in scope). GET /… (collection)
watch Stream changes to a collection (open a watch). GET …?watch=true
create Create a new object. POST
update Replace an entire object. PUT
patch Partially modify an object (incl. kubectl rollout restart, label edits). PATCH
delete Delete a single named object. DELETE /…/name
deletecollection Delete all objects of a type in scope at once. DELETE (collection)

Plus special, non-CRUD verbs that are real escalation paths — covered in Security notes:

Special verb On resource Effect
bind roles, clusterroles Bind that (possibly higher-privilege) role to a subject.
escalate roles, clusterroles Create/update a role with more rights than you currently hold (bypasses the escalation check).
impersonate users, groups, serviceaccounts Act as another subject — effectively become them.
use podsecuritypolicies (legacy) / certain admission policy resources Permission to use a policy object.
approve / sign certificatesigningrequests (subresources) Approve or sign CSRs.

A crucial subtlety: get does not imply list. They are independent verbs against different endpoints. A role with only get lets you read a pod if you already know its name, but kubectl get pods (which lists) will return Forbidden. Grant ["get","list","watch"] together for normal read access. Likewise, list returns full objects — there is no “list names only”, so anyone with list secrets can read secret values.

Worked rule: every field in play

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payments-operator
  namespace: team-payments
rules:
  # Full lifecycle on deployments in this namespace
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  # Scale subresource is a SEPARATE grant
  - apiGroups: ["apps"]
    resources: ["deployments/scale"]
    verbs: ["update", "patch"]
  # Read pod logs (subresource), but not exec
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
  # Read EXACTLY one configmap by name, nothing else
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["payments-feature-flags"]
    verbs: ["get", "update", "patch"]

Read it as four independent grants OR’d together. Note that pods/log (read logs) and pods/exec (shell in) are different subresources — granting one never grants the other.

Subjects: who a binding grants to

The subjects list of a binding holds one or more of three kinds. Each is authenticated differently before RBAC ever sees it.

kind What it is How it’s authenticated apiGroup value Is it a real object?
User A human (or external automation), identified by a username string. Client certificate CN, OIDC token, or auth proxy header. rbac.authorization.k8s.io No — just a string the authenticator asserts.
Group A set of users, identified by a group-name string. Cert O field, OIDC groups claim, or built-in groups. rbac.authorization.k8s.io No — also just a string.
ServiceAccount A namespaced identity for workloads (pods, controllers). A bearer token (bound/projected) mounted into the pod. "" (the core group) — and you give name + namespace. Yes — a v1 ServiceAccount object.

Two facts that catch people out:

The full SA username form, which you use with impersonation and in some bindings, is:

system:serviceaccount:<namespace>:<name>

And every SA automatically belongs to two groups: system:serviceaccounts (all SAs) and system:serviceaccounts:<namespace> (all SAs in that namespace).

Built-in users and groups you’ll meet

Identity Kind Meaning
system:masters Group Super-group — bound to cluster-admin by default; cluster super-user. Anyone with a cert in O=system:masters is god.
system:authenticated Group Every successfully authenticated request.
system:unauthenticated Group Anonymous requests (if anonymous auth is on).
system:serviceaccounts Group All ServiceAccounts cluster-wide.
system:serviceaccounts:<ns> Group All ServiceAccounts in one namespace.
system:node:<name> User A kubelet’s identity (handled mostly by the Node authorizer).

Default ClusterRoles: don’t reinvent these

Kubernetes ships default (built-in) ClusterRoles so you rarely write read/write roles from scratch. The four “user-facing” ones are the ones to know.

ClusterRole Grants Typical use
view read-only (get/list/watch) on most resources except Secrets/roles/bindings. Auditors, dashboards, junior read access.
edit read/write on most resources (create/update/delete), but not roles/bindings and not quota/limits. Can read Secrets (historically) and exec into pods. Developers in their own namespace (bind with a RoleBinding).
admin everything edit does plus managing Roles/RoleBindings within a namespace (but not the namespace object or resource quota). Namespace owner / team lead.
cluster-admin everything, everywhere — all verbs on all resources in all namespaces + non-resource URLs. Break-glass only. Bound to system:masters by default.

These built-ins are aggregated (they absorb labelled add-on ClusterRoles automatically) — a powerful but double-edged feature covered in the advanced RBAC lesson. For most teams the pattern is: bind view/edit/admin with a RoleBinding per namespace, and reserve cluster-admin for emergencies.

Never bind cluster-admin with a ClusterRoleBinding to a person or a workload SA “to make the error go away”. The whole point of RBAC is least privilege; cluster-admin discards it.

ServiceAccounts: identity for workloads

A User is for humans; a ServiceAccount (SA) is for workloads. Every pod runs as exactly one SA, and that SA is how the pod authenticates to the API server (and how RBAC decides what the pod may do).

apiVersion: v1
kind: ServiceAccount
metadata:
  name: order-processor
  namespace: team-payments
automountServiceAccountToken: false   # see "automounting" below

A pod selects its SA with spec.serviceAccountName. If you omit it, the pod uses the namespace’s default SA.

apiVersion: apps/v1
kind: Deployment
metadata: { name: order-processor, namespace: team-payments }
spec:
  template:
    spec:
      serviceAccountName: order-processor   # else "default" is used
      containers:
        - name: app
          image: ghcr.io/acme/orders:1.4.0

The default ServiceAccount

Every namespace gets a default SA automatically (created by a controller). Two important points:

Tokens: bound/projected vs the legacy Secret token

How a pod actually authenticates is the part that changed most in modern Kubernetes. There are three eras; know all three because you’ll meet old manifests.

Token type How obtained Lifetime Audience-scoped? Auto-rotated? Status
Bound / projected token (TokenRequest API) The kubelet requests a token for the pod’s SA and projects it into the pod via a projected volume. Short (default ~1h; kubelet refreshes before expiry) Yes (default audience = API server; can request others) Yes Default since v1.22+; the right way.
Legacy auto-created Secret token Pre-1.24, the SA controller auto-created a kubernetes.io/service-account-token Secret and mounted it. Never expires (static) No No Removed as default in v1.24+. SAs no longer auto-get a Secret.
Manually created Secret token You create a kubernetes.io/service-account-token Secret with kubernetes.io/service-account.name annotation. Never expires (static) No No Still possible, but discouraged — treat as a static credential.

Inside a pod, the projected token (and CA cert + namespace) lands at:

/var/run/secrets/kubernetes.io/serviceaccount/token
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/namespace

To mint a token from the command line (for testing or short-lived automation), use the TokenRequest API directly:

# Short-lived token for an SA (audience = API server by default)
kubectl create token order-processor -n team-payments

# With a custom expiry and audience
kubectl create token order-processor -n team-payments \
  --duration=30m --audience=https://kubernetes.default.svc

kubectl create token is the modern replacement for the old trick of reading a long-lived token out of a Secret. Prefer it everywhere. If you truly need a non-expiring token (rare — e.g. an external CI system that can’t refresh), you must create the Secret explicitly; do so knowing you’ve created a static credential to guard and rotate.

Automounting the token

By default, the SA token is mounted into every pod — handy if the workload calls the API, but a needless credential for one that doesn’t (a web frontend that only serves HTTP has no reason to hold a cluster token). Anyone who lands code execution in that container gets a ready-made credential.

Control it at two levels (pod-level wins if both are set):

# On the ServiceAccount: default for all pods using this SA
apiVersion: v1
kind: ServiceAccount
metadata: { name: web-frontend, namespace: shop }
automountServiceAccountToken: false
---
# On the Pod: overrides the SA setting for this pod
apiVersion: apps/v1
kind: Deployment
metadata: { name: web-frontend, namespace: shop }
spec:
  template:
    spec:
      serviceAccountName: web-frontend
      automountServiceAccountToken: false   # belt-and-braces

Rule of thumb: default to false, and switch it on only for workloads that genuinely talk to the API.

imagePullSecrets on a ServiceAccount

A ServiceAccount can also carry image pull credentials so pods using it can pull from a private registry without each pod specifying the secret. First create a docker-registry secret, then attach it to the SA:

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=acme-bot \
  --docker-password='<token>' \
  -n team-payments
apiVersion: v1
kind: ServiceAccount
metadata:
  name: order-processor
  namespace: team-payments
imagePullSecrets:
  - name: regcred        # every pod using this SA inherits this pull credential

Now any pod with serviceAccountName: order-processor can pull private images without its own imagePullSecrets. (Note imagePullSecrets is a list of {name} objects; it references docker-registry secrets, not RBAC.)

The RBAC model at a glance

Kubernetes RBAC model

The diagram traces a request from a subject (a User, Group, or a pod’s ServiceAccount) through a binding (RoleBinding for one namespace, or ClusterRoleBinding cluster-wide) to a permission set (Role inside a namespace, or ClusterRole spanning the cluster), where the matching ruleapiGroups × resources × verbs — is what finally allows or forbids the action at the API server. Read it left to right: whovia which bindinggets which rulesover which resources.

Common beginner mistakes

These are conceptual traps — the mental-model errors that lead beginners astray — as opposed to the symptom→cause→fix table further down. Each item is the mistaken belief that causes a whole class of errors.

“I’ll just let pods use the default ServiceAccount.” Why it bites: the default SA is shared by every pod in the namespace that doesn’t name another. The moment anyone binds a permission to default (often to make a Forbidden go away quickly), all of those pods silently inherit it — a blast radius you never see until something is over-privileged. Right model: one ServiceAccount per workload, named explicitly in the Deployment, and never bind anything to default. Treat default as “no identity,” not “the normal identity.”

“Mounting the token everywhere is harmless — most pods ignore it.” Why it bites: a mounted token is a live credential sitting in the container’s filesystem at a well-known path. A frontend that only serves HTTP has no reason to hold a cluster credential, but if it does and an attacker gets code execution (an SSRF, an RCE, a leaked log), they inherit a ready-made key to the API with whatever that SA was granted. Right model: automountServiceAccountToken: false by default, switched on only for the workloads that actually call the API.

“If a pod has a ServiceAccount token, it can access the API.” Why it bites: a token proves identity (authentication), not permission (authorisation). The default SA’s token is perfectly valid and authenticates fine — and can do essentially nothing, because no RBAC binding grants it any verbs. Conversely, an SA is powerless until a binding points at it. Right model: token = identity; binding = power. You need both to act.

“Users and ServiceAccounts are basically the same kind of account.” Why it bites: they are authenticated in completely different ways, and one of them isn’t even a Kubernetes object. A User is just a string an external authenticator (a client certificate, your company OIDC/SSO) asserts — there is no User object and no kubectl create user. A ServiceAccount is a real, namespaced v1 object whose token Kubernetes itself issues. Right model: Users come from outside and are for humans; ServiceAccounts live inside the cluster and are for workloads. Bind humans via their IdP groups; bind workloads via their SA.

get on a resource lets me kubectl get (list) it.” Why it bites: get and list are separate verbs on separate endpoints, and kubectl get pods performs a list. Grant ["get","list","watch"] together for ordinary read access. (This one straddles concept and symptom — it appears in the troubleshooting table too, because it is that common.)

“I’ll grant cluster-admin for now and tighten it later.” Why it bites: “later” rarely comes, and a ClusterRoleBinding to cluster-admin discards the whole point of RBAC — least privilege — in a single line. Right model: when something is Forbidden, add the one missing verb/resource to a scoped Role, proven with kubectl auth can-i. Never widen to cluster-admin to make an error disappear.

Hands-on lab: build and test RBAC end to end

Everything here runs on a free local cluster and is fully reversible. We will create a namespace, a ServiceAccount, a least-privilege Role, bind it, mint a token, and prove the boundaries with kubectl auth can-i.

0. Start a cluster

kind create cluster --name rbac-lab        # or: minikube start  / k3d cluster create
kubectl cluster-info

1. Namespace and ServiceAccount

kubectl create namespace team-payments
kubectl create serviceaccount ci-deployer -n team-payments

2. A least-privilege Role and a RoleBinding

Save as rbac-lab.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: team-payments
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
  # Deliberately NO secrets, NO pods/exec
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-binding
  namespace: team-payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: deployer
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: team-payments
kubectl apply -f rbac-lab.yaml

Expected:

role.rbac.authorization.k8s.io/deployer created
rolebinding.rbac.authorization.k8s.io/ci-deployer-binding created

3. Prove the permissions with auth can-i (impersonation)

kubectl auth can-i asks the real authoriser, so it is ground truth. Impersonate the SA with --as:

# Should be ALLOWED
kubectl auth can-i create deployments -n team-payments \
  --as=system:serviceaccount:team-payments:ci-deployer        # -> yes

kubectl auth can-i get pods -n team-payments \
  --as=system:serviceaccount:team-payments:ci-deployer        # -> yes

# Should be FORBIDDEN (we never granted these)
kubectl auth can-i get secrets -n team-payments \
  --as=system:serviceaccount:team-payments:ci-deployer        # -> no

kubectl auth can-i create pods/exec -n team-payments \
  --as=system:serviceaccount:team-payments:ci-deployer        # -> no

# Wrong namespace -> the Role doesn't reach there
kubectl auth can-i create deployments -n default \
  --as=system:serviceaccount:team-payments:ci-deployer        # -> no

List everything the SA can do in the namespace:

kubectl auth can-i --list -n team-payments \
  --as=system:serviceaccount:team-payments:ci-deployer

Impersonation itself is a privileged action. As the cluster-admin in your kubeconfig you can --as anyone; a normal user needs the impersonate verb. That’s why auth can-i --as works for you here without extra setup.

4. Mint a real token and call the API as the SA

TOKEN=$(kubectl create token ci-deployer -n team-payments --duration=10m)

# Use the token directly against the API server
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
curl -sk -H "Authorization: Bearer $TOKEN" \
  "$APISERVER/apis/apps/v1/namespaces/team-payments/deployments" | head

# Same token trying secrets -> 403 Forbidden body
curl -sk -H "Authorization: Bearer $TOKEN" \
  "$APISERVER/api/v1/namespaces/team-payments/secrets"

The first call returns a deployment list (or an empty list); the second returns a 403 Forbidden status object — RBAC enforcing exactly what you bound.

5. Validation

kubectl describe rolebinding ci-deployer-binding -n team-payments
kubectl get role deployer -n team-payments -o yaml

You should see the binding’s roleRefdeployer and the single SA subject.

6. Cleanup

kubectl delete -f rbac-lab.yaml
kubectl delete serviceaccount ci-deployer -n team-payments
kubectl delete namespace team-payments
kind delete cluster --name rbac-lab        # or: minikube delete / k3d cluster delete

Cost note: zero — everything is local. No cloud account, no charges.

Common mistakes & troubleshooting

Symptom Likely cause Fix
Forbidden even though “the Role looks right” The core API group was written as "core" or omitted. Use apiGroups: [""] (empty string) for pods/services/secrets/configmaps/nodes.
kubectl get pods is Forbidden but get a named pod works Granted get but not list. Add list (and watch); these are independent verbs.
Role + RoleBinding created, subject still Forbidden Role/RoleBinding live in the wrong namespace, or the SA’s namespace in the subject is wrong. Ensure the RoleBinding is in the namespace where access is needed; verify subjects[].namespace.
ClusterRoleBinding to a Role rejected A ClusterRoleBinding can only reference a ClusterRole. Reference a ClusterRole, or switch to a RoleBinding.
Changed which role a binding points at, got an error roleRef is immutable. Delete and recreate the binding.
Pod can read Secrets you never granted Pod uses the default SA which someone bound, or you granted broad list that includes secrets. Give the pod its own SA; exclude secrets from broad read roles; never bind default.
resourceNames rule ignored for kubectl get <type> resourceNames does not apply to list/watch/create. Use get with the explicit name, or accept that listing returns all.
Cannot reach nodes / persistentvolumes These are cluster-scoped; a namespaced Role can’t grant them. Use a ClusterRole + ClusterRoleBinding.
Old manifest reads a token from a Secret that no longer exists Auto-created SA token Secrets were removed in v1.24+. Use kubectl create token (TokenRequest) or projected tokens.

The fastest debugging loop is always: reproduce the exact decision with kubectl auth can-i <verb> <resource> -n <ns> --as=<subject>, then add the specific missing verb/resource to a scoped Role — never widen to cluster-admin.

Best practices

Security notes

RBAC is a primary security boundary, so a few risks deserve emphasis even at the fundamentals level:

These threads are developed fully in Least-Privilege RBAC design (aggregation footguns, escalation-path hunting, continuous auditing) and in Pod Security Admission.

Going deeper

The fundamentals above are enough to pass an exam and write correct RBAC. This section is for the reader who wants to know how the machinery actually works — what a token is, how the API server trusts it, and how that in-cluster identity becomes a cloud credential. None of it needs a live cluster to understand, and it is exactly the material that separates “I can copy a Role” from “I can reason about an escalation.”

What a bound token really is, and how the API server trusts it

A modern ServiceAccount token is a signed JWT (JSON Web Token). When the kubelet projects it into a pod, the file at /var/run/secrets/kubernetes.io/serviceaccount/token is three base64url segments — header, claims, signature — joined by dots. Decode the middle segment and you see roughly:

{
  "aud": ["https://kubernetes.default.svc"],
  "exp": 1770000000,
  "iat": 1769996400,
  "iss": "https://kubernetes.default.svc",
  "sub": "system:serviceaccount:team-payments:order-processor",
  "kubernetes.io": {
    "namespace": "team-payments",
    "serviceaccount": { "name": "order-processor", "uid": "d4e5..." },
    "pod": { "name": "order-processor-7d9f8", "uid": "a1b2..." }
  }
}

(representative decoded claims)

When the pod sends Authorization: Bearer <jwt>, the API server’s ServiceAccount token authenticator does four things: verifies the signature against the SA signing keys (published at /openid/v1/jwks), checks exp (not expired) and aud (the audience is one this API server accepts), and — because this is a bound token — confirms the referenced pod and ServiceAccount still exist. If the pod named under kubernetes.io.pod has been deleted, the token is dead even before it expires. That “bound to the pod’s lifetime” property is precisely why these are called bound tokens, and it is something a static Secret token can never do.

Only after all that does the authenticator hand RBAC an identity: username system:serviceaccount:team-payments:order-processor, plus the groups system:serviceaccounts, system:serviceaccounts:team-payments, and system:authenticated. RBAC then matches that against bindings. Authentication proved the badge is real; RBAC decides what it opens.

The three token eras, precisely

The fundamentals table listed bound, legacy-auto, and manual Secret tokens. Here is what actually changed across versions, because you will meet all three in real clusters:

Audience, expiry, and requesting your own projected token

When a workload needs a token for something other than the API server — say, to prove its identity to Vault or a cloud IAM — you request a second projected token with a different audience. You can see the exact shape in a pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: token-demo
  namespace: team-payments
spec:
  serviceAccountName: order-processor
  containers:
    - name: app
      image: ghcr.io/acme/orders:1.4.0
      volumeMounts:
        - name: vault-token
          mountPath: /var/run/secrets/vault
          readOnly: true
  volumes:
    - name: vault-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: vault           # NOT the API server
              expirationSeconds: 600     # floor is 600 (10 minutes)

The kubelet mints and rotates that token independently of the default one. expirationSeconds has a floor of 600 (ten minutes); the admission-injected default token expires after about an hour. From the CLI the same TokenRequest is kubectl create token order-processor -n team-payments --audience=vault --duration=10m — handy for inspecting exactly what a downstream verifier will receive.

From cluster identity to cloud IAM: IRSA, Workload Identity, and friends

This is where ServiceAccounts become genuinely powerful. Each managed Kubernetes exposes the cluster’s SA-token OIDC issuer (discoverable at /.well-known/openid-configuration and /openid/v1/jwks) so a cloud IAM system will trust tokens the cluster signs. The pattern — workload-identity federation — lets a pod obtain cloud credentials with no long-lived cloud keys stored anywhere:

The through-line: the humble in-cluster projected token — a JWT the cluster signs — is exchanged for cloud credentials by each provider’s federation. Same primitive you already learned, one new audience.

imagePullSecrets, the admission plugin, and automount precedence

Three quieter mechanics run automatically and are worth making explicit:

Escalation paths that run through ServiceAccount tokens

Finally, the reason all of the above is a security topic. An attacker who lands code execution in a pod inherits that pod’s SA. From there the classic escalation chains are:

  1. Over-mounted token → API access. The pod never needed the token, but it was mounted; now the attacker has the SA’s granted verbs for free. (Fix: automount off.)
  2. get/list secrets → other identities. A “read-only” role that includes secrets can read any manually-created SA-token Secret in the namespace and then become that SA. (Fix: exclude secrets from broad reads; grant by resourceNames.)
  3. create pods + a privileged SA in the namespace → token theft. If a subject can create pods, they can schedule a pod that runs as a more-privileged SA and mounts its token — an indirect escalation even without direct access to that SA. (Fix: scope pod-creation in shared namespaces; keep privileged SAs out of them.)
  4. escalate / bind → self-granted power. These verbs bypass the built-in check that stops you granting more than you hold. Treat create/update on roles, clusterroles, and their bindings as equivalent to the permissions they can mint.
  5. impersonate → become anyone. The impersonate verb on users/groups/serviceaccounts lets a subject act as another — including highly-privileged ones. Audit it like a super-power.

Every one of these is blunted by the same three habits from the fundamentals: automount off, one least-privileged SA per workload, and never bind to default or cluster-admin. The advanced Least-Privilege RBAC design lesson turns these into a repeatable escalation-path hunt.

Interview & exam questions

  1. What is the difference between authentication and authorisation in Kubernetes, and where does RBAC sit? Authentication establishes who you are (cert/OIDC/token → username + groups) and fails with 401. Authorisation decides whether you may act and fails with 403. RBAC is an authoriser in the authorisation stage; admission control runs after, and is a separate concern.

  2. Role vs ClusterRole — when must you use a ClusterRole? Use a ClusterRole for cluster-scoped resources (nodes, PVs, namespaces, the RBAC objects themselves), for non-resource URLs (/healthz, /metrics), or to define a reusable permission set you’ll bind per namespace.

  3. A RoleBinding references a ClusterRole. What does the subject get? Only the permissions of that ClusterRole within the binding’s namespace — the cluster-wide reach is clipped to that one namespace. This is the workhorse multi-tenant pattern.

  4. Can a ClusterRoleBinding reference a Role? No. A ClusterRoleBinding can only reference a ClusterRole. A namespaced Role can only be referenced by a RoleBinding.

  5. What are the components of an RBAC rule? apiGroups × resources × verbs, optionally narrowed by resourceNames, or nonResourceURLs (+ verbs) for non-resource endpoints. The core group is the empty string "".

  6. Does get imply list? No. They are separate verbs on different endpoints. get reads one named object; list reads a collection. kubectl get <type> needs list.

  7. Is RBAC additive, and can you write a deny rule? RBAC is purely additive — effective permission is the union of all matching bindings. There are no deny rules; you reduce access only by removing/narrowing bindings.

  8. What is the default ServiceAccount and what can it do? Every namespace has a default SA, used by any pod that doesn’t name another. It has no RBAC permissions by default. Best practice: give workloads their own SA and never bind to default.

  9. How do modern pods authenticate to the API server, and how did this change in v1.24? Via short-lived, audience-scoped bound/projected tokens from the TokenRequest API, auto-rotated by the kubelet. Before v1.24, the SA controller auto-created a non-expiring service-account-token Secret; that auto-creation was removed in v1.24.

  10. How do you mint a token for a ServiceAccount from the CLI, and why prefer it? kubectl create token <sa> -n <ns> [--duration --audience]. It returns a short-lived bound token instead of a static secret, so there’s no long-lived credential to leak.

  11. What does automountServiceAccountToken: false do and when should you set it? It stops the SA token from being mounted into the pod. Default it to false; enable it only for workloads that actually call the API, reducing the credential blast radius if a container is compromised.

  12. Name three RBAC verbs/permissions that are privilege-escalation paths. escalate and bind on roles/clusterroles (grant yourself more than you hold / bind a high-priv role), and impersonate on users/groups/serviceaccounts (become another subject). list/get on secrets is an underrated one too.

Quick check

  1. Which API group string covers pods, services, configmaps and secrets?
  2. You created a Role + RoleBinding in default, but your workload runs in team-a and gets Forbidden. Why?
  3. True or false: a RoleBinding can reference a ClusterRole.
  4. Where does a pod’s projected ServiceAccount token appear inside the container?
  5. Which command proves, as ground truth, whether a specific subject may delete secrets in team-a?

Answers

  1. The core group — the empty string "".
  2. A Role/RoleBinding only grant in their own namespace. They must live in team-a (or be a ClusterRole bound via a RoleBinding in team-a).
  3. True. The subject then gets that ClusterRole’s permissions only in the binding’s namespace.
  4. /var/run/secrets/kubernetes.io/serviceaccount/token (alongside ca.crt and namespace).
  5. kubectl auth can-i delete secrets -n team-a --as=<subject> (e.g. --as=system:serviceaccount:team-a:ci).

Practice challenges

Work these in a throwaway local cluster (kind create cluster). They escalate from reading RBAC to reasoning about escalation. Each solution is one command or manifest plus a one-line why — try it before you open the toggle.

Challenge 1 — Beginner · Read a rule. Given this rule, can the subject run kubectl get pods -n web? Can it read a pod’s logs?

- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch"]

<details> <summary>Solution</summary>

No to kubectl get pods — that is a list, and only get/watch are granted — and no to logs, because pods/log is a separate subresource that was not granted. The rule only lets you get/watch a pod by name. Why: list and each subresource are independent grants. </details>

Challenge 2 — Beginner · Give a workload its own identity. Create a ServiceAccount reporter in namespace analytics, and a Deployment that runs as it with the token not mounted.

<details> <summary>Solution</summary>

kubectl create namespace analytics
kubectl create serviceaccount reporter -n analytics
spec:
  template:
    spec:
      serviceAccountName: reporter
      automountServiceAccountToken: false
      containers: [{ name: app, image: ghcr.io/acme/reporter:1.0 }]

Why: an explicit SA plus automount off is the safe default for a workload that does not call the API. </details>

Challenge 3 — Intermediate · Reusable role, scoped per namespace. You need identical read/write on deployments for teams blue and green, authored once. Which objects — and why not just write two Roles?

<details> <summary>Solution</summary>

One ClusterRole (the rule set) plus one RoleBinding per namespace referencing it:

kubectl create clusterrole deploy-rw \
  --verb=get,list,watch,create,update,patch,delete --resource=deployments
kubectl create rolebinding deploy-rw -n blue  --clusterrole=deploy-rw --serviceaccount=blue:ci
kubectl create rolebinding deploy-rw -n green --clusterrole=deploy-rw --serviceaccount=green:ci

Why: a RoleBinding → ClusterRole clips the cluster-wide role to just its own namespace — author once, grant per tenant. </details>

Challenge 4 — Intermediate · Prove a boundary as ground truth. Without applying anything new, show that analytics:reporter cannot read secrets in analytics.

<details> <summary>Solution</summary>

kubectl auth can-i get secrets -n analytics \
  --as=system:serviceaccount:analytics:reporter   # -> no

Why: auth can-i --as asks the real authoriser, so it is ground truth — assert the cannot, not only the can. </details>

Challenge 5 — Advanced · Find the escalation. Namespace sandbox has a ServiceAccount builder bound to a Role with create on pods, and it also contains a privileged SA deployer (bound to the built-in edit). Explain the escalation and the fix.

<details> <summary>Solution</summary>

builder can create a pod with serviceAccountName: deployer, then read that pod’s mounted token (or simply act through it) to gain deployer’s edit rights — indirect privilege escalation via create pods plus a privileged SA sharing the namespace. Fix: do not co-locate privileged SAs with pod-create grants; set automountServiceAccountToken: false on deployer; restrict which SAs a subject may run pods as. </details>

Challenge 6 — Advanced · Federate to the cloud (design). A pod must read an AWS S3 bucket with no static AWS keys in the container. Outline the moving parts.

<details> <summary>Solution</summary>

Register the cluster OIDC issuer as an IAM identity provider; create an IAM role whose trust policy allows sts:AssumeRoleWithWebIdentity from that issuer; annotate the SA with eks.amazonaws.com/role-arn; the injected projected token (audience sts.amazonaws.com) is exchanged by the AWS SDK for temporary credentials. Why: workload-identity federation swaps a cluster-signed JWT for short-lived cloud creds — zero long-lived keys. See EKS IRSA → Pod Identity. </details>

Exercise

In a fresh local cluster, build a read-only auditor and a scoped operator, then prove the boundaries:

  1. Create namespace shop and two ServiceAccounts: auditor and operator.
  2. Bind auditor to the built-in view ClusterRole using a RoleBinding scoped to shop (not a ClusterRoleBinding).
  3. Write a custom Role deploy-manager granting full lifecycle on deployments and deployments/scale, plus read on pods/pods/log, and bind it to operator.
  4. Using kubectl auth can-i --as, assert: auditor can list pods but cannot create deployments and cannot get secrets; operator can patch deployments and update deployments/scale but cannot get secrets or create pods/exec.
  5. Mint a 5-minute token for operator with kubectl create token and confirm it can list deployments but is 403 on secrets via curl.
  6. Set automountServiceAccountToken: false on the auditor SA and verify a pod using it has no token file mounted.
  7. Clean everything up.

Bonus: try to bind operator to a Role via a ClusterRoleBinding and observe the rejection — then explain why.

Certification mapping

Exam Where this maps
CKA “Security → RBAC” objective — create Roles/ClusterRoles and bindings, use auth can-i. Core, frequently tested.
CKAD Application security — ServiceAccounts for pods, serviceAccountName, token mounting, imagePullSecrets.
CKS Cluster setup & hardening — least-privilege RBAC, minimising SA tokens, restricting secrets/exec, escalation paths (deepened in the advanced lesson).
KCNA Cloud-native security fundamentals — recognising the RBAC model and the four objects.

Glossary

Next steps

KubernetesRBACServiceAccountsSecurityAuthorisationCKA
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments