Containerization Lesson 14 of 113

Kubernetes Labels, Selectors, Annotations & Field Selectors, In Depth

In a nutshell

Picture a giant warehouse of boxes. On every box you can stick small coloured sticky notes — “fragile”, “kitchen”, “ship-Monday”. Later you never hunt for a box by its serial number; you just say “bring me everything tagged kitchen.” That is almost exactly how Kubernetes finds things. The sticky notes are labels, and a request like “everything tagged X” is a selector. A Service does not know its Pods by name — it says “send traffic to everything wearing the app=web sticky note,” and whatever carries that note right now gets the traffic. Scale from 3 Pods to 30 and the Service never changes: the new Pods simply show up wearing the same note.

Annotations are a different thing that lives right next to labels and looks almost identical. They are the little notes field on the box — “packed by Sam, checked 3 May, handle per SOP-42.” Nobody sorts the warehouse by that field; it is there for people and tools to read once they already have the box in hand. So the single rule to carry through this whole lesson is: if something needs to find the object, use a label; if something just needs to read a note about it, use an annotation.

Field selectors are a fourth idea that sounds like the others but is unrelated to your sticky notes. They filter on facts the system has already stamped on the box itself — “is it Running?”, “which shelf (node) is it on?” You will meet them last, on purpose, so you keep them mentally separate.

Don’t worry about syntax yet; the rest of the lesson builds each idea up slowly with the exact kubectl you would type, then goes deep enough (how selectors are indexed and evaluated, why some are immutable, what the platform labels for you) that an experienced engineer will still learn something. If you remember only one sentence, make it the labels-vs-annotations rule above.

Level: Beginner → Advanced · Time: ~45 min · You’ll need: a rough idea of Pods/Deployments/Services and a free local cluster (kind or minikube). No cloud account, no cost.

Almost nothing in Kubernetes is wired together by name. A Service does not know the names of the Pods it sends traffic to; a Deployment does not own its Pods by listing their identities; a NetworkPolicy does not enumerate the workloads it protects. Instead, Kubernetes is held together by labels — small key/value tags you stick on objects — and label selectors — queries that say “everything carrying these tags”. This loose, query-based coupling is what lets you scale a Deployment from three Pods to thirty without touching the Service in front of it, roll out a new version by swapping which Pods match a selector, and apply policy to a category of workloads rather than a hand-maintained list. Understanding labels and selectors is therefore not optional trivia: it is the single mental model that explains how Services, controllers, scheduling and policy all find each other.

This lesson takes that model apart completely. We cover the exact syntax rules for labels (what characters are legal, how prefixes work, the standard app.kubernetes.io/* set everyone should adopt), both flavours of label selector (the simple equality-based form and the richer set-based form, plus matchLabels vs matchExpressions as they appear in YAML), and then we walk through every place a selector wires two objects together — Services to Pods, ReplicaSets/Deployments to Pods, nodeSelector and node/pod affinity, NetworkPolicy, and PodDisruptionBudget. We then draw the sharp line between labels and annotations (the other kind of metadata, the non-identifying kind that nothing selects on), and finish with field selectors — a separate, server-side filtering mechanism that queries object fields such as status.phase and spec.nodeName rather than labels. Everything runs on a free local cluster (kind or minikube), and every concept comes with the exact kubectl you would type. By the end you will be able to read any selector in any manifest and say precisely which objects it matches and why.

Learning objectives

By the end of this lesson you will be able to:

Prerequisites & where this fits

You should already know what a Pod, Deployment, and Service are at a basic level (covered in Pods, Deployments & Services) and have kubectl talking to a cluster. Any free local cluster works — kind, minikube, or k3d. This lesson sits early in the Fundamentals module of the Kubernetes Zero-to-Hero course: it is the connective tissue you need before the Services, Deployments, scheduling, and NetworkPolicy lessons, because every one of those objects is defined in terms of selectors. Examples target Kubernetes v1.30+, where set-based selectors and field selectors behave as described.

Core concept: identifying vs non-identifying metadata

Every Kubernetes object carries a metadata block, and inside it live two maps that look almost identical but do completely different jobs:

Labels (metadata.labels) Annotations (metadata.annotations)
Purpose Identifying metadata — used to select and group Non-identifying metadata — arbitrary notes for humans and tools
Queried by selectors? Yes — this is their whole reason to exist No — nothing selects on annotations
Used by controllers to find objects? Yes (Services, ReplicaSets, etc.) No (but tools read specific keys)
Value constraints Short, strict syntax (≤63 chars, limited charset) Free-form, large (total ≤256 KB)
Typical content app=nginx, env=prod, tier=frontend change-cause, last-applied-config, ingress hints, checksums
Indexed for fast lookup? Yes (label index) No

The rule of thumb that almost never fails: if something needs to find this object, use a label; if something just needs to read a note about this object, use an annotation. A version you route traffic to is a label; a git commit SHA you display in a dashboard is an annotation. Both are metadata; only one is queryable.

A third, separate mechanism — field selectors — is easy to confuse with label selectors but is unrelated: field selectors filter on the object’s real fields (like status.phase or spec.nodeName), not on the labels map. We treat them at the end, deliberately, so you keep them mentally distinct.

Labels: syntax rules in full

A label is a key: value pair. Both halves have strict, enforced syntax — the API server rejects anything that violates it, so it is worth learning exactly.

The key has two parts: an optional prefix and a required name, written prefix/name.

The value must be ≤63 characters, and if non-empty must begin and end with an alphanumeric, with dashes/underscores/dots allowed between. A value may be the empty string (""), which is meaningful — an existence check (exists) treats a key with an empty value as present.

Two reserved prefixes carry special meaning and you should not invent labels under them: kubernetes.io/ and k8s.io/ are reserved for core Kubernetes components (for example nodes auto-carry kubernetes.io/hostname, kubernetes.io/arch, kubernetes.io/os, and the well-known topology labels topology.kubernetes.io/zone and topology.kubernetes.io/region).

Here is the rule set as a quick table:

Element Max length Allowed characters Must start/end with
Key name (after /) 63 alphanumeric, -, _, . alphanumeric
Key prefix (before /) 253 (DNS subdomain) lowercase alphanumeric, -, . alphanumeric
Value 63 (empty allowed) alphanumeric, -, _, . alphanumeric (if non-empty)

A few worked examples of invalid labels, because the error messages are common:

The recommended common labels: app.kubernetes.io/*

Kubernetes does not require any particular labels, but it publishes a recommended set under the app.kubernetes.io/ prefix so that tools (dashboards, Helm, kubectl, operators) can understand applications they did not create. Adopt these as a baseline on every workload:

Label Meaning Example
app.kubernetes.io/name The name of the application mysql
app.kubernetes.io/instance A unique name for this deployment of the app mysql-prod
app.kubernetes.io/version The application version (often a SemVer or image tag) 8.0.36
app.kubernetes.io/component The role this piece plays within the app database, cache, web
app.kubernetes.io/part-of The higher-level application this belongs to wordpress
app.kubernetes.io/managed-by The tool managing the object helm, kustomize, argocd

name is what the software is; instance is which install of it this is (so two MySQLs in one namespace can be told apart). Helm sets several of these automatically. The benefit is interoperability — kubectl get pods -l app.kubernetes.io/part-of=wordpress works the same way for anyone’s WordPress.

Note: many older charts and examples use a bare app: <name> label instead. That still works perfectly as a selector; the app.kubernetes.io/* set is simply the standardised convention. You will see both in the wild, and you can carry both during a migration.

Setting and reading labels with kubectl

Labels are usually written declaratively in metadata.labels, but kubectl label is invaluable for ad-hoc work:

# Add (or change) a label on a live object
kubectl label pod my-pod tier=frontend

# Overwrite an existing label (without --overwrite this errors if the key exists)
kubectl label pod my-pod tier=backend --overwrite

# Remove a label (note the trailing minus)
kubectl label pod my-pod tier-

# Label many objects at once by selector
kubectl label pods -l app=nginx environment=staging

# Label everything of a kind
kubectl label nodes --all gpu=false

To see labels:

kubectl get pods --show-labels          # append a LABELS column with everything
kubectl get pods -L tier -L environment # add named columns for specific keys

--show-labels dumps the full map; -L <key> (capital L) promotes specific label keys to their own columns, which is far more readable when you only care about one or two.

Label selectors: querying by label

A label selector is a query over the labels map. Kubernetes supports two syntaxes, and both ultimately answer the same question — which objects carry the labels I asked for?

Equality-based selectors

Equality-based selectors match on exact values (or inequality). The operators are:

Operator Meaning Example
= or == key exists and value equals environment=production
!= key exists with a different value, or (per kubectl semantics) is absent tier!=frontend

Multiple equality requirements are comma-separated and AND together:

kubectl get pods -l 'environment=production,tier=frontend'

That returns Pods that are both environment=production and tier=frontend. There is no OR in a single selector — comma is always AND. (If you genuinely need OR, you use a set-based in with multiple values, below.)

Set-based selectors

Set-based selectors are richer: they match against sets of values and can test for the mere presence or absence of a key. The operators:

Operator Meaning Example
in key’s value is one of the listed set environment in (production, staging)
notin key’s value is not in the set, or the key is absent tier notin (frontend, backend)
<key> (exists) the key is present (any value, including empty) partition
!<key> (not exists) the key is absent !partition

On the command line, set-based requirements are also comma-separated and AND together, and you can freely mix them with equality terms:

# value is in a set
kubectl get pods -l 'environment in (production, staging)'

# key must exist at all
kubectl get pods -l 'partition'

# key must NOT exist
kubectl get pods -l '!partition'

# mix: exists + in + equality, all ANDed
kubectl get pods -l 'partition, environment in (prod,staging), tier=frontend'

in (a, b) is the idiomatic way to express OR over values of one key. Note the subtle but important point about absence: tier!=frontend and tier notin (frontend) both match objects that lack the tier key entirely, not just those with a different value — a frequent source of “why did that get matched?” surprises.

The two syntaxes are not interchangeable everywhere

Crucially, not every API object accepts both syntaxes. The split is historical:

So a Service selector is always a flat AND of equalities; a Deployment selector can be far richer. We will see both in the wiring section.

matchLabels vs matchExpressions in YAML

When an object uses the structured LabelSelector (Deployments, NetworkPolicy, affinity, PDBs, and so on), the selector has two optional fields:

selector:
  matchLabels:
    app: web
    tier: frontend
  matchExpressions:
    - key: environment
      operator: In
      values: ["production", "staging"]
    - key: track
      operator: NotIn
      values: ["canary"]
    - key: partition
      operator: Exists
    - key: deprecated
      operator: DoesNotExist

The combination rule is simple and worth committing to memory:

Every requirement across both matchLabels and matchExpressions must be satisfied — they are ANDed together. There is no way to OR two requirements except In over the values of a single key.

Two edge cases that trip people up:

How selectors wire objects together

This is the heart of the lesson. Each of the following objects uses a selector to find other objects at runtime. Same idea everywhere, different field names and slightly different rules.

Services → Pods (endpoints)

A Service’s spec.selector is a plain equality map. The control plane continuously evaluates it and writes the matching Pods’ IP:port pairs into an EndpointSlice (the modern replacement for the older Endpoints object). kube-proxy then programs those endpoints into the dataplane, so traffic to the Service load-balances across exactly the Pods that match right now.

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:                 # equality-only map; ANDed
    app.kubernetes.io/name: web
    app.kubernetes.io/component: frontend
  ports:
    - port: 80
      targetPort: 8080

Key facts:

For the full Services treatment see Services, networking, endpoints & DNS.

Deployments / ReplicaSets → Pods

A ReplicaSet (and the Deployment that manages it) owns Pods via spec.selector (a structured LabelSelector) and stamps new Pods using spec.template.metadata.labels. Two rules make this safe:

  1. The template’s labels must match the selector. If spec.selector.matchLabels asks for app=web but the Pod template does not carry app=web, the API server rejects the object — the controller would create Pods it then could not recognise as its own.
  2. The selector is immutable after creation on Deployments, ReplicaSets, StatefulSets, and DaemonSets. You cannot edit it in place; you must delete and recreate the object. This is deliberate: changing the selector would orphan the existing Pods (they would no longer match) and the controller would spin up a whole new set, causing a silent double-deploy.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: web      # immutable; must match template below
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web    # the Pods this Deployment will own
        app.kubernetes.io/version: "1.4.0"
    spec:
      containers:
        - name: web
          image: nginx:1.27

How ownership actually works: a ReplicaSet adopts any Pod that (a) matches its selector and (b) has no controlling ownerReference. This is why a stray hand-made Pod that happens to match a ReplicaSet’s selector can get adopted (and counted, and possibly deleted to satisfy replicas) — a classic gotcha. The Deployment additionally adds a pod-template-hash label to each ReplicaSet and its Pods so that the old and new ReplicaSets during a rollout select disjoint Pod sets. See Deployments, ReplicaSets, rollouts & rollback.

nodeSelector: Pods → Nodes (simplest form)

spec.nodeSelector on a Pod is the most basic node-targeting mechanism: a plain equality map matched against node labels. A Pod is only schedulable onto nodes that carry all the listed labels.

spec:
  nodeSelector:
    disktype: ssd
    topology.kubernetes.io/zone: eu-west-1a

It is equality-only and a hard requirement — if no node matches, the Pod stays Pending. For anything richer (preferences, set logic, “spread”, “avoid”), you use affinity.

Node affinity / anti-affinity: Pods → Nodes (rich form)

Node affinity is nodeSelector with the full structured selector plus soft preferences. It lives under spec.affinity.nodeAffinity and has two modes:

Field Hardness Behaviour
requiredDuringSchedulingIgnoredDuringExecution Hard Must be satisfied or the Pod won’t schedule (like nodeSelector, but with matchExpressions/operators incl. Gt/Lt)
preferredDuringSchedulingIgnoredDuringExecution Soft Scheduler prefers matching nodes, weighted 1–100, but will schedule elsewhere if needed
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:                # terms are ORed
          - matchExpressions:             # expressions within a term are ANDed
              - key: topology.kubernetes.io/zone
                operator: In
                values: ["eu-west-1a", "eu-west-1b"]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 50
          preference:
            matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd"]

Note the OR/AND structure unique to node affinity: multiple nodeSelectorTerms are ORed, while matchExpressions within a single term are ANDed. IgnoredDuringExecution means the rule is only checked at scheduling time — relabelling a node later will not evict an already-running Pod.

Pod affinity / anti-affinity: Pods → Pods (via topology)

Pod (anti-)affinity places Pods relative to other Pods, using a labelSelector to identify those other Pods and a topologyKey to define the domain (“same node”, “same zone”). This is what powers “spread my replicas across zones” (anti-affinity) and “co-locate the cache with the web Pod” (affinity).

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: web   # avoid other 'web' Pods...
          topologyKey: kubernetes.io/hostname   # ...on the same node

Here the labelSelector selects Pods, and topologyKey names a node label whose value defines the topology domain. The optional namespaceSelector/namespaces fields control which namespaces’ Pods are considered. The deep treatment, including topology spread constraints, lives in Scheduling: affinity, topology spread, priority & preemption.

NetworkPolicy: selecting subjects and peers

NetworkPolicy uses label selectors in three distinct places, and getting them straight is the key to reading any policy:

Field What it selects Scope
spec.podSelector The Pods this policy applies to (the subjects) Always within the policy’s own namespace
podSelector inside an ingress/egress from/to peer The peer Pods allowed to talk The peer’s namespace (default: same namespace)
namespaceSelector inside a peer The namespaces whose Pods are peers Cluster-wide (matches namespace labels)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend
  namespace: shop
spec:
  podSelector:                      # this policy governs 'api' Pods in 'shop'
    matchLabels:
      app.kubernetes.io/name: api
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:        # from any namespace labelled team=web
            matchLabels:
              team: web
          podSelector:              # AND only its 'frontend' Pods
            matchLabels:
              app.kubernetes.io/component: frontend

The two subtleties that cause the most confusion:

PodDisruptionBudget: selecting the protected set

A PodDisruptionBudget (PDB) protects a set of Pods from voluntary disruptions (node drains, cluster upgrades) by guaranteeing a minimum availability. It selects its protected Pods with a structured LabelSelector:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2          # or maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: web

The eviction API (used by kubectl drain) consults every PDB whose selector matches the Pod being evicted and refuses the eviction if it would breach the budget. The gotcha: a PDB whose selector matches no Pods, or overlapping PDBs that both match a Pod, lead to confusing drains — keep one PDB per logical workload and make sure its selector actually matches the running Pods.

The pattern, summarised

Object Selector field Selector type Selects Mutable?
Service spec.selector Equality map Pods (→ endpoints) Yes
ReplicaSet / Deployment / StatefulSet / DaemonSet spec.selector LabelSelector Pods (ownership) No (immutable)
Job spec.selector (usually auto) LabelSelector Pods No
nodeSelector spec.nodeSelector Equality map Nodes Per-Pod (immutable post-create)
Node affinity spec.affinity.nodeAffinity LabelSelector + weights Nodes No
Pod (anti-)affinity spec.affinity.pod(Anti)Affinity LabelSelector + topologyKey Pods No
NetworkPolicy podSelector, namespaceSelector LabelSelector Pods / Namespaces Yes
PodDisruptionBudget spec.selector LabelSelector Pods No

Notice the recurring theme: controllers and the dataplane re-evaluate selectors continuously. Add a label to a Pod and it can join a Service, a NetworkPolicy’s protected set, or be adopted by a ReplicaSet — without you touching those objects. This dynamism is the power and the danger of label-based wiring.

Annotations: the non-identifying metadata

Annotations live in metadata.annotations and look syntactically like labels (the key follows the same prefix/name rules as label keys), but they are a fundamentally different tool:

Use an annotation when the data is descriptive rather than selectable. Common, real-world annotations you will meet:

Annotation Set by Purpose
kubernetes.io/change-cause kubectl ... --record / you Text shown in kubectl rollout history for each revision
kubectl.kubernetes.io/last-applied-configuration kubectl apply Snapshot of the last applied manifest, used for 3-way merge
deployment.kubernetes.io/revision Deployment controller The revision number of a ReplicaSet
kubectl.kubernetes.io/default-container you Which container kubectl logs/exec defaults to
nginx.ingress.kubernetes.io/rewrite-target (and many *.ingress.*) you Per-Ingress controller behaviour (rewrites, TLS, timeouts)
service.beta.kubernetes.io/aws-load-balancer-* (and cloud equivalents) you Cloud LoadBalancer configuration
kubectl.kubernetes.io/restartedAt kubectl rollout restart Timestamp injected into the Pod template to force a rollout

That last one is a neat illustration of the labels/annotations divide: kubectl rollout restart works by writing a changing annotation into the Pod template. Because the template changed, the Deployment rolls; because it is an annotation (not a label), it does not alter what any selector matches. A label could not be used here without risking the selector.

Managing annotations mirrors labels:

kubectl annotate pod my-pod owner="payments-team"
kubectl annotate pod my-pod owner="platform-team" --overwrite
kubectl annotate pod my-pod owner-                 # remove
kubectl annotate deploy/web kubernetes.io/change-cause="bump to nginx:1.27"

There is no --show-annotations; to view them, use kubectl get pod my-pod -o jsonpath='{.metadata.annotations}' or kubectl describe.

Field selectors: filtering by object fields

Field selectors are the other server-side filter, and they are not labels. A field selector filters the list returned by the API server according to the value of a field of the resource — typically a status or spec field — rather than the labels map. You pass them with --field-selector.

# Pods that are actually running
kubectl get pods --field-selector status.phase=Running

# Pods NOT running (negation works)
kubectl get pods --field-selector status.phase!=Running

# Pods on a specific node (great for "what's on this node?")
kubectl get pods --all-namespaces --field-selector spec.nodeName=node-3

# Combine terms (comma = AND); combine with a label selector too
kubectl get pods \
  --field-selector status.phase=Running,spec.nodeName=node-3 \
  -l app.kubernetes.io/name=web

# Everything EXCEPT a namespace — handy in scripts
kubectl get pods --all-namespaces --field-selector metadata.namespace!=kube-system

# Non-namespaced example: filter events by the object they reference
kubectl get events --field-selector involvedObject.kind=Pod,type=Warning

Critically, which fields are selectable is fixed per resource type by the API server — you cannot field-select on arbitrary fields the way you can label-select on arbitrary labels. The supported set is small and resource-specific. The most useful, by resource:

Resource Commonly supported field selectors
Pod metadata.name, metadata.namespace, status.phase, status.podIP, status.nominatedNodeName, spec.nodeName, spec.restartPolicy, spec.schedulerName, spec.serviceAccountName
Node metadata.name, spec.unschedulable
Event involvedObject.kind/name/namespace/uid, reason, reportingComponent, source, type, metadata.namespace
Service metadata.name, metadata.namespace, spec.clusterIP, spec.type
Secret / ConfigMap metadata.name, metadata.namespace, plus Secret supports type
ReplicaSet / Deployment / Job / etc. metadata.name, metadata.namespace, status.successful (Job) — mostly the universal name/namespace pair
Almost every resource metadata.name, metadata.namespace (these two are universal)

Operators are limited compared with label selectors: field selectors support =/== and != only (no set-based in/notin, no existence checks), and multiple terms AND with commas. Using an unsupported field returns a clear field label not supported error — that error is itself how you discover what is and isn’t selectable.

Field selectors vs label selectors at a glance

Label selector (-l) Field selector (--field-selector)
Operates on metadata.labels (arbitrary, user-defined) Specific object fields (status.phase, spec.nodeName, …)
Extensible? Yes — any label you invent No — only the server-allowlisted fields per resource
Operators =, !=, in, notin, exists, ! =, != only
Combination comma = AND comma = AND
Used by controllers to wire objects? Yes No — purely a client/list-time filter
Typical use “all Pods of app X” “all Pods on node Y”, “Running Pods”, “Warning events”

You can and often will combine them: -l to pick the application, --field-selector to narrow by runtime state.

Kubernetes core objects

The diagram above shows the core objects — Pods, ReplicaSets/Deployments, Services and the rest — and the way they connect: every one of those connections is, under the hood, a label selector resolving “which objects match these tags right now”, which is exactly the machinery this lesson has been pulling apart.

Going deeper

You can use everything above productively without reading this section. It is for when you want to know how the machinery actually works — how a selector is evaluated at scale, which filters are cheap and which are expensive, the labels the platform sets without asking you, and the precise rules behind “immutable” and “256 KiB.” None of it is required to pass an exam, but it is the difference between using selectors and understanding them.

How a selector is actually evaluated (and why it stays fast)

We casually say labels are “indexed,” but it is worth being precise, because it explains a lot of behaviour. etcd — the datastore behind the API server — does not maintain a secondary index on your labels. It stores each object under a key path like /registry/pods/<namespace>/<name>. When you run kubectl get pods -l app=web, the API server retrieves the candidate objects for that resource-and-namespace key range and then applies your label selector as an in-memory filter, returning only the matches. Two consequences fall out of this:

Controllers do not re-list on every reconcile. Each controller runs an informer — a local, continuously-updated cache of the objects it cares about — and evaluates its selector against that in-memory shared cache. So a Deployment matching Pods, or a Service resolving endpoints, is comparing labels in RAM, not hammering etcd. This is the real reason label-based wiring scales.

The “Indexed for fast lookup? Yes (label index)” row in the Core-concept table earlier is a useful shorthand — labels behave as if indexed because filtering is fast and server-side — but strictly, the speed comes from namespace/resource scoping and in-memory caches, not from a secondary label index in etcd.

Which field selectors are index-backed — and why the kubelet relies on it

Label selectors accept any key you invent. Field selectors are the opposite: each resource type declares a small, fixed allowlist of selectable fields, and only a handful of those are backed by a real index in the API server’s watch cache. The most important indexed field selector in the whole system is spec.nodeName on Pods.

Why does that matter? Every kubelet watches the API server for “the Pods assigned to my node” using a field selector — effectively --field-selector spec.nodeName=<myNode>. On a 5,000-node cluster that is 5,000 kubelets each asking a filtered question. Because Pods are indexed by nodeName in the watch cache, the API server answers each one without scanning all Pods. Take that index away and per-node watches would degrade badly. So when you type:

kubectl get pods -A --field-selector spec.nodeName=node-3

you are using the same index the control plane depends on. By contrast, a field selector on a supported-but-unindexed field is evaluated by streaming and filtering — correct, but not free — and a field that is not on the allowlist at all returns a hard field label not supported error rather than silently scanning. That error is a feature: it stops you writing a query the server cannot answer efficiently.

matchExpressions has powers a plain LabelSelector does not

The structured LabelSelector operators are only In, NotIn, Exists, DoesNotExist. But node affinity uses a richer requirement type (NodeSelectorRequirement) that adds two more operators, Gt and Lt (numeric greater-/less-than, with a single integer-valued string in values), and a whole second field, matchFields, for matching on a node’s fields (in practice metadata.name) rather than its labels:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/arch
              operator: In
              values: ["amd64"]
            - key: example.com/gpu-count
              operator: Gt          # Gt/Lt exist ONLY in node (anti-)affinity
              values: ["0"]         # exactly one value, must parse as an integer
          matchFields:
            - key: metadata.name    # match a node FIELD, not a label
              operator: NotIn
              values: ["control-plane-1"]

Do not expect Gt/Lt/matchFields in a Service, Deployment, NetworkPolicy or PDB selector — they are specific to the scheduling nodeAffinity/podAffinity requirement types. This is the single most common “but the docs said set-based selectors support…” confusion.

Ownership: the selector finds candidates, ownerReferences claims them

Earlier we said a ReplicaSet “owns” Pods via its selector. The full mechanism has two moving parts, and the distinction explains the adoption gotcha:

  1. The selector finds candidate Pods — everything currently wearing the right labels.
  2. metadata.ownerReferences records the claim. When a ReplicaSet creates a Pod it stamps an owner reference with controller: true pointing back to itself. A controller will adopt a matching Pod that has no controller owner, and release (orphan) a Pod that stops matching its selector.

So a stray hand-made Pod that merely matches a ReplicaSet’s selector — and has no controller of its own — can be adopted, counted toward replicas, and even deleted to satisfy the desired count. Conversely, kubectl delete deployment web --cascade=orphan removes the Deployment but leaves the ReplicaSet and Pods, precisely by severing those owner references while the labels stay put. The pod-template-hash label the Deployment adds is what keeps the old and new ReplicaSets’ selectors disjoint during a rollout, so the two never fight over the same Pods.

The labels and annotations the platform sets for you

A surprising amount of the label space is populated by Kubernetes. Knowing these lets you select on them and read policies others wrote:

Set on Key Meaning / use
Every Node kubernetes.io/hostname, kubernetes.io/os, kubernetes.io/arch Basic node identity; target with nodeSelector/affinity
Every Node (cloud) topology.kubernetes.io/region, topology.kubernetes.io/zone, node.kubernetes.io/instance-type Zone/region spread, instance-type targeting
Every Namespace kubernetes.io/metadata.name Auto-set to the namespace’s own name — the clean way to target a namespace by name in a namespaceSelector
Every Namespace (opt-in) pod-security.kubernetes.io/enforce (/warn, /audit) Pod Security Admission level — policy driven entirely by namespace labels
Deployment’s Pods & RS pod-template-hash Distinguishes rollout generations
StatefulSet’s Pods statefulset.kubernetes.io/pod-name, apps.kubernetes.io/pod-index Per-Pod identity and ordinal (enables per-replica Services)
DaemonSet / StatefulSet controller-revision-hash Which revision a Pod belongs to
Job’s Pods batch.kubernetes.io/job-name, batch.kubernetes.io/controller-uid Auto-generated Job ownership (see below)
EndpointSlice kubernetes.io/service-name Which Service the slice belongs to (you used this in the lab)

That kubernetes.io/metadata.name auto-label is a genuinely useful trick. To let a NetworkPolicy accept traffic from exactly the payments namespace by name, you do not need to label that namespace yourself:

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: payments   # auto-set on every Namespace

And because Pod Security Admission is configured with labels, hardening a namespace to the restricted standard is a kubectl label away — covered in Pod Security Admission: baseline & restricted.

Selector immutability, object by object

“The selector is immutable” is true for more objects than just Deployments, and false for a couple you might assume. The precise picture:

Object spec.selector mutable? Notes
Deployment, ReplicaSet, StatefulSet, DaemonSet No Changing it would orphan the current Pods and double-deploy; API rejects the edit — delete & recreate
Job No Normally auto-generated; to set your own you must also set spec.manualSelector: true (rarely a good idea)
Service Yes Editing it re-points endpoints live — the basis of some manual blue/green flips
NetworkPolicy Yes podSelector/peer selectors can be edited freely
PodDisruptionBudget Yes (avoid it) Editable, but changing which Pods a PDB protects mid-drain is a foot-gun
Pod nodeSelector / affinity No Pods are essentially immutable after creation; you replace, not edit

The Job case is worth seeing, because most people never write a Job selector at all:

apiVersion: batch/v1
kind: Job
metadata:
  name: import
spec:
  # No manualSelector -> the controller auto-generates a unique selector on
  #   batch.kubernetes.io/controller-uid and stamps it onto the template,
  #   guaranteeing this Job never adopts another Job's Pods.
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: import
          image: busybox:1.36
          command: ["sh", "-c", "echo done"]

Setting manualSelector: true and supplying your own selector defeats that safety and lets two Jobs fight over Pods — which is exactly why the field is gated behind an explicit opt-in.

The 256 KiB cap, and the last-applied / Server-Side Apply story

The annotation limit is precise: the sum of all annotation keys and values on one object must be ≤ 256 KiB (262,144 bytes), enforced by API validation. That is separate from — and smaller than — the ceiling on the whole object, which is bounded by etcd’s request size (roughly 1.5 MiB by default, --max-request-bytes). Blow either and writes fail.

This is why one legacy annotation used to cause real pain. Client-side kubectl apply stores your entire last manifest in the kubectl.kubernetes.io/last-applied-configuration annotation so it can compute a three-way merge — effectively doubling your object into its own metadata. On large ConfigMaps or CRDs that pushed objects toward the size ceiling. Server-Side Apply (SSA) fixed this: it tracks ownership in metadata.managedFields instead, which is not an annotation (so it does not count against the 256 KiB annotation budget, though it still adds to overall object size). The old kubectl apply --record flag that wrote kubernetes.io/change-cause is deprecated for the same family of reasons. Practical rules: keep bulky or binary data out of annotations entirely — reference a ConfigMap or Secret instead — and prefer kubectl apply --server-side for large declarative objects.

Hands-on lab

We will create a tiny multi-version app, see selectors wire a Service to Pods, watch what set-based selectors do, prove the Deployment-selector immutability rule, and use field selectors. Everything is free and local.

1. Start a cluster

kind create cluster --name labels-lab
# or: minikube start -p labels-lab
kubectl get nodes

2. Create two versioned Deployments sharing one app label

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: web
      app.kubernetes.io/version: "1"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
        app.kubernetes.io/version: "1"
        track: stable
    spec:
      containers:
        - name: web
          image: registry.k8s.io/echoserver:1.10
          ports: [{containerPort: 8080}]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: web
      app.kubernetes.io/version: "2"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
        app.kubernetes.io/version: "2"
        track: canary
    spec:
      containers:
        - name: web
          image: registry.k8s.io/echoserver:1.10
          ports: [{containerPort: 8080}]
EOF

3. Explore equality and set-based selectors

# Everything in the app (both versions) — shared label
kubectl get pods -l app.kubernetes.io/name=web --show-labels

# Only v1 (equality, ANDed)
kubectl get pods -l 'app.kubernetes.io/name=web,app.kubernetes.io/version=1'

# v1 OR v2 via set-based 'in' (here, trivially all)
kubectl get pods -l 'app.kubernetes.io/version in (1,2)'

# Only Pods that have a 'track' label at all (exists)
kubectl get pods -l 'track'

# Only Pods that are NOT canary (note: also matches Pods with no 'track')
kubectl get pods -l 'track!=canary'

# Promote labels to columns
kubectl get pods -L app.kubernetes.io/version -L track

You should see 3 Pods total under the shared app.kubernetes.io/name=web label (2 from v1, 1 from v2), and the filters narrow as described.

4. A Service that spans both versions

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app.kubernetes.io/name: web    # equality only — spans v1 AND v2
  ports:
    - port: 80
      targetPort: 8080
EOF

kubectl get endpointslices -l kubernetes.io/service-name=web

The EndpointSlice should list 3 ready endpoints — the Service load-balances across both versions because they share app.kubernetes.io/name=web. This is the canonical pattern: a stable Service over a shared label, with version labels used for finer-grained selection elsewhere.

5. Prove the Deployment selector is immutable

kubectl patch deployment web-v1 --type merge \
  -p '{"spec":{"selector":{"matchLabels":{"app.kubernetes.io/name":"web2"}}}}'

Expected: the API server rejects the patch with a message that spec.selector is immutable / field is immutable. This is the safety rule from earlier — selectors cannot be edited in place on Deployments.

6. Field selectors

# Running Pods only (a field, not a label)
kubectl get pods --field-selector status.phase=Running

# Which node each Pod landed on, then filter to one node
kubectl get pods -o wide
NODE=$(kubectl get pods -o jsonpath='{.items[0].spec.nodeName}')
kubectl get pods --field-selector spec.nodeName="$NODE"

# Combine field + label selectors
kubectl get pods --field-selector status.phase=Running -l track=canary

# Trigger the 'unsupported field' error to see the allowlist in action
kubectl get pods --field-selector spec.restartPolicy=Always   # supported
kubectl get pods --field-selector spec.nodeNameXYZ=foo        # error: not supported

7. Annotations vs labels in practice

# Annotate (descriptive note) vs label (selectable tag)
kubectl annotate deployment web-v1 kubernetes.io/change-cause="initial echoserver"
kubectl label deployment web-v1 owner=platform     # this you COULD select on

# A rollout-restart writes a changing ANNOTATION into the template
kubectl rollout restart deployment web-v1
kubectl get deploy web-v1 -o jsonpath='{.spec.template.metadata.annotations}'
# -> shows kubectl.kubernetes.io/restartedAt — note: an annotation, so selectors are untouched

Validation

Cleanup

kubectl delete deployment web-v1 web-v2
kubectl delete service web
kind delete cluster --name labels-lab     # or: minikube delete -p labels-lab

Cost note

Everything here runs on a local kind/minikube cluster on your laptop — zero cloud cost. The echoserver image is small and the workloads are trivial.

Practice challenges

Work these before peeking. Each has a collapsible solution with a one-line why. They assume the two-Deployment setup from the lab (web Pods labelled app.kubernetes.io/name=web with an app.kubernetes.io/version of 1 or 2, and a track of stable or canary), but you can reason about most on paper.

Challenge 1 — Beginner: AND two equalities. Write the kubectl get pods command that returns only Pods that are both version 1 and on the stable track.

<details><summary>Solution</summary>

kubectl get pods -l 'app.kubernetes.io/version=1,track=stable'

The comma is AND; there is no OR in an equality selector, so both requirements must hold. </details>

Challenge 2 — Beginner: existence, not value. List every Pod that carries a track label at all, regardless of its value — then list every Pod that does not have one.

<details><summary>Solution</summary>

kubectl get pods -l 'track'      # has the key (any value, incl. empty)
kubectl get pods -l '!track'     # does NOT have the key

Bare key is the exists check; !key is does-not-exist. Neither looks at the value. </details>

Challenge 3 — Intermediate: OR over one key. Using a single selector, return Pods whose version is 1 or 2 (but which would exclude a future version 3) — and say why you cannot do this on a Service.

<details><summary>Solution</summary>

kubectl get pods -l 'app.kubernetes.io/version in (1,2)'

in (…) is the only way to express OR over a key’s values. A Service spec.selector is an equality-only map, so it cannot express in — you would instead give both versions a shared label (e.g. app.kubernetes.io/name=web) and select on that. </details>

Challenge 4 — Intermediate: build a matchExpressions. Write the matchExpressions block for a Deployment/NetworkPolicy selector meaning: environment is prod or staging, the track is not canary, and a region label is present (any value).

<details><summary>Solution</summary>

matchExpressions:
  - key: environment
    operator: In
    values: ["prod", "staging"]
  - key: track
    operator: NotIn
    values: ["canary"]
  - key: region
    operator: Exists      # note: NO values field for Exists/DoesNotExist

All three requirements AND together. Exists/DoesNotExist must omit values; In/NotIn must supply a non-empty values list. </details>

Challenge 5 — Intermediate: label + field in one query. With one kubectl get, list only the Running Pods that are part-of=shop — and explain which mechanism did which half of the work.

<details><summary>Solution</summary>

kubectl get pods \
  -l app.kubernetes.io/part-of=shop \
  --field-selector status.phase=Running

The label selector (-l) picks the application (an arbitrary user label); the field selector narrows by runtime state (status.phase, a real object field, not a label). They combine with AND, both applied server-side. </details>

Challenge 6 — Advanced: roll without touching the selector. You changed a ConfigMap and need the web Deployment’s Pods to restart to pick it up — but you must not alter anything a selector matches. Give two ways.

<details><summary>Solution</summary>

# (a) let kubectl write a changing annotation into the template:
kubectl rollout restart deployment web-v1

or (b) add/update a checksum/config annotation on spec.template.metadata.annotations (e.g. a hash of the ConfigMap) in your manifest and re-apply. Either way the Pod template changes → a rollout triggers, but because the change is an annotation, no selector’s result moves. A label here would risk the immutable selector. </details>

Common beginner mistakes

These are misconceptions, not symptom→fix items (those are in the troubleshooting table below). Each is a wrong mental model and the right one to replace it with.

“My extra Pod matches the ReplicaSet’s labels — that’s fine, it’ll just join.” It will not just join harmlessly. A controller adopts any matching Pod that has no controller owner of its own, counts it toward replicas, and may delete it to hit the desired count — so your carefully hand-made debug Pod vanishes. Right model: selector labels are a controller’s territory. Give ad-hoc Pods a label no controller selects on (e.g. debug=true and nothing the RS wants), and they stay yours.

“I’ll just edit the Deployment’s selector to re-point it.” You can’t — spec.selector is immutable on Deployments, ReplicaSets, StatefulSets and DaemonSets, and the API server rejects the edit. This is a guardrail, not a bug: changing the selector would orphan the running Pods (they’d stop matching) and spawn a whole fresh set, a silent double-deploy. Right model: pick selector labels once, keep them stable forever, and put anything that changes (build SHA, timestamp) in annotations or non-selector labels. If you truly must change a selector, delete and recreate the object deliberately.

“I’ll store the build number / commit SHA / owner email as a label so it shows up.” Labels are the wrong home for descriptive data. They are length- and charset-constrained (≤63 chars, no spaces, no @ or : in the value), they get indexed and selected on, and a value that changes every build means churn on something meant to be stable. Right model: if nothing needs to find the object by it, it belongs in an annotation — free-form, large, ignored by every selector. A version you route traffic to is a label; a git-commit=… you merely display is an annotation.

tier!=frontend returns Pods whose tier is something else.” It returns those and every Pod that has no tier label at all. != and notin treat an absent key as “not equal,” so they match more than beginners expect. Right model: to mean “has a tier, but not frontend,” require the key too: -l 'tier,tier!=frontend'.

“Annotations can be selected on if I really need to.” There is no annotation selector anywhere in Kubernetes — not in kubectl, not in a Service, not in a controller. Annotations are never indexed and never queried. Right model: the moment you find yourself wishing you could select on an annotation, that fact wanted to be a label all along. Promote it.

“The Service selector and the Deployment selector should be identical.” They often aren’t, and shouldn’t have to be. A Service selector is an equality-only map and is frequently broader (e.g. app.kubernetes.io/name=web to span v1 and v2), while a Deployment’s structured selector is narrower and immutable. Right model: the Service selects “everything I should load-balance across”; the Deployment selects “the Pods I own.” Different questions, sometimes-overlapping selectors.

namespaceSelector and podSelector in a NetworkPolicy always combine.” Only when they sit in the same from/to list item (then they AND). As two separate list items they OR — a common way to accidentally allow far more traffic than intended. Right model: indentation is semantics here; one dash versus two changes AND into OR.

Common mistakes & troubleshooting

Symptom Likely cause Fix
Service has no endpoints; traffic 503s Service selector does not match any Pod’s labels (typo, wrong key, or Pods not ready) Compare kubectl get svc web -o jsonpath='{.spec.selector}' with kubectl get pods --show-labels; fix the labels or selector; check readiness
selector does not match template labels on apply Deployment spec.selector.matchLabels not a subset of spec.template.metadata.labels Make the template carry every label the selector requires
Cannot change a Deployment’s selector spec.selector is immutable Delete and recreate the Deployment (plan for a brief replacement)
A hand-made Pod got deleted/adopted unexpectedly Its labels matched a ReplicaSet’s selector and it had no controller owner Use distinct labels for ad-hoc Pods; never reuse a controller’s selector labels
tier!=frontend matched Pods with no tier label != / notin also match objects where the key is absent Add an exists term: -l 'tier,tier!=frontend' to require the key
field label not supported error The field is not in the API server’s allowlist for that resource Use a supported field (see the table) or filter client-side with -o jsonpath/grep
NetworkPolicy allows too much / too little namespaceSelector and podSelector placed in separate from items (ORed) vs one item (ANDed) Put both in the same list item to AND them; separate items to OR
error: 'tier' already has a value on kubectl label Setting an existing key without --overwrite Add --overwrite
Annotation rejected: too long Total annotations exceed 256 KiB Move bulky data into a ConfigMap/Secret and reference it

Best practices

Security notes

Interview & exam questions

  1. What is the difference between a label and an annotation, and when would you use each? Labels are identifying metadata used by selectors to find and group objects; they are short and strictly formatted. Annotations are non-identifying, free-form, larger metadata for humans and tools that nothing selects on. Use a label when something must find the object; an annotation when it just needs a note about it.

  2. Explain equality-based vs set-based selectors and which objects support each. Equality-based uses =/==/!= (ANDed by commas). Set-based adds in, notin, exists/!. Older objects (Service, ReplicationController) take only an equality map; newer objects (Deployment, ReplicaSet, NetworkPolicy, PDB, affinity) take the structured LabelSelector supporting both.

  3. What is the relationship between matchLabels and matchExpressions? Both live in a structured LabelSelector. matchLabels is a map of equality requirements; matchExpressions is a list of {key, operator, values} using In/NotIn/Exists/DoesNotExist. All requirements across both are ANDed. matchLabels: {a: b} equals a matchExpressions In [b].

  4. Why is a Deployment’s spec.selector immutable, and what must be true about it relative to the Pod template? It is immutable because changing it would orphan the existing Pods (they would stop matching) and spawn a fresh set, causing a silent double-deploy. The template’s labels must be a superset of the selector, or the API rejects the object.

  5. A Service has no endpoints. How do you debug it? Compare the Service’s spec.selector with the Pods’ actual labels (--show-labels); confirm the Pods exist and are Ready (unready Pods are excluded). Check you are in the right namespace. Remember a selector-less Service requires manually managed EndpointSlices.

  6. What is the difference between a label selector and a field selector? A label selector queries the arbitrary metadata.labels map (with set logic). A field selector queries specific, server-allowlisted object fields (status.phase, spec.nodeName), supports only =/!=, and is a client list-time filter — controllers do not wire objects with field selectors.

  7. How do you find every Pod scheduled on a particular node, including across all namespaces? kubectl get pods --all-namespaces --field-selector spec.nodeName=<node>. spec.nodeName is one of the supported Pod field selectors.

  8. Why might kubectl get pods -l tier!=frontend return Pods you didn’t expect? != (and notin) match objects where the key is absent, not only those with a different value. To require the key, add an existence term: -l 'tier,tier!=frontend'.

  9. What does an empty selector ({}) mean, and where does that matter? An empty LabelSelector matches everything in scope. It is the idiom for “all Pods” — e.g. a NetworkPolicy podSelector: {} applies to every Pod in the namespace (the basis of default-deny). A null selector, by contrast, typically matches nothing.

  10. How does kubectl rollout restart force a restart without changing what selectors match? It writes a changing annotation (kubectl.kubernetes.io/restartedAt) into the Pod template. The template change triggers a rollout; because it is an annotation rather than a label, no selector’s result changes.

  11. In a NetworkPolicy, what is the difference between putting namespaceSelector and podSelector in one from item vs two? In one item they are ANDed (the peer must be in that namespace and match the pod labels). In two separate items under from they are ORed. Indentation changes the policy’s meaning.

  12. What is the recommended set of common labels and why use them? The app.kubernetes.io/* set: name, instance, version, component, part-of, managed-by. They give tools a shared vocabulary to understand applications they did not create, enabling consistent grouping, dashboards, and cost allocation.

  13. Which field selector does the kubelet depend on, and why is it index-backed? spec.nodeName on Pods. Each kubelet watches only “Pods on my node” with --field-selector spec.nodeName=<self>; because the API server’s watch cache indexes Pods by nodeName, it serves thousands of per-node watches without scanning every Pod.

  14. How do you target a specific namespace by name in a namespaceSelector without labelling it yourself? Match the auto-set label kubernetes.io/metadata.name: <namespace>. Kubernetes stamps it on every Namespace, so no manual labelling is needed.

Quick check

  1. Which selector syntax can a Service use — equality-based, set-based, or both?
  2. True or false: annotations can be used in a label selector.
  3. Write a matchExpressions entry meaning “the environment key is present with value prod or staging”.
  4. What does kubectl get pods --field-selector status.phase!=Running return?
  5. You add the label app.kubernetes.io/name=web to a stray standalone Pod while a ReplicaSet selects on that label. What can happen?

Answers

  1. Equality-based only. Service spec.selector is a plain map; it cannot express in/notin/exists.
  2. False. Nothing selects on annotations; they are non-identifying. Only labels are selectable.
  3. - {key: environment, operator: In, values: ["prod", "staging"]}.
  4. All Pods whose status.phase is not Running (Pending, Succeeded, Failed, Unknown) — a field selector with !=.
  5. The ReplicaSet may adopt the Pod (it matches the selector and has no controlling owner), count it toward replicas, and potentially delete it to satisfy the desired count. Give ad-hoc Pods labels no controller selects on.

Exercise

On a fresh kind/minikube cluster:

  1. Deploy two Deployments, cache and web, each with the full app.kubernetes.io/* label set (name, instance, component, part-of=shop, managed-by=manual, version).
  2. Create one Service that targets only the web component (not cache) using an equality selector, and verify its EndpointSlice contains only the web Pods.
  3. Write a nodeSelector and an equivalent requiredDuringSchedulingIgnoredDuringExecution node-affinity rule that both target nodes labelled disktype=ssd; label a node and confirm both schedule.
  4. Add pod anti-affinity so the two web replicas avoid sharing a node (topologyKey: kubernetes.io/hostname) and confirm they land on different nodes (or one stays Pending on a single-node cluster — explain why).
  5. Use a single command combining a label selector and a field selector to list only the Running Pods of part-of=shop.
  6. Attempt to edit the web Deployment’s spec.selector; capture the error and explain it.
  7. Add a kubernetes.io/change-cause annotation and confirm it appears in kubectl rollout history while not affecting any selector.

Write down, for each step, which metadata mechanism (label, annotation, field) did the work and why.

Certification mapping

Glossary

Next steps

KubernetesLabelsSelectorsAnnotationskubectlCKAD
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments