Policy-as-Code with Kyverno: Validate, Mutate, Generate, and Verify Image Signatures Admission-Time

In a nutshell

Imagine your cluster has a doorman standing at the only entrance. Every time someone tries to run something — kubectl apply, an Argo sync, a Helm upgrade — the request has to walk past this doorman before it becomes real. Kyverno is that doorman, and it can do four different jobs, all described in plain YAML files you check into git:

• Validate — the bouncer. It blocks anything that breaks the rules (“no :latest images”, “every container needs resource limits”) and hands back a clear reason.
• Mutate — the proofreader. Instead of just rejecting, it quietly fixes the request: fills in a missing securityContext, adds a label, injects a sidecar — so the bouncer rarely has to say no.
• Generate — the concierge. When something new arrives (a fresh namespace), it creates the companion things that should always come with it — a default-deny NetworkPolicy, a registry pull-secret.
• verifyImages — the ID checker. It refuses to admit a container image unless it carries a valid cryptographic signature (and, if you insist, proof of how and where it was built).

The single most important idea: because mutate runs before validate, one Kyverno install can fix a field and then enforce it in the same breath. And every rule is just a Kubernetes resource — no separate policy language to learn, which is exactly why beginners can start here and why it out-powers the alternatives for real platform work.

Level: Advanced — but written with a beginner on-ramp · Time: ~35 min

Read the diagram left to right: the API server calls Kyverno’s webhooks on every write, the request passes through mutate (change), generate (create companions), then validate and verifyImages (gate); the outcome is admitted-and-patched, rejected under Enforce, or — under Audit — admitted with the result recorded in a PolicyReport.

Prerequisites and what you’ll be able to do

Know first: how to kubectl apply a YAML manifest and read one; roughly what Kubernetes admission control is (the API server calling out to a webhook before writing to etcd — covered in Admission Controllers: Validating & Mutating Webhooks); what a Pod securityContext, a namespace, and RBAC roughly are. You do not need to know Rego, Go, or any policy language.

After this lesson you can:

• Read and write a ClusterPolicy that uses all four rule types — validate, mutate, generate, and verifyImages.
• Pick the correct anchor (?*, =(), (), +()) and scope a rule with match/exclude.
• Roll a policy out safely: Audit → read the PolicyReport → promote to Enforce, one namespace at a time.
• Require cosign-signed images and SLSA provenance at admission, and pin the verified tag to a digest.
• Default missing fields with a mutate rule so validate rules almost never fire.
• Unit-test policies offline with the Kyverno CLI and gate every PR in CI.

Every cluster accumulates rules that live in tribal memory: images must come from our registry, every Pod needs resource limits, no latest tags, each namespace gets a default-deny NetworkPolicy. Documented in a wiki, enforced by hope. Policy-as-code moves those rules into the admission path so they are evaluated on every kubectl apply, every Argo sync, every Helm upgrade – before the object is persisted. Kyverno is the policy engine I reach for first because its policies are Kubernetes resources written in YAML, not a separate language, and because it does something Gatekeeper cannot: it mutates and generates resources, not just validates them. This is the end-to-end workflow, from a first validate rule to verifying cosign signatures inline.

1. Kyverno vs OPA/Gatekeeper, and the admission flow

Both Kyverno and OPA/Gatekeeper run as ValidatingWebhookConfiguration (and Kyverno also MutatingWebhookConfiguration) targets that the API server calls during admission. The difference is the authoring model and the verb coverage.

If you have already met OPA Gatekeeper, this lesson is the Kyverno-native counterpart to OPA Gatekeeper: Policy-as-Code Admission Gating; the two engines occupy the same admission slot but trade Rego’s generality for Kyverno’s YAML familiarity and its mutate/generate/verifyImages reach.

The admission flow is the same shape for both. The API server authenticates and authorizes the request, runs mutating webhooks (Kyverno injects defaults here), persists nothing yet, runs validating webhooks (Kyverno enforces here), and only then writes to etcd. Order matters: a mutate rule that adds runAsNonRoot: true runs before a validate rule that requires it, so a single Kyverno install can both fix and gate the same field.

Why “before” is the whole trick. Beginners often ask why you would bother mutating if you are also going to validate. The answer is that mutate turns a failure into a silent fix. Without the mutate rule, a developer who forgets runAsNonRoot gets their deploy rejected and has to go edit YAML. With it, Kyverno fills the field in for them and the validate rule they never saw simply passes. Fewer rejected deploys, same guarantee.

Install with the official Helm chart. Pin the chart version in production.

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm install kyverno kyverno/kyverno \
  --namespace kyverno --create-namespace \
  --version 3.2.6 \
  --set admissionController.replicas=3 \
  --set backgroundController.replicas=2

kubectl -n kyverno get pods
kubectl get crd | grep kyverno.io

Since Kyverno 1.10 the controllers are split – admission, background, cleanup, and reports controllers run as separate Deployments. Run at least 3 admission replicas so a node drain never leaves the webhook unbacked; an unbacked webhook with failurePolicy: Fail blocks the API server for the resources it matches.

The four controllers map cleanly onto the four powers, which is worth internalising early because it tells you where to look when something misbehaves:

2. Writing validate rules with patterns, anchors, and good messages

A ClusterPolicy contains rules; each rule has a match block and exactly one of validate, mutate, generate, or verifyImages. The most common validate style is pattern: you write a fragment that the resource must match. This policy requires CPU and memory limits on every container.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-limits
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "CPU and memory limits are required on every container."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

The anchors are the part people get wrong. ?* means “any non-empty value.” Inside a list, the default behavior makes the pattern apply to every element – so the rule above requires limits on all containers, not just the first. Other anchors:

• =(field) – existence-optional but if present must match. Useful for fields that may be absent.
• () (conditional anchor) – “if this matches, then the sibling keys must also match.” Drives conditional logic.
• X(field) (negation) – the field must not exist.
• +(field) (add anchor) – mutate-only, adds if absent.

A conditional example: only require a read-only root filesystem when the container is not explicitly privileged.

      validate:
        message: "Non-privileged containers must set readOnlyRootFilesystem: true."
        pattern:
          spec:
            containers:
              - =(securityContext):
                  =(privileged): "false"
                  readOnlyRootFilesystem: true

For logic that patterns cannot express, use deny with conditions. This blocks the :latest tag and bare tags using JMESPath against the image string.

      validate:
        message: "Using a mutable ':latest' or untagged image is not allowed."
        deny:
          conditions:
            any:
              - key: "{{ images.containers.*.tag }}"
                operator: AnyIn
                value:
                  - "latest"
              - key: "{{ images.containers.*.tag || '' }}"
                operator: AnyIn
                value:
                  - ""

Failure messages are a UX surface. The message is what a developer sees when their deploy is rejected. “validation error: rule require-limits failed” plus a clear sentence beats a wall of YAML. Write the message as an instruction, not a complaint.

Pattern vs deny: which one do I use?

This is the single most common authoring fork, so hold the distinction firmly:

A rule of thumb: if you can point at the field and say “this must look like that”, reach for pattern; if you catch yourself wanting an if, a ||, a contains, or a comparison, reach for deny.

Scoping every rule: match, exclude, and foreach

match and exclude are not validate-specific — they scope every rule type, and getting them right is more than half of using Kyverno safely. Each takes any (OR — at least one operand matches) or all (AND — every operand matches), and each operand can select on far more than kinds:

This rule requires an owner label on production Deployments, only on create/update, and exempts the Argo CD controller that legitimately creates unlabeled scaffolding:

      match:
        all:
          - resources:
              kinds: [Deployment]
              operations: [CREATE, UPDATE]
              namespaceSelector:
                matchLabels:
                  stage: prod
      exclude:
        any:
          - subjects:
              - kind: ServiceAccount
                name: argocd-applicationset-controller
                namespace: argocd
      validate:
        message: "Production Deployments must carry an owner label."
        pattern:
          metadata:
            labels:
              owner: "?*"

When a rule needs to check each element of a list independently — every container’s image, every volume, every port — reach for foreach. It iterates a JMESPath list and applies a pattern, deny, or (in mutate) a patch to each element. This validates that every container image comes from an approved registry:

      validate:
        message: "Images may only come from registry.internal or ghcr.io/acme."
        foreach:
          - list: "request.object.spec.containers"
            pattern:
              image: "registry.internal/* | ghcr.io/acme/*"

foreach beats a top-level pattern whenever you need per-element messages, per-element context lookups, or to combine the check with preconditions that skip some elements. It is the workhorse for “for every X in this resource, assert Y.”

3. Mutate rules: inject defaults, labels, and sidecars

Mutation is where Kyverno pulls ahead of Gatekeeper. The most useful pattern is defaulting – supplying a sane value so the validate rule never has to fail. Here we set imagePullPolicy: IfNotPresent and a default runAsNonRoot when they are missing, using the add anchor +.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: default-security-context
spec:
  rules:
    - name: default-run-as-non-root
      match:
        any:
          - resources:
              kinds: [Pod]
      mutate:
        patchStrategicMerge:
          spec:
            +(securityContext):
              +(runAsNonRoot): true
            containers:
              - (name): "?*"
                +(imagePullPolicy): IfNotPresent

The (name): "?*" is a conditional anchor that matches every container by name, then + adds the field only if absent – so we never clobber an explicit setting. Strategic merge respects list merge keys, which is why we anchor on name.

For injecting a sidecar or any structural change, patchesJson6902 (JSON Patch, RFC 6902) gives precise control:

      mutate:
        patchesJson6902: |-
          - op: add
            path: "/spec/containers/-"
            value:
              name: logging-sidecar
              image: registry.internal/fluent-bit:2.2
              resources:
                limits: { cpu: "100m", memory: "128Mi" }

Mutate also supports foreach, which is the clean way to patch a variable number of list elements — every container, however many there are — without a fragile strategic-merge anchor:

      mutate:
        foreach:
          - list: "request.object.spec.containers"
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    +(imagePullPolicy): IfNotPresent

A cleaner pattern for shared mutations is mutateExistingOnPolicyUpdate plus a targets block, which lets a policy retroactively patch resources that already exist when the policy changes – handy for rolling a new label across live namespaces without re-applying every manifest. Use it sparingly; it generates write load against the API server. (We take this apart in Going deeper.)

4. Generate rules: auto-create NetworkPolicies and ConfigMaps per namespace

generate rules create downstream resources in response to a trigger. The canonical use is a default-deny NetworkPolicy in every new namespace, so security posture is correct by construction rather than by a checklist.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: default-deny-netpol
spec:
  rules:
    - name: deny-all-ingress
      match:
        any:
          - resources:
              kinds: [Namespace]
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{ request.object.metadata.name }}"
        synchronize: true
        data:
          spec:
            podSelector: {}
            policyTypes:
              - Ingress
              - Egress

Two flags carry the weight. synchronize: true means Kyverno reconciles the generated resource – if someone deletes or edits the NetworkPolicy, the background controller restores it from the policy definition. data embeds the resource inline; the alternative, clone, copies from an existing source object, which is the right choice for distributing a shared registry pull-secret or CA bundle into every namespace:

      generate:
        apiVersion: v1
        kind: Secret
        name: regcred
        namespace: "{{ request.object.metadata.name }}"
        synchronize: true
        clone:
          namespace: platform
          name: regcred

Generate rules need RBAC. The background controller can only create what its ServiceAccount is permitted to, so to generate NetworkPolicies you must grant the kyverno:background-controller an aggregated ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:generate-netpol
  labels:
    app.kubernetes.io/component: background-controller
    rbac.kyverno.io/aggregate-to-background-controller: "true"
rules:
  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies"]
    verbs: ["create", "update", "delete", "get", "list", "watch"]

The rbac.kyverno.io/aggregate-to-background-controller label is what wires this into Kyverno’s role. Forget it and your generate rule silently produces no resources – check kubectl describe clusterpolicy events and the background controller logs first when generation appears to do nothing.

5. Image verification: enforcing cosign signatures and attestations inline

This is the supply-chain payoff. The verifyImages rule blocks any image that does not carry a valid cosign signature, evaluated at admission against the registry’s OCI signature artifact. Keyless verification (Fulcio/Rekor) keys on the signing identity rather than a static public key:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: verify-signed-by-ci
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "registry.internal/*"
          mutateDigest: true
          verifyDigest: true
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/acme/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Three behaviors are worth calling out. mutateDigest: true rewrites the verified tag to its immutable digest in the admitted Pod spec, closing the tag-mutability gap – the image that was verified is provably the image that runs. verifyDigest: true rejects images referenced by tag only when a digest cannot be resolved. And count: 1 against an attestors list lets you require m-of-n signers, e.g. CI plus a release-approver identity.

For static keys, swap keyless for keys:

                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----
                    rekor:
                      url: https://rekor.sigstore.dev

You can go further and require attestations – demand that an SBOM or SLSA provenance predicate exists and satisfies a condition, for example that the build ran on a hosted runner:

      verifyImages:
        - imageReferences: ["registry.internal/*"]
          attestations:
            - type: https://slsa.dev/provenance/v1
              attestors:
                - count: 1
                  entries:
                    - keyless:
                        issuer: "https://token.actions.githubusercontent.com"
                        subject: "https://github.com/acme/*"
              conditions:
                - all:
                    - key: "{{ buildDefinition.externalParameters.workflow.repository }}"
                      operator: Equals
                      value: "https://github.com/acme/payments"

This couples admission to provenance: not just “is it signed” but “was it built from the repo we expect, on the runner we expect.” That is the difference between supply-chain theater and supply-chain control. If the terms cosign, SBOM, and SLSA provenance are new, the producing side of this pipeline — how images get signed and how attestations are generated in CI — is covered in Container Image Supply Chain: Cosign, SBOM & SLSA; Kyverno is the consuming side that enforces it at admission.

6. Policy reporting, background scans, and audit vs enforce

validationFailureAction is the single most important field for safe rollout. Audit allows the resource and records a result; Enforce rejects it. Always start Audit.

In Audit mode (and for background: true policies generally), Kyverno’s reports controller writes results to PolicyReport (namespaced) and ClusterPolicyReport objects, and re-scans existing resources on a schedule – so you see which already-running workloads would fail before you flip to Enforce.

# Per-namespace pass/fail/warn/error tallies
kubectl get policyreport -A

# Drill into one namespace's failing results
kubectl get policyreport -n payments -o yaml \
  | yq '.results[] | select(.result == "fail")'

# Cluster-scoped resources (namespaces, nodes, CRDs)
kubectl get clusterpolicyreport

The background controller is what makes Audit useful: it evaluates policies against the existing cluster on a periodic scan, not only on admission. So a policy you apply today immediately tells you your historical debt, not just your future compliance. Watch the pass/fail columns trend toward zero failures, then promote.

A note on the newer field name. Since Kyverno 1.10, spec.validationFailureAction is being superseded by a per-rule validate.failureAction, with validate.failureActionOverrides to vary the action by namespace. The spec-level field still works (and every example above using it is correct), but new policies increasingly set the action on the rule so a single policy can enforce in prod while only auditing in staging. We show the graduated form in Going deeper.

7. Testing policies with the Kyverno CLI and wiring into CI

Never let a cluster be the first place a policy meets a manifest. The kyverno CLI applies policies to resource files offline and kyverno test runs a declarative test suite with expected results.

# Apply a policy to a manifest and print the verdict
kyverno apply policies/require-limits.yaml \
  --resource manifests/deployment.yaml

# Declarative test suite defined in kyverno-test.yaml
kyverno test .

The test file pins expected outcomes so a policy change that flips a verdict fails CI:

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: limits-suite
policies:
  - policies/require-limits.yaml
resources:
  - manifests/deployment.yaml
results:
  - policy: require-resource-limits
    rule: require-limits
    resource: web
    kind: Deployment
    result: fail

Wire it into the pipeline so policies are unit-tested like any other code, and so application manifests are linted against the live policy set before merge:

# .github/workflows/policy.yaml
name: kyverno-policy
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Kyverno CLI
        run: |
          curl -sL https://github.com/kyverno/kyverno/releases/download/v1.12.0/kyverno-cli_v1.12.0_linux_x86_64.tar.gz \
            | tar -xz kyverno
          sudo mv kyverno /usr/local/bin/
      - name: Run policy tests
        run: kyverno test .
      - name: Lint app manifests against policies
        run: kyverno apply policies/ --resource manifests/ --warn-exit-code 0

Running the same policies in CI that run in the cluster collapses the feedback loop from “deploy rejected at 2am” to “PR check failed in 30 seconds.”

8. Performance, failurePolicy, and safe rollout across many namespaces

failurePolicy decides what happens when the webhook itself is unreachable. Fail (the default for security-critical policies) means the API server rejects the request if Kyverno cannot answer – correct for image verification, dangerous if Kyverno is undersized. Ignore fails open. The honest engineering position: run image-verify policies as Fail and run enough admission replicas with a PodDisruptionBudget that the webhook is never down.

spec:
  failurePolicy: Fail
  webhookTimeoutSeconds: 15        # cap latency added to every matched request
  rules:
    - name: verify
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaceSelector:
                matchExpressions:
                  - key: kyverno.io/enforce
                    operator: In
                    values: ["true"]

Three rollout levers keep this safe at scale:

1. Scope the webhook. A namespaceSelector on the match means the webhook is only invoked for opted-in namespaces. Roll out by labeling namespaces, not by editing the policy.
2. Exclude system namespaces. Always exclude kube-system and kyverno from broad Pod policies, or a Kyverno restart can deadlock on its own webhook.
3. Tune the timeout. Image verification reaches out to a registry and Rekor; a slow registry plus failurePolicy: Fail is an outage. Set webhookTimeoutSeconds deliberately and monitor admission latency.

Exclude protected namespaces directly in the match block:

      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: ["kube-system", "kyverno", "kube-node-lease"]

Going deeper

Everything above is the working surface. This section is for the reader who already ships Kyverno and wants the internals, the newer APIs, and the knobs that matter at scale.

Mutate on admission vs mutate-existing

There are two fundamentally different mutation timings, and conflating them is a classic mistake.

• Mutate on admission (everything in §3) fires synchronously in the mutating webhook, on the incoming object, before it is stored. It is cheap and instant, but it only ever touches the resource being admitted.
• Mutate-existing fires asynchronously in the background controller, against resources that already live in etcd, triggered either by an admission event on a different resource or by a policy update. You opt in with mutateExistingOnPolicyUpdate: true and a targets block that names what to patch:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: propagate-label
spec:
  mutateExistingOnPolicyUpdate: true
  rules:
    - name: add-managed-label
      match:
        any:
          - resources:
              kinds: [Namespace]
      mutate:
        targets:
          - apiVersion: v1
            kind: ConfigMap
            namespace: "{{ request.object.metadata.name }}"
        patchStrategicMerge:
          metadata:
            labels:
              managed-by: platform

The trade-off is load: mutate-existing issues real UPDATE calls against the API server for every matched target, and needs its own RBAC (an aggregated ClusterRole with update on the target kind). Use it for one-off retroactive rollouts, not as a continuous reconciler — that is what an operator is for.

Generate lifecycle: synchronize, clone, cloneList, and orphans

generate has more lifecycle nuance than §4 lets on:

• generateExisting: true makes the rule fire for triggers that already exist when the policy is applied, not only for new ones — so labeling a policy on cannot leave old namespaces without their default-deny.
• cloneList clones a set of source objects selected by label, instead of a single named object — the clean way to fan a family of secrets or configmaps into every namespace:

      generate:
        namespace: "{{ request.object.metadata.name }}"
        synchronize: true
        cloneList:
          namespace: platform
          kinds:
            - v1/Secret
          selector:
            matchLabels:
              distribute: "true"

• synchronize and orphans. With synchronize: true, deleting the policy or the trigger garbage-collects the generated children (they carry an owner reference back to the policy). With synchronize: false, the child is created once and then belongs to no one — edits and deletions stick, and you own the drift. Choose true for anything security-relevant.

verifyImages: required, caching, private registries, and m-of-n

Production verifyImages rules almost always need three things the §5 starter omits:

      verifyImages:
        - imageReferences: ["registry.internal/*"]
          required: true               # fail closed if NO signature data is present
          useCache: true               # cache verification results (default) to cut registry calls
          imageRegistryCredentials:
            secrets:
              - regcred                 # pull secret so Kyverno can read a private registry
          attestors:
            - count: 1                  # m-of-n: require 1 of the listed identities…
              entries:
                - keyless:
                    subject: "https://github.com/acme/*"
                    issuer: "https://token.actions.githubusercontent.com"

required: true is the difference between “reject images with a bad signature” and “reject images with no signature” — the latter is what you actually want. imageRegistryCredentials is mandatory for private registries or Kyverno cannot even fetch the signature artifact. And count against multiple entries gives you true m-of-n signing: require CI and a release approver, or any 2 of 3 named identities. When you also assert attestations, the predicate JSON is exposed as JMESPath variables (buildDefinition, predicate, metadata) that your conditions interrogate — that is how you enforce “built from repo X on runner type Y”, not merely “signed”.

JMESPath, $()/{{ }} variables, and the CEL direction

Kyverno’s dynamic power comes from variable substitution. {{ }} interpolates a JMESPath expression evaluated against a context that includes request (the AdmissionReview), images, element (inside foreach), and any context entries you declare. Those context entries can pull data at admission time:

      context:
        - name: allowed
          configMap:                       # read a ConfigMap
            name: registry-allowlist
            namespace: kyverno
        - name: nsinfo
          apiCall:                         # call the API server
            urlPath: "/api/v1/namespaces/{{ request.namespace }}"
            jmesPath: "metadata.labels"

For lookups that would otherwise hammer the API server on every admission, a GlobalContextEntry (Kyverno 1.11+) caches an API or registry response on a refresh interval, and rules reference it by name — turning a per-request call into a per-minute one:

apiVersion: kyverno.io/v2alpha1
kind: GlobalContextEntry
metadata:
  name: deployments-count
spec:
  apiCall:
    urlPath: "/apis/apps/v1/deployments"
    refreshInterval: 1m

The larger trend: the Kubernetes project standardised on CEL (Common Expression Language) for its built-in ValidatingAdmissionPolicy, and Kyverno has followed. Since 1.11 you can drop CEL straight into a traditional rule:

      validate:
        cel:
          expressions:
            - expression: "object.spec.replicas >= 3"
              message: "Production Deployments need at least 3 replicas."

And newer Kyverno (1.14+) introduces first-class CEL-native policy CRDs — ValidatingPolicy and ImageValidatingPolicy — that mirror the upstream VAP shape and can even be offloaded to the API server’s own admission machinery for a subset of rules, cutting webhook latency to zero:

apiVersion: policies.kyverno.io/v1alpha1
kind: ValidatingPolicy
metadata:
  name: require-team-label
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods]
  validations:
    - expression: "has(object.metadata.labels) && 'team' in object.metadata.labels"
      message: "Every Pod must carry a team label."

You do not have to migrate — the YAML ClusterPolicy model remains fully supported — but knowing CEL is now table stakes for admission policy on Kubernetes generally, so it is worth learning here.

PolicyReports and background scans, in detail

The reports controller writes two flavours of result. Admission reports capture the verdict at the moment of a write; background reports come from the periodic re-scan of existing resources. Both aggregate into PolicyReport (namespaced) and ClusterPolicyReport (cluster-scoped) objects, which conform to the open Kubernetes Policy WG report schema — so tooling like Policy Reporter can render them independently of Kyverno. At scale the reports themselves cost memory; Kyverno exposes reportsConfig and chunking knobs to keep the reports controller from ballooning on a cluster with hundreds of thousands of resources. The operational rule stands: promote a policy to Enforce only when its fail count on the background scan is a number you have consciously accepted (usually zero, sometimes “zero except these known exceptions”).

Exceptions: waiving a rule without weakening it

Real clusters have legitimate exceptions — a legacy workload that cannot yet meet a rule. The wrong fix is to broaden the policy (which weakens it everywhere); the right fix is a scoped, auditable PolicyException:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: allow-legacy-registry
  namespace: legacy
spec:
  exceptions:
    - policyName: restrict-registries
      ruleNames:
        - allowed-registries
        - autogen-allowed-registries       # exempt the auto-generated Deployment/DaemonSet rule too
  match:
    any:
      - resources:
          kinds: [Pod]
          namespaces: [legacy]

An exception is a first-class object: it is namespaced, reviewable in a PR, and can be time-boxed by pairing it with a cleanup policy. Note the autogen- rule name — Kyverno auto-generates Pod-controller variants of Pod rules (for Deployment, DaemonSet, Job, …), and an exception must name those variants too or it will only cover bare Pods.

Cleanup policies: TTL without a controller

Kyverno’s fourth controller runs CleanupPolicy / ClusterCleanupPolicy — declarative, cron-scheduled garbage collection for resources Kubernetes will not clean up itself:

apiVersion: kyverno.io/v2
kind: ClusterCleanupPolicy
metadata:
  name: cleanup-completed-jobs
spec:
  match:
    any:
      - resources:
          kinds: [Job]
  conditions:
    all:
      - key: "{{ target.status.succeeded || `0` }}"
        operator: Equals
        value: 1
  schedule: "*/10 * * * *"

This is a genuinely different capability from the other three powers — it deletes on a schedule rather than gating a write — and it replaces a pile of bespoke CronJobs that used to prune completed Jobs, stale ephemeral namespaces, or expired test resources.

HA and performance tuning

The performance story in §8 is the beginning; at scale the levers are:

• Narrow the match. Every kind/operation your policies match is a kind/operation the API server forwards to the webhook. Match operations: [CREATE, UPDATE] (not DELETE/CONNECT) and specific kinds to shrink webhook traffic.
• resourceFilters. The Kyverno ConfigMap ships a resourceFilters list that globally excludes noisy or system objects (events, leases, kube-system) from admission before any policy runs — the cheapest possible optimisation.
• Right-size the controllers. The admission controller is CPU/latency-sensitive (run 3+ replicas, a PDB, and generous CPU); the background and reports controllers are memory-sensitive on large clusters (they hold resource state). They scale independently precisely because their pressure profiles differ.
• Cache aggressively. useCache on verifyImages and GlobalContextEntry for hot API lookups turn per-request work into per-interval work.
• Prefer offloaded CEL policies for simple, high-volume validations where the API server’s native VAP can do the job without a webhook round-trip at all.

Enterprise scenario

A fintech platform team I worked with had a hard control from their auditors: production workloads must run only images signed by the central CI identity, and every Pod must carry the owning team’s cost-center label. They had ~140 namespaces across three clusters and could not afford a big-bang cutover that risked blocking deploys org-wide.

The constraint that bit them was ordering and false negatives. Their first attempt enforced verifyImages cluster-wide on day one with failurePolicy: Fail. Within an hour a registry hiccup combined with the fail-closed webhook blocked every deploy across all 140 namespaces – including the platform team’s own fix. They rolled it back and rebuilt the rollout as a labeled opt-in.

The fix had three parts. First, a mutate rule defaulted the cost-center label from an existing namespace annotation, so teams that had set it once on the namespace never had to repeat it on every Pod – eliminating the most common validate failure before it happened. Second, the verifyImages policy was scoped by namespaceSelector so only namespaces labeled verify=enforce were gated; everything else ran in Audit and reported via PolicyReport. Third, they watched the cluster reports until failures hit zero per namespace, then flipped that namespace’s label.

    - name: default-cost-center-from-ns
      match:
        any:
          - resources:
              kinds: [Pod]
      context:
        - name: ns
          apiCall:
            urlPath: "/api/v1/namespaces/{{ request.namespace }}"
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(cost-center): "{{ ns.metadata.annotations.\"acme.io/cost-center\" || 'unassigned' }}"

The context.apiCall fetches the namespace object at admission so the rule can read its annotation; the + add anchor defaults the label only when absent. Per-namespace promotion meant the blast radius of any mistake was one team, not the company. Full enforcement across all 140 namespaces took three weeks of label flips, with zero deploy-blocking incidents after the rebuild.

Practice challenges

Work these in order — each builds a rule you would actually ship. Try before opening the solution; the one-line why matters more than the YAML.

Challenge 1 (beginner) — Default a restricted securityContext. Write a mutate ClusterPolicy that, on every Pod, sets runAsNonRoot: true and seccompProfile.type: RuntimeDefault at the Pod level, and on every container sets allowPrivilegeEscalation: false and drops all capabilities — but only when those fields are absent, never overwriting an explicit value.

<details> <summary>Solution</summary>

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: default-restricted-securitycontext
spec:
  rules:
    - name: add-default-securitycontext
      match:
        any:
          - resources:
              kinds: [Pod]
      mutate:
        patchStrategicMerge:
          spec:
            +(securityContext):
              +(runAsNonRoot): true
              +(seccompProfile):
                +(type): RuntimeDefault
            containers:
              - (name): "?*"
                +(securityContext):
                  +(allowPrivilegeEscalation): false
                  +(capabilities):
                    +(drop): [ALL]

Why: every field uses the add anchor +(), and (name): "?*" iterates all containers by merge key — so the rule defaults the restricted profile without ever clobbering a deliberate setting, turning a would-be Pod Security rejection into a silent fix. </details>

Challenge 2 (intermediate) — Generate a default-deny NetworkPolicy per namespace. Write a generate rule that creates a default-deny NetworkPolicy (deny all ingress and egress) in every namespace, reconciles it if edited or deleted, and excludes the system namespaces. Assume the background-controller RBAC label is already granted.

<details> <summary>Solution</summary>

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-netpol
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds: [Namespace]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, kube-node-lease, kube-public]
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{ request.object.metadata.name }}"
        synchronize: true
        data:
          spec:
            podSelector: {}
            policyTypes: [Ingress, Egress]

Why: synchronize: true is what makes it self-healing (the background controller restores drift), and the exclude block keeps Kyverno from locking down kube-system — add generateExisting: true if you also need it to backfill namespaces that predate the policy. </details>

Challenge 3 (advanced) — Verify signatures only in opted-in namespaces. Write a verifyImages rule that requires images from registry.internal/* to be keyless-signed by your GitHub Actions identity, pins the verified tag to a digest, runs fail-closed — but only in namespaces labeled verify: enforce.

<details> <summary>Solution</summary>

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-team-images
spec:
  validationFailureAction: Enforce
  webhookConfiguration:
    failurePolicy: Fail
  rules:
    - name: verify-signed
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaceSelector:
                matchLabels:
                  verify: enforce
      verifyImages:
        - imageReferences: ["registry.internal/*"]
          mutateDigest: true
          verifyDigest: true
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/acme/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Why: the namespaceSelector is the safety valve — it lets you roll signature enforcement out one labeled namespace at a time, so a registry outage plus failurePolicy: Fail can only ever block the namespaces you have consciously opted in, never the whole cluster. </details>

Challenge 4 (advanced) — Graduated enforcement in one policy. Take the resource-limits rule from §2 and make it Enforce everywhere except staging and dev, which should only Audit — without writing two policies.

<details> <summary>Solution</summary>

      validate:
        failureAction: Enforce
        failureActionOverrides:
          - action: Audit
            namespaces: [staging, dev]
        message: "CPU and memory limits are required on every container."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

Why: the per-rule validate.failureAction plus failureActionOverrides (Kyverno 1.10+) varies enforcement by namespace inside a single rule — the modern replacement for the spec-level validationFailureAction, and the clean way to keep dev fast while prod is strict. </details>

Common beginner mistakes

• Enforcing on day one. Applying a new policy with Enforce (or spec-level validationFailureAction: Enforce) straight away is how you block a deploy you did not anticipate. Right model: every policy starts in Audit, you read the PolicyReport background scan, and you promote only when fail is a number you accept. The cluster tells you your debt for free — use it.
• patchStrategicMerge without an add anchor — the clobber and the loop. Writing imagePullPolicy: IfNotPresent (no +) overwrites an explicit value the developer set; worse, a mutate that unconditionally sets a field it also matches on can re-trigger itself. Right model: default with +(field) so you only add when absent, which is both non-destructive and loop-safe.
• Generate without synchronize, then surprise at the drift. A generated NetworkPolicy created once with synchronize: false will not come back after someone deletes it — and people delete things. Right model: set synchronize: true for anything security-relevant so the background controller reconciles it, and add generateExisting: true if pre-existing namespaces must be backfilled.
• verifyImages with the wrong attestor — or no required. Two traps: a keyless subject/issuer that does not exactly match your CI’s OIDC identity silently fails every image; and omitting required: true means an image with no signature at all sails through (the rule only catches bad signatures). Right model: copy the exact issuer from your CI’s token, and set required: true to fail closed on unsigned images.
• Reaching for deny when a pattern fits (and vice-versa). Beginners write JMESPath deny.conditions to assert a field exists — which a one-line pattern with ?* does more readably — or try to express a comparison inside a pattern, which cannot do logic. Right model: pattern for shape, deny for logic (see the table in §2).
• Forgetting the autogen rules exist. Kyverno auto-expands a Pod rule to cover Deployment/DaemonSet/Job/etc. as autogen-<rule>. A PolicyException or a kyverno test result that names only the Pod rule will not match those variants. Right model: name the autogen- rule too, or scope the rule to controllers explicitly.
• Not excluding kube-system and kyverno. A broad Pod policy that matches everything can deadlock the cluster — Kyverno cannot restart because its own Pods hit its own webhook. Right model: always exclude the system namespaces (or use the ConfigMap resourceFilters) on any cluster-wide rule.

Verify

Confirm the whole pipeline end to end before declaring a policy live.

# 1. Policies are loaded and Ready
kubectl get clusterpolicy
# READY column should be true for each

# 2. A non-compliant Pod is rejected under Enforce
kubectl run bad --image=registry.internal/web:latest --dry-run=server
# Expect: admission webhook denied (latest tag / unsigned)

# 3. A mutate rule actually patched the object
kubectl get pod good -o jsonpath='{.spec.securityContext.runAsNonRoot}'
# Expect: true

# 4. Generated NetworkPolicy exists in a fresh namespace
kubectl create namespace probe
kubectl get networkpolicy -n probe default-deny
# Expect: the synchronized default-deny NetworkPolicy

# 5. Image verification rewrote the tag to a digest
kubectl get pod signed -o jsonpath='{.spec.containers[0].image}'
# Expect: registry.internal/web@sha256:...

# 6. Reports show the historical picture
kubectl get policyreport -A

If step 4 shows nothing, check background-controller RBAC and logs. If step 5 keeps the tag, mutateDigest is off or the digest could not be resolved.

Checklist

• Kyverno installed via pinned Helm chart; admission controller at 3+ replicas with a PodDisruptionBudget
• kube-system, kyverno, and other protected namespaces excluded from broad Pod policies
• Every new policy starts at validationFailureAction: Audit and is promoted only after PolicyReport failures reach zero
• Validate rules use the correct anchors (?*, =(), (), +()) and ship instruction-style failure messages
• Defaulting mutate rules supply sane values so validate rules rarely fail
• Generate rules for NetworkPolicies/Secrets have synchronize: true and the aggregate-to-background-controller RBAC label
• verifyImages policies set mutateDigest: true, run failurePolicy: Fail, and have a tuned webhookTimeoutSeconds
• Critical images additionally verify SLSA/SBOM attestations, not just signatures
• kyverno test and kyverno apply run in CI on every PR
• Enforcement rolled out by namespaceSelector label, one team at a time, never cluster-wide on day one

Glossary

• Admission control — the phase where the API server calls out to webhooks before persisting a write, so policy can mutate or reject the object. Kyverno lives here.
• ClusterPolicy / Policy — Kyverno’s policy custom resources. ClusterPolicy is cluster-scoped; Policy is namespaced. Each holds rules.
• Rule — one unit of policy: a match block plus exactly one of validate, mutate, generate, or verifyImages.
• match / exclude — the selector that decides which requests a rule sees (by kind, operation, namespace, label, or requesting identity). any = OR, all = AND.
• Pattern — a validate style: a resource fragment the target must structurally match, using anchors for optionality.
• Anchor — a marker in a pattern: ?* (any non-empty), =() (optional-but-if-present), () (conditional), X() (must-not-exist), +() (mutate: add if absent).
• deny + conditions — a validate style for logic patterns cannot express, evaluated with JMESPath or CEL.
• foreach — iterate a list (containers, volumes) and apply a check or patch to each element.
• Mutate — a rule that changes the incoming object (defaulting, labels, sidecars) before it is stored.
• Mutate-existing — a background mutation of resources that already exist, triggered by a policy update or another resource’s admission.
• Generate — a rule that creates companion resources (NetworkPolicy, Secret) in response to a trigger such as a new namespace.
• synchronize — on a generate rule, whether Kyverno reconciles the generated child if it is edited or deleted.
• clone / cloneList — generate by copying an existing source object (or a labeled set of them) instead of embedding the spec inline.
• verifyImages — a rule that checks an image’s cosign signature (and optional attestations) at admission.
• cosign — the Sigstore tool that signs and verifies OCI artifacts; the signature is stored alongside the image in the registry.
• Keyless signing — cosign signing that uses a short-lived certificate tied to an OIDC identity (Fulcio) and a transparency log (Rekor), instead of a long-lived private key.
• Attestor — an identity (a public key, or a keyless subject/issuer) that a signature must come from. count enables m-of-n.
• Attestation — signed metadata about an image, e.g. an SBOM or SLSA provenance predicate, that a rule can assert conditions against.
• SBOM / SLSA provenance — a software bill of materials (what is in the image) / a record of how and where the image was built.
• mutateDigest — rewrites a verified image tag to its immutable @sha256: digest, so the verified image is provably the one that runs.
• failurePolicy — what the API server does when the webhook is unreachable: Fail (reject, fail-closed) or Ignore (admit, fail-open).
• validationFailureAction / failureAction — whether a failed validation Enforces (rejects) or Audits (admits and records). Audit first, always.
• PolicyReport / ClusterPolicyReport — the objects that record per-resource pass/fail results, populated by admission and by background scans.
• Background scan — the periodic re-evaluation of existing resources against policies, which surfaces pre-existing violations that admission (only firing on writes) never would.
• PolicyException — a scoped, reviewable waiver that exempts specific resources from named rules without weakening the policy globally.
• CleanupPolicy / ClusterCleanupPolicy — cron-scheduled declarative deletion of matching resources; Kyverno’s fourth controller.
• CEL — Common Expression Language, the Kubernetes-standard expression language now usable inside Kyverno rules and in its newer ValidatingPolicy CRDs.
• GlobalContextEntry — a cached API/registry lookup, refreshed on an interval, that rules reference by name to avoid per-request API calls.

Related lessons

• Admission Controllers: Validating & Mutating Webhooks — the mechanism Kyverno plugs into; read this if the mutating-then-validating flow is fuzzy.
• OPA Gatekeeper: Policy-as-Code Admission Gating — the Rego-based alternative in the same admission slot; the comparison in §1 lives here in depth.
• Container Image Supply Chain: Cosign, SBOM & SLSA — the producing side of §5: how images get signed and how attestations are generated before Kyverno enforces them.