Containerization Lesson 43 of 113

Deploy Kyverno Policies to Enforce Image Signing, Resource Limits, and Pod Security

Jun 10, 2026 32 min read Advanced Vinod

Level: Advanced · Time: ~35 min · You’ll want first: a working feel for how the API server calls out to admission webhooks — see Admission controllers: validating & mutating webhooks.

In a nutshell

Think of your Kubernetes API server as the front door to a secure building, and every kubectl apply as someone trying to walk in carrying a workload. Kyverno is the security officer you post at that door — except the rulebook they check against is written in ordinary Kubernetes YAML, not a specialised programming language. When a manifest arrives, Kyverno reads it against your house rules and can do one of four things: wave it through (validate–pass), quietly fix it up before it enters (mutate — e.g. add the CPU/memory limits the author forgot), demand proof of identity before admitting the image (verifyImages — check the Cosign signature), or turn it away at the door (validate–deny). It can even furnish a new room the moment it appears (generate — drop a default-deny NetworkPolicy into every fresh namespace).

Those four verbs — validate, mutate, generate, verifyImages — are the whole engine, and you write them inside a ClusterPolicy, which is just a normal Kubernetes object. That is Kyverno’s headline difference from the other big policy engine, OPA Gatekeeper: with Gatekeeper you learn Rego, a dedicated policy language; with Kyverno you stay in the YAML you already know. The trade is raw expressiveness for approachability, and for the three controls in this lesson — signed images, resource limits, and Pod Security — YAML is more than enough.

The other idea to hold onto is audit versus enforce. A brand-new policy should first run in Audit mode: it watches, records every pass and fail in a report, and blocks nothing. Only once you have read the report and fixed the offenders do you flip it to Enforce, where a failing manifest is actually rejected. Skipping the audit step is the single most common way teams take their own cluster down — we will do it the safe way.

Why a beginner should care: without an admission gate, “our cluster only runs signed, limited, non-root workloads” is a hope propped up by code review and good intentions. With Kyverno it becomes a property the API server guarantees — the bad manifest is refused before it is ever written to etcd, every time, for everyone.

By the end of this lesson you will be able to:

A payments platform team gets the finding back from their first real supply-chain audit: anyone with kubectl apply can run :latest from an arbitrary public registry, half the pods have no CPU/memory limits so one bad deploy noisy-neighbours an entire node, and a third of workloads run as root with hostPath mounts. The CISO’s instruction is blunt — “nothing runs in production unless it is our signed image, it stays inside its limits, and it cannot get root on the node.” You can chase that with code review and good intentions, or you can make the cluster itself refuse the bad manifest at the API server. This guide does the latter with Kyverno, the Kubernetes-native policy engine, enforcing three controls as a single admission gate: image signature verification (Cosign), resource limits (mutate + validate), and the restricted Pod Security Standard. Every command below is real and runnable against any conformant cluster (AKS, EKS, GKE, or vanilla).

Prerequisites

Target topology

Deploy Kyverno Policies to Enforce Image Signing, Resource Limits, and Pod Security — topology

Kyverno installs as a set of controllers in the kyverno namespace and registers two webhooks with the API server: a ValidatingWebhookConfiguration (deny on policy violation) and a MutatingWebhookConfiguration (inject defaults, verify-and-rewrite image digests). Every CREATE/UPDATE of a Pod-bearing resource flows API server → Kyverno admission controller → your ClusterPolicy rules → allow / mutate / deny. A separate reports controller writes PolicyReport objects continuously so you have a posture view even for resources admitted before a policy existed. Three independent control planes feed in:

1. Install Kyverno

Install via the official Helm chart. Run admission in high availability (3 replicas) for any cluster that matters — a single Kyverno pod is a single point of admission failure.

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm install kyverno kyverno/kyverno \
  --namespace kyverno --create-namespace \
  --version 3.3.4 \
  --set admissionController.replicas=3 \
  --set backgroundController.replicas=2 \
  --set reportsController.replicas=2 \
  --set cleanupController.replicas=2

Wait for the controllers and confirm the webhooks registered:

kubectl -n kyverno rollout status deploy/kyverno-admission-controller
kubectl get pods -n kyverno
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep kyverno

Notice you just installed four deployments, not one. Since Kyverno 1.10 the controllers are split so each responsibility can scale and fail independently — this matters the moment you care about performance or HA:

Controller Job Why it is separate
admission Answers the webhook in the hot path of every matching CREATE/UPDATE It is latency-critical and must be HA — this is the one that can block your API server
background Re-scans existing resources for background: true policies, and runs generate/mutateExisting Runs off the hot path, on a schedule and via informers
reports Aggregates rule results into PolicyReport / ClusterPolicyReport objects Report writing is chatty; isolating it keeps admission fast
cleanup Runs CleanupPolicy TTL deletions of stale resources A janitor, unrelated to admission

Only the admission controller sits in the request path, which is why you gave it the most replicas. The two webhook configurations you just confirmed are the only wiring between Kubernetes and Kyverno: the API server calls out to them for every object that matches, and everything Kyverno does — allow, mutate, deny, verify — happens inside that call. If you want the mental model of how that callout works, it is the standard validating & mutating admission webhook mechanism; Kyverno is a generic, policy-driven implementation of it.

A critical safety setting before you write any policy: decide what happens if Kyverno itself is down. The default failurePolicy: Fail means admission requests are rejected when the webhook is unreachable — safe, but it can wedge a cluster. Set it deliberately per policy (below). Also confirm Kyverno excludes its own and system namespaces so you cannot deadlock the control plane:

kubectl get configmap kyverno -n kyverno -o jsonpath='{.data.webhooks}' ; echo
# Expect kube-system / kyverno excluded by namespaceSelector

2. Set up Cosign signing in CI

Image-signature enforcement is worthless if your own images are unsigned, so build the signing side first. Generate a key pair, or — preferred — use keyless signing where Cosign gets a short-lived certificate from Fulcio bound to your CI’s OIDC identity, leaving no long-lived key to leak.

Before the commands, hold the mental model, because “signing an image” confuses almost everyone at first. Cosign does not modify your image. It computes a signature over the image’s content-addressable digest (sha256:...) and pushes that signature alongside the image in the same registry as a separate artifact, then (by default) records the event in Rekor, a public append-only transparency log. Verification later re-derives the digest, finds the signature artifact, and checks it against either a public key (keyed) or a certificate identity issued by Fulcio for your CI (keyless). Keyless is preferred precisely because there is no private key sitting anywhere to be stolen — the “key” is your pipeline’s OIDC identity, and it exists only for the seconds the signing runs.

Key-based, with the private key stored in HashiCorp Vault (never in the repo or a plain CI secret):

# One-time: generate and push the public half to the registry/Git; private half to Vault
cosign generate-key-pair
vault kv put secret/cosign/payments cosign.key=@cosign.key password='<passphrase>'
shred -u cosign.key            # do not keep the private key on disk

The CI job pulls the key from Vault at build time and signs the digest (never a tag):

# .github/workflows/build-sign.yml  (GitHub Actions)
permissions:
  contents: read
  id-token: write          # required for keyless / Vault OIDC auth
jobs:
  build-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        id: build
        with: { push: true, tags: "ghcr.io/kloudvin/api:${{ github.sha }}" }
      - uses: sigstore/cosign-installer@v3
      # Option A — keyless (recommended): identity is the GitHub OIDC token
      - run: |
          cosign sign --yes \
            "ghcr.io/kloudvin/api@${{ steps.build.outputs.digest }}"
      # Option B — key from Vault:
      # - run: cosign sign --yes --key "hashivault://payments/cosign" \
      #     "ghcr.io/kloudvin/api@${{ steps.build.outputs.digest }}"

Signing the digest and not the tag is not a style choice — a tag is a mutable pointer, so a signature “on a tag” secures nothing once someone re-pushes it. Signing @sha256:... binds the signature to exact bytes.

Verify locally so you know the exact identity strings the cluster policy must match:

cosign verify \
  --certificate-identity-regexp "https://github.com/kloudvin/.+" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/kloudvin/api@<digest> | jq '.[0].optional.Subject'

Copy those two strings down verbatim. The cluster policy in the next step must match the subject (the certificate identity, e.g. the workflow ref) and the issuer (https://token.actions.githubusercontent.com) exactly — a stray character in either silently fails every verification, and it is the single most common reason a working signing pipeline still gets its pods blocked.

3. Enforce image signatures with verifyImages

Now the gate. This ClusterPolicy uses Kyverno’s verifyImages rule to require a valid Cosign signature for any image from your registry. Start in Audit so you can see the blast radius before you block anything.

# policies/verify-images.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
  annotations:
    policies.kyverno.io/severity: high
spec:
  validationFailureAction: Audit        # flip to Enforce in step 7
  failurePolicy: Fail
  webhookTimeoutSeconds: 30             # signature checks are slower than plain validation
  background: false                     # verifyImages cannot run as a background scan
  rules:
    - name: verify-ghcr-cosign-keyless
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "ghcr.io/kloudvin/*"      # only OUR registry; pin public ones separately
          failureAction: Audit
          mutateDigest: true            # rewrite the verified tag to an immutable @sha256
          required: true
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/kloudvin/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Read the rule the way Kyverno does. imageReferences scopes it to your registry only — a wildcard * here would try to verify every public sidecar and fail the lot, so keep third-party images to their own, separately-pinned policy. attestors is the list of parties whose signature you will accept, and count: 1 means “at least one of the entries must match.” A keyless entry says “accept a Fulcio certificate whose subject and issuer match these” — the two strings you copied in step 2. required: true (the default) means an image with no signature at all is a failure, not a skip.

Two per-rule fields carry the safety of this policy. failureAction: Audit is the modern, per-verifyImages control for what happens on a mismatch; it supersedes the spec-level validationFailureAction (which primarily governs validate rules and is being deprecated) — when both appear, prefer the per-rule one and keep them in step. And background: false is not optional here: signature verification has to reach a registry and Rekor, which the background scanner cannot do, so verifyImages runs only at admission.

If you signed with a Vault/KMS key instead of keyless, swap the attestor entry for the public key:

              entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQ...your cosign.pub...
                      -----END PUBLIC KEY-----
                    rekor:
                      url: https://rekor.sigstore.dev

Apply it and watch the reports:

kubectl apply -f policies/verify-images.yaml
kubectl get clusterpolicy require-signed-images
kubectl get policyreport -A | head        # PASS/FAIL counts per namespace

mutateDigest: true is doing quiet, important work: once verified, Kyverno rewrites :tag to the pinned @sha256:... digest in the pod spec, so what runs is provably the bytes you signed — closing the tag-mutation window where an attacker re-pushes a tag after verification.

4. Mutate in default resource limits

A pod with no limits can starve a node. Use a mutate rule to inject sane defaults when the author omits them — non-destructive, and far better adoption than rejecting every under-specified deployment on day one.

# policies/default-resources.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-resources
spec:
  rules:
    - name: set-default-requests-limits
      match:
        any:
          - resources:
              kinds: [Pod]
      mutate:
        foreach:
          - list: "request.object.spec.containers"
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    resources:
                      requests:
                        +(memory): "128Mi"     # +(...) = add only if absent
                        +(cpu): "100m"
                      limits:
                        +(memory): "512Mi"
                        +(cpu): "500m"

Two pieces of Kyverno syntax do the work here. foreach with list: "request.object.spec.containers" walks each container in turn, exposing it as {{ element }} — that is how one rule patches every container by name rather than only the first. The +(...) anchor is the crucial one: +(memory) is an add-if-absent anchor, so Kyverno writes the default only when the field is missing and never clobbers a value the author set deliberately. (Kyverno has a small family of these anchors — () conditional, +() add-if-absent, X() “must not exist” — worth knowing when you read other people’s policies.)

5. Require resource limits with validate

Defaulting is a safety net, not a rule. Pair it with a validate rule so a container that explicitly omits limits in a namespace you care about is rejected outright — defence in depth against someone setting limits: null.

# policies/require-limits.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-cpu-mem-limits
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "CPU and memory limits are required on every container."
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                any:
                  - key: "{{ element.resources.limits.memory || '' }}"
                    operator: Equals
                    value: ""
                  - key: "{{ element.resources.limits.cpu || '' }}"
                    operator: Equals
                    value: ""

The deny.conditions block is a small boolean expression: any means “deny if any of these is true.” The {{ element.resources.limits.memory || '' }} is JMESPath — Kyverno’s expression language — where || '' supplies an empty-string default if the field is missing, so a container with no memory limit evaluates to "" Equals "" → true → denied. Note this rule sets background: true (unlike verifyImages), which means the reports controller can also flag already-running pods that violate it, not just new admissions.

Order matters: Kyverno runs mutate rules before validate, so the step-4 defaults are applied first and only a container that cannot be defaulted (e.g. an explicit null) trips this deny.

6. Enforce restricted Pod Security

Replace the deprecated PodSecurityPolicy with Kyverno’s podSecurity subrule, which maps directly to the upstream Pod Security Standards. This single rule enforces the entire restricted profile — no root, no privilege escalation, dropped capabilities, seccomp, no host namespaces.

# policies/pod-security-restricted.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pod-security-restricted
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: restricted-profile
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        podSecurity:
          level: restricted
          version: latest
          # Targeted, auditable exemptions instead of a blanket opt-out:
          exclude:
            - controlName: "Capabilities"
              images: ["ghcr.io/kloudvin/net-tools:*"]

The Pod Security Standards define three cumulative levels — privileged (anything goes), baseline (blocks the well-known escapes), and restricted (hardened current best practice: runAsNonRoot, allowPrivilegeEscalation: false, drop: ["ALL"], a seccompProfile, no host namespaces). The podSecurity subrule enforces a whole level in one line rather than hand-writing a dozen deny conditions, and — unlike the built-in Pod Security Admission — its exclude lets you relax one named control for one set of images while every other control stays enforced. Here only the net-tools image may keep extra capabilities; it still may not run as root or escape its namespace.

Why Kyverno over the built-in Pod Security Admission: PSA only operates per-namespace at fixed levels and cannot make exceptions, mutate, or report centrally. Kyverno gives you per-image exemptions, the same PolicyReport stream as your other controls, and a single place security reviews. Apply all the policies through Argo CD rather than kubectl in production so the policy set is the Git-tracked source of truth:

kubectl apply -f policies/        # or sync the Argo CD Application
kubectl get cpol                  # all four ClusterPolicies, READY=true

7. Promote from Audit to Enforce

Never go straight to Enforce on a live cluster. Run in Audit, read the reports, fix the offenders, then flip. Find what would be blocked:

# Aggregate failing rules across the cluster
kubectl get policyreport -A -o json \
  | jq -r '.items[].results[] | select(.result=="fail")
      | "\(.policy)/\(.rule)\t\(.resources[0].namespace)/\(.resources[0].name)"' \
  | sort | uniq -c | sort -rn

Once the failures are down to known exemptions, flip each policy to enforcing:

kubectl patch clusterpolicy require-signed-images \
  --type merge -p '{"spec":{"validationFailureAction":"Enforce"}}'
# repeat for the verifyImages rule's own failureAction: Enforce

validationFailureAction (and its per-rule successor failureAction) is the single most important field for a safe rollout, and it has exactly two settings that matter: Audit records a result and admits the resource; Enforce rejects it. The discipline is always the same loop — apply in Audit, watch PolicyReport failures fall to a known set of intended exemptions, then patch to Enforce. There is no reason to ever start at Enforce, and every reason not to.

For high-risk control-plane namespaces, keep failurePolicy: Fail; for application namespaces during rollout, Ignore avoids an outage if Kyverno blips. Make that choice consciously, per policy.

Going deeper

You have four working policies. This section is for the engineer who has to run Kyverno in production — the webhook internals, the scale knobs, the verification depth, and the “which tool” decisions the happy path glosses over.

The admission-webhook architecture: failurePolicy, timeouts, and matchConditions

Everything Kyverno does happens inside a webhook callout from the API server, and three fields govern that callout’s behaviour under stress. failurePolicy decides what happens when Kyverno is unreachable (crashed, mid-rollout, network blip): Fail rejects the request (fail-closed, secure), Ignore admits it (fail-open, available). webhookTimeoutSeconds caps how long the API server waits for an answer — the default is 10 and the hard maximum is 30, which is exactly why the verifyImages policy in step 3 sets 30: a signature check must round-trip to a registry and Rekor, and a slow registry behind failurePolicy: Fail is an outage waiting to happen.

The dangerous interaction is Fail + an under-provisioned or matched-too-broadly webhook. If Kyverno’s own pods are being rescheduled and a fail-closed policy matches their namespace, the API server can refuse to admit the very pods that would bring Kyverno back — a self-inflicted deadlock. Kyverno ships with kube-system and kyverno excluded by namespaceSelector for this reason; never remove those exclusions. Beyond that, tighten what the webhook is even invoked for, so you are not paying admission latency on objects you do not police:

# Only invoke the webhook for pods that actually carry your app label
spec:
  webhookConfiguration:
    matchConditions:
      - name: only-labelled
        expression: "has(object.metadata.labels) && 'app' in object.metadata.labels"

matchConditions are CEL expressions evaluated by the API server before it ever calls Kyverno, so a non-matching object skips the network hop entirely — the cheapest possible way to shrink your admission blast radius and latency.

Background scans and PolicyReports

A validate policy with background: true is evaluated in two places: at admission (new/changed objects) and on a schedule by the reports controller, which re-scans everything already running. That second path is what gives you posture on workloads admitted before the policy existed. Results land in two CRDs from the open PolicyReport working group: PolicyReport (namespaced, one per namespace) and ClusterPolicyReport (cluster-scoped resources). They are your queryable source of truth:

# Every failing result, grouped, without opening a single manifest
kubectl get policyreport -A -o json \
  | jq -r '.items[].results[] | select(.result=="fail") | .policy' \
  | sort | uniq -c | sort -rn

The catch you already met in step 3: verifyImages and mutate rules cannot background-scan — signature checks need live registry access and mutation only makes sense at write time — so those policies run background: false and only ever see new admissions. Existing pods keep their unverified images until you re-roll them. Treat “policy applied” and “fleet compliant” as two different states, and roll deployments deliberately after enabling a verify policy.

Image verification in depth: keyed, keyless, and attestations

The attestors/entries structure is more expressive than the single-keyless example suggests. count: N under an attestor set means “at least N of these entries must match,” so you can require, say, two independent signatures — one from CI, one from a release approver — before an image is admitted (dual control for production). Each entry is either keys (a static public key, from Vault/KMS) or keyless (a Fulcio certificate identity), and both can pin rekor.url so verification also insists the signing event was logged in the transparency log.

Signatures prove who built it; attestations prove facts about it. Cosign can attach signed predicates — an SBOM, SLSA provenance, a passing vulnerability scan — and Kyverno can require them under verifyImages[].attestations, checking the predicate content with conditions:

      verifyImages:
        - imageReferences: ["ghcr.io/kloudvin/*"]
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/kloudvin/*"
                    issuer: "https://token.actions.githubusercontent.com"
          attestations:
            - type: https://slsa.dev/provenance/v1
              attestors:
                - count: 1
                  entries:
                    - keyless:
                        subject: "https://github.com/kloudvin/*"
                        issuer: "https://token.actions.githubusercontent.com"
              conditions:
                - all:
                    - key: "{{ buildDefinition.externalParameters.workflow.ref || '' }}"
                      operator: Equals
                      value: "refs/heads/main"

That policy admits an image only if it carries SLSA provenance, signed by your CI, asserting it was built from main — supply-chain enforcement well past “is it signed.”

Mutate and generate beyond defaults — and synchronize

mutate is not just for resource defaults: common production uses include injecting imagePullSecrets, adding standardised labels/annotations for cost allocation, or setting automountServiceAccountToken: false by default. mutateExisting can even patch resources already in the cluster when a trigger fires. generate goes further — it creates new objects in response to another object appearing, the classic case being “every new namespace gets a default-deny NetworkPolicy and a copy of the registry pull-secret.” The field that makes generate trustworthy is synchronize: true: with it, Kyverno’s background controller reconciles the generated object, restoring it if someone deletes or edits it, and cleaning it up if the source policy or trigger goes away. clone copies from a live source object (right for distributing a shared secret/CA bundle), while data embeds the resource inline. These mutate/generate patterns are a lesson in themselves; treat this as the pointer and go deep separately.

CEL vs Kyverno syntax (and ValidatingAdmissionPolicy)

There are now three ways to express admission policy on a modern cluster, and they are not competitors so much as a spectrum:

Kyverno vs Gatekeeper vs Pod Security Admission — when each

Pod Security Admission OPA Gatekeeper Kyverno
Language none (namespace labels) Rego Kubernetes YAML (+ optional CEL)
Scope Pod security only anything (general policy) anything (general policy)
Mutate / generate no mutation (limited); no generate yes, both
Image signature verify no via external data / not native native verifyImages
Exceptions no (namespace level only) selectors / referential exclude + PolicyException CRD
Reporting audit annotations only constraint status / audit PolicyReport / ClusterPolicyReport
Learning curve trivial steep (Rego) moderate (YAML)

The decision rule: PSA is the floor — free, built-in, dependency-free, so keep it enforcing baseline/restricted at the namespace level as a backstop that survives a Kyverno outage (see Pod Security Admission migration). Reach for Kyverno when you need mutation, generation, image verification, per-image exceptions, or central reporting — which is most real programs. Choose Gatekeeper if your team already lives in Rego or shares OPA policy across non-Kubernetes systems; its expressiveness is unmatched but the Rego learning curve is real. Running PSA plus one policy engine is not redundant — PSA is the guarantee, the engine is the expressive layer on top.

Policy exceptions instead of weakening the policy

When one workload legitimately cannot satisfy a rule, do not loosen the rule for everyone. Kyverno’s PolicyException is a separate, RBAC-controllable object that exempts a named resource from a named rule, leaving the policy itself strict and the exception greppable and auditable:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: allow-legacy-caps
  namespace: legacy-app
spec:
  exceptions:
    - policyName: pod-security-restricted
      ruleNames: ["restricted-profile"]
  match:
    any:
      - resources:
          kinds: ["Pod"]
          namespaces: ["legacy-app"]
          names: ["legacy-migration-*"]

This is far better than editing pod-security-restricted to add a namespace to its exclude: the exception is a distinct object you can put behind its own RBAC, expiry review, and Git history, so “who is exempt from what, and who approved it” is a query, not archaeology. Exceptions are enabled by default in recent Kyverno; on locked-down installs confirm the admission controller’s --enablePolicyException flag (and the namespace it watches) is set.

Performance and HA

The admission controller is in the synchronous path of every matching request, so its health is your API server’s health for those objects. The production posture: run 3+ admission replicas with a PodDisruptionBudget so a node drain never leaves the webhook unbacked; keep failurePolicy: Fail for security-critical policies but earn the right to by ensuring the webhook is genuinely never down; and shrink the matched set with tight match blocks and matchConditions so you are not evaluating policy on irrelevant objects. Watch admission latency (Kyverno exports Prometheus metrics) — a verify policy’s registry/Rekor round-trip is the usual culprit, mitigated with a generous webhookTimeoutSeconds and an in-cluster registry mirror. The background and reports controllers use leader election, so their extra replicas are for failover, not throughput. One more autogen subtlety worth knowing: when you match on kinds: [Pod], Kyverno automatically synthesises equivalent rules for the pod controllers (Deployment, StatefulSet, DaemonSet, Job, CronJob) so authors get the error at kubectl apply on the Deployment, not later on the Pod — controlled by the pod-policies.kyverno.io/autogen-controllers annotation if you ever need to narrow it.

Validation

Prove the gate works with a deliberately bad pod — every one of these must be rejected once policies are enforcing:

# 1. Unsigned / wrong-registry image -> blocked by verifyImages
kubectl run bad-unsigned --image=nginx:latest
# Error: ... require-signed-images: image is not signed

# 2. Signed image with NO limits -> defaulted by mutate, or denied if null
kubectl run noreq --image=ghcr.io/kloudvin/api@<digest> --dry-run=server -o yaml \
  | grep -A4 resources                      # see injected requests/limits

# 3. Root / privileged pod -> blocked by restricted profile
kubectl run rooty --image=ghcr.io/kloudvin/api@<digest> \
  --privileged --dry-run=server
# Error: ... pod-security-restricted: privileged containers are not allowed

# 4. A correctly signed, limited, non-root pod -> ADMITTED
kubectl apply -f tests/good-pod.yaml        # should succeed

Confirm the digest rewrite actually happened on the admitted pod:

kubectl get pod good-pod -o jsonpath='{.spec.containers[0].image}'; echo
# Expect ghcr.io/kloudvin/api@sha256:...  (a digest, not a tag)

Run Kyverno’s own test harness in CI so policy changes are unit-tested before they ship via Argo CD:

kyverno test ./policies/          # asserts expected pass/fail per fixture

Rollback / teardown

Policies are declarative, so rollback is fast — switch back to Audit first if a policy is over-blocking in production, then remove if needed:

# Soft rollback: stop denying, keep reporting
for p in require-signed-images require-resource-limits pod-security-restricted; do
  kubectl patch cpol "$p" --type merge -p '{"spec":{"validationFailureAction":"Audit"}}'
done

# Remove a single policy
kubectl delete clusterpolicy pod-security-restricted

# Full uninstall (also removes both webhooks, so admission stops gating)
helm uninstall kyverno -n kyverno
kubectl delete ns kyverno

If you delivered policies via Argo CD, do the rollback in Git (revert the commit) and let the sync remove them — never kubectl delete out of band, or Argo will flag drift and may re-create them.

Practice challenges

Work these top to bottom — they escalate from “read the report” to “design the rollout.” Try each before opening the solution.

1. Beginner — see what would be blocked without blocking anything. You just applied require-resource-limits in Audit. Write the one pipeline that lists which policy/rule is failing and how many resources trip it, using only PolicyReports.

<details> <summary>Solution</summary>

kubectl get policyreport -A -o json \
  | jq -r '.items[].results[] | select(.result=="fail") | .policy + "/" + .rule' \
  | sort | uniq -c | sort -rn

Why: Audit mode admits everything but records each result in a PolicyReport; the reports are the blast-radius preview you read before flipping to Enforce. No cluster change, full visibility. </details>

2. Beginner — why is my “Enforce” policy letting everything through? A colleague swears their validate policy is enforcing, yet bad pods still get in. Name the two most likely one-line causes and the field to check.

<details> <summary>Solution</summary>

Either the policy is still validationFailureAction: Audit (records but never rejects), or it uses the newer per-rule form and the rule’s validate.failureAction is Audit while the spec-level field looks like Enforce. Check the effective action: kubectl get cpol <name> -o yaml | grep -i -E 'failureAction|validationFailureAction'. Why: Audit vs Enforce is the behavioural switch; a policy can be perfectly correct and simply not be denying because it was never promoted. </details>

3. Intermediate — default a label without overwriting it. Write a mutate rule that adds team: unassigned to a Pod’s labels only if no team label exists, leaving any author-set value untouched.

<details> <summary>Solution</summary>

      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(team): "unassigned"

Why: the +(...) add-if-absent anchor is the whole trick — +(team) writes the default only when the key is missing, so a pod that already declares team: payments is left alone. Overwriting deliberate values is the classic mutate footgun. </details>

4. Intermediate — pin the two identity strings. Your keyless verify policy blocks a correctly-signed image. Which two keyless fields must match the Cosign certificate exactly, and what command produces their expected values?

<details> <summary>Solution</summary>

subject (the certificate identity, e.g. https://github.com/kloudvin/...) and issuer (https://token.actions.githubusercontent.com). Get the truth from the image itself: cosign verify --certificate-identity-regexp ... --certificate-oidc-issuer ... <image@digest> | jq '.[0].optional'. Why: keyless verification is an exact string match on the Fulcio cert; a stray branch in the subject or a wrong issuer URL silently fails every verify, and it is the most common “signing works but pods are blocked” cause. </details>

5. Advanced — exempt one workload without weakening the policy. A legacy-migration-* Deployment in namespace legacy-app genuinely needs an extra Linux capability and cannot pass restricted. Grant it an exception without editing pod-security-restricted or relaxing it for anyone else.

<details> <summary>Solution</summary>

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: allow-legacy-caps
  namespace: legacy-app
spec:
  exceptions:
    - policyName: pod-security-restricted
      ruleNames: ["restricted-profile"]
  match:
    any:
      - resources:
          kinds: ["Pod"]
          namespaces: ["legacy-app"]
          names: ["legacy-migration-*"]

Why: a PolicyException is a separate, RBAC-controllable, Git-tracked object scoped to named resources, so the strict policy stays strict for everyone else and “who is exempt and who approved it” is a query — far better than adding a namespace to the policy’s exclude. </details>

6. Advanced — design the safe rollout and pick failurePolicy. You are enabling require-signed-images (verifyImages) across 140 namespaces. Give the ordered rollout and justify the failurePolicy choice so a registry blip cannot wedge the fleet.

<details> <summary>Solution</summary>

  1. Apply with failureAction: Audit and background: false; collect PolicyReport fails for a full sprint. 2. Fix/exempt offenders (re-roll deployments so existing pods are re-verified — verifyImages does not background-scan). 3. Promote to Enforce namespace by namespace, not fleet-wide in one patch. 4. Keep failurePolicy: Fail for the guarantee, but earn it: 3+ admission replicas, a PodDisruptionBudget, a generous webhookTimeoutSeconds (30) and an in-cluster registry mirror so a slow registry never combines with fail-closed into an outage. Why: the real-world failure is a registry hiccup plus fail-closed plus a broad match rejecting every deploy at once — including your own fix; a staged, well-provisioned rollout removes each ingredient. </details>

Common beginner mistakes

These are misconceptions — the wrong mental model, not just a wrong command. Fix the model and the commands follow.

Common pitfalls

Security notes

This is a Zero-Trust admission control: the cluster trusts no image it cannot cryptographically tie to your CI identity, runs nothing as root, and pins every workload to a signed digest. Keep the Cosign private key in HashiCorp Vault or use keyless signing so there is no long-lived secret to steal; rotate the key and update the policy’s public key together. Feed every PolicyReport to Wiz (to correlate admission posture with cloud misconfig and attack paths) and your SIEM, and let a hard denial open a ServiceNow incident so security gets a ticket, not just a log line. Remember the boundary: Kyverno gates admission — CrowdStrike Falcon on the nodes covers runtime (a compromise after a pod is admitted), and the two together close the gap.

Cost notes

Kyverno’s own footprint is small — the HA controllers run comfortably in roughly 0.5 vCPU / 512Mi per replica, a rounding error against the workloads they protect. The real saving is indirect: the step-4/5 resource-limit policies stop unbounded pods from triggering node autoscale events and cluster overprovisioning, which is usually a far larger line item than the controller. Watch one operational cost — verifyImages adds a Rekor/registry round-trip per new image, so size webhookTimeoutSeconds (step 3) generously and run an in-cluster registry mirror if your image pull volume is high, both to cut latency and to avoid public-registry rate limits.

Glossary

KubernetesKyvernoCosignPod SecuritySupply ChainAdmission Control
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me
VH

Written by Vinod

Senior Cloud Architect with 22+ years of experience. Sharing practical knowledge from production environments.

Comments