Policy-as-Code Guardrails with OPA Gatekeeper: Constraint Templates, Mutation, and CI Gating

Every cluster eventually accumulates a folklore of rules nobody enforces: “always set resource limits,” “only pull from our registry,” “tag everything with a cost-center.” These live in wikis and code review comments until the day a Deployment with no limits OOM-kills a node, or an unscanned image from Docker Hub lands in production. Guardrails that depend on human vigilance are not guardrails — they are suggestions.

OPA Gatekeeper turns those suggestions into admission-time controls. It plugs Open Policy Agent into the Kubernetes API server as a validating (and mutating) webhook, so a non-compliant object is rejected before it is persisted to etcd. This article builds a real guardrail program: ConstraintTemplates in Rego, parameterized Constraints, mutation defaults, safe staged rollout with dryrun, referential checks against synced data, and — critically — the same policies running in CI so violations surface on a pull request instead of at kubectl apply.

In a nutshell

OPA Gatekeeper is a policy engine for your cluster’s front door. Every time someone runs kubectl apply (or a controller creates an object), the API server can pause and ask an external opinion: is this object allowed? Gatekeeper is that opinion. It evaluates rules — written in a small language called Rego — and either lets the object through, admits it with a warning, or rejects it outright. Because the check happens at admission time, a bad object never reaches the cluster’s database in the first place.

You do not hand Gatekeeper raw Rego. You work with two objects, and the whole system clicks once you separate them:

• A ConstraintTemplate is the rule logic — “objects must carry these labels,” expressed once in Rego, with a small schema describing what can be parameterized. Applying a template generates a new resource type (a CRD) named after it.
• A Constraint is an instance of that generated type. It says where the rule applies (which kinds, which namespaces) and with what values (which labels, which registries). It contains no Rego at all — it is pure configuration.

A useful analogy is airport security. The ConstraintTemplate is the rulebook the security agency writes once — “no liquids over this limit” — as a reusable, parameterized procedure. A Constraint is the sign posted at a specific checkpoint that switches that rule on for a specific lane, with the numbers filled in (“100 ml here; staff lane exempt”). Gatekeeper is the scanner at the gate: every bag (kubectl apply) is checked against the active signs before it reaches the plane (before it is written to etcd). And the audit controller is the guard who periodically walks the terminal re-checking bags that were already let through — in case a rule was posted after they passed.

Level: Advanced · Time: ~30 min · this lesson ramps from the basics, so a careful beginner can follow it end to end while an experienced platform engineer still finds depth in the “Going deeper” section.

The diagram below is the whole mental model on one page — keep it in view as you read; every section fills in one of these boxes.

Prerequisites & what you’ll be able to do

Know this first. You should be comfortable reading Kubernetes YAML (Deployment, Pod, Namespace) and driving a cluster with kubectl. Two companion lessons make everything here land faster: Admission controllers: validating & mutating webhooks explains the exact point in the request path where Gatekeeper plugs in, and CRDs, operators & the controller pattern explains why a ConstraintTemplate can generate a new resource kind — the single most confusing idea for newcomers. You do not need a running cluster to follow the reasoning: every manifest and command here is real and current for Gatekeeper 3.16.x, and any output is labelled representative.

After this you can:

• Explain the split between OPA, Gatekeeper, ConstraintTemplates, and Constraints — and why the two-tier design exists.
• Author a ConstraintTemplate in Rego with a typed parameter schema, and declare parameterized Constraints against it.
• Roll a policy out safely with the dryrun → warn → deny path, reading the audit report to size the blast radius first.
• Default missing fields with mutation (Assign, ModifySet) instead of only rejecting.
• Write referential policies that reason about other objects via synced data.inventory.
• Run the same intent in CI with Conftest, and unit/integration-test policies with opa test and gator.
• Choose deliberately between Gatekeeper and Kyverno for a given requirement.

1. Architecture: webhook, constraint framework, and audit

First, the relationship that trips up newcomers. OPA (Open Policy Agent) is a general-purpose policy engine: give it some JSON input and a Rego rule, and it tells you whether the input satisfies the rule. It knows nothing about Kubernetes. Gatekeeper is the Kubernetes-native packaging around OPA: it wires OPA into the API server as a webhook, gives you Kubernetes-shaped CRDs to author policies (so kubectl get works on your policies like any other object), and adds a background auditor. In short — OPA is the brain; Gatekeeper is the body that puts the brain in the right place at the right time.

Gatekeeper has three moving parts, and understanding the split prevents most production surprises.

The admission webhook. Gatekeeper registers a ValidatingWebhookConfiguration (and a MutatingWebhookConfiguration). On every CREATE/UPDATE for matched resources, the API server calls Gatekeeper synchronously. Gatekeeper evaluates the request against all active Constraints and returns allow/deny. Because this is in the critical path of every write, two settings matter enormously: failurePolicy and timeoutSeconds. We tune those in section 4.

The constraint framework. You do not write raw Rego against the webhook. You write a ConstraintTemplate — Rego plus a CRD schema — which generates a new custom resource kind. You then create Constraints (instances of that kind) that say “apply this logic to these resources with these parameters.” This two-tier design is the whole point: platform engineers author templates once; application teams (or you) declare cheap, declarative Constraints without touching Rego.

Audit. A background controller periodically re-evaluates existing cluster objects against all Constraints and writes results to each Constraint’s status.violations. This catches resources that predate a policy, or that were admitted while a Constraint was in dryrun. Audit is how you measure blast radius before flipping to enforce.

It helps to see the division of labour on one line each:

The generated-CRD step is the concrete magic worth stating plainly: when you apply a ConstraintTemplate whose spec.crd.spec.names.kind is K8sRequiredLabels, Gatekeeper creates a new CRD so that K8sRequiredLabels becomes a first-class resource kind in your cluster. From then on kubectl get k8srequiredlabels works, and every object of that kind you create is a Constraint that Gatekeeper enforces. You are, in effect, generating a tiny purpose-built API for each policy.

Install with the released manifest (pin the version — never track latest for an admission controller):

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.16.3/deploy/gatekeeper.yaml
kubectl -n gatekeeper-system rollout status deploy/gatekeeper-controller-manager
kubectl get crd | grep gatekeeper

2. Authoring a ConstraintTemplate in Rego

Start with the canonical guardrail: required labels. The template defines the Rego logic and the parameter schema that becomes the Constraint CRD.

# templates/k8srequiredlabels.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels      # this becomes the Constraint kind
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("missing required labels: %v", [missing])
        }

The contract is fixed and worth memorizing: the rule must be named violation, it returns a set of objects with a msg (string) and optional details, and a non-empty set means “reject.” The admission payload is at input.review.object; parameters from the Constraint are at input.parameters. Set arithmetic (required - provided) is idiomatic Rego — far cleaner than iterating.

If you have never read Rego, walk the rule body line by line — every line is an implicit AND, and the head is emitted only when all of them hold:

• violation[{"msg": msg, ...}] { — a partial set rule. Its head is one element of the violation set; the body decides whether (and how many times) that element gets added. Zero elements means “allowed.”
• provided := {label | input.review.object.metadata.labels[label]} — a set comprehension. Iterating a map with [label] binds label to each key, so provided becomes the set of label names actually present on the incoming object.
• required := {label | label := input.parameters.labels[_]} — the same idea over the parameter array; [_] iterates every element. required is the set of label names the Constraint demands.
• missing := required - provided — plain set difference: what was required but not present.
• count(missing) > 0 — a guard. If the set is empty the expression is false, the body fails, and no violation is produced (i.e. the object is compliant).
• msg := sprintf("missing required labels: %v", [missing]) — build the human-readable message the user will see in the admission error. Because the head references msg and missing, both must be bound for the head to be valid.

A common mistake is naming the rule deny. That is the Conftest convention (section 7), not Gatekeeper. Gatekeeper only collects violation. Mixing them silently disables enforcement.

3. Parameterized Constraints: required labels, registries, resource limits

With the template applied, a Constraint is pure declaration. No Rego, just scope and parameters:

# constraints/require-owner-label.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-owner-and-costcenter
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]
  parameters:
    labels: ["owner", "cost-center"]

The match block is your scoping surface: kinds, namespaces, excludedNamespaces, labelSelector, and namespaceSelector. Always exclude system namespaces — locking kube-system out of mutating its own pods is a self-inflicted outage.

The match block deserves a reference, because getting it wrong is the difference between a targeted guardrail and a cluster-wide outage:

An empty match means everything — a foot-gun. Prefer the narrowest scope that satisfies the policy, and put system namespaces in excludedNamespaces on day one.

Two more guardrails platform teams ship on day one. A registry allowlist keeps images on your trusted path:

# templates/k8sallowedrepos.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          satisfied := [good | repo := input.parameters.repos[_]; good := startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("container image %v is not from an allowed registry: %v", [container.image, input.parameters.repos])
        }

For resource limits, lean on Gatekeeper’s maintained library rather than hand-rolling. The community ships a battle-tested K8sContainerLimits template (and many others) at github.com/open-policy-agent/gatekeeper-library. Vendoring proven templates beats reinventing CPU/memory parsing in Rego, which is a notorious source of off-by-one bugs around binary vs. decimal suffixes.

4. Safe rollout: enforcementAction dryrun, warn, and audit

Never ship a new Constraint straight to deny in a busy cluster — you will discover the long tail of non-compliant workloads by paging the on-call. Gatekeeper gives three enforcementAction values:

The disciplined rollout is dryrun -> read audit -> warn -> deny. Start here:

spec:
  enforcementAction: dryrun

Apply, wait one audit cycle (default ~60s), then inspect what would have been blocked:

kubectl get k8srequiredlabels require-owner-and-costcenter \
  -o jsonpath='{.status.totalViolations}{"\n"}'

kubectl get k8srequiredlabels require-owner-and-costcenter \
  -o jsonpath='{range .status.violations[*]}{.namespace}{"/"}{.name}{": "}{.message}{"\n"}{end}'

Drive that count to zero (or to a known, accepted set) before promoting. Equally important is the webhook’s behavior when Gatekeeper itself is unavailable. The default failurePolicy: Ignore fails open — safer for cluster availability but it means an outage silently disables your guardrails. For genuinely security-critical policies, set failurePolicy: Fail on the webhook (fail closed), but only after you trust Gatekeeper’s HA and have budgeted for a tight timeoutSeconds (3 seconds is a sane ceiling; a slow webhook stalls every write).

There is also a middle setting for teams that want hard enforcement on new writes but only reporting from the periodic scan: Gatekeeper supports scoped enforcement actions, where a single Constraint can deny at the admission webhook while merely auditing at the audit controller. Reach for it when a policy must block deploys immediately, yet you want the audit dashboard to keep counting pre-existing violations without any of them being treated as a hard failure.

5. Mutation: defaults with Assign and ModifySet

Validation rejects; mutation fixes. Instead of denying a Pod that omits seccompProfile, you can inject a default. Mutators are separate CRDs and run before validation, so a mutation can bring an object into compliance with a Constraint that would otherwise reject it.

Assign sets a scalar or object field. Defaulting the seccomp profile cluster-wide:

# mutations/default-seccomp.yaml
apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: default-seccomp-profile
spec:
  applyTo:
    - groups: [""]
      kinds: ["Pod"]
      versions: ["v1"]
  match:
    scope: Namespaced
    excludedNamespaces: ["kube-system"]
  location: "spec.securityContext.seccompProfile.type"
  parameters:
    assign:
      value: "RuntimeDefault"
    pathTests:
      - subPath: "spec.securityContext.seccompProfile.type"
        condition: MustNotExist

pathTests with MustNotExist is what makes this a default rather than an override: the mutation only fires when the user has not already set the field. Without it you would stomp on teams that legitimately chose a localhost profile.

ModifySet manages list membership idempotently — adding to or pruning from arrays. To strip a debug flag teams keep copy-pasting:

# mutations/strip-debug-args.yaml
apiVersion: mutations.gatekeeper.sh/v1
kind: ModifySet
metadata:
  name: strip-debug-args
spec:
  applyTo:
    - groups: [""]
      kinds: ["Pod"]
      versions: ["v1"]
  location: "spec.containers[name: *].args"
  parameters:
    operation: prune
    values:
      fromList:
        - "--debug"

Use operation: merge to add an element. The [name: *] wildcard applies the change to every container. There is also AssignMetadata (labels/annotations only) and AssignImage (image fields), but Assign and ModifySet cover the overwhelming majority of defaulting needs.

6. Syncing data for referential constraints

Some rules cannot be decided from the incoming object alone — they need cluster context. “An Ingress host must be unique across all namespaces” requires knowing every other Ingress. Gatekeeper solves this by replicating selected objects into OPA’s in-memory cache via a Config (or SyncSet), then exposing them in Rego under data.inventory.

# config/sync.yaml
apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: gatekeeper-system
spec:
  sync:
    syncOnly:
      - group: "networking.k8s.io"
        version: "v1"
        kind: "Ingress"

A referential template then reads the cache. Namespace-scoped objects live at data.inventory.namespace[<ns>][<groupVersion>][<kind>][<name>]; cluster-scoped at data.inventory.cluster[<groupVersion>][<kind>][<name>]:

package k8suniqueingresshost

identical(obj, review) {
  obj.metadata.namespace == review.object.metadata.namespace
  obj.metadata.name == review.object.metadata.name
}

violation[{"msg": msg}] {
  input.review.kind.kind == "Ingress"
  host := input.review.object.spec.rules[_].host
  other := data.inventory.namespace[_]["networking.k8s.io/v1"]["Ingress"][_]
  other.spec.rules[_].host == host
  not identical(other, input.review)
  msg := sprintf("ingress host %v is already claimed", [host])
}

The identical guard is essential — without it the object always collides with itself on UPDATE. Two operational caveats: only sync what you query (the cache costs memory and watch load), and remember the cache is eventually consistent. Under a burst of simultaneous Ingress creates, two could momentarily both pass. Treat referential uniqueness as defense-in-depth, not a hard transactional guarantee.

7. Shift-left: the same policies in CI with Conftest

Admission control is your last line of defense. It is a poor first one — by the time kubectl apply is rejected, the developer has already context-switched. The fix is running policy in CI against rendered manifests, so the feedback lands on the pull request.

Conftest runs Rego against structured config (YAML, JSON, HCL). Its convention differs from Gatekeeper: rules are named deny/violation/warn in package main, and the document under test is input directly (no review.object wrapper). You can share the core logic by factoring it into a library package both call, or maintain a thin Conftest mirror:

# policy/deny_registry.rego
package main

allowed_repos := ["registry.internal.example.com/", "ghcr.io/acme/"]

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not any([startswith(container.image, r) | r := allowed_repos[_]])
  msg := sprintf("%v: image %v not from an allowed registry", [input.metadata.name, container.image])
}

Wire it into the pipeline. Render Helm/Kustomize first so you test what actually deploys:

# render, then gate
helm template ./chart --values values-prod.yaml > /tmp/rendered.yaml
conftest test /tmp/rendered.yaml --policy policy/

# .github/workflows/policy.yml
name: policy-gate
on: [pull_request]
jobs:
  conftest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install conftest
        run: |
          VER=0.56.0
          curl -sSfL "https://github.com/open-policy-agent/conftest/releases/download/v${VER}/conftest_${VER}_Linux_x86_64.tar.gz" \
            | tar -xz -C /usr/local/bin conftest
      - name: Render manifests
        run: kustomize build overlays/prod > rendered.yaml
      - name: Conftest gate
        run: conftest test rendered.yaml --policy policy/

Conftest exits non-zero on any deny, failing the job. Now a bad registry reference is a red check on the PR, not a production incident.

8. Testing Rego and gating the policies themselves

Policies are code, and untested policy code rots. Two complementary tools cover the two layers.

Unit-test the Rego with OPA’s built-in framework. Rules prefixed test_ are auto-discovered; with input as mocks the document:

# policy/deny_registry_test.rego
package main

test_denies_external_image {
  deny[_] with input as {
    "kind": "Deployment",
    "metadata": {"name": "web"},
    "spec": {"template": {"spec": {"containers": [{"image": "docker.io/nginx"}]}}},
  }
}

test_allows_internal_image {
  count(deny) == 0 with input as {
    "kind": "Deployment",
    "metadata": {"name": "web"},
    "spec": {"template": {"spec": {"containers": [{"image": "ghcr.io/acme/web:1.2.3"}]}}},
  }
}

opa test policy/ -v

Integration-test the Gatekeeper templates with gator, which evaluates real ConstraintTemplates + Constraints against fixtures without a cluster. A Suite declares cases and asserts the expected violation count:

# test/suite.yaml
kind: Suite
apiVersion: test.gatekeeper.sh/v1alpha1
tests:
  - name: required-labels
    template: ../templates/k8srequiredlabels.yaml
    constraint: ../constraints/require-owner-label.yaml
    cases:
      - name: missing-owner-is-rejected
        object: fixtures/deploy-no-owner.yaml
        assertions:
          - violations: yes
      - name: compliant-is-allowed
        object: fixtures/deploy-compliant.yaml
        assertions:
          - violations: no

gator verify test/suite.yaml

Both tools exit non-zero on failure, so they drop straight into the CI job from section 7. This closes the loop: a change to a ConstraintTemplate cannot merge unless its tests pass, exactly like application code.

Going deeper

You now have a working guardrail program. This section is for when you operate it at scale — the internals that explain the surprises, and the decisions that separate a demo from a platform.

Under the hood: the webhook and the audit loop

Gatekeeper ships as two workloads in gatekeeper-system: gatekeeper-controller-manager (which serves the admission webhook and runs the template/constraint controllers) and gatekeeper-audit (the periodic scanner). Keeping them separate matters — a storm of audit work never slows the in-path webhook.

The webhook side manages its own ValidatingWebhookConfiguration (and mutating equivalent). Gatekeeper’s cert controller generates and rotates the TLS caBundle automatically, which is why you never mint certificates by hand — and why a stuck cert-rotation, not your Rego, is a common cause of sudden x509 errors in the API server log. Every matched CREATE/UPDATE becomes a synchronous AdmissionReview call; the object you evaluate is exactly input.review.object.

The audit side re-lists cluster objects on an interval (--audit-interval, default 60s) and evaluates them against every Constraint, writing up to a capped number of offenders (--constraint-violations-limit, default 20) into each Constraint’s status.violations plus a status.totalViolations count. Two things follow: audit results lag reality by up to one interval, and the per-constraint list is truncated — a Constraint showing 20 violations may have thousands. Trust totalViolations for the true count, and export it to a dashboard rather than eyeballing the list.

Rego in five minutes, for policy authors

Rego rewards a small vocabulary. For Gatekeeper you almost never need more than this:

• Packages and rules. Each template’s Rego lives in a package. A rule is head { body }; the body is a conjunction of expressions, all of which must be true.
• Partial set rules. violation[x] { ... } builds a set. Gatekeeper reads that set: empty means allow, non-empty means deny, and each element is one reported problem.
• The inputs. input.review.object is the incoming Kubernetes object; input.review.kind, input.review.operation, and input.review.userInfo carry request metadata; input.parameters is your Constraint’s parameters.
• Iteration. _ is “some element” — input.parameters.repos[_] visits each repo. some i names an index when you need it.
• Comprehensions. {x | ...} builds a set, [x | ...] a list, {k: v | ...} an object — the workhorse for gathering “all the things that match.”
• Negation as a guard. not any(satisfied) succeeds only when nothing satisfied the condition — the idiom behind “none of the allowed registries matched.”
• Handy builtins. startswith, endswith, contains, count, sprintf, and set arithmetic (-, &, |).

A minimal “no :latest tags” rule shows the shape end to end:

package k8sdisallowlatesttag

violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  endswith(container.image, ":latest")
  msg := sprintf("image %v uses the mutable :latest tag", [container.image])
}

Referential policies, data.inventory, and external data

Section 6 syncs cluster objects into data.inventory so a rule can see its neighbours. When the answer lives outside the cluster — an image signature to verify, an allow-list maintained by a security team, a CMDB — Gatekeeper offers external data providers. You register a Provider (an HTTPS endpoint) and call it from Rego with the external_data builtin; Gatekeeper batches the keys, calls the provider during admission, and feeds the response back to your rule. It is powerful and it is a footgun: every external call adds latency to the write path, so it runs behind a feature flag, wants a strict timeout, and should cache aggressively. Prefer syncing when the data is already in the cluster; reserve external data for facts that genuinely cannot be.

Mutation ordering and idempotency

Mutation is a separate webhook that runs before validation, so a defaulted object is what your Constraints actually see. When several mutators touch one object, Gatekeeper applies them in a defined, stable order (by mutator type and then by name), and it re-runs them until the result is stable — which is exactly why every mutator must be idempotent. Assign with pathTests: MustNotExist is idempotent by construction; ModifySet is idempotent because it manages set membership. A mutator that blindly appends to a list on every pass is a bug waiting to double your args.

Gatekeeper vs Kyverno

Both are CNCF admission-policy engines; the honest answer to “which one” is “it depends on your team and your requirements.” The core split is the policy language and model:

Reach for Gatekeeper when you want one policy language spanning admission and CI and other systems, when your rules are genuinely complex (Rego’s expressiveness pays off), or when you already have Rego skills. Reach for Kyverno when you want the shallowest learning curve, when generating or synchronizing resources is a first-class need, or when native image verification matters — see Kyverno: policy-as-code with mutate, generate & image verification for its model in depth. Many platforms even run both; there is no rule against it, though two engines is two things to operate.

Testing at every layer: gator verify, gator test, gator expand

gator is more than the Suite runner from section 8:

• gator verify runs a Suite of declared cases with expected violation counts — the assertion-style tests you commit alongside a template.
• gator test pipes a stream of resources plus your templates and constraints and prints violations, the way Conftest does for plain Rego. It is the fastest way to gate rendered manifests in CI against the real Gatekeeper templates.
• gator expand expands a workload into what it will actually create (a Deployment into its Pod) and applies your mutators first, so you test the object that will truly be admitted — not the template you wrote.

# Evaluate rendered manifests against real templates + constraints (CI-friendly)
kustomize build overlays/prod | gator test --filename=- \
  --filename=templates/ --filename=constraints/

Performance and HA

Because the webhook is in the path of every write, treat it like a tier-0 service:

• Run replicas + a PodDisruptionBudget for gatekeeper-controller-manager so a node drain never takes the gate offline.
• Keep timeoutSeconds tight (≤ 3s). A slow webhook does not just delay one request — it delays every matching write cluster-wide.
• Choose failurePolicy deliberately. Ignore protects availability at the cost of silently skipping policy during an outage; Fail protects security at the risk of blocking writes if Gatekeeper is down. Reserve Fail for a small set of security-critical Constraints, and back them with RBAC too.
• Exempt system namespaces globally, not just per-Constraint, so Gatekeeper can never wedge cluster components (including its own).
• Narrow every match. Evaluation cost scales with how many Constraints a request matches; a broad match makes Gatekeeper evaluate rules on objects that could never violate them.
• Watch the metrics. Gatekeeper exports Prometheus series (request latency, evaluation counts, constraint totals) — alert on webhook latency the same way you alert on the API server’s.

Verify

Confirm the full guardrail program end to end against a live cluster:

# 1. Constraints are registered and active
kubectl get constrainttemplates
kubectl get constraints -A

# 2. A non-compliant object is actually rejected (expect an error)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rogue
  namespace: default
spec:
  selector: { matchLabels: { app: rogue } }
  template:
    metadata: { labels: { app: rogue } }
    spec:
      containers:
        - name: app
          image: docker.io/library/nginx:latest
EOF
# -> admission webhook "validation.gatekeeper.sh" denied the request:
#    missing required labels: {"cost-center", "owner"}; image ... not from an allowed registry

# 3. Mutation applied a default
kubectl run probe --image=ghcr.io/acme/probe:1.0 --restart=Never
kubectl get pod probe -o jsonpath='{.spec.securityContext.seccompProfile.type}{"\n"}'
# -> RuntimeDefault

# 4. Audit surfaces pre-existing violations
kubectl get k8srequiredlabels -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.totalViolations}{"\n"}{end}'

# 5. CI gate runs locally
opa test policy/ -v && gator verify test/suite.yaml && conftest test rendered.yaml --policy policy/

If step 2 admits the Deployment instead of rejecting it, the most likely causes are: the Constraint is still in dryrun, the match block does not cover apps/Deployment, or the namespace is in excludedNamespaces.

Enterprise scenario

A platform team running multi-tenant clusters for ~40 product squads hit a recurring class of incident: teams shipped Deployments with no memory limits, and a single runaway pod would consume a node’s memory and trigger noisy-neighbor evictions across unrelated tenants. The wiki said “always set limits.” Nobody did.

Going straight to a hard deny was politically and operationally untenable — an audit showed roughly 60% of existing workloads lacked limits, so a same-day enforce would have blocked the next deploy for two-thirds of the org. They ran a staged program instead.

First, they applied the gatekeeper-library container-limits template with the Constraint in dryrun, exported status.totalViolations to a dashboard, and pushed the per-squad list into each team’s channel. Second — the move that made it land — they added an Assign mutation that injected conservative default limits only when absent, so new workloads became compliant automatically while teams tuned real values:

apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: default-mem-limit
spec:
  applyTo:
    - groups: [""]
      kinds: ["Pod"]
      versions: ["v1"]
  match:
    scope: Namespaced
    excludedNamespaces: ["kube-system", "gatekeeper-system"]
  location: "spec.containers[name: *].resources.limits.memory"
  parameters:
    assign:
      value: "512Mi"
    pathTests:
      - subPath: "spec.containers[name: *].resources.limits.memory"
        condition: MustNotExist

Mutation drove the dry-run violation count down on its own as workloads rolled. Six weeks later, with the dashboard reading near-zero and the same checks already failing PRs in CI via Conftest, they promoted the Constraint to deny during a change window. There was no flag day and no spike of blocked deploys — the gate had effectively already closed.

Practice challenges

Work these in order; each builds on the last. Try before opening the solution — the point is the muscle memory, not the answer. If you have no cluster, gator and opa test run every graded check locally.

1. Beginner — install and confirm. Install Gatekeeper from a pinned release and prove the CRDs registered.

<details> <summary>Solution</summary>

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.16.3/deploy/gatekeeper.yaml
kubectl -n gatekeeper-system rollout status deploy/gatekeeper-controller-manager
kubectl get crd | grep gatekeeper.sh   # expect constrainttemplates + config + mutation CRDs

Why: pinning the version keeps an in-path admission controller from changing under you; the rollout status gate ensures the webhook pods are actually ready before you rely on them. </details>

2. Beginner — a required-labels ConstraintTemplate + Constraint. Require every Deployment in the apps group to carry a team label. Ship the Constraint in dryrun first.

<details> <summary>Solution</summary>

# template
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          required := {l | l := input.parameters.labels[_]}
          provided := {l | input.review.object.metadata.labels[l]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("missing required labels: %v", [missing])
        }
---
# constraint (dryrun)
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-team-label
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]
  parameters:
    labels: ["team"]

Why dryrun: it admits everything but records offenders in status.violations, so you learn how many existing Deployments would break before anything is blocked. </details>

3. Intermediate — a disallowed-registry policy. Reject any Pod whose containers pull from outside ghcr.io/acme/. Enforce with deny.

<details> <summary>Solution</summary>

# template
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          satisfied := [ok | repo := input.parameters.repos[_]; ok := startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("image %v is not from an allowed registry %v", [container.image, input.parameters.repos])
        }
---
# constraint (deny)
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: only-acme-ghcr
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]
  parameters:
    repos: ["ghcr.io/acme/"]

Why the list comprehension + not any(...): it means “none of the allowed prefixes matched this image” — the clean Rego idiom for an allow-list. Go to deny here only because a registry rule usually has a small, known violation set; confirm with audit first if unsure. </details>

4. Intermediate — promote safely. You shipped challenge 2 in dryrun. Read the audit report, then promote to deny once the count is acceptable.

<details> <summary>Solution</summary>

# wait ~1 audit interval, then read the true count + the offenders
kubectl get k8srequiredlabels require-team-label \
  -o jsonpath='{.status.totalViolations}{"\n"}'
kubectl get k8srequiredlabels require-team-label \
  -o jsonpath='{range .status.violations[*]}{.namespace}{"/"}{.name}{"\n"}{end}'

# once zero (or a known, accepted set), flip to enforce
kubectl patch k8srequiredlabels require-team-label --type=merge \
  -p '{"spec":{"enforcementAction":"deny"}}'

Why read totalViolations and not just the list: the per-constraint list is truncated at --constraint-violations-limit (default 20), so the count is the trustworthy number. </details>

5. Advanced — test the policy without a cluster. Add a gator Suite that asserts the registry policy rejects a Docker Hub image and admits an ghcr.io/acme one.

<details> <summary>Solution</summary>

# test/suite.yaml
kind: Suite
apiVersion: test.gatekeeper.sh/v1alpha1
tests:
  - name: allowed-repos
    template: ../templates/k8sallowedrepos.yaml
    constraint: ../constraints/only-acme-ghcr.yaml
    cases:
      - name: dockerhub-is-rejected
        object: fixtures/pod-dockerhub.yaml
        assertions:
          - violations: yes
      - name: acme-image-is-allowed
        object: fixtures/pod-acme.yaml
        assertions:
          - violations: no

gator verify test/suite.yaml   # exits non-zero if either assertion fails

Why gator: it runs the real template + constraint against fixtures with no cluster, so the same check gates the policy repo in CI exactly like application tests. </details>

6. Advanced — default instead of deny. Rather than rejecting Pods that omit seccompProfile, mutate them to RuntimeDefault — but never overwrite a profile a team set on purpose.

<details> <summary>Solution</summary>

apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: default-seccomp-profile
spec:
  applyTo:
    - groups: [""]
      kinds: ["Pod"]
      versions: ["v1"]
  match:
    scope: Namespaced
    excludedNamespaces: ["kube-system"]
  location: "spec.securityContext.seccompProfile.type"
  parameters:
    assign:
      value: "RuntimeDefault"
    pathTests:
      - subPath: "spec.securityContext.seccompProfile.type"
        condition: MustNotExist

Why pathTests: MustNotExist: it turns an override into a default — the mutation fires only when the field is absent, so a team that chose a Localhost profile keeps it. </details>

Common beginner mistakes

• Shipping straight to deny. The misconception is that a policy is only “real” if it blocks. In a cluster with existing workloads, a same-day deny pages the on-call as two-thirds of deploys start failing. The right mental model: a policy’s first job is to measure. Always dryrun → read audit → warn → deny.
• Naming the rule deny instead of violation. Gatekeeper only collects a set named violation; deny is the Conftest convention. A template with a deny rule parses fine and enforces nothing — the most dangerous failure mode, because it looks installed. Memorize: Gatekeeper = violation, Conftest = deny.
• Forgetting to sync data for referential rules. A rule that reads data.inventory returns nothing if you never created a Config/SyncSet for that GVK — so a uniqueness policy silently passes everything. The cache is opt-in; if a rule reasons about other objects, it needs a sync entry, and the rule needs an identical guard so an object does not collide with itself on UPDATE.
• An over-broad match (or none). An empty match means every object, including system namespaces. Locking kube-system or gatekeeper-system out of their own writes is a self-inflicted outage. Scope to the narrowest kinds, and put system namespaces in excludedNamespaces on day one.
• Assuming mutation overrides. Without pathTests: MustNotExist, an Assign stomps a field every team set deliberately. Mutators must also be idempotent — Gatekeeper re-applies them until stable, so a mutator that blindly appends will duplicate. Default, don’t overwrite.
• Not testing policies. Treating Rego as config you eyeball, not code you test, means a refactor quietly inverts a rule and nobody notices until production. Commit opa test unit tests and a gator Suite, and gate the policy repo on them like any service.
• failurePolicy: Fail without HA. Failing closed is correct for security-critical rules — but if Gatekeeper has one replica and it restarts, every matching write in the cluster starts failing. Fail closed only after replicas, a PDB, and a tight timeoutSeconds are in place.
• Expecting Gatekeeper to block objects already in the cluster. Admission only gates new writes. A policy added today does nothing to yesterday’s non-compliant Deployment until that object is next updated — the audit controller only reports it. To remediate existing objects you must re-apply or mutate them.

Glossary

• OPA (Open Policy Agent): A general-purpose policy engine that evaluates Rego rules against JSON input. Knows nothing about Kubernetes on its own.
• Gatekeeper: The Kubernetes-native integration of OPA — it wires OPA into the API server as an admission webhook, gives you CRDs to author policy, and adds an audit controller.
• Rego: OPA’s declarative policy language. Rules are conjunctions of expressions; a partial set rule named violation is the Gatekeeper contract.
• ConstraintTemplate: A reusable policy definition — Rego logic plus an openAPIV3Schema for its parameters. Applying one generates a new CRD kind.
• Constraint: An instance of the CRD a template generated. Declares match (scope), parameters (values), and enforcementAction. Contains no Rego.
• CRD (CustomResourceDefinition): A way to add a new resource type to the Kubernetes API. Each ConstraintTemplate produces one.
• Admission webhook: An HTTPS callback the API server invokes on writes. Validating webhooks allow/deny; mutating webhooks can change the object.
• enforcementAction: Per-Constraint behavior on violation — dryrun (record only), warn (admit + warn the client), or deny (reject).
• Audit controller: A background Gatekeeper component that periodically re-evaluates existing objects and writes status.violations / status.totalViolations.
• violation: The Rego set rule Gatekeeper reads. A non-empty set means reject; each element is one reported problem with a msg.
• input.review.object: The incoming Kubernetes object under evaluation. input.review also carries kind, operation, and userInfo.
• input.parameters: The parameters block from the Constraint, made available to the template’s Rego.
• match: The Constraint’s scoping surface — kinds, namespaces, excludedNamespaces, labelSelector, namespaceSelector, scope, name.
• Referential constraint: A policy that reasons about objects other than the one being admitted, reading them from data.inventory.
• data.inventory: The in-memory cache of synced cluster objects that referential Rego reads.
• Config / SyncSet: The objects that tell Gatekeeper which GVKs to replicate into data.inventory.
• Mutation (Assign, ModifySet, AssignMetadata, AssignImage): CRDs that change an object before validation — setting defaults, pruning list entries, editing labels/annotations or image fields.
• pathTests / MustNotExist: A mutation guard that fires only when a field is absent, turning an override into a default.
• failurePolicy: Webhook behavior when Gatekeeper is unreachable — Ignore (fail open) or Fail (fail closed).
• timeoutSeconds: How long the API server waits for the webhook before applying failurePolicy. Keep it tight (≤ 3s).
• gator: Gatekeeper’s CLI for testing templates + constraints without a cluster — verify (Suites), test (evaluate a resource stream), expand (expand workloads + apply mutators).
• Conftest: A CLI that runs Rego against config files (YAML/JSON/HCL) in CI. Uses deny/violation/warn rules in package main with the document at input.
• Constraint framework: The shared Go library that powers both Gatekeeper and gator, so a template behaves identically in-cluster and in tests.
• gatekeeper-library: The community repository of maintained ConstraintTemplates (container limits, allowed repos, and more) you should vendor rather than re-author.
• External data provider: A registered HTTPS Provider that Rego can call via the external_data builtin to pull facts from outside the cluster during admission.

Checklist

• Gatekeeper installed from a pinned release; gatekeeper-system excluded from all Constraints
• ConstraintTemplates use the violation rule contract and a typed openAPIV3Schema
• Every new Constraint starts in dryrun; audit totalViolations driven to a known set before promotion
• dryrun -> warn -> deny promotion path documented per policy
• failurePolicy chosen deliberately (Ignore to fail open vs. Fail for security-critical) with a tight timeoutSeconds
• Mutations use pathTests: MustNotExist so they default rather than override
• Config/SyncSet syncs only the GVKs referential policies actually query
• Referential rules guard against self-collision on UPDATE
• The same intent enforced in CI with Conftest against rendered manifests
• Rego unit-tested with opa test; templates integration-tested with gator verify; both gate the policy repo