A flat pod network is the default failure mode of every Kubernetes cluster. Out of the box any pod can reach any other pod, any node, the API server, and the cloud metadata endpoint — which means one compromised container is one curl away from lateral movement across your entire estate. Zero-trust pod networking inverts that: deny everything, then allow named, intended flows. This guide builds a default-deny posture with stock NetworkPolicy, then extends it with Cilium’s identity model and L7-aware rules so you can write policy in terms of what a workload may do, not which ephemeral IP it happens to have today.
In a nutshell
Level: Advanced · Time: ~30 min
Here is the whole idea in one breath: by default, every pod in a Kubernetes cluster can talk to every other pod. There is no wall between your payment service and your marketing blog — they share one big flat network, and any pod can open a connection to any other. A NetworkPolicy is a firewall for pods: an object that says who is allowed to talk to whom. The winning move is to start by denying everything (a “default-deny” posture), then add back only the specific conversations your apps actually need.
Think of a locked building. With no policy, it is a field: anyone can walk up to any door and go in. A default-deny policy puts up walls and a single guarded door, and hands the guard a guest list. Traffic on the list gets in; everything else is turned away. There is no “banned list” — you never write “block X.” You only write “allow Y,” and anyone not on the allow-list is refused by default. That inversion — deny first, then allow by name — is the entire mental model of zero-trust pod networking.
Two directions matter, and they are controlled separately. Ingress is traffic coming into a pod (who may call me?). Egress is traffic going out of a pod (where may I connect to?). A policy can lock down one, the other, or both. The classic beginner trap lives here: the moment you lock down egress, your pods can no longer reach DNS, so they cannot resolve names like postgres or api.stripe.com — and the app looks broken for reasons that have nothing to do with the app. You will see this gotcha called out repeatedly below, because it bites everyone exactly once.
One caveat that saves hours of confusion: a NetworkPolicy is only enforced if your cluster’s networking plugin (the CNI) actually implements it. Some plugins accept the YAML and quietly do nothing. So the first job is always to prove enforcement, not assume it. If pods and labels are new to you, skim Pods, Deployments & Services and the CNI pod-networking model first — policy is written against pod labels, and enforced by the CNI, so both concepts are load-bearing here.
After this lesson you will be able to:
- Apply a namespace-wide default-deny for ingress and egress without locking yourself out.
- Re-permit exactly the flows a real workload needs — DNS, the API server, a database, one external SaaS.
- Read a
NetworkPolicyspec field by field and predict what it allows before you apply it. - Reach for
CiliumNetworkPolicywhen you need identity-based rules, L7 HTTP filtering, or allow-by-DNS-name egress. - Debug “why is this blocked?” with Hubble instead of guessing.
1. How NetworkPolicy actually works: additive allow, and the default-allow trap
NetworkPolicy is the upstream API, but it has two properties that bite everyone exactly once.
First, rules are purely additive whitelists. There is no deny rule. A policy only ever adds permitted traffic; the effective allowance for a pod is the union of every policy that selects it. You restrict traffic not by writing denials but by writing a policy that selects a pod and permits nothing.
Second — the trap — a pod is “default-allow” until at least one policy selects it for a given direction. The moment a single NetworkPolicy with policyTypes: [Ingress] selects a pod, that pod’s ingress flips to default-deny and only the listed rules are allowed. Egress is independent: selecting a pod for ingress does nothing to its egress. Each direction is gated separately.
| Pod state | Ingress | Egress |
|---|---|---|
| No policy selects it | Allow all | Allow all |
| Selected by an Ingress policy | Only listed ingress rules |
Still allow all |
| Selected by an Egress policy | Still allow all | Only listed egress rules |
| Selected by both | Both locked down | Both locked down |
The other detail people miss: NetworkPolicy is enforced by the CNI, not the API server. kubectl apply succeeds even if your CNI ignores the object entirely. Flannel, for instance, does not enforce policy at all — the YAML applies cleanly and does nothing. Confirm enforcement before you trust it.
Selectors resolve against pod labels, not names or IPs. An empty
podSelector: {}selects every pod in the policy’s namespace — that is the lever you pull for a namespace-wide default-deny.
Anatomy of a NetworkPolicy object
Before writing policy in anger, learn the five load-bearing parts of the spec so every field has a name. A NetworkPolicy never “attaches” to a pod — it selects pods by label, in its own namespace, and declares which directions it governs and which peers are permitted:
apiVersion: networking.k8s.io/v1 # the stable upstream API group/version
kind: NetworkPolicy
metadata:
name: api-allow-frontend
namespace: shop # a policy ONLY affects pods in its own namespace
spec:
podSelector: # (1) WHICH pods this policy governs (by label)
matchLabels:
app: api
policyTypes: # (2) WHICH directions flip to default-deny
- Ingress
- Egress
ingress: # (3) allowed INBOUND sources (who may call app=api)
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
egress: # (4) allowed OUTBOUND destinations (where app=api may go)
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
Read it as four questions. (1) podSelector — which pods does this govern? An empty {} means every pod in the namespace; a label match means just those. (2) policyTypes — which directions does it lock down? Crucially, listing Egress here with no egress rules is how you deny all egress; the presence of the direction in policyTypes is what flips the switch, not the presence of rules. (3) ingress and (4) egress — the allow-lists themselves, each a list of from/to peers plus optional ports. Omit a ports block and every port to that peer is allowed; include it and only those ports pass.
The podSelector accepts the full Kubernetes label-selector grammar, so matchExpressions lets you select set-wise — useful when one policy should cover several apps:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: select-two-apps
namespace: shop
spec:
podSelector:
matchExpressions:
- key: app
operator: In
values: ["checkout", "ledger"]
policyTypes:
- Ingress
That single policy selects both app=checkout and app=ledger pods and flips both to default-deny ingress. matchLabels and matchExpressions may be combined in one selector, and when both are present they are AND-ed together — a pod must satisfy every clause to be selected.
2. Roll out a namespace-scoped default-deny, safely
The safe ordering is: turn on default-deny for ingress first, verify nothing broke, then do egress — because egress lockdown breaks DNS, and a pod that cannot resolve names fails in confusing ways that look like application bugs.
Start with ingress only, scoped to one namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: payments
spec:
podSelector: {} # every pod in this namespace
policyTypes:
- Ingress
# no ingress rules => deny all ingress
Apply it, then confirm your services still answer intended callers (they will not yet — you have not written allow rules — so do this in a non-prod namespace or pair it with the allow rules from the next step). Once ingress is understood, layer egress:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: payments
spec:
podSelector: {}
policyTypes:
- Egress
# no egress rules => deny all egress (including DNS!)
The instant this lands, every pod in payments loses DNS, loses the API server, loses everything. That is correct — but only acceptable if you apply the allow-list from step 3 in the same change. Treat default-deny-egress and its DNS allow as an atomic unit; never merge one without the other.
A useful discipline: keep these two deny policies in every namespace as a baseline, managed by your platform layer (a Kustomize base or a Helm chart applied to all tenant namespaces), and let application teams add only allow policies on top.
You can also express both directions in a single object —
podSelector: {}withpolicyTypes: [Ingress, Egress]and no rules — which some teams prefer as one unmistakable “this namespace is locked” baseline. Two separate objects (as above) let you roll ingress and egress out on different days; one object is tidier once you are confident. Both are equivalent in effect.
3. Allow DNS, kube-api, and metadata without opening the cluster
With egress denied, you must explicitly re-permit the handful of things almost every pod needs. The trick is to scope each one tightly.
DNS is the first casualty. CoreDNS runs in kube-system; allow egress to it on UDP/TCP 53. Because you need to select pods in another namespace, use namespaceSelector + podSelector together (an AND, not an OR, when in the same from/to element):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: payments
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns # CoreDNS keeps this legacy label
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
The kubernetes.io/metadata.name label is set automatically by the API server on every namespace, so you can rely on it without labeling namespaces yourself.
The API server is reached via the in-cluster kubernetes.default.svc ClusterIP, which is a stable virtual IP but not a pod — so a podSelector cannot match it. Stock NetworkPolicy can only express this as an ipBlock to the service CIDR or the control-plane endpoint, which is brittle. This is the first concrete place plain NetworkPolicy runs out of road; Cilium has a first-class toEntities: [kube-apiserver] selector that solves it (step 5).
Cloud metadata (169.254.169.254) is the endpoint you most want to block by default — SSRF to it is how attackers steal node IAM credentials. With egress default-deny you get this for free: if no rule permits 169.254.169.254, it is denied. If a workload legitimately needs IMDS, allow it narrowly and prefer IMDSv2 / hop-limit hardening at the node level too:
egress:
- to:
- ipBlock:
cidr: 169.254.169.254/32
ports:
- protocol: TCP
port: 80
ipBlockmatches the post-SNAT source/destination as seen by the CNI. Pod-to-pod traffic is not expressed viaipBlockreliably because pod IPs are ephemeral — reserveipBlockfor genuinely external, stable CIDRs.
AND vs OR: the selector combination that catches everyone
This is the single most error-prone corner of the whole API, so it earns its own worked example. Inside one from (or to) block, each list item (each leading -) is OR-ed with the others, but selectors nested under the same list item are AND-ed together. The layout of your dashes literally changes the meaning.
OR — “from pods labelled app=frontend in any namespace, or from anything in the monitoring namespace”:
ingress:
- from:
- podSelector: # ← list item A
matchLabels:
app: frontend
- namespaceSelector: # ← list item B (separate dash = OR)
matchLabels:
kubernetes.io/metadata.name: monitoring
AND — “only from pods labelled app=frontend that also live in the web namespace”:
ingress:
- from:
- namespaceSelector: # ← same list item…
matchLabels:
kubernetes.io/metadata.name: web
podSelector: # …no dash = AND with the line above
matchLabels:
app: frontend
The two blocks differ only by one dash and two spaces, yet the OR version admits every pod in monitoring (a much wider hole) while the AND version admits only frontend pods from web. When a policy is mysteriously too permissive or too strict, check this first: count the dashes. The DNS rule in the snippet above deliberately uses the AND form — you want kube-dns pods in kube-system specifically, not “any pod in kube-system” and not “any pod named kube-dns anywhere.”
4. The Cilium identity model: labels over IPs, and why it survives churn
Everything above leans on IPs for anything non-pod, and that is the structural weakness of NetworkPolicy. Cilium replaces IP-based matching with security identities: Cilium hashes the set of security-relevant labels on a pod into a numeric identity, and the eBPF datapath enforces policy on identity, not IP.
The payoff is direct. When a Deployment rolls and 30 pods get 30 new IPs, their identity is unchanged because their labels are unchanged. No policy update, no datapath churn, no window where a new pod IP is briefly unmatched. Identity also makes policy readable — you allow app=frontend to talk to app=backend, and that sentence is the policy.
Inspect identities directly on any node:
# list identities and the label sets that define them
kubectl -n kube-system exec ds/cilium -- cilium identity list
# what identity does this endpoint (pod) have?
kubectl -n kube-system exec ds/cilium -- cilium endpoint list
Cilium also ships reserved identities for things that are not pods: reserved:host (the node itself), reserved:remote-node, reserved:world (anything outside the cluster), reserved:kube-apiserver, and reserved:health. These are how you express “the API server” or “the internet” without hardcoding IPs — exactly the gap stock NetworkPolicy left open.
Cilium enforces stock NetworkPolicy objects too, so your step 2-3 work is not wasted. CiliumNetworkPolicy (CNP) is a superset you reach for when you need identity entities, L7 rules, or FQDN matching. For a deeper tour of the eBPF datapath and flow visibility, see Cilium eBPF, NetworkPolicy & Hubble observability.
5. Writing CiliumNetworkPolicy with L7 HTTP and DNS FQDN rules
Two CNP capabilities are why teams adopt Cilium: L7 HTTP filtering and DNS-based egress.
L7 HTTP: method and path enforcement
A normal policy can say “frontend may reach backend on 8080.” An L7 policy says “frontend may only GET /api/v1/products on backend.” Cilium transparently redirects the matched traffic through its per-node Envoy and enforces the HTTP rules:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: frontend-to-backend-l7
namespace: shop
spec:
endpointSelector:
matchLabels:
app: backend
ingress:
- fromEndpoints:
- matchLabels:
app: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: "GET"
path: "/api/v1/products"
- method: "POST"
path: "/api/v1/orders"
Anything outside those two method/path pairs is dropped at L7 with an HTTP 403 — the connection is allowed at L4 but the request is denied, which is a far better failure signal than a TCP reset. Note port is a string in CNP, a common gotcha versus the integer used in stock NetworkPolicy.
DNS-aware egress: allow by FQDN
For egress to external services, IPs are hopeless — api.stripe.com resolves to a rotating CDN range. Cilium solves this by observing DNS responses and pinning the returned IPs to an FQDN rule. You allow the DNS lookup and the destination by name:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: egress-to-stripe
namespace: payments
spec:
endpointSelector:
matchLabels:
app: checkout
egress:
# 1. permit DNS to kube-dns AND snoop the answers
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
rules:
dns:
- matchPattern: "*.stripe.com"
# 2. permit egress to whatever those names resolved to
- toFQDNs:
- matchName: "api.stripe.com"
- matchPattern: "*.stripe.com"
toPorts:
- ports:
- port: "443"
protocol: TCP
The two blocks are both required: the dns proxy rule lets Cilium see the resolution and learn the IPs; toFQDNs then permits traffic to exactly those IPs for the rule’s TTL. Without the DNS visibility rule, toFQDNs has nothing to pin and the connection is denied.
FQDN policy enforces on the IPs Cilium observed in DNS answers. If a pod hardcodes an IP or uses its own resolver that bypasses the proxy,
toFQDNscannot match it. Force all DNS through CoreDNS and the Cilium DNS proxy, or the model leaks.
The API-server problem from step 3 also disappears here:
egress:
- toEntities:
- kube-apiserver
6. Observe allowed and dropped flows with Hubble
You cannot write tight policy blind. Hubble is Cilium’s flow observability layer; it shows you the verdict (FORWARDED / DROPPED) and the reason for every flow, which turns policy debugging from guesswork into reading.
Enable it (if not already) and open the relay:
cilium hubble enable --ui # one-time, via the cilium CLI
cilium hubble port-forward & # exposes the relay locally
Then watch live, filtered to what you care about:
# every dropped flow in a namespace — your policy-gap finder
hubble observe --namespace payments --verdict DROPPED --follow
# why was a specific pod denied? show the policy verdict + L7
hubble observe --pod payments/checkout-7d9 --verdict DROPPED -o json
# confirm an L7 rule is doing what you think
hubble observe --protocol http --to-label app=backend --follow
A DROPPED flow with reason Policy denied and a source/destination identity tells you exactly which allow rule is missing. This is the loop: apply default-deny, drive real traffic, watch DROPPED, add the minimal allow, repeat until the drop stream is quiet except for genuine intrusions.
7. Cluster-wide policy, host firewall, and external CIDR egress
Per-namespace policy does not cover non-namespaced concerns. Cilium adds two cluster-scoped tools.
CiliumClusterwideNetworkPolicy (CCNP) has no namespace and applies across the cluster — ideal for platform-wide guardrails like “no workload may reach the metadata IP, ever,” which no tenant policy can override:
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: deny-cloud-metadata
spec:
endpointSelector: {} # all endpoints, all namespaces
egressDeny:
- toCIDR:
- 169.254.169.254/32
Note egressDeny — CNP/CCNP do support explicit deny rules (unlike stock NetworkPolicy), and deny takes precedence over any allow. This is how you write non-negotiable backstops.
The host firewall extends policy to the node itself. By default Cilium policies govern pod (endpoint) traffic; the node’s host network is separate. Enable hostFirewall and use a CCNP with nodeSelector to lock down what can reach node ports such as the kubelet (10250) or SSH:
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: host-fw-lockdown
spec:
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
ingress:
- fromEntities:
- remote-node # other cluster nodes
- kube-apiserver
toPorts:
- ports:
- port: "10250"
protocol: TCP
For egress to external CIDRs, prefer toFQDNs when you can, and toCIDR / toCIDRSet (with except for carve-outs) when you are pinned to known IP ranges — a partner’s static VPN range, say:
egress:
- toCIDRSet:
- cidr: 10.20.0.0/16
except:
- 10.20.5.0/24 # this subnet stays denied
Going deeper
The sections above get you a working posture. This section is for when you have to reason about why a policy behaves the way it does, defend the design in a review, or scale it across hundreds of namespaces without it quietly rotting.
How multiple policies combine: it is a union, always
Because every rule is additive and there is no ordering or priority in stock NetworkPolicy, the effective allowance for a pod is the set union of every policy that selects it, per direction. Work a concrete case. Suppose three policies all select app=api in shop:
default-deny-ingress(podSelector{}, ingress, no rules) — contributes nothing, but flips ingress to default-deny.allow-frontend— ingress fromapp=frontendon 8080.allow-monitoring— ingress fromapp=prometheuson 9090.
The pod’s inbound allow-set is {frontend:8080} ∪ {prometheus:9090}. Everything else is denied — not because any policy said “deny,” but because ingress is default-deny (thanks to the first policy) and nothing else is in the union. Remove default-deny-ingress and the whole thing collapses: with no policy flipping ingress to deny for the other ports, the pod is back to default-allow for anything not… no — subtle point: once any ingress policy selects the pod, ingress is default-deny; the deny-all policy is often redundant if another ingress policy already selects the same pods. Teams keep the explicit deny-all anyway because it makes the intent legible and survives someone deleting the narrower policy. There is no way for one policy to subtract from another’s allowance — that limitation is exactly why explicit-deny primitives (egressDeny, AdminNetworkPolicy Deny) exist outside the stock API.
L3/L4 vs L7: two different enforcement engines
Stock NetworkPolicy and the L3/L4 parts of Cilium policy are enforced in the kernel — Cilium compiles the allow-set into eBPF programs attached to the pod’s virtual interface, so an allowed or denied packet is decided at line rate with no userspace hop. The unit of matching is identity + protocol + port. This is cheap and it is where the vast majority of your rules should live.
L7 rules are different. When a toPorts block carries an http: or dns: rules section, Cilium transparently redirects that connection through a per-node Envoy proxy (for HTTP/gRPC/Kafka) or its built-in DNS proxy (for DNS). The proxy terminates and inspects the request, applies the method/path or FQDN rules, and forwards or 403s. That inspection is not free — an L7-filtered connection pays a proxy hop and Envoy CPU, so treat L7 as a scalpel for the handful of sensitive edges (the payment API, the admin path, egress-by-name) rather than a blanket you drape over every service. A good pattern: L3/L4 identity rules everywhere, L7 only where “which HTTP method/path” or “which hostname” is genuinely the security boundary.
IP-based vs identity-based enforcement, side by side
| Concern | Stock NetworkPolicy (IP-based) |
Cilium (identity-based) |
|---|---|---|
| Match key | Pod CIDR / ipBlock, plus label selectors resolved to IPs |
Numeric security identity from labels |
| Pod churn | New IPs must re-resolve; brief unmatched windows possible | Identity unchanged across rollouts; no churn |
| Non-pod peers (API server, world) | ipBlock only — brittle |
toEntities reserved identities |
| L7 (HTTP/DNS) | Not expressible | rules.http / rules.dns via Envoy/DNS proxy |
| Explicit deny | None (additive-allow only) | egressDeny / ingressDeny, deny wins |
| External by name | Not expressible | toFQDNs with DNS snooping |
| Cross-cluster | Not expressible | Identities propagate in ClusterMesh |
Identity allocation has limits worth knowing at scale: each unique set of security-relevant labels consumes one cluster-wide identity, and the default identity space is bounded (on the order of 64k). Label explosions — a per-pod-hash label, say — can burn identities fast, so keep security-relevant labels stable and coarse. External CIDRs and FQDN results become their own CIDR identities, which is how the eBPF datapath enforces toCIDR/toFQDNs on the same identity machinery as pods.
The DNS-egress gotcha, in full
It is worth understanding why egress lockdown breaks DNS so completely. A pod resolving postgres sends a query to the cluster DNS service (CoreDNS) — that is an egress connection to kube-system on port 53. The instant an egress policy selects the pod, that connection is no longer allowed unless you listed it. Because almost every outbound action starts with a name lookup, losing DNS makes everything fail, and the failure surfaces as connection timeouts deep in the app, not as “DNS denied.” Three details catch people:
- Both UDP and TCP 53. Most lookups are UDP, but large answers (and some resolvers) fall back to TCP. List both or intermittent lookups fail.
- NodeLocal DNSCache. If your cluster runs the node-local DNS cache, queries go to a link-local address on the node, not the CoreDNS pod — your
namespaceSelector/podSelectorto kube-dns will not match it, and you need a rule for the node-local path instead. Check before assuming the kube-dns selector is enough. - Cilium DNS visibility vs stock allow. Stock
NetworkPolicycan only allow DNS to the resolver; it cannot see inside the query. Cilium’srules.dnsboth allows the query and snoops the answer sotoFQDNscan pin the resulting IPs. If you want allow-by-hostname egress at all, DNS must flow through the Cilium proxy.
The ingress-controller and load-balancer exceptions
Traffic arriving through an Ingress controller or a Service type=LoadBalancer does not carry the original client’s identity to your pod — by the time it reaches your backend, the source is the ingress controller’s pod (or the node, after SNAT). So an ingress policy on your backend must allow the controller’s labels/identity, not the end user’s; the real client only survives at L7 as an X-Forwarded-For header. Likewise hostNetwork pods and host-sourced health checks appear with reserved:host, and traffic from other nodes with reserved:remote-node — if kubelet or a LoadBalancer health check is being dropped, those reserved identities are usually the missing allow. Map your ingress path deliberately: client → LB/controller → backend, and write the backend’s ingress rule against whatever the previous hop’s identity actually is.
API and version caveats
-
endPort(port ranges) graduated to GA and lets one rule cover a contiguous range instead of listing every port:ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8000 endPort: 8080 -
Named ports are matched against the target pod’s container port names, and support across CNIs has historically been uneven — prefer numeric ports for portable policy.
-
AdminNetworkPolicy(ANP) is the newer upstream answer to stockNetworkPolicy’s missing pieces: a cluster-scoped, admin-owned API (policy.networking.k8s.io) with priority ordering and realAllow/Deny/Passactions — a portable, CNI-agnostic cousin of Cilium’sCiliumClusterwideNetworkPolicy. It is still alpha and only works on CNIs that implement it, but it is where portable cluster-wide guardrails are heading:apiVersion: policy.networking.k8s.io/v1alpha1 kind: AdminNetworkPolicy metadata: name: cluster-egress-baseline spec: priority: 20 # lower number = evaluated first subject: namespaces: {} # applies across the cluster egress: - name: "allow-dns" action: Allow to: - namespaces: matchLabels: kubernetes.io/metadata.name: kube-system ports: - portNumber: protocol: UDP port: 53 - name: "deny-metadata" action: Deny to: - networks: - 169.254.169.254/32Because ANP has priority and explicit
Deny, an admin can set floors and ceilings that a namespace-ownedNetworkPolicycannot loosen — closing the exact “a tenant widened their own allow-list” gap you will meet in the enterprise scenario below.BaselineAdminNetworkPolicyis its sibling that sets a default a tenant policy can override.
Testing policy before it reaches production
Do not ship policy on faith. Layer three checks:
- Schema/dry-run —
kubectl apply --dry-run=server -f policy.yamlcatches typos like an integer port in a CNP, a bad selector key, or an unknown field, without changing cluster state. - Behavioural tests — tools such as
cyclonusand the upstreamnetpolconformance suite generate a full connectivity matrix and diff allowed vs actual, surfacing rules that are too tight or too loose across every pod pair. - Continuous observation — pipe Hubble’s
--verdict DROPPEDstream into a canary/CI stage so a policy that would break a real path shows up as drops before promotion, not as a pager alert after.
Verify
Prove the posture rather than assume it.
# 1. A pod with NO allow rules cannot egress (should hang/fail):
kubectl -n payments run probe --image=nicolaka/netshoot --rm -it --restart=Never \
-- curl -m 5 https://example.com ; echo "exit=$?"
# expect a timeout / non-zero exit
# 2. DNS works (you allowed it) but the destination is still denied at L4:
kubectl -n payments exec deploy/checkout -- nslookup api.stripe.com # resolves
kubectl -n payments exec deploy/checkout -- curl -m 5 https://api.stripe.com # allowed only if FQDN policy applied
# 3. L7 enforcement: the wrong path is 403, the right one is 200:
kubectl -n shop exec deploy/frontend -- curl -s -o /dev/null -w "%{http_code}\n" \
http://backend:8080/api/v1/products # 200
kubectl -n shop exec deploy/frontend -- curl -s -o /dev/null -w "%{http_code}\n" \
http://backend:8080/admin # 403
# 4. Watch the verdicts that explain all of the above:
hubble observe --namespace payments --verdict DROPPED --last 50
# 5. Validate a policy parses before you ship it:
kubectl apply --dry-run=server -f policy.yaml
If step 1 returns 200, your default-deny-egress is not in effect (or your CNI is not enforcing). If step 3’s /admin returns 200, L7 redirection is not active for that endpoint — check that the CNP toPorts.rules.http block actually selects the pod.
Practice challenges
Work these in a scratch namespace. Each has a graded difficulty and a worked solution — try it before you open the toggle. Assume a shop namespace with app=frontend, app=api, and a postgres pod, and a CNI that enforces policy.
Challenge 1 (beginner) — lock a namespace, open one door. Make app=api reachable only by app=frontend on TCP 8080, and nothing else. Everything not on that list must be refused.
<details> <summary>Solution</summary>
Two objects: a namespace-wide deny-ingress to flip every pod to default-deny, then a narrow allow for the one path. (The deny-all makes the intent explicit even though the allow policy already flips app=api.)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: shop
spec:
podSelector: {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-api
namespace: shop
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
Why: the allow-list is a union; only frontend→api:8080 is in it, so every other caller is dropped by the default-deny.
</details>
Challenge 2 (beginner) — stop breaking DNS. You locked egress on the whole namespace and now nothing resolves names. Restore DNS without opening anything else.
<details> <summary>Solution</summary>
Allow egress to CoreDNS in kube-system on UDP and TCP 53, selecting the kube-dns pods specifically (AND form):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: shop
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Why: name resolution is an egress connection to kube-dns on 53; both transports are needed because large answers fall back to TCP. </details>
Challenge 3 (intermediate) — reach the API server without an IP. A controller pod needs the Kubernetes API but you refuse to hardcode the service CIDR. Allow it cleanly.
<details> <summary>Solution</summary>
This is the case stock NetworkPolicy can only express as a brittle ipBlock; use Cilium’s reserved entity instead:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-apiserver-egress
namespace: shop
spec:
endpointSelector: {}
egress:
- toEntities:
- kube-apiserver
Why: kube-apiserver is a reserved identity that tracks the control-plane endpoints for you, so the rule survives control-plane IP changes.
</details>
Challenge 4 (advanced) — GET-only at L7. app=frontend may read from app=api but must never write. Allow only GET under /api/v1/ and 403 everything else, at L7.
<details> <summary>Solution</summary>
An L7 CiliumNetworkPolicy — method GET, path as a regex prefix; anything else is denied by Envoy:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: api-get-only
namespace: shop
spec:
endpointSelector:
matchLabels:
app: api
ingress:
- fromEndpoints:
- matchLabels:
app: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: "GET"
path: "/api/v1/.*"
Why: the http rule set is an allow-list too — a POST, or a GET outside /api/v1/, matches nothing and is dropped with a 403. Remember port is a string in CNP. (Cilium HTTP path is a regex, so /api/v1/.* covers the whole prefix.)
</details>
Challenge 5 (advanced) — an un-overridable metadata block. Guarantee that no workload in any namespace can reach 169.254.169.254, and that no tenant policy can loosen it.
<details> <summary>Solution</summary>
A cluster-wide explicit deny — because deny wins over any allow and CCNP is not namespace-scoped, no tenant NetworkPolicy can override it:
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: deny-cloud-metadata
spec:
endpointSelector: {}
egressDeny:
- toCIDR:
- 169.254.169.254/32
Why: egressDeny is a hard deny evaluated above additive allows; stock NetworkPolicy has no equivalent, which is exactly why platform guardrails live in CCNP (or upstream AdminNetworkPolicy).
</details>
Common beginner mistakes
-
“I applied a NetworkPolicy but nothing is blocked.” Almost always one of two things: your CNI does not enforce policy (Flannel, or a misconfigured plugin —
kubectl applystill succeeds and does nothing), or no policy actually selects the pod for that direction, so it is still default-allow. The right mental model: applying YAML is necessary but not sufficient — you must prove a deny drops traffic, and confirm thepodSelectormatches the pod’s labels. -
“Where do I write the deny rule?” You do not. Stock
NetworkPolicyhas no deny primitive. Deny is implicit, and it only switches on for a pod+direction once at least one policy selects that pod for that direction. You “block” by selecting a pod and not listing the traffic — the absence is the block. (OnlyCiliumNetworkPolicy/CCNP andAdminNetworkPolicyadd explicit deny.) -
Locking egress and forgetting DNS. The most common self-inflicted outage. The moment egress is default-deny, name resolution stops, and the app fails with timeouts that look like its bug. Always ship the DNS allow (UDP+TCP 53 to kube-dns) in the same change as any default-deny-egress.
-
Mixing up
fromandto(ingress vs egress direction).ingress.fromis who may call this pod;egress.tois where this pod may connect. Beginners write an egress rule to “let the frontend reach me” — but that belongs in the callee’s ingress. Ask “whose pod does this policy select, and is the traffic coming in or going out of that pod?” -
Getting the AND/OR selector wrong. Two dashes under
fromis OR (widens); one dash with two nested selectors is AND (narrows). AnamespaceSelectorandpodSelectoras separate list items admits every pod in the namespace — a much bigger hole than intended. -
Thinking a policy in namespace A governs traffic in namespace B. A policy only selects pods in its own namespace, and it governs that pod’s ingress/egress. To control
A→B, you write the rule where the selected pod lives: an ingress policy in B (to gate who reaches B) or an egress policy in A (to gate where A may go). -
Integer vs string ports. Stock
NetworkPolicyuses an integerport: 8080;CiliumNetworkPolicyuses a stringport: "8080". Copy-pasting between the two silently fails schema validation or matches nothing —--dry-run=servercatches it. -
Expecting
ipBlockto match pods.ipBlockis for stable external CIDRs. Pod IPs are ephemeral and are matched by label/identity, notipBlock; using a pod CIDR inipBlockbreaks the first time a pod reschedules.
Enterprise scenario
A fintech platform team ran a 200-namespace, multi-tenant cluster on Cilium. A PCI audit required that the cardholder-data environment (CDE) namespaces could egress only to a named token-vault service and the payment processor, and that this could not be loosened by a tenant. Their first attempt used per-namespace CiliumNetworkPolicy with toFQDNs, but two findings broke it: (1) one tenant added a permissive egress policy in their own CDE namespace that widened the allow-list, and (2) an app used a baked-in IP for the processor, which toFQDNs could not match, so it silently relied on a leftover allow-all during a migration window.
The fix was a two-layer model. They moved the hard boundary into a CiliumClusterwideNetworkPolicy with an explicit egressDeny that no namespace policy can override, scoped by a cde=true label the platform applies to namespaces (tenants cannot relabel their own namespace — that label is managed by the platform’s admission policy). Tenants could still author allow rules, but never escape the deny. They also forced all DNS through the Cilium proxy and used Hubble’s DROPPED stream in CI to catch the hardcoded-IP app before promotion.
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: cde-egress-lockdown
spec:
endpointSelector:
matchLabels:
io.cilium.k8s.namespace.labels.cde: "true"
# explicit deny wins over any tenant allow
egressDeny:
- toEntities:
- world
egress:
- toFQDNs:
- matchName: "vault.internal.example.com"
- matchName: "api.processor.example.com"
toPorts:
- ports: [{ port: "443", protocol: TCP }]
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports: [{ port: "53", protocol: UDP }]
rules:
dns:
- matchPattern: "*.internal.example.com"
- matchPattern: "*.processor.example.com"
The egressDeny: toEntities: [world] is the backstop; the egress allows are the only permitted exits. Because the boundary lives in a cluster-wide object keyed off a platform-managed label, tenant changes cannot regress it — which is precisely what the auditor wanted to see in writing.
Checklist
Glossary
- Pod — the smallest deployable unit in Kubernetes; one or more containers sharing an IP. Policy is written against a pod’s labels, never its name or IP.
- Label / selector — key-value tags on objects. A
podSelector/namespaceSelectorchooses which pods or namespaces a rule applies to by matching labels. - NetworkPolicy — the upstream, namespaced API for allowing pod traffic. Additive-allow only; enforced by the CNI.
- CNI (Container Network Interface) — the networking plugin that wires up pods and, if it supports it, enforces NetworkPolicy. If it does not (e.g. Flannel), policies apply but do nothing.
- Ingress — traffic into a pod (who may connect to it). Governed by
spec.ingressand theIngressentry inpolicyTypes. - Egress — traffic out of a pod (where it may connect to). Governed by
spec.egressand theEgressentry inpolicyTypes. podSelector— chooses which pods a policy governs. Empty{}= every pod in the namespace.policyTypes— declares which directions (Ingress,Egress) the policy locks to default-deny; listing a direction with no rules denies all of it.- Default-deny — the posture where a pod refuses all traffic in a direction except what is explicitly listed. Switches on the moment any policy selects the pod for that direction.
- Default-allow — the out-of-the-box state: a pod not selected by any policy accepts and initiates all traffic.
namespaceSelector— selects peer namespaces by label; combined withpodSelectorin the same list item it is AND-ed (pods matching both), in separate items it is OR-ed.ipBlock— a CIDR peer for stable external addresses; not reliable for ephemeral pod IPs.- CiliumNetworkPolicy (CNP) — Cilium’s namespaced superset of NetworkPolicy adding identity entities, L7 rules, FQDN egress, and explicit deny.
- CiliumClusterwideNetworkPolicy (CCNP) — the non-namespaced version for cluster-wide guardrails; can select nodes and use
egressDeny. - AdminNetworkPolicy (ANP) — the newer upstream, cluster-scoped, admin-owned API with priorities and real
Allow/Deny/Passactions; a portable cousin of CCNP. - Security identity — a numeric ID Cilium derives from a pod’s security-relevant labels; policy is enforced on identity, so it survives IP churn.
- Reserved identity — Cilium’s built-in identities for non-pod peers:
host,remote-node,world,kube-apiserver,health. - L3/L4 — network/transport layer: IP + protocol + port. Where most rules live; enforced in-kernel by eBPF.
- L7 — application layer: HTTP method/path, DNS names. Enforced by redirecting through a per-node Envoy (HTTP) or DNS proxy.
toFQDNs— a Cilium egress rule that permits traffic to a hostname by pinning the IPs Cilium observed in DNS answers.toEntities— a Cilium rule targeting reserved identities (e.g.kube-apiserver,world) instead of IPs.egressDeny/ingressDeny— Cilium explicit-deny rules that win over any additive allow; the way to write non-negotiable backstops.- eBPF — the in-kernel virtual machine Cilium uses to enforce L3/L4 policy at line rate.
- Envoy (L7 proxy) — the per-node proxy Cilium redirects L7-filtered connections through to enforce HTTP/gRPC rules.
- Hubble — Cilium’s flow-observability layer; shows each flow’s
FORWARDED/DROPPEDverdict and reason. - FQDN (Fully Qualified Domain Name) — a complete hostname like
api.stripe.com; the unit oftoFQDNsanddnsrules. - SNAT — source network address translation; why
ipBlocksees post-translation addresses and why ingress-controller traffic loses the original client IP. - IMDS / metadata endpoint — the
169.254.169.254link-local address that returns node cloud credentials; a prime SSRF target you block by default. - Microsegmentation — dividing the flat pod network into many small, individually-gated segments so a breach cannot move laterally.
- Zero-trust — the principle of trusting no flow by default and permitting only explicitly-verified, intended connections.