Cilium and eBPF Network Policy: L3-L7 Segmentation and Hubble Flow Visibility

In a nutshell

Cilium is a CNI – the plugin Kubernetes calls to give every pod an IP address and wire up pod-to-pod networking. What makes Cilium different is how it does that job. Instead of the classic Linux tools (a mountain of iptables rules managed by kube-proxy), it loads tiny sandboxed programs called eBPF into the Linux kernel, right next to where packets actually arrive. Those in-kernel programs do the networking, enforce your security policy, and record what happened – all without a packet ever leaving the kernel to be dragged through a slow chain of firewall rules.

Picture the old model as an airport where every passenger walks past a single guard who reads a giant, ever-growing binder of “who may go where” from page one, every single time – correct, but slower the more rules you add. eBPF is like giving each gate its own smart-card reader: the decision is a one-touch lookup, and the “card” is the pod’s identity (its labels), not its seat number (its IP address). Because the check is on identity, the rule keeps working after a pod restarts onto a brand-new IP – the exact thing that breaks IP-based firewalls constantly in Kubernetes.

Three payoffs follow from that one idea. Policy gets richer: not just “allow port 8080” but “allow HTTP GET /orders and nothing else,” or “allow egress only to *.stripe.com, by name.” Speed: policy and load-balancing become O(1) hash lookups in kernel maps instead of an O(n) walk down an iptables chain that grows with every Service. And observability: because Cilium sees every packet in the kernel, Hubble can tell you the exact verdict for any flow – forwarded, or dropped and which policy decided it – something stock Kubernetes simply cannot do.

Level: Expert · Time: ~31 min · Prerequisites: you should already understand pods, Services, labels/selectors, and the idea of a Kubernetes NetworkPolicy and the CNI pod-networking model. After this you will be able to: install Cilium in kube-proxy-replacement mode, put a namespace into default-deny, write identity-based L3/L4 and L7 (HTTP/DNS/Kafka) CiliumNetworkPolicy, allowlist external egress by DNS name, and use Hubble to prove every allowed and dropped flow.

Kubernetes network policy on a stock cluster is a stack of compromises. The native NetworkPolicy object is L3/L4 only – it cannot say “allow GET but not DELETE,” it cannot match a destination by DNS name, and the moment a pod gets a new IP your policy is reasoning about a label selector that the kube-proxy iptables backend has flattened into a linear chain of rules that grows with every service. At a few thousand services that chain becomes a measurable latency and a CPU sink, and when a packet gets dropped you have no native way to learn why. Cilium replaces both halves of that problem: an eBPF dataplane that enforces policy on a stable pod identity rather than an IP, and Hubble, a flow-observability layer that tells you the exact policy verdict for every packet. This walkthrough installs Cilium in kube-proxy-replacement mode, builds a default-deny posture, layers L3 through L7 and FQDN-based egress policy on top, and uses Hubble to prove the result.

1. Why eBPF beats iptables for policy enforcement

kube-proxy in iptables mode translates every Service into a set of rules in the nat table. Packet matching there is O(n) in the number of rules – the kernel walks the chain top to bottom for each new connection. Each Service adds rules; each endpoint adds more. On a large cluster this chain reaches tens of thousands of rules, and the per-connection walk shows up as setup latency and softirq CPU. ipvs mode improves the lookup to a hash, but you still carry the conntrack and rule-management overhead, and policy enforcement still happens on IP tuples.

Cilium attaches eBPF programs at the network device and socket layers. Service translation and policy verdicts are hash lookups in eBPF maps, effectively O(1) regardless of cluster size, and for in-cluster traffic Cilium can perform the load-balancing translation at the socket layer (connect() time) so the packet never carries a service VIP into the network at all. The decisive difference for policy: Cilium does not enforce on IP. Every pod is assigned a numeric security identity derived from its labels, and that identity travels with the packet (in the IP option or VXLAN/Geneve header). A policy verdict is “does identity A allow identity B on this port,” resolved once and cached – which is exactly why the same policy keeps working when a pod restarts onto a new IP.

The mental model shift: stop thinking “allow 10.0.3.0/24 to 10.0.4.7:443” and start thinking “allow identity app=frontend to identity app=api on 443.” The IP is an implementation detail Cilium manages; the identity is the policy.

What eBPF actually is (the 60-second version)

eBPF lets you load a small program into the running Linux kernel and attach it to a hook – a point the kernel passes through, such as “a packet just arrived on this network device” or “a process just called connect().” The program runs in kernel space at that moment, at near-native speed, and it cannot crash the kernel: before it is allowed to load, an in-kernel verifier proves it terminates and touches no memory it should not. A rejected program never runs. That safety property is why eBPF is trusted in production for the network datapath, for tracing, and for security tooling. Cilium is one large, purpose-built set of eBPF programs plus the control plane that compiles your Kubernetes objects into them.

iptables vs IPVS vs eBPF at a glance

Where Cilium sits: the CNI slot

Kubernetes does not do pod networking itself. When the kubelet starts a pod, it calls whatever CNI (Container Network Interface) plugin is installed to allocate the pod’s IP and connect its virtual interface to the node. Cilium is that plugin – but it programs the connection with eBPF instead of Linux bridges and iptables, and it keeps a per-node agent (a DaemonSet pod) that watches the Kubernetes API, compiles your policies into eBPF maps, and runs the Hubble and Envoy machinery. Replacing kube-proxy is an option Cilium offers because, once it owns the datapath, doing Service load-balancing too is a small additional step rather than a separate component.

2. Install Cilium in kube-proxy-replacement mode

Replacing kube-proxy is the highest-leverage step. On a fresh cluster, bring the nodes up without kube-proxy (kubeadm init --skip-phases=addon/kube-proxy, or set the equivalent in your managed-cluster bootstrap). Then point Cilium at the API server directly, because with no kube-proxy there is no in-cluster kubernetes Service VIP to reach it through:

# Install the Cilium CLI (verifies and templates the Helm install for you)
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

# API_SERVER_IP / PORT are your control-plane endpoint, since there is no kube-proxy
cilium install \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT} \
  --set routingMode=tunnel \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Validate that the dataplane is healthy and that kube-proxy replacement is actually active before you trust it with policy:

cilium status --wait
# Look for: KubeProxyReplacement:  True
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement

If you are migrating a live cluster rather than building fresh, do not flip this in place – see the migration section. Confirm end-to-end connectivity with the built-in suite, which spins up a set of client/server pods and exercises pod-to-pod, pod-to-service, and policy paths:

cilium connectivity test

3. CiliumNetworkPolicy vs native NetworkPolicy

Cilium enforces the upstream NetworkPolicy object faithfully, so existing policies keep working. But the native object tops out at L4 and IP/label selectors. CiliumNetworkPolicy (CNP, cilium.io/v2) adds the capabilities that make segmentation real: identity-based endpointSelector, toEntities for well-known peers (world, cluster, host, kube-apiserver), DNS-aware toFQDNs, and L7 rules for HTTP, Kafka, and DNS. A side-by-side on the same intent:

Crucially, CNP and native policy compose under Kubernetes’ additive allow semantics: with multiple policies selecting a pod, the union of their ingress/egress allows applies, and anything not allowed is denied. There is no rule-ordering or priority to reason about, unlike iptables – a packet is allowed if any selecting policy permits it.

The security identity model

The word “identity” is doing real work here, so it is worth making concrete. When a pod starts, Cilium takes its security-relevant labels (namespace, app=, and so on – deliberately not volatile labels like a pod-template hash) and maps that exact label set to a small integer, the security identity. Every pod with the same relevant labels shares one identity. Cilium persists these as CiliumIdentity objects (or entries in a kvstore) so every node agrees on the mapping. On the wire, the sending node stamps the source identity into the packet – in the encapsulation header (VXLAN/Geneve) in tunnel mode, or an IP option in native routing – so the receiving node makes the verdict without a lookup round-trip.

A handful of identities are reserved for things that are not pods, and you reference them by name with toEntities/fromEntities:

External IPs get CIDR identities, and names learned via toFQDNs get FQDN identities – which is how a DNS-based egress rule becomes an identity the datapath can match on. So a rule like “reach the API server and stay inside the cluster” is just two named entities:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-to-apiserver
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: api
  egress:
    - toEntities:
        - kube-apiserver
        - cluster

The practical upshot: you author policy in terms of what a workload is, and Cilium handles the churn of where it currently lives.

4. Default-deny per namespace, then build allows from observed traffic

The first policy in a namespace should lock it down. An empty CiliumNetworkPolicy that selects everything (endpointSelector: {}) but specifies the Ingress/Egress policy types with no allow rules puts every pod in that namespace into default-deny for that direction:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  endpointSelector: {}
  ingress:
    - {}
  egress:
    - {}

The - {} under each is an empty rule that selects no peers – it denies all ingress and egress while engaging the policy engine for those directions. (Selecting a pod with an ingress rule at all is what flips it from “default allow” to “default deny” for ingress; the empty rule is the explicit, readable way to say so.)

Do not author the allow rules from memory. Turn on policy audit mode so Cilium reports what would be dropped without actually dropping it, run real traffic, and read the verdicts out of Hubble:

# Per-endpoint audit: log denied flows as "would-be-dropped" but let them pass
CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium \
  -o jsonpath='{.items[0].metadata.name}')
ENDPOINT_ID=$(kubectl -n kube-system exec $CILIUM_POD -- \
  cilium-dbg endpoint list -o jsonpath='{[0].id}')
kubectl -n kube-system exec $CILIUM_POD -- \
  cilium-dbg endpoint config $ENDPOINT_ID PolicyAuditMode=Enabled

# Watch what the default-deny WOULD drop, with full identity context
hubble observe --namespace payments --verdict AUDIT -f

Every line is a flow you must consciously allow or leave blocked. Promote the legitimate ones into explicit rules and turn audit mode off. Here a frontend may reach the API on 8080, and the API may reach Postgres on 5432 – nothing else moves:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-allow
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            app: postgres
      toPorts:
        - ports:
            - port: "5432"
              protocol: TCP

Default-deny is only safe if you also allow DNS egress. Pods that cannot reach kube-dns fail every name resolution and look “broken” in ways unrelated to your real policy. Allow kube-system/kube-dns on UDP/TCP 53 in the same pass – and you will want an L7 DNS rule there anyway for FQDN policy (next section).

5. L7 policy: HTTP method/path, Kafka, and gRPC at the sidecar-free proxy

This is the capability native policy cannot touch. Cilium runs an embedded Envoy proxy inside the agent – no per-pod sidecar, no injection – and transparently redirects L7-scoped traffic to it for parsing. You add a rules block under toPorts, and Cilium only forwards requests that match.

Restrict the frontend to read-only HTTP against the API: allow GET /api/v1/orders and the health endpoint, drop everything else (including any POST/DELETE) with an explicit 403 from the proxy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-http-l7
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/orders"
              - method: "GET"
                path: "/healthz"

gRPC is HTTP/2, so you express gRPC method authorization with the same http rules – the path is /<package>.<Service>/<Method>:

          rules:
            http:
              - method: "POST"
                path: "/payments.PaymentService/GetStatus"

Kafka gets first-class L7 enforcement – you can authorize specific API keys and topics so a producer cannot consume, or a client cannot touch a topic it has no business reading:

          rules:
            kafka:
              - role: "produce"
                topic: "payment-events"
              - role: "consume"
                topic: "payment-events-dlq"

When the proxy denies an L7 request, the connection is not reset at L4 – it completes the TCP handshake and the application gets a protocol-level rejection (HTTP 403, a Kafka authorization error). That is a feature: callers get a clean, debuggable error instead of a mysterious timeout, and Hubble records the L7 verdict with the method and path.

6. Egress control with FQDN policies and toFQDNs DNS interception

Allowing egress to “the internet” by CIDR is hopeless – destinations move and you cannot allowlist a SaaS API by IP. Cilium solves this by intercepting DNS at L7: a toFQDNs rule works only if the same policy also allows DNS to a resolver with an L7 dns rule. Cilium snoops those DNS responses, learns the IPs the name currently resolves to, and programs exactly those IPs into the egress allow – so the policy tracks DNS as it changes.

This pairing is mandatory and the most common thing people get wrong. The DNS rule both permits resolution and feeds the IP learning:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-egress-fqdn
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: api
  egress:
    # 1. Allow + intercept DNS so toFQDNs can learn the answers
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
          rules:
            dns:
              - matchPattern: "*.stripe.com"
              - matchName: "api.stripe.com"
    # 2. Now allow egress only to those learned FQDNs, on 443
    - toFQDNs:
        - matchName: "api.stripe.com"
        - matchPattern: "*.stripe.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

The dns matchPattern/matchName in step 1 governs which names the pod may even resolve; toFQDNs in step 2 governs which resolved IPs it may then connect to. If a name is not allowed in the DNS rule, Cilium never sees its answer and toFQDNs can never permit it – a name you forgot to allow simply fails closed. Inspect what Cilium has learned for an endpoint:

kubectl -n kube-system exec ds/cilium -- \
  cilium-dbg fqdn cache list | grep stripe

7. ClusterwideNetworkPolicy and the host firewall

Some controls are not namespace-scoped. A baseline that must apply to every namespace – “no pod may egress to RFC1918 ranges outside the cluster,” or “all pods may always reach kube-dns” – belongs in a CiliumClusterwideNetworkPolicy (CCNP), which has no namespace field and selects across the whole cluster:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-dns-cluster-wide
spec:
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP

Cilium can also firewall the nodes themselves, not just pods. The host firewall enforces policy on the host network namespace, letting you lock down node ports (kubelet 10250, etcd, SSH) with the same CRD model. Enable it, then select the host with the reserved node label and a nodeSelector:

cilium config set enable-host-firewall true   # or set hostFirewall.enabled via Helm

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: lock-down-nodes
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  ingress:
    - fromEntities:
        - cluster
    - fromCIDR:
        - 10.20.0.0/16     # management/bastion range allowed to SSH
      toPorts:
        - ports:
            - port: "22"
              protocol: TCP

Host policies are default-deny the instant any host policy selects a node, exactly like pod policies. Apply a host firewall in audit mode first (PolicyAuditMode on the host endpoint) – a too-tight host policy can sever kubelet from the API server and cordon the node out from under you.

8. Hubble: tracing a dropped packet to the exact policy decision

The native stack can drop a packet and tell you nothing. Hubble’s entire reason to exist is to answer “why was this dropped, and which policy decided it.” Enable it (done in the install above) and open a port-forward, or use the CLI directly against the relay:

cilium hubble enable --ui           # if not enabled at install
cilium hubble port-forward &        # exposes the relay locally on :4245
hubble status                       # confirm flows are being collected

Watch only the denials, with both endpoints’ identities and the drop reason:

hubble observe --verdict DROPPED -f \
  --namespace payments -o compact
# Example line:
# payments/frontend-xxxx -> payments/api-yyyy  http-request  DROPPED  (Policy denied)

For an L7 verdict you get the method and path that was rejected, which makes “my GET works but my POST 403s” a five-second diagnosis instead of a packet-capture expedition:

hubble observe --namespace payments --protocol http --verdict DROPPED \
  --http-method POST -o json | jq '.l7.http'

You can pivot on identity directly. To see everything a specific pod tried and what happened to it:

hubble observe --pod payments/api-yyyy --last 200

The Hubble UI (cilium hubble ui) renders the same data as a live service map: a green line is an allowed flow, a red line is a drop, and clicking the flow shows the verdict and the policy. For an on-call engineer, the service map is the fastest way to spot “this new policy broke a dependency we forgot about” – the broken edge turns red in real time.

Hubble’s moving parts

Hubble is not a separate agent sniffing the network – it is a read-out of the flow events the eBPF datapath already produces. Three pieces matter:

• Hubble (in the agent) exposes the flows for one node. hubble observe against a single agent shows only that node’s traffic.
• Hubble Relay fans out to every agent and presents one cluster-wide flow API. cilium hubble port-forward targets the Relay; this is what you want for “show me every dropped flow in the whole cluster.”
• Hubble UI renders the Relay’s data as the live service map.

Separately, Hubble can export metrics to Prometheus (hubble.metrics.enabled), turning flows into aggregate series – drop counts by identity, HTTP status-code rates, DNS error rates – that you can alert on. A flow line carries much more than source and destination; the fields you will actually use:

9. Migrating from kube-proxy without an outage

You cannot safely delete kube-proxy and install Cilium on a running cluster in one step – there is a window where neither is fully programming service translation and connections break. Cilium supports a controlled migration via kubeProxyReplacement together with a per-node label gate so the eBPF dataplane and kube-proxy coexist while you cut over node by node:

# 1. Install/upgrade Cilium with kube-proxy replacement, but gated to nodes
#    carrying a label, so existing kube-proxy keeps running everywhere else.
helm upgrade --install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT} \
  --set nodeSelector."io\.cilium\.migration/a-node"="cilium-after-migration"

# 2. Cut over one node: cordon/drain, label it so Cilium takes the dataplane,
#    restart the agent, then uncordon. Validate, then proceed to the next node.
NODE=worker-1
kubectl cordon $NODE
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
kubectl label node $NODE --overwrite io.cilium.migration/a-node=cilium-after-migration
kubectl -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName=$NODE
kubectl -n kube-system rollout status ds/cilium
kubectl uncordon $NODE

Once every node is labeled and validated, remove the original kube-proxy DaemonSet and the nodeSelector gate so Cilium owns service translation cluster-wide:

kubectl -n kube-system delete ds kube-proxy
# Clear stale iptables rules kube-proxy left behind, on each node:
kubectl -n kube-system exec ds/cilium -- \
  nsenter -t 1 -m -- bash -c 'iptables-save | grep -v KUBE | iptables-restore'

Going deeper

This section is the internals – for the reader who wants to know what the agent is actually doing under each command above. None of it is required to ship a working policy, but it is what separates “I applied a YAML” from “I know why the packet went where it went.”

The eBPF datapath: hooks, maps, and XDP

Cilium’s programs attach at three kinds of hook. tc BPF at the ingress and egress of each pod’s virtual interface (veth) is where per-endpoint policy and identity handling happen. tc/eBPF on the node’s physical NIC handles encapsulation and node-to-node forwarding. And XDP, the earliest possible hook (in the NIC driver, before an skb is even allocated), is where Cilium can do line-rate work like dropping a DDoS flood or bouncing a load-balanced packet to another node – enabled with loadBalancer.acceleration=native on supported drivers.

State lives in eBPF maps, the shared memory between the kernel programs and the agent. The ones worth knowing:

A verdict, end to end: parse the packet, resolve the destination identity from cilium_ipcache, look up (source identity, destination identity, port) in the endpoint’s cilium_policy_<id> map, and allow or deny – with conntrack short-circuiting packets that belong to an already-approved flow. You can dump these live with cilium-dbg bpf ... or bpftool map dump.

Identity vs IP enforcement, deeper

The reason identity scales where IPs do not: policy in the datapath is one map entry per identity pair, not per IP pair. A Deployment scaling from 3 to 300 pods adds zero policy-map entries, because all 300 pods share one identity. The iptables model would add rules for every new pod IP. The cilium_ipcache map is the bridge that makes this work at the edges – it maps every known IP (local pods, remote-node pods, CIDR ranges, learned FQDNs) back to an identity, so when a packet arrives carrying only an IP the datapath can recover the identity and make the same verdict. This is also why a policy “follows” a pod across restarts: the IP changed, but the label set – and therefore the identity, and therefore the verdict – did not.

L7 policy and the Envoy path

When a toPorts.rules block names an L7 protocol, Cilium marks that traffic for proxy redirect: the datapath transparently steers the connection to a per-node Envoy instance, which parses HTTP/gRPC/Kafka and applies your rules, injecting a 403 or a protocol-level error on denial. Envoy is embedded in the agent by default; for blast-radius isolation you can run it as a separate DaemonSet with envoy.enabled=true. The important performance point: L3/L4-only flows never touch Envoy. There is no proxy tax unless a policy explicitly asked for L7 on that traffic, so you pay for parsing only where you need authorization on it. The toFQDNs DNS interception uses this same proxy path – the DNS proxy is what snoops responses to learn the IPs behind a name.

Hubble, Relay, and metrics at scale

Each agent keeps recent flows in an in-memory ring buffer (hubble.eventBufferCapacity); it is a rolling window, not a database, so for durable evidence you either scrape it continuously or turn on flow-log export (hubble.export) to write JSON to disk for shipping to a SIEM. Hubble metrics are powerful but a cardinality trap: enabling rich context labels (source/destination identity, HTTP path) multiplies Prometheus series fast, so scope the label set deliberately on a large cluster. Relay is the cluster-wide aggregator; if hubble observe shows only one node’s traffic, you are talking to a single agent instead of the Relay.

Cluster mesh and the egress gateway (forward reference)

Two capabilities extend this model past a single cluster’s edge. ClusterMesh connects multiple clusters so that identities and Services become global – a Service annotated service.cilium.io/global: "true" load-balances across clusters, and a CiliumNetworkPolicy can allow an identity that lives in a different cluster. The egress gateway solves the opposite problem: an external system that only allows a static source IP, when your pods have ephemeral ones. A CiliumEgressGatewayPolicy forces selected pods’ egress out through a fixed gateway node and its stable IP, so the third party’s firewall has something durable to allowlist:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: egress-via-gateway
spec:
  selectors:
    - podSelector:
        matchLabels:
          app: api
  destinationCIDRs:
    - "0.0.0.0/0"
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"
    interface: eth0

Both, along with the BGP control plane for advertising Service and pod ranges to a physical network, are covered in depth in Cilium cluster mesh, egress gateway, and BGP control plane.

Transparent encryption: WireGuard vs IPsec

Cilium can encrypt pod-to-pod traffic between nodes with no application changes, in one of two modes. WireGuard (encryption.type=wireguard) is the modern default choice – simple, fast, per-node keys managed automatically, nothing to rotate by hand. IPsec (encryption.type=ipsec) exists mainly for compliance regimes that mandate it (FIPS-validated cipher suites) and offers more tuning knobs at the cost of more operational surface. Both add CPU and a little latency, so reserve them for the traffic that genuinely needs wire encryption rather than blanketing a cluster that already runs inside a trusted VPC.

The bandwidth manager and BBR

Kubernetes lets a pod request an egress rate cap via the kubernetes.io/egress-bandwidth annotation, but the legacy enforcement (HTB/tc policing) is inefficient and bursty. Cilium’s bandwidth manager (bandwidthManager.enabled=true) enforces those caps with EDT (earliest-departure-time) pacing on top of the fq qdisc, which is both cheaper and smoother. With it on, you can also enable BBR congestion control (bandwidthManager.bbr=true) for pods, which typically improves throughput and tail latency for external-facing traffic on lossy paths – BBR depends on the bandwidth manager being enabled first.

Verify

Prove the policy does what you claim – both the allow and the drop – with live traffic, not by reading YAML.

# 1. Dataplane is in kube-proxy-replacement mode and healthy
cilium status --wait | grep KubeProxyReplacement   # -> True

# 2. The default-deny works: an unlabeled pod canNOT reach the API
kubectl -n payments run probe --rm -it --image=nicolaka/netshoot --restart=Never -- \
  curl -m 3 http://api:8080/api/v1/orders        # expect: timeout (no allow)

# 3. The L7 allow works: a frontend pod CAN GET but cannot POST
kubectl -n payments exec deploy/frontend -- curl -s -o /dev/null -w "%{http_code}\n" \
  http://api:8080/api/v1/orders                  # expect: 200
kubectl -n payments exec deploy/frontend -- curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST http://api:8080/api/v1/orders          # expect: 403 (proxy denied)

# 4. FQDN egress: allowed name connects, an un-allowed name fails closed
kubectl -n payments exec deploy/api -- curl -s -o /dev/null -w "%{http_code}\n" \
  https://api.stripe.com                          # expect: 200/3xx
kubectl -n payments exec deploy/api -- curl -m 3 https://example.com  # expect: timeout

# 5. Hubble confirms the verdicts with identities and the drop reason
hubble observe --namespace payments --verdict DROPPED --last 50 -o compact
hubble observe --namespace payments --protocol http --http-method POST --last 20

If step 2 returns 200, your default-deny is not engaged – check that an ingress rule actually selects the API pod. If step 3’s POST returns 200, the L7 rule is not redirecting to the proxy – confirm the http block is under toPorts.rules, not at the wrong nesting level. Hubble in step 5 is the source of truth: if a flow you expected to drop shows FORWARDED, some other policy is allowing it (additive semantics), and cilium-dbg policy selectors list on the endpoint will show you which.

Enterprise scenario

A payments platform team ran a PCI-scoped cluster where a QSA finding was blunt: the cardholder-data service could egress to the internet on 443 with no restriction, because the only available control was a native NetworkPolicy, which is IP/port-only. The service legitimately needed to reach exactly one external dependency – the card processor’s tokenization API at *.cardprocessor.com – and nothing else. The team could not allowlist by IP: the processor published a wide, frequently-rotating set of addresses behind their CDN, and any static CIDR allow was stale within a week and too broad to pass audit. Their stopgap, a default-deny with a 0.0.0.0/0 egress hole on 443, was precisely what got flagged.

The fix was a toFQDNs egress policy paired with L7 DNS interception, which let them allowlist the dependency by name and let Cilium track the rotating IPs automatically. They scoped it to the cardholder-data pods only, allowed DNS resolution for just that domain, and connected on 443 solely to the learned answers:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: chd-egress-processor-only
  namespace: cde
spec:
  endpointSelector:
    matchLabels:
      app: cardholder-data
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*.cardprocessor.com"
    - toFQDNs:
        - matchPattern: "*.cardprocessor.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

They shipped it in policy audit mode for 48 hours, watched hubble observe --namespace cde --verdict AUDIT to confirm the only external destinations the service actually reached were *.cardprocessor.com (catching one forgotten metrics endpoint in the process, which they added explicitly), then enforced. For the audit evidence they exported hubble observe --namespace cde --type drop -o json showing every attempt to any other destination being dropped with a policy verdict, plus cilium-dbg fqdn cache list proving the allow tracked the processor’s rotating IPs. The QSA closed the finding: egress was now restricted to a named dependency, the evidence was per-flow and continuous, and no human had to chase a CIDR list. The native NetworkPolicy simply could not have expressed it.

Practice challenges

Work these in order; each builds on the last. Try before opening the solution.

1. (Beginner) Prove kube-proxy is really gone. You just installed Cilium with kubeProxyReplacement=true. Confirm the datapath actually took over Service translation.

<details> <summary>Solution</summary>

cilium status | grep KubeProxyReplacement    # -> True
# or, from inside an agent pod:
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement

If it reports False or Disabled, Cilium is running but kube-proxy still owns Services – your policies apply, but Service load-balancing is not on the eBPF path. </details>

2. (Beginner) Watch a live drop. In namespace shop, stream only the dropped flows and read out the two identities and the reason.

<details> <summary>Solution</summary>

hubble observe --namespace shop --verdict DROPPED -f -o compact
# shop/web-abc -> shop/catalog-def  http-request  DROPPED  (Policy denied)

-f follows live; -o compact gives the one-line source→dest form. The parenthetical is the drop reason – Policy denied means no allow rule matched. </details>

3. (Intermediate) Fix the DNS outage you caused. You applied this to shop and now every pod reports name-resolution failures:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: locked-down
  namespace: shop
spec:
  endpointSelector: {}
  egress:
    - {}

Explain what happened and restore DNS without opening egress wider than necessary.

<details> <summary>Solution</summary>

The empty egress rule flipped every pod in shop to default-deny egress, which silently includes DNS to kube-dns. Add an explicit DNS allow – ideally cluster-wide so you never repeat the mistake:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-dns
spec:
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
          rules:
            dns:
              - matchPattern: "*"

The dns rule also positions you to add toFQDNs egress later without touching the DNS allow again. </details>

4. (Advanced) Author a GET-only L7 policy. Let app=web reach app=catalog on TCP 8080, but allow only HTTP GET – any POST/PUT/DELETE must be rejected by the proxy, not time out.

<details> <summary>Solution</summary>

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: web-get-only
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: catalog
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: web
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"

Omitting path matches any path; the single method: "GET" entry means every other method falls through to a proxy 403. Note rules sits under toPorts – the single most common nesting error. </details>

5. (Advanced) Prove challenge 4 without reading YAML. Show, with live traffic and Hubble, that GET is allowed and POST is denied at L7.

<details> <summary>Solution</summary>

kubectl -n shop exec deploy/web -- curl -s -o /dev/null -w "%{http_code}\n" \
  http://catalog:8080/items                         # expect: 200
kubectl -n shop exec deploy/web -- curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST http://catalog:8080/items                 # expect: 403
hubble observe --namespace shop --protocol http --http-method POST --verdict DROPPED --last 10

A 403 (not a timeout) confirms the flow reached Envoy and was rejected at L7; the Hubble line shows the method and path that was denied. </details>

6. (Advanced) The static-IP partner problem. A third party will only accept traffic from one fixed source IP, but your app=api pods land on random node IPs. Which Cilium feature fixes this, and sketch the object.

<details> <summary>Solution</summary>

The egress gateway: it forces selected pods’ egress out through a designated gateway node and its stable IP, giving the partner a durable address to allowlist.

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: egress-via-gateway
spec:
  selectors:
    - podSelector:
        matchLabels:
          app: api
  destinationCIDRs:
    - "0.0.0.0/0"
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"
    interface: eth0

Full treatment (plus ClusterMesh and BGP) in Cilium cluster mesh, egress gateway, and BGP control plane. </details>

Common beginner mistakes

• Mixing CNIs, or expecting native NetworkPolicy and Cilium to both “own” enforcement. A cluster has exactly one CNI. Install Cilium as the CNI; it will also enforce upstream NetworkPolicy objects, so you do not need a second policy engine. Layering Cilium on top of another CNI’s policy enforcer produces contradictory, undebuggable verdicts.
• Expecting L7 rules to work with the rules block in the wrong place. The http/dns/kafka block must be nested under toPorts (toPorts[].rules.http). Put it one level too high and the policy still applies as an L4 allow – so your GET works, your POST also works, and it looks like L7 “did nothing.” A POST that returns 200 instead of 403 is this bug until proven otherwise.
• Thinking L7 enforcement happens “in the kernel like everything else.” L3/L4 is pure eBPF; L7 is enforced by the Envoy proxy the datapath redirects to. If Envoy is disabled or the proxy path is misconfigured, L7 rules cannot be honored. This is also why L7 adds latency where L4 does not.
• Default-deny egress without DNS egress. The most common self-inflicted outage. A default-deny egress policy silently blocks kube-dns, so every name lookup fails and the app looks broken for reasons unrelated to the app. Always pair default-deny with a DNS allow (challenge 3).
• Turning on kubeProxyReplacement without meeting its prerequisites. It needs a recent enough kernel, kube-proxy actually absent (or the label-gated migration), and k8sServiceHost/k8sServicePort set – because with no kube-proxy there is no in-cluster VIP to reach the API server through. Flip it without those and Services half-work in confusing ways.
• Confusing identity with IP. Writing toCIDR rules for in-cluster pods (use toEndpoints with labels), or being surprised that a policy keeps working after a pod’s IP changes. In-cluster is identity; toCIDR/toFQDNs are for things outside the cluster that have no Kubernetes identity.
• Reading a DROPPED as “Cilium is broken.” Additive-allow means a flow is dropped because nothing allowed it, and forwarded if anything did. If a flow you meant to block shows FORWARDED, hunt for the other policy that permits it (cilium-dbg policy selectors list) rather than assuming the drop logic failed.

Checklist

• Cilium installed with kubeProxyReplacement=true; cilium status reports KubeProxyReplacement: True and cilium connectivity test passes.
• On a fresh cluster, kube-proxy was never installed (--skip-phases=addon/kube-proxy); on a live one, the node-by-node label-gated migration was used and stale KUBE- iptables rules cleared.
• Every workload namespace has an explicit default-deny CNP engaging both ingress and egress.
• DNS egress to kube-system/kube-dns (53 UDP/TCP) is allowed in every default-deny namespace – ideally as a cluster-wide policy.
• Allow rules were derived from observed traffic via PolicyAuditMode + hubble observe --verdict AUDIT, not written from memory.
• Policies use identity-based fromEndpoints/toEndpoints (label selectors), not pod IPs or CIDRs, for in-cluster flows.
• L7 rules (http method/path, kafka, gRPC-as-http2) applied wherever protocol-level authorization is required, and verified to return 403 / protocol errors, not timeouts.
• External egress is toFQDNs paired with an L7 dns rule on the resolver – never a 0.0.0.0/0 hole; cilium-dbg fqdn cache list confirms IP learning.
• Cluster-wide baselines live in CiliumClusterwideNetworkPolicy; node ports protected by the host firewall, rolled out in audit mode first.
• Hubble Relay + UI enabled; on-call can pull --verdict DROPPED flows with identities and drop reason, and read the live service map.
• A verification run on record proves both a sanctioned flow (200) and a blocked flow (timeout / 403) end to end, with the matching Hubble verdicts captured.

Glossary

• eBPF – a way to load small, verified programs into the running Linux kernel and attach them to hooks (packet arrival, connect(), and more). They run at near-native speed and cannot crash the kernel. Cilium’s datapath is built from eBPF programs.
• CNI (Container Network Interface) – the plugin Kubernetes calls to give a pod its IP and connect it to the network. Cilium is a CNI.
• kube-proxy-replacement – Cilium doing Service load-balancing itself in eBPF, so the kube-proxy component (and its iptables rules) is not needed. Reported by cilium status as KubeProxyReplacement: True.
• Security identity – a small integer Cilium assigns to a set of security-relevant pod labels. Every pod with the same labels shares one identity; policy verdicts are made on identity, not IP.
• ipcache (cilium_ipcache) – the eBPF map that maps every known IP (pods, nodes, CIDRs, learned FQDNs) to its identity, so the datapath can recover an identity from an on-wire IP.
• CiliumNetworkPolicy (CNP) – Cilium’s cilium.io/v2 policy object, adding identity selectors, toEntities, toFQDNs, and L7 rules on top of native NetworkPolicy.
• CiliumClusterwideNetworkPolicy (CCNP) – a CNP with no namespace, selecting across the whole cluster; used for baselines like cluster-wide DNS allow and the host firewall.
• endpointSelector – the label selector picking which pods a CNP applies to; {} means “every pod in scope.”
• fromEndpoints / toEndpoints – identity-based (label) selectors for the peers a rule allows, in vs out.
• toEntities / fromEntities – named non-pod peers: world, cluster, host, remote-node, kube-apiserver.
• toFQDNs – egress allow by DNS name; works only when paired with an L7 dns rule that lets Cilium learn the name’s current IPs.
• L3 / L4 / L7 – network layers: L3 = IP/identity, L4 = protocol + port, L7 = application protocol (HTTP, DNS, Kafka). Cilium enforces all three.
• Envoy – the proxy Cilium embeds (one per node) to parse and enforce L7 rules; the datapath transparently redirects L7-scoped flows to it.
• Hubble – Cilium’s flow-observability layer; reads out the flow events the eBPF datapath produces, with verdict and drop reason.
• Hubble Relay – the component that aggregates every node’s Hubble into one cluster-wide flow API (what cilium hubble port-forward targets).
• XDP – the earliest eBPF hook, in the NIC driver; used for line-rate load-balancing and drops (loadBalancer.acceleration=native).
• tc BPF – eBPF at the traffic-control ingress/egress of an interface; where Cilium does per-endpoint policy.
• conntrack (cilium_ct4_global) – the connection-tracking map that lets already-approved flows skip re-evaluation.
• Policy audit mode (PolicyAuditMode) – per-endpoint mode where would-be drops are logged as AUDIT but still forwarded; used to derive allow rules from real traffic before enforcing.
• Default-deny – the posture where, once any policy selects a pod for a direction, only explicitly allowed peers pass; there is no deny rule to write.
• Egress gateway (CiliumEgressGatewayPolicy) – forces selected pods’ egress out through a fixed node/IP so external firewalls can allowlist a stable source. See the cluster mesh and egress lesson.
• ClusterMesh – connecting multiple clusters so identities and Services become global and policy spans clusters.
• Transparent encryption – pod-to-pod wire encryption with no app changes, via WireGuard (encryption.type=wireguard) or IPsec (encryption.type=ipsec).
• Bandwidth manager – Cilium’s EDT/fq-based enforcement of pod egress-bandwidth caps (bandwidthManager.enabled=true), and the prerequisite for enabling pod BBR congestion control.