Cilium Beyond CNI: Cluster Mesh, Egress Gateway, and the BGP Control Plane

In a nutshell

Think of a plain CNI as the wiring inside one office building: it connects the rooms — your pods — so they can talk to each other. This lesson is about three upgrades that turn that single-building wiring into a campus network run by the same crew.

• Cluster Mesh knocks doorways between two buildings. A team in Building A can be served by a colleague who has moved to Building B, and neither the caller nor the code has to know it happened — there is one shared staff directory across both buildings. In Kubernetes terms, two clusters become one service space: a Service in cluster A can be answered by pods in cluster B.
• The Egress Gateway makes everyone who leaves the campus exit through one guarded gate with one fixed street address. A partner who only buzzes in visitors from a short list of known addresses recognises you every single time, no matter which room you started in. In Kubernetes terms, selected pods leave the cluster with a stable, allowlistable source IP instead of a random node address.
• The BGP control plane tells the city’s road signs — your physical routers — how to reach every room and every public service counter inside your campus, automatically, so deliveries arrive without a hand-drawn map that goes stale the moment a room is renumbered. In Kubernetes terms, Cilium advertises PodCIDRs and LoadBalancer IPs to your network so they are reachable without a cloud load balancer.

The reason all three belong in one lesson is that they run on one shared engine: the same identity-aware eBPF datapath that already forwards your pod traffic. Cluster Mesh extends the service catalogue across clusters, the Egress Gateway changes the source IP of chosen flows, and BGP changes what the physical network knows how to reach. Learn the datapath once and the three features stop being three unrelated products.

Level: Expert · Time: ~34 min read

Before this, you should be comfortable with: the Cilium CNI, NetworkPolicy, and — most importantly — Cilium’s identity model and kube-proxy replacement, all covered in Cilium eBPF: NetworkPolicy and Hubble observability; Kubernetes Service/Endpoints basics; and the default-deny pattern from Cilium L7 network policies and default-deny.

After this you will be able to:

• Federate two or more clusters with Cluster Mesh and decide, per Service, whether it should be global, local, or affinity-pinned.
• Force selected pods to egress through a fixed IP a partner has allowlisted, scoped so you do not funnel unrelated traffic through one node.
• Advertise PodCIDR and LoadBalancer IPs to top-of-rack routers with the BGP control plane, hand out VIPs with LB-IPAM, and choose BGP vs L2 for bare metal.
• Write cluster-aware network policy so federation does not silently widen your blast radius.
• Debug each layer independently — the single most useful skill once the three are composed.

The diagram above is the entire lesson on one canvas. Two Cilium clusters are federated by the mesh apiserver (blue → teal → accent), a global checkout Service is served from either cluster, selected pods leave through a gateway node’s fixed egress IP (amber), and the BGP control plane advertises PodCIDR and LoadBalancer IPs back to the physical router (purple). One eBPF datapath sits under all three. As you read the numbered sections, trace the matching badges on the diagram — each badge is a control point or a place the design commonly breaks.

Most teams install Cilium, get a working CNI plus NetworkPolicy, and stop there. That leaves the most valuable half of the product on the shelf. The same eBPF dataplane that forwards your pod traffic can federate clusters into a single service namespace, force selected pods to leave the cluster through a stable source IP a partner has allowlisted, and speak BGP to your top-of-rack switches so PodCIDRs and LoadBalancer IPs are reachable from the physical network without an external load balancer. This guide wires all three together and shows how to debug them when they misbehave.

Everything here targets Cilium 1.16/1.17 with the BGP control plane v2 API (CiliumBGPClusterConfig and friends) and Cluster Mesh as it ships today. Commands are real and current; where a feature has a sharp edge I name it instead of papering over it.

1. The dataplane recap that makes the rest make sense

Three properties of the Cilium dataplane explain why the platform features behave the way they do.

Identities, not IPs. Cilium assigns every pod a numeric security identity derived from its labels, and the dataplane enforces policy on identities, not IP addresses. A flow is “frontend talking to payments,” not “10.0.3.7 talking to 10.0.9.4.” That indirection is what makes Cluster Mesh possible: if two clusters agree on what label set maps to which identity, a policy written in cluster A applies to the same workload in cluster B.

kube-proxy replacement. Cilium can fully replace kube-proxy, implementing Service load balancing in eBPF at the socket and tc layers instead of with iptables/IPVS. This removes a large iptables ruleset, lowers latency, and — critically for this article — is the mechanism that lets a ClusterIP resolve to backends in another cluster.

# Confirm kube-proxy replacement is actually active
cilium config view | grep -i kube-proxy-replacement
# kube-proxy-replacement   true

cilium status --verbose | grep -A2 KubeProxyReplacement

Routing mode. Cilium runs in tunnel mode (VXLAN/Geneve, the default and most portable) or native routing, where pod traffic is routed without encapsulation and the underlying network is expected to know the PodCIDRs. Native routing is faster and is the natural pairing with the BGP control plane — you advertise the PodCIDRs precisely so the fabric can route them. Check yours:

cilium config view | grep -E 'routing-mode|tunnel-protocol'
# routing-mode   tunnel   (or "native")

Mental model for the whole article: the eBPF datapath already does identity-aware L3/L4 load balancing. Cluster Mesh extends the identity and service catalog across clusters; Egress Gateway changes the source IP of selected egress flows; BGP changes what the physical network knows how to reach. They compose because they all sit on the same datapath.

2. Cluster Mesh: shared identities, global services, cross-cluster LB

Cluster Mesh connects two or more Cilium clusters so that Services, identities, and policy span all of them. Each cluster runs a clustermesh-apiserver (etcd plus a sync agent) that exposes its state; every other cluster’s agents read it.

Two invariants must hold before you connect anything, and getting them wrong is the most common failure:

1. Unique, stable cluster-id and cluster-name per cluster. IDs are integers. With the default identity allocation, the usable range is 1–255 and IDs must be unique across the mesh. (Raising max-connected-clusters to 511 is possible but reshapes the identity bit layout and must be set identically and from install time on every cluster — not a knob to flip on a live mesh.)
2. Non-overlapping PodCIDRs across all clusters. Cluster Mesh routes pod-to-pod by IP; overlapping ranges are unroutable. Plan this at install.

Set the identity at install (Helm) so it is baked into every agent:

# Cluster 1
helm upgrade --install cilium cilium/cilium --namespace kube-system \
  --set cluster.name=cluster-east \
  --set cluster.id=1

# Cluster 2 — different name AND id
helm upgrade --install cilium cilium/cilium --namespace kube-system \
  --set cluster.name=cluster-west \
  --set cluster.id=2

Enable the apiserver and connect the clusters. The CLI handles cert/secret exchange and writes the peer config into both clusters:

# Run against each cluster context
cilium clustermesh enable --context cluster-east --service-type LoadBalancer
cilium clustermesh enable --context cluster-west --service-type LoadBalancer

# Wait for the apiserver to be Ready in both
cilium clustermesh status --context cluster-east --wait

# Bi-directional connect (run once; it configures both sides)
cilium clustermesh connect --context cluster-east --destination-context cluster-west

--service-type LoadBalancer exposes the apiserver via a cloud LB; on bare metal use NodePort or, better, a LoadBalancer IP that you then advertise with BGP (section 5 — this is one reason the features pair so well). The apiserver endpoint must be reachable from the other cluster’s nodes, so a private NLB or an advertised VIP is the production-grade choice over NodePort.

Global services

A normal Service is local. You opt a Service into the mesh with an annotation, and Cilium unions the endpoints from every cluster that defines a Service of the same name and namespace:

apiVersion: v1
kind: Service
metadata:
  name: checkout
  namespace: shop
  annotations:
    service.cilium.io/global: "true"
spec:
  selector:
    app: checkout
  ports:
    - port: 8080

Deploy an identically named checkout Service in both clusters. Now checkout.shop.svc.cluster.local resolves locally as always, but the eBPF load balancer’s backend set for that ClusterIP includes pods from both clusters. A caller in cluster-east can be served by a checkout pod in cluster-west transparently — no DNS tricks, no second hostname.

How identities stay unique across the mesh

Here is the mechanical detail the “identities, not IPs” idea glosses over, and it is the reason the two invariants above are non-negotiable. Inside one cluster, a security identity is a small integer allocated from a workload’s labels. Across a mesh, two independent clusters could easily both hand out identity 12345 to different label sets. Cluster Mesh avoids the clash by making identities cluster-scoped: Cilium combines the local identity with the cluster-id, so cluster-east/frontend and cluster-west/frontend are distinguishable by origin while a policy that names the frontend labels still applies to both.

That is why a cluster-id collision is catastrophic: two clusters both claiming id 1 alias each other’s identities, and policy verdicts silently apply to the wrong workloads — the worst kind of failure, because nothing errors, traffic just goes where it should not. It is also why raising max-connected-clusters to 511 is an install-time, all-clusters decision: it moves the boundary between the cluster-id bits and the identity bits, so a live change would renumber identities underneath running policy. Set cluster.id/cluster.name once, uniquely, and never touch them on a running mesh.

3. Designing global vs local services, and failover affinity

“Make everything global” is the wrong instinct. The right unit of decision is per-Service, and the lever is affinity.

By default a global service load-balances across all backends in all clusters equally. That is rarely what you want for latency-sensitive paths — you do not want half your checkout calls crossing a region boundary in steady state. The service.cilium.io/affinity annotation fixes this:

metadata:
  annotations:
    service.cilium.io/global: "true"
    # Prefer local backends; spill to remote only if no healthy local exist
    service.cilium.io/affinity: "local"

affinity: local is the workhorse for active/active with regional failover: traffic stays in-cluster for latency and egress cost, and the mesh silently fails the Service over to the peer cluster only when local endpoints disappear. There is also service.cilium.io/shared: "false", which lets a cluster consume a global service but not export its own backends into it — useful for a cluster that should call a shared service without advertising its own pods as backends.

A design rule that saves incidents: keep stateful backends local. Global services are an L3/L4 endpoint union with no awareness of data locality. Federating a database Service so writes can land in either region is a correctness bug, not a feature. Make stateless, idempotent services global; pin stateful ones with affinity: local or keep them out of the mesh entirely.

4. Egress Gateway: fixed source IPs for partner allowlists

The recurring enterprise problem: a partner, a payment processor, or a legacy mainframe will only accept connections from a short list of source IPs. Pods get ephemeral, node-dependent addresses that change on every reschedule, so they cannot be allowlisted. Egress Gateway solves exactly this — it forces traffic from selected pods, to selected destinations, to leave the cluster SNAT’d to a stable IP owned by a designated gateway node.

Prerequisites: enable the feature and bpf-masquerade (Egress Gateway depends on eBPF masquerading).

helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
  --set egressGateway.enabled=true \
  --set bpf.masquerade=true \
  --set kubeProxyReplacement=true

Define the policy. This says: pods labelled app=billing in namespace shop, when talking to the partner CIDR, egress via the node matching egress-node=true, SNAT’d to 203.0.113.10.

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: billing-to-partner
spec:
  selectors:
    - podSelector:
        matchLabels:
          io.kubernetes.pod.namespace: shop
          app: billing
  destinationCIDRs:
    - "198.51.100.0/24"        # the partner's network
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"
    egressIP: 203.0.113.10     # must live on an interface of the gateway node

The egressIP must be a real address configured on an interface of the gateway node (commonly a secondary IP). The gateway node becomes a chokepoint and a SPOF for that traffic class, so in production you select a small set of gateway nodes and front the egress IP with a mechanism that can move it on failure — on bare metal, BGP/L2 announcement (section 6); in cloud, a floating/secondary IP you can reassign.

Sharp edge: only traffic matching both the pod selector and a destinationCIDR is redirected and SNAT’d. Everything else egresses normally with the node’s IP. Do not write 0.0.0.0/0 as the destination unless you genuinely intend to funnel all egress from those pods through one node — that is a bandwidth and blast-radius decision, not a default. Also note return traffic must route back to the gateway node, so the egress IP and the gateway node must share a subnet/routing domain that the upstream network honors.

5. BGP control plane: advertise PodCIDR and LoadBalancer IPs

In native-routing or bare-metal clusters, the physical network does not inherently know how to reach pod IPs or LoadBalancer VIPs. Cilium’s BGP control plane peers each node with your routers (ToR switches, a route reflector) and advertises that reachability dynamically. No more static routes that rot when a node is replaced.

The v2 API splits configuration into composable resources: a CiliumBGPClusterConfig (which nodes peer, with whom), a CiliumBGPPeerConfig (timers, families, reusable peer template), and CiliumBGPAdvertisement (what to advertise). Enable the feature first:

helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
  --set bgpControlPlane.enabled=true

Define a reusable peer config — graceful restart matters so a Cilium agent restart does not blackhole traffic while sessions re-establish:

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeerConfig
metadata:
  name: tor-peer
spec:
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: bgp        # ties to the Advertisement below

Cluster config: which nodes peer, and the router’s ASN/address. peerASN is the upstream router; localASN is what Cilium presents. Use private ASNs (64512–65534) unless your network team assigned one.

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchLabels:
      bgp-enabled: "true"        # only label nodes that should peer
  bgpInstances:
    - name: instance-65000
      localASN: 65000
      peers:
        - name: tor-switch
          peerASN: 64512
          peerAddress: 10.0.0.1          # the ToR/router
          peerConfigRef:
            name: tor-peer

Finally, what to advertise. Here both PodCIDR and LoadBalancer service IPs:

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPAdvertisement
metadata:
  name: pod-and-lb
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: PodCIDR
    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchLabels:
          announce: "bgp"        # only Services with this label get advertised

The Service advertisement only fires for Service objects that carry the matching label and have an assigned LoadBalancer IP — which is where IPAM comes in.

6. LoadBalancer IPAM and L2 announcements for bare metal

On bare metal there is no cloud controller to fill in status.loadBalancer.ingress, so type: LoadBalancer Services sit <pending> forever. Cilium ships its own IPAM to hand out those IPs from pools you define — this is the MetalLB-equivalent, built into Cilium.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: lb-pool
spec:
  blocks:
    - cidr: "203.0.113.0/24"
  serviceSelector:
    matchLabels:
      announce: "bgp"            # scope this pool to Services we also advertise

Now a LoadBalancer Service gets an IP from the pool automatically. You then make it reachable one of two ways:

• BGP (section 5): the assigned IP is advertised as a /32, routers learn it, traffic reaches whichever node the eBPF LB is happy with. This is the scalable, ECMP-friendly path.
• L2 announcement: Cilium answers ARP/NDP for the VIP on a chosen node — no router config required, but it is single-node-active (failover is an ARP move) and confined to one L2 segment. Good for small clusters or where you cannot get BGP peering from the network team.

apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: l2-default
spec:
  loadBalancerIPs: true          # announce LB IPs (not externalIPs here)
  interfaces:
    - ^eth[0-9]+$                 # interfaces eligible to answer ARP
  nodeSelector:
    matchLabels:
      l2-announce: "true"

L2 announcement needs kubeProxyReplacement=true and the l2announcements feature enabled; it uses a leader-election lease per Service, so exactly one node answers at a time. Pick BGP for scale and ECMP, L2 for simplicity when BGP peering is not available. Do not enable both for the same VIP. The bare-metal LB story — MetalLB, kube-vip, BGP vs L2 trade-offs — is covered end to end in Deploy MetalLB and kube-vip for bare-metal Kubernetes; Cilium’s LB-IPAM plays the same role as MetalLB but inside the CNI.

7. Securing cross-cluster traffic with cluster-aware policy

Federating clusters quietly widens your blast radius: a global service means a pod in cluster-west can now reach a backend in cluster-east. Standard CiliumNetworkPolicy still applies on the receiving side, and you should tighten it to be cluster-aware rather than leaving the mesh open. Cilium exposes the source cluster as the well-known label io.cilium.k8s.policy.cluster:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: checkout-allow-east-only
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: checkout
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
            io.cilium.k8s.policy.cluster: cluster-east   # only this cluster
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP

This permits frontend to reach checkout only when the caller originates in cluster-east, even though checkout is a global service reachable from the whole mesh. Treat the mesh as you would any trust expansion: default-deny on the receiving namespace, then allow named cross-cluster flows explicitly, qualified by source cluster. The identity model makes this exact and durable — it survives pod IP churn on both sides because it is written in labels.

Going deeper

The seven sections above are enough to ship. This one is for the reader who has to operate the result at 2 a.m. — the internals, the HA edges, the version caveats, and the one thread that ties all three features together.

Cluster Mesh architecture: what actually syncs, and at what scale

The clustermesh-apiserver is a Deployment per cluster that runs an etcd plus an apiserver container. It publishes a subset of that cluster’s state — nodes, security identities, endpoints, and Services marked global — and every remote cluster’s cilium-agent connects to it over mTLS and watches those keys. It is not a full replica of the cluster; it is the minimum needed for another cluster to route to your pods and resolve your global Services.

Two scaling facts follow from that design:

• Connection fan-out. With N clusters, each agent connects to every remote apiserver — an N-way mesh of watch connections. At larger scale, enable KVStoreMesh, which caches remote state in each cluster’s local kvstore so agents read locally instead of holding a live connection to every peer. It trades a little staleness for far fewer long-lived connections.
• Identity pressure. Every distinct label set across the mesh is an identity, and identities are global. High-churn, high-cardinality label sets (per-pod labels, per-build labels) inflate the identity count the apiserver must sync. Keep label sets stable and meaningful.

Tunnel vs native routing, revisited for the cross-cluster case. In tunnel mode, cross-cluster pod traffic is encapsulated node-to-node, so the only requirement is node-to-node IP reachability on the tunnel port (VXLAN 8472 / Geneve 6081). In native routing, the fabric between clusters must actually route the remote PodCIDRs — via VPC peering, a transit gateway, or, yes, BGP advertising each cluster’s PodCIDR to a shared fabric. That is the third time these features pair: native-routed Cluster Mesh often needs the BGP control plane to make remote pod IPs routable.

The standards-based alternative to the annotation. Newer Cilium also supports the Kubernetes Multi-Cluster Services API (MCS-API). Instead of the service.cilium.io/global annotation you create a ServiceExport, and importing clusters see a ServiceImport. Same outcome, portable across implementations:

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: checkout
  namespace: shop

Egress Gateway internals and real HA

Under the hood, the agent programs an eBPF egress policy map: a packet whose (identity, destinationCIDR) matches a policy is redirected to the elected gateway node and SNAT’d to the egressIP; the reply comes back to the gateway and is reverse-translated via conntrack. This is why the feature requires bpf.masquerade=true and kube-proxy replacement — the SNAT and the redirect are the eBPF masquerade path, not iptables.

You have two ways to name the source address. Pin an explicit egressIP, or name an interface and let Cilium use that interface’s address — handy when the gateway node’s stable IP is managed elsewhere:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: billing-ha
spec:
  selectors:
    - podSelector:
        matchLabels:
          io.kubernetes.pod.namespace: shop
          app: billing
  destinationCIDRs:
    - "198.51.100.0/24"
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"     # may match several nodes
    interface: eth1             # use this interface's IP as the source

HA is failover, not load-sharing. If the nodeSelector matches several nodes, Cilium elects a single active gateway per policy (a deterministic choice) and fails over to another matched node if the active one dies. It does not spread a policy’s egress across multiple nodes — there is one active egress IP at a time per policy. To actually distribute egress load, split the traffic into multiple policies (different pod selectors) pointing at different gateway nodes/IPs. And the egress IP has to move on failover: on bare metal, advertise it via BGP or L2 so it re-homes to the surviving node; in cloud, you must reassign the secondary/floating IP yourself (Cilium does not call the cloud API to move it).

How it differs from a cloud NAT gateway is the whole point:

One more sharp edge: with a broad destinationCIDRs, use excludedCIDRs to carve intra-cluster or infra ranges back out of the redirect, so you do not accidentally SNAT node-to-node or metadata traffic through the gateway.

The BGP control plane: v1 → v2, ECMP, BFD, and timers

The older v1 API was a single CiliumBGPPeeringPolicy CRD — node selection, peers, and advertisements all in one object. It still works but is frozen; new work targets v2. The v2 API is deliberately decomposed:

Version caveat: on Cilium 1.16/1.17 these v2 resources are served under cilium.io/v2alpha1 (the group used in every example here). They are on the path to promotion, so run kubectl api-resources | grep -i bgp on your cluster before copying manifests, and match the group your CRDs actually expose.

ECMP is the scale story. When several nodes each advertise the same LoadBalancer /32, the router installs multiple equal-cost next hops and hashes flows across them; whichever node receives a flow runs the eBPF LB and forwards to a backend. That is how you get horizontally scaled north-south load balancing with no appliance. For clean ECMP each node needs a distinct BGP routerID, which is exactly what the per-node override is for:

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPNodeConfigOverride
metadata:
  name: bgp-node-a          # named after the node
spec:
  bgpInstances:
    - name: instance-65000
      routerID: "10.0.1.5"  # unique per node
      peers:
        - name: tor-switch
          localAddress: "10.0.1.5"

Failure detection: BFD vs hold timers. Plain BGP notices a dead peer only when the hold timer expires — commonly 90s (3× a 30s keepalive). That is a 90-second blackhole for anything relying on the withdrawn route. BFD (Bidirectional Forwarding Detection) gives sub-second detection where both Cilium and the router support it; pair it with BGP for unplanned failures, and keep gracefulRestart for planned ones (an agent restart during an upgrade), where the router keeps forwarding on last-known routes for restartTimeSeconds instead of withdrawing. Different problems, different mechanisms — you generally want both.

Auth. Many enterprise ToR configs require a TCP MD5 password on the session. If the router expects MD5 and Cilium is not configured with it, the session hangs at active/connect and never establishes — a classic, silent misconfiguration. Set the password in the peer config/secret to match the router.

LB-IPAM in depth

CiliumLoadBalancerIPPool hands out the addresses that BGP or L2 then advertises. Blocks can be a CIDR or an explicit start/stop range, and a serviceSelector scopes which Services may draw from a pool — so you can keep a “partner-facing” pool distinct from a “general” pool:

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: partner-pool
spec:
  blocks:
    - start: "203.0.113.10"
      stop: "203.0.113.20"
  serviceSelector:
    matchLabels:
      announce: "bgp"

The controller watches type: LoadBalancer Services, matches a pool by selector, assigns a free address, and writes status.loadBalancer.ingress. Overlapping pools are detected and the offending pool is marked conflicting rather than double-allocating. After assignment, BGP (ECMP, multi-node, needs peering) or L2 (ARP/NDP, single-node-active, no router config, one segment) makes the VIP reachable — never both for one VIP.

The kube-proxy-replacement thread that ties it all together

Notice what section 1 quietly set up: all three features depend on the eBPF service LB that kube-proxy replacement provides. Global services resolve cross-cluster because the eBPF ClusterIP backend set can include remote endpoints. Egress Gateway’s redirect and SNAT ride the same eBPF masquerade path. An advertised LoadBalancer VIP reaches a node, but it is the eBPF LB that then picks a healthy backend. If kube-proxy-replacement is partial or false, expect the classic confusing symptoms: global services that only ever resolve locally, egress redirects that misbehave, and advertised VIPs that reach a node but never load-balance. Verifying it is true on every node is the precondition for the whole platform — not an optional optimisation. Cilium’s identity model and kube-proxy replacement are covered from the ground up in Cilium eBPF: NetworkPolicy and Hubble observability.

Verify

Prove each layer independently; debugging a federated, BGP-advertised, egress-pinned cluster is miserable if you cannot isolate which feature broke.

# --- Cluster Mesh ---
cilium clustermesh status --context cluster-east        # peers Ready, tunnels up
# Confirm a global service unioned backends from both clusters:
kubectl exec -n kube-system ds/cilium -- \
  cilium-dbg service list | grep -A4 checkout
# A global service shows backends with IPs from BOTH PodCIDRs.

# --- Egress Gateway ---
kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf egress list
# Then from a billing pod, hit a destination that echoes the source IP:
kubectl exec -n shop deploy/billing -- curl -s https://ifconfig.me
# Must print the egressIP (203.0.113.10), not a node IP.

# --- BGP ---
cilium bgp peers                  # Session State: established; per-peer
cilium bgp routes advertised ipv4 unicast    # PodCIDR + /32 LB IPs present

# --- LoadBalancer IPAM ---
kubectl get svc -n shop -o wide   # EXTERNAL-IP populated from the pool, not <pending>

# --- End-to-end flow visibility ---
hubble observe --namespace shop --follow
# Cross-cluster flows are tagged with the source/destination cluster.

For BGP specifically: if Session State is active or idle and never reaches established, it is almost always (a) a wrong peerASN/peerAddress, (b) the node not matching the nodeSelector (check the bgp-enabled label), or © the upstream router not configured to accept the session / expecting MD5 auth. cilium bgp peers shows the negotiated families and received/advertised route counts, which disambiguates “session up but no routes” from “no session.”

Enterprise scenario

A payments platform team ran two regional EKS-on-self-managed-nodes clusters (us-east, us-west) behind a single product. Two constraints collided. First, their acquiring bank’s firewall allowlisted exactly two source IPs and refused to add more — so every settlement call, from any cluster, had to appear to come from one of those two addresses. Second, the business wanted regional active/active: a settlement request landing in us-west should be served locally for latency, but fail over to us-east if the west settlement workers were down, without a DNS change.

They solved it by composing the three features. Cluster Mesh federated the clusters, and the settlement Service was made global with service.cilium.io/affinity: "local" so steady-state traffic stayed regional and failover to the peer cluster was automatic. The two allowlisted bank IPs were configured as egress IPs on a pair of dedicated gateway nodes per region, and a CiliumEgressGatewayPolicy funneled only the settlement pods, only to the bank’s CIDR, through them. The gateway IPs themselves were advertised via the BGP control plane so they survived gateway-node replacement without the bank ever seeing a new source.

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: settlement-egress
spec:
  selectors:
    - podSelector:
        matchLabels:
          io.kubernetes.pod.namespace: payments
          app: settlement
  destinationCIDRs:
    - "192.0.2.0/24"            # the bank's published ingress range
  egressGateway:
    nodeSelector:
      matchLabels:
        role: egress-gw
    egressIP: 198.51.100.7      # one of the two bank-allowlisted IPs

The decisive design choice was scoping destinationCIDRs to the bank’s range only. An earlier draft used 0.0.0.0/0, which pushed all egress from the settlement pods — telemetry, package pulls, DNS to external resolvers — through the two gateway nodes and saturated them during a deploy. Narrowing the CIDR to just the bank dropped gateway throughput by an order of magnitude and made the SPOF acceptable. The failover affinity earned its keep three weeks later when a us-west node group rolled badly: settlement traffic shifted to us-east automatically, the bank saw the same two source IPs throughout, and no customer transaction failed.

Practice challenges

Work these in order — they escalate from beginner to advanced. Try each before opening the solution. If you have no cluster, treat them as design exercises: write the manifest and predict the verification output.

1. (Beginner) State the two Cluster Mesh invariants and verify one. Before connecting cluster-east and cluster-west, name the two things that must be true, and give the command that confirms kube-proxy replacement is active on a cluster.

<details> <summary>Solution</summary>

The invariants are unique, stable cluster.id (1–255) and cluster.name per cluster and non-overlapping PodCIDRs across the mesh. Confirm kube-proxy replacement with:

cilium config view | grep -i kube-proxy-replacement   # want: true

Why: global services and Egress Gateway both ride the eBPF service LB that kube-proxy replacement provides; a cluster-id collision or a PodCIDR overlap silently breaks routing and identity. </details>

2. (Beginner → Intermediate) Consume a global service without exporting your own pods. cluster-west should be able to call the global catalog Service but must not advertise its own catalog pods as backends. Write the Service.

<details> <summary>Solution</summary>

apiVersion: v1
kind: Service
metadata:
  name: catalog
  namespace: shop
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/shared: "false"
spec:
  selector:
    app: catalog
  ports:
    - port: 8080

Why: global: "true" lets the cluster resolve the unioned backend set; shared: "false" stops it contributing its own endpoints into that set — consume, do not export. </details>

3. (Intermediate) Two services, two affinities. checkout is latency-sensitive and must stay regional with automatic failover. reporting is a batch job that should prefer the other cluster’s spare capacity. Give the annotation for each.

<details> <summary>Solution</summary>

checkout uses service.cilium.io/affinity: "local" (regional steady state, fail over only when no local endpoint remains). reporting uses service.cilium.io/affinity: "remote":

apiVersion: v1
kind: Service
metadata:
  name: reporting
  namespace: shop
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/affinity: "remote"
spec:
  selector:
    app: reporting
  ports:
    - port: 9090

Why: affinity is the per-Service lever. local optimises latency/egress cost; remote deliberately offloads to the peer, falling back locally only when no remote endpoint is healthy. </details>

4. (Intermediate) Scope an egress pin correctly. Pin only billing-pod traffic to the partner through a gateway node, and explain why the destination CIDR must not be 0.0.0.0/0.

<details> <summary>Solution</summary>

Use a CiliumEgressGatewayPolicy with destinationCIDRs set to the partner’s range (e.g. 198.51.100.0/24), a podSelector for app=billing, and a nodeSelector for the gateway node (see section 4). Writing 0.0.0.0/0 would funnel all egress from those pods — telemetry, image pulls, DNS — through one gateway node, saturating it and turning a scoped chokepoint into a cluster-wide SPOF. Only podSelector ∩ destinationCIDR is redirected, so a tight CIDR keeps blast radius small. </details>

5. (Advanced) Advertise a VIP with ECMP across two nodes. A LoadBalancer Service must be reachable via BGP with equal-cost paths through nodes A and B. What has to be true, and why does the router show two next-hops?

<details> <summary>Solution</summary>

Label both nodes bgp-enabled: "true" so both peer; both must establish sessions with the ToR; and a CiliumBGPAdvertisement of type Service/LoadBalancerIP (label-selected) must match the Service. Each node advertises the same /32, so the router installs two equal-cost next-hops and hashes flows across them — that is ECMP. Give each node a distinct routerID via CiliumBGPNodeConfigOverride for clean sessions. Advertising ExternalIP too is a one-line addition:

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPAdvertisement
metadata:
  name: lb-only
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
          - ExternalIP
      selector:
        matchLabels:
          announce: "bgp"

Why: ECMP is emergent — it happens precisely because N nodes advertise the identical prefix and the router load-shares. No node is “the” LB. </details>

6. (Advanced) Debug a stuck BGP session. cilium bgp peers shows Session State: active and it never reaches established, even though the CRDs applied cleanly. List the three most likely causes and how you would confirm each.

<details> <summary>Solution</summary>

(a) Wrong peerASN/peerAddress — compare the CRD against the router’s neighbour config. (b) Node not selected — the node lacks the bgp-enabled label, so no session is even attempted; check the nodeSelector and kubectl get nodes --show-labels. © Router not accepting the peer / expecting MD5 — the ToR has no matching neighbour statement, or requires a TCP-MD5 password Cilium is not sending; check with the network team and the router’s BGP logs. active (as opposed to idle) means Cilium is trying to connect and the far side is not completing the handshake — that points hard at (a) or ©. </details>

Common beginner mistakes

These are misconceptions, not just symptoms — each one comes from an incorrect mental model, so the fix is a corrected model, not just a command.

• “I’ll just NAT between clusters if the PodCIDRs overlap.” Cluster Mesh routes pod-to-pod by real IP and aliases identities by cluster-id; overlapping PodCIDRs are unroutable and there is no NAT shim to save you. Right model: plan disjoint PodCIDRs at install. Overlap is not fixable post-hoc without re-IPing a cluster.
• “One gateway node is fine.” A single Egress Gateway node is a SPOF, and if the egress IP cannot move, failing the node over accomplishes nothing. Right model: a small set of gateway nodes plus a movable egress IP (BGP/L2 on bare metal, or an automated cloud IP reassignment) and tight destinationCIDRs so one node is never saturated.
• “I applied the BGP CRDs — why are there no routes?” BGP is a two-sided handshake. Cilium peering does nothing until the ToR is configured to accept the neighbour (matching ASN, neighbour statement, sometimes MD5). Right model: the session must reach established before any prefix is exchanged; coordinate with the network team and watch cilium bgp peers.
• “I copied the same Helm values to both clusters.” Identical cluster.id/cluster.name on two clusters aliases their identities, so policy verdicts apply to the wrong workloads — silently. Right model: unique name AND id per cluster, set once at install and never changed live.
• “I’ll bump max-connected-clusters on the running mesh.” That flag changes the split between cluster-id bits and identity bits; flipping it live renumbers identities under running policy and causes an outage. Right model: choose the mode at install, uniform across every cluster.
• “Make everything global.” Global services are an L3/L4 endpoint union with no data-locality awareness; federating a stateful/database Service so writes can land in either region is a correctness bug. Right model: stateless/idempotent → global; stateful → affinity: local or out of the mesh.
• “The Egress Gateway redirects all the pod’s traffic.” Only flows matching both the pod selector and a destinationCIDR are SNAT’d; everything else uses the node IP. Right model: it is selective by design — verify with the source-echo curl and expect the egress IP only for partner-bound traffic.

Checklist

• Every cluster has a unique, stable cluster.id (1–255) and cluster.name, set at install via Helm.
• PodCIDRs are non-overlapping across all meshed clusters (verified before connecting).
• kube-proxy replacement is true on every cluster that uses global services or Egress Gateway.
• Global services are limited to stateless/idempotent workloads; stateful backends use affinity: local or stay out of the mesh.
• Latency-sensitive global services set service.cilium.io/affinity: "local" for regional failover instead of even cross-cluster spread.
• Egress Gateway policies scope destinationCIDRs to the specific partner/processor range — never 0.0.0.0/0 by default.
• egressIP is a real, reachable address on the gateway node(s); gateway nodes are a known SPOF with a failover plan.
• BGP peers reach established; PodCIDR and labelled LoadBalancer IPs appear in cilium bgp routes advertised.
• gracefulRestart is enabled on BGP peer configs so agent restarts do not blackhole advertised routes.
• CiliumLoadBalancerIPPool is defined for bare metal; each VIP uses BGP or L2 announcement, not both.
• Receiving namespaces are default-deny, with cross-cluster ingress allowed explicitly and qualified by io.cilium.k8s.policy.cluster.
• Verification covered each layer in isolation: cilium clustermesh status, cilium-dbg bpf egress list, cilium bgp peers, Hubble flows.

Glossary

• CNI (Container Network Interface): the plugin that gives pods their networking. Cilium is a CNI, but this lesson is about the platform features layered on top of it.
• eBPF: the in-kernel technology Cilium uses to forward and filter traffic without iptables. The “one engine” under all three features.
• Security identity: a small integer Cilium derives from a pod’s labels; policy is enforced on identities, not IPs. Made globally unique across a mesh by combining it with the cluster-id.
• kube-proxy replacement: Cilium doing Service load balancing in eBPF instead of kube-proxy’s iptables/IPVS. Prerequisite for global services, Egress Gateway, and eBPF LB behind advertised VIPs.
• Tunnel vs native routing: tunnel mode encapsulates pod traffic (VXLAN/Geneve); native routing sends it unencapsulated and expects the network to know the PodCIDRs (the natural pairing with BGP).
• Cluster Mesh: connecting two or more Cilium clusters so Services, identities, and policy span all of them.
• clustermesh-apiserver: the per-cluster etcd + apiserver that publishes a cluster’s state for remote clusters to read.
• KVStoreMesh: an option that caches remote mesh state locally to cut per-agent cross-cluster connections at scale.
• cluster-id / cluster-name: the unique integer (1–255) and name each cluster must have; collisions alias identities across the mesh.
• Global service: a Service annotated service.cilium.io/global: "true" (or exported via MCS-API) whose backends are unioned across clusters.
• Affinity (local/remote/none): the per-Service lever that decides whether a global service prefers local backends, remote backends, or spreads evenly.
• shared: "false": a global service that a cluster consumes but does not contribute backends to.
• MCS-API (ServiceExport/ServiceImport): the Kubernetes standard for multi-cluster services; Cilium’s portable alternative to the global annotation.
• Egress Gateway: forcing selected pods, to selected destinations, to leave the cluster SNAT’d to a stable IP on a gateway node.
• SNAT (Source NAT): rewriting the source IP of a packet — here, to the fixed egressIP a partner can allowlist.
• egressIP / gateway node: the stable address, and the node that owns it, through which pinned egress traffic leaves.
• destinationCIDRs / excludedCIDRs: which destination ranges an egress policy applies to, and which to carve back out.
• BGP (Border Gateway Protocol): how routers exchange reachability; Cilium’s control plane speaks it to advertise cluster routes.
• ASN (localASN/peerASN): the autonomous-system numbers identifying Cilium and the router in a BGP session (use private 64512–65534 unless assigned one).
• ToR (top-of-rack): the switch/router each node peers with.
• PodCIDR: the IP range a node/cluster allocates pod addresses from; advertised over BGP so the fabric can route pods.
• LoadBalancer IP / LB-IPAM (CiliumLoadBalancerIPPool): Cilium’s built-in allocator that hands type: LoadBalancer Services an IP on bare metal (the MetalLB-equivalent).
• L2 announcement (CiliumL2AnnouncementPolicy): answering ARP/NDP for a VIP on one elected node — simple, single-node-active, no router config.
• ECMP (Equal-Cost Multi-Path): when many nodes advertise the same /32, the router load-shares across them — Cilium’s scale-out north-south LB.
• BFD (Bidirectional Forwarding Detection): a sub-second link-failure detector that avoids waiting on BGP’s ~90s hold timer.
• Graceful restart: a BGP feature letting the router keep forwarding on last-known routes during a Cilium agent restart, avoiding a blackhole.
• CiliumBGPClusterConfig / PeerConfig / Advertisement / NodeConfigOverride: the composable v2 BGP CRDs — who peers, reusable peer settings, what to advertise, and per-node overrides.
• Hubble: Cilium’s flow-observability layer; cross-cluster flows are tagged with source/destination cluster.