Rollouts Traffic Management: NGINX, Istio, ALB & the Gateway API

In a nutshell

A canary release ships a new version to a small slice of real users first, watches it, and only widens the slice if it behaves. This lesson is about the one piece that makes that slice honest: the traffic router. Argo Rollouts is the decision-maker (“send 20% to the new version”), but on its own it can only change how many pods run the new version — and a pod count is a clumsy stand-in for a traffic percentage. To split real requests exactly, Rollouts hands the weight to a router — NGINX, Istio, an AWS ALB, or a Gateway API implementation — and the router does the actual splitting.

Think of a dimmer switch versus a bank of light bulbs. Without a router you can only “dim the room” by switching whole bulbs on and off: with five bulbs the finest step is 20%, and if someone adds bulbs the brightness drifts. A traffic router is a real dimmer — it sets exactly 20% no matter how many bulbs are installed. setWeight: 20 without a router flips bulbs; setWeight: 20 with a router turns the dimmer to 20.

The twist that makes this an expert lesson: the “dimmer” lives at your cloud’s networking edge, the least portable part of Kubernetes. The same Rollout object drives an ALB on EKS, an in-cluster NGINX/Istio on AKS (Application Gateway has no native integration), and a Gateway API HTTPRoute on GKE. Learn the four routers and the three cloud edges and you can run progressive delivery anywhere.

Level: Expert · Time: ~54 min

Prerequisites & outcomes: You should already know the Rollout object and canary/blue-green basics from Argo Rollouts: Canary & Blue-Green Deployments. After this you can tell a real traffic canary from a pod-ratio one, wire trafficRouting for NGINX, Istio, ALB and the Gateway API plugin, map each onto EKS/AKS/GKE, and reach for header routing, mirroring and canary-scale decoupling when a plain percentage is not enough.

A canary deployment is a promise: send a small slice of real users to the new version, watch it, and only widen the slice if it behaves. The trap almost everyone falls into is believing Argo Rollouts keeps that promise the moment you write setWeight: 20. It does not. On its own, that line changes a pod ratio, and a pod ratio is only a rough, quantised approximation of a traffic percentage. To shift a true 20% of requests — not 33%, not 50%, but twenty — you need a traffic router: NGINX, Istio, an AWS ALB, or a Gateway API implementation, told by Rollouts exactly how to weight the split. This lesson is about that layer, and about the fact that the router is a cloud edge — so it looks different on EKS, AKS and GKE.

Why this matters

Every serious progressive-delivery story eventually collides with one question from a skeptical SRE: “when you say 20% canary, is that 20% of traffic or 20% of pods?” If you cannot answer crisply, your canary is theatre. A canary that routes by pod count can only move in coarse steps (with five replicas the smallest non-zero slice is 20%), it drifts every time the Horizontal Pod Autoscaler resizes the fleet, and it gives you no way to do header-based or mirrored routing at all. The difference between “roughly a fifth of pods” and “exactly a fifth of requests, verifiable at the load balancer” is the difference between a demo and a control you would stake a Friday deploy on.

The reason this is a whole lesson — and an expert-tier one — is that the router lives at the networking edge, and the edge is the least portable part of Kubernetes. Argo Rollouts is beautifully cloud-agnostic right up until trafficRouting, at which point the honest answer to “how do I weight traffic?” becomes “which mesh or ingress or managed load balancer are you on?” On EKS that might be the AWS Load Balancer Controller driving an ALB’s weighted target groups; on AKS it is usually NGINX or Istio in front of Application Gateway, because AGIC has no native Rollouts integration; on GKE it is increasingly the Gateway API plugin driving a Google Cloud Load Balancer. Same Rollout object, three different edges. Getting this mapping right — and knowing which combinations are mature versus experimental — is the skill that separates “I followed a tutorial” from “I run progressive delivery across three clouds.”

The mental model to hold: Rollouts is the brain, the router is the hands. Rollouts decides the weight at each step; the router is the thing that actually splits packets. If the hands are missing or mis-wired, the brain’s decisions never reach real users, and setWeight: 20 quietly degrades into “scale the canary ReplicaSet to about a fifth and hope kube-proxy load-balances evenly.” This lesson wires up real hands, four different kinds, and then shows you which pair of hands each cloud hands you.

setWeight is a lie without a router: pods vs real traffic

Start with the uncomfortable truth. A Rollout with a plain canary strategy and no trafficRouting does not touch your network path at all. It manipulates two ReplicaSets — a stable one and a canary one — behind a single Service. When you say setWeight: 20, the controller scales the canary ReplicaSet to roughly 20% of the desired replicas and the stable ReplicaSet to the remaining 80%. The Service has one selector that matches both ReplicaSets’ pods, so traffic distribution is whatever kube-proxy (or your CNI’s service proxy) does across the combined endpoint list — approximately even per endpoint, which only approximates your intended weight because it is really “1 canary pod out of 5 total.”

That approximation breaks in three ways that matter in production:

Here is the arithmetic that makes it concrete. Suppose desired replicas: 4 and you want a 10% canary. Pod-ratio math forces you to round: 10% of 4 is 0.4 pods, which Rollouts rounds up to 1 canary pod — that is 25% of pods, not 10%, and with per-connection balancing your canary may see anywhere from 15% to 35% of requests depending on connection reuse. Now add trafficRouting. The controller still runs a canary pod (it must, to have somewhere to send traffic), but the split is enforced by the router at exactly 10%, regardless of whether that canary pod is 25% or 5% of the fleet. The replica count becomes a capacity decision; the weight becomes a routing decision; the two are finally decoupled.

That decoupling is controlled by a small set of canary knobs you should know before touching any provider:

The canaryService/stableService pair is the linchpin. Without a router, one Service fronts everything. With a router, Rollouts injects the rollouts-pod-template-hash label into each of those two Services’ selectors so that canaryService resolves to only the new pods and stableService to only the old ones. The router then has two clean backends to weight between. If you forget to create these two Services, or their selectors overlap and both match every pod, the router faithfully splits traffic — to two backends that are secretly identical, and your “canary” is meaningless. That selector hygiene is the single most common wiring bug, and we will hit it again in troubleshooting.

The trafficRouting field: one canary, many routers

trafficRouting lives at spec.strategy.canary.trafficRouting and is a oneof-ish map: you populate exactly one provider block (plus optional managedRoutes), and that provider becomes the thing Rollouts programs at every setWeight. The full menu of first-class and plugin providers:

Two structural rules cut across every provider. First, all of them need the canary/stable Service model — a canaryService, a stableService, and for some a root Service that receives all traffic before the split. Second, capabilities differ: not every router can do header routing or mirroring, and that gap decides which advanced steps you can use. Keep this capability matrix nearby; it will save you from writing a setMirrorRoute step against a provider that cannot honour it.

And the Service model, provider by provider, because “which Services do I actually create?” is a question every one of these raises:

With the shape of the field understood, we can wire each router for real. The pattern repeats: define the two Services, define the router’s native object, and point trafficRouting.<provider> at it. Everything else is detail — but the detail is where trust is won or lost, so we go field by field.

NGINX Ingress: canary annotations and the shadow ingress

ingress-nginx has a built-in canary feature driven entirely by annotations, and the Rollouts NGINX provider is a thin, clever wrapper over it. You author one Ingress — the stable Ingress, pointing at your stableService. Rollouts, at sync time, generates a second, shadow Ingress — the canary Ingress — that is a near-copy of your stable one but carries nginx.ingress.kubernetes.io/canary: "true" and nginx.ingress.kubernetes.io/canary-weight: "<N>", and points at your canaryService. ingress-nginx sees the two Ingresses for the same host/path, and routes N% of matching requests to the canary backend. You never write the canary Ingress; you never hand-edit the weight; Rollouts owns both.

The trafficRouting.nginx block is small:

A complete, schema-correct NGINX Rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
  namespace: demo
spec:
  replicas: 5
  strategy:
    canary:
      canaryService: rollouts-demo-canary   # Service selecting only canary pods
      stableService: rollouts-demo-stable    # Service selecting only stable pods
      trafficRouting:
        nginx:
          stableIngress: rollouts-demo-stable # the Ingress you author, below
      steps:
        - setWeight: 10
        - pause: {}                            # hold for a human or an analysis
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
        - name: rollouts-demo
          image: argoproj/rollouts-demo:blue
          ports:
            - containerPort: 8080

The two Services are ordinary ClusterIP Services; Rollouts injects the pod-template-hash into their selectors at runtime, so you write only the app label:

apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-stable
  namespace: demo
spec:
  selector:
    app: rollouts-demo        # Rollouts adds rollouts-pod-template-hash here
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-canary
  namespace: demo
spec:
  selector:
    app: rollouts-demo        # same base selector; hash makes it canary-only
  ports:
    - port: 80
      targetPort: 8080

And the stable Ingress — the one you author, pointing at the stable Service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rollouts-demo-stable
  namespace: demo
spec:
  ingressClassName: nginx
  rules:
    - host: rollouts-demo.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: rollouts-demo-stable
                port:
                  number: 80

When the Rollout reaches setWeight: 10, inspect the generated Ingress and you will see the canary annotations Rollouts stamped on it. The generated Ingress is named after the stable one with a canary suffix:

# List the ingresses; note the second, generated one
kubectl -n demo get ingress
# NAME                                     CLASS   HOSTS                 ...
# rollouts-demo-stable                     nginx   rollouts-demo.local   ...
# rollouts-demo-stable-rollouts-demo-canary  nginx rollouts-demo.local   ...   (generated)

# The generated canary Ingress carries the weight (representative output)
kubectl -n demo get ingress rollouts-demo-stable-rollouts-demo-canary \
  -o jsonpath='{.metadata.annotations}' | tr ',' '\n'
# "nginx.ingress.kubernetes.io/canary":"true"
# "nginx.ingress.kubernetes.io/canary-weight":"10"

The canary annotation family is worth memorising, because additionalIngressAnnotations lets you use the header/cookie variants for targeted testing even though the NGINX provider does not implement the setHeaderRoute step:

The NGINX provider has a well-known sharp edge: weight precision and total-weight are ingress-controller settings, not Rollouts settings. ingress-nginx defaults to integer weights out of 100. If you need sub-1% canaries you must raise nginx.ingress.kubernetes.io/canary-weight-total (and a matching controller flag) — Rollouts will happily set canary-weight: 0 for a setWeight: 0 step, but a setWeight finer than the controller’s granularity silently rounds. Test your smallest intended step against the real controller before you trust it in prod.

Istio: VirtualService weights and subsets

Istio is the most capable router in this lesson: it does weighting, header routing, and true traffic mirroring, all at L7, all per-request. The cost is that every pod in the path needs an Istio sidecar (or you run ambient mode), which means the namespace must be mesh-enrolled. The Rollouts Istio provider does not invent any Istio concept — it edits the weight fields inside a VirtualService you author, and optionally flips pods between DestinationRule subsets.

There are two ways to model the split, and choosing between them is the first Istio decision:

The trafficRouting.istio fields:

A host-based Rollout and its VirtualService:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
  namespace: demo
spec:
  replicas: 5
  strategy:
    canary:
      canaryService: rollouts-demo-canary
      stableService: rollouts-demo-stable
      trafficRouting:
        istio:
          virtualService:
            name: rollouts-demo-vsvc
            routes:
              - primary            # the named HTTP route to manage
      steps:
        - setWeight: 20
        - pause: { duration: 2m }
        - setWeight: 60
        - pause: { duration: 2m }
        - setWeight: 100
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
        - name: rollouts-demo
          image: argoproj/rollouts-demo:blue
          ports:
            - containerPort: 8080

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: rollouts-demo-vsvc
  namespace: demo
spec:
  gateways:
    - rollouts-demo-gateway
  hosts:
    - rollouts-demo.local
  http:
    - name: primary               # matches trafficRouting.istio ... routes[0]
      route:
        - destination:
            host: rollouts-demo-stable
          weight: 100              # Rollouts drives this down
        - destination:
            host: rollouts-demo-canary
          weight: 0                # ...and this up

You author the weights as 100/0; Rollouts rewrites them at every step. The critical, non-obvious rule: the route name in the VirtualService must match a name in routes, and the two destination entries must already exist — Rollouts edits weights, it does not add or remove destinations. If your VirtualService has a single unnamed route, you may omit routes, but naming it is far less error-prone.

The subset variant swaps the two hosts for one host plus a DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: rollouts-demo-destrule
  namespace: demo
spec:
  host: rollouts-demo               # one Service fronting all pods
  subsets:
    - name: stable
      labels: {}                    # Rollouts injects rollouts-pod-template-hash
    - name: canary
      labels: {}                    # ...into these subset label maps

# In the Rollout, reference the DestinationRule:
trafficRouting:
  istio:
    virtualService:
      name: rollouts-demo-vsvc
      routes: [primary]
    destinationRule:
      name: rollouts-demo-destrule
      canarySubsetName: canary
      stableSubsetName: stable

Knowing which parts of the VirtualService you own versus which Rollouts rewrites keeps the two from fighting — anything Rollouts owns, you must not template or a sync will thrash:

Istio’s most infamous canary bug is a host/gateway mismatch you cannot see in the Rollout. If the client’s Host: header (or the Gateway’s hosts) does not exactly match the VirtualService hosts, Istio never applies the route, the weights are irrelevant, and 100% of traffic sails through on the default routing — your canary receives zero. ActualWeight in kubectl argo rollouts get will say 20 because Rollouts set the VirtualService correctly; the mesh simply isn’t matching it. Always verify with istioctl proxy-config route <pod> that the route and its weights are actually programmed into Envoy, not just present in the VirtualService YAML.

AWS ALB: weighted target groups and the action annotation

On EKS the native answer is the AWS Load Balancer Controller, which turns an Ingress into a real Application Load Balancer. The ALB supports weighted target groups natively, and Rollouts drives them through the controller’s action annotation. This is the most AWS-specific wiring in the lesson, so go slowly.

The controller reads an annotation of the form alb.ingress.kubernetes.io/actions.<action-name>, whose value is a JSON forward action listing target groups with weights. The Ingress rule’s backend then references that action by name (with port: use-annotation). Rollouts, at each setWeight, rewrites the two weights inside that JSON. The rootService field tells Rollouts which Service/action is the entry point.

The trafficRouting.alb fields:

A representative ALB Ingress. Note target-type: ip — weighted target groups require IP targets (pods registered directly), not instance targets:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rollouts-demo-ingress
  namespace: demo
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip           # required for weighting
    # The action Rollouts rewrites; you seed it, Rollouts owns the weights:
    alb.ingress.kubernetes.io/actions.rollouts-demo-root: >
      {"type":"forward","forwardConfig":{"targetGroups":[
        {"serviceName":"rollouts-demo-stable","servicePort":"80","weight":100},
        {"serviceName":"rollouts-demo-canary","servicePort":"80","weight":0}]}}
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: rollouts-demo-root      # the action name above
                port:
                  name: use-annotation        # tells the controller to read the annotation

# The matching Rollout trafficRouting block:
trafficRouting:
  alb:
    ingress: rollouts-demo-ingress
    servicePort: 80
    rootService: rollouts-demo-root

The forward action’s JSON is the thing Rollouts edits in place; know its shape so you can read it in a debug session:

Two annotations, two behaviours, and one CRD, all worth a row:

The ALB provider’s classic failure is silent: weights that never apply. The three usual causes are (1) the AWS Load Balancer Controller isn’t installed or lacks IAM to modify the ALB, so the annotation changes but the ALB does not; (2) target-type is instance, so you are weighting nodes and every node runs both versions; or (3) rootService/action name mismatch, so Rollouts edits an annotation the Ingress rule never references. Check the controller logs and confirm the ALB’s listener rule shows two target groups with the expected weights in the AWS console or aws elbv2 describe-rules — the cluster-side annotation is necessary but not sufficient.

SMI, Gateway API, and the plugin ecosystem

Three more routing worlds round out the picture — one fading, one ascending, and a long tail reached through plugins.

SMI (TrafficSplit) — legacy, know it, avoid it for greenfield. The Service Mesh Interface defined a portable TrafficSplit CRD, and Rollouts’ smi provider writes backend weights into it. It was the neutral option when Linkerd and others implemented SMI. The honest status in 2026: SMI is dormant — the project has not moved in years and few meshes still push it. If you already run a TrafficSplit-based setup it works, but do not build new pipelines on it.

trafficRouting:
  smi:
    rootService: rollouts-demo-root        # fronts the TrafficSplit
    trafficSplitName: rollouts-demo-split  # optional; generated if omitted

Gateway API — the emerging standard, via a plugin. The Kubernetes Gateway API (HTTPRoute, Gateway) reached GA (v1) and is the industry’s convergence point for L7 routing — it replaces the annotation soup of Ingress with typed, portable resources, and HTTPRoute backendRefs carry a first-class weight. Rollouts supports it through the argoproj-labs/gatewayAPI plugin, not a built-in provider, so there is an install step. You register the plugin in the argo-rollouts-config ConfigMap and grant RBAC over httproutes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-config
  namespace: argo-rollouts
data:
  trafficRouterPlugins: |-
    - name: "argoproj-labs/gatewayAPI"
      location: "https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/download/v0.5.0/gatewayapi-plugin-linux-amd64"

# The Rollout references the plugin by its registered name:
trafficRouting:
  plugin:
    argoproj-labs/gatewayAPI:
      httpRoute: rollouts-demo-route     # the HTTPRoute to weight
      namespace: demo

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: rollouts-demo-route
  namespace: demo
spec:
  parentRefs:
    - name: rollouts-demo-gateway
  rules:
    - backendRefs:
        - name: rollouts-demo-stable
          port: 80
          weight: 100                    # Rollouts drives these two weights
        - name: rollouts-demo-canary
          port: 80
          weight: 0

The plugin field keys off the exact registered name, and the config keys on this one table:

Gateway API maturity, honestly: the API is GA and stable; the Rollouts plugin is a community (argoproj-labs) component still on a 0.x line, and support for setMirrorRoute/setHeaderRoute depends on both the plugin version and the specific Gateway implementation (Istio’s gateway, Envoy Gateway, NGINX Gateway Fabric, and each cloud’s controller differ). Treat weighting as solid and the advanced routes as “verify on your exact stack.” This is the direction the ecosystem is heading, but it is not yet the safest default on every cloud.

The plugin long tail. The same plugin mechanism reaches routers that never had a built-in provider:

Kong is typically driven via its Ingress or Gateway API support rather than a bespoke weight object, and the community list grows over time — the point is that “my router isn’t in the built-in list” no longer means “I can’t do real traffic canaries.”

The multi-cloud LB edge: ALB, Application Gateway, GCLB

This is the crux of the lesson and the reason it is tagged multi-cloud. Everything above is a router type; a managed Kubernetes service also hands you a cloud load balancer, and the interesting question is how the router maps onto that specific LB. The three big managed platforms answer very differently, and — critically — one of them has no native Rollouts integration at all.

Read the diagram left to right: the Rollout’s trafficRouting sets a weight, a router turns that weight into a real request split, the split lands on whichever managed LB your cloud gives you, and an AnalysisRun decides whether the weight advances or the whole thing rolls back. The per-cloud table is the reference you will actually come back to:

Now the paired blocks — the same canary, expressed three ways, one per cloud edge.

EKS → ALB (native). Use the alb provider straight onto the AWS Load Balancer Controller. This is the smoothest of the three because the ALB weights are a first-class AWS feature:

# EKS: Rollouts drives ALB weighted target groups directly
trafficRouting:
  alb:
    ingress: rollouts-demo-ingress   # ALB Ingress with target-type: ip
    servicePort: 80
    rootService: rollouts-demo-root  # the forward action's root
# Prereqs: AWS Load Balancer Controller installed, IRSA/Pod Identity granting
# elasticloadbalancing:* on the ALB, and target-type=ip on the Ingress.

AKS → Application Gateway (no native provider — front it). This is the honest, load-bearing caveat of the whole lesson. AGIC (the Application Gateway Ingress Controller) has no Argo Rollouts traffic-router provider. There is no appgw key. So on AKS you do not point Rollouts at Application Gateway; you run a router inside the cluster — almost always NGINX or Istio — and let Application Gateway (or a plain Azure Load Balancer) sit in front as the L4/L7 entry point that forwards to the in-cluster router:

# AKS: Rollouts drives NGINX (or Istio) INSIDE the cluster;
# Application Gateway / Azure LB is just the entry point in front of it.
trafficRouting:
  nginx:
    stableIngress: rollouts-demo-stable   # ingress-nginx, exposed via App Gateway/ALB
# There is NO trafficRouting.appgw. Do not expect AGIC to weight for you.
# Emerging native path: Application Gateway for Containers (AGC) speaks Gateway API,
# so AGC + the argoproj-labs/gatewayAPI plugin can weight HTTPRoutes natively.

GKE → GCLB (via Gateway API). Classic GKE Ingress (the gce class) provisions a Google Cloud Load Balancer but cannot weight between backends from annotations. The modern path is the GKE Gateway controller, which implements Gateway API and can weight HTTPRoute backends — so you drive it with the Gateway API plugin:

# GKE: the GKE Gateway controller implements Gateway API; the plugin weights HTTPRoute
trafficRouting:
  plugin:
    argoproj-labs/gatewayAPI:
      httpRoute: rollouts-demo-route   # HTTPRoute bound to a gke-l7-* GatewayClass
      namespace: demo
# Prereqs: Gateway API CRDs + GKE Gateway controller enabled, the plugin registered
# in argo-rollouts-config, and a Gateway using a GKE GatewayClass (e.g. gke-l7-global-external-managed).

The through-line: only EKS gives you a first-class, native trafficRouting provider for its managed LB. AKS makes you run an in-cluster mesh/ingress and treats the cloud LB as plumbing. GKE routes you through Gateway API. If a stakeholder says “we’ll just use the cloud load balancer’s canary feature on all three,” this table is your evidence that the story is uneven and AKS in particular needs an in-cluster router. Standardising on Istio or the Gateway API plugin across all three clouds is the pragmatic way to make your Rollout manifests portable — the router becomes the same object everywhere, and only the entry-point LB differs. The IAM and controller wiring behind each edge is a lesson in itself: the ALB controller’s IRSA setup is covered in Argo CD on EKS: IRSA, Secrets Manager, ECR & ALB, and the GKE Gateway plus Workload Identity path in Argo CD on GKE: Workload Identity, Secret Manager & Artifact Registry.

Before you pick, weigh the operational prerequisites and what each edge actually bills for — the router is rarely free, and the cheapest-to-run option is not always the most portable:

Beyond weight: header routing, mirroring, and canary scale

Weighting is the backbone, but the providers that support it also unlock routing primitives that make canaries far safer than “N% of everyone.” These are expressed as steps, and they require declaring the routes you will manage under trafficRouting.managedRoutes so Rollouts knows to create and clean them up.

Header routing (setHeaderRoute) sends only requests carrying a specific header to the canary — perfect for “route my QA team to v2, everyone else stays on v1” before you shift any percentage of the public:

strategy:
  canary:
    trafficRouting:
      managedRoutes:
        - name: canary-by-header      # declare the route Rollouts manages
      istio:
        virtualService:
          name: rollouts-demo-vsvc
          routes: [primary]
    steps:
      - setHeaderRoute:
          name: canary-by-header
          match:
            - headerName: X-Canary
              headerValue:
                exact: "true"
      - pause: { duration: 30m }       # internal testing window, 0% public traffic
      - setHeaderRoute:                 # remove the header route
          name: canary-by-header
      - setWeight: 10                   # now begin the real percentage rollout

Traffic mirroring (setMirrorRoute), supported by Istio and (implementation-dependent) the Gateway API plugin, shadows a copy of live requests to the canary without the client ever seeing the canary’s response — the ultimate zero-risk smoke test under real load:

steps:
  - setMirrorRoute:
      name: mirror-canary
      percentage: 35                    # mirror 35% of matching requests to canary
      match:
        - method:
            exact: GET                  # mirror only idempotent GETs
  - pause: { duration: 10m }
  - setMirrorRoute:                     # stop mirroring
      name: mirror-canary
  - setWeight: 20

The match grammar these two steps share is worth a table, because it is how you scope which requests get diverted or mirrored:

Canary scale decoupling is the capacity side of the same idea. By default Rollouts scales the canary ReplicaSet to roughly match the weight, but setCanaryScale and dynamicStableScale let you break that link:

setMirrorRoute and setHeaderRoute are the two steps most likely to fail silently on the wrong provider. If you write them against the nginx provider (which implements neither step), Rollouts does not error loudly — the step is a no-op and your “header canary” routes nobody. Cross-check the capability matrix from earlier before you design a pipeline around these, and prefer Istio or the Gateway API plugin when header/mirror routing is a requirement rather than a nicety.

Traffic plus metrics: where analysis comes in

A weighted split answers “how much traffic hits the canary?” It does not answer “is the canary healthy enough to widen the split?” That second question is what turns a traffic router from a manual dial into an autopilot, and it is the job of Rollouts analysis — AnalysisTemplate, ClusterAnalysisTemplate, and the AnalysisRun they spawn. The pattern is to interleave setWeight steps with analysis steps so that each traffic increase is gated on a metric query (Prometheus success rate, latency SLO, a Datadog monitor) rather than a human staring at a dashboard.

steps:
  - setWeight: 10
  - analysis:                          # gate the next step on live metrics
      templates:
        - templateName: success-rate
      args:
        - name: canary-service
          value: rollouts-demo-canary
  - setWeight: 50
  - analysis:
      templates:
        - templateName: success-rate
  - setWeight: 100

The analysis step’s fields are few but load-bearing — this is the seam where traffic hands off to metrics:

When the AnalysisRun a step spawns reports Failed, the Rollout aborts and rolls back to stable automatically; Successful lets the next setWeight proceed. The synergy is the whole point of progressive delivery: traffic gives the canary real users to be judged on; metrics judge it; the two together let promotion and rollback happen without a human in the loop. A 10% weight with no analysis is just a slower way to ship a bug to 10% of users; a 10% weight plus an AnalysisRun watching error rate is a control that catches the bug and rolls back automatically. We build those templates, background vs inline analysis, metric providers, and auto-rollback in Rollouts Analysis: Metrics, AnalysisTemplates & Automated Rollback; for now, hold the shape — a setWeight you trust is one an AnalysisRun is watching. And if you have not yet met the Rollout object and its canary/blue-green strategies from the ground up, Argo Rollouts: Canary & Blue-Green Deployments is the prerequisite that introduces the CRD this lesson has been driving.

Going deeper

The body wired each router in isolation and mapped it to a cloud. Six things the happy path skips will bite you at scale — production nuance, not new theory.

One canary, two routers at once

The body called trafficRouting “oneof-ish,” and for most people it is a single provider block. But it is not literally a oneof: Rollouts can drive multiple providers in the same step. A common real case is a service mesh and an ingress — Istio splitting east-west (service-to-service) traffic while an SMI TrafficSplit handles a Linkerd-based edge. You list both blocks, and at every setWeight Rollouts programs all of them to the same weight:

trafficRouting:
  istio:
    virtualService:
      name: rollouts-demo-vsvc
      routes: [primary]
  smi:
    trafficSplitName: rollouts-demo-split
    rootService: rollouts-demo-root

The catch is capability alignment: a step is only as capable as the least capable provider you listed. Add a setMirrorRoute and it silently no-ops on the providers that cannot mirror while working on the ones that can — so your shadow traffic ends up half-configured. Keep multi-provider setups to setWeight unless every listed provider supports the advanced step.

ALB without the blip: ping-pong services

The ALB section used rootService and a single forward action. That model has a known race: the AWS Load Balancer Controller reconciles the two target groups inside one annotation, and during a weight change there is a window where the ALB briefly serves a stale split. The fix is the ALB provider’s ping-pong mode — two services, ping and pong, that swap stable/canary roles on each rollout so a promotion never rewrites the currently-live target group in place:

trafficRouting:
  alb:
    ingress: rollouts-demo-ingress
    servicePort: 80
    pingPong:
      pingService: rollouts-demo-ping
      pongService: rollouts-demo-pong

With ping-pong you drop rootService; the ping/pong pair is the pair of weighted backends, and Rollouts records in the Rollout status which one is currently stable. It is the production-grade ALB wiring — reach for it whenever a customer-visible blip during promotion is unacceptable.

Gateway API weights are ratios, not percentages

A subtle correctness trap: an HTTPRoute backendRefs[].weight is not a percentage, and the weights need not sum to 100 — the spec defines them as relative proportions. weight: 1 and weight: 3 means 25/75, and weight: 100 / weight: 0 means 100/0 only because the ratio works out. Rollouts writes numbers that produce the ratio you asked for, so you rarely compute this by hand — but when you read an HTTPRoute to debug, do not expect the two weights to add to 100, and never hard-code a “total” the way NGINX’s canary-weight-total works. They are different number systems.

The Gateway API plugin also handles managedRoutes precedence. Header and mirror routes are inserted as additional rules that must out-rank the weighted split, so Rollouts places them ahead of the stable/canary rule in the HTTPRoute, in the order you list them under managedRoutes. List a broad match before a narrow one and the broad rule wins — order managedRoutes most-specific-first, exactly as you would hand-order Gateway API rules.

Draining the edge: scale-down delays

When trafficRouting is in play, Rollouts does not rip down the old ReplicaSet the instant a promotion finishes. Two fields govern the drain:

These exist because the router and the endpoints are only eventually consistent: the ALB or Envoy needs a moment to stop sending to a target group before its pods disappear, or clients hit connection resets. Raise scaleDownDelaySeconds for long-lived connections (gRPC streams, websockets); the default 30s suits ordinary HTTP.

Istio without sidecars: ambient mode

The Istio section assumed the classic sidecar model, and its worst trap — pods that predate the istio-injection label get no proxy. Ambient mesh removes per-pod sidecars entirely: L4 runs through a node-level ztunnel, and L7 features like VirtualService weighting need a waypoint proxy deployed for the namespace or service account. Rollouts is unchanged — it still edits the VirtualService — but enforcement moves from a sidecar to the waypoint. The injection trap becomes a “is there a waypoint for this service?” trap: no waypoint, no L7 weighting, and (like the host/gateway mismatch) ActualWeight reads correct while real traffic ignores it. Verify the waypoint exists before trusting an ambient canary.

Closing the loop: verifying the weight landed

Everything so far trusts that writing the annotation or VirtualService is the split. The ALB provider can go further and verify the weight actually took at the load balancer — the controller calls the AWS ELBv2 API to confirm the listener rule shows the expected target-group weights before advancing the step. That verification is exactly why the controller’s IAM needs elasticloadbalancing:Describe* on top of the modify permissions, and it is what turns “I set the annotation” into “the ALB is provably serving 20%.” When a step must not advance until the edge agrees, weight verification is the seam that enforces it; the by-hand equivalent is the istioctl proxy-config route / aws elbv2 describe-rules check from the lab, promoted into the controller loop.

Hands-on lab

We wire a true traffic-shifted canary two ways on a free local cluster — first with NGINX, then with Istio — then map the result onto each cloud’s LB. No cloud spend is required for the two working labs; the per-cloud mapping is read-only reference you would apply on a real EKS/AKS/GKE cluster.

⚠️ Cost note: Parts A and B run entirely on a local kind cluster and bill nothing. The per-cloud blocks in Part C, if you apply them on a real cluster, provision a managed load balancer (ALB / Application Gateway / GCLB) that bills by the hour plus data processing — do not leave them running. Everything in Parts A and B is torn down at the end.

Prerequisites. A local Docker, kind, kubectl, helm, and the Argo Rollouts kubectl plugin (kubectl argo rollouts version). Install the plugin from the Argo Rollouts releases page if kubectl argo rollouts is not found.

Part A — NGINX traffic-shifted canary

Step 1 — Create the cluster and install ingress-nginx + Argo Rollouts.

kind create cluster --name traffic-lab
# Creating cluster "traffic-lab" ...
#  ✓ Ready after a few seconds

# ingress-nginx (the router)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl -n ingress-nginx wait --for=condition=available deploy/ingress-nginx-controller --timeout=180s
# deployment.apps/ingress-nginx-controller condition met

# Argo Rollouts controller
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
kubectl -n argo-rollouts wait --for=condition=available deploy/argo-rollouts --timeout=180s
# deployment.apps/argo-rollouts condition met

What just happened: you now have a router (ingress-nginx) and the brain (the Rollouts controller). Neither knows about the other yet — the Rollout will connect them.

Step 2 — Apply the demo namespace, Services, Ingress and Rollout (the four manifests from the NGINX section; save them as nginx-canary.yaml).

kubectl create namespace demo
kubectl apply -n demo -f nginx-canary.yaml
# service/rollouts-demo-stable created
# service/rollouts-demo-canary created
# ingress.networking.k8s.io/rollouts-demo-stable created
# rollout.argoproj.io/rollouts-demo created

kubectl argo rollouts -n demo get rollout rollouts-demo --watch

Representative output once the initial revision is healthy:

Name:            rollouts-demo
Namespace:       demo
Status:          ✔ Healthy
Strategy:        Canary
  Step:          5/5
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       5
  Ready:         5
  Available:     5

What just happened: the first revision is fully stable at 100% — there is nothing to canary yet. The router is wired but idle.

Step 3 — Trigger a canary by changing the image, then watch the split.

# Ship a new version — this starts the canary at step 1 (setWeight: 10)
kubectl argo rollouts -n demo set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:yellow
# rollout "rollouts-demo" image updated

kubectl argo rollouts -n demo get rollout rollouts-demo

Representative paused-at-10% output:

Name:            rollouts-demo
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          2/5
  SetWeight:     10
  ActualWeight:  10
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)

What just happened: Rollouts scaled up a canary pod, generated the shadow canary Ingress, set its canary-weight to 10, and paused. Now prove the weight is real traffic, not pods:

# Confirm the generated canary Ingress and its weight
kubectl -n demo get ingress
# rollouts-demo-stable                          nginx  rollouts-demo.local
# rollouts-demo-stable-rollouts-demo-canary     nginx  rollouts-demo.local

# Send 200 requests through the ingress and count versions (representative)
for i in $(seq 200); do
  curl -s -H 'Host: rollouts-demo.local' http://localhost/color
done | sort | uniq -c
#   180 "blue"
#    20 "yellow"     <- ~10% real traffic to canary, on 1 canary pod out of ~5

What just happened: roughly 10% of requests — about 20 of 200 — hit the yellow canary, enforced by ingress-nginx, not by pod arithmetic. That is the whole lesson in one loop.

Step 4 — Promote through the remaining steps.

kubectl argo rollouts -n demo promote rollouts-demo   # advance past the pause to setWeight: 50
# rollout "rollouts-demo" promoted
kubectl argo rollouts -n demo promote rollouts-demo   # advance to setWeight: 100

Re-run the curl loop between promotions and you will see the yellow share climb to ~50%, then 100%. The stable image flips to yellow once the rollout completes.

Part B — the Istio variant

Step 5 — Install Istio and enable sidecar injection.

# istioctl install --set profile=demo -y   (install istioctl first from Istio releases)
istioctl install --set profile=demo -y
# ✔ Istio core installed
# ✔ Istiod installed
# ✔ Ingress gateways installed

kubectl label namespace demo istio-injection=enabled --overwrite
# namespace/demo labeled

What just happened: pods created in demo from now on get an Envoy sidecar — the thing that actually enforces Istio weights. Existing pods do not get a sidecar retroactively, which is exactly the trap in troubleshooting row 8.

Step 6 — Apply the Istio Rollout, VirtualService and a Gateway (from the Istio section, plus a Gateway named rollouts-demo-gateway), then restart the Rollout so its pods are injected:

kubectl apply -n demo -f istio-canary.yaml
kubectl argo rollouts -n demo restart rollouts-demo   # get sidecars into the pods

Step 7 — Trigger the canary and verify the VirtualService weights.

kubectl argo rollouts -n demo set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:yellow

# Rollouts rewrites the VirtualService weights; confirm them (representative)
kubectl -n demo get virtualservice rollouts-demo-vsvc -o jsonpath=\
'{.spec.http[0].route[*].weight}'
# 80 20        <- stable=80, canary=20 at step 1

# And confirm Envoy actually programmed the route:
istioctl -n demo proxy-config route deploy/istio-ingressgateway | grep rollouts-demo
# rollouts-demo.local  ...  weighted_clusters 80/20

What just happened: Rollouts edited the VirtualService, Istiod pushed the config, and Envoy is now splitting real requests 80/20. The second command is the one that catches host/gateway mismatches — if it shows no weighted clusters, your Host header or Gateway hosts don’t match.

Part C — map it onto each cloud (reference)

You now have a working, router-enforced canary. On a real cluster the only thing that changes is the trafficRouting provider and the entry-point LB. Apply the paired block for your cloud from the multi-cloud section: EKS keeps NGINX/Istio or switches to the native alb provider; AKS keeps the exact NGINX/Istio setup from Parts A/B behind Application Gateway (there is no AGIC provider to switch to); GKE switches to the argoproj-labs/gatewayAPI plugin driving a GKE Gateway. The Rollout’s steps — your weights, pauses, and analysis gates — do not change at all. That portability is the payoff of separating the brain from the hands.

Teardown.

kind delete cluster --name traffic-lab
# Deleting cluster "traffic-lab" ...

What just happened: the entire lab — cluster, router, controller, Rollout — is gone, and because it was all local kind, your cloud bill is untouched.

Practice challenges

Work these in order — each assumes the last. The first two need only a reading of a manifest; the rest are runnable on the Part A/B kind lab.

1. Beginner — spot the fake canary. You inherit this Rollout. Is setWeight: 25 moving real traffic, or just pods? How can you tell in one glance?

spec:
  strategy:
    canary:
      steps:
        - setWeight: 25
        - pause: {}

<details> <summary>Solution</summary>

Pods only. There is no trafficRouting block and no canaryService/stableService, so this is a pod-ratio canary: Rollouts scales the canary ReplicaSet to ~25% of replicas behind one Service and lets kube-proxy approximate the split. The one-glance tell is the missing trafficRouting key.

</details>

Why: the presence of trafficRouting plus the two Services is the entire difference between a real and an approximate canary.

2. Beginner — make it real (NGINX). Add the minimum to the Rollout above to get a true 25% split through ingress-nginx, assuming an Ingress named demo-stable already points at the stable Service.

<details> <summary>Solution</summary>

spec:
  strategy:
    canary:
      canaryService: demo-canary
      stableService: demo-stable
      trafficRouting:
        nginx:
          stableIngress: demo-stable
      steps:
        - setWeight: 25
        - pause: {}

Plus the two ClusterIP Services (demo-canary, demo-stable) carrying the app’s base selector — Rollouts injects the pod-template-hash to make them canary-only and stable-only.

</details>

Why: every provider needs the canary/stable Service pair; stableIngress is the only NGINX-specific field.

3. Intermediate — host split → subset split (Istio). Convert a two-host VirtualService canary to the DestinationRule subset model. What do you add, and what must the subset labels contain?

<details> <summary>Solution</summary>

Add a DestinationRule with stable and canary subsets whose labels are left empty ({}) — Rollouts injects rollouts-pod-template-hash into them. Point the Rollout at it:

trafficRouting:
  istio:
    virtualService:
      name: rollouts-demo-vsvc
      routes: [primary]
    destinationRule:
      name: rollouts-demo-destrule
      canarySubsetName: canary
      stableSubsetName: stable

The VirtualService route destinations now target one host with subset: stable / subset: canary instead of two hostnames.

</details>

Why: subsets let one Service front all pods while Rollouts weights by label — the model you want when versions are already modelled as subsets or need per-subset mTLS.

4. Intermediate — internal testers first. Write the steps that send only requests with header X-Canary: true to the canary for 15 minutes at 0% public traffic, then begin a 10% public rollout. Which providers support this?

<details> <summary>Solution</summary>

trafficRouting:
  managedRoutes:
    - name: canary-by-header
  istio:
    virtualService:
      name: rollouts-demo-vsvc
      routes: [primary]
steps:
  - setHeaderRoute:
      name: canary-by-header
      match:
        - headerName: X-Canary
          headerValue:
            exact: "true"
  - pause: { duration: 15m }
  - setHeaderRoute: { name: canary-by-header }   # bare name removes the route
  - setWeight: 10

Supported on Istio, ALB, and the Gateway API plugin — not NGINX’s step, not SMI. A setHeaderRoute with only a name removes the previously-added route.

</details>

Why: header routing decouples “who sees v2” from “how much traffic,” so you can dogfood before exposing any public users.

5. Advanced — the ALB that won’t shift. On EKS you setWeight: 30, the action annotation updates, but the ALB serves 100% stable. List the three usual causes and how you would confirm each.

<details> <summary>Solution</summary>

1. Controller / IAM — the AWS Load Balancer Controller isn’t installed or lacks IAM to modify the ALB. Confirm: controller pod logs; aws elbv2 describe-rules shows one target group, not two.
2. target-type: instance — you’re weighting nodes, and every node runs both versions. Confirm: read the Ingress annotation; switch it to ip.
3. rootService / action-name mismatch — Rollouts edits an action the Ingress rule never references. Confirm: the backend service.name must equal the actions.<name> and rootService.

Bonus resilience: switch to pingPong services to remove the in-place-rewrite blip entirely.

</details>

Why: the ALB annotation is necessary but not sufficient — the proof is at the load balancer (aws elbv2 describe-rules), not in the cluster.

6. Expert — one manifest, three clouds. You must run the same Rollout across EKS, AKS and GKE with the least per-cloud drift. Which router do you standardise on, what changes per cloud, and why can’t you just “use each cloud’s LB canary”?

<details> <summary>Solution</summary>

Standardise on Istio or the Gateway API plugin — both exist identically on all three clouds, so trafficRouting and every step stay byte-identical. What changes per cloud is only the entry-point LB and its Gateway/ingress: ALB on EKS, Application Gateway / Azure LB in front on AKS, GCLB via GKE Gateway on GKE. You cannot rely on a native LB canary because AKS’s AGIC has no Rollouts provider at all, and GKE needs Gateway API while EKS uses the ALB action model — three different edges, no common native feature. See the multi-cloud capstone for the full three-cloud build and Argo CD on AKS for why Application Gateway needs an in-cluster router.

</details>

Why: portability comes from making the router the same object everywhere and letting only the cloud entry-point differ — the payoff of separating the brain (Rollouts) from the hands (the router).

Common mistakes and troubleshooting

The failure modes here cluster around one theme: the weight is set correctly in the Rollout but never reaches real traffic, because the router is missing, mis-wired, or not matching. Use the ActualWeight field as your first signal — if it advances but traffic doesn’t shift, the problem is below Rollouts, in the router or the LB.

Three of these deserve a longer look because they cost the most hours.

1. The invisible pod-ratio canary. The nastiest failure is the one that looks like it works. Omit trafficRouting and everything is green — the Rollout progresses, setWeight steps advance, kubectl argo rollouts get shows a tidy canary. But there is no router, so “20%” means “one canary pod among five, load-balanced by kube-proxy.” Under keep-alive or gRPC connections that pod can see 5% or 40% of real traffic, and it drifts every time the HPA fires. The tell is that ActualWeight matches SetWeight yet your load balancer’s own metrics show a different split. The fix is conceptual, not a flag: decide up front whether you need real traffic weighting, and if you do, wire a provider. A canary without trafficRouting is a slow rollout, not a controlled one.

2. Host/gateway mismatch in Istio. Rollouts writes the VirtualService weights perfectly and reports success, because from its perspective the job is done — the object has the right numbers. But Istio only applies a VirtualService when the request matches its hosts and arrives through a listed gateway. A trailing-dot hostname, a mismatched Host: header from your test client, or a Gateway serving a different host means Envoy never selects the weighted route, and 100% of traffic takes the default path. Because Rollouts is “correct” and Istio is “correct,” nothing errors — you just get no canary. Always confirm at the data plane: istioctl proxy-config route <gateway-pod> must show your weighted_clusters with the expected split. If it doesn’t, the problem is your host/gateway plumbing, not the Rollout.

3. AKS expecting a native provider. Teams moving from EKS to AKS assume symmetry: “ALB has a provider, so Application Gateway must too.” It does not. AGIC exposes no annotations Rollouts can weight, and there is no trafficRouting.appgw. The result is an afternoon lost searching for a provider that was never built. The correct architecture on AKS is an in-cluster router — NGINX or Istio, wired exactly as in this lab — with Application Gateway or an Azure Load Balancer as the L7/L4 entry point in front of it. The genuinely native Azure path is newer: Application Gateway for Containers (AGC) speaks Gateway API, so AGC plus the argoproj-labs/gatewayAPI plugin can weight HTTPRoutes — but that is a deliberate architecture choice, not a drop-in AGIC feature. Plan for the in-cluster router on AKS and you will never hit this wall.

Common beginner mistakes

These are mental-model errors, not symptoms — the wrong belief that leads you to build the wrong thing. (For symptom → cause → fix, see the troubleshooting table above.)

“More replicas give me a finer canary.” The instinct after learning that 5 pods can only do 20% steps is to run 20 or 100 replicas for 5% or 1% granularity. Wrong lever. Once you add a router, weight is a percentage set at the router, completely independent of replica count — setWeight: 1 means 1% whether you run 3 pods or 300. Replicas are a capacity decision; weight is a routing decision; the router decouples them. Scaling the fleet to buy precision just wastes compute.

“The canary and stable Services need different label selectors.” Beginners hand-write version: canary on one Service and version: stable on the other, then wonder why both match everything. Rollouts does this for you: you give both Services the same base selector (app: demo), and Rollouts injects rollouts-pod-template-hash at runtime so one resolves to only-new pods and the other to only-old. Adding your own version label defeats the mechanism. Right model: author the base selector, let Rollouts own the discriminator.

“With a traffic router, the canary pod count tracks the weight exactly.” At setWeight: 5, people expect exactly 5% of pods. But the router split and the ReplicaSet size are decoupled — Rollouts runs enough canary pods to serve the weighted traffic (at least one), which on a 4-pod fleet is 25% of pods serving 5% of traffic. That is fine and intentional. If you want to control canary capacity, that is what setCanaryScale is for — the weight will not do it.

“Gateway API is GA, so it’s the safe default on every cloud.” The Gateway API is GA and stable. The Rollouts plugin that drives it is a separate argoproj-labs component still on a 0.x line, and setHeaderRoute/setMirrorRoute support depends on both the plugin version and your specific Gateway implementation (Istio gateway, Envoy Gateway, NGINX Gateway Fabric, each cloud’s controller). Treat weighting as solid and advanced routes as “verify on my exact stack.” GA-the-API is not GA-the-integration.

“Traffic mirroring shows the canary’s response to users.” Mirroring (setMirrorRoute) is fire-and-forget shadowing: the router copies the request to the canary and discards the canary’s response — the client only ever sees stable’s answer, at stable’s latency. That is what makes it a zero-user-impact smoke test. The real constraint is side effects, not latency: mirror only idempotent requests (match on GET) so the shadow copy does not double-write to a database or re-charge a card.

“The cloud load balancer does the canary for me.” Especially on AKS: teams assume Application Gateway will weight traffic the way the ALB does. It will not — AGIC has no Rollouts provider, and there is no trafficRouting.appgw. The thing that actually splits traffic is either an in-cluster mesh/ingress (NGINX, Istio) or, on EKS, the ALB action annotation Rollouts rewrites. The cloud LB is the front door; the router is what divides the hallway. Assuming the front door also splits is the single most expensive wrong belief on AKS.

Cheat-sheet

The trafficRouting providers at a glance:

Per-cloud LB mapping, the reference you’ll reach for most:

The kubectl argo rollouts commands you’ll live in:

Canary steps and scale knobs:

Interview and exam questions

Q: A colleague says “we’re running a 20% canary” but there’s no trafficRouting in the Rollout. What is actually happening, and why might it not be 20%? A: It’s a pod-ratio canary: Rollouts scales the canary ReplicaSet to ~20% of replicas behind a single Service, and traffic distribution is whatever the service proxy does across the combined endpoints. With five replicas that’s one canary pod, and per-connection balancing (worse with keep-alive/gRPC) means real traffic can land anywhere from ~5% to ~40%. It also drifts when the HPA resizes the fleet. True 20% of requests requires a trafficRouting provider that weights at L7.

Q: What is the role of canaryService and stableService, and what breaks if their selectors overlap? A: They are two Services that must resolve to only canary pods and only stable pods respectively; Rollouts injects the rollouts-pod-template-hash into their selectors to achieve that. The router weights traffic between these two backends. If the selectors overlap and both match every pod, the router still splits traffic — but to two identical backends, so the “canary” is meaningless and you’re testing nothing.

Q: Walk through how the NGINX provider actually shifts traffic. A: You author one stable Ingress pointing at stableService. At each setWeight, Rollouts generates (and owns) a second, canary Ingress — a copy carrying nginx.ingress.kubernetes.io/canary: "true" and canary-weight: "N" — pointing at canaryService. ingress-nginx sees two Ingresses for the same host/path and routes N% of requests to the canary backend. You never hand-edit the canary Ingress or its weight.

Q: On EKS you set setWeight: 30, the annotation updates, but the ALB doesn’t shift traffic. Name three likely causes. A: (1) The AWS Load Balancer Controller isn’t installed or lacks IAM (IRSA/Pod Identity) to modify the ALB, so the annotation changes but the ALB doesn’t; (2) alb.ingress.kubernetes.io/target-type is instance instead of ip, so you’re weighting nodes that both run both versions; (3) rootService/action-name mismatch, so Rollouts edits an action annotation the Ingress rule never references. Verify at the ALB with aws elbv2 describe-rules.

Q: Why is there no native Argo Rollouts provider for AKS’s Application Gateway, and what do you do instead? A: AGIC (the classic Application Gateway Ingress Controller) exposes no annotations Rollouts can weight, and there’s no trafficRouting.appgw. On AKS you run an in-cluster router — NGINX or Istio — wired normally, with Application Gateway or an Azure LB as the entry point in front. The emerging native path is Application Gateway for Containers (AGC), which speaks Gateway API, so AGC plus the argoproj-labs/gatewayAPI plugin can weight HTTPRoutes.

Q: How does the Gateway API plugin differ operationally from the built-in nginx/istio providers? A: It’s not compiled into the controller — it’s a plugin you register in the argo-rollouts-config ConfigMap (trafficRouterPlugins with a name and a binary location), and the controller needs RBAC over httproutes. Once registered, you reference it under trafficRouting.plugin by the exact registered name, and it weights HTTPRoute backendRefs. The API is GA; the plugin is a 0.x community component, so verify advanced features on your specific Gateway implementation.

Q: You need to send only your QA team to v2 before exposing any public traffic. Which primitive, and on which providers? A: setHeaderRoute — route requests carrying a specific header (e.g. X-Canary: true) to the canary while 0% of public traffic shifts. It’s supported on Istio, ALB, and the Gateway API plugin (not NGINX’s step, not SMI). Declare the route under trafficRouting.managedRoutes, add the setHeaderRoute step, then remove it before the first setWeight.

Q: What is traffic mirroring, when would you use it, and what’s the constraint? A: setMirrorRoute shadows a copy of live requests to the canary; the client never sees the canary’s response. It’s the zero-user-impact way to test the new version under real production load and traffic shapes. The constraint: it’s Istio (and implementation-dependent Gateway API) only, and you should mirror idempotent requests (match on GET) so the shadow doesn’t cause side effects like duplicate writes.

Q: How do traffic weighting and analysis combine, and why is weighting alone insufficient? A: Weighting controls how much traffic hits the canary; analysis (AnalysisTemplate/AnalysisRun) controls whether it should advance, by querying metrics (success rate, latency) and failing the step if the canary is unhealthy. Weighting alone is just a slow way to ship a bug to N% of users; interleaving analysis steps between setWeight steps gates each increase on live health and enables automatic rollback — that’s what makes it progressive delivery, not just a staged rollout.

Q: You want the same Rollout manifests to work across EKS, AKS, and GKE. What’s the pragmatic strategy? A: Standardise on a router that exists identically on all three — Istio, or the Gateway API plugin — so trafficRouting and the steps are the same object everywhere, and only the entry-point LB (ALB/App Gateway/GCLB) differs per cloud. Relying on each cloud’s native LB feature doesn’t port, because AKS has no native provider and GKE needs Gateway API while EKS uses the ALB action model.

Q: kubectl argo rollouts get shows ActualWeight: 20 on Istio, but the canary receives no traffic. Where do you look? A: The data plane, not the Rollout. Rollouts set the VirtualService weights correctly (hence ActualWeight: 20), but Istio only applies the route if the request’s Host: and the Gateway match the VirtualService hosts/gateways. Run istioctl proxy-config route <gateway-pod> — if it doesn’t show your weighted_clusters 80/20, the host/gateway plumbing is the culprit, or the pods lack sidecars.

Key takeaways

• setWeight without trafficRouting is a pod-ratio approximation, not a traffic percentage. It’s coarse, drifts under autoscaling, and can’t do header or mirror routing. Real traffic weighting needs a router.
• Every provider needs the canary/stable Service model, and some need a root Service. Rollouts injects the pod-template-hash so those Services resolve to only canary or only stable pods — selector hygiene is the most common wiring bug.
• Each router weights a different native object: NGINX a generated canary Ingress, Istio a VirtualService (or DestinationRule subsets), ALB the action annotation’s target groups, Gateway API an HTTPRoute’s backendRefs. SMI’s TrafficSplit is legacy — avoid it for new work.
• The load-balancer edge is per-cloud and uneven: EKS has a native alb provider; AKS has no native Application Gateway provider (front it with NGINX/Istio, or use AGC + Gateway API); GKE routes through the Gateway API plugin on GKE Gateway. Standardise on Istio or the Gateway API plugin for portable manifests.
• Header routing and mirroring are provider-dependent (Istio and Gateway API are the capable ones); writing setHeaderRoute/setMirrorRoute against NGINX or SMI fails silently. Check the capability matrix before designing a pipeline around them.
• Weighting answers “how much”; analysis answers “should it advance.” Interleave analysis steps between setWeight steps so each traffic increase is gated on live metrics — that’s the difference between a slow rollout and an autopilot with automatic rollback.
• Debug below Rollouts. When ActualWeight matches SetWeight but traffic doesn’t move, the fault is in the router or LB — verify at the data plane (istioctl proxy-config route, aws elbv2 describe-rules, the generated canary Ingress), not in the Rollout YAML.

Glossary

• Traffic router — the L7 component (NGINX, Istio, an ALB, or a Gateway API implementation) that actually splits requests by weight. Rollouts decides the weight; the router enforces it.
• Pod-ratio canary — a canary with no trafficRouting: the split is approximated by how many canary vs stable pods exist, load-balanced by kube-proxy. Coarse and drift-prone.
• trafficRouting — the field under strategy.canary naming the router(s) Rollouts programs at each setWeight.
• setWeight: N — a canary step meaning “send N% of traffic to the canary.” Real only when a router is wired; otherwise it just scales pods.
• SetWeight vs ActualWeight — status fields: SetWeight is what Rollouts intends; ActualWeight is what it has programmed. Both can read correct while the data plane ignores the weight (host mismatch, missing sidecar/waypoint).
• canaryService / stableService — two Services that must resolve to only canary and only stable pods. Rollouts injects the pod-template-hash into their selectors to guarantee that.
• rollouts-pod-template-hash — the label Rollouts adds to each ReplicaSet’s pods and to the two Services’ selectors, so canary/stable Services never overlap.
• rootService — a Service that receives all traffic before the split, used by the alb and smi providers as the fan-out point.
• Canary (shadow) Ingress — for NGINX, the second Ingress Rollouts generates and owns, carrying canary: "true" and canary-weight: "N". You never author it.
• VirtualService / DestinationRule — Istio objects; Rollouts rewrites HTTP route weights in the VirtualService, and optionally flips pods between stable/canary subsets defined in a DestinationRule.
• Action annotation — the ALB’s alb.ingress.kubernetes.io/actions.<name> JSON whose target-group weights Rollouts rewrites; requires target-type: ip.
• pingPong — the ALB provider’s two-Service mode (ping/pong) that swaps stable/canary roles each rollout to avoid an in-place target-group rewrite blip.
• HTTPRoute / Gateway — the GA, portable Gateway API successor to Ingress; backendRefs[].weight are relative ratios (need not sum to 100). Rollouts drives them via the argoproj-labs/gatewayAPI plugin.
• TrafficSplit (SMI) — the legacy Service Mesh Interface weight object; the smi provider still writes it, but SMI is dormant — avoid for new work.
• Plugin (traffic router) — an out-of-tree provider (Gateway API, APISIX, Traefik, Contour, Gloo) registered in the argo-rollouts-config ConfigMap and referenced under trafficRouting.plugin.
• managedRoutes — the list of extra routes (header/mirror) Rollouts creates and cleans up; inserted ahead of the weighted split so they win, in the order you list them.
• setHeaderRoute — route only requests matching a header to the canary (Istio/ALB/Gateway API), for testing at 0% public traffic.
• setMirrorRoute (shadow traffic) — copy a % of live requests to the canary and discard its response; the client is unaffected. Istio and (implementation-dependent) Gateway API only.
• setCanaryScale — decouple canary replica count from traffic weight (pre-warm capacity, or fix a canary size).
• dynamicStableScale — shrink the stable ReplicaSet as canary weight rises, keeping total pods ~constant during a long bake.
• scaleDownDelaySeconds — how long the old stable pods linger after full promotion so in-flight/keep-alive connections drain before the router’s endpoints vanish (default 30).
• Sidecar vs waypoint (Istio) — classic Istio enforces weights via a per-pod Envoy sidecar; ambient mode uses a shared waypoint proxy instead. Missing either means the weight is set but not enforced.
• AGIC / AGC — Azure’s Application Gateway Ingress Controller (no Rollouts provider — front it with an in-cluster router) versus Application Gateway for Containers, which speaks Gateway API and can be driven by the plugin.
• AnalysisTemplate / AnalysisRun — the metric-query objects that gate each setWeight on canary health, turning a manual dial into an autopilot with automatic rollback.