Istio Ambient Mesh in Practice: Zero-Trust mTLS, Traffic Management & L7 Authorization

In a nutshell

A service mesh gives every service three things — encryption in transit (mTLS), traffic control (routing, retries, canaries), and observability (uniform metrics, logs, traces) — without changing a single line of application code. The mesh does its work in the network layer, so your app keeps speaking plain HTTP while the platform transparently wraps, routes, and watches every call between services.

The classic way to build a mesh is the sidecar: an Envoy proxy injected into every pod, next to your container. It works, but you pay for a full Layer-7 proxy on every pod whether that pod needs one or not, and a pod has to restart to join or leave the mesh. Istio’s ambient mode removes the per-pod sidecar entirely. In its place:

• A shared ztunnel (“zero-trust tunnel”) runs once per node as a DaemonSet and hands every enrolled pod mTLS and an L4 identity for free — no sidecar, no restart.
• An optional waypoint proxy is added only for the handful of services that genuinely need Layer-7 features (HTTP routing, path/method authorization, JWT validation). Everything else stays L4-only and cheap.

Analogy — the office building. The sidecar model gives every single room its own airport-style security scanner: thorough, but wildly expensive, and you have to close the room to install it. Ambient mode instead posts one trusted guard on each floor (ztunnel) who checks everyone’s badge and encrypts every corridor, and only wheels a full X-ray machine (a waypoint) up to the few doors — payroll, the vault — that truly need to inspect what is inside each package. Most rooms are protected without ever paying for the X-ray.

Level: Advanced · Time: ~33 min · You’ll need: comfort with Pods, Deployments, Services and ServiceAccounts (see Pods, Deployments & Services), plus a nodding acquaintance with TLS and HTTP. A cluster is not required to read along — every manifest and command below is real and current for Istio 1.24+ ambient. When you want hands-on reps, run them against a local kind or minikube cluster installed with the ambient profile.

After this lesson you can:

• Explain when a sidecar mesh versus an ambient mesh is the right call, in concrete cost and blast-radius terms.
• Enroll a namespace into the mesh with a single label and prove mTLS is live — with zero pod restarts.
• Enforce STRICT mTLS and a default-deny authorization baseline, then punch explicit L4 holes by workload identity.
• Add a waypoint only where L7 is actually needed, and run weight- and header-based canary routing behind it.
• Debug the two-proxy data plane — ztunnel for L4, the waypoint for L7 — the way an on-call operator does.

The diagram is the whole lesson in one picture — read it left to right. A pod joins the mesh by a label and immediately sits behind its node’s ztunnel, which gives it mTLS and an L4 identity: the zero-trust floor every workload gets. Only services that need HTTP-level control get a waypoint; the rest of your fleet stays L4-only. Hold the L4-floor-versus-opt-in-L7 split in your head — almost everything in this lesson hangs off it.

Ambient mode splits the Istio data plane in two: a per-node L4 proxy (ztunnel) that does mTLS and identity for every enrolled pod for free, and an optional per-service L7 proxy (waypoint) you pay for only where you need HTTP routing or rich authorization. No sidecars, no pod restarts to join the mesh, and a CPU/memory bill that scales with traffic instead of pod count. This guide takes a workload from unmeshed to zero-trust, then routes, secures, and debugs it the way you would on call.

Everything here targets ambient as it shipped GA in Istio 1.24 and the APIs stable since. Commands are real and current; where a feature has a sharp edge I call it out rather than paper over it.

1. Sidecar vs ambient: what you are actually deploying

In the sidecar model every pod carries its own Envoy. That Envoy does L4 and L7, costs ~50-100Mi of memory per pod whether or not the pod needs HTTP features, and requires a pod restart to inject or upgrade. Ambient unbundles that:

The trade-off is honest, not free. ztunnel tunnels traffic over HBONE (HTTP/2 CONNECT on port 15008, mTLS-wrapped), which adds a hop and a small latency tax versus a direct sidecar-to-sidecar path. The big win is that pods needing only encryption-in-transit and L4 policy never pay for an L7 proxy at all. You add a waypoint only when a service genuinely needs L7 — and a waypoint is a normal Deployment you can scale and schedule independently of the apps behind it.

Mental model: ztunnel is the zero-trust floor every workload gets. The waypoint is an L7 upgrade you bolt on per service. Traffic only traverses a waypoint when the destination it was originally addressed to has one configured.

The cost model, concretely

The abstract “scales with nodes, not pods” line becomes obvious the moment you put numbers on it. Picture a cluster with 50 nodes and 2,000 meshed pods, each sidecar reserving a conservative 60Mi of memory:

• Sidecar mesh: 2,000 × 60Mi ≈ 117Gi of memory spent purely on proxies, plus a slice of CPU per pod, all of it duplicated whether the pod ever makes an HTTP-aware decision or not.
• Ambient mesh: 50 ztunnels (one per node) plus, say, 8 waypoints for the services that actually need L7. Call it ~58 proxy instances instead of 2,000. Even generously sized, that is single-digit Gi, roughly an order of magnitude less, and it does not grow when you scale a Deployment from 3 replicas to 300.

That is the headline, but the operational savings are just as real: enrolling a namespace is a label edit, so you never schedule a maintenance window to roll 2,000 pods just to inject or upgrade a proxy. You upgrade the L4 layer by rolling one DaemonSet.

Ambient is not automatically the right answer, though. Here is where each model still wins:

2. Install ambient mode

Install with the ambient profile. This lays down istiod, the Istio CNI node agent (which sets up traffic redirection without NET_ADMIN in your app pods), and the ztunnel DaemonSet.

istioctl install --set profile=ambient --skip-confirmation

# Confirm the control plane and data plane are up
kubectl get pods -n istio-system
kubectl get daemonset ztunnel -n istio-system

You should see istiod, istio-cni-node (one per node), and ztunnel (one per node). If you run a managed cluster, check the CNI chaining mode for your platform — on some distributions the Istio CNI must be ordered after the primary CNI. Install the Kubernetes Gateway API CRDs too; waypoints are Gateway API resources:

kubectl get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1 || \
  kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

3. Enroll namespaces incrementally

This is ambient’s headline feature: enrollment is a label, and existing pods join with zero restarts.

kubectl label namespace shop istio.io/dataplane-mode=ambient

# Verify workloads are now seen by ztunnel as HBONE-capable
istioctl ztunnel-config workloads --workload-namespace shop

Every pod in shop is now inside the L4 mesh. ztunnel-config workloads (aliased istioctl zc workloads) lists each workload’s address, the protocol (HBONE once enrolled), and any assigned waypoint. Roll out namespace by namespace — because there is no sidecar to inject, you can mesh a busy namespace during business hours without a rolling restart, which is the single biggest operational difference from the sidecar model.

What “enrolled” actually buys you: the pod’s outbound and inbound traffic is now transparently captured by its node’s ztunnel and carried inside mutually-authenticated HBONE tunnels. That means the connection is encrypted and identity-tagged — but it is not yet denied to anyone. Encryption and authorization are two separate switches; §4 is where you turn on the second one. Read “enrolled” as “on the encrypted network”, not “secured”.

To exclude a specific pod (say a job that opens raw TCP that does not tolerate redirection), label the pod:

kubectl label pod <pod> istio.io/dataplane-mode=none --overwrite

4. Enforce strict mTLS and a default-deny posture

Enrollment already encrypts pod-to-pod traffic with mTLS via HBONE, but it does not yet forbid plaintext. Lock that down with a mesh-wide PeerAuthentication in STRICT mode, then layer a default-deny AuthorizationPolicy.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system   # mesh-wide root config namespace
spec:
  mtls:
    mode: STRICT

Now the deny-all baseline. An AuthorizationPolicy with an empty spec (no rules) and the default ALLOW action denies everything, because “allow nothing” is the result of an allow-policy with zero matching rules.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: shop
spec:
  {}   # ALLOW action, zero rules => deny everything in this namespace

From here you punch holes with explicit allows. This L4 policy lets only the web service account reach orders, matched by SPIFFE identity rather than IP. It is enforced by ztunnel because it uses a selector and references only L4 attributes (principals, ports) — no HTTP:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-allow-web
  namespace: shop
spec:
  selector:
    matchLabels:
      app: orders
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
        - cluster.local/ns/shop/sa/web

Key distinction: a policy using selector and only L4 attributes is enforced at ztunnel. The moment you need HTTP methods, paths, or headers you must target a waypoint with targetRefs (Section 7). ztunnel is L4-only and physically cannot read HTTP.

New to mTLS migration? Do not jump straight to STRICT on a live mesh. PeerAuthentication supports three modes — PERMISSIVE (accept both mTLS and plaintext; this is the default while you migrate), STRICT (reject all plaintext), and DISABLE. Roll out PERMISSIVE first, confirm from telemetry that every real caller is already speaking mTLS, then flip to STRICT. Flipping to STRICT while a non-mesh client (a monitoring probe, a legacy job, a cross-namespace caller you forgot about) is still connecting in plaintext is the single most common way to take an ambient rollout down. You can also scope the mode per namespace during migration instead of mesh-wide.

5. Add a waypoint for L7

L7 routing and L7 authorization require a waypoint. Deploy one for the namespace:

istioctl waypoint apply -n shop --enroll-namespace
kubectl get gateways.gateway.networking.k8s.io -n shop

Under the hood this creates a Gateway API Gateway with gatewayClassName: istio-waypoint. If you manage manifests in Git, generate the YAML instead of applying imperatively:

istioctl waypoint generate --for service -n shop > waypoint.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: shop
  labels:
    istio.io/waypoint-for: service   # service | workload | all | none
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

Wait for the Gateway to report PROGRAMMED=True, then point services at it. --enroll-namespace adds istio.io/use-waypoint: waypoint to the namespace so all services route through it; you can scope it to a single service instead:

kubectl label service orders -n shop istio.io/use-waypoint=waypoint

The --for service choice matters: a service-scoped waypoint intercepts traffic addressed to a Service VIP (the common case for routing and splits). A workload waypoint intercepts pod-IP traffic. Pick service unless you specifically need to govern direct pod-to-pod calls.

6. L7 routing: weight and header-based splits

With a waypoint in place, classic VirtualService and DestinationRule work as they always have. Define subsets, then split. Here is a canary: 90/10 by weight, with an escape hatch that sends anyone carrying x-canary: always straight to v2.

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: orders
  namespace: shop
spec:
  host: orders.shop.svc.cluster.local
  subsets:
  - name: v1
    labels: { version: v1 }
  - name: v2
    labels: { version: v2 }
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: orders
  namespace: shop
spec:
  hosts:
  - orders.shop.svc.cluster.local
  http:
  - match:
    - headers:
        x-canary:
          exact: always
    route:
    - destination: { host: orders.shop.svc.cluster.local, subset: v2 }
  - route:
    - destination: { host: orders.shop.svc.cluster.local, subset: v1 }
      weight: 90
    - destination: { host: orders.shop.svc.cluster.local, subset: v2 }
      weight: 10

You can drive routing with Gateway API HTTPRoute instead of VirtualService. Pick one per service — mixing VirtualService and Gateway API route objects on the same host is not supported and produces undefined precedence. I use VirtualService when I need fault injection or circuit breaking (next section), because the Gateway API does not yet cover the full Istio resilience surface.

7. L7 authorization at the waypoint

Now the policy that needs HTTP semantics. This allows the web identity to GET and POST only under /api/, and is attached to the waypoint via targetRefs (note targetRefs, not selector). The to.operation block with methods and paths is exactly what forces waypoint enforcement.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-l7
  namespace: shop
spec:
  targetRefs:
  - kind: Service
    group: ""
    name: orders
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
        - cluster.local/ns/shop/sa/web
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/*"]

For end-user identity, validate JWTs at the waypoint with RequestAuthentication, then require a valid token in an AuthorizationPolicy. RequestAuthentication only defines how to validate a token — it does not reject requests on its own. The companion policy below denies any request lacking an authenticated principal (requestPrincipals of ["*"] means “any valid issuer/subject”).

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: orders-jwt
  namespace: shop
spec:
  targetRefs:
  - kind: Service
    group: ""
    name: orders
  jwtRules:
  - issuer: "https://accounts.example.com"
    jwksUri: "https://accounts.example.com/.well-known/jwks.json"
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-require-jwt
  namespace: shop
spec:
  targetRefs:
  - kind: Service
    group: ""
    name: orders
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]

Order of operations at the waypoint: JWT is validated, then CUSTOM (ext-authz) policies, then DENY, then ALLOW. Deny always wins over allow, so a deny rule cannot be overridden by a broader allow.

8. Resilience: timeouts, retries, circuit breaking, fault injection

These ride on the same waypoint. Timeouts and retries live in VirtualService; connection-pool limits and outlier detection (circuit breaking) live in DestinationRule.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: orders-resilience
  namespace: shop
spec:
  hosts: ["orders.shop.svc.cluster.local"]
  http:
  - timeout: 2s
    retries:
      attempts: 3
      perTryTimeout: 500ms
      retryOn: 5xx,reset,connect-failure
    route:
    - destination: { host: orders.shop.svc.cluster.local }
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: orders-cb
  namespace: shop
spec:
  host: orders.shop.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp: { maxConnections: 100 }
      http: { http2MaxRequests: 200, maxRequestsPerConnection: 10 }
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Fault injection is a separate VirtualService — inject a delay or an abort to test that your retries and timeouts actually behave:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: orders-fault
  namespace: shop
spec:
  hosts: ["orders.shop.svc.cluster.local"]
  http:
  - fault:
      delay:
        percentage: { value: 25 }
        fixedDelay: 3s
      abort:
        percentage: { value: 10 }
        httpStatus: 503
    route:
    - destination: { host: orders.shop.svc.cluster.local }

Sharp edge: Istio will not let you combine fault with retries/timeout on the same VirtualService. Keep fault injection in its own object, or you will get a config rejection. Remove it before promoting to prod.

9. Observability without sidecars

The waypoint exports the full standard Istio telemetry set, so a waypointed service has the same metrics, access logs, and tracing surface a sidecar would — just emitted by a shared proxy. Services that are L4-only (ztunnel alone) get connection-level metrics from ztunnel.

# Standard Istio request metrics, served by the waypoint
kubectl exec -n shop deploy/waypoint -c istio-proxy -- \
  pilot-agent request GET stats/prometheus | grep istio_requests_total

# ztunnel exposes its own metrics endpoint on 15020
kubectl exec -n istio-system ds/ztunnel -- curl -s localhost:15020/metrics | grep istio_tcp_connections

Turn on mesh-wide access logging and tracing with the Telemetry API rather than per-proxy flags:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
  - providers:
    - name: envoy
  tracing:
  - randomSamplingPercentage: 5.0

Tail a waypoint’s access logs directly when you are debugging a specific call path:

kubectl logs -n shop deploy/waypoint -c istio-proxy --tail=50

Verify

Prove each layer independently.

# 1. mTLS / L4: ztunnel sees the workload over HBONE
istioctl ztunnel-config workloads --workload-namespace shop

# 2. The waypoint is healthy and programmed
kubectl get gateways.gateway.networking.k8s.io waypoint -n shop \
  -o jsonpath='{.status.conditions[?(@.type=="Programmed")].status}{"\n"}'

# 3. Authz works: allowed identity gets 200, others get 403
kubectl exec -n shop deploy/web -- curl -s -o /dev/null -w '%{http_code}\n' http://orders/api/health
kubectl run probe --rm -it --image=curlimages/curl -n shop --restart=Never -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://orders.shop/api/health   # expect 403

# 4. Routing split is live (waypoint metrics by subset)
kubectl exec -n shop deploy/waypoint -c istio-proxy -- \
  pilot-agent request GET stats/prometheus | grep 'destination_version="v2"'

# 5. Static config sanity check
istioctl analyze -n shop

A clean run: workloads show HBONE, the Gateway is True, the allowed call returns 200 and the unauthorized one 403, v2 shows ~10% of requests, and analyze reports no errors.

10. Debugging the data plane

Two proxies means two places to look. Triage L4 at ztunnel, L7 at the waypoint.

# Is the pod actually enrolled? Look for the redirection annotation
kubectl get pod <pod> -n shop -o yaml | grep ambient.istio.io/redirection

# ztunnel's view of services and which waypoint each maps to
istioctl ztunnel-config services -n shop

# ztunnel's view of workloads (address, protocol, waypoint)
istioctl ztunnel-config workloads --workload-namespace shop

# Crank ztunnel logging for one node's pod (scoped, revert after)
istioctl ztunnel-config log <ztunnel-pod> --level=info,access=debug
kubectl logs -n istio-system <ztunnel-pod> --tail=100

# Inspect the waypoint's Envoy like any proxy: listeners, routes, clusters
istioctl proxy-config routes deploy/waypoint -n shop
istioctl proxy-config clusters deploy/waypoint -n shop

The usual culprits, in the order I check them:

• 403 you did not expect — your default-deny is doing its job but the explicit allow has the wrong principal. Double-check the SPIFFE string: cluster.local/ns/<ns>/sa/<serviceaccount>, and confirm the source pod runs under that ServiceAccount.
• L7 policy “ignored” — the policy uses selector instead of targetRefs, so it landed on ztunnel, which silently drops the L7 parts because it cannot evaluate them. Switch to targetRefs pointing at the Service or the Gateway.
• Routing not applied — no waypoint on the destination service, or you mixed VirtualService with an HTTPRoute on the same host. One routing API per host.
• PROGRAMMED=False — Gateway API CRDs missing, or the waypoint Deployment cannot schedule. kubectl describe gateway waypoint -n shop.

Enterprise scenario

A payments platform team migrated ~400 namespaces off sidecars to bank the memory savings. Enrollment went clean until the fraud-scoring service started returning intermittent 502s under load — but only for calls that crossed availability zones. The constraint: that service ran a StatefulSet with mutual TLS terminated inside the app (legacy, pre-mesh) and clients addressed pods directly by their stable pod DNS, not the Service VIP. They had applied a service-scoped waypoint namespace-wide via --enroll-namespace, so pod-IP traffic never traversed the waypoint, yet the double mTLS (HBONE plus app-level) was fighting ztunnel’s redirection on the stateful pods.

The fix was twofold. First, they excluded the StatefulSet pods from the L7 path and let ztunnel handle L4 only, then scoped the waypoint to the Services that actually needed routing rather than the whole namespace:

# Stop blanket namespace enrollment; opt in per Service instead
kubectl label namespace payments istio.io/use-waypoint-

# Pods addressed directly need a workload waypoint, not a service one
istioctl waypoint apply -n payments --name pods-wp --for workload
kubectl label statefulset fraud-score -n payments istio.io/use-waypoint=pods-wp

Second, they kept the app-level TLS but stopped HBONE from re-wrapping by confirming the redirection annotation and sizing the waypoint against measured cross-AZ RPS, not pod count. The lesson the team wrote into their runbook: --for service and --for workload are not interchangeable, and any workload that bypasses the Service VIP needs a workload waypoint or no waypoint at all. The 502s disappeared once pod-IP traffic stopped hitting an L7 proxy that was never on its path.

Going deeper

This section is for the reader who wants the internals — how the L4 floor actually moves bytes, exactly when a waypoint earns its place, and the tradeoffs a platform team weighs when migrating a real fleet.

The ambient data plane: HBONE tunnels and SPIFFE identity

When a pod is enrolled, the Istio CNI node agent reconfigures that pod’s networking so its traffic is transparently captured by the node-local ztunnel — no NET_ADMIN in the app pod, no init container, no restart. You can see the marker it leaves behind:

kubectl get pod <pod> -n shop -o yaml | grep ambient.istio.io/redirection

From there, ztunnel carries the connection over HBONE — HTTP-Based Overlay Network Environment. Concretely, the source node’s ztunnel opens an mTLS-secured HTTP/2 CONNECT tunnel to the destination node’s ztunnel on port 15008, and the original TCP connection is tunneled inside that stream. Two things ride along for free: the traffic is encrypted end to end between nodes, and each end presents a cryptographic identity.

That identity is SPIFFE. Every workload’s ServiceAccount is minted into a SPIFFE ID of the form spiffe://cluster.local/ns/<namespace>/sa/<serviceaccount>, delivered as a short-lived X.509 SVID certificate that istiod’s CA issues over the Secret Discovery Service. When you write an AuthorizationPolicy that matches principals: ["cluster.local/ns/shop/sa/web"], ztunnel is checking the peer’s SVID, not its IP address. This is why ambient policy survives pod rescheduling, IP churn, and NAT — identity is bound to the workload, not the network location. It is also why ztunnel is strictly L4: it terminates and re-originates mTLS and can reason about identity, source/destination, and ports, but the application bytes stay inside the tunnel, opaque to it. To read an HTTP method or path, something has to terminate L7 — and that is the waypoint’s job.

When you actually need a waypoint

The decision rule is short: you need a waypoint the instant a policy or route depends on Layer-7 semantics. That means any of:

• HTTP-aware authorization — matching on methods, paths, or request headers.
• End-user authentication — RequestAuthentication (JWT) and requiring a requestPrincipal.
• L7 traffic management — weighted or header-based routing, retries, timeouts, circuit breaking, fault injection, mirroring.
• Per-request telemetry — istio_requests_total with response codes and per-route labels.

If a service needs none of those — it only wants encryption, identity, and coarse L4 allow/deny — it never gets a waypoint, and that is the point. A waypoint attaches with targetRefs (to a Service or a Gateway), never selector, and it is a first-class scaling unit: a plain Deployment you can give HPA, resource requests, and its own node affinity. Because it sits on the request path for everything behind it, an undersized waypoint is a shared latency-and-availability bottleneck — capacity-plan it against peak L7 RPS, independent of how many app pods it fronts.

The --for service vs --for workload choice (the crux of the Enterprise scenario) turns on how clients address the destination: service intercepts traffic to the Service VIP — the common case, and what you want for routing and splits — while workload intercepts traffic to individual pod IPs, which you need for direct pod addressing like headless StatefulSet DNS.

The sidecar → ambient migration, and the cost/perf tradeoff

You can run sidecar and ambient in the same mesh, but not for the same workload — a given pod is one mode or the other. A pragmatic migration looks like:

1. Install the ambient components alongside your existing control plane (ztunnel DaemonSet + CNI node agent), leaving current sidecar namespaces untouched.
2. Migrate one namespace at a time: remove the sidecar injection label, add istio.io/dataplane-mode=ambient, and do a single rolling restart so the sidecars drop out. New pods come up sidecar-free and immediately meshed.
3. Re-create only the L7 features that were actually in use as waypoints on the specific services that needed them — not blanket across the namespace.

Cross-mode traffic keeps working during the migration because sidecars and ztunnel both speak mTLS; a sidecar pod and an ambient pod can call each other. What you must not do is straddle a single host with two routing APIs or two data-plane modes at once.

On performance, be precise rather than tribal. Section 1 noted a small latency tax for the extra hop out to the node-local ztunnel — that is real, because a sidecar does its work over localhost inside the pod. The flip side is that you drop the second in-pod proxy, so for L4-only traffic the node-to-node ztunnel↔ztunnel path is usually a wash with two sidecars, and often wins on tail latency because the proxies are warm, shared, and few. The genuine, opt-in cost is the waypoint: when a service has one, a request goes source ztunnel → waypoint → destination ztunnel, a deliberate extra L7 hop you pay for only where you asked for L7. Benchmark the HBONE path under realistic load before you commit it to a latency-critical route, and remember EnvoyFilter is not supported against waypoints — if your sidecar setup leaned on EnvoyFilter, that customization needs a deliberate rethink, not a lift-and-shift.

AuthorizationPolicy: L4 at ztunnel vs L7 at the waypoint

The same AuthorizationPolicy kind is enforced in two different places depending on what it matches and how it attaches. Internalize this table and most “my policy is being ignored” tickets solve themselves:

The trap: put L7 fields (methods, paths) in a policy that uses selector, and it lands on ztunnel, which cannot evaluate them — the L7 parts are silently dropped and the rule does not do what you think. Always attach L7 policy with targetRefs. At the waypoint the evaluation order is fixed: JWT validation, then CUSTOM (external authorization), then DENY, then ALLOW — and DENY always beats ALLOW, so you cannot widen a deny with a broader allow. Keep a mesh- or namespace-wide default-deny and express access as narrow, additive allows.

Canary and traffic-splitting patterns

Behind a waypoint, three splitting patterns cover almost everything:

• Weighted — the 90/10 in §6; ramp the weights over time for a progressive rollout.
• Header/identity-based — route x-canary: always (or a specific user cohort) to v2 while everyone else stays on v1; the safest way to dogfood.
• Mirroring (shadow traffic) — send a copy of live traffic to v2 and discard its responses, so you exercise the new version with production load without any user seeing its output:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: orders-mirror
  namespace: shop
spec:
  hosts: ["orders.shop.svc.cluster.local"]
  http:
  - route:
    - destination: { host: orders.shop.svc.cluster.local, subset: v1 }
    mirror: { host: orders.shop.svc.cluster.local, subset: v2 }
    mirrorPercentage: { value: 20.0 }

In production you rarely hand-edit weights. A progressive-delivery controller (Argo Rollouts or Flagger) drives the VirtualService weights automatically, watching success-rate and latency metrics and rolling back on a breach. The mesh provides the dial; the controller provides the hand on the dial. (See the sibling lesson on Gateway API traffic splitting for the HTTPRoute-native equivalent.)

The Gateway API integration

Ambient leans on the Kubernetes Gateway API more than sidecar Istio ever did. A waypoint is a Gateway (gatewayClassName: istio-waypoint). North-south ingress is a Gateway with gatewayClassName: istio plus HTTPRoutes. East-west (service-to-service) L7 routing can be expressed either with an HTTPRoute attached to a Service — the GAMMA initiative, “Gateway API for mesh” — or with the classic VirtualService/DestinationRule pair.

The practical guidance: Gateway API is becoming the primary, portable config surface, and you should reach for it first. But VirtualService/DestinationRule are not going anywhere yet, because they still own resilience features the Gateway API has not standardized — fault injection, circuit breaking (outlierDetection), connection pools, and traffic mirroring. Pick exactly one routing API per host: mixing a VirtualService and an HTTPRoute on the same host yields undefined precedence, and it is a top cause of “my route silently isn’t applied”.

Observability across the two proxies

Because the data plane is split, so is the telemetry:

• ztunnel (L4) emits connection-level metrics — istio_tcp_connections_opened_total, istio_tcp_connections_closed_total, bytes sent/received — on its :15020 endpoint. For an L4-only service, this is all you get, and it is correct: there are no HTTP requests to count because nothing is terminating HTTP.
• The waypoint (L7) emits the full request-level set — istio_requests_total labelled with response_code, source/destination workload, and route — exactly like a sidecar would.

This is the source of a classic beginner confusion: “my service has no request metrics / an empty Kiali graph”. Nine times out of ten the service simply has no waypoint, so there is no L7 proxy to produce request-level data — the fix is to add a waypoint, not to debug Prometheus. Use the Telemetry API (see §9) to configure access logging and tracing mesh-wide rather than per-proxy, and remember cardinality is now per shared proxy, not per pod, which is friendlier on your metrics backend.

Ambient GA status and version notes

Ambient mode was declared GA in Istio 1.24 (November 2024): ztunnel, waypoints, and the security and telemetry behaviors described here are production-supported, not experimental. The Kubernetes Gateway API is the recommended way to define waypoints and ingress. A few operational notes that age well:

• Keep istioctl and the control-plane version on matching minors; upgrade the L4 layer by rolling the ztunnel DaemonSet and each waypoint by rolling its Deployment — no application pods restart for a data-plane upgrade.
• Validate CNI chaining on managed platforms (AKS/EKS/GKE and their default CNIs each have ordering caveats).
• EnvoyFilter against waypoints remains unsupported — plan any migration that relied on it deliberately.
• Read the release notes for the minor you run; ambient is evolving quickly and each release tends to sand down a specific sharp edge.

Practice challenges

Work these in order against a kind/minikube cluster installed with the ambient profile (or read the solution and reason it through). Each escalates from the last.

1. (Beginner) Prove enrollment needs no restart. Create a namespace shop, run two pods in it, then enroll the namespace and prove the pods are in the L4 mesh without any restart.

<details> <summary>Solution</summary>

kubectl create namespace shop
kubectl label namespace shop istio.io/dataplane-mode=ambient
istioctl ztunnel-config workloads --workload-namespace shop

The existing pods show protocol HBONE in the output, and their AGE/restart count is unchanged. Why: the Istio CNI node agent redirects existing pods’ traffic to ztunnel in place — enrollment is a label, not an injection. </details>

2. (Beginner) Opt one pod out. A batch job in shop opens a raw TCP protocol that dislikes redirection. Exclude just that pod from the mesh, leaving the rest enrolled.

<details> <summary>Solution</summary>

kubectl label pod <job-pod> -n shop istio.io/dataplane-mode=none --overwrite

Why: the pod-level dataplane-mode=none label overrides the namespace default for that one pod, so ztunnel stops capturing its traffic. </details>

3. (Intermediate) Lock down with STRICT + default-deny, then allow one caller at L4. Make the mesh reject plaintext, deny everything in shop by default, then allow only the web ServiceAccount to reach orders — enforced at ztunnel (no waypoint).

<details> <summary>Solution</summary>

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: istio-system }
spec: { mtls: { mode: STRICT } }
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: deny-all, namespace: shop }
spec: {}
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: orders-allow-web, namespace: shop }
spec:
  selector: { matchLabels: { app: orders } }
  action: ALLOW
  rules:
  - from: [ { source: { principals: ["cluster.local/ns/shop/sa/web"] } } ]

Why: STRICT forbids plaintext, the empty-spec policy denies all, and the third policy uses selector + principals only (pure L4), so ztunnel enforces it — no waypoint required. </details>

4. (Intermediate) Add L7 authorization. Now require that web may only GET/POST under /api/* on orders. What must you deploy first, and why does the policy shape change?

<details> <summary>Solution</summary>

istioctl waypoint apply -n shop --enroll-namespace   # or label just the orders Service

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: orders-l7, namespace: shop }
spec:
  targetRefs:
  - { kind: Service, group: "", name: orders }
  action: ALLOW
  rules:
  - from: [ { source: { principals: ["cluster.local/ns/shop/sa/web"] } } ]
    to:   [ { operation: { methods: ["GET","POST"], paths: ["/api/*"] } } ]

Why: methods/paths are L7, which ztunnel cannot read, so you need a waypoint. The policy attaches with targetRefs (not selector) so it lands on that waypoint. </details>

5. (Advanced) Canary with an escape hatch. Split orders 80/20 v1/v2, but always send requests carrying x-canary: always to v2 regardless of weight.

<details> <summary>Solution</summary>

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: orders, namespace: shop }
spec:
  host: orders.shop.svc.cluster.local
  subsets:
  - { name: v1, labels: { version: v1 } }
  - { name: v2, labels: { version: v2 } }
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: orders, namespace: shop }
spec:
  hosts: ["orders.shop.svc.cluster.local"]
  http:
  - match: [ { headers: { x-canary: { exact: always } } } ]
    route: [ { destination: { host: orders.shop.svc.cluster.local, subset: v2 } } ]
  - route:
    - { destination: { host: orders.shop.svc.cluster.local, subset: v1 }, weight: 80 }
    - { destination: { host: orders.shop.svc.cluster.local, subset: v2 }, weight: 20 }

Why: the header match block is evaluated first and short-circuits to v2; the weighted route is the fallthrough. This all requires a waypoint on orders (L7 routing). </details>

6. (Advanced) Diagnose a silent L7 policy. A teammate reports their orders-l7 policy (matching paths: ["/admin/*"]) “does nothing” — every caller still gets through. The waypoint is PROGRAMMED=True. What is the most likely bug, and how do you confirm it?

<details> <summary>Solution</summary>

Most likely the policy attaches with selector instead of targetRefs, so it landed on ztunnel, which cannot evaluate paths and silently ignores the L7 rule.

kubectl get authorizationpolicy orders-l7 -n shop -o yaml | grep -A2 -E 'selector|targetRefs'
istioctl proxy-config listeners deploy/waypoint -n shop   # confirm the waypoint has the rule

Fix: replace selector with targetRefs pointing at the orders Service (or the waypoint Gateway). Re-test that /admin/* now returns 403 for unauthorized callers. </details>

Common beginner mistakes

• Expecting L7 rules to work without a waypoint. The misconception is “ambient does HTTP”. It does not — ztunnel is strictly L4 and cannot read a method, path, or header. Any rule that references those needs a waypoint, attached with targetRefs. Right mental model: L4 floor is free and automatic; L7 is an opt-in you deploy where you need it.

• Reaching for a mesh to protect a single service. A service mesh earns its complexity across a fleet of services that all need mTLS, identity-based policy, and uniform telemetry. For one or two apps, a NetworkPolicy plus application-level TLS is often simpler and cheaper. Right mental model: adopt a mesh for fleet-wide zero-trust, not to secure one Deployment.

• Flipping mTLS to STRICT while non-mesh clients still call in plaintext. STRICT rejects all plaintext, so the instant you apply it, any caller still speaking plaintext — a legacy probe, a job, a service you forgot was outside the mesh — starts failing. Right mental model: migrate through PERMISSIVE, verify from telemetry that everyone is already on mTLS, then flip to STRICT — and scope it per namespace while you migrate.

• Mixing sidecar and ambient on the same workload, or two routing APIs on one host. A pod is either sidecar-meshed or ambient-meshed, never both, and a single host must be governed by one routing API (VirtualService or Gateway API HTTPRoute, not both) or precedence is undefined. Right mental model: one data-plane mode per workload, one routing API per host.

• Treating “enrolled” as “secured”. Labelling a namespace encrypts traffic and tags it with identity, but it does not deny anything — plaintext is still accepted and every identity is still allowed until you say otherwise. Right mental model: encryption and authorization are two separate switches; enrollment flips the first, STRICT + default-deny flips the second.

• Putting a --for service waypoint where pods are addressed directly. If clients dial pods by their IP or headless pod DNS (common with StatefulSets), a service-scoped waypoint never sees that traffic. Right mental model: --for service intercepts Service-VIP traffic; direct pod addressing needs --for workload — or no waypoint at all.

Glossary

• Service mesh — an infrastructure layer that adds mTLS, traffic control, and observability to service-to-service calls without changing application code.
• Data plane — the proxies that actually carry and control traffic (ztunnel and waypoints in ambient; per-pod Envoy in sidecar mode).
• Control plane (istiod) — the Istio component that configures the data plane and issues workload identity certificates.
• Sidecar — an Envoy proxy injected into each pod that handles both L4 and L7; the classic, per-pod mesh model.
• Ambient mode — Istio’s sidecar-less data plane: a shared per-node L4 proxy plus optional per-service L7 proxies.
• ztunnel (zero-trust tunnel) — the per-node DaemonSet proxy that provides mTLS and L4 identity/authorization for every enrolled pod. L4-only.
• Waypoint — an optional, per-service (or per-namespace) L7 proxy, deployed as a Gateway API Gateway, that does HTTP routing and L7 authorization.
• HBONE — HTTP-Based Overlay Network Environment; the mTLS-wrapped HTTP/2 CONNECT tunnel (port 15008) ztunnel uses to carry traffic between nodes.
• SPIFFE / SVID — a portable workload-identity standard; Istio issues each workload a SPIFFE ID (cluster.local/ns/<ns>/sa/<sa>) as a short-lived X.509 SVID certificate.
• mTLS (mutual TLS) — both sides of a connection present certificates and verify each other, giving encryption and authentication.
• PeerAuthentication — the API that sets mTLS mode: PERMISSIVE (both), STRICT (mTLS only), or DISABLE.
• AuthorizationPolicy — the API that allows/denies traffic; enforced at ztunnel (L4) or a waypoint (L7) depending on what it matches and how it attaches.
• RequestAuthentication — defines how to validate end-user JWTs at a waypoint; it validates but does not reject on its own.
• L4 vs L7 — Layer 4 is TCP-level (identities, ports, connections); Layer 7 is HTTP-level (methods, paths, headers, JWTs).
• VirtualService — Istio routing rules: match traffic and route it to destinations/subsets, with retries, timeouts, and fault injection.
• DestinationRule — defines subsets (by pod labels) and traffic policy (connection pools, outlier detection / circuit breaking).
• Subset — a named slice of a Service’s pods (e.g. version: v2) that routing rules can target.
• Gateway API — the Kubernetes-native, portable API for routing; ambient uses it for waypoints (istio-waypoint) and ingress (istio).
• HTTPRoute — the Gateway API object for HTTP routing; the GAMMA initiative extends it to east-west (mesh) traffic.
• targetRefs vs selector — how a policy attaches: targetRefs (Service/Gateway) lands on a waypoint for L7; selector (pod labels) lands on ztunnel for L4.
• gatewayClassName — the Gateway API field that picks the implementation: istio-waypoint for a mesh waypoint, istio for ingress.
• Outlier detection / circuit breaking — automatically ejecting an unhealthy backend from the pool after repeated errors, configured in DestinationRule.
• Telemetry API — configures mesh-wide access logging and tracing sampling instead of per-proxy flags.

Checklist

• Ambient profile installed; ztunnel and istio-cni-node running on every node
• Gateway API CRDs present
• Namespace labeled istio.io/dataplane-mode=ambient, workloads showing HBONE
• Mesh-wide PeerAuthentication set to STRICT
• Default-deny AuthorizationPolicy in place, holes punched explicitly
• Waypoint deployed and PROGRAMMED=True only where L7 is needed
• Routing via one API per host (VirtualService or HTTPRoute, not both)
• L7 authz attached with targetRefs; JWT validated via RequestAuthentication
• Resilience (timeouts/retries/outlier detection) and verified with fault injection
• Telemetry API configured; access logs and traces flowing
• Load-test the HBONE path to size the latency tax for your SLOs
• Capacity-plan waypoint replicas against peak L7 RPS, separate from app pods

Pitfalls and next steps

The mistakes that bite in production: treating “enrolled” as “secured” (enrollment encrypts but does not deny — you still need STRICT plus default-deny); putting an L7 policy on ztunnel and wondering why HTTP rules vanish; and forgetting the waypoint is on the request path, so an undersized one becomes a latency and availability bottleneck for every service behind it. EnvoyFilter is not supported against waypoints — if you relied on it with sidecars, plan that migration deliberately.

From here: add a waypoint only to services that earn it, keep the long tail L4-only to bank the savings, wire Telemetry output into your existing Prometheus/Grafana and tracing backend, and benchmark the HBONE hop under realistic load before committing it to a latency-sensitive path. For a deeper drill on waypoint-based L7 authorization, see the sibling lesson Istio Ambient: waypoint proxies & L7 authorization; to contrast ambient’s approach with a different mesh, compare Linkerd: mTLS, retries & multi-cluster failover.