Linkerd in Production: Automatic mTLS, Retry/Timeout Budgets, and Multicluster Failover

Linkerd’s pitch is restraint: a Rust micro-proxy that does mTLS, load balancing, retries, and metrics in single-digit megabytes, with no Lua, no WASM, and a CLI that tells you the truth when something is misconfigured. You give up the kitchen-sink extensibility of an Envoy mesh; you get a data plane you can reason about at 3am. This guide takes a workload from unmeshed to zero-trust, layers on retry and timeout budgets that prevent retry storms, then links two clusters and wires automatic failover.

Everything here targets the stable 2.x line (validated against 2.15/2.16 behavior). Where a feature has a sharp edge — ServiceProfiles vs Gateway API, the 64KiB retry ceiling, gateway-mode latency — I call it out rather than paper over it.

In a nutshell

Level: Advanced · Time: ~35–40 min

A service mesh solves a problem you feel the moment you have more than a handful of services talking to each other: every one of them needs the same boring plumbing — encrypt the connection, retry a dropped request, give up on a call that hangs, and record how it went. Writing that into every app, in every language, and keeping it consistent, is a losing battle. A mesh moves that plumbing out of the app and onto the network.

Linkerd is the deliberately small one. It attaches a tiny, identical helper — an ultralight Rust micro-proxy called linkerd2-proxy — right next to each of your containers. That helper is the whole trick. From the instant it is there, every call your service makes to another meshed service is automatically encrypted and mutually authenticated (mTLS), retried within a safe budget, timed out if it hangs, and measured (success rate, requests per second, latency percentiles) — and you did not change a single line of application code to get any of it.

The mental model: imagine giving every service its own post-office clerk who sits at the door. The clerk seals and stamps every outgoing letter so only the right recipient can open it (mTLS), re-sends a letter that got lost up to a sensible limit (retries), stops waiting on a reply that is taking far too long (timeouts), and keeps a tally of every letter’s fate (golden metrics). The clerks are identical, cheap, and managed centrally by a small back office (the control plane) that hands out their ID badges and collects their tallies. Your app just writes letters like it always did.

Where Linkerd differs from the bigger mesh (Istio) is restraint by design: fewer knobs, a proxy small enough to reason about, and secure defaults you cannot forget to turn on. You trade deep extensibility for a mesh that a small team can actually operate. This lesson takes you from that idea to a production deployment: custom certificates, retry/timeout budgets that cannot cause a retry storm, canary releases gated on the mesh’s own metrics, and two clusters wired so one automatically covers for the other.

Prerequisites and what you’ll be able to do

You will get the most from this lesson if you are comfortable with Kubernetes Services and the ClusterIP/port model, know what a sidecar container is, and have at least seen TLS certificates (a root CA signing an intermediate signing a leaf). You do not need any prior mesh experience — we build it up from zero. If you want the wider “how many clusters, and why” picture, The architecting ladder: single cluster to multi-region is good background, and the Gateway API / HTTPRoute lesson covers the routing API Linkerd now reuses.

After working through it you will be able to:

Explain what the control plane and the linkerd2-proxy data plane each do, and why the proxy being a Rust micro-proxy matters operationally.
Install Linkerd with your own trust anchor, and rotate the issuer without disrupting traffic.
Mesh a workload and prove mTLS is really happening with linkerd viz edges and tap — not just assume it.
Set retry budgets and timeouts that make a service more reliable without ever amplifying an outage.
Run a canary and gate promotion on Linkerd’s golden metrics.
Link two clusters, mirror a service, and configure automatic failover that survives losing a whole cluster.
Reason about Linkerd vs Istio, and lock down traffic with Server/ServerAuthorization policy.

The mesh at a glance

Two meshed pods each with a linkerd2 Rust micro-proxy sidecar: automatic mTLS and retry/timeout budgets on the pod-to-pod hop, identity and golden metrics exchanged with the control plane, and a cross-cluster failover edge to a mirrored standby service on another cluster

Read it left to right. Pod A and Pod B are ordinary application pods, except each has picked up a second container — the linkerd2-proxy sidecar (badge 1). Your app in Pod A makes a normal HTTP call; iptables quietly routes it through its proxy, which opens a mutually authenticated, encrypted connection to Pod B’s proxy (badge 2, automatic mTLS) and applies retry and timeout budgets on the way (badge 3). Both proxies talk to the control plane in the linkerd namespace: the identity service issues each proxy a short-lived certificate tied to its ServiceAccount (badge 4), and every proxy streams golden metrics — success rate, RPS, p50/p95/p99 — that linkerd viz reads back (badge 5). Finally, when Pod B is the primary backend of a failover split and its success rate collapses, a linkerd-failover controller shifts weight to a mirrored standby service on another cluster (badge 6), reachable over the shared trust anchor. Every numbered badge is a place the lesson returns to and makes concrete.

1. Architecture: why the micro-proxy is different

Linkerd has two layers. The control plane (destination, identity, proxy-injector) runs in the linkerd namespace. The data plane is linkerd2-proxy, a purpose-built Rust proxy injected as a sidecar next to each application container.

If you are new to meshes, the two words to anchor on are data plane and control plane. The data plane is the fleet of proxies that actually sit in the request path and move your bytes — one per pod. The control plane is the small set of central components that configure and support those proxies (hand out identities, answer “where does this service live?”, inject the sidecar) but never touch an individual request. A useful analogy: the data plane is every traffic light on every corner; the control plane is the traffic-management office that programs them. Break the office and the lights keep their last program running — which is exactly why a control-plane outage does not immediately drop your traffic.

The design choices that matter operationally:

Concern	Linkerd	Typical Envoy mesh
Data plane	Rust micro-proxy, ~10-20Mi/pod	Envoy, ~50-100Mi/pod
Config surface	Opinionated, few knobs	xDS, near-infinite knobs
mTLS	On by default, zero config	Opt-in policy, PeerAuthentication
Extensibility	Intentionally limited	Lua / WASM / ext_authz
Identity	Per-workload cert from `identity` service	Per-workload SPIFFE

The identity component is a CA that issues short-lived (24h default) leaf certs to each proxy, keyed to the pod’s Kubernetes ServiceAccount; proxies rotate them automatically. Because identity is bootstrapped from the ServiceAccount token, mTLS is on the moment a pod is meshed — there is no separate “turn on mTLS” step like Istio’s PeerAuthentication. The secure default is the only default, and that is the single biggest reason teams reach for Linkerd.

Mental model: the trust anchor (root CA) is long-lived and you guard it like a crown jewel. The issuer (intermediate CA) is what actually signs proxy certs, lives in a Kubernetes Secret, and you rotate it on a schedule. Leaf certs are ephemeral and you never touch them.

What “meshing” actually does to a pod. When a pod is meshed, the proxy-injector webhook mutates its spec to add two things: the linkerd-proxy sidecar container, and a tiny linkerd-init init container (or a CNI plugin) that installs the iptables rules routing the pod’s inbound and outbound TCP through the proxy on ports 4143/4140. Your application container is byte-for-byte unchanged — it still thinks it is calling http://emoji.emojivoto.svc:8080 directly. That transparency is the point: the mesh is something you add to the platform, not something your developers import.

2. Install with a custom trust anchor

Never run a production mesh on the auto-generated certs from linkerd install with no arguments — they expire in a year and you cannot rotate the issuer independently of the anchor. Generate your own with the step CLI.

A quick decode of the three-layer certificate chain before the commands, because it is where a beginner gets lost: the trust anchor is the root certificate — the one every proxy is configured to trust as the ultimate authority. The issuer is an intermediate certificate the root signs, and it is what actually signs the per-proxy leaf certs day to day. You want the root to live a very long time and stay offline (compromising it compromises the whole mesh), and you want the intermediate to be short-lived and routinely rotated (it lives online in a Secret, so it is the exposed one). This is the same pattern the public web PKI uses.

# Long-lived root (trust anchor) — 10 years, kept OFFLINE after this
step certificate create root.linkerd.cluster.local ca.crt ca.key \
  --profile root-ca --no-password --insecure --not-after=87600h

# Issuer (intermediate) — 1 year, signed by the root
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
  --profile intermediate-ca --not-after=8760h --no-password --insecure \
  --ca ca.crt --ca-key ca.key

Install the CRDs first, then the control plane wired to those certs:

linkerd install --crds | kubectl apply -f -

linkerd install \
  --identity-trust-anchors-file ca.crt \
  --identity-issuer-certificate-file issuer.crt \
  --identity-issuer-key-file issuer.key \
  | kubectl apply -f -

linkerd check

For GitOps, prefer Helm so the chart owns the lifecycle:

helm install linkerd-crds linkerd/linkerd-crds -n linkerd --create-namespace

helm install linkerd-control-plane -n linkerd \
  --set-file identityTrustAnchorsPEM=ca.crt \
  --set-file identity.issuer.tls.crtPEM=issuer.crt \
  --set-file identity.issuer.tls.keyPEM=issuer.key \
  linkerd/linkerd-control-plane

Rotating the issuer

The issuer is the cert you rotate routinely. Because every proxy already trusts the root, swapping the intermediate is non-disruptive — you do not need to bundle anything.

# New intermediate, same root
step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key \
  --profile intermediate-ca --not-after 8760h --no-password --insecure \
  --ca ca.crt --ca-key ca.key

linkerd upgrade \
  --identity-issuer-certificate-file=./issuer-new.crt \
  --identity-issuer-key-file=./issuer-new.key \
  | kubectl apply -f -

# Proxies pick up the new issuer on their next cert rotation; force it to verify
kubectl -n emojivoto rollout restart deploy
linkerd check --proxy

Rotating the trust anchor is the harder case — you bundle old + new so proxies trust both during the transition, roll everything, re-issue the intermediate from the new root, roll again, then drop the old anchor. That four-step dance is why you give the root a 10-year life and rotate the issuer instead.

Automate the expiry watch, not just the rotation. linkerd check --proxy reports issuer and trust-anchor expiry; a common production setup runs it on a schedule and alerts at, say, 30 days out. A silently expired issuer means new pods cannot get an identity and fail to start meshed — a slow, confusing outage that a one-line cron check prevents.

3. Mesh workloads and prove mTLS

Meshing is injection: the proxy-injector webhook adds the sidecar when it sees the linkerd.io/inject: enabled annotation. Annotate the namespace so every new pod is meshed:

kubectl annotate namespace emojivoto linkerd.io/inject=enabled
kubectl -n emojivoto rollout restart deploy   # existing pods need a restart to get the sidecar

For one-off or pipeline use, inject at apply time:

kubectl apply -k github.com/BuoyantIO/emojivoto/kustomize/deployment
kubectl -n emojivoto get deploy -o yaml | linkerd inject - | kubectl apply -f -

The annotation is inherited but overridable: a namespace annotated enabled meshes every pod in it, but an individual workload can opt out with linkerd.io/inject: disabled on its pod template. That per-workload override is how you keep a Job or a debug pod out of the mesh without unmeshing the whole namespace.

Verify with Viz

Install the observability extension, then prove mTLS is actually happening rather than assuming it.

linkerd viz install | kubectl apply -f -
linkerd viz check

edges shows you which traffic is secured. The SECURED column is the source of truth:

linkerd viz -n emojivoto edges deployment

SRC          DST          SRC_NS     DST_NS     SECURED
web          emoji        emojivoto  emojivoto  √
web          voting       emojivoto  emojivoto  √
vote-bot     web          emojivoto  emojivoto  √

tap streams live requests with a tls field. tls=true means the connection was mTLS’d between two meshed identities; tls=no_tls_from_remote is normal for kubelet health probes, which have no mesh identity:

linkerd viz -n emojivoto tap deploy/web
# req id=0:1 proxy=in  src=10.1.0.5:48000 dst=10.1.0.9:8080 tls=true :method=GET :path=/api/list

If SECURED is blank or tls=disabled for app-to-app traffic, the destination pod is not meshed — check the injector annotation and that you restarted the deployment.

4. Retry budgets and timeouts

This is where teams get reliability right or badly wrong. A naive “retry on 5xx, 3 attempts” turns a brief dependency blip into a self-inflicted DDoS: every failure triples load exactly when the system can least absorb it. Linkerd’s answer is a retry budget — retries are capped as a fraction of live traffic, not a fixed per-request count, so they can never become a runaway multiplier.

To make the budget idea concrete: with the default 20% budget, a service doing 1,000 req/s may add at most ~200 retries/s no matter how many requests are failing. A fixed “3 retries” policy on the same service during a 50% failure event would try to add ~1,000 extra req/s (500 failures × up to 2 more attempts each) — on top of the original load, aimed at the already-struggling dependency. The budget converts “retry everything three times” into “retry as much as is safe, then stop.” That single design choice is why the mesh makes you more reliable instead of amplifying the next outage.

There are two APIs. Gateway API (HTTPRoute annotations) is the current path for new clusters; as of 2.16 it supplants ServiceProfile for routing, timeouts, and retries. ServiceProfile is the legacy API that remains supported and is still the way to express an explicit retryBudget object. I cover both because brownfield clusters run a mix.

Retries and timeouts via HTTPRoute (current)

Retries and timeouts are plain annotations on an HTTPRoute. Note: route annotations are incompatible with a ServiceProfile for the same service — if a ServiceProfile exists, it wins and these are ignored.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: web-default
  namespace: emojivoto
  annotations:
    retry.linkerd.io/http: 5xx        # also: gateway-error (502-504), or a range like 500-504
    retry.linkerd.io/limit: "2"       # max attempts per request
    retry.linkerd.io/timeout: 300ms   # cancel+retry a slow attempt after this
    timeout.linkerd.io/request: 2s    # total budget across all attempts
    timeout.linkerd.io/response: 1s   # single backend response ceiling
spec:
  parentRefs:
    - name: web
      kind: Service
      group: core
      port: 80
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: "/api"

Two ceilings worth memorizing because they bite people: requests larger than 64KiB are not retried (the proxy will not buffer an unbounded body), and retry.linkerd.io/limit is a per-request attempt cap that operates underneath the global budget — the budget is the backstop that prevents aggregate retry traffic from exceeding the ratio.

Explicit retry budget via ServiceProfile

When you need to tune the budget itself — the ratio, the free-retry floor, the accounting window — that lives in a ServiceProfile. The default budget is generous: 20% extra load plus 10 free retries/sec. Tighten it for sensitive backends:

apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: web-svc.emojivoto.svc.cluster.local
  namespace: emojivoto
spec:
  routes:
    - name: GET /api/list
      condition:
        method: GET
        pathRegex: /api/list
      isRetryable: true        # GET is idempotent — safe to retry
      timeout: 600ms
    - name: POST /api/vote
      condition:
        method: POST
        pathRegex: /api/vote
      isRetryable: false       # never blind-retry a non-idempotent write
  retryBudget:
    retryRatio: 0.1            # retries may add at most 10% to live traffic
    minRetriesPerSecond: 5     # plus this free floor for low-RPS routes
    ttl: 10s                   # rolling window for computing the ratio

The discipline that prevents outages: only mark idempotent routes isRetryable: true. A retried POST /vote double-counts; a retried POST /charge double-bills. Idempotency is a correctness property of the endpoint, not a mesh setting — the mesh only enforces what you assert.

5. Traffic splitting for canary by golden metrics

Linkerd does weighted splits with the SMI TrafficSplit resource, and the data plane already emits the golden metrics (success rate, RPS, p50/p95/p99) you need to gate a canary. Run web stable at 90% and web-canary at 10%:

apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
  name: web-split
  namespace: emojivoto
spec:
  service: web                 # the apex Service clients address
  backends:
    - service: web
      weight: 90
    - service: web-canary
      weight: 10

Watch the canary’s golden metrics live before you shift more weight:

linkerd viz -n emojivoto stat deploy/web-canary --from deploy/vote-bot
# NAME        MESHED   SUCCESS   RPS   LATENCY_P95   LATENCY_P99
# web-canary     1/1   100.00%  4.2   12ms          28ms

The promotion loop is mechanical: hold weight, watch success rate and p99 for a few minutes, step the weight up, repeat. Flagger automates exactly this loop against Linkerd if you want it controller-driven — the metric source is the same proxy data.

“Golden metrics” is not Linkerd jargon — it is the industry’s shorthand (from Google’s SRE book, the “golden signals”) for the four numbers that tell you a service’s health without knowing anything about its internals: success rate, traffic (RPS), latency, and saturation. Linkerd gives you the first three for every meshed workload for free, because every request already passes through a proxy that can count it. That is the observability dividend of a mesh: you get uniform, per-route golden metrics you would otherwise have to instrument by hand in every service.

6. Multicluster: gateway and service mirroring

Linkerd connects clusters with a gateway plus a service-mirror controller. The mirror watches a target cluster for exported services and creates local mirror Services (named <svc>-<cluster>) that resolve through the remote gateway. Cross-cluster traffic stays mTLS’d end to end, with the gateway as the only exposed ingress.

Both clusters must share the same trust anchor — that is what lets a proxy in west validate a proxy in east. Install the multicluster extension on both, with identical ca.crt.

# Install the gateway + service-mirror on each cluster
for ctx in west east; do
  linkerd --context=${ctx} multicluster install | \
    kubectl --context=${ctx} apply -f -
done

linkerd --context=west multicluster check
linkerd --context=east multicluster check

Link west so it mirrors from east. The link command, run against the target, emits a Link CR (gateway address, gateway identity, mirror credentials) you apply on the source:

linkerd --context=east multicluster link --cluster-name east | \
  kubectl --context=west apply -f -

linkerd --context=west multicluster gateways
# CLUSTER  ALIVE  NUM_SVC  LATENCY
# east     True         2     31ms

Nothing mirrors until you explicitly export a service — this is opt-in by design, so you never accidentally expose an internal service cross-cluster:

# On east: expose podinfo to linked clusters
kubectl --context=east -n test label svc podinfo mirror.linkerd.io/exported=true

A podinfo-east Service now appears in west and resolves to the east gateway:

kubectl --context=west -n test get svc
# NAME           TYPE        CLUSTER-IP      PORT(S)
# podinfo-east   ClusterIP   10.43.81.12     9898/TCP

Call it like any local Service: http://podinfo-east.test.svc.cluster.local:9898. Be honest about the cost: gateway mode adds a network hop and the gateway’s latency to every cross-cluster call — fine for failover, not where you want chatty east-west traffic by default.

7. Automatic failover when a cluster degrades

The linkerd-failover operator turns a TrafficSplit into an active/standby controller. You declare a primary backend; when its success rate collapses, the operator gradually shifts weight to the standby backends — including a mirrored service on another cluster — and shifts back when the primary recovers.

Install the operator (requires linkerd-smi on 2.12+ since SMI is no longer bundled):

helm repo add linkerd https://helm.linkerd.io/stable
helm repo update
helm install linkerd-failover -n linkerd-failover --create-namespace \
  linkerd/linkerd-failover

Declare local podinfo as primary, with the mirrored podinfo-east as standby (weight 0 until needed). Two annotations/labels drive it: failover.linkerd.io/primary-service names the primary, and the controlled-by label tells the operator to manage this split.

apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
  name: podinfo
  namespace: test
  annotations:
    failover.linkerd.io/primary-service: podinfo
  labels:
    failover.linkerd.io/controlled-by: linkerd-failover
spec:
  service: podinfo
  backends:
    - service: podinfo           # local primary
      weight: 1
    - service: podinfo-east      # mirrored standby on east
      weight: 0

When local podinfo starts failing health checks, the operator drains weight off the primary and onto podinfo-east; when it recovers, weight returns. The failover is gradual on purpose — dumping 100% onto the standby in one step can overload the surviving cluster or blow your latency SLO from the added gateway hop. Let it ramp.

Going deeper

Everything above is enough to run Linkerd. This section is for when you need to defend the design in a review, tune it at scale, or debug the parts that only bite in production.

The Rust micro-proxy, and why it is lighter than Envoy

linkerd2-proxy is a bespoke L7 proxy written in Rust on the Tokio async runtime, using Hyper for HTTP and a rustls-based TLS stack. It is not a general-purpose proxy that happens to be small — it is scoped to exactly what a mesh sidecar needs (HTTP/1.1, HTTP/2, gRPC, opaque TCP, mTLS, retries, timeouts, metrics, tap) and nothing else. Envoy is a magnificent general proxy with an xDS configuration API, WASM/Lua extensibility, and dozens of filters; that generality is what makes it 50–100Mi per pod and gives it a configuration surface measured in thousands of fields. Linkerd’s bet is that a mesh sidecar should be a micro-proxy: no dynamic config language, no plugins, just a fast, memory-safe forwarder that the control plane drives with a narrow gRPC discovery API.

The operational consequences are concrete. Per-pod memory in the low tens of megabytes means the sidecar tax on a 5,000-pod cluster is gigabytes, not tens of gigabytes. Rust’s memory safety removes an entire class of CVE that has historically dogged C++ proxies. And because there is no config language, there is no “the proxy is doing something and I cannot tell what” — the behavior is the code, and the CLI (linkerd check, linkerd diagnostics) exposes it. The flip side, which you must accept going in: if you need an ext_authz callout to a custom policy engine, a WASM filter, or exotic load-balancing algorithms, Linkerd will not do it, and that is a deliberate omission, not a roadmap gap.

One performance detail worth knowing: Linkerd’s load balancing is EWMA (exponentially-weighted moving average) latency-aware at L7, and it balances over individual endpoints (pods), not over the Service VIP. That means it can route around a slow pod within a Service automatically — a reliability win that kube-proxy’s random/round-robin L4 balancing cannot give you.

Automatic mTLS: identity, SPIFFE, and rotation internals

Each proxy’s identity is a certificate whose subject is a SPIFFE-style identity derived from the pod’s ServiceAccount, of the form <serviceaccount>.<namespace>.serviceaccount.identity.linkerd.cluster.local. At startup the proxy generates a private key, creates a CSR, proves who it is by presenting its ServiceAccount projected token to the identity service, and gets back a 24h leaf cert. The proxy re-requests a fresh cert well before expiry (roughly at the cert’s midpoint), so leaf rotation is continuous and invisible. Private keys never leave the pod; the identity service only ever sees CSRs.

This is why mTLS “just works”: the trust root is the same thing Kubernetes already uses to establish workload identity (the ServiceAccount), so there is nothing for you to distribute. It is also why the trust anchor is the crown jewel — it is the single root that every proxy in the mesh (and, in multicluster, every proxy in every linked cluster) validates against. Rotating it is the one genuinely disruptive certificate operation, which is the whole argument for a 10-year offline root and a routinely-rotated online issuer.

Two production facts to internalize: (1) certificate validity depends on reasonably synchronized clocks — significant node clock skew shows up as spurious identity failures, so NTP is a hard dependency, not a nicety; (2) linkerd check --proxy surfaces both issuer and anchor expiry, and wiring it into CI or a scheduled job is the difference between a planned rotation and a 2am outage when the issuer silently lapses.

Retry budgets vs fixed retries, and how timeouts bound them

The budget’s three fields encode a control loop. retryRatio is the steady-state ceiling (retries as a fraction of successful requests); minRetriesPerSecond is a free floor so a low-traffic route can still retry a little without needing a large base to draw a ratio from; ttl is the accounting window over which the ratio is computed. Together they guarantee that aggregate retry traffic is self-limiting — as failures rise and successes fall, the number of allowed retries falls with them, which is the exact opposite of a fixed-count policy that ramps retries up precisely when the system is drowning.

Timeouts are the other half. timeout.linkerd.io/response bounds a single backend attempt; retry.linkerd.io/timeout cancels a slow attempt so a retry can be tried; timeout.linkerd.io/request is the hard ceiling across all attempts. Get the arithmetic right: if a request timeout is 2s and each attempt can take up to 1s, you have room for one retry, not three. Set the total-request timeout below your client’s own timeout, or the client gives up first and the mesh’s retry does nothing but add load. And remember the 64KiB rule — a large-body POST is never retried regardless of policy, because the proxy will not buffer an unbounded request body to replay it.

Traffic splitting: SMI TrafficSplit vs Gateway API weights

The TrafficSplit resource above is SMI (split.smi-spec.io), and SMI is effectively legacy — on 2.12+ it needs the separate linkerd-smi extension because it is no longer bundled. The modern way to express a weighted split is Gateway API HTTPRoute backendRefs with weight, the same mechanism the Gateway API lesson covers, which Linkerd’s destination controller now understands natively. New clusters should reach for HTTPRoute weights for routing and canaries; the reason this lesson still shows TrafficSplit is that linkerd-failover is built around it, so failover remains an SMI story even as ordinary canaries move to Gateway API. Keep that split in your head: canary routing → Gateway API; automatic failover → SMI TrafficSplit + linkerd-failover.

Multicluster networking: gateway mode vs flat (pod-to-pod) mode

The gateway model in section 6 is the portable default: cross-cluster calls hop through the remote cluster’s linkerd-gateway, which is the only thing exposed between clusters, and it works even when the two pod networks cannot reach each other. The cost is a network hop and the gateway’s latency on every cross-cluster request, plus the gateway being a throughput chokepoint for east-west traffic.

When your clusters share flat, routable pod networking (for example, VPC-peered subnets or a CNI that spans clusters), Linkerd also supports pod-to-pod multicluster (link with --gateway=false), where a proxy in west dials the actual pod IP in east directly, mTLS end to end, with no gateway hop. That removes the latency tax and the chokepoint, at the cost of requiring real network reachability between pods. The decision is a networking one, not a mesh one: gateway mode for isolated networks and simple failover; flat mode when you have the connectivity and want cross-cluster traffic to feel local.

Observability: how viz, tap, and the metrics pipeline fit

linkerd-viz is an extension, not the core — it bundles (or points at an external) Prometheus that scrapes every proxy’s /metrics, plus the web dashboard, the tap API, and the metrics-api. linkerd viz stat/edges/routes are queries against that Prometheus; they are aggregate and cheap. tap is different in kind: it opens an on-demand live stream of individual request metadata straight from the proxies (method, path, tls, latency, response code) — invaluable for “is this specific call mTLS’d / why is it 500ing” but not something you leave running, since it taxes the proxies. For long-term retention, point viz at an external Prometheus (the bundled one is in-memory and ephemeral) or federate it into your existing stack; Linkerd also emits everything you need to build Grafana dashboards, which it no longer ships by default.

Policy: Server, ServerAuthorization, and L7 authz

mTLS proves who is calling; policy decides whether they are allowed. Linkerd’s policy layer is default-allow until you opt a workload into a stricter posture (per-workload or cluster-wide via proxy.defaultInboundPolicy, e.g. deny, all-authenticated, cluster-authenticated). You then describe intent with CRDs: a Server selects a specific port on a set of pods and declares it a protected resource; a ServerAuthorization (or the newer AuthorizationPolicy + MeshTLSAuthentication/NetworkAuthentication) says which meshed identities may reach that Server. For L7 rules — “only GET /api/list from the web identity, deny the rest” — you attach an HTTPRoute to the Server and authorize per route. The result is identity-based, mTLS-backed authorization enforced by the proxy itself, without the app knowing. Start default-allow, lock down one sensitive Server at a time, and verify each with tap and linkerd viz authz before tightening the default.

Linkerd vs Istio: choosing on purpose

Dimension	Linkerd	Istio
Data plane	Bespoke Rust micro-proxy (~10-20Mi)	Envoy (~50-100Mi), or ztunnel + waypoints in ambient mode
Philosophy	Minimal, opinionated, secure-by-default	Feature-rich, highly configurable
mTLS	On automatically when meshed	Opt-in via `PeerAuthentication` (STRICT/PERMISSIVE)
Extensibility	Intentionally none	Lua, WASM, `ext_authz`, EnvoyFilter
Sidecarless option	No (sidecar model)	Yes — ambient mesh
Learning/ops cost	Low	Higher, broad surface area
Governance	CNCF graduated; stable builds via Buoyant	CNCF graduated

Choose Linkerd when you want mTLS, reliability, and golden metrics with the smallest possible operational and resource footprint, and you do not need Envoy’s extensibility. Choose Istio when you need rich traffic management (fault injection, mirroring, complex routing), WASM/ext_authz extensibility, VM/multi-mesh topologies, or the sidecarless ambient model. One licensing nuance to factor into a production decision: Buoyant now publishes the stable, numbered Linkerd releases under terms that ask larger production users to buy Buoyant Enterprise for Linkerd, while the open-source edge channel (and self-built stable releases) remain freely available — so “which release channel, under what support model” is a real planning question, not just a technical one.

Verify

Run this top to bottom after any mesh change; treat a non-green check as a release blocker.

# Control plane, data plane, and extension health
linkerd check
linkerd check --proxy
linkerd viz check

# mTLS is live for app-to-app traffic (SECURED = √)
linkerd viz -n emojivoto edges deployment
linkerd viz -n emojivoto tap deploy/web | grep -m1 'tls=true'

# Retry/timeout policy resolves for a route
linkerd viz -n emojivoto stat deploy/web --from deploy/vote-bot

# Multicluster link is alive and gateways reachable
linkerd --context=west multicluster check
linkerd --context=west multicluster gateways

# Failover split is being reconciled
kubectl -n test get trafficsplit podinfo -o jsonpath='{.spec.backends}'

Wire linkerd check --proxy into CI as a gate so a cert nearing expiry or a half-meshed deployment fails the pipeline instead of paging you.

Common beginner mistakes

Meshing kube-system (or the control plane’s own namespace). The instinct is “mesh everything.” Do not mesh kube-system, linkerd, or linkerd-viz, and be careful with the API server, DNS, and CNI pods. These components can start before the proxy-injector or the identity service is ready, so injecting them risks a chicken-and-egg deadlock where the thing that issues identities cannot start because it is waiting for an identity. The right mental model: mesh your application namespaces; leave cluster infrastructure unmeshed unless a doc explicitly says otherwise.

Making a non-idempotent route retryable. Setting isRetryable: true on a POST /charge or POST /vote feels like more reliability; it is a correctness bug. A retried write can double-bill or double-count, because the mesh cannot know your write is unsafe to repeat — you assert that. Right model: retries are only ever safe on idempotent operations (most GETs, and writes you have deliberately made idempotent with an idempotency key). Default to isRetryable: false and opt in per route.

Expecting Istio features. Reaching for a WASM filter, an ext_authz callout, request mirroring, or fault injection and being surprised Linkerd “can’t do it” is not a Linkerd bug — it is Linkerd doing its job. The whole value proposition is a small, non-extensible proxy. If your design genuinely needs those features, that is a signal to choose Istio, not to fight Linkerd. Right model: pick the mesh from your requirements before you build on it.

Treating the trust anchor as install-and-forget. Because mTLS “just works,” it is easy to forget there are certificates with expiry dates underneath. A lapsed issuer stops new pods from getting an identity; a lapsed anchor breaks the whole mesh. Right model: the issuer is a routinely-rotated online secret and the anchor is a long-lived offline root, and linkerd check --proxy (on a schedule, alerting weeks ahead) is a required operational control, not an optional one.

Assuming multicluster “just routes” without network planning. Labeling a service exported=true and expecting cross-cluster calls to flow ignores the network layer. In gateway mode the remote linkerd-gateway must be reachable (LoadBalancer/IP, firewall rules) from the source cluster, and every cross-cluster call pays a gateway hop; in flat mode the pod networks must actually be routable to each other. Right model: multicluster is a networking project with a mesh on top — decide gateway vs flat mode from your real connectivity, and verify with linkerd multicluster gateways showing ALIVE True before you rely on it.

Practice challenges

Work these in order; each builds on the last. Solutions are collapsed — try first, then check. If you do not have two clusters, k3d/kind (two clusters) or a single cluster (challenges 1–4) is enough.

1. (Beginner) Mesh a namespace and prove mTLS. Deploy emojivoto, mesh it, and produce evidence that web → emoji traffic is actually encrypted — not just assume it.