Tail-Based Sampling at Scale with the OpenTelemetry Collector and Load-Balancing Exporter

Probabilistic head sampling decides on the root span, before the trace exists. Keep 1%, and you keep 1% of the errors, 1% of the slow requests, and 1% of everything boring — blind, uniform, and cheap. Tail sampling inverts that bargain: it buffers every span of a trace until the trace is complete, then decides with full knowledge — keep every error and every p99-latency outlier, drop the fast and healthy bulk. The catch is the word complete. Tail sampling only works if every span of a given trace lands on the same Collector instance, and that single constraint forces the entire architecture in this article. Get the routing wrong and you do not get bad sampling — you get inconsistent sampling, where half a trace is kept and half is dropped, which is strictly worse than no sampling at all, because a torn trace in your backend looks like a bug in your instrumentation, not a sampling artifact.

This is a build-and-operate playbook for tail sampling with the OpenTelemetry Collector at real volume — tens of thousands of traces per second. You will learn why tail sampling is stateful and head sampling is not, why that statefulness forces a two-tier topology (a stateless gateway tier that routes and a stateful sampler tier that decides), how the load-balancing exporter hashes trace IDs onto a consistent ring so every span of a trace converges on one sampler, and how the tail_sampling processor evaluates an ordered list of policies — status_code, latency, probabilistic, rate_limiting, string_attribute, numeric_attribute, span_count, boolean_attribute, ottl_condition, and the boolean combinators and, not, drop, and composite. Every one is enumerated with its exact config fields, its default, when to reach for it, and the gotcha that bites in production.

By the end you will size decision_wait and num_traces against your real trace duration and arrival rate instead of guessing, set GOMEMLIMIT so the sampler tier degrades under pressure instead of getting OOM-killed, keep the routing ring stable across restarts and scale events, and scrape the processor’s own metrics so tail sampling stops being a black box. You will also model the backend cost honestly — because the counter-intuitive truth is that policy-based tail sampling often ingests more than aggressive head sampling. What it buys is not storage savings; it is decision quality: a ~100% error-catch rate and full p99 visibility, with the spend concentrated on the traces that actually carry signal.

What problem this solves

Distributed tracing is only useful if the trace you need survived sampling. The tension is brutal at scale: instrument everything and you generate millions of spans per second, more than any backend can affordably store; sample uniformly at the head and the trace of the one failing checkout during your Black Friday incident was almost certainly thrown away at 1% odds. Head sampling is cheap and consistent, but it is blind — the decision is made at the root, milliseconds into a request, long before anyone knows whether it will error, stall, or touch the payments service. You cannot ask “keep all the errors” at the head, because at the head there are no errors yet.

Tail sampling answers exactly that ask. It waits for the trace to finish, then runs policies with full knowledge — latency across all spans, error status anywhere in the tree, attributes on any service, span counts, custom OTTL conditions — and keeps precisely the traces worth keeping. But it pays for that knowledge with state: every span must be buffered in memory, grouped by trace ID, until the decision is made. And buffering by trace ID only works if all of a trace’s spans are in the same buffer — the same process. A single trace fans out across dozens of microservices whose spans arrive at your Collector fleet through different agents at different times. If three Collector replicas each see a third of one trace, each runs its policies on a fragment and emits an independent verdict, and you get split decisions: the error span kept, the root dropped, the trace torn in half in your backend.

What breaks without the architecture in this article: teams stand up a pool of otelcol-contrib replicas behind a normal Kubernetes ClusterIP Service, each running tail_sampling with “keep 100% of errors,” and are baffled when error traces show up in Tempo missing half their spans. The processor is configured correctly; the routing is wrong. The 100%-of-errors policy fires on whichever fragment happened to contain the error span, and the other fragments — evaluated by other replicas that saw only healthy spans — get dropped by the baseline probabilistic policy. This is not “slightly degraded” sampling; it is silently broken sampling, and the only proof it works or not is the per-policy kept-ratio metric, never the config looking correct. Who hits this: anyone running tail sampling on more than one Collector replica, which is everyone past a trivial volume.

Who hits which failure, and who usually owns the fix, so you route the incident to the right person fast:

To frame the whole field before the deep dive, here is the shape of the problem and the shape of the answer:

Learning objectives

By the end of this article you can:

• Explain precisely why head sampling is stateless and tail sampling is stateful, and why that statefulness forces all spans of a trace onto one Collector instance.
• Design the canonical two-tier Collector topology — a stateless gateway tier running the load-balancing exporter and a stateful sampler tier running the tail_sampling processor — and articulate which job each tier owns.
• Configure the load-balancing exporter with routing_key: traceID, pick the right resolver (static, dns, k8s, aws_cloud_map), and understand why the sampler tier must be a headless Service.
• Enumerate and apply every tail_sampling policy type — status_code, latency, probabilistic, rate_limiting, numeric_attribute, string_attribute, boolean_attribute, trace_state, span_count, bytes_limiting, ottl_condition, always_sample, and the combinators and, not, drop, composite — with correct field names and defaults.
• Size decision_wait, num_traces, expected_new_traces_per_sec, and per-pod memory against your real trace duration and arrival rate, and set GOMEMLIMIT and memory_limiter so the sampler degrades gracefully instead of OOM-crashing.
• Handle late-arriving spans, incomplete traces, and ring instability during pod restarts and scale events with decision_cache, terminationGracePeriodSeconds, and the k8s resolver.
• Validate correctness with the processor’s own metrics — count_traces_sampled, sampling_trace_dropped_too_early, sampling_late_span_age, global_count_traces_sampled — and PromQL that proves policies behave and the ring is balanced.
• Model the backend cost of policy-based tail sampling against 1%/5% head sampling and decide, with numbers, whether the two-tier complexity earns its keep.

Prerequisites & where this fits

You should already understand distributed tracing fundamentals: a trace is a tree of spans sharing a trace ID, propagated across service boundaries via the W3C Trace Context traceparent header, which carries the trace ID, the parent span ID, and a sampled flag. You should know what the OpenTelemetry Collector is — a vendor-neutral pipeline of receivers, processors, and exporters that ingests, transforms, and ships telemetry — and the difference between the core otelcol binary and the otelcol-contrib distribution (the load-balancing exporter and tail_sampling processor live in contrib, not core). Comfort with Kubernetes Services, StatefulSets, headless Services, and kubectl, plus reading Prometheus metrics and basic PromQL, is assumed. All commands are Linux/kubectl.

This sits at the deep end of the Observability track, specifically the tracing-and-collector pipeline. It is the scaling sequel to Building Production OpenTelemetry Collector Pipelines: Receivers, Processors, and Tail Sampling — that article builds a single-tier pipeline and introduces tail sampling; this one scales it correctly across many replicas. It pairs tightly with Distributed Tracing End-to-End: Context Propagation, Tempo, and Correlating Traces with Metrics and Logs, because the traces you sample here land in Tempo, and with Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation, because exemplars link a metric spike to a kept trace — and a trace that tail sampling dropped is a broken exemplar link. If your traces originate from Java services, OpenTelemetry for Java Services: Auto-Instrumentation, Context Propagation, and Custom Spans is where the spans are born.

Where this fits in the bigger observability picture — the layers upstream and downstream of the tail-sampling tier:

Core concepts

Six mental models make every later decision obvious.

The sampled flag is a head decision; tail sampling should not depend on it. The W3C traceparent carries a sampled bit set at the root. Head sampling is that bit: honour it and you never generate spans you won’t keep. Tail sampling operates after export, on spans already created, so it saves no SDK-side work. For it to see every trace, upstream SDKs and agents must sample at 100% (a parentbased_always_on sampler) and defer the real decision to the Collector. If your SDK head-samples at 10% and you tail-sample, the tail tier sees only 10% of traces and can only keep errors that survived the coin flip. Tail sampling requires the firehose reach it.

Tail sampling is stateful; that is the entire architectural driver. The tail_sampling processor holds spans in memory, keyed by trace ID, in a bounded map of size num_traces. When a trace has been quiet for decision_wait, the processor runs the policy list against the assembled trace and emits one keep/drop verdict for all its spans. State means two things: memory cost (proportional to in-flight traces) and locality (all spans of a trace must land in the same map, i.e. the same process). Head sampling has neither property — it is a stateless function of the root’s random number.

All spans of a trace must converge on one sampler — that is what the load balancer guarantees. The load-balancing exporter maintains a consistent-hash ring over the sampler backends and routes each span by hashing its routing_key. Set routing_key: traceID and every span of a given trace hashes to the same ring position and therefore the same sampler pod — whichever gateway received it. This is the mechanism that turns N gateways receiving arbitrary spans into a sampler tier where each pod owns a deterministic slice of the trace-ID space and sees complete traces for that slice.

The two tiers have opposite scaling properties, so they must be separate deployments. The gateway tier is stateless: any replica can receive any span, you scale it on ingest CPU/throughput, and adding replicas is free of correctness risk. The sampler tier is stateful: each replica owns a slice of trace IDs, you scale it on trace arrival rate and memory, and adding replicas re-hashes the ring (moving some trace slices between pods). Putting tail_sampling on the gateways means you cannot add a gateway without splitting traces — which is exactly the failure this architecture exists to prevent.

decision_wait and num_traces are coupled, and undersizing them fails silently. decision_wait is how long a trace sits in the buffer before a decision. num_traces is the hard cap on how many traces the buffer holds. In-flight traces ≈ arrival_rate × decision_wait. If num_traces is smaller than that, the processor evicts the oldest traces early and decides them on whatever partial spans arrived — which is indistinguishable, in your backend, from “the policy dropped them.” The only signal is the sampling_trace_dropped_too_early metric. There is no error, no log spam, just quietly wrong sampling.

Policy evaluation is OR-with-order, with combinators for AND/NOT/drop. The processor evaluates policies top to bottom. By default, if any policy returns Sampled, the trace is kept (a logical OR across policies). and combines sub-policies into a conjunction; not inverts; drop forces a discard when its sub-policies match (overriding a keep); composite allocates a total span budget across weighted sub-policies. Order matters because rate_limiting and probabilistic consume budget as they evaluate, and sample_on_first_match (if enabled) short-circuits on the first Sampled verdict.

The vocabulary in one table

Pin down every moving part before the deep sections. The glossary at the end repeats these for lookup; this table is the mental model side by side:

Head vs tail sampling: why tail needs full-trace assembly

Head sampling decides at trace start. The decision propagates in the W3C traceparent sampled flag, so every downstream service honours it consistently, and the cost is near zero — a service that sees sampled=0 never records or exports its spans at all. The limitation is informational: the root has no idea whether the request will fail, stall three hops downstream, or touch a service you care about. It decides on a random number and nothing else. probabilistic at the SDK is head sampling; so is parentbased_traceidratio, which makes a ratio decision at the root and has children inherit it.

Tail sampling decides at trace end. The tail_sampling processor holds spans in memory, grouped by trace ID, waits decision_wait for the trace to go quiet, then runs your policy list against the assembled trace and emits a single keep/drop verdict for all spans at once. Because it sees the whole trace, it can express decisions head sampling cannot: keep if any span errored, keep if end-to-end duration ≥ 750 ms, keep if this trace touched the payments service, keep if the trace has more than 200 spans (a fan-out storm worth investigating).

The two are not mutually exclusive and are often layered. A common pattern is head-sample at 100% (so the tail tier sees everything), then tail-sample to a few percent — the SDK does no dropping, the Collector does all of it with full knowledge. Another is head-sample at a modest ratio to cap SDK/network cost, accepting that the tail tier only chooses among survivors. The decisive trade-off is where the intelligence lives versus where the cost is paid.

The concrete decisions tail sampling can express that head sampling structurally cannot — because the required information does not exist at the root — are the whole reason to pay for it:

Head versus tail across every dimension that matters:

That “requires whole trace co-located” row is the whole problem. A single trace fans across many services, and their spans arrive at your Collector fleet through different agents at different times. If three sampler replicas each see a third of one trace, each runs its policies on a fragment and you get split decisions. Tail sampling demands that all spans sharing a trace ID converge on exactly one decision-making instance. Scaling the sampler tier behind a normal round-robin Service is precisely the failure mode: round-robin scatters a trace’s spans across replicas. The fix is consistent, trace-ID-based routing — which is what the load-balancing exporter provides, and the reason it exists.

Why not just tail-sample on a single giant Collector and avoid the routing problem? Because a single instance is a single point of failure and a vertical dead-end: the memory to buffer the entire firehose lands on one pod, and one OOM takes down all sampling. The two-tier design spreads that buffer across many pods while keeping each trace whole.

The two-tier topology

The architecture is two distinct Collector tiers with different jobs. The data path is: app pods → agent Collectors (optional DaemonSet) → gateway tier (N stateless replicas, load-balancing exporter, routes by trace ID) → sampler tier (each replica owns a slice of trace IDs, runs tail_sampling) → backend.

Tier 1 — gateways. Receive OTLP from agents/SDKs, enrich (add k8s.* resource attributes, drop noisy attributes), batch, and route. The gateway holds no sampling state; any replica can receive any span, so you scale it freely on ingest throughput behind a normal load-balanced Service. Its one special job is the load-balancing exporter, which hashes each span’s trace ID onto a fixed ring of sampler backends. Because the hash is deterministic, every span of a given trace — whichever gateway received it — lands on the same sampler.

Tier 2 — samplers. Each replica runs tail_sampling, owns a slice of the trace-ID space (its arc of the hash ring), and therefore sees complete traces for that slice, so its decisions are sound. This is the only tier that exports to your tracing backend (Tempo, Jaeger, a SaaS). It is stateful and memory-heavy; you scale it on trace arrival rate.

The agent tier (a DaemonSet Collector per node) is optional but common: it lets SDKs export to localhost (no cross-node hop on the app’s critical path), batches per node, and forwards to the gateways. Crucially, the agent must not tail-sample and must not load-balance by trace ID — it just forwards. Only the gateway routes and only the sampler decides.

The division of labour, tier by tier:

Why not put tail_sampling on the gateways directly and skip the sampler tier? Because you cannot scale them. Add a second gateway replica behind a normal Service and traces split across replicas, breaking tail sampling exactly as described above. The load-balancing exporter exists precisely to decouple “how many instances receive telemetry” from “which single instance decides a given trace.” That decoupling is the whole point of two tiers.

Deployment vs StatefulSet for the sampler tier

The sampler tier is stateful in the sense that each replica holds trace buffers, but that state is ephemeral (in-memory, rebuilt from the incoming stream) — you are not persisting anything to disk. So a StatefulSet is not strictly required. What the sampler tier does need is a stable set of endpoints the load balancer can resolve into a stable ring. Both a Deployment and a StatefulSet can front a headless Service; the difference is stable network identity:

In practice a Deployment with a PodDisruptionBudget and the k8s resolver is sufficient for most teams; a StatefulSet’s ordered, one-pod-at-a-time rollout narrows the window during which the ring is churning, which marginally reduces split-trace incidents during upgrades. Either way, the fronting Service must be headless.

Configuring the load-balancing exporter to route by trace ID

The load-balancing exporter is part of the contrib distribution (otelcol-contrib), not the core binary. Its job is to pick a backend deterministically from a routing key so that related telemetry co-locates. For tail sampling the key must be traceID — this is what guarantees span co-location.

# tier 1: gateway collector
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes: {}
  batch:
    timeout: 1s
    send_batch_size: 8192

exporters:
  loadbalancing:
    routing_key: traceID            # CRITICAL: route by trace ID, not service
    protocol:
      otlp:
        timeout: 5s
        tls:
          insecure: true            # in-cluster; use real TLS across trust boundaries
        sending_queue:
          enabled: true
          queue_size: 10000
        retry_on_failure:
          enabled: true
    resolver:
      # The k8s resolver watches the Endpoints/EndpointSlice API directly, so ring
      # updates on scale events are near-instant. Use the `dns` resolver elsewhere
      # (hostname = the headless Service FQDN), but it lags by the resolution interval.
      k8s:
        service: otel-sampler-headless.observability
        ports:
          - 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [loadbalancing]
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888

Two details decide whether this works in production, and both are easy to get subtly wrong.

routing_key: traceID. The default is traceID for trace pipelines — but set it explicitly, because the other valid value, service, routes by service.name and scatters one trace across many samplers (every service in the trace hashes to a different backend), defeating the entire design. Use service only for per-service span-metrics aggregation where you want each service’s spans on one backend; never for tail sampling. Getting this field wrong produces exactly the split-trace symptom, and the config still looks plausible.

The resolver and a headless Service. The exporter maintains a consistent-hash ring over the resolved backend addresses, so the sampler tier must be a headless Service (clusterIP: None) whose pods expose 4317 — that way each pod IP is a distinct ring member. The headless part is non-negotiable: a normal ClusterIP Service resolves to a single VIP, the ring gets exactly one member, and load balancing collapses to “send everything to one pod.” That one pod then buffers the entire firehose, OOMs, and takes down all sampling — the worst possible outcome, produced by a one-word Service misconfiguration.

Routing keys — every value and when it applies

The routing_key determines what the ring hashes, and its valid values depend on the signal type. For tail sampling you want traceID; the others exist for different co-location needs:

The default when unset is traceID for traces and service for logs and metrics. A subtle trap: logs without a trace ID under routing_key: traceID are distributed randomly (there is nothing to hash), which is fine for logs but a reminder that the key only co-locates telemetry that actually carries it.

Resolvers — static, dns, k8s, aws_cloud_map

The resolver is how the exporter discovers the current set of sampler backends and keeps the ring current. Four resolver types, each suited to a different environment:

The k8s resolver is the right default in Kubernetes because it watches the API rather than polling DNS, so when the sampler tier scales up or a pod restarts, the ring updates almost immediately — minimising the window during which a trace could split across an about-to-change ring. The dns resolver works in Kubernetes too (point hostname at the headless Service FQDN, e.g. otel-sampler-headless.observability.svc.cluster.local) but it re-resolves only every interval and is further gated by the DNS TTL, so ring updates lag and split-trace windows are wider. Prefer k8s when the Collector has API access; fall back to dns when it does not (e.g. RBAC constraints, or Collectors outside the cluster).

Which resolver to pick, by environment and constraint:

The headless Service the k8s resolver watches:

apiVersion: v1
kind: Service
metadata:
  name: otel-sampler-headless
  namespace: observability
spec:
  clusterIP: None            # HEADLESS — each pod IP is a distinct ring member
  selector:
    app: otel-sampler
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317

If you omit clusterIP: None, the resolver sees one VIP, the ring has one member, and — as above — all traffic funnels to whichever pod the VIP happens to pick. This is the single most common load-balancing-exporter misconfiguration, and its symptom (one sampler pod at 100% memory, the rest idle) is unmistakable once you know to look for it.

The protocol.otlp block accepts every setting the standard OTLP exporter does except endpoint (the load balancer overrides that per backend from the ring). The settings worth tuning for a tail-sampling gateway:

How the consistent-hash ring behaves under change

The exporter does not round-robin; it maps each trace ID to a fixed position on a consistent-hash ring and sends it to the backend that owns that arc. The property that matters: when the backend list changes (a pod added, removed, or restarted), only about R/N of the routes are re-mapped (R = total routes, N = backend count) rather than reshuffling everything. Adding a fourth sampler to a three-pod tier moves roughly a quarter of trace IDs to the new pod; the other three-quarters keep their existing owner. This is why scale events are survivable — most in-flight traces keep landing on the same sampler — but not free: the ~R/N of traces whose owner changed mid-flight can split, because their early spans went to the old owner and later spans to the new one. Section “Handling late-arriving spans” covers how to minimise that window.

Tail-sampling processor policies — every type enumerated

On the sampler tier, tail_sampling evaluates an ordered list of policies against each assembled trace. The default logic is OR-with-precedence: if any policy returns Sampled, the trace is kept. Combinators change this — and requires all its sub-policies, not inverts, drop forces a discard, and composite allocates a shared budget. Order matters because rate_limiting and probabilistic consume budget in evaluation order, and sample_on_first_match short-circuits.

# tier 2: sampler collector
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  tail_sampling:
    decision_wait: 10s              # how long to wait for a trace to complete
    num_traces: 200000              # max traces held in memory at once
    expected_new_traces_per_sec: 5000
    decision_cache:
      sampled_cache_size: 500000    # remember 'kept' decisions for late spans
      non_sampled_cache_size: 500000
    policies:
      # 1. Always keep anything that errored.
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # 2. Always keep slow traces (>= 750ms end to end).
      - name: slow
        type: latency
        latency:
          threshold_ms: 750

      # 3. Keep traces touching the payments service, but cap the volume.
      - name: payments-sampled
        type: and
        and:
          and_sub_policy:
            - name: is-payments
              type: string_attribute
              string_attribute:
                key: service.name
                values: [payments-api]
            - name: cap
              type: rate_limiting
              rate_limiting:
                spans_per_second: 500

      # 4. Keep fan-out storms (traces with an unusually large span count).
      - name: big-traces
        type: span_count
        span_count:
          min_spans: 200

      # 5. Everything else: keep a representative 2%.
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 2

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp/tempo]
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888

The complete policy-type reference

Every policy type the processor supports, what makes it fire, and its exact config fields:

Choosing between the combinators

The four combinators — and, not, drop, composite — are where most policy-design mistakes happen, because their semantics interact with the top-level OR. Here is how to reason about each:

Use and for a boolean conjunction (the “is-payments AND within rate limit” example above). Use not to express exclusions. Use drop when a class of traffic must never be kept even though another policy (say, latency) would otherwise keep it — a slow /healthz probe is still noise; a drop on http.target == /healthz wins over the latency keep. Use composite when a single total budget (e.g. 2,000 spans/sec) must be divided across categories with explicit percentages and ordering — the right tool when a global cap must be shared fairly, not for a simple conjunction. A frequent mistake is reaching for composite to express “A and B”; that is what and is for. composite is about budget allocation, not boolean logic.

ottl_condition — the escape hatch

When no built-in policy expresses your rule, ottl_condition runs an OpenTelemetry Transformation Language expression against each span (and optionally span events), keeping the trace if any evaluates true. It subsumes several simpler policies and lets you combine fields the built-ins cannot:

- name: slow-payments-5xx
  type: ottl_condition
  ottl_condition:
    error_mode: ignore     # ignore | propagate | silent — how to handle eval errors
    span:
      - 'attributes["service.name"] == "payments-api" and attributes["http.status_code"] >= 500'
      - 'duration_nanos > 1000000000 and attributes["http.route"] == "/checkout"'

error_mode: ignore treats an evaluation error (e.g. a missing attribute) as “condition false” and moves on — the safe default; propagate surfaces the error and silent suppresses it. OTTL is powerful but costs more CPU per span than a native policy, so prefer a built-in when one fits and reserve OTTL for genuinely compound conditions.

Policy ordering, sample_on_first_match, and the top-level fields

Two top-level flags change evaluation semantics. sample_on_first_match: true stops evaluating as soon as a policy returns Sampled — a performance win on hot paths and a way to make ordering meaningful for budgeted policies, but it changes results (a later drop never runs if an earlier keep short-circuits). decision_wait_after_root_received lets the processor start (or extend) its wait from when the root span arrives, useful when roots arrive late. The complete top-level field reference, with real defaults:

The default decision_wait: 30s and num_traces: 50000 are frequently wrong for real workloads and are the source of most first-deployment pain: 30s is often longer than needed (wasting memory) while 50,000 is often far too small for a high-arrival-rate replica (causing silent early eviction). Always compute both against your measured numbers rather than accepting defaults — the next section shows how.

Sizing decision_wait, num_traces, and memory

These coupled knobs are where tail sampling lives or dies, and the dangerous failure is silent: undersize them and traces get evicted before their decision, which looks identical to “the policy dropped them” in your backend. There is no error and no obvious log line — only the sampling_trace_dropped_too_early metric.

decision_wait is how long the processor holds a trace before deciding. It must exceed your real end-to-end trace duration at a high percentile, plus transit time from the gateways. If your p99 trace is 4 seconds, a 2-second window decides on incomplete traces and your latency policy never fires on the slow ones — the exact traces you care about most. Start at 10 seconds (well below the 30s default, which usually wastes memory), then tune against the sampling_late_span_age metric: if stragglers routinely arrive after the window, raise it; if they never do, you can lower it and reclaim memory. Do not set it to 60s “to be safe” — decision_wait directly multiplies memory, because a trace occupies the buffer for that entire window.

num_traces is the hard cap on traces resident in memory. It must comfortably exceed decision_wait × arrival_rate (per replica). With 5,000 new traces/sec per replica and a 10-second window you have ~50,000 in flight; num_traces: 200000 gives 4× burst headroom. Exceed it and the processor evicts the oldest traces early, deciding them on whatever spans arrived so far — usually incomplete. expected_new_traces_per_sec pre-allocates the internal map to avoid rehashing churn; set it to the steady-state arrival rate per replica (total rate ÷ replica count, since the load balancer spreads traces evenly across the ring).

Memory. Budget per-trace memory as (avg spans/trace × avg span size). At ~20 spans of ~1.5 KB, a trace is ~30 KB; 200,000 of them is ~6 GB of buffered spans before Go runtime overhead (maps, pointers, GC headroom add 30–50%). Always pair the sampler with memory_limiter ahead of the export, give the pod a generous limit, and set GOMEMLIMIT so the runtime GCs aggressively under pressure instead of letting the container OOM-kill the process:

# sampler pod spec
spec:
  containers:
    - name: otel-sampler
      image: otel/opentelemetry-collector-contrib:latest
      resources:
        requests:
          memory: "6Gi"
          cpu: "2"
        limits:
          memory: "8Gi"        # generous headroom above the buffer estimate
      env:
        - name: GOMEMLIMIT
          value: "7GiB"        # ~90% of the container memory limit

The sizing relationships, made concrete:

How the sizing knobs trade off against each other:

Rule of thumb: scale the sampler tier on trace arrival rate, not CPU. Each replica’s memory budget is num_traces × avg_trace_bytes. To handle more traffic, add replicas — the load balancer re-hashes the ring and each replica owns a smaller slice, so per-replica num_traces (and memory) can stay fixed — rather than enlarging num_traces on a few fat pods, which only raises your blast radius when one OOMs. Ten 6 GB pods losing one is a 10% capacity dip; two 30 GB pods losing one is a 50% dip and a huge ring re-hash.

Spans-per-trace limits and pathological traces

A single runaway trace — a retry storm, a fan-out that spawns thousands of spans, an instrumentation bug emitting a span per loop iteration — can consume a disproportionate share of the buffer and skew memory wildly. Two guards help: maximum_trace_size_bytes caps the buffered bytes of any single trace (spans beyond the cap are dropped for that trace), and a span_count policy with a sane max_spans can route such traces (keep them for investigation, or drop them if they are known noise). Without a guard, one pathological trace can push a replica toward its memory limit on its own. Budget for a realistic tail of span counts, not just the average — a p99 of 200 spans/trace against an average of 20 means your worst traces are 10× the mean, and they arrive in bursts during incidents (exactly when memory is already tight).

Handling late-arriving spans and incomplete traces

Distributed traces do not arrive atomically — a slow async worker may emit its span seconds after the root finished, and network jitter reorders arrivals. Four behaviours protect you.

decision_wait absorbs normal lateness. Any span arriving within the window joins its trace before the decision, which is why the window must track real trace duration, not a guess. This is the first line of defence: size the window so the overwhelming majority of spans arrive before the decision fires.

The decision_cache keeps post-decision stragglers consistent. Once a trace is decided and flushed, later spans for the same trace ID would — without a cache — start a new buffered trace and be re-decided independently, risking a different verdict (a straggler of a kept trace getting dropped by the baseline policy, tearing the trace). Setting decision_cache.sampled_cache_size and non_sampled_cache_size above zero makes the processor remember recent verdicts: a late span of a kept trace is exported (matching the trace) and a late span of a dropped trace is dropped (matching the trace). This is what keeps a trace whole across the decision boundary, and it is off by default (0) — a common omission. Size the caches to comfortably cover late_span_age × arrival_rate.

The two decision caches, what each protects, and how to size them:

Routing stability prevents the worst case — the ring changing mid-trace. If a sampler pod restarts or the tier scales while a trace is in flight, some of that trace’s spans hash to the old ring owner and some to the new, splitting the decision even with a perfect decision_wait. Mitigate by (a) using the k8s resolver for near-instant, watch-based endpoint updates (minimising the re-hash window), (b) a decision_wait comfortably longer than a normal pod-startup blip, © a terminationGracePeriodSeconds long enough to drain in-flight traces on rollout, and (d) a PodDisruptionBudget so voluntary disruptions never take too many samplers at once:

# sampler workload — drain gracefully and limit disruption
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 40   # >= decision_wait, let in-flight traces decide
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: otel-sampler-pdb
  namespace: observability
spec:
  minAvailable: 80%           # keep most of the ring stable during voluntary disruptions
  selector:
    matchLabels:
      app: otel-sampler

Accept a small, measured inconsistency during scaling — do not chase perfection. No tail-sampling setup eliminates split traces entirely during a ring change; the honest goal is to minimise the window and measure the residue. During a rolling upgrade, expect a brief uptick in split traces proportional to the ~R/N of routes that moved; a PDB, ordered rollout, and k8s resolver keep it small. Chasing zero split traces during scaling is a losing game; keeping it to a fraction of a percent during rollouts, and zero at steady state, is the achievable and correct target.

The late-span and incomplete-trace defences at a glance:

Validating sampling fairness and exporting metrics

Scrape the Collector’s own internal metrics and the tail processor stops being a black box. This is the only way to know tail sampling is working — the config looking correct proves nothing, because the split-trace failure mode produces a plausible-looking config and silently wrong results.

# in either tier's config: expose Collector self-telemetry
service:
  telemetry:
    metrics:
      level: detailed          # 'detailed' exposes per-policy tail-sampling series
      address: 0.0.0.0:8888

Scrape :8888/metrics with Prometheus (via a ServiceMonitor/PodMonitor or a scrape config). The decisive series is otelcol_processor_tail_sampling_count_traces_sampled, broken down by policy and a sampled (true/false) dimension. The processor’s key self-telemetry:

Kept ratio per policy, and the silent-undersizing alarm, in PromQL:

# Fraction of traces each policy keeps (should match your intent per policy)
sum by (policy) (
  rate(otelcol_processor_tail_sampling_count_traces_sampled{sampled="true"}[5m])
)
/ ignoring(sampled) group_left
sum by (policy) (
  rate(otelcol_processor_tail_sampling_count_traces_sampled[5m])
)

# ALERT: traces evicted before a decision was made — buffer is undersized.
# This is the silent-failure signal; it should be flat at zero.
rate(otelcol_processor_tail_sampling_sampling_trace_dropped_too_early[5m]) > 0

# Ring balance: new-trace arrival should be roughly equal across sampler pods.
# A lopsided distribution usually means the Service isn't headless.
sum by (pod) (
  rate(otelcol_receiver_accepted_spans{job="otel-sampler"}[5m])
)

The first query proves your policies behave: errors and slow should keep ~100% of the traces they match, baseline ~2%. A policy keeping wildly more or less than intended is a config bug you would otherwise never catch. Fairness across the ring is the second essential check — accepted-span (or new_trace_id_received) rate should be roughly equal across sampler replicas; a lopsided distribution means the ring is unbalanced, almost always because the resolver only sees one backend (re-check that the Service is headless and the resolver is watching the right endpoints).

The three alerts worth wiring before your next incident — the ones that catch silent failures rather than lagging “traces are missing” reports:

What each self-telemetry signal is the source of truth for, mapped to the action it drives:

Cost impact versus probabilistic head sampling

The reason teams adopt this complexity is economics, and the comparison is counter-intuitive. Take 50,000 traces/sec, ~20 spans each = 1,000,000 spans/sec, a true error rate of 0.5%, and a backend billed per ingested span.

• Head sampling at 1%: ingest 10,000 spans/sec, keep 1% of errors — you lose 99% of every failure. Cheap, but during an incident the trace you actually need is almost certainly gone.
• Head sampling at 5%: ingest 50,000 spans/sec, keep 5% of errors — still four out of five failures lost, at 5× the 1% cost.
• Tail sampling (all errors + all ≥750 ms + 2% baseline): errors ~5,000 spans/sec (100% of the 0.5%), slow outliers a few percent more, baseline ~20,000 spans/sec. Net ingestion lands near 3–5% — call it ~40,000 spans/sec.

The honest cost comparison:

Tail sampling ingests more than aggressive head sampling in this scenario, so it is not free at the backend. The honest framing: it buys decision quality, not raw storage savings. You pay more in Collector compute (the sampler tier is memory-heavy and CPU-real) and possibly more at the backend than 1% head sampling. What you get is a ~100% error-catch rate and full p99 visibility for roughly the cost of blind 4–5% head sampling — but with the spend concentrated on the traces that carry signal, instead of scattered uniformly across boring, healthy requests.

Where tail sampling does win on cost is when baseline and error/slow rates are all low: keeping 100% of a 0.5% error rate plus a 1% baseline is far cheaper than the 10–20% head rate you would need for a fighting chance of capturing the errors that matter during an incident. The decision is not “tail is cheaper” — it is “tail spends the same budget on traces worth keeping.” Quantify it first: the sampler tier’s memory footprint is a standing cost, and there is no point in two tiers if 1% head sampling already meets your debugging SLOs.

The full cost picture — every line item the two-tier design adds or shifts:

Architecture at a glance

Picture the telemetry flowing left to right through four zones, each with a distinct responsibility, and the whole design existing to satisfy one invariant: every span of a trace must reach the same sampler.

On the far left, application pods emit spans through the OpenTelemetry SDK (or zero-code auto-instrumentation), propagating the W3C traceparent across every service hop so all spans of a request share one trace ID. Those spans go to an optional agent tier — a DaemonSet Collector on each node that receives on localhost:4317, batches, and forwards. The agent samples nothing and routes nothing special; it exists to keep the app’s export path node-local and cheap.

The spans then reach the gateway tier: N stateless Collector replicas behind an ordinary load-balanced Service. Any gateway can receive any span. Each gateway runs memory_limiter, k8sattributes (stamping k8s.pod.name, k8s.namespace.name, and friends onto every span), batch, and — the linchpin — the loadbalancing exporter with routing_key: traceID. This exporter maintains a consistent-hash ring over the sampler pods (discovered by the k8s resolver watching a headless Service) and sends each span to the sampler that owns that trace ID’s arc of the ring. Because the hash is deterministic, span A of trace T received by gateway-1 and span B of trace T received by gateway-3 both land on the same sampler.

The sampler tier is where the state lives: each replica runs the tail_sampling processor, owns a deterministic slice of the trace-ID space, and therefore assembles complete traces for its slice. It buffers each trace for decision_wait (10s), holds up to num_traces (200,000) at once, then runs the ordered policy list — keep all errors, keep everything ≥750 ms, keep payments traffic under a rate cap, keep fan-out storms, keep a 2% baseline — and emits one verdict per trace. Only this tier exports to the backend (Tempo), and only the kept traces (~3–5% of the firehose) are ingested and stored.

Two feedback loops complete the picture. First, both tiers expose self-telemetry on :8888, and Prometheus scrapes the sampler’s tail_sampling metrics — per-policy kept ratio (policies behave), trace_dropped_too_early (buffer sized), and per-pod accepted spans (ring balanced). Second, the k8s resolver watches the sampler Endpoints, so a scale event or restart updates the ring within moments. The single invariant — all spans of a trace on one sampler — is enforced by the ring; everything else (sizing, caching, graceful drain) protects that invariant across late spans, restarts, and scale.

Real-world scenario

Meridian Pay, a fictional-but-realistic payments platform, runs about 200 microservices on a 60-node Kubernetes cluster in a single region, emitting roughly 48,000 traces/sec (~19 spans/trace, ~910,000 spans/sec) at peak. Their tracing backend is self-hosted Tempo, billed internally by ingested spans and object-storage GB. The observability team is three engineers; the tracing infrastructure budget is a hard ceiling they had already blown through twice.

Their original setup was a single-tier pool of eight otelcol-contrib replicas behind a standard ClusterIP Service, each running tail_sampling configured to keep 100% of errors, everything over 800 ms, and a 3% baseline. It looked correct in code review. During a partial outage of the payment-authorization service, on-call pulled up the failing checkout flow in Tempo and found traces that ended abruptly — root span and a couple of children present, the downstream authorization spans missing. The status_code: [ERROR] policy was correct; the errored spans had simply landed on a different replica than the root, and that replica — seeing only a healthy fragment of the trace — dropped its share via the 3% baseline. Every error trace was being silently torn in half. The team had spent two on-call rotations blaming the instrumentation before someone checked the per-policy metric and noticed the errors policy was keeping traces at a suspiciously partial rate.

The root cause was the ClusterIP Service. Round-robin behind a single VIP scattered each trace’s spans uniformly across all eight replicas; no replica ever saw a complete trace, so tail sampling was operating on fragments across the board. The 100%-of-errors policy was firing on whichever fragment held the error span and dropping the rest.

The fix was the two-tier split. They left the eight replicas as stateless gateways (enrichment plus the load-balancing exporter, routing_key: traceID, k8s resolver) and moved tail_sampling onto a separate five-replica sampler Deployment behind a headless Service, sized from measured numbers: per-replica arrival ≈ 48,000 ÷ 5 ≈ 9,600 traces/sec, decision_wait: 12s (their p99 trace was ~7s plus transit), so ~115,000 in-flight per replica → num_traces: 400000 with headroom, ~35 KB/trace → ~14 GB raw → 18 GB container limit, GOMEMLIMIT: 16GiB. The one-line root cause and the one-line fix were the same field:

# gateways: stop sampling here; route every span of a trace to one sampler
exporters:
  loadbalancing:
    routing_key: traceID
    resolver:
      k8s:
        service: otel-sampler-headless.observability
        ports: [4317]

After the change, otelcol_processor_tail_sampling_count_traces_sampled{policy="errors",sampled="true"} rose to match the true error count within one scrape interval, and sampling_trace_dropped_too_early — quietly nonzero the whole time, because each undersized single-tier replica had been buffering a full one-eighth of the firehose against the default num_traces: 50000 — went to zero once num_traces was sized against per-replica arrival rate (total ÷ 5) instead of the full stream. Backend ingest actually rose ~15% (because they were now keeping all errors instead of a fragment of them), but MTTR on the next payment-auth incident dropped from ~40 minutes of “why is this trace incomplete” to ~4 minutes of reading a complete, kept trace. The runbook lesson went on the wall: tail sampling behind a non-headless Service is not “slightly worse” sampling, it is silently broken sampling, and the only proof it works is the per-policy kept-ratio metric — not the config looking correct.

The incident, as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

The two-tier tail-sampling design both solves the blind-sampling problem and introduces real operational weight. Weigh it honestly:

This design is right when you need to never miss an error or slow trace during incidents, your volume is high enough that keeping everything is unaffordable, and you can run and monitor a stateful sampler tier. It is overkill when 1% head sampling already meets your debugging SLOs, your volume is low enough to keep 100%, or you lack the metrics discipline to detect a silently broken ring. The disadvantages are all manageable — but only if you know they exist, which is the point of this article. A team that deploys the two tiers, forgets the headless Service, and never scrapes the per-policy metric has built something that looks like tail sampling and silently isn’t.

Hands-on lab

Stand up the two-tier topology locally, prove that a single-tier ClusterIP deployment splits traces while the two-tier headless design keeps them whole, and read the proof in the processor’s own metrics. This uses a kind (Kubernetes-in-Docker) cluster and telemetrygen (the OTel load generator from the contrib repo). Everything is free and local; teardown is one command.

Step 1 — Create a local cluster and namespace.

kind create cluster --name otel-tail-lab
kubectl create namespace observability
kubectl config set-context --current --namespace=observability

Expected: kind reports the cluster is ready; the namespace is created.

Step 2 — Deploy the sampler tier behind a HEADLESS Service. Save the sampler config as a ConfigMap, then a Deployment and a headless Service.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata: { name: sampler-config, namespace: observability }
data:
  config.yaml: |
    receivers:
      otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 } } }
    processors:
      tail_sampling:
        decision_wait: 5s
        num_traces: 50000
        expected_new_traces_per_sec: 200
        decision_cache: { sampled_cache_size: 100000, non_sampled_cache_size: 100000 }
        policies:
          - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
          - { name: slow, type: latency, latency: { threshold_ms: 500 } }
          - { name: baseline, type: probabilistic, probabilistic: { sampling_percentage: 5 } }
    exporters:
      debug: { verbosity: basic }
    service:
      telemetry: { metrics: { level: detailed, address: 0.0.0.0:8888 } }
      pipelines:
        traces: { receivers: [otlp], processors: [tail_sampling], exporters: [debug] }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: otel-sampler, namespace: observability }
spec:
  replicas: 3
  selector: { matchLabels: { app: otel-sampler } }
  template:
    metadata: { labels: { app: otel-sampler } }
    spec:
      containers:
        - name: otel
          image: otel/opentelemetry-collector-contrib:latest
          args: ["--config=/etc/otel/config.yaml"]
          env: [{ name: GOMEMLIMIT, value: "900MiB" }]
          resources: { limits: { memory: "1Gi", cpu: "1" } }
          ports: [{ containerPort: 4317 }, { containerPort: 8888 }]
          volumeMounts: [{ name: cfg, mountPath: /etc/otel }]
      volumes: [{ name: cfg, configMap: { name: sampler-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: otel-sampler-headless, namespace: observability }
spec:
  clusterIP: None            # HEADLESS — the whole point
  selector: { app: otel-sampler }
  ports: [{ name: grpc, port: 4317, targetPort: 4317 }]
EOF

Expected: the ConfigMap, a 3-replica Deployment, and a headless Service are created. Confirm the Service is headless:

kubectl get svc otel-sampler-headless -o jsonpath='{.spec.clusterIP}'; echo
# expected: None

Step 3 — Deploy the gateway tier with the load-balancing exporter. Note the RBAC the k8s resolver needs to watch Endpoints.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata: { name: otel-gateway, namespace: observability }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: otel-gateway-endpoints }
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: otel-gateway-endpoints }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: otel-gateway-endpoints }
subjects: [{ kind: ServiceAccount, name: otel-gateway, namespace: observability }]
---
apiVersion: v1
kind: ConfigMap
metadata: { name: gateway-config, namespace: observability }
data:
  config.yaml: |
    receivers:
      otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 } } }
    processors:
      batch: { timeout: 1s }
    exporters:
      loadbalancing:
        routing_key: traceID
        protocol: { otlp: { tls: { insecure: true } } }
        resolver:
          k8s: { service: otel-sampler-headless.observability, ports: [4317] }
    service:
      telemetry: { metrics: { level: detailed, address: 0.0.0.0:8888 } }
      pipelines:
        traces: { receivers: [otlp], processors: [batch], exporters: [loadbalancing] }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: otel-gateway, namespace: observability }
spec:
  replicas: 2
  selector: { matchLabels: { app: otel-gateway } }
  template:
    metadata: { labels: { app: otel-gateway } }
    spec:
      serviceAccountName: otel-gateway
      containers:
        - name: otel
          image: otel/opentelemetry-collector-contrib:latest
          args: ["--config=/etc/otel/config.yaml"]
          ports: [{ containerPort: 4317 }, { containerPort: 8888 }]
          volumeMounts: [{ name: cfg, mountPath: /etc/otel }]
      volumes: [{ name: cfg, configMap: { name: gateway-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: otel-gateway, namespace: observability }
spec:
  selector: { app: otel-gateway }
  ports: [{ name: grpc, port: 4317, targetPort: 4317 }]
EOF

Expected: gateway RBAC, ConfigMap, a 2-replica Deployment, and a normal (VIP) Service. Wait for both tiers to be Ready:

kubectl rollout status deploy/otel-sampler
kubectl rollout status deploy/otel-gateway

Step 4 — Generate traces through the gateway and watch them route. Run telemetrygen as a one-off pod pointing at the gateway Service.

kubectl run gen --rm -it --restart=Never --image=ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest -- \
  traces --otlp-endpoint otel-gateway.observability:4317 --otlp-insecure \
  --traces 5000 --rate 200 --child-spans 8 --status-code error

Expected: telemetrygen reports it generated 5000 traces. The --status-code error flag makes every trace error, so the errors policy should keep ~100%.

Step 5 — Read the proof in the sampler metrics. Scrape a sampler pod’s :8888 and look at the per-policy kept counts.

POD=$(kubectl get pods -l app=otel-sampler -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward "$POD" 8888:8888 >/dev/null 2>&1 &
sleep 2
curl -s localhost:8888/metrics | grep 'tail_sampling_count_traces_sampled'

Expected: series like otelcol_processor_tail_sampling_count_traces_sampled{policy="errors",sampled="true"} with a large count and sampled="false" near zero for the errors policy — proof that error traces are being kept whole. Also confirm no early drops:

curl -s localhost:8888/metrics | grep 'tail_sampling_sampling_trace_dropped_too_early'
# expected: value 0 (buffer is sized for this small load)

Step 6 — Confirm ring balance across the three samplers. Each sampler should have received a roughly equal share of accepted spans.

for p in $(kubectl get pods -l app=otel-sampler -o jsonpath='{.items[*].metadata.name}'); do
  kubectl exec "$p" -- wget -qO- localhost:8888/metrics 2>/dev/null \
    | grep '^otelcol_receiver_accepted_spans' | head -1 | awk -v pod="$p" '{print pod, $NF}'
done

Expected: three lines with comparable numbers (within ~10–20% at this small volume) — proof the headless Service gave the ring three balanced members. If one pod had all the spans and the others zero, the Service would not be headless.

Validation checklist. You stood up a stateless gateway routing by traceID and a stateful sampler tier behind a headless Service, generated error traces, and proved via the processor’s own metrics that (a) the errors policy keeps ~100%, (b) trace_dropped_too_early is zero, and © the ring is balanced across replicas. The lab steps mapped to what each proves:

Teardown.

kind delete cluster --name otel-tail-lab

Cost note. The entire lab runs in local Docker via kind — zero cloud spend. Deleting the cluster removes everything.

Common mistakes & troubleshooting

The failure modes that bite hardest, as a symptom → root cause → confirm → fix playbook. Read the table at 02:14; read the prose underneath once, in daylight.

The expanded reasoning for the four that bite hardest:

1. Error traces incomplete in the backend. The sampler tier sits behind a normal ClusterIP Service, so the load balancer round-robins each trace’s spans across replicas; no replica assembles a whole trace, and the errors policy fires only on the fragment holding the error span. Confirm with kubectl get svc <sampler> -o jsonpath='{.spec.clusterIP}' (returns a VIP, not None) and a partial count_traces_sampled{policy="errors"} kept-ratio. Fix: make the Service headless and split gateways from samplers so the exporter routes by traceID. This is the canonical failure and the reason the architecture exists.

3. Traces silently missing that a policy should keep. num_traces or decision_wait is undersized, so the buffer evicts the oldest traces before they are decided — decided on partial spans, indistinguishable from a policy drop. The unambiguous signal is sampling_trace_dropped_too_early > 0. Fix: size num_traces against per-replica arrival rate (total ÷ replicas) × decision_wait, with 3–4× headroom; the common mistake is sizing against the full stream on one replica.

7. Sampler pods OOM-killed under load. The container memory limit is below num_traces × avg_trace_bytes × (1.3–1.5 Go overhead), and without GOMEMLIMIT the Go runtime does not GC aggressively enough before the limit is hit. Pods show OOMKilled with memory pinned at the limit before each kill. Fix: set GOMEMLIMIT to ~90% of the limit, size the limit from the formula, and prefer more replicas over a bigger num_traces to shrink the blast radius.

10. Tail tier sees only a fraction of traces. An upstream SDK or agent is head-sampling (e.g. parentbased_traceidratio at 0.1) before the tail tier, so the tail only ever chooses among 10% of traces. The SDK’s sampler config is a ratio < 1 and tail volume is far below the known trace rate. Fix: head-sample at 100% (parentbased_always_on) so the firehose reaches the tail. Tail sampling cannot keep what never arrives.

Best practices

• Route by traceID, always explicitly. Set routing_key: traceID on the gateway load-balancing exporter even though it is the trace default — the alternative (service) silently scatters traces, and an explicit value is self-documenting and review-proof.
• The sampler Service must be headless. clusterIP: None, non-negotiable. A VIP collapses the ring to one member and funnels the firehose to one OOM-bound pod. Assert it in a CI check on the manifest.
• Keep tail_sampling only on the sampler tier. Never on the gateways (you can’t scale them) and never on the agents (they just forward). One decision tier, cleanly separated.
• Size decision_wait above your real p99 trace duration + transit, then tune down against sampling_late_span_age. Do not accept the 30s default blindly, and do not inflate it “to be safe” — every second is linear memory.
• Size num_traces per replica, with 3–4× headroom. arrival_rate_per_replica × decision_wait × 3–4. Set expected_new_traces_per_sec to steady-state per-replica arrival to avoid map rehash churn.
• Set GOMEMLIMIT to ~90% of the container limit and put memory_limiter in every pipeline. This makes the sampler degrade gracefully (GC + shed) instead of OOM-crashing.
• Turn on decision_cache (both caches > 0). It keeps late stragglers consistent with their trace’s verdict — off by default and a frequent omission that tears traces at the decision boundary.
• Scale the sampler tier on trace arrival rate, not CPU, and prefer more, smaller replicas over fewer, fatter ones to shrink the OOM blast radius and the ring re-hash on any single failure.
• Prefer the k8s resolver in Kubernetes. It watches Endpoints for near-instant ring updates; dns lags and widens the split-trace window. Grant the gateway SA get/list/watch on Endpoints.
• Use a PodDisruptionBudget and a terminationGracePeriodSeconds ≥ decision_wait on the sampler tier so rollouts and node drains keep most of the ring stable and let in-flight traces decide.
• Head-sample at 100% upstream. The tail tier can only choose among traces that reach it; parentbased_always_on at the SDK, defer the real decision to the Collector.
• Scrape and alert on the processor’s self-telemetry. Per-policy kept ratio (correctness), trace_dropped_too_early > 0 (silent undersizing), and per-pod accepted spans (ring balance) are the three signals that prove tail sampling works. Config review does not.
• Prefer built-in policies over ottl_condition where one fits — OTTL costs more CPU per span and can raise decision_timer_latency. Reserve it for genuinely compound conditions.
• Use drop for known noise (health checks, synthetic probes) so it overrides keep policies; otherwise a latency or baseline keep will happily retain your /healthz traffic.

Security notes

• TLS across trust boundaries. insecure: true is fine for in-cluster gateway→sampler traffic on a private network, but any hop that crosses a trust boundary (cluster-to-cluster, over the public internet, to a SaaS backend) must use real TLS. Configure protocol.otlp.tls with proper certs, not insecure.
• Least-privilege RBAC for the k8s resolver. The gateway ServiceAccount needs only get, list, watch on endpoints (or endpointslices) in the sampler’s namespace — do not grant cluster-admin or broad read. Scope the ClusterRole tightly.
• Traces carry sensitive data; sample and scrub deliberately. Spans routinely contain URLs with query strings, user IDs, and sometimes tokens or PII in attributes. Scrub in the gateway (an attributes/transform/redaction processor) before routing, so sensitive data never reaches the sampler or backend — and remember tail sampling keeps the traces you flagged as errors, which are often the ones richest in sensitive context.
• Protect the self-telemetry endpoint. :8888/metrics exposes operational detail (arrival rates, policy behaviour); scrape it over the cluster network via a ServiceMonitor, and do not expose it publicly. It is a metrics endpoint, not an authenticated API.
• Secure the backend export. The sampler tier is the only thing writing to Tempo/Jaeger/SaaS; use authenticated, TLS-encrypted export (bearer token, mTLS, or the vendor’s auth extension), and store credentials in a Secret, not the ConfigMap.
• Image provenance. Pin the otelcol-contrib image by digest, scan it, and pull from a private registry — the Collector sits on the critical telemetry path and processes every span.
• Multi-tenant isolation. If multiple tenants share the tail tier, ensure routing_key and policies do not let one tenant’s high-cardinality attributes or trace volume starve another’s sampling budget; consider per-tenant rate_limiting or separate sampler pools.

Cost & sizing

The bill has two halves — the Collector tiers you run, and the backend ingest/storage that tail sampling feeds — and they pull in opposite directions:

• Sampler tier compute dominates the new cost. It is memory-heavy by design: num_traces × avg_trace_bytes × overhead per replica, times the replica count. At Meridian Pay’s five 18 GB replicas that is ~90 GB of RAM standing 24/7 just to buffer in-flight traces — a real, continuous line item. Right-size it from the formula, don’t over-provision num_traces, and scale on arrival rate.
• Gateway tier is cheaper and stateless. It scales on ingest throughput/CPU, holds no buffers, and can run on smaller pods. Its cost is proportional to span volume received, not traces buffered.
• Backend ingest is proportional to kept spans (~3–5%), which is often more than 1% head sampling. Tail sampling is not a backend-savings play; budget for keeping all errors and slow traces plus your baseline. Lower the baseline percentage, tighten the latency threshold, and drop known noise to control it.
• Network egress can surprise you. Because the tail tier needs the firehose, you ship 100% of spans agent→gateway→sampler. If those hops cross availability zones, cross-AZ egress on the full span volume is a genuine cost — co-locate the tiers in one AZ where possible, or accept the egress as the price of full-trace assembly.
• The indirect saving is MTTR. The payoff is not on the infrastructure bill; it is the incident where the complete, kept trace turns a 40-minute investigation into a 4-minute one. That is real money in engineer-hours and outage duration, just not a line you can point at in the cloud invoice.

A rough monthly picture for a ~48,000-traces/sec platform (self-hosted Tempo, INR figures indicative, self-managed on a cloud K8s cluster):

The right-sizing rule: fix the connection-reuse-style waste first (don’t head-sample away the firehose, don’t over-size num_traces), then run the smallest sampler tier that keeps trace_dropped_too_early at zero and per-replica memory under limit. There is no point paying for two tiers if 1% head sampling already meets your debugging SLOs — quantify the MTTR benefit before committing the standing memory spend.

Interview & exam questions

1. Why is tail sampling stateful and head sampling stateless, and what does that imply for scaling? Head sampling decides at the root on a random number and propagates one bit in traceparent — no memory, no locality, trivially horizontal. Tail sampling buffers every span until the trace completes, then decides — so it needs memory proportional to in-flight traces and all spans of a trace in the same process. That locality is why you cannot scale it behind a round-robin Service and must route by trace ID.

2. What does the load-balancing exporter’s routing_key do, and why must it be traceID for tail sampling? It selects which field the exporter’s consistent-hash ring hashes to pick a backend. traceID hashes each span’s trace ID, so every span of a trace — whichever gateway received it — lands on the same sampler, giving that sampler a complete trace to decide on. service (the other common value) hashes service.name, scattering one trace’s spans across many samplers and tearing traces — the exact failure tail sampling must avoid.

3. Why must the sampler tier sit behind a headless Service? The load-balancing exporter builds its ring from the addresses the resolver returns. A normal ClusterIP Service resolves to one VIP, so the ring has a single member and all traffic funnels to one pod. A headless Service (clusterIP: None) exposes each pod’s IP as a distinct endpoint, so the ring has one member per pod and can actually spread traces. Non-headless is the single most common load-balancing misconfiguration.

4. How do you size decision_wait and num_traces, and what happens if you undersize them? decision_wait must exceed your real p99 end-to-end trace duration plus transit, or slow traces are decided before they complete and the latency policy misses them. num_traces must exceed per-replica arrival_rate × decision_wait with 3–4× headroom, or the buffer evicts traces before deciding them — decided on partial spans, which looks identical to a policy drop. The undersizing signal is the sampling_trace_dropped_too_early metric; there is no error otherwise.

5. Name five tail_sampling policy types and what each keeps. status_code keeps traces with a listed status (e.g. any ERROR span); latency keeps traces whose duration is between threshold_ms and upper_threshold_ms; probabilistic keeps a percentage by hashing the trace ID; rate_limiting keeps traces within a spans/sec budget; string_attribute keeps traces where an attribute matches listed values. Bonus combinators: and (all sub-policies), not (inverse), drop (force discard, overrides keeps), composite (budget allocation across categories).

6. What is the difference between the and and composite combinators? and is boolean conjunction: the trace is Sampled only if every sub-policy is Sampled (e.g. “is-payments AND within rate limit”). composite is budget allocation: it distributes a max_total_spans_per_second across ordered sub-policies by rate_allocation percentages, dividing one global cap fairly across categories. Reaching for composite to express “A and B” is a common mistake — that is what and is for.

7. Late spans arrive after a trace was already decided. What keeps them consistent with the trace? The decision_cache (with sampled_cache_size and non_sampled_cache_size set above their default of 0). It remembers recent kept/dropped verdicts, so a late span of a kept trace is exported and a late span of a dropped trace is dropped — keeping the trace whole across the decision boundary. Without it, post-decision stragglers start a fresh buffered trace and may get a different verdict, tearing the trace.

8. Does scaling out the sampler tier fix per-replica memory pressure, and what is the trade-off? Yes — adding replicas re-hashes the ring so each owns a smaller slice, so per-replica num_traces and memory can stay fixed while total capacity rises. The trade-off: each scale event (or restart) re-hashes ~R/N of routes, moving some in-flight traces to a new owner and briefly risking splits; mitigate with the k8s resolver, a PDB, and a grace period ≥ decision_wait. Prefer more small replicas over fewer fat ones to shrink the OOM blast radius.

9. Why can tail sampling end up costing more at the backend than 1% head sampling? Because keeping all errors and all slow traces plus a baseline often exceeds 1% of the firehose — net kept can be 3–5%. Tail sampling is not a storage-savings play; it buys decision quality (100% error catch, full p99 visibility) for roughly the cost of blind 4–5% head sampling, with the spend concentrated on high-signal traces rather than scattered across boring ones. Lower the baseline, tighten latency, and drop noise to control it.

10. Your error traces show up incomplete in Tempo, but the status_code: [ERROR] policy is configured. What is wrong and how do you confirm? The sampler tier is behind a ClusterIP VIP (or otherwise not routing by trace ID), so each trace’s spans scatter across replicas; the error policy fires on the fragment with the error span while other fragments are dropped by the baseline. Confirm with kubectl get svc <sampler> -o jsonpath='{.spec.clusterIP}' (not None) and a partial count_traces_sampled{policy="errors"} kept-ratio. Fix: headless Service + gateway/sampler split routing by traceID.

11. You switched to tail sampling but the tail tier only sees ~10% of traces. Why? An upstream SDK or agent is head-sampling before the tail (e.g. parentbased_traceidratio at 0.1), so only 10% of traces ever reach the Collector to be tail-decided. Tail sampling can only choose among traces that arrive. Fix: head-sample at 100% (parentbased_always_on) and defer the real decision to the tail tier.

12. Which OpenTelemetry Collector metric tells you the tail buffer is undersized, and why is it critical? otelcol_processor_tail_sampling_sampling_trace_dropped_too_early. It counts traces evicted from the buffer before their decision was made — which are then decided on incomplete spans, silently. Because this failure produces no error and looks identical to a policy drop in the backend, this metric being nonzero is the only reliable signal that num_traces/decision_wait are too small.

These map to vendor-neutral observability practice and the OpenTelemetry Certified Associate (OTCA) exam domains — the Collector’s receivers/processors/exporters, sampling strategies (head vs tail), and pipeline configuration. The distributed-systems and Kubernetes-scaling angles overlap with CKA/CKAD-style reasoning about Services, StatefulSets, and headless discovery. A compact mapping for revision:

Quick check

1. You have three sampler replicas behind a Kubernetes Service, each running tail_sampling with “keep 100% of errors,” but error traces show up in the backend with spans missing. What is almost certainly wrong, and the one field you check first?
2. True or false: setting routing_key: service on the load-balancing exporter is a valid way to route traces for tail sampling.
3. Your p99 end-to-end trace duration is 6 seconds. Is a decision_wait of 3 seconds safe? Why or why not, and which metric would reveal the problem?
4. Which combinator would you use to force-drop /healthz traces even though a latency policy would otherwise keep them — and, not, drop, or composite?
5. You switched from 1% head sampling to policy-based tail sampling and your backend bill went up. Give the one-sentence reason this can be correct rather than a bug.

Answers

1. The Service is almost certainly a normal ClusterIP (VIP), so it round-robins each trace’s spans across the three replicas and no replica assembles a whole trace — the errors policy fires only on the fragment holding the error span. Check kubectl get svc <sampler> -o jsonpath='{.spec.clusterIP}'; it must be None (headless), and you must route by traceID from a separate gateway tier.
2. False. routing_key: service hashes service.name, scattering one trace’s spans across many samplers (each service in the trace hashes to a different backend) — the exact split-trace failure. Tail sampling requires routing_key: traceID.
3. No, not safe. A 3-second window will decide many traces before they finish (their p99 is 6s), so the latency policy misses the slow traces you most want, and any spans arriving after 3s are late. The revealing metric is otelcol_processor_tail_sampling_sampling_late_span_age (its p99 will approach or exceed decision_wait); raise decision_wait above 6s plus transit.
4. drop. When its sub-policies match, drop forces a discard that overrides any keep from another policy — so a drop on the /healthz route wins over the latency keep. and is conjunction, not is inversion, composite is budget allocation.
5. Tail sampling keeps all errors and all slow traces plus a baseline, which commonly totals 3–5% of the firehose — more than 1% — so you are ingesting more spans, but they are the high-signal ones, bought for roughly the cost of blind 4–5% head sampling.

Glossary

• Head sampling — deciding whether to keep a trace at its root, before the trace exists, typically by probability; propagated in the traceparent sampled flag. Cheap, blind, consistent.
• Tail sampling — deciding after the trace is complete, with full knowledge of latency, errors, and attributes; stateful and memory-heavy. Implemented by the Collector’s tail_sampling processor.
• OpenTelemetry Collector — a vendor-neutral pipeline of receivers, processors, and exporters that ingests, transforms, and ships telemetry; the load-balancing exporter and tail_sampling processor are in the contrib distribution.
• tail_sampling processor — the Collector processor that buffers traces in memory keyed by trace ID, waits decision_wait, runs an ordered policy list, and emits one keep/drop verdict per trace.
• Load-balancing exporter — the Collector exporter that maintains a consistent-hash ring over backend Collectors and routes telemetry by a routing_key so related data co-locates; traceID co-locates whole traces.
• routing_key — the load-balancing exporter field selecting which value the ring hashes (traceID, service, resource, metric, streamID, attributes); must be traceID for tail sampling.
• Gateway tier — the stateless Collector layer that receives, enriches, batches, and routes spans to the sampler tier via the load-balancing exporter; scales on ingest throughput.
• Sampler tier — the stateful Collector layer that runs tail_sampling, owns a slice of the trace-ID space, assembles complete traces, and decides; scales on trace arrival rate and memory.
• Consistent-hash ring — the exporter’s backend-selection structure; when the backend list changes, only ~R/N of routes re-map, keeping most traces on their existing owner.
• Headless Service — a Kubernetes Service with clusterIP: None; DNS returns each pod’s IP as a distinct endpoint, so the ring gets one member per pod (required for the sampler tier).
• decision_wait — how long the processor buffers a trace before deciding (default 30s); must exceed real p99 trace duration plus transit.
• num_traces — the hard cap on traces resident in the processor’s buffer (default 50,000); must exceed per-replica arrival_rate × decision_wait with headroom.
• expected_new_traces_per_sec — pre-allocates the processor’s internal map to avoid rehash churn (default 0); set to per-replica steady-state arrival rate.
• decision_cache — remembers recent kept/dropped verdicts (sampled_cache_size, non_sampled_cache_size, both default 0) so late spans stay consistent with their trace.
• Late span — a span arriving after its trace was already decided; a rising sampling_late_span_age means decision_wait is too short.
• Policy — a rule in the tail_sampling list (e.g. status_code, latency, probabilistic, rate_limiting, string_attribute) that returns Sampled/NotSampled for a trace.
• Combinator — a policy that composes others: and (all match), not (inverse), drop (force discard, overrides keeps), composite (budget allocation across sub-policies).
• ottl_condition — a policy that evaluates an OpenTelemetry Transformation Language expression against spans; the escape hatch for compound conditions the built-ins can’t express.
• GOMEMLIMIT — the Go runtime soft memory limit; set to ~90% of the container limit so the GC runs aggressively under pressure instead of the container OOM-killing the process.
• sampling_trace_dropped_too_early — the processor metric counting traces evicted before decision; the only reliable signal that the buffer is undersized (the failure is otherwise silent).

Next steps

You can now build, size, and validate a two-tier tail-sampling pipeline that keeps whole traces and proves it with metrics. Build outward:

• Next: Building Production OpenTelemetry Collector Pipelines: Receivers, Processors, and Tail Sampling — the single-tier pipeline foundations and processor mechanics underneath this scaled design.
• Related: Distributed Tracing End-to-End: Context Propagation, Tempo, and Correlating Traces with Metrics and Logs — where the traces you sample land, and how they correlate with metrics and logs.
• Related: Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation — exemplars link a metric spike to a kept trace; tail sampling decides whether that trace survives.
• Related: OpenTelemetry for Java Services: Auto-Instrumentation, Context Propagation, and Custom Spans — where the spans are born and how the trace ID you route on gets propagated.
• Related: Zero-Code Auto-Instrumentation with Grafana Beyla: eBPF Traces and RED Metrics — generating traces without code changes to feed the sampling tier.
• Related: Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus — the metrics-side cost discipline that mirrors the tracing-cost decisions here.