Distributed Tracing End-to-End: Context Propagation, Tempo, and Correlating Traces with Metrics and Logs

A dashboard tells you the p99 latency on the checkout endpoint just doubled. A trace tells you which of the eleven downstream calls caused it, on which pod, with which tenant ID in the baggage. Distributed tracing is the only observability signal that reconstructs a single request’s path across service boundaries — and it is worthless if context drops on the first hop, if your spans are named so badly the backend can’t aggregate them, or if the trace sits in Tempo while you stare at a Grafana panel with no way to jump to it. Most teams that “have tracing” are in exactly that state: traces exist, nobody looks at them, and every incident still starts with grep.

This article is the end-to-end wiring, treated as a system rather than a product install: the byte layout of the traceparent header, deliberate span design with OpenTelemetry, a hard-nosed Grafana Tempo vs Jaeger comparison (storage model, components, TraceQL versus tag search), and the three correlation links that make tracing pay rent — exemplars (metrics → trace), trace_id in logs (trace ↔ log), and service graphs + span metrics (RED derived from traces). Plus head versus tail sampling, multi-tenancy and retention on both backends, and the five-pivot debugging workflow — alert → metric → exemplar → trace → log — that turns a page at 02:14 into a root cause by 02:25.

This is the concept-and-decision piece. When you are ready to stand up the Tempo backend itself — S3 block storage, compaction, metrics-generator tuning, TraceQL metrics — the implementation companion is Configure Grafana Tempo with TraceQL, Metrics-Generator, and S3 Block Storage. Here we build the mental model that makes every config line in that guide obvious.

What problem this solves

Metrics tell you that something is wrong, aggregated across everything. Logs tell you what one process said, in isolation. Neither answers the question that costs you money: which hop, in which service, on which code path, for which subset of requests? In a monolith you attached a debugger. In a system of 40 services, 3 message buses and 2 clouds, a checkout request touches 15 processes — and the slow one is three hops from the one that alerted.

Without tracing done properly, three failure patterns repeat everywhere. The war-room fan-out: an SLO alert fires at the gateway, nobody can see the request path, so every downstream team gets paged and declares “our dashboards look fine” — mean time to innocence beats mean time to resolution. The unaggregatable trace swamp: Jaeger was installed two years ago, spans are named GET /users/8231 and process, and sampling keeps 1% of everything — so 0 of the 30 errors that mattered. The disconnected pillars: traces, metrics and logs in three tools, with the on-call human as the join engine, copy-pasting timestamps between tabs.

The fix is not “buy an APM” — it is six contracts applied ruthlessly: W3C propagation on every hop including queues; semantic-convention span names and attributes; a sampling policy that guarantees errors and outliers are kept; RED metrics generated from spans; exemplars so every latency panel is a door into a trace; and trace_id on every log line. Wire those six and the three pillars collapse into one investigation surface.

Who hits this: platform teams standardising observability across polyglot services, SREs whose reviews keep saying “detection fast, localisation slow”, and architects choosing between Jaeger-on-Elasticsearch and Tempo-on-S3 with a real budget attached.

Learning objectives

By the end of this article you can:

• Explain span anatomy (trace_id, span_id, parent, kind, status, attributes, events, links) and read a traceparent header byte by byte.
• Propagate context across HTTP, gRPC, and message queues (Kafka/SQS/RabbitMQ), and choose parent-child versus span links for fan-out and batch consumers.
• Instrument services with the OTel SDK and zero-code agents, tune the batch processor, and apply semantic conventions so backends can aggregate your spans.
• Compare Tempo and Jaeger component by component — ingest, storage, index, query language, scale model — and defend a backend choice in an architecture review.
• Design a two-stage sampling policy: consistent head sampling in the SDK plus tail sampling in a Collector gateway that keeps 100% of errors and latency outliers.
• Generate RED metrics and service graphs from spans (Collector spanmetrics/servicegraph connectors or Tempo’s metrics-generator) without double counting.
• Wire exemplars end to end (SDK → Collector → Prometheus exemplar-storage → Grafana exemplarTraceIdDestinations) and both directions of trace↔log correlation (Loki derived fields, tracesToLogsV2).
• Operate the system: multi-tenant isolation with X-Scope-OrgID, per-tenant retention and rate limits, and a troubleshooting playbook for broken traces, missing exemplars and empty correlations.

Prerequisites & where this fits

You should be comfortable with Kubernetes (Deployments, Services, Helm), the Prometheus data model (labels, histograms, rate()), and one language well enough to read its OTel snippets — and know the OpenTelemetry Collector at block-diagram level: receivers, processors, exporters. The deep dive lives in Building Production OpenTelemetry Collector Pipelines: Receivers, Processors, and Tail Sampling; this article uses the Collector but does not re-teach it.

This piece sits at the centre of the observability track. Upstream of it: PromQL fluency (PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work) and log pipeline design (Grafana Loki Deep Dive: LogQL, Label Cardinality, and Chunk Storage Tuning). Downstream: the Tempo implementation guide already mentioned, tail sampling at fleet scale (Tail-Based Sampling at Scale with the OpenTelemetry Collector and Load-Balancing Exporter), and language-specific instrumentation (OpenTelemetry for Java Services: Auto-Instrumentation, Context Propagation, and Custom Spans).

Core concepts: spans, traces, and span context

A span is one timed unit of work. A trace is a tree of spans sharing one 16-byte trace_id. Each span carries an 8-byte span_id and (except the root) a parent_span_id. That is the whole data model; everything else — waterfalls, service graphs, RED metrics, exemplars — is derived from it.

trace_id = 4bf92f3577b34da6a3ce929d0e0e4736

  [HTTP GET /checkout]              span A (root, parent=none, kind=SERVER)
    [validate-cart]                 span B (parent=A, kind=INTERNAL)
    [POST inventory.Reserve]        span C (parent=A, kind=CLIENT)
       [grpc inventory.Reserve]     span D (parent=C, kind=SERVER, other process)
          [SELECT stock]            span E (parent=D, kind=CLIENT)
    [POST payments.Charge]          span F (parent=A, kind=CLIENT)

Spans C and D are the crux of distributed tracing: the caller and callee each create a span, in different processes, and D’s parent is C only because C’s context rode across the wire. Every field of a span has an operational consequence:

SpanKind deserves its own table because service graphs and the client/server latency split are computed from it, and getting it wrong silently breaks both:

Two rules complete the model. Status discipline: leave status Unset for normal flow, set Error only for genuine failures and record the exception as a span event — a 404 on a lookup endpoint is a valid outcome, and marking it Error corrupts every error-rate metric derived from spans. Span context: only trace_id + span_id + trace flags + trace state travel between processes. Attributes, events and names do not propagate — anything downstream must see goes in baggage, explicitly.

Context propagation: W3C traceparent and where traces break

Propagation is serializing the active span context into an outbound carrier (HTTP headers, gRPC metadata, Kafka record headers) and deserializing it on the other side so the remote span parents correctly. Drop it on any hop and the trace splits — the most common tracing failure in production fleets.

The standard is W3C Trace Context: two headers, parsed byte-for-byte by every SDK and proxy:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ ^------------------------------^ ^--------------^ ^^
             ver           trace-id (16B)          parent-id (8B)  flags

tracestate: vendorA=opaqueValue,vendorB=opaqueValue
baggage:    tenant.id=acme,checkout.experiment=b

W3C is the standard, but it is not the only format alive in real fleets — and mismatched propagators are the number-one cause of silently broken traces at organisational boundaries:

Spec-compliant OTel SDKs default to tracecontext,baggage. Breakage comes from mixed fleets: a Sleuth service emitting only B3 calls an OTel service expecting traceparent, and the trace forks. The fix is either standardising on W3C everywhere or running a composite propagator that reads all formats during the migration:

# Transitional setting on every legacy-adjacent service:
export OTEL_PROPAGATORS="tracecontext,baggage,b3multi"

Carriers: how context physically travels per transport

For synchronous HTTP/gRPC, instrumentation libraries inject and extract automatically once the propagator is set. Everything else needs deliberate work:

Async messaging is where traces die most often: the producer’s HTTP context is long gone when a consumer picks up the message, so context must ride inside the message:

# Producer: inject context into Kafka record headers
from opentelemetry.propagate import inject

carrier = {}
inject(carrier)                       # writes traceparent/baggage into the dict
producer.send("orders",
              value=payload,
              headers=[(k, v.encode()) for k, v in carrier.items()])

# Consumer: extract, then decide — child (1:1) or links (batch/fan-out)
from opentelemetry.propagate import extract
from opentelemetry import trace

tracer = trace.get_tracer("orders.consumer")

# 1:1 processing → parent-child is honest
ctx = extract({k: v.decode() for k, v in msg.headers})
with tracer.start_as_current_span("orders process", context=ctx,
                                  kind=trace.SpanKind.CONSUMER):
    handle(msg)

# Batch of 500 → parenting one poll span under 500 producers is meaningless.
# Use span LINKS: causality without a fake tree.
links = [trace.Link(extract(dict(r.headers)).get_span_context())
         for r in records]
with tracer.start_as_current_span("orders settle-batch", links=links,
                                  kind=trace.SpanKind.CONSUMER) as span:
    span.set_attribute("messaging.batch.message_count", len(records))
    settle(records)

Where traces break: the boundary audit

Walk every boundary against this table once and you will find the split traces before an incident does:

Two boundary rules. The sampled bit is part of the contract: ParentBased samplers downstream honour the 01 flag — that is what keeps traces complete instead of half-sampled. Baggage is a loaded weapon: the only way to move business context (tenant, experiment, priority) to every hop, and the easiest way to leak PII into every hop’s logs — budget it to a few short keys and strip it at the trust boundary with a Collector attributes/redaction processor.

Instrumenting with OpenTelemetry: SDK, auto-instrumentation, Collector

OpenTelemetry won the instrumentation war: vendor-neutral API + SDK + wire protocol (OTLP), with Tempo and Jaeger both OTLP-native. Instrumenting well means understanding the SDK pipeline you are configuring, by code or by environment variables:

The full SDK bootstrap in Python — every line maps to a row above:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "checkout",
    "service.version": "1.42.0",
    "deployment.environment.name": "prod",
})

provider = TracerProvider(
    resource=resource,
    sampler=ParentBased(root=TraceIdRatioBased(0.25)),
)
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://otel-agent:4317", insecure=True),
))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("checkout.orders", "1.0.0")

In containerised fleets you rarely hardcode this — spec-defined environment variables that every SDK honours make instrumentation a deployment concern:

Zero-code versus manual instrumentation

You almost never write spans for HTTP servers, clients, and database drivers — auto-instrumentation covers those. You do write spans for business logic worth seeing on a timeline:

# Java: full tracing with zero code changes (-Dotel.* mirrors the env vars above)
java -javaagent:/otel/opentelemetry-javaagent.jar \
     -Dotel.service.name=checkout \
     -Dotel.traces.sampler=parentbased_traceidratio \
     -Dotel.traces.sampler.arg=0.25 \
     -jar checkout.jar

The Kubernetes-native version of this — the OTel Operator injecting agents via pod annotations — is covered in Deploy the OpenTelemetry Operator with Target Allocator and Auto-Instrumentation, and the eBPF path in Zero-Code Instrumentation with eBPF: Grafana Beyla and OpenTelemetry.

Semantic conventions: the attribute contract

Backends aggregate spans by attributes. If every team invents its own keys (http_status, statusCode, code), TraceQL queries, span-metrics dimensions and vendor integrations all break. Semantic conventions are the shared vocabulary — use them verbatim:

The naming rule that protects aggregation — the span name is a low-cardinality template; variables are attributes:

Finally, avoid span explosion: no span per loop iteration or per row — processing 10,000 records is one span with batch.size=10000. A span should represent work worth seeing on a timeline (a network call, a transaction, a meaningful compute phase); everything finer is pure ingest, storage and query cost.

Tempo vs Jaeger: architecture, storage, and query

Both are open-source, OTLP-native trace backends, and both render a waterfall. The architectural bets underneath are near-opposite, and they decide your cost curve and your query ergonomics. The headline matrix first, then the mechanics:

Storage and index: why the cost curves diverge

Jaeger writes each span into a database and maintains an index over service names, operation names, tags and durations. Elasticsearch/OpenSearch is the common production choice: one index per day (jaeger-span-2026-06-04), shards and replicas to size, sub-second searches over any indexed tag. The price is the database itself: at tens of thousands of spans per second the cluster needs multiple data nodes with fast disks running around the clock — you pay for peak, always.

Tempo writes spans into columnar Parquet blocks in object storage with only a tiny bloom-filter/index footprint per block. Trace-by-ID lookup is cheap; search means queriers scanning block columns in parallel — brute force made affordable by Parquet’s column pruning and object-storage bandwidth. The price moves from storage to query-time CPU — elastic and mostly idle — and retention costs S3 prices, not ES-node prices. The block/compaction/caching machinery is what Configure Grafana Tempo with TraceQL, Metrics-Generator, and S3 Block Storage walks through.

Query: TraceQL versus Jaeger search

Jaeger’s search is a form: service, operation, tags (error=true http.status_code=503), min/max duration, lookback, limit (default 20, UI caps at 1500 results). It is fast and fine for “show me recent errors in checkout”. It cannot express structure — “traces where checkout calls payments AND the payments span was slow” — because the index is per-span, not per-tree.

TraceQL is a pipeline language over trace structure, and it is Tempo’s decisive UX advantage:

Capability side by side:

Deployment shapes and the v2 note

Tempo ships one binary in two modes: monolithic (everything in one process — labs, small shops) and microservices (distributor → ingester → querier/query-frontend → compactor + metrics-generator) for scale, deployed via the tempo-distributed Helm chart. Jaeger v1’s shape is collector → storage ← query, optionally with Kafka and an ingester between collector and storage as a buffer. Jaeger v2 rebuilds the whole project as an OpenTelemetry Collector distribution: the same binary hosts receivers, processors, a jaeger_storage extension and the jaeger_query UI — which means Collector-style YAML config, first-class OTLP, and any Collector processor available natively in Jaeger:

# jaeger-v2.yaml — Jaeger as an OTel Collector distribution (Elasticsearch backend)
extensions:
  jaeger_query:
    storage:
      traces: es_main
  jaeger_storage:
    backends:
      es_main:
        elasticsearch:
          server_urls: ["http://elasticsearch:9200"]
          indices:
            index_prefix: "jaeger-prod"

exporters:
  jaeger_storage_exporter:
    trace_storage: es_main

service:
  extensions: [jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]

And the Tempo equivalent skeleton, showing the object-storage bet directly:

# tempo.yaml (excerpt) — object storage + metrics-generator
multitenancy_enabled: false          # flip to true for X-Scope-OrgID isolation

distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: "0.0.0.0:4317" }
        http: { endpoint: "0.0.0.0:4318" }

storage:
  trace:
    backend: s3
    s3:
      bucket: kloudvin-tempo-traces
      endpoint: s3.ap-south-1.amazonaws.com
      region: ap-south-1
    wal:
      path: /var/tempo/wal

compactor:
  compaction:
    block_retention: 336h            # 14 days, the global default

metrics_generator:
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus.observability.svc:9090/api/v1/write
        send_exemplars: true

overrides:
  defaults:
    metrics_generator:
      processors: [span-metrics, service-graphs]

The decision table

From here the examples default to Tempo, whose correlation features are the tightest fit with Grafana — but every concept maps to Jaeger, and the tables call out where mechanics differ.

Sampling strategies: head, tail, and remote

Storing every span of every request is financially silly at scale: healthy traffic is overwhelmingly self-similar. The question is what to keep, and there are two places to decide — at trace start (head) or after completion (tail) — plus a hybrid where the backend pushes rates to SDKs (remote).

Head sampling happens in the SDK at root-span creation. The standard samplers:

Two properties make head sampling safe in a distributed system. Parent-based composition: every non-root service honours the inbound sampled flag, or you get half-traces. Consistency: traceidratio derives the decision from the trace_id itself, so every service computing the same ratio reaches the same answer (the W3C Level-2 random flag formalises the entropy). What head sampling cannot do is see the future — a 5% head sample throws away 95% of your errors, because the decision precedes the failure.

Tail sampling decides after the whole trace is assembled, in a Collector gateway tier, and can therefore keep all the interesting traces:

# Collector gateway: guarantee errors + outliers, sample the boring 95%
processors:
  tail_sampling:
    decision_wait: 10s          # buffer window before deciding (default 30s)
    num_traces: 200000          # in-flight traces held in memory
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: healthy-baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Tail sampling has real costs: the gateway buffers every in-flight trace in RAM for decision_wait, and all spans of a trace must reach the same Collector instance — at fleet scale that forces a two-tier layout with the loadbalancing exporter routing by trace ID, the subject of Tail-Based Sampling at Scale with the OpenTelemetry Collector and Load-Balancing Exporter.

How the pieces combine in practice:

One principle above all of it: metrics at 100%, traces sampled. RED metrics generated from spans (next section) are cheap aggregates — never sample those. Sample the examples, keep the totals. And generate the metrics before the tail-sampling processor in the pipeline graph, or your “request rate” will silently become “request rate of kept traces”.

Span metrics and service graphs: RED from traces

Every span already encodes RED: it happened (Rate), it has a status (Errors), it has a duration (Duration). Aggregating spans into Prometheus series yields request metrics with the same dimensions as your traces — precisely what makes exemplars and tracesToMetrics line up later. Two places can run this aggregation; run exactly one:

The Collector-side wiring, including the fan-out that lets metrics see 100% of spans while Tempo receives only the tail-sampled subset:

connectors:
  spanmetrics:
    histogram:
      exemplars:
        enabled: true                       # carries trace_id samples → step 6
      explicit:
        buckets: [5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s]
    dimensions:
      - name: http.route
      - name: http.response.status_code
  servicegraph:
    latency_histogram_buckets: [10ms, 50ms, 100ms, 500ms, 1s, 5s]

service:
  pipelines:
    traces/full:                            # 100% of spans → RED metrics
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [spanmetrics, servicegraph]
    traces/store:                           # sampled subset → Tempo
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics/red:
      receivers: [spanmetrics, servicegraph]
      processors: [batch]
      exporters: [prometheusremotewrite]

What comes out the other side — the series you will alert and dashboard on:

Service graphs are computed by pairing each CLIENT span with the matching remote SERVER span (via trace_id + parent relationship) and emitting an edge client→server. Both sides must be instrumented and correctly kinded or the edge never forms, and chronic one-sided instrumentation surfaces as unpaired_spans_total — one of the most useful “is propagation healthy” meta-signals you own. Grafana renders the map from these series via serviceMap.datasourceUid pointing at your Prometheus/Mimir.

Dimension discipline is the same as any Prometheus work: every dimensions: entry multiplies series count. http.route (bounded) yes; url.path, user.id, tenant.id for thousands of tenants — no. The governance maths lives in Prometheus Cardinality Control: Relabeling and Cost Governance, and dashboarding these RED series properly is Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code.

Exemplars: the metrics-to-trace jump

An exemplar is a sample attached to a histogram bucket carrying the trace_id (plus value and timestamp) of one real request that landed in that bucket. On a Grafana latency panel exemplars render as diamonds under the line; clicking one opens that request’s trace. It is the highest-leverage click in observability — a p99 spike stops being a statistic and becomes a specific slow trace — and it fails unless all four stages of a chain hold:

The storage flag and the datasource link, concretely:

# Prometheus with exemplar storage (feature flag) and remote-write receiver
prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --enable-feature=exemplar-storage \
  --web.enable-remote-write-receiver

# Grafana provisioning: Prometheus → Tempo via exemplars
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      exemplarTraceIdDestinations:
        - name: trace_id          # the exemplar's label key
          datasourceUid: tempo

Any histogram panel then does the work — toggle “Exemplars” on in the query options:

histogram_quantile(0.99,
  sum by (le, http_route) (
    rate(traces_span_metrics_duration_milliseconds_bucket[5m])
  )
)

Two caveats. Exemplars are not a statistical sample — Prometheus keeps a bounded circular buffer of recent representatives, which is what debugging needs but not what reporting should be built on. And an exemplar pointing at a trace tail sampling dropped is a dead link — one more reason “keep all errors + slow traces” is the right tail policy, since those are exactly the traces anomalous buckets point to. The application-side view of this plumbing (SDK histograms attaching exemplars directly) is covered in Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation.

Trace-log correlation: trace_id everywhere

The third correlation is logs, and the rule is brutally simple: every structured log line emitted inside an active span carries trace_id and span_id. Do that and both pivots open — span to exactly its log lines, log line to the full distributed trace. Miss it and you are back to timestamp archaeology.

Injection is a logging-layer concern, one small shim per language:

# Python: stamp trace context onto every log record
import logging
from opentelemetry import trace

class TraceContextFilter(logging.Filter):
    def filter(self, record):
        ctx = trace.get_current_span().get_span_context()
        record.trace_id = format(ctx.trace_id, "032x") if ctx.is_valid else ""
        record.span_id  = format(ctx.span_id,  "016x") if ctx.is_valid else ""
        return True

{"ts":"2026-06-04T10:41:03Z","level":"error","service":"checkout",
 "msg":"inventory reserve failed","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
 "span_id":"00f067aa0ba902b7"}

(Java: the OTel agent auto-populates MDC keys trace_id/span_id; Go: wrap slog/zap with a handler reading trace.SpanContextFromContext; .NET: Activity.Current.TraceId flows into ILogger scopes.)

With Loki storing logs, wire both directions in Grafana — pure data-source configuration, no code:

# Loki → Tempo: derived field lifts trace_id out of the line and links it
  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: '"trace_id":"([0-9a-f]+)"'
          url: '$${__value.raw}'        # $$ escapes $ in provisioning files
          datasourceUid: tempo

# Tempo → Loki: from a span, query logs scoped by labels + trace_id + time
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: '-5m'
        spanEndTimeShift: '5m'
        filterByTraceID: true
        tags: [{ key: 'service.name', value: 'service_name' }]
      tracesToMetrics:
        datasourceUid: prom
        spanStartTimeShift: '-2m'
        spanEndTimeShift: '2m'
        queries:
          - name: 'Route p99'
            query: 'histogram_quantile(0.99, sum by (le) (rate(traces_span_metrics_duration_milliseconds_bucket{$$__tags}[5m])))'
      serviceMap:
        datasourceUid: prom
      nodeGraph:
        enabled: true

The details that decide whether these links return results or an empty pane:

Where should trace_id live on the Loki side? Not as a stream label — trace IDs are unique per request, and unique-valued labels are the cardinality catastrophe explained in Grafana Loki Deep Dive: LogQL, Label Cardinality, and Chunk Storage Tuning. Keep it in the log line (filtered with |=/| trace_id="") or, on Loki 3.x, in structured metadata — the indexed-but-not-a-label middle ground this problem needed; Loki’s OTLP endpoint puts it there automatically.

The complete correlation matrix — six pivots, each with its own mechanism:

The debugging workflow: alert to root cause in five pivots

Everything above exists to make one on-call motion fast. The drill, with the artefact each pivot hands the next:

Ninety seconds of clicking, zero grep, and one person does it without paging four teams. The reverse entry points work too: support ticket with request ID → log search → derived field → trace → service map; suspicious deploy → tracesToMetrics from a canary span → route history. The alert layer feeding step 1 is covered in SLOs and Error Budgets in Practice: Defining SLIs and Building Multi-Window Burn-Rate Alerts. Three checkable invariants keep the workflow trustworthy: exemplars point at kept traces (tail policy retains errors/outliers), span metrics are generated before sampling (or panels lie), and clocks are disciplined (skew breaks both the log window and waterfall ordering).

Multi-tenancy and retention

The moment tracing serves more than one team, two governance questions arrive: whose spans are whose (isolation, limits) and how long do they live (retention, cost). The backends answer very differently:

The per-tenant overrides file is where platform teams encode the contract — limits, retention, metric generation, per X-Scope-OrgID:

# tempo.yaml
multitenancy_enabled: true
overrides:
  per_tenant_override_config: /conf/overrides.yaml

# overrides.yaml — the platform contract, per tenant
overrides:
  "team-payments":                  # X-Scope-OrgID: team-payments
    ingestion:
      rate_limit_bytes: 30000000    # 30 MB/s (global default 15 MB/s)
      burst_size_bytes: 40000000
    global:
      max_bytes_per_trace: 10000000 # pathological traces rejected at 10 MB
    compaction:
      block_retention: 720h         # 30 days — they pay for it
    metrics_generator:
      processors: [span-metrics, service-graphs]
  "team-internal-tools":
    compaction:
      block_retention: 168h         # 7 days is plenty
    metrics_generator:
      processors: []                # no RED generation for this tenant

The Collector side stamps the tenant on the way in — one gateway per team namespace, or the headers_setter extension mapping a resource attribute to the header:

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc:4317
    headers:
      X-Scope-OrgID: team-payments
    tls:
      insecure: true    # in-cluster only; mTLS at the mesh layer

Retention mechanics: “we keep traces 30 days” is a config line in Tempo and an operational subsystem in Jaeger:

Retention strategy itself: traces age badly — nobody opens a 60-day-old waterfall. Keep the aggregates long-term instead (span metrics in Mimir, retained 13 months cheaply — see Running Grafana Mimir: Multi-Tenant, Horizontally Scalable Prometheus Storage); 14–30 days of traces with 100%-of-errors sampling covers real debugging at a fraction of the cost of “keep everything for a year”.

Architecture at a glance

Picture the pipeline left to right in four tiers. Tier 1 — services: every pod runs an OTel SDK (or agent) with W3C propagation; HTTP/gRPC hops carry traceparent automatically, Kafka producers inject it into record headers, and every log line is stamped with trace_id/span_id. Spans batch out via OTLP to a node-local Collector agent. Tier 2 — Collector gateway: agents forward to a small gateway fleet (routed by trace ID when tail sampling is on), where the traffic forks deliberately: one pipeline feeds the spanmetrics and servicegraph connectors so RED metrics and edges are computed from 100% of spans and remote-written to Prometheus/Mimir with exemplars; the other applies tail sampling — every error, every trace over the latency threshold, a 5% healthy baseline — and exports the survivors to the trace backend.

Tier 3 — backends: Tempo’s distributor receives OTLP (per-tenant via X-Scope-OrgID), ingesters cut Parquet blocks to S3, the compactor enforces per-tenant retention, queriers serve TraceQL; in a Jaeger deployment the same position is held by collectors writing to Elasticsearch/Cassandra with the query service in front. Loki holds the logs alongside; Prometheus/Mimir holds the RED series and service-graph edges. Tier 4 — Grafana: one pane wired with the six correlation links — exemplar diamonds from any latency panel into the exact trace, tracesToLogsV2 from any span into its Loki lines, derived fields from any log line back to the full trace, tracesToMetrics and the service map closing the loop. The topology is a lambda shape: aggregates flow unsampled to the metrics store, examples flow sampled to the trace store, and trace_id is the foreign key joining every surface.

Real-world scenario

A payments platform (“FinRail”, ~90 services, 60k spans/s peak, AWS ap-south-1) ran Jaeger 1.x on a 6-node OpenSearch cluster: roughly $1,900/month to hold 7 days of head-sampled traces at 10%. Incident reviews kept repeating one line: “detection 3 minutes, localisation 45”. The 10% head sample meant the trace for any given failed settlement existed one time in ten — the search index was fast, but the trace people needed usually wasn’t there. Checkout p99 alerts pointed at dashboards with no path into traces; engineers searched Jaeger by service + time window and scrolled.

The platform team rebuilt over six weeks. Weeks 1–2: standardised on W3C propagation (three Spring services still emitted only B3 — the composite propagator bridged the migration) and pushed trace_id into all JSON logs via the Java agent’s MDC. Weeks 3–4: a two-tier Collector layout — node agents plus a 6-pod gateway with the loadbalancing exporter — with RED generation moved into spanmetrics/servicegraph connectors before tail sampling, and sampling switched to head 100% → tail “all errors + traces > 800ms + 5% baseline”. Weeks 5–6: Tempo (microservices mode) on S3, 30-day retention for the payments tenant and 14 for everyone else via X-Scope-OrgID, and Grafana provisioned with the full correlation mesh.

The numbers after: stored trace volume dropped 84% versus “keep 10% of everything” (tail policies kept ~5.8% of traces, but the right 5.8% — error-trace coverage went from ~10% to 100%). Backend cost fell from ~$1,900/mo to ~$410/mo (S3 ~2.1 TB steady-state + Tempo and gateway pods). The cultural change came from exemplars: median “alert → trace in hand” went from 14 minutes to under 60 seconds, and localisation MTTR dropped 58% over the next quarter. The regression they accepted knowingly: ad-hoc tag search over long windows got slower (TraceQL scan vs OpenSearch index) — a fair trade, since 95% of trace opens now arrive via an exemplar or log link carrying an exact trace_id, where Tempo is instant.

Advantages and disadvantages

The honest summary: the system is strictly better than disconnected pillars, and the cost is that it is a system — a dozen configs across SDKs, Collectors, backends and Grafana that must agree. The troubleshooting table exists because they occasionally won’t.

Hands-on lab

Build the correlation mesh on your laptop: Tempo (OTLP ingest + metrics-generator) remote-writing to Prometheus (exemplar storage), Loki and Grafana alongside, traces generated with telemetrygen, then click every link. Docker Compose, no cloud spend.

1. Layout.

mkdir -p tracing-lab/{tempo,prometheus,grafana/provisioning/datasources} && cd tracing-lab

2. Tempo config — monolithic, local block storage, metrics-generator remote-writing RED + service graphs with exemplars:

# tempo/tempo.yaml
server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: "0.0.0.0:4317" }
storage:
  trace:
    backend: local
    local: { path: /var/tempo/blocks }
    wal:   { path: /var/tempo/wal }
compactor:
  compaction:
    block_retention: 24h
metrics_generator:
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
overrides:
  defaults:
    metrics_generator:
      processors: [span-metrics, service-graphs]

3. Prometheus config (scrapes nothing; receives remote_write):

# prometheus/prometheus.yml
global:
  scrape_interval: 15s

4. Grafana datasources — the whole point of the lab, every correlation in one file:

# grafana/provisioning/datasources/ds.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    url: http://prometheus:9090
    jsonData:
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo
  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: '"trace_id":"([0-9a-f]+)"'
          url: '$${__value.raw}'
          datasourceUid: tempo
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: '-5m'
        spanEndTimeShift: '5m'
        filterByTraceID: true
        tags: [{ key: 'service.name', value: 'service_name' }]
      serviceMap: { datasourceUid: prom }
      nodeGraph: { enabled: true }

5. Compose file. (Tempo’s image runs as a non-root user; for a lab, running as root sidesteps volume-permission setup — do not do this in production.)

# docker-compose.yaml
services:
  tempo:
    image: grafana/tempo:latest
    user: "0"
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo/tempo.yaml:/etc/tempo.yaml
      - tempo-data:/var/tempo
    ports: ["3200:3200", "4317:4317"]
  prometheus:
    image: prom/prometheus:latest
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --enable-feature=exemplar-storage
      - --web.enable-remote-write-receiver
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports: ["9090:9090"]
  loki:
    image: grafana/loki:latest
    ports: ["3100:3100"]
  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
    ports: ["3000:3000"]
volumes:
  tempo-data:

docker compose up -d
docker compose ps          # expect 4 services Up

6. Generate traces with telemetrygen (the Collector project’s load tool):

docker run --rm --network tracing-lab_default \
  ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest \
  traces --otlp-endpoint tempo:4317 --otlp-insecure \
  --service checkout --rate 20 --duration 120s --child-spans 4

The run ends by reporting traces generated (≈ 2,400).

7. Validate each link in the chain.

# a) Tempo has traces (TraceQL search over everything)
curl -s "http://localhost:3200/api/search?q=%7B%7D&limit=3" | jq '.traces[].traceID'
# → three 32-hex trace IDs

# b) metrics-generator produced RED series in Prometheus (allow ~60s)
curl -s "http://localhost:9090/api/v1/query?query=traces_spanmetrics_calls_total" \
  | jq '.data.result | length'
# → non-zero

# c) exemplars are stored alongside the latency histogram
curl -s "http://localhost:9090/api/v1/query_exemplars" \
  --data-urlencode 'query=traces_spanmetrics_latency_bucket' \
  --data-urlencode "start=$(date -u -v-1H +%s 2>/dev/null || date -u -d '-1 hour' +%s)" \
  --data-urlencode "end=$(date -u +%s)" | jq '.data | length'
# → non-zero; each exemplar carries labels.trace_id

8. Prove log↔trace correlation — push a log line into Loki carrying a real trace_id, then click through:

TRACE_ID=$(curl -s "http://localhost:3200/api/search?q=%7B%7D&limit=1" | jq -r '.traces[0].traceID')
curl -s -X POST http://localhost:3100/loki/api/v1/push \
  -H "Content-Type: application/json" \
  -d "{\"streams\":[{\"stream\":{\"service_name\":\"checkout\",\"level\":\"error\"},
       \"values\":[[\"$(date +%s)000000000\",
       \"{\\\"msg\\\":\\\"payment failed\\\",\\\"trace_id\\\":\\\"${TRACE_ID}\\\"}\"]]}]}"

9. Click the mesh in Grafana (http://localhost:3000): Explore → Prometheus → query histogram_quantile(0.99, sum by (le) (rate(traces_spanmetrics_latency_bucket[5m]))), enable Exemplars → diamonds render → click one → the Tempo trace opens. In the trace view, open a span → Logs for this span → your pushed Loki line appears. In Explore → Loki → {service_name="checkout"}, expand the line — the TraceID derived field links back. Explore → Tempo → Service Graph renders checkout and its synthetic children.

10. Teardown.

docker compose down -v && cd .. && rm -rf tracing-lab

If step 7c returns 0, you have reproduced the most common exemplar failure — work the troubleshooting table from row 6.

Common mistakes & troubleshooting

The playbook, symptom-first. Confirm before you fix; every row has a concrete check.

Meta-lesson from rows 6–8: correlation features fail silently — nothing errors, the click just returns empty. Treat the correlation mesh as a deployable artefact with its own smoke test (lab step 7, scriptable against staging).

Best practices

• Standardise W3C tracecontext,baggage fleet-wide; run composite propagators only as a bounded migration, with an end date.
• Name spans as low-cardinality operation templates; push all variables into semconv attributes. Review span names in PRs like metric label sets.
• Set status Error only for real failures and record exceptions as events — your error-rate metrics are only as honest as this discipline.
• Instrument once, derive thrice: spans → span metrics → service graphs. Never hand-maintain a parallel RED instrumentation that can drift from the traces.
• Generate metrics before sampling, sample traces aggressively: 100% metrics, tail-kept errors + outliers + small baseline; audit monthly that every SEV kept its trace.
• Make exemplars a tested contract: a CI/staging smoke test that queries query_exemplars and resolves one trace end-to-end.
• Stamp trace_id/span_id into logs via shared logging config (platform-owned library), not per-team copy-paste.
• Budget baggage (≤ 3 keys, short values, no PII) and enforce centrally with Collector redaction/attributes processors at the egress gateway.
• Watch the meta-signals: unpaired_spans_total, tempo_discarded_spans_total, Collector queue/drop counters — they catch broken tracing before humans notice.
• Pin per-tenant limits and retention in the overrides file under version control; changes go through PR review like any capacity contract.
• Keep clocks disciplined (chrony everywhere, including VMs and on-prem) — skewed waterfalls destroy trust in the tool faster than any outage.
• Run game days on the pivot chain: hand an engineer an alert and require root cause via exemplar→trace→log only; any dead pivot is the action item.

Security notes

Traces are an under-audited data store that routinely ends up containing what your DLP policy forbids. Attributes and baggage are the leak surface: URLs with tokens in query strings (url.full), raw SQL with literals (db.query.text unsanitised), enduser.id as plain email, request bodies dumped into events. Enforce redaction centrally in the Collector gateway — attributes (delete/hash specific keys) and redaction (allowlist mode) processors — so a single misbehaving team can’t leak into the shared backend. Baggage deserves special paranoia: it propagates to every downstream service, crosses trust boundaries you forgot existed, and lands in logs; strip it at the network edge in both directions.

Transport and access: OTLP from agents to gateway to backend rides mTLS (mesh-issued certs in-cluster; never insecure: true across nodes in production). Tempo’s X-Scope-OrgID is an identifier, not authentication — anyone who reaches the distributor can claim any tenant, so front it with an authenticating proxy and give Grafana per-tenant datasources locked with RBAC so team A cannot query team B’s traces. Jaeger’s UI has no native authz — front it with an OIDC proxy. Retention is a security control too: traces older than your incident-review horizon are pure liability — let the compactor/ILM delete them, keep the bucket private with public access blocked, and use SSE-KMS if compliance asks. Finally, traceparent on outbound requests to third parties leaks request structure — usually harmless, but strip tracestate/baggage at egress and know the header crosses the boundary.

Cost & sizing

The bill drivers, in order: span volume (spans/s × bytes/span), retention days, span-metrics cardinality, and the always-on compute of your backend. Working numbers: a typical instrumented request produces 5–20 spans; a span serialises to ~300–1,000 bytes on the wire and compacts to roughly a third in columnar storage.

Right-sizing heuristics: start tail sampling at “errors + >p99-ish threshold + 5% baseline” and tune until kept-traffic is 5–10% of raw; set retention at 14 days and extend only per tenant, with a use case (the override makes them pay their own delta); review span-metrics dimensions quarterly exactly like scrape cardinality. For a small platform (≤ 5k spans/s) the entire Tempo stack fits in three small pods + a bucket — comfortably inside a lab budget. Grafana Cloud’s free tier (which includes a traces allowance) is a legitimate zero-ops way to run this architecture while you learn — it is Tempo underneath, so everything transfers 1:1.

Interview & exam questions

Q1. Walk me through what happens to trace context when service A calls service B over HTTP. A’s SDK has an active span; the HTTP client instrumentation asks the global propagator to inject its span context into headers — traceparent: 00-<trace_id>-<A's span_id>-<flags> plus tracestate/baggage. B’s server instrumentation extracts it, creates a SERVER span whose parent is A’s span_id, same trace_id, honouring the sampled flag via its ParentBased sampler. Attributes and events do not travel; only the context tuple does.

Q2. Why can’t head sampling guarantee you keep error traces, and what’s the fix? The head decision is made at root-span creation, before any downstream failure exists, and it’s derived from the trace_id ratio — so errors are kept at exactly the baseline rate. The fix is tail sampling in a Collector gateway: buffer the complete trace (decision_wait), then apply status_code and latency policies that deterministically keep failures and outliers, plus a probabilistic baseline.

Q3. Tempo versus Jaeger: argue the storage trade-off in one minute. Jaeger indexes spans in a database (ES/Cassandra), so tag search is fast but you pay for an always-on cluster sized for peak. Tempo writes columnar Parquet blocks to object storage with minimal index; trace-by-ID is cheap, search is a parallel scan (TraceQL), and cost scales with bytes at S3 prices. Tempo’s bet: you usually arrive carrying a trace_id (exemplar, log link), so a heavy index is wasted spend. Choose Jaeger for indexed ad-hoc search you truly need; Tempo for volume and Grafana-native correlation.

Q4. What is an exemplar, end to end? A sample attached to a histogram bucket carrying the trace_id (plus value/timestamp) of one real request in that bucket. Chain: metrics source attaches it (spanmetrics connector or SDK) → remote_write with send_exemplars → Prometheus stores it behind --enable-feature=exemplar-storage in a bounded buffer → Grafana’s exemplarTraceIdDestinations renders diamonds that deep-link into the trace datasource.

Q5. Your consumer spans all start new traces. Diagnose. Context isn’t riding the messages: HTTP auto-instrumentation doesn’t cover queue carriers. Producer must inject() the context into message headers; consumer must extract() and start its CONSUMER span against that context — or, for batch consumption, attach span links to each message’s context instead of faking a single parent.

Q6. How do service-graph metrics get computed, and what breaks them? The processor pairs a CLIENT span with the remote SERVER span it spawned (same trace, parent relationship) and emits client→server edge counters and latency histograms from both viewpoints. Broken by: missing instrumentation on one side, wrong span kinds (INTERNAL everywhere), or propagation drops — all visible as traces_service_graph_unpaired_spans_total rising.

Q7. Why must span metrics be generated before tail sampling? Tail sampling drops most healthy traces by design. Metrics generated after it measure only survivors — request rates ~20× low, latency distributions biased toward the slow traces you kept. Fan the raw stream into spanmetrics in one pipeline and sample in another; metrics stay 100%-accurate while traces stay cheap.

Q8. How does multi-tenancy work in Tempo, and what’s the equivalent story in Jaeger? Tempo: multitenancy_enabled: true; every ingest and query carries X-Scope-OrgID; per-tenant overrides govern rate limits, max trace size, metrics-generator processors and block_retention. The header is unauthenticated, so an authenticating gateway fronts it. Jaeger has no native tenancy — you isolate by running instances per team or splitting ES index prefixes, with access control at the proxy layer.

Q9. A latency panel’s exemplar click says “trace not found”. Explain the failure and two fixes. Metrics (and their exemplars) were generated from 100% of spans, but tail sampling dropped that particular trace before Tempo stored it — the exemplar is a dangling pointer. Fixes: ensure tail policies keep the populations exemplars point at on anomalous buckets (all errors + all slow), or generate the exemplar-bearing metrics from the post-sampling stream (accepting slightly biased exemplar coverage of healthy traffic).

Q10. What belongs in baggage, and what must never be there? Belongs: tiny, non-sensitive routing/business context needed by many hops — tenant tier, experiment arm, synthetic-traffic flag. Never: PII, tokens, anything large — baggage propagates to every downstream service (including third parties unless stripped at egress) and leaks into logs.

Q11. Certification mapping: where does this show up? CNCF OTCA (OpenTelemetry Certified Associate) covers propagation, the SDK pipeline, sampling and Collector config directly; PCA (Prometheus Certified Associate) covers the metrics-side mechanics (histograms, remote_write, exemplars). Cloud certs (AWS DevOps Pro, Azure AZ-400) test the managed equivalents — X-Ray, Application Insights — where the same head/tail sampling and correlation concepts transfer.

Quick check

1. In traceparent: 00-4bf9…4736-00f0…02b7-01, what does the trailing 01 mean and which sampler type honours it downstream?
2. You see traces_span_metrics_calls_total and traces_spanmetrics_calls_total both in Prometheus. What happened and why is it bad?
3. Which TraceQL operator finds traces where a checkout span has a slow payments descendant, and can Jaeger’s search express the same?
4. Name the four stages of the exemplar chain that must all be enabled.
5. Why is trace_id stored in the log line (or structured metadata) rather than as a Loki stream label?

Answers

1. It’s the sampled flag (bit 0 of trace-flags). ParentBased samplers downstream honour it, which is what keeps distributed traces complete instead of partially sampled.
2. Both the Collector spanmetrics connector and Tempo’s metrics-generator are enabled — every RED number is double-counted. Disable one; keep the connector if you tail-sample so metrics see 100% of spans.
3. The descendant operator >>: { resource.service.name="checkout" } >> { resource.service.name="payments" && duration > 500ms }. Jaeger’s tag search cannot express structural (ancestor/descendant) conditions — its index is per-span, not per-tree.
4. Source emits exemplars (connector/SDK) → transport preserves them (send_exemplars on remote_write / OTLP native) → Prometheus stores them (--enable-feature=exemplar-storage) → Grafana links them (exemplarTraceIdDestinations on the Prometheus datasource).
5. Trace IDs are unique per request; as a label each would mint a one-entry stream — unbounded cardinality that destroys Loki’s index and chunk efficiency. In the line (or structured metadata) it stays filterable without minting streams.

Glossary

• Span: one timed unit of work with name, attributes, events, status; the atom of tracing.
• Trace: the tree of spans sharing one trace_id across processes.
• Span context: the propagated tuple — trace_id, span_id, trace flags, trace state.
• traceparent / tracestate / baggage: the W3C headers carrying context (and key-value baggage) between processes.
• Propagator: SDK component that injects/extracts context into carriers (W3C, B3, Jaeger, X-Ray formats).
• SpanKind: SERVER/CLIENT/PRODUCER/CONSUMER/INTERNAL — drives service graphs and latency attribution.
• Semantic conventions (semconv): OTel’s standard attribute names (http.route, db.system.name…) that make spans aggregatable.
• OTLP: the OpenTelemetry wire protocol (gRPC 4317 / HTTP-protobuf 4318) spoken by SDKs, Collectors, Tempo and Jaeger.
• Head / tail sampling: keep-decision at trace start in the SDK vs after trace completion in a gateway.
• TraceQL: Tempo’s structural query language over stored traces (filters, >>, aggregates, metrics functions).
• Metrics-generator: Tempo component deriving span metrics and service graphs from ingested spans, remote-written to Prometheus.
• Exemplar: a histogram-bucket sample carrying a trace_id — the metrics→trace hyperlink.
• Derived field: Loki datasource feature extracting a value (trace_id) from log lines and linking it to another datasource.
• X-Scope-OrgID: the tenant header used by Tempo (and Loki/Mimir) for multi-tenant ingest and query scoping.
• Service graph: the directed service-to-service edge map computed from paired CLIENT/SERVER spans.

Next steps

• Stand the backend up properly: Configure Grafana Tempo with TraceQL, Metrics-Generator, and S3 Block Storage — blocks, compaction, caching, TraceQL metrics.
• Build the pipeline tier this article assumes: Building Production OpenTelemetry Collector Pipelines: Receivers, Processors, and Tail Sampling, then scale the sampling gateway with Tail-Based Sampling at Scale with the OpenTelemetry Collector and Load-Balancing Exporter.
• Finish the correlation mesh from the metrics side: Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation and the log side: Deploy Loki in Distributed Microservices Mode with S3 Chunk Storage and Index Gateway.
• Add the fourth signal — from a slow span straight into a flame graph: Continuous Profiling in Production with eBPF: Parca, Pyroscope, and Flame Graphs.