Distributed Tracing on AWS with X-Ray: Service Maps, Segments, and ADOT on EKS

X-Ray is the only tracing backend on AWS that the managed services themselves understand. ALB stamps a trace header, API Gateway continues it, Lambda creates a segment automatically, and the X-Ray service map draws the whole call graph without you instrumenting the edges. The catch is that X-Ray speaks its own segment document format, not OTLP — so on EKS you do not point your OpenTelemetry SDKs at X-Ray directly. You run the AWS Distro for OpenTelemetry (ADOT) Collector, take OTLP in, and let the awsxray exporter translate spans into segments. Get the IAM, the sampling, and the header propagation right and you get one service graph spanning Lambda, API Gateway, and your EKS workloads; get any one wrong and you get orphaned segments, blown sampling budgets, or a map with holes where the managed services should be.

1. X-Ray’s data model: segments, subsegments, traces, and the service graph

X-Ray does not store OpenTelemetry spans. It stores segments. A segment is the work a single service did for one request — roughly an OTel server span plus everything local to that service. Inside it, subsegments record downstream calls: an outbound HTTP request, a DynamoDB query, an SQS publish. An OTel client span maps to a subsegment.

A trace is the set of all segments and subsegments that share one trace ID, and here is the first hard constraint: X-Ray trace IDs are not W3C trace IDs. An X-Ray trace ID has a fixed format — 1-{8 hex of epoch seconds}-{24 hex random} — for example 1-67c0a1f2-5e1b2a3c4d5e6f7081920a3b. The first segment’s start time is literally encoded in the ID, which is why X-Ray can shard and expire traces cheaply. The awsxray exporter and the AWS X-Ray ID generator reformat between the 32-hex W3C ID and this layout so the epoch-second prefix is preserved.

The service graph (the service map) is derived, not stored separately. X-Ray reads segment fields — name, origin, namespace, error/fault/throttle flags, subsegment name and namespace=aws — and aggregates them into nodes and edges with rolled-up latency and error statistics. You do not draw the map; you emit correctly shaped segments and the map falls out.

The annotation-versus-metadata split is load-bearing for triage. Annotations are indexed and filterable in the console and via filter expressions (annotation.tenant = "acme"). Metadata is attached to the trace but not indexed — readable on a trace you already found, but not searchable. The exporter only promotes the attributes you name to annotations, so choose them deliberately (section 6).

2. Deploying the ADOT Collector on EKS

The ADOT Collector is AWS’s supported, security-patched build of the upstream OpenTelemetry Collector with the AWS components (awsxray, awsemf, awsxrayreceiver) compiled in, from the public ECR repo public.ecr.aws/aws-observability/aws-otel-collector. Two deployment shapes exist; the choice is about where the receiver lives.

• DaemonSet (gateway-per-node). One Collector per node; pods send OTLP to the node-local Collector over the host IP. Fewer Collectors, efficient batching, the right default for EKS at scale. This is what we build here.
• Sidecar. One Collector container per pod — strong isolation and per-workload config, but multiplies Collector count and overhead. Reserve it for workloads with bespoke pipelines.

First, the receiver pipeline. Take OTLP on the standard ports and export to X-Ray:

# adot-collector-config (ConfigMap data)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch/traces:
    timeout: 1s
    send_batch_size: 50          # X-Ray PutTraceSegments takes batches; keep them modest
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

exporters:
  awsxray:
    region: eu-west-1
    indexed_attributes:          # which span attributes become searchable annotations
      - tenant.id
      - http.route
      - deployment.environment

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch/traces]
      exporters: [awsxray]
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888

The DaemonSet exposes the OTLP ports on the host so pods can reach their node-local Collector via the Kubernetes Downward API (status.hostIP):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: adot-collector
  namespace: observability
spec:
  selector:
    matchLabels: { app: adot-collector }
  template:
    metadata:
      labels: { app: adot-collector }
    spec:
      serviceAccountName: adot-collector   # IRSA-bound, see section 3
      containers:
        - name: aws-otel-collector
          image: public.ecr.aws/aws-observability/aws-otel-collector:v0.43.0
          args: ["--config=/conf/otel-config.yaml"]
          ports:
            - { name: otlp-grpc, containerPort: 4317, hostPort: 4317 }
            - { name: otlp-http, containerPort: 4318, hostPort: 4318 }
          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits:   { cpu: "1",  memory: 512Mi }
          volumeMounts:
            - { name: config, mountPath: /conf }
      volumes:
        - name: config
          configMap:
            name: adot-collector-config
            items: [{ key: otel-config.yaml, path: otel-config.yaml }]

Applications point their OTLP exporter at the node IP, injected as an env var:

# in the application Deployment's pod spec
env:
  - name: HOST_IP
    valueFrom:
      fieldRef: { fieldPath: status.hostIP }
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(HOST_IP):4318"

3. OTLP-to-X-Ray export, IAM permissions, and the awsxray exporter

The awsxray exporter calls the X-Ray API directly, so it needs credentials, and on EKS the correct mechanism is IRSA (IAM Roles for Service Accounts) or EKS Pod Identity — never long-lived keys in the pod. The Collector’s service account is bound to an IAM role granting exactly the X-Ray write and sampling-read actions. The managed policy AWSXRayDaemonWriteAccess is the standard grant; the underlying actions are worth seeing explicitly so you know what the exporter actually exercises:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords",
        "xray:GetSamplingRules",
        "xray:GetSamplingTargets",
        "xray:GetSamplingStatisticSummaries"
      ],
      "Resource": "*"
    }
  ]
}

PutTraceSegments is the write path — every batch of translated segments goes through it. The three Sampling* actions let the Collector pull centralized sampling rules from X-Ray (section 4); omit them and the Collector silently falls back to its local rule. X-Ray API actions do not support resource-level scoping, so Resource: "*" is expected — scope access through the role trust policy and the IRSA binding instead.

Bind the role to the service account with IRSA (eksctl does the OIDC plumbing in one command):

eksctl create iamserviceaccount \
  --cluster my-eks \
  --namespace observability \
  --name adot-collector \
  --attach-policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess \
  --approve --region eu-west-1

A subtle but critical detail: the awsxray exporter expects the spans it receives to carry an X-Ray-compatible trace ID. If your SDK mints standard random W3C IDs whose first 4 bytes are not a valid recent epoch timestamp, X-Ray rejects the segment because the trace ID’s time prefix sits in the distant past. The fix is the AWS X-Ray ID generator in your SDK, so root trace IDs carry the correct epoch prefix:

# Python SDK: use the X-Ray ID generator + propagator so IDs are X-Ray-shaped
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.aws import AwsXRayPropagator

provider = TracerProvider(id_generator=AwsXRayIdGenerator())
set_global_textmap(AwsXRayPropagator())

The AwsXRayPropagator reads and writes the X-Amzn-Trace-Id header — the format ALB, API Gateway, and Lambda all speak — so your EKS services stitch into segments those managed services started. Without it your services emit W3C traceparent only, the AWS edges propagate X-Amzn-Trace-Id, the two never join, and the map shows your mesh disconnected from the front door.

4. Centralized sampling rules and reservoir/fixed-rate configuration

X-Ray sampling is not a single percentage. Each rule combines a reservoir — a fixed number of traces kept per second, a sample floor even at low traffic — with a fixed rate, a percentage of everything above the reservoir. The reservoir guarantees you always have some traces from a quiet endpoint; the fixed rate scales sampling with volume on a busy one.

Define rules centrally so every Collector and SDK in the account shares one policy. That is the point of the GetSamplingRules/GetSamplingTargets API: the Collector polls X-Ray, the service hands each Collector its slice of the reservoir, and the budget is coordinated across the fleet instead of each instance keeping its own reservoir locally.

# Terraform: a default rule plus a high-priority rule for the checkout route
resource "aws_xray_sampling_rule" "default" {
  rule_name      = "Default"
  priority       = 10000          # highest number = lowest precedence
  reservoir_size = 1              # 1 trace/sec floor
  fixed_rate     = 0.05           # then 5% of the rest
  host           = "*"
  http_method    = "*"
  url_path       = "*"
  service_name   = "*"
  service_type   = "*"
  resource_arn   = "*"
  version        = 1
}

resource "aws_xray_sampling_rule" "checkout" {
  rule_name      = "checkout-high-fidelity"
  priority       = 100            # lower number = evaluated first
  reservoir_size = 5              # always keep 5 checkout traces/sec
  fixed_rate     = 0.20           # plus 20% above the reservoir
  host           = "*"
  http_method    = "POST"
  url_path       = "/checkout*"
  service_name   = "*"
  service_type   = "*"
  resource_arn   = "*"
  version        = 1
}

Rules are evaluated by priority ascending — the lowest priority number that matches wins, so put specific high-fidelity rules at low numbers and the catch-all Default at the maximum. Tell the Collector to use centralized (remote) sampling rather than its local rule:

# in the application SDK environment, or the Collector if it samples
env:
  - name: OTEL_TRACES_SAMPLER
    value: xray                   # delegate to X-Ray centralized rules

The reservoir is per-rule and per-second, coordinated by the service across all reporters. With a reservoir of 5 and three Collectors, X-Ray allocates the 5/sec budget across the three so the aggregate floor is 5, not 15. This is exactly why the GetSamplingTargets IAM action matters: without it, each Collector keeps its own local reservoir and you over-sample by the number of reporters.

5. Propagating X-Ray trace headers across ALB, API Gateway, and Lambda

The managed services use X-Amzn-Trace-Id, not W3C traceparent. The header looks like:

X-Amzn-Trace-Id: Root=1-67c0a1f2-5e1b2a3c4d5e6f7081920a3b;Parent=53995c3f42cd8ad8;Sampled=1

Root is the X-Ray trace ID, Parent is the upstream segment/subsegment ID, and Sampled is the decision bit (0, 1, or absent meaning “decide downstream”). Each AWS edge handles it differently, and the behaviour tells you where the trace starts:

• ALB. With X-Ray enabled, the ALB injects X-Amzn-Trace-Id if absent and propagates it if present. It does not appear as a segment, but it sets the root ID your first service inherits.
• API Gateway. Enable Active Tracing on the stage. API Gateway then creates its own segment (it shows up as a service-map node) and passes the trace context to the integration — Lambda, an HTTP backend, or a VPC link to your EKS ALB.
• Lambda. With Active Tracing on the function, Lambda creates the function’s segment automatically and exposes the header to your code via the _X_AMZN_TRACE_ID environment variable. In-function instrumentation adds subsegments under it.

Enabling the edges is a one-liner each:

# API Gateway stage: turn on Active Tracing (emits an API Gateway segment)
aws apigateway update-stage \
  --rest-api-id abc123 --stage-name prod \
  --patch-operations op=replace,path=/tracingEnabled,value=true

# Lambda: turn on Active Tracing (Lambda creates the function segment)
aws lambda update-function-configuration \
  --function-name checkout-handler \
  --tracing-config Mode=Active

What ties it together on EKS is the AwsXRayPropagator from section 3. When a request flows API Gateway -> EKS service, API Gateway sends X-Amzn-Trace-Id; the propagator extracts Root as the trace ID and Parent as the parent span, so your service’s segment attaches under the API Gateway node. Configure only W3C propagation and that extraction never happens — you get two disconnected traces for one request, the classic “the map shows API Gateway, then nothing” symptom.

6. Reading the service map, annotations, and trace groups for triage

The service map is the triage entrypoint. Each node is a service (or an AWS resource like a DynamoDB table); each edge carries a response-time distribution and counts of OK / error (4xx) / fault (5xx) / throttle (429). A node ringed in red is signalling faults, and the ring fill encodes the error-class proportions, so you see at a glance whether a node is throwing 5xx or being throttled.

To go from “this edge is red” to “these traces,” you filter. X-Ray filter expressions query the indexed fields:

service("checkout-api") AND fault = true
annotation.tenant = "acme" AND responsetime > 2
http.url CONTAINS "/v2/orders" AND error = true

This is where the indexed_attributes from section 2 pay off — only those attributes are searchable as annotation.<key>. A common principal-level move is to promote tenant.id so you can answer “show me the failing traces for this one customer” during an incident, which metadata cannot do.

Trace groups persist a filter expression as a named, monitored group. X-Ray continuously evaluates it, emits CloudWatch metrics for its matching trace volume and fault rate, and lets you alarm on them. Create a group for the flows that matter:

aws xray create-group \
  --group-name checkout-faults \
  --filter-expression 'service("checkout-api") AND fault = true'

Once the group exists, you can pin its sub-map in the console and CloudWatch publishes ApproximateTraceCount, ErrorRate, FaultRate, and ThrottleRate dimensioned by GroupName — turning an ad hoc filter into an SLO signal you can alert on.

7. Correlating X-Ray traces with CloudWatch Logs and metrics

A trace tells you where a request slowed; the logs from that exact span tell you why. The link is the trace ID, and the discipline is to emit it in structured logs in a field X-Ray and CloudWatch both recognise. Emit the X-Ray-format trace ID (with the 1- prefix) as a field named AWS.XRayTraceId; CloudWatch Logs Insights and the X-Ray console both use it to jump from a trace to its logs and back:

{
  "timestamp": "2026-06-08T10:32:11Z",
  "level": "error",
  "msg": "inventory reserve failed",
  "tenant.id": "acme",
  "AWS.XRayTraceId": "1-67c0a1f2-5e1b2a3c4d5e6f7081920a3b"
}

With that field present, pivot from an incident trace to every log line for the same request:

fields @timestamp, level, msg, `tenant.id`
| filter `AWS.XRayTraceId` = "1-67c0a1f2-5e1b2a3c4d5e6f7081920a3b"
| sort @timestamp asc

For metrics, X-Ray’s derived series (latency percentiles and fault rate per node, plus the trace-group metrics from section 6) live in CloudWatch dimensioned by service or group. So you build one dashboard showing service-graph fault rate, trace-group fault count, and the application’s RED metrics side by side — three pillars correlated by service name and trace ID, which is the whole point of doing this on AWS rather than bolting on a separate stack.

8. Cost controls and trace retention strategy

X-Ray pricing has two meters: traces recorded (segments and subsegments written via PutTraceSegments) and traces retrieved/scanned (segments read by the console, filter queries, and GetTraceSummaries). The dominant control on the write side is sampling — so the reservoir-plus-fixed-rate rules in section 4 are not just a fidelity dial, they are your primary cost lever. Drop the Default fixed rate from 5% to 1% and you cut recorded-trace cost on the noisy majority by 5x; keep the checkout reservoir high so the flows that matter stay fully visible.

Retention is fixed. X-Ray traces are retained for 30 days, with no configurable period — you cannot pay to keep them longer or shorten it to save money. So if you need trace data beyond 30 days (audit, capacity planning, regression baselines), export it. Two patterns:

• Periodic export via the API. A scheduled job calls GetTraceSummaries to find traces of interest and BatchGetTraces to pull the full segment documents, then writes them to S3 under a lifecycle policy — keeping only the long-term traces you actually need, cheaply.
• OTLP fan-out at the Collector. The data already flows through ADOT, so add a second exporter and a copy of every sampled trace lands in a backend whose retention you control. The X-Ray copy stays the 30-day operational view; the durable copy is yours.

# fan out: X-Ray for operations, a second backend for long-term retention
exporters:
  awsxray:
    region: eu-west-1
  otlp/longterm:
    endpoint: traces-archive.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch/traces]
      exporters: [awsxray, otlp/longterm]

The governance point: sampling controls recorded cost, uncontrolled console querying controls retrieved cost. A team that scripts GetTraceSummaries polls every few seconds across a wide window can run up the retrieval meter as fast as ingestion — so budget both, and prefer trace groups (server-side metrics) over repeated broad scans.

Enterprise scenario

A retail platform team ran checkout on EKS behind API Gateway, with two Lambda functions for payment callbacks. Active Tracing was on for API Gateway and both Lambdas, the EKS services were instrumented with the OpenTelemetry SDK, and an ADOT DaemonSet ran the awsxray exporter. The service map showed API Gateway and the two Lambdas as a clean connected graph — and a completely separate disconnected cluster for the EKS services. One request was producing two traces. During a payment incident, the on-call could see the API Gateway node throwing faults but could not follow the trace into the EKS service that actually failed: the trace ended at the gateway.

The constraint was that the front door (ALB, API Gateway, Lambda) speaks X-Amzn-Trace-Id, but the EKS services had only the default W3C TraceContext propagator. The gateway’s Root trace ID was never extracted, so each EKS service minted a fresh trace ID. Root cause and fix were one line each — the propagator:

# EKS services: extract X-Amzn-Trace-Id so segments attach under the API Gateway node
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.aws import AwsXRayPropagator
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator
from opentelemetry.sdk.trace import TracerProvider

set_global_textmap(AwsXRayPropagator())
provider = TracerProvider(id_generator=AwsXRayIdGenerator())

After the change, the map collapsed into one connected graph — API Gateway -> EKS checkout-api -> payment Lambda — with the EKS segments parented under the gateway. They also promoted tenant.id to indexed_attributes so triage could filter annotation.tenant = "acme" AND fault = true and pull only the affected customer’s broken traces. The runbook line: on AWS, a disconnected service map is almost never a sampling problem — it is a propagator mismatch between X-Amzn-Trace-Id and W3C, and the only proof it is fixed is seeing the edge actually drawn between the managed service and your workload.

Verify

# 1. Confirm the Collector authenticated to X-Ray (IRSA working, no AccessDenied).
kubectl -n observability logs ds/adot-collector | grep -iE "xray|accessdenied|signature"

# 2. Confirm segments are being written: watch the exporter's success counter.
kubectl -n observability port-forward ds/adot-collector 8888:8888 &
curl -s localhost:8888/metrics | grep -E "otelcol_exporter_sent_spans|otelcol_exporter_send_failed_spans"

# 3. Confirm the centralized sampling rules are present and being polled.
aws xray get-sampling-rules --region eu-west-1 \
  --query 'SamplingRuleRecords[].SamplingRule.{name:RuleName,prio:Priority,rate:FixedRate,res:ReservoirSize}'

# 4. Pull a recent trace summary and confirm the service graph has the expected nodes.
aws xray get-trace-summaries --region eu-west-1 \
  --start-time "$(date -u -d '10 minutes ago' +%s)" --end-time "$(date -u +%s)" \
  --query 'TraceSummaries[0].ServiceIds[].Name'

In the X-Ray console, open the service map and confirm there is a single connected graph from the front-door service (API Gateway or ALB-fronted EKS) through to your downstream Lambdas and data stores, with no orphaned cluster. Then run a filter expression on an indexed annotation (annotation.tenant = "acme") and confirm it returns traces — that proves both the connected graph and the annotation indexing are live. otelcol_exporter_send_failed_spans flat at zero confirms the IAM grant and trace-ID format are correct.

Checklist

• ADOT Collector deployed (DaemonSet on EKS), image pinned, OTLP ports exposed on the host.
• awsxray exporter configured with the correct region and explicit indexed_attributes.
• Service account bound via IRSA/Pod Identity to a role with xray:PutTraceSegments and the three Sampling* actions.
• SDK uses AwsXRayIdGenerator so root trace IDs carry a valid epoch prefix X-Ray accepts.
• SDK uses AwsXRayPropagator so X-Amzn-Trace-Id is read/written across ALB, API Gateway, and Lambda.
• Active Tracing enabled on API Gateway stages and Lambda functions that must appear as nodes.
• Centralized sampling rules defined (reservoir + fixed rate), high-fidelity flows at low priority numbers.
• OTEL_TRACES_SAMPLER=xray (or Collector remote sampling) so the reservoir is coordinated, not per-reporter.
• Structured logs carry AWS.XRayTraceId for trace-to-logs correlation in CloudWatch Logs Insights.
• Trace groups created for SLO-critical flows; alarms wired to their CloudWatch fault/throttle metrics.
• Long-term retention strategy decided (30-day X-Ray is fixed; export to S3 or a second OTLP backend if needed).