Zero-Code Auto-Instrumentation with Grafana Beyla: eBPF Traces and RED Metrics

Every platform team carries a tier of services that will never be instrumented by hand: a Rust proxy nobody owns, a Go binary whose maintainer left two reorgs ago, a vendor container you are contractually forbidden to rebuild, a Python monolith where adding an OpenTelemetry SDK means a six-month change-management fight. The classic answers all have a tax. A service mesh sees only what crosses the sidecar, doubles your data plane, and is blind to the TLS-terminated calls a process makes to a local cache. Adding an SDK means a code change, a redeploy, and an owner — three things the long tail of services does not have. Grafana Beyla takes a third route: it loads small eBPF programs into the Linux kernel, attaches them to the syscalls and library functions a process already calls, watches the request and response bytes flow past, and emits RED metrics and distributed-trace spans over OTLP. No code change, no recompile, no language SDK, no mesh.

This is an Intermediate, production-grade walk-through of how that works and where it stops working: what eBPF is and why it observes a process without modifying it; the two hook families (kprobes/tracepoints on kernel functions, uprobes on user-space library functions) and what each buys you; how it reads cleartext from an HTTPS connection by hooking SSL_read/SSL_write at the libssl boundary with no key; exactly what it captures (HTTP/1.x, HTTP/2, gRPC, SQL, Kafka, Redis; server and client RED and spans); how it exports to an OpenTelemetry Collector or a native Prometheus endpoint; the single most misunderstood thing about it — the real limits of automatic distributed-trace context propagation across non-Go services; how to deploy it on Kubernetes as a DaemonSet or a sidecar with the precise capabilities and pod-security settings; how it compares to the OTel SDK and the OTel eBPF operator; what it costs; and how to lock it down.

By the end you will be able to put RED metrics and per-request server spans on a service you cannot modify, in any language, in under an hour — and you will know precisely what Beyla does not give you for free, so you deploy it with correct expectations instead of filing a confused ticket three days later.

What problem this solves

The pain is the long tail of un-instrumented services and the cost of the alternatives. In any estate older than about two years, a meaningful fraction of running services have no telemetry — no request rate, no error rate, no latency, no traces. They are black boxes, and when one is on the critical path of an incident you are flying blind through the most important hop. The reasons are organisational, not technical: no owner, no build pipeline you control, a runtime (Rust, C++, an old Go binary) where wiring an SDK is real work, or a vendor’s sealed container.

The conventional fixes each break down. Manual instrumentation (an OTel SDK plus code) is the gold standard for depth but needs a code change and an owner per service — exactly what the long tail lacks. A service mesh gives L7 metrics for free, but only for traffic that crosses the sidecar proxy; it is blind to in-process calls, doubles the data plane, and the sidecar overhead is not free on a CPU-bound fleet. Wire sniffing (tcpdump-style capture) sees network bytes but is useless the moment the service speaks TLS — it captures ciphertext.

Beyla closes this gap from underneath. Because eBPF runs in the kernel and attaches to functions the process already calls, it needs nothing from the application: no SDK, no recompile, no redeploy, no mesh, no owner. It produces accurate server- and client-side RED metrics and per-request spans for HTTP/1.x, HTTP/2, gRPC and database calls, in any language, and — the part that beats wire sniffing — it reads cleartext from encrypted connections by hooking the TLS library after decryption, never touching a key.

Who hits this: platform and SRE teams running polyglot Kubernetes estates with a tail of unowned or unmodifiable services; teams wanting fleet-wide RED as a baseline before (or instead of) a per-service SDK rollout; anyone needing telemetry for a vendor container or legacy binary with zero application change. Who should not reach for it first: a greenfield service you own — there the SDK gives richer, business-level telemetry the kernel can never see.

To frame the whole field before the deep dive, here is what each approach can and cannot do — Beyla’s place is the “no code change, sees TLS, breadth not depth” corner:

Learning objectives

By the end of this article you can:

• Explain what eBPF is, why it can observe a process without modifying it, and the safety model (the verifier, the JIT, the fixed hook points) that makes loading code into the kernel survivable.
• Distinguish Beyla’s two hook families — kprobes/tracepoints on kernel socket functions and uprobes on user-space library functions — and state precisely what each one captures.
• Describe how Beyla reads cleartext out of an HTTPS or Go-TLS connection by hooking the encryption library at the buffer boundary, without ever holding a private key.
• Enumerate exactly what Beyla captures: HTTP/1.x, HTTP/2, gRPC, SQL, Redis and Kafka; server and client RED metrics; server and client spans; and the OTel semantic-convention attributes it emits.
• State the real limits of automatic distributed-trace context propagation — why it is clean for Go and within a Beyla-instrumented estate, and why a non-Go hop generally breaks the chain — and how to bridge it.
• Deploy Beyla on Kubernetes as a DaemonSet (fleet-wide) or a sidecar (per-service), with the exact Linux capabilities, hostPID/shareProcessNamespace settings, RBAC, and discovery configuration.
• Export Beyla telemetry to an OpenTelemetry Collector over OTLP, or expose a native Prometheus scrape endpoint, and reason about sampling and cardinality at the source.
• Choose correctly between Beyla, the OTel SDK, and a mesh for a given service — and combine Beyla (breadth) with the SDK (depth) without double-counting the entry span.
• Quantify the overhead and the security blast radius, and lock Beyla down to least privilege.

Prerequisites & where this fits

You should be comfortable with Kubernetes workloads (Deployments, DaemonSets, pods, namespaces, labels, RBAC) and kubectl, and you should understand the three observability pillars — metrics, logs, traces — at the level of “a span is a timed operation with a parent, and a trace is a tree of spans sharing a trace ID.” Familiarity with OpenTelemetry concepts (OTLP, the Collector, semantic conventions, the W3C traceparent header) is assumed; if any of that is fuzzy, read Distributed Tracing End-to-End: Context Propagation, Tempo, and Correlating Traces with Metrics and Logs first — context propagation is the exact concept that bounds what Beyla can do automatically. You do not need to know eBPF; this article builds the model you need.

This sits in the Observability track as the zero-code on-ramp to instrumentation. It is upstream of the deeper SDK and pipeline articles and downstream of nothing — it is often the first telemetry a service ever gets. It pairs tightly with Building Production OpenTelemetry Collector Pipelines: Receivers, Processors, and Tail Sampling, because Beyla emits native OTLP and the Collector is its destination, and with Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code, because the RED metrics Beyla produces are exactly what those dashboards consume. Where it crosses into network-flow visibility, Network Observability with Cilium Hubble: Flow Logs, L7 Visibility, and Service Maps is the eBPF sibling that watches connections rather than producing application RED.

A quick map of who owns what when you run Beyla, so you scope it correctly:

Core concepts

Six mental models make every later decision obvious.

eBPF is sandboxed code that runs in the kernel on events. eBPF (extended Berkeley Packet Filter) lets you load a small program into the running Linux kernel and attach it to a hook — a kernel function entry, a tracepoint, a user-space function, a network event. When the hook fires, the program runs in kernel context, reads arguments and memory, writes to maps (shared kernel/user-space data structures), and returns. It cannot crash the kernel, loop forever, or read arbitrary memory: before load the verifier statically proves the program terminates and touches only permitted memory, then a JIT compiles it to native code — which is why loading code into the kernel is sane and not a recipe for panics. Beyla is, at its core, a set of these programs plus a user-space agent that drains their maps and turns raw events into telemetry.

Beyla hooks two kinds of points. The first is kprobes and tracepoints on kernel functions — chiefly the socket send/receive path (tcp_sendmsg, tcp_recvmsg, the accept/connect/close calls) — so it sees request and response bytes crossing any TCP connection, parses the HTTP/1.x request line or the HTTP/2 frames, and times the round trip. This is language-agnostic and needs nothing from the binary. The second is uprobes on user-space library functions — it inspects a target binary’s symbol table and attaches to specific functions at specific offsets. The uprobe path is what unlocks two things the socket path cannot do alone: reading TLS cleartext, and following a request across Go’s goroutine scheduler.

TLS is observed at the library boundary, not on the wire. Wire bytes are useless if the service speaks HTTPS — they are ciphertext. So Beyla attaches uprobes to SSL_read and SSL_write inside the process’s OpenSSL (libssl), and to the equivalents in Go’s crypto/tls. Those functions handle the buffer after decryption on read and before encryption on write — exactly the moment the application itself sees cleartext. Beyla reads the same plaintext the app reads, at the same instant, without ever touching a private key, performing a MITM, or terminating TLS. This is the decisive advantage over packet capture and over a mesh that only sees mTLS at the proxy.

Go is a special case, by necessity. The Go runtime multiplexes goroutines onto OS threads with its own scheduler, so one logical request can hop across threads and the generic socket view cannot reliably stitch it into a single server span. Beyla detects Go binaries (via ELF build info) and attaches uprobes to specific Go runtime, net/http and gRPC symbols, so it follows a request across goroutines, reconstructs the server span correctly, and — uniquely — reads and writes the traceparent header inside the running Go process. Every other language (Python, Node, Java, Ruby, .NET, Rust, C++) is served by the generic socket and libssl layer.

RED is what Beyla produces, and what your dashboards consume. The two telemetry pillars Beyla emits are RED metrics — request Rate, Error rate, and request Duration (as a histogram) — and spans. RED is the standard service-health signal: how much traffic, how many failures, how slow, sliced by route, method and status. Beyla follows OTel semantic conventions, so the series are http.server.request.duration, http.client.request.duration, the rpc.* equivalents for gRPC, and database/messaging conventions for SQL and Kafka. These feed straight into a RED dashboard or an SLO with no translation.

Context propagation is the line between “free” and “needs help”. A distributed trace requires the W3C traceparent header to be injected into each outbound request and read from each inbound one, so the next hop continues the same trace ID. Beyla does this end-to-end for Go (its uprobes rewrite headers in-process) and within a fully Beyla-instrumented estate (its own propagation). For a non-Go service it observes the request and produces correct local spans but generally cannot rewrite outbound headers from the kernel — so the chain breaks there unless an SDK or Beyla-to-Beyla propagation bridges it. This is the single most common source of “why is my trace fragmented?” tickets.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the model side by side:

How eBPF lets Beyla see without changing code

Start with the mechanism, because everything else follows from it. A traditional profiler or APM agent changes the thing it observes — it links a library, rewrites bytecode, or wraps the process. eBPF does not. You load a program into the kernel, attach it to a function the system already calls millions of times a second, and it runs in addition to the normal code path. The observed process is unaware and unmodified; it simply keeps calling tcp_sendmsg and SSL_read as it always did, and Beyla’s probes ride along.

What makes this safe enough to run in production on every node is the kernel’s contract. When Beyla loads a probe, the verifier walks every possible path through the program and rejects it unless it can prove the program terminates (bounded loops only), never dereferences an invalid pointer, and only reads memory it is permitted to. A rejected program never loads — there is no “it crashed the kernel” failure mode for a verified program. Accepted programs are JIT-compiled to native instructions, so the overhead of the probe firing is small. Data flows from kernel to user space through maps (ring buffers, hash maps), which Beyla’s agent drains and turns into spans and metrics.

The two hook families differ in what they can see and what they require. Knowing which family produces which signal removes the mystery from “why does HTTPS work but cross-service tracing doesn’t”:

The practical consequence is a clean split. The socket layer (kprobes) gives you breadth — RED and spans for any process speaking plaintext TCP, in any language. The libssl uprobe extends that breadth to encrypted traffic without keys. The Go uprobes add depth that only Go can have through eBPF: correct request tracking across goroutines and true in-process header propagation. Everything you read later about “automatic for Go, limited for others” traces directly back to which of these hooks is available for a given binary.

A few hard requirements fall out of the mechanism. eBPF needs a reasonably modern kernel — 5.8+ is the comfortable floor for Beyla’s feature set, and BTF (BPF Type Format) should be available so probes resolve kernel structures portably (CO-RE: “compile once, run everywhere”); most managed Kubernetes node images today ship both. The agent needs elevated Linux capabilities to load programs and read other processes’ memory (detailed under deployment). And in containers it needs to see the target — sharing the node’s PID view (hostPID) for a DaemonSet, or the pod’s process namespace (shareProcessNamespace) for a sidecar.

What Beyla actually captures

Be precise about the surface area, because “it instruments everything” is both the promise and the trap. Beyla produces application-level telemetry for the protocols it can parse out of the byte stream, and it produces two kinds of signal — metrics and spans — on both the server side (requests this process received) and the client side (requests this process made).

The protocols it understands, and what it extracts from each:

The two telemetry pillars, side by side — what each is for and what it costs you downstream:

What Beyla cannot capture is as important as what it can, because it sets the boundary with the SDK. The kernel sees bytes and timings; it does not see your application’s intent. So Beyla gives you the entry and exit of a request and the calls it makes, but never the inside — no custom business spans (settle_payment, score_fraud), no application-level attributes (payment.provider, tenant.id, cart.item_count), no log correlation you did not wire, and no view of work that never crosses a socket or a TLS boundary (pure in-memory computation). That interior detail is exactly what the SDK adds, which is why the mature pattern is Beyla for breadth and the SDK for depth — covered later.

The headline metric series, so you know what to query and graph:

Exporting to OpenTelemetry and Prometheus

Beyla speaks native OTLP, so the destination is an OpenTelemetry Collector, not Beyla-specific plumbing. This is a deliberate design choice and a strength: nothing downstream is “Beyla-aware.” You point Beyla’s OTLP exporter at a Collector’s OTLP receiver, and from there metrics flow to Prometheus/Mimir and traces to Tempo/Jaeger through ordinary pipelines. Beyla can export metrics and traces to the same endpoint or to different ones, and it honours the standard OTEL_EXPORTER_OTLP_* environment variables in addition to its own YAML config.

The two pillars enabled explicitly in Beyla’s config — note that endpoint and protocol are set per signal:

# beyla-config.yml — export RED metrics and traces to a Collector over OTLP/gRPC
otel_metrics_export:
  endpoint: http://otel-collector.observability:4317
  protocol: grpc
otel_traces_export:
  endpoint: http://otel-collector.observability:4317
  protocol: grpc

# Enrich every metric and span with Kubernetes metadata (namespace, pod, deployment, node).
attributes:
  kubernetes:
    enable: true

# Local span printing is for debugging only — leave disabled in production.
trace_printer: disabled

On the Collector side this is an ordinary OTLP receiver fanning out to a metrics backend and a traces backend. Nothing in this pipeline knows or cares that Beyla produced the data:

# otel-collector.yaml — receive Beyla's OTLP, route metrics to Mimir and traces to Tempo
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch: {}
exporters:
  otlphttp/tempo:
    endpoint: http://tempo-distributor.observability:4318
  prometheusremotewrite:
    endpoint: http://mimir-nginx.observability/api/v1/push
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]

If your metrics path is already pull-based, Beyla can expose a native Prometheus scrape endpoint instead of (or alongside) OTLP — handy when you only want to push traces but pull metrics into an existing Prometheus:

# beyla-config.yml — expose a /metrics endpoint for Prometheus to scrape...
prometheus_export:
  port: 9090
  path: /metrics
# ...while still pushing traces over OTLP.
otel_traces_export:
  endpoint: http://otel-collector.observability:4317
  protocol: grpc

The export-path decision, laid out — most teams push OTLP to a Collector, but the pull mode is a clean fit for existing Prometheus estates:

Control telemetry volume at the source rather than shipping everything and dropping it downstream. Beyla honours the standard OTel sampler variables, so you can sample 100% at the agent and let a Collector do tail sampling centrally, or thin at the source if you have no central sampling:

# Sample everything at Beyla; decide centrally (e.g. tail-based) in the Collector.
export OTEL_TRACES_SAMPLER=parentbased_always_on

# OR thin at the source when the Collector is not doing tail sampling:
# export OTEL_TRACES_SAMPLER=parentbased_traceidratio
# export OTEL_TRACES_SAMPLER_ARG=0.05

For the trade-offs between head sampling at the agent and tail sampling in the Collector — and why tail sampling needs the load-balancing exporter to keep a trace’s spans together — see Tail-Based Sampling at Scale with the OpenTelemetry Collector and Load-Balancing Exporter. Metrics are not sampled the same way; you control their volume by collapsing high-cardinality labels (routes) at the source, covered next.

Service identity, Kubernetes enrichment, and route cardinality

Three configuration concerns separate a usable Beyla deployment from a useless one: every service must have a real name (not unknown_service), every signal should carry Kubernetes metadata (so it joins your dashboards and routing), and high-cardinality URL paths must be collapsed into low-cardinality routes (or your metrics bill explodes).

Service name resolution follows a priority order. An explicit OTEL_SERVICE_NAME/BEYLA_SERVICE_NAME wins; failing that, in Kubernetes Beyla derives the name from the workload (Deployment/StatefulSet/DaemonSet) that owns the pod; failing that, it falls back to the executable name. For anything you care about, set it explicitly or rely on the Kubernetes decorator — never ship data labelled unknown_service:

Kubernetes enrichment decorates every metric and span with k8s.namespace.name, k8s.pod.name, k8s.deployment.name, and node — the attributes your dashboards and routing depend on. Enabling it requires Beyla to query the Kubernetes API to map PIDs to pods, which needs RBAC:

# beyla-config.yml — turn on k8s metadata enrichment
attributes:
  kubernetes:
    enable: true
    cluster_name: prod-eu-west-1   # otherwise inferred where possible

# RBAC: Beyla queries the API to map PIDs → pods/workloads
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: beyla
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "daemonsets", "statefulsets"]
    verbs: ["get", "list", "watch"]

Route cardinality is where teams melt their metrics backend. Left alone, /orders/12345 and /orders/67890 are distinct http.route values, and the label set grows without bound — every order ID becomes a new series. Give Beyla route patterns so it collapses them into one low-cardinality route, and set unmatched: heuristic as the safety net for anything you did not enumerate:

# beyla-config.yml — collapse high-cardinality URL paths into stable routes
routes:
  patterns:
    - /orders/{id}
    - /users/{id}/cart
  unmatched: heuristic   # auto-wildcard numeric/UUID-like segments not matched above

The unmatched options and what each does to your series count — heuristic is almost always the right default for a public API:

On a public-facing API, unmatched: heuristic is the difference between a few hundred route series and a few hundred thousand. If you want to go deeper on controlling series explosions — relabeling, limits, and the cost governance around them — Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus is the companion piece.

TLS decryption and the context-propagation limit

This section separates people who deploy Beyla successfully from people who file confused tickets. Two facts, one good and one constraining.

TLS is handled, and handled well. Because Beyla hooks SSL_read/SSL_write (and Go’s crypto/tls) at the library boundary, HTTPS services produce the same clean RED data and spans as plaintext ones. There are no keys to manage, no MITM, no certificate juggling, no mesh — Beyla simply reads the cleartext buffer at the instant the application does. This is the single biggest advantage over wire-sniffing approaches, which see only ciphertext, and over a mesh, which only sees mTLS at the proxy and is blind to in-pod TLS calls. The one requirement is that Beyla can find and attach to the TLS library; for dynamically-linked OpenSSL and Go this is routine, and statically-linked exotic TLS stacks are the rare edge case.

Automatic cross-service trace propagation is the real limit, and it is asymmetric by language. A distributed trace lives or dies on the traceparent header being propagated hop to hop. Here is the honest matrix of when Beyla stitches a trace automatically and when it does not:

For a non-Go service, Beyla produces correct local server and client spans — you still get per-service RED and per-hop latency — but those spans do not automatically link into one trace across the hop, because Beyla generally cannot rewrite the outbound headers from the kernel the way it can inside a Go process. The supported way to bridge this without code changes is Beyla’s own context propagation between Beyla-instrumented services; enable it deliberately, knowing it is strongest within a fully Beyla-instrumented set:

# beyla-config.yml — stitch spans across Beyla-instrumented services
ebpf:
  context_propagation: all   # 'headers' for HTTP only; 'ip' as an L4 fallback

The propagation modes and their reach:

The honest mental model to carry away: Beyla gives you excellent per-service RED and per-hop spans for free, and automatic cross-service traces that are cleanest among Go services or within a fully Beyla-instrumented set. Where a request crosses into a service that already speaks W3C Trace Context via an SDK, you want those two worlds to interoperate — which is the next section.

Combining Beyla breadth with SDK depth

Beyla and the OpenTelemetry SDK compose rather than compete. The pattern in mature estates is Beyla for breadth, SDK for depth: Beyla blankets every service in RED metrics and server spans, and the few services that need business-level detail also run the SDK to add child spans, baggage, and attributes the kernel can never see. The key to producing one trace across both is shared context.

Because Beyla emits and (for Go / within its instrumented set) honours the W3C traceparent, a downstream SDK-instrumented service finds the header on the inbound request and continues the trace rather than starting a new one — as long as both ends use the W3C propagator, which is the OTel default:

# On the SDK-instrumented service, keep the default W3C propagators so it
# continues a trace context that Beyla (or another SDK) started.
export OTEL_PROPAGATORS=tracecontext,baggage

Inside that service, the manual span you add nests under the active server span:

from opentelemetry import trace

tracer = trace.get_tracer("checkout")

def settle_payment(order):
    # Automatically becomes a child of the active server span (Beyla- or SDK-created).
    with tracer.start_as_current_span("settle_payment") as span:
        span.set_attribute("payment.provider", order.provider)
        span.set_attribute("payment.amount_cents", order.amount_cents)
        return _charge(order)

The one rule that prevents the most common composition bug is never let two sources emit the entry span for the same process. If Beyla and an SDK auto-instrumentation agent both run on the same service and both emit the server span, you get two spans for one request and double-counted RED. Pick exactly one source for the entry span per service — usually Beyla for the long tail, the SDK for the few services that already have it — and use the other only to add detail.

The division of labour, and how to wire each side so they cooperate:

If your owned services are JVM-based, the SDK side of this story — auto-instrumentation, context propagation, and adding custom spans — is covered in OpenTelemetry for Java Services: Auto-Instrumentation, Context Propagation, and Custom Spans.

Beyla vs the OTel SDK vs other eBPF tooling

Choosing the right instrument per service is the difference between clean telemetry and a mess of duplicate spans and gaps. Beyla is not a replacement for the SDK; it is a different tool with a different sweet spot, and there is also a second eBPF auto-instrumentation effort in the OTel ecosystem that overlaps with Beyla.

Beyla against the SDK, head to head:

Beyla against the other eBPF instrumentation in the ecosystem — they solve the same problem and the choice is mostly about which platform you are standing on:

Two of those are complementary, not alternatives, and worth naming so you do not conflate signals. Network-flow observability answers “who is talking to whom and is it allowed” rather than “what is this service’s request rate and latency” — that is Network Observability with Cilium Hubble: Flow Logs, L7 Visibility, and Service Maps. Continuous profiling answers “where is this service spending CPU” — that is Continuous Profiling in Production with eBPF: Parca, Pyroscope, and Flame Graphs. All three are eBPF; they produce different pillars and you typically run more than one.

The decision rule, distilled to one table — match the service to the instrument:

Architecture at a glance

Beyla’s architecture is best understood as a flow from the kernel up to your backend, with the observed application sitting entirely outside the instrumentation path. Picture three planes stacked vertically on a single Kubernetes node, with the telemetry backend off to one side in the cluster.

At the bottom is the kernel plane. The Linux kernel on the node runs the normal socket and TLS code paths every process uses — tcp_sendmsg/tcp_recvmsg on the socket layer, SSL_read/SSL_write inside each process’s libssl, and the Go runtime’s TLS and HTTP functions inside Go binaries. Beyla’s eBPF programs are attached here as kprobes on the kernel socket functions and uprobes on the user-space library functions. They fire whenever the observed process sends or receives bytes, read the request/response data (cleartext, even for HTTPS, because the uprobe sits after decryption), capture timing and metadata, and write events into eBPF maps — the kernel-to-user-space conduit. Crucially, nothing about the application changed: the app keeps calling the same functions, unaware that probes ride along.

In the middle is the Beyla agent plane — Beyla’s user-space process, running as a DaemonSet pod (one per node) or a sidecar (one per app pod). It drains the eBPF maps, reassembles the raw byte events into protocol-aware records (parsing HTTP/1.x request lines, HTTP/2 and gRPC frames, SQL and Redis and Kafka wire formats), and turns them into two things: RED metrics (rate, errors, duration histograms by route/method/status) and spans (one server span per inbound request, client spans for outbound calls). Here it also applies the discovery rules that decide which processes to watch, resolves each process to a service name, and — querying the Kubernetes API — decorates every record with k8s.namespace.name, k8s.pod.name, k8s.deployment.name and node. For Go processes it additionally tracks requests across goroutines and reads/writes the traceparent header so Go-to-Go chains stitch into one trace.

At the top is the export plane, and off to the side the backend. Beyla’s OTLP exporter pushes metrics and traces to an OpenTelemetry Collector (or exposes a Prometheus /metrics endpoint for a scrape). The Collector — entirely unaware that Beyla produced the data — fans the metrics to Prometheus/Mimir and the traces to Tempo/Jaeger, where your RED dashboards and trace search live. Follow the path end to end and the whole system reads left-to-right from a byte on a socket to a panel on a dashboard: kernel hook → eBPF map → Beyla agent (parse, name, enrich) → OTLP → Collector → Prometheus + Tempo → Grafana. The observed application is never on that path; it sits beside it, untouched, which is the entire point of the design.

Real-world scenario

Saffron Pay, a fictional payments platform, ran a polyglot estate on a 40-node EKS cluster: the customer-facing APIs were Go, instrumented beautifully with the OTel Go SDK into existing Tempo and Mimir, but the fraud-scoring service was a Python monolith and the ledger was a vendor-supplied JVM container the team was contractually forbidden to modify or rebuild. The Go traces were rich — until the instant a request crossed into Python or the JVM black box, where every trace went dark, and the vendor container had no telemetry at all: no request rate, no error rate, no latency. During a payment-failure incident, the on-call team could see the Go front end was healthy and the trace ended at “calls fraud-scoring” with nothing beyond — a forty-minute blind spot on the exact hop that mattered. A service mesh had been floated and rejected twice: it would have doubled the data plane on a CPU-bound cluster and still missed the TLS-terminated, in-pod calls the fraud service made to a local Redis cache.

They deployed Beyla as a DaemonSet scoped to the payments and fraud namespaces, with Kubernetes enrichment and route patterns, in an afternoon. Two outcomes landed almost immediately. The unmodifiable JVM vendor container went from zero signal to full RED metrics and per-request server spans — including its HTTPS calls to the payment provider, decrypted at the libssl boundary with no keys handed over and no vendor ticket. And the Python fraud service, previously a black hole, now showed its request rate, its p95 latency, and — critically — client spans for its Redis and Postgres calls, so the team could finally see that fraud-scoring p99 latency spiked because a Redis call was timing out under load. The route patterns mattered: the fraud API embedded a transaction ID in the path (/score/{txn}), and without unmatched: heuristic that one endpoint would have produced hundreds of thousands of http.route series and blown the Mimir ingestion budget; with it, a few dozen.

The one deliberate engineering decision was avoiding double entry spans. The Go APIs already emitted entry spans via their SDK, so for the go-apis namespace they ran Beyla in metrics-only mode — RED for the fleet-wide baseline — and let the SDK own traces, while the Python and JVM namespaces used Beyla for both metrics and traces. Because both Beyla and the SDK spoke W3C traceparent with the default propagator, the Go entry spans and the Beyla-observed downstream hops shared a trace ID where the chain was Go-originated; where it crossed into Python they accepted per-hop RED plus local spans, because automatic cross-service propagation does not span a non-Go hop. The split was a few lines of config:

# beyla-config.yml on the go-apis node selection: RED metrics only,
# let the existing OTel SDK own the spans to avoid duplicate entry spans.
discovery:
  instrument:
    - k8s_namespace: go-apis
otel_metrics_export:
  endpoint: http://otel-collector.observability:4317
  protocol: grpc
# (otel_traces_export intentionally omitted here)

Total time to first useful dashboard for the previously-invisible vendor container: under a day, with no application change, no vendor ticket, and no mesh. The lesson the team wrote on the wall: “You do not need to own a service to observe it — you need to get under it.” Measured eBPF overhead settled at low single-digit percent CPU per node, and the fraud-Redis timeout the new client spans surfaced — invisible for over a year — was finally root-caused and fixed.

Advantages and disadvantages

The eBPF-from-underneath model both enables observing the un-ownable and constrains what it can see. Weigh it honestly:

The model is right for the long tail of services you cannot or will not instrument by hand, and for getting a fleet-wide RED baseline fast before any per-service SDK work. It bites hardest when teams expect SDK-grade traces (full cross-language stitching, business spans) from a kernel-level tool, when discovery is left too broad and overhead/cardinality climb, or on locked-down clusters where granting eBPF capabilities is a fight. Every disadvantage is manageable — scope discovery, set route patterns, pick one entry-span owner, and grant least-privilege capabilities — but only if you know they exist, which is the point of deploying it with eyes open.

Hands-on lab

Deploy Beyla as a DaemonSet on a Kubernetes cluster, instrument a sample HTTP service with zero code change, and confirm RED metrics and spans flowing. Works on any cluster with kernel 5.8+ nodes (kind, minikube, or a managed cluster); we tear everything down at the end. Run from a machine with kubectl pointed at the cluster.

Step 1 — Create a namespace and deploy a sample target (a plain HTTP service, uninstrumented).

kubectl create namespace demo
kubectl -n demo create deployment httpbin --image=mccutchen/go-httpbin --port=8080
kubectl -n demo expose deployment httpbin --port=8080

Expected: a httpbin Deployment and Service. This is the service we will instrument without touching it — it has no SDK, no telemetry.

Step 2 — Create the Beyla config as a ConfigMap. Scope discovery to the demo namespace, enable Kubernetes enrichment and route heuristics, and (for the lab) print spans to stdout instead of needing a backend.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: beyla-config
  namespace: demo
data:
  beyla-config.yml: |
    discovery:
      instrument:
        - k8s_namespace: demo
    attributes:
      kubernetes:
        enable: true
    routes:
      unmatched: heuristic
    # Lab only: print spans to stdout so we can see them without a backend.
    trace_printer: text
    # Expose a Prometheus endpoint so we can curl RED metrics.
    prometheus_export:
      port: 9090
      path: /metrics
EOF

Step 3 — Grant RBAC so Beyla can map PIDs to pods.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata: { name: beyla, namespace: demo }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: beyla }
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "daemonsets", "statefulsets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: beyla }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: beyla }
subjects:
  - { kind: ServiceAccount, name: beyla, namespace: demo }
EOF

Step 4 — Deploy Beyla as a DaemonSet with hostPID and the eBPF capabilities, mounting the config.

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata: { name: beyla, namespace: demo }
spec:
  selector: { matchLabels: { app: beyla } }
  template:
    metadata: { labels: { app: beyla } }
    spec:
      serviceAccountName: beyla
      hostPID: true                       # see processes across the node
      containers:
        - name: beyla
          image: grafana/beyla:2.6.0      # pin; never float on :latest
          securityContext:
            runAsUser: 0
            readOnlyRootFilesystem: true
            capabilities:
              add: ["BPF","PERFMON","SYS_PTRACE","NET_RAW","CHECKPOINT_RESTORE","DAC_READ_SEARCH"]
          env:
            - { name: BEYLA_CONFIG_PATH, value: /config/beyla-config.yml }
          ports:
            - { containerPort: 9090, name: metrics }
          volumeMounts:
            - { name: config, mountPath: /config }
      volumes:
        - { name: config, configMap: { name: beyla-config } }
EOF
kubectl -n demo rollout status ds/beyla

Expected: the DaemonSet reaches Ready. If a pod crash-loops, see Step 7’s troubleshooting (capabilities or kernel are the usual cause).

Step 5 — Generate traffic against the target.

kubectl -n demo run load --image=curlimages/curl --restart=Never --command -- \
  sh -c 'for i in $(seq 1 200); do curl -s httpbin:8080/status/200 >/dev/null; curl -s httpbin:8080/status/500 >/dev/null; curl -s httpbin:8080/get >/dev/null; sleep 0.2; done'

Step 6 — Confirm RED metrics and spans. First the spans, in Beyla’s logs:

kubectl -n demo logs ds/beyla | grep -i "GET /status\|GET /get" | head
# You should see one line per observed request: method, route, status, duration.

Then the RED metrics, by port-forwarding Beyla’s Prometheus endpoint:

kubectl -n demo port-forward ds/beyla 9090:9090 &
sleep 2
curl -s localhost:9090/metrics | grep http_server_request_duration_seconds_count | head
# Series carry http_route, http_response_status_code, service_name, and k8s_* labels.

You should see request counts split by route and status — including the 500s — with service_name="httpbin" and k8s_namespace_name="demo". That is RED on a service you never modified.

Validation checklist. You deployed a service with no telemetry, attached Beyla from outside via eBPF, and got per-route RED metrics and per-request spans — all without a single line of application code. The steps mapped to what each proves:

Cleanup.

kubectl delete namespace demo
kubectl delete clusterrole beyla
kubectl delete clusterrolebinding beyla

Note on managed clusters. On a hardened cluster with Pod Security Admission enforcing restricted, the Beyla DaemonSet needs the demo namespace labelled to permit hostPID and the added capabilities (e.g. pod-security.kubernetes.io/enforce=privileged on that namespace, or a dedicated namespace). That permission is exactly the security trade-off discussed below.

Common mistakes & troubleshooting

This is the part you bookmark. First a scannable table, then the full reasoning for the entries that bite hardest.

The expanded form for the entries that cost the most time:

1. Beyla pod CrashLoopBackOff immediately on start. Cause: missing eBPF capabilities, a kernel below 5.8, or no BTF. Confirm: logs show operation not permitted / failed to load BPF program; check the node’s uname -r. Fix: add the capability set (BPF, PERFMON, SYS_PTRACE, NET_RAW, CHECKPOINT_RESTORE, DAC_READ_SEARCH) or run privileged; use node images with kernel 5.8+ and BTF; on PSA-restricted namespaces grant a privileged enforcement label.

2. Beyla is healthy but produces no telemetry. Cause: discovery.instrument matches nothing — wrong namespace, a label no pod carries, or an exe_path regex matching no binary. Confirm: kubectl logs ds/beyla | grep -i "found process" returns nothing. Fix: correct the selector; in DaemonSet mode prefer k8s_namespace or k8s_pod_labels, and verify the pods actually carry that label.

3. All services show up as unknown_service. Cause: no explicit name and enrichment off, so Beyla falls back to the executable name. Confirm: metrics carry service_name="unknown_service:<exe>". Fix: set OTEL_SERVICE_NAME (sidecar) or enable attributes.kubernetes.enable: true (DaemonSet), and apply the RBAC so the API lookup succeeds.

4. The metrics backend cardinality alarms or OOMs after enabling Beyla. Cause: URL paths with IDs (/orders/12345) emitted as distinct http.route values — unbounded series. Confirm: the series count for http_server_request_duration_seconds_count is in the hundreds of thousands. Fix: add routes.patterns and set unmatched: heuristic so ID-bearing paths are auto-wildcarded — the single most important production setting on a public API.

6. A trace fragments the moment it crosses a particular service. Cause: that hop is non-Go and uninstrumented, so Beyla cannot rewrite the outbound traceparent from the kernel — a new trace ID starts downstream. Confirm: in Tempo/Jaeger the trace ID before the hop differs from the one after, though both span sets are individually correct. Fix: add an OTel SDK at that hop (it continues the W3C context), or run Beyla everywhere on the path with ebpf.context_propagation: all; accept that a pure non-Go, non-Beyla hop gives per-hop RED only.

7. Two server spans per request and doubled RED. Cause: Beyla and an SDK agent both run on the process and both emit the entry span. Confirm: one request shows two root-level server spans; RED counts roughly double the real rate. Fix: choose one owner per service — where the SDK owns traces, run Beyla metrics-only (omit otel_traces_export); reserve Beyla-as-entry-span for the long tail.

8. Beyla’s own CPU/memory climbs on busy nodes. Cause: the discovery selector is too broad, so Beyla instruments system pods, the kubelet, and high-throughput infrastructure. Confirm: kubectl top pod shows Beyla high; logs show many unexpected attached processes. Fix: scope discovery.instrument to the namespaces and labels you need — overhead scales with instrumented throughput, not cluster size.

Best practices

• Pin the Beyla image to a digest, never :latest. This is privileged, kernel-adjacent software; a surprise version bump is a kernel-compatibility and security risk. Pin and roll forward deliberately.
• Scope discovery tightly. Match real namespaces and pod labels, not the whole node. Broad selectors are the root cause of both overhead and cardinality — instrument what you need, opt others in by label.
• Set route patterns + unmatched: heuristic from day one on anything with IDs in the URL. It is the difference between hundreds and hundreds of thousands of series, and the cheapest disaster you can prevent.
• Always give services a real name. Explicit OTEL_SERVICE_NAME in sidecar mode, Kubernetes enrichment in DaemonSet mode. Never ship unknown_service.
• Pick exactly one entry-span owner per service. Beyla for the long tail, the SDK for already-instrumented services. Run the other metrics-only to avoid double-counting.
• Enable Kubernetes enrichment with matching RBAC. The k8s_* labels are what join Beyla’s data to your dashboards, routing, and SLOs; without them the data is hard to use.
• Prefer DaemonSet for fleet coverage, sidecar for hard isolation. DaemonSet is one deploy and fleet-wide RED; reserve sidecars for services that need strict per-pod ownership or where node-wide privileges are disallowed.
• Grant least-privilege capabilities, not blanket privileged, where your platform allows it. The fine-grained capability set is auditable and smaller blast radius than privileged: true.
• Keep trace_printer disabled in production. It is a local-debugging aid; leaving it on adds log volume and noise.
• Sample at the source or tail-sample centrally — decide deliberately. 100% at the agent plus Collector tail sampling for fidelity, or traceidratio at the source when there is no central sampler.
• Validate each layer independently after deploy: Beyla attached (logs), metrics flowing (query the histogram), spans flowing and TLS decrypted (search a trace on an HTTPS endpoint), and overhead sane (Beyla’s own CPU/mem).
• Treat Beyla as the breadth baseline, the SDK as the depth layer. Roll out Beyla fleet-wide first for instant RED, then add SDK depth to the services that earn it.

Security notes

• Beyla is privileged, kernel-adjacent software — treat it as such. Loading eBPF programs and reading other processes’ memory requires elevated Linux capabilities (CAP_BPF, CAP_PERFMON, CAP_SYS_PTRACE, CAP_NET_RAW, and a couple more), and in containers it needs hostPID (DaemonSet) or shareProcessNamespace (sidecar). That is real privilege; review and approve it like any node-level agent.
• Grant the minimum capability set, not privileged: true, wherever possible. The explicit capability list is auditable and narrower than blanket privilege; reserve privileged for platforms that cannot grant fine-grained caps.
• Beyla reads cleartext, including from HTTPS connections. Because it hooks SSL_read/SSL_write after decryption, request and response bodies and headers pass through Beyla in plaintext. It does not export bodies by default, but the capability exists — so anyone who can read Beyla’s configuration or its telemetry stream is in a sensitive position. Restrict who can edit the ConfigMap and who can read the backend.
• Mind PII in routes, attributes, and any captured content. A naive route or an enabled body-capture can pull tokens, emails, or account numbers into your metrics and traces. Use route patterns to strip IDs, and never enable content capture on endpoints that carry secrets without redaction.
• Lock down the RBAC. Beyla’s ClusterRole needs only get/list/watch on pods, nodes, and the workload resources — not write, not secrets. Keep it read-only and scoped; it should never need to mutate anything.
• Secure the export path. Use TLS and authentication on the OTLP endpoint to the Collector, and protect the Prometheus scrape endpoint with network policy — Beyla’s telemetry reveals your internal topology, routes, and traffic patterns.
• Pin and scan the image, restrict who can deploy the DaemonSet. A compromised or trojaned Beyla image runs with kernel-level reach on every node; digest-pin it, scan it, source it from a trusted registry, and gate the DaemonSet behind change control.
• Run on a dedicated, labelled namespace so the Pod Security Admission exception (allowing hostPID and the capabilities) is contained to Beyla and not granted broadly.

Cost & sizing

The bill and the budget here are mostly overhead and cardinality, not licence — Beyla is open source:

• Compute overhead is per node (DaemonSet) or per pod (sidecar), and scales with instrumented throughput. eBPF probes are JIT-compiled and cheap per event, and the user-space agent’s cost tracks the number and request rate of instrumented processes — typically low single-digit percent CPU on a busy node when discovery is scoped, plus a modest, bounded memory footprint for the agent and maps. A DaemonSet on a 40-node cluster is 40 small agents; a sidecar model multiplies by pod count, which is why sidecars cost more at scale.
• The real cost driver is telemetry volume, especially metric cardinality. Every distinct http.route × method × status × k8s_* combination is a time series your backend stores and bills. Un-collapsed URL IDs are the classic budget-killer — one endpoint can mint hundreds of thousands of series. Route patterns + unmatched: heuristic are the cost control, not an afterthought.
• Trace volume costs storage and ingest. A blanket fleet of server and client spans at 100% sampling is a lot of trace data; sample at the source or tail-sample in the Collector to keep Tempo/storage spend in check.
• Sidecar vs DaemonSet is a cost decision as much as an architectural one. DaemonSet amortises one agent across all pods on a node; a sidecar per pod is N agents and N times the baseline memory — choose DaemonSet for fleet coverage unless isolation forces sidecars.
• No per-host licence (it is open source), so spend is infrastructure (the agents’ CPU/RAM) plus your existing observability backend’s ingest/storage. The marginal cost of turning on Beyla is small if you scope discovery and collapse cardinality; it is large if you instrument everything and emit raw paths.

The cost drivers and how to control each:

A rough picture: on a mid-size cluster (30–50 nodes), a scoped Beyla DaemonSet adds a small, predictable compute overhead (each agent a fraction of a core and tens-to-low-hundreds of MB) and its dominant cost lands in your metrics backend — which is entirely under your control via route patterns and sampling. The expensive failure mode is the avoidable one: broad discovery plus raw URL paths. Scope it and collapse it, and Beyla is one of the cheapest ways to get RED across an entire fleet.

Interview & exam questions

1. What is eBPF and why can it observe a process without modifying it? eBPF lets you load small, verifier-checked programs into the running Linux kernel and attach them to hooks (kernel functions, tracepoints, user-space functions) that the system already executes. When the hook fires, the program runs in addition to the normal code path, reads arguments/memory, and writes to maps — the observed process keeps calling the same functions, unaware and unchanged. The verifier proves the program is safe (terminates, valid memory only) before load, so it cannot crash the kernel.

2. What are the two hook families Beyla uses, and what does each capture? kprobes/tracepoints on kernel socket functions (tcp_sendmsg/tcp_recvmsg) capture plaintext request/response bytes and connection timing for any language. uprobes on user-space library functions capture two things the socket layer cannot: TLS cleartext (by hooking SSL_read/SSL_write in libssl after decryption) and, for Go binaries, goroutine-correct request tracking plus in-process traceparent read/write.

3. How does Beyla observe HTTPS traffic without holding any keys? It attaches uprobes to SSL_read/SSL_write (and Go’s crypto/tls), which handle the buffer after decryption on read and before encryption on write — exactly where the application sees cleartext. Beyla reads the same plaintext the app reads, at the same moment, never terminating TLS, performing a MITM, or touching a private key. This beats wire sniffing, which only sees ciphertext.

4. What is the single biggest limitation of Beyla, and for which services does it bite? Automatic distributed-trace context propagation. Beyla stitches traces end-to-end for Go (its uprobes rewrite traceparent in-process) and within a fully Beyla-instrumented estate, but for a non-Go service it generally cannot rewrite outbound headers from the kernel — so the trace fragments at that hop. You still get correct per-service RED and local spans; you lose automatic cross-service linkage unless an SDK or Beyla-to-Beyla propagation bridges it.

5. What are RED metrics and what series does Beyla emit? RED = Rate, Errors, Duration — the standard service-health signal. Beyla emits OTel-semantic series: http.server.request.duration and http.client.request.duration (histograms with count, plus http.route/method/status), the rpc.* equivalents for gRPC, and database/messaging conventions for SQL, Redis, and Kafka. These feed RED dashboards and SLOs directly.

6. DaemonSet vs sidecar for Beyla — when do you pick which? DaemonSet = one agent per node instrumenting many processes; needs hostPID: true; one deploy gives fleet-wide RED; the high-leverage choice for platform teams. Sidecar = one Beyla per app pod; needs shareProcessNamespace: true; strong per-service isolation at the cost of an extra container per pod and N× the agent overhead. DaemonSet for coverage, sidecar for isolation or where node-wide privileges are disallowed.

7. Why might every Beyla-instrumented service show up as unknown_service, and how do you fix it? No explicit service name was set and Kubernetes enrichment is off, so Beyla falls back to the executable name or unknown_service. Fix by setting OTEL_SERVICE_NAME/BEYLA_SERVICE_NAME (sidecar) or enabling attributes.kubernetes.enable: true so the name derives from the owning workload — and applying the RBAC that lets the API lookup succeed.

8. How do you keep Beyla from exploding your metrics cardinality? URL paths with IDs (/orders/12345) become distinct http.route values and the series count grows unbounded. Define routes.patterns for known templates and set unmatched: heuristic so unmatched ID-bearing paths are auto-wildcarded. This collapses hundreds of thousands of potential series into a stable few hundred.

9. How do Beyla and the OpenTelemetry SDK work together without double-counting? Beyla provides breadth (fleet-wide RED + server spans, no code change); the SDK provides depth (business spans, app attributes) on services you own. They share context via the W3C traceparent (default propagator), so spans join into one trace. The rule: exactly one source emits the entry span per service — run the other metrics-only — or you get two server spans and doubled RED.

10. Beyla, Cilium Hubble, and a profiler are all eBPF — what different signals do they produce? Beyla produces application RED metrics and traces (request rate/errors/latency, spans). Cilium Hubble produces network-flow observability (who talks to whom, L7 flow logs, service map, policy). An eBPF profiler (Parca/Pyroscope) produces continuous CPU/memory profiles (flame graphs). Different pillars; you typically run more than one.

11. What kernel and privilege requirements does Beyla have, and why? A kernel of roughly 5.8+ with BTF available (for portable probe attachment), and elevated Linux capabilities — CAP_BPF, CAP_PERFMON, CAP_SYS_PTRACE, CAP_NET_RAW and a couple more — to load eBPF programs and read other processes’ memory, plus hostPID/shareProcessNamespace to see the targets. These are inherent to running code in the kernel and inspecting processes from outside.

12. Why is Beyla preferable to a service mesh for observing in-pod TLS calls? A mesh only sees traffic that traverses its sidecar proxy and is blind to calls a process makes inside the pod (e.g. to a local cache over TLS); it also doubles the data plane. Beyla hooks the TLS library inside the process, so it sees those in-pod, TLS-terminated calls in cleartext — with no extra proxy and no data-plane duplication.

These map most directly to vendor-neutral observability and CNCF/Kubernetes practitioner knowledge (OpenTelemetry, eBPF, Prometheus, Grafana). The eBPF, kernel-capability, and Kubernetes-deployment angles align with CKA/CKS-style platform-security reasoning; the OTel/OTLP and RED/SLO angles align with observability-engineer interview tracks.

Quick check

1. Beyla can produce clean RED metrics and spans for an HTTPS-only service without any keys. Which exact functions does it hook to do this, and at what moment in the data path do they sit?
2. A request that starts in a Go service traces perfectly through two more Go hops, then “goes dark” the instant it reaches a Python service. Why, and what are your two options to fix it?
3. True or false: running Beyla and an OTel SDK auto-instrumentation agent on the same process is the recommended way to get both breadth and depth.
4. After enabling Beyla on a public API, your Mimir ingestion alarms on series count. What is the most likely cause and the one setting that fixes it?
5. You want fleet-wide RED across a 40-node cluster with one deploy. Which Beyla topology, and which one pod-spec field must you set for it to see the processes?

Answers

1. Beyla hooks SSL_read and SSL_write inside the process’s OpenSSL (libssl) — and Go’s crypto/tls equivalents. These sit at the library buffer boundary: after decryption on read and before encryption on write, i.e. exactly where the application itself handles cleartext. Beyla reads the same plaintext at the same instant, never touching a key.
2. The Python hop is non-Go and uninstrumented, so Beyla generally cannot rewrite the outbound traceparent from the kernel — the trace fragments and a new trace ID starts downstream. Options: (a) add an OTel SDK at that hop so it continues the W3C context, or (b) run Beyla everywhere on the path with ebpf.context_propagation: all so Beyla-to-Beyla propagation bridges it.
3. False. If both emit the entry (server) span you get two spans per request and doubled RED. The correct pattern is exactly one entry-span owner per service — Beyla for the long tail, the SDK for already-instrumented services — running the other in metrics-only mode.
4. Route cardinality — URL paths with IDs (/orders/12345) emitted as distinct http.route values, minting unbounded series. Fix with routes.patterns plus unmatched: heuristic so ID-bearing segments are collapsed/auto-wildcarded.
5. A DaemonSet (one agent per node). The pod spec must set hostPID: true so the agent can see processes across the node (a sidecar would instead need shareProcessNamespace: true).

Glossary

• eBPF (extended Berkeley Packet Filter) — a mechanism to load small, sandboxed programs into the Linux kernel and run them on hooks (kernel functions, tracepoints, user-space functions, network events) without changing the observed process.
• Verifier — the kernel component that statically proves an eBPF program is safe (bounded loops, valid memory only) before it loads; why kernel eBPF can’t panic the box.
• JIT (just-in-time compilation) — compiles a verified eBPF program to native machine code so probe overhead is small.
• kprobe / tracepoint — an eBPF hook on a kernel function or predefined kernel event; Beyla uses these on the socket send/receive path to capture bytes language-agnostically.
• uprobe — an eBPF hook on a user-space library function (at a symbol offset); Beyla uses these on libssl (TLS cleartext) and Go runtime symbols (request tracking + header propagation).
• libssl / crypto/tls hook — uprobes on SSL_read/SSL_write (OpenSSL) and Go’s TLS read/write, sitting after decryption / before encryption, so Beyla reads HTTPS cleartext without keys.
• BTF (BPF Type Format) — kernel type metadata that lets eBPF programs resolve kernel structures portably (CO-RE: compile once, run everywhere).
• Map — a kernel/user-space shared data structure (ring buffer, hash map) eBPF programs write to and the Beyla agent drains.
• RED metrics — Rate, Errors, Duration (as a histogram): the standard service-health signal Beyla emits per route/method/status.
• Span — one timed operation in a trace; Beyla emits a server span per inbound request and client spans for outbound calls.
• traceparent — the W3C Trace Context header carrying trace and span IDs across hops; propagating it is what stitches spans into one distributed trace.
• Context propagation — injecting/reading traceparent hop to hop; automatic in Beyla for Go and within a Beyla-instrumented estate, limited for non-Go services.
• OTLP (OpenTelemetry Protocol) — OpenTelemetry’s native wire protocol; Beyla’s primary export format to a Collector.
• OpenTelemetry Collector — a vendor-neutral pipeline that receives OTLP and fans telemetry to backends (Tempo, Mimir/Prometheus); Beyla’s usual destination.
• DaemonSet — a Kubernetes workload running one pod per node; Beyla’s fleet-wide topology (needs hostPID: true).
• Sidecar — a second container in the same pod as the app; Beyla’s per-service isolation topology (needs shareProcessNamespace: true).
• Discovery (discovery.instrument) — Beyla’s rules selecting which processes to instrument (by namespace, pod labels, or executable); the lever for scope, overhead, and cardinality.
• CAP_BPF / CAP_PERFMON / CAP_SYS_PTRACE — Linux capabilities Beyla needs to load eBPF programs and read other processes’ memory; the least-privilege alternative to privileged.
• unmatched: heuristic — the Beyla route setting that auto-wildcards ID-like URL segments not matched by an explicit pattern, the key cardinality control.
• unknown_service — the fallback service name Beyla uses when no explicit name is set and Kubernetes enrichment is off; a bug to fix, never to accept.

Next steps

You can now put RED metrics and per-request spans on any service, in any language, with no code change — and you know exactly where the kernel-level view stops and the SDK begins. Build outward:

• Next: Building Production OpenTelemetry Collector Pipelines: Receivers, Processors, and Tail Sampling — the pipeline that receives everything Beyla emits and routes it to your backends.
• Related: Distributed Tracing End-to-End: Context Propagation, Tempo, and Correlating Traces with Metrics and Logs — the propagation concept that bounds what Beyla can stitch automatically.
• Related: OpenTelemetry for Java Services: Auto-Instrumentation, Context Propagation, and Custom Spans — the SDK depth layer to pair with Beyla’s breadth on services you own.
• Related: Network Observability with Cilium Hubble: Flow Logs, L7 Visibility, and Service Maps — the eBPF sibling that watches network flows rather than application RED.
• Related: Continuous Profiling in Production with eBPF: Parca, Pyroscope, and Flame Graphs — the third eBPF pillar: where your services spend CPU.
• Related: Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus — keep the RED series Beyla produces from exploding your bill.
• Related: Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code — turn Beyla’s RED metrics into dashboards people actually open.