Continuous Profiling in Production with eBPF: Parca, Pyroscope, and Flame Graphs

Metrics tell you a service is burning 80% of a core. Traces tell you which request is slow. Neither tells you which line of code is on CPU when that happens. That gap used to be closed by attaching a profiler to one process, for one minute, after the incident — by which point the pathological allocation is gone. Continuous profiling closes it permanently: an eBPF agent on every node samples the on-CPU stack of every process, dozens to hundreds of times per second, for roughly 1% overhead and zero application changes. The output is a flame graph you can rewind to 03:00 last Tuesday, keyed by namespace, pod, container, node, and function.

This is a reference you will keep open mid-investigation, so it is deliberately table-dense: option matrices for sample rate and unwinding, decision tables for which symbolizer a language needs, a symptom-cause-confirm-fix playbook for the failure modes that make a flame graph lie to you, and real parca, pyroscope, alloy, and go tool pprof commands with expected output. By the end you will deploy an eBPF profiling agent as a DaemonSet, symbolize native and JIT code correctly, read a flame graph without fooling yourself, diff two windows to find a regression in seconds, correlate a profile with the exact slow span that produced it, size the pprof storage bill, and turn aggregate CPU-share into cores you can deprovision.

The hard truth this article keeps returning to is that a profiler does not watch every instruction. It interrupts the CPU at a fixed frequency and records the current stack. The relative width of a frame is an unbiased estimate of the fraction of CPU time spent there; absolute attribution of a single rare event is not what this tool is for. Every capability below — the DWARF unwinder, the build-ID symbolization, the differential view, the span labels — exists to make that statistical picture trustworthy and actionable at fleet scale.

What problem this solves

The three classic pillars answer “what”, “when”, and “across what”. Metrics say error rate rose at 03:00. Logs say which request threw. Traces say the process-cart span took 50 ms. None of them say that 40 ms of that span was a JSON serializer reallocating a buffer inside a loop, on line 214 of encode.go. Profiling answers “where in the code” — it is the missing fourth pillar, and until continuous profiling it was the one you only had after an incident, if you were lucky enough to be profiling when it struck.

What breaks without it: an engineer sees a p99 regression after a Graviton migration, cannot reproduce it in staging (which runs a different architecture), and has no code change to blame because CPU rose uniformly. Or a finance review asks “why are we running 400 cores for a service that handles 2,000 requests per second” and nobody can point at the specific functions eating the budget. Or a memory alert fires and the on-call restarts pods for a week because there is no way to see which allocation site drives the heap growth. Every one of these is a “where in the code” question, and every one is answerable in seconds if you already have always-on profiles keyed by time and labels.

Who hits this hardest: teams running large fleets (hundreds to thousands of pods) where the cost of CPU is real money; polyglot shops where a regression can hide in a shared vendored library across Go, Python, and Java services simultaneously; latency-sensitive services under change freeze where you cannot bolt a manual profiler onto a PCI-scoped process; and any platform team that has been asked to turn “we’re spending a lot on compute” into a ranked, defensible optimization backlog. The property that makes continuous profiling different from perf record on one box is that it is always-on, low-overhead, and fleet-wide — the question becomes a query (“show me the flame graph for payments between 03:00 and 03:05 UTC”) instead of a reproduction exercise.

To frame the whole field before the deep dive, here is the landscape of profiling approaches and where continuous eBPF profiling sits among them:

The eBPF row is the one that turns profiling from an incident-time reaction into a standing dataset. The rest of this article is how to run that row well.

Learning objectives

By the end of this article you can:

• Explain how a sampling profiler works — perf events, the interrupt-and-record model, why frame width is a statistical estimate and the x-axis is share not time.
• Distinguish the three stack-unwinding strategies — frame-pointer, DWARF/.eh_frame CFI, and ORC (kernel) — and know which one an eBPF agent uses for which binary and why release builds break naive unwinding.
• Deploy an eBPF profiling agent (Grafana Alloy/pyroscope.ebpf, Parca Agent, or the OpenTelemetry eBPF profiler) as a privileged DaemonSet with hostPID, and validate the eBPF program actually loaded.
• Symbolize native (Go/Rust/C++), JIT/managed (Python, JVM, Node, .NET), and stripped binaries correctly using in-binary tables, per-runtime unwinders, debuginfod, and build-ID upload.
• Read a flame graph and an icicle/differential view without lying to yourself, and pull two pprof profiles to pprof -diff_base a regression across a deploy boundary.
• Choose between CPU, allocation (alloc_space/alloc_objects), in-use heap (inuse_space), and off-CPU profiling, and know which question each answers.
• Model pprof storage, retention, and the sample-rate/overhead/accuracy trade-off, and size the object-store bill.
• Correlate a profile with a trace via span_id/trace_id pprof labels to get span-scoped flame graphs, and drive a fleet-wide CPU-attribution review that feeds a cost-optimization backlog.

Prerequisites & where this fits

You should be comfortable operating Kubernetes (DaemonSets, ServiceAccounts, securityContext, hostPID), reading YAML and Helm values, and running kubectl. You need a working mental model of what a stack trace is (a chain of return addresses from a leaf function back to a thread entry point) and roughly how a program’s memory is laid out (text/code, stack, the rbp/rsp registers on x86-64). Familiarity with the pprof format from Go (go tool pprof) helps but is not required — this article teaches it. You do not need to know eBPF internals; you need to know what the agent does on your behalf and how to tell when it fails.

This sits in the Observability track, specifically the profiling corner that most teams reach last, after metrics, logs, and traces are already in place. It assumes you have a metrics stack (Prometheus/Mimir) and ideally distributed tracing, because the highest-value workflow — span-scoped profiling — links profiles to traces. It pairs directly with eBPF Zero-Code Instrumentation with Grafana Beyla & OpenTelemetry (the same eBPF, applied to RED metrics and spans instead of stacks), with Distributed Tracing with Tempo & Jaeger: Exemplars & Correlation (the traces you correlate against), and with Cilium Hubble: Network Flow Observability & Service Map (eBPF at the network layer). If your telemetry pipeline runs through the collector, OpenTelemetry Collector Pipelines in Production is where the OTel eBPF profiler’s output lands.

A quick map of who owns what in a continuous-profiling deployment, so you route problems to the right person:

Core concepts

Six mental models make every later section obvious.

A profiler samples; it does not trace every instruction. The agent arms a perf event that fires at a fixed frequency per CPU. On each fire the kernel runs a tiny eBPF program in the context of whatever was executing and records the current stack. Over a window, the count of samples whose leaf (or any frame) is function F divided by total samples is an unbiased estimate of the fraction of CPU time spent in F. This is why a flame graph’s x-axis is share of samples, not time, and why the tool is superb at “where do cycles go across a week” and poor at “what happened in this one 5 ms request” (that is a trace’s job).

Aggregation happens in the kernel, which is why it’s cheap. The naive approach (perf record) ships one event per sample to user space — thousands per second per CPU — then aggregates offline. An eBPF profiler instead increments a counter in a BPF hash map keyed by (pid, user_stack_id, kernel_stack_id). User space reads that map every few seconds and gets pre-counted stacks. You never move per-sample data across the kernel boundary, which is what keeps overhead near 1% even at fleet scale.

Unwinding the user stack is the hard part. Getting from “the CPU was at address 0x4f21a0” to a stack of frames requires walking parent frames. Frame-pointer walking follows the rbp chain and is trivial — but only works if the binary kept frame pointers (-fno-omit-frame-pointer), and most release/distro builds omit them to free a register. So modern agents parse the binary’s .eh_frame DWARF call-frame information (CFI), precompute a compact unwind table, load it into BPF maps, and unwind in-kernel without frame pointers. The kernel’s own stack uses ORC (a kernel-specific unwind format). No recompilation, no LD_PRELOAD, no sidecar.

Symbolization is a separate step from unwinding, with its own failure modes. Unwinding gives you a list of addresses. Symbolization maps each address to function + file:line. Where that mapping lives depends on the language: in-binary tables (Go’s .gopclntab), a separated .debug ELF fetched by build-ID from a debuginfod server, a per-runtime interpreter walk (CPython frame objects), or a JIT map file (perf-<pid>.map). A flame graph full of [unknown] or bare hex is almost always a symbolization failure (missing debug info for a build-ID), not a broken agent.

“On-CPU” is not “slow.” A CPU profile shows where cycles burn. It is blind to off-CPU time — blocked on locks, disk, network, epoll. If a service is slow but its CPU flame graph is nearly empty, the cost is off-CPU; you need off-CPU profiling (eBPF tracing sched_switch to measure time threads spend descheduled) or a trace, not a wider CPU flame graph.

pprof is the lingua franca. Parca, Pyroscope, the Go runtime, the OTel profiler — they all speak pprof (profile.proto), the compressed protobuf go tool pprof has read for a decade. A profile is a set of samples, each a stack of locations (each resolving to functions/lines) plus one or more values (nanoseconds of CPU, bytes allocated, objects) and a bag of labels. Because it’s one format, a Pyroscope profile, a Parca profile, and a net/http/pprof dump are the same shape and interoperate.

The vocabulary in one table

Pin down every moving part before the deep sections; the glossary at the end repeats these for lookup.

How sampling profilers work

A sampling profiler answers “where does the CPU spend time” by taking a large number of instantaneous snapshots of what is executing and counting them. The statistical claim is simple: if you sample the program counter and its call stack uniformly in time, the fraction of samples in which function F appears equals the fraction of wall-CPU-time spent in (or under) F, in expectation. Double the time in a function and you double its expected sample count. This is why you can trust relative widths and must not read absolute single events.

The perf event that drives it

The clock is a perf event created with perf_event_open(2). Two flavours matter:

Most fleet agents default to the software CPU clock at a modest frequency, because hardware PMU counters are frequently unavailable inside cloud VMs and containers. The event is configured with a sample frequency (e.g. freq = 19 meaning “aim for 19 samples per second per CPU”). On each fire the kernel raises an interrupt, and — because an eBPF program is attached to this perf event — runs that program synchronously in the interrupted context.

The interrupt-and-record loop

Here is the loop, concretely, for an eBPF CPU profiler:

1. The perf event fires on CPU n. The kernel is now executing the eBPF program in the context of whatever task was on CPU n.
2. The program reads the current task’s pid/tgid.
3. It calls bpf_get_stackid() (or, in newer agents, bpf_get_stack() plus a custom unwinder) to capture the user stack and the kernel stack, returning numeric stack IDs that index into a BPF_MAP_TYPE_STACK_TRACE map.
4. It increments a counter in a BPF_MAP_TYPE_HASH keyed by (pid, user_stack_id, kernel_stack_id).
5. Every few seconds, user space reads the hash map, resolves stack IDs to address lists, symbolizes, and emits a pprof profile.

The genius is step 4: counting in the kernel. The alternative — one perf record per sample copied to a ring buffer and drained to user space — is what makes perf record expensive at high rates. By counting stacks in a map, the agent moves only distinct stacks and their counts, which compresses enormously because hot paths repeat.

Why width is share, not time (and what that forbids)

The x-axis of a flame graph is the count of samples in which a frame appears, normalized. It is not a timeline. Two consequences engineers get wrong:

• A wide frame low in the stack is not itself expensive. main is always 100% wide because every sample descends through it; that tells you nothing. The cost is in its children. Follow width upward to the leaf.
• You cannot read the duration of a single request off a flame graph. The graph is an aggregate over many samples across many goroutines/threads. “This function took 30 ms” is a trace question; “this function was on CPU 12% of the time this week” is the profile’s answer.

The statistical resolution improves with sample count, which is frequency × duration × CPUs. A rare function that is on CPU 0.1% of the time needs a lot of samples before its width is stable — which is exactly why you crank frequency for a targeted, short investigation and keep it low fleet-wide. The confidence-interval trade-off is quantified later in the storage section.

The clearest way to keep the tool honest is to hold in mind what a sampling profile can and cannot tell you — most misreadings come from asking it a question from the wrong column:

Stack unwinding: frame-pointer vs DWARF vs ORC

Capturing a program counter is trivial. Capturing the stack — the chain of callers back to the thread root — requires unwinding, and this is where naive profilers fail silently on modern binaries. There are three mechanisms, and an eBPF agent chooses per binary.

Frame-pointer unwinding

On x86-64, a function that keeps a frame pointer sets rbp to point at its stack frame, and the previous rbp and the return address sit at known offsets. Unwinding is then a trivial loop: read [rbp] for the caller’s rbp, read [rbp+8] for the return address, repeat until you reach the top. It costs a few memory reads per frame and is what bpf_get_stackid() uses by default.

The catch: compilers omit frame pointers by default at -O1 and above to free rbp as a general-purpose register (worth a small percentage of performance). So most release builds, most distro-shipped shared libraries, and the C library itself have no frame pointers, and frame-pointer unwinding stops at the first such frame — you get a truncated stack (often just the leaf, with everything below it missing). This is the number-one reason a naive eBPF profiler produces shallow, useless stacks.

DWARF CFI (.eh_frame) unwinding

The fix that needs no recompilation: the compiler emits Call Frame Information in the ELF .eh_frame section (present even in stripped binaries, because it’s needed for C++ exceptions and debuggers). CFI is a per-PC table that says, at this instruction, “the return address is at CFA-8, the caller’s rbp is at CFA-16, and the Canonical Frame Address is rsp+32.” An unwinder can reconstruct the full stack from CFI without frame pointers.

The complication: .eh_frame is a compact bytecode (a virtual machine of DWARF opcodes) that is expensive to interpret, and you cannot run a DWARF interpreter safely inside the eBPF verifier. So agents like Parca Agent and the Grafana/OTel eBPF profiler do this: in user space, parse .eh_frame into a flattened unwind table (sorted array of {pc, cfa_rule, rbp_rule, ra_rule} rows), load that table into BPF maps, and run a simple in-kernel unwinder that binary-searches the table by PC and applies the rule — no DWARF VM in the kernel. This is what lets a frame-pointer-less Go, Rust, or C++ release binary unwind fully at ~1% overhead.

ORC unwinding (the kernel side)

The Linux kernel does not use .eh_frame for itself; it uses ORC (Oops Rewind Capability), a purpose-built, compact, fast unwind format generated at kernel build time (CONFIG_UNWINDER_ORC). When your eBPF program captures the kernel portion of a stack (syscalls, kernel functions above the user frames), the kernel unwinds it with ORC internally and hands you resolved kernel frames. You do not manage ORC; it is why kernel frames in your flame graph are usually well-symbolized (against /proc/kallsyms) even when user frames are not.

Choosing per binary — the decision table

The agent inspects each mapped binary and picks a strategy:

The practical upshot: for your own services, keeping frame pointers on (-fno-omit-frame-pointer, ~1% runtime cost) makes unwinding bulletproof and cheap. For everything else — dependencies, base images, stripped release binaries — you rely on the agent’s DWARF-CFI path, which is why a modern agent and a modern kernel matter.

Kernel and capability requirements

Unwinding and stack capture depend on kernel features and privileges. The realistic floor:

eBPF whole-system profiling: the three agents

Three production-grade eBPF profilers dominate, and they share the same core (perf event → in-kernel aggregation → pprof) while differing in storage, symbolization model, and ecosystem. You will pick one; knowing how they differ prevents a costly re-platform later.

Parca (Agent + Server)

Parca is the CNCF-adjacent, pprof-native pair: Parca Agent (the eBPF DaemonSet) samples every process on a node and pushes to the Parca server, which stores profiles in FrostDB (an embeddable columnar store) and does server-side symbolization — agents upload build-IDs, and the server resolves them once, centrally, against uploaded debug info or debuginfod. The design philosophy is “pprof all the way down”: everything is pprof, the query language is label-selector-based, and the server exposes a pprof-compatible API you can point go tool pprof at.

Grafana Pyroscope (+ Alloy / eBPF)

Grafana Pyroscope is the object-store-backed, horizontally scalable profiling database modeled on the Grafana LGTM stack (same microservices shape as Loki/Mimir/Tempo). You feed it either from language SDKs (pyroscope-go, Python, Java, Ruby, .NET) or from Grafana Alloy running the pyroscope.ebpf component (the eBPF whole-system profiler). Storage is S3/GCS/ABS, so retention is an object-store bill, and it integrates tightly with Grafana’s flame-graph panel, Explore Profiles, and trace-to-profile links in Tempo.

The OpenTelemetry eBPF profiler

The OpenTelemetry eBPF profiler (donated from Elastic’s universal profiler) is the emerging vendor-neutral standard: a whole-system, multi-runtime eBPF profiler that emits the OTLP profiles signal, so profiles flow through the OpenTelemetry Collector alongside metrics/logs/traces and can be routed to any OTLP-profiles backend (including Pyroscope). It unwinds native (DWARF), Go, Python, JVM (via JIT integration), Ruby, PHP, Node/V8, and .NET, and is designed to be the eBPF profiling source once the OTel profiling signal stabilizes.

Agent comparison — the decision matrix

Agent configuration — the load-bearing knobs

Whichever agent you pick, the same set of knobs governs behaviour. Getting these right is the difference between trustworthy profiles and a fleet-wide overhead surprise:

Deploying the agent as a DaemonSet

One agent per node, profiling every container on it. Below is a Grafana Alloy DaemonSet running the eBPF profiler and remote-writing to Pyroscope — Alloy is the supported way to ship the Grafana eBPF profiler.

The Alloy configuration (a .alloy river file) discovers pods, relabels metadata into query labels, runs pyroscope.ebpf, and writes to a Pyroscope endpoint:

// alloy-config.alloy
discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label  = "node"
  }
}

pyroscope.ebpf "instance" {
  forward_to       = [pyroscope.write.endpoint.receiver]
  targets          = discovery.relabel.pods.output
  default_target   = "false"     // only profile discovered/labeled targets
  collect_interval = "15s"
  sample_rate      = 19          // Hz per CPU
}

pyroscope.write "endpoint" {
  endpoint {
    url = "http://pyroscope.monitoring.svc.cluster.local:4040"
  }
  external_labels = {
    "cluster" = "prod-eu-west-1",
  }
}

The agent needs host access and elevated privileges. The pod spec is the load-bearing part — get hostPID and the capabilities right or nothing works:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: alloy-profiler
  namespace: monitoring
spec:
  selector:
    matchLabels: { app: alloy-profiler }
  template:
    metadata:
      labels: { app: alloy-profiler }
    spec:
      hostPID: true                       # resolve sampled PIDs to host containers
      serviceAccountName: alloy-profiler
      containers:
        - name: alloy
          image: grafana/alloy:v1.5.0
          args:
            - run
            - /etc/alloy/config.alloy
            - --server.http.listen-addr=0.0.0.0:12345
          securityContext:
            privileged: true              # or the scoped caps below
            # capabilities:
            #   add: ["BPF", "PERFMON", "SYS_PTRACE", "SYS_RESOURCE"]
          volumeMounts:
            - { name: config, mountPath: /etc/alloy }
      volumes:
        - name: config
          configMap: { name: alloy-profiler-config }

hostPID: true is what lets the agent resolve a sampled PID back to the right container and binary on the host. Without it you get stacks you cannot attribute — the PID inside the agent’s namespace does not match the host PID the perf event reported.

If you prefer Parca, the model is identical — a DaemonSet of parca-agent pushing to a parca server:

# parca-agent container args (DaemonSet)
args:
  - /bin/parca-agent
  - --node=$(NODE_NAME)                              # from fieldRef spec.nodeName
  - --remote-store-address=parca.monitoring.svc:7070
  - --remote-store-insecure
  - --profiling-cpu-sampling-frequency=19
  - --dwarf-unwinding=enabled                        # DWARF CFI for FP-less binaries

For the OpenTelemetry eBPF profiler, you run it as a DaemonSet exporting OTLP profiles to a collector, which routes them onward:

# otel-ebpf-profiler args (DaemonSet), exporting OTLP profiles to a local collector
args:
  - -collection-agent=otel-collector.monitoring.svc:4317
  - -samples-per-second=19
  - -reporter-interval=15s

All three encode profiles as pprof (or OTLP profiles, which carries the same sample model) under the hood — the format go tool pprof has read for a decade. That portability is why a Pyroscope profile, a Parca profile, and a Go runtime profile are all the same shape, and why you can pull any of them into go tool pprof for diffing.

The RBAC and PodSecurity requirements are non-negotiable; if your cluster enforces a restricted Pod Security Standard, the profiler namespace needs a privileged label exception:

Symbolization: turning addresses into names

A raw sample is a list of virtual addresses. Symbolization maps each to function + file:line. Where that mapping lives is entirely language-dependent, and the failure modes differ, so treat symbolization as its own subsystem.

Native, stripped (Go, Rust, C/C++)

The binary on the node either has a symbol table (.symtab/.debug_*) or it does not. Release builds are usually stripped. Three options, in order of preference:

1. debuginfod. The agent extracts the binary’s build-ID (the hash in the ELF NT_GNU_BUILD_ID note) and fetches the matching separated debug info from a debuginfod server over HTTP. Point the agent (or your tooling) at one or more:

# Public elfutils federation + your internal server (comma-separated, tried in order)
export DEBUGINFOD_URLS="https://debuginfod.elfutils.org/ https://debuginfod.internal.corp/"

Parca runs its own symbolizer that uploads build-IDs and resolves them server-side, so symbolization happens once, centrally, not on every node.

2. Ship debug info to the server ahead of time. With Parca, parca-debuginfo upload pushes the .debug ELF for a build-ID into Parca’s store. CI is the right place — right after the build, before the artifact ships:

# In CI, after building the release binary
parca-debuginfo upload \
  --store-address=parca.monitoring.svc:7070 \
  ./bin/payments-service
# Parca now has debug info keyed by this binary's build-ID; any node running it
# will symbolize centrally with zero on-node debug data.

3. Don’t strip / keep frame pointers. For your own services, keep a separated .debug file and frame pointers. The runtime overhead of -fno-omit-frame-pointer is ~1% and it makes unwinding bulletproof.

JIT and managed runtimes — per-language unwinders

Managed runtimes don’t put native function names at fixed addresses; the agent needs a per-runtime strategy:

The build-ID pipeline as a whole

Symbolization at fleet scale is a pipeline, not a per-node accident. The reliable pattern:

The failure mode you see first is a flame graph full of bare hex addresses or [unknown] frames stacked under a recognizable native function. That always means missing debug info for that build-ID — not a broken agent. Resolve the build-ID (upload debug info, fix the debuginfod URL, enable the runtime’s perf map), not the agent config.

CPU vs memory/alloc vs off-CPU profiling

“Profiling” is not one thing. The value type recorded per sample defines the question you’re answering, and mixing them up leads to wrong conclusions (“I see a leak” when the panel shows allocation rate, not retained heap). Know exactly which profile you’re reading.

The profile types

On-CPU: the default, and its blind spot

The eBPF CPU profile is what you get out of the box, and it is the right first look for compute-bound cost. Its blind spot is fundamental: it cannot see time spent off-CPU. A thread blocked on a mutex, a disk read, or epoll_wait is not on any CPU, so no perf-event interrupt catches it, so it contributes zero to the CPU flame graph. If a service’s p99 is 500 ms but its CPU flame graph is nearly empty, the latency is off-CPU — do not widen the CPU profile looking for it; switch to off-CPU or read the trace.

Memory: allocation rate vs retained heap (the trap)

Continuous memory profiling almost always defaults to allocation profiles — alloc_space / alloc_objects, the total bytes/objects allocated over the window. That is the right signal for GC pressure and allocation churn (the thing that makes a Go service spend 30% of CPU in the garbage collector). It is not the same as inuse_space (what is retained right now). A function can dominate alloc_space (allocates a lot, all short-lived) and contribute nothing to inuse_space — that is churn, not a leak. Conversely, a slow leak shows in inuse_space climbing over time, barely visible in per-window alloc_space. Before you say “leak,” confirm which value the panel shows:

Off-CPU: seeing the waits

Off-CPU profiling answers “where is my program not running when it should be.” An eBPF program hooks sched_switch (the kernel scheduler descheduling a task) and records the stack at the moment the thread goes off-CPU plus the duration until it comes back on. Aggregate that and you get a flame graph of wait time — dominated by lock acquisition, blocking syscalls, and channel/queue waits. It is heavier than CPU profiling (context-switch events are frequent) and not every eBPF profiler ships it, but it is the missing half of latency analysis for I/O- or lock-bound services.

Which to reach for — the decision table

Reading flame graphs without lying to yourself

A flame graph is an aggregation of every sampled stack over a time window, drawn as a stack of rectangles. Reading it wrong wastes hours; reading it right finds the fix in seconds.

The axes and the rules

• The x-axis is NOT time. It is the count of samples (share of CPU). Width = share. A frame twice as wide was on-CPU twice as often. Left-to-right order within a parent is alphabetical (or by another stable key) — meaningless as a sequence.
• The y-axis is stack depth. The bottom frame is the entry point (main, a thread root); the top frame is what was actually executing when the sample fired.
• The top edge is where the CPU was. Trace the topmost frames and you’re looking at the leaf functions consuming cycles. Wide and high = a hot leaf you can optimize directly.
• A wide frame low down is not itself expensive — it’s a frame whose children are expensive. Don’t optimize the caller; follow the width upward to the leaf.

Flame vs icicle vs differential

The same data, oriented and colored differently, answers different questions:

The differential view — the single most valuable read

Pick two windows — before a deploy and after — and the renderer colors each frame by the change in its sample share. Red/wider = this function got more expensive; blue/narrower = cheaper. This is how you find a regression in seconds. Pull two pprof profiles and diff them locally with go tool pprof:

# Pull two pprof profiles from Parca's pprof-compatible API and diff them locally.
curl -s "http://parca:7070/api/v1/query?query=<selector>&time=<BEFORE_TS>" -o before.pb.gz
curl -s "http://parca:7070/api/v1/query?query=<selector>&time=<AFTER_TS>"  -o after.pb.gz

# Positive (red) nodes are the NEW cost introduced after the deploy.
go tool pprof -http=:8080 -diff_base=before.pb.gz after.pb.gz

pprof -diff_base produces exactly this: a graph where positive (red) nodes are the new cost. If a single function lit up red across a deploy boundary, that is your regression, with the commit pinned by the deploy timestamp. In Grafana/Pyroscope and Parca UIs the same diff is a first-class panel — select baseline and comparison ranges and it colors the flame graph for you.

Two more semantics people get wrong

• CPU profile vs wall-clock. Covered above: an on-CPU profile is blind to off-CPU time. A near-empty CPU flame graph on a slow service means the cost is off-CPU.
• Aggregation hides bimodality. A flame graph over an hour blends a fast steady state with a rare slow path. If you suspect a tail behaviour, narrow the window (or filter by a label like a slow endpoint, or use span-scoped profiling) so the aggregate isn’t dominated by the common case that hides the pathology.

Common misreadings, tabulated

Correlating profiles with traces

The next leap is span-scoped profiling: instead of asking “what was hot in payments from 03:00 to 03:05”, ask “what was hot during this specific slow span.” The mechanism is a label. OpenTelemetry’s profiling integration / the Pyroscope SDK pushes the active span’s span_id and trace_id into the runtime’s pprof labels, so each CPU sample taken while that span is active carries the span it belongs to.

In Go, the runtime’s pprof.Labels/pprof.Do machinery is what the SDK rides on. With the Pyroscope Go SDK:

import "github.com/grafana/pyroscope-go"

// Attach span context as a profiling label for the duration of the work.
// Every CPU sample taken inside this closure is tagged with this span_id.
pyroscope.TagWrapper(
  ctx,
  pyroscope.Labels("span_id", span.SpanContext().SpanID().String()),
  func(c context.Context) {
    processCart(c)
  },
)

Or, at a lower level, the standard-library primitive the SDK builds on:

import "runtime/pprof"

pprof.Do(ctx, pprof.Labels("span_id", spanID, "trace_id", traceID), func(ctx context.Context) {
    processCart(ctx) // samples here carry span_id and trace_id in the pprof profile
})

Now in the trace UI (Tempo/Grafana) you click a 50 ms span and get the flame graph of just that span’s CPU time. The trace tells you which call was slow; the embedded profile tells you why, down to the function — one click, no reproduction. This is the payoff that justifies the whole stack.

The correlation is bidirectional and rests on a small set of conventions:

Two caveats keep this from backfiring. First, cardinality: trace_id is unbounded, so you do not want it as a storage label on every profile series — it’s carried on the sample and used for lookup, not as a top-level series dimension you group by. Second, coverage: span labels only appear for services running an SDK that sets them (typically your Go/Java/etc. services), while the eBPF agent still profiles everything — so you get whole-fleet flame graphs from eBPF and span-scoped detail on the instrumented subset. Wire the labels for the latency-critical services first.

For the trace side of this correlation, see Distributed Tracing with Tempo & Jaeger: Exemplars & Correlation and OpenTelemetry Metrics: Exemplars & Prometheus Trace Correlation, which cover the exemplar links that let a metric spike jump straight to the trace and then the span-scoped flame graph.

The pprof storage model, retention, and the sampling trade-off

Profiles are bulkier than metrics but compress hard because stacks repeat. Understanding the storage model tells you what to tune and what the bill will be.

The pprof data model

A pprof profile (profile.proto) is a self-contained, gzip-compressed protobuf with these tables:

Because locations, functions, and strings are deduplicated and referenced by index, and because hot stacks repeat across samples, a profile compresses far better than its logical size. Storage engines exploit this further: Parca’s FrostDB stores profiles columnar (stacks as columns, values as columns), and Pyroscope stores in an object-store-backed format that dedupes symbols across profiles.

Retention and backend configuration

Pyroscope’s object-store backend makes retention an S3/GCS bill rather than a node-local-disk problem:

# pyroscope values.yaml (microservices mode) — object store + limits
pyroscope:
  structuredConfig:
    storage:
      backend: s3
      s3:
        bucket_name: prod-pyroscope-profiles
        endpoint: s3.eu-west-1.amazonaws.com
    limits:
      max_query_lookback: 720h        # 30 days queryable
      ingestion_rate_mb: 32           # per-tenant ingest cap
      max_query_length: 24h           # cap a single query's time span

The knobs that govern retention, ingest, and query cost:

The sample-rate trade-off, quantified

The single dial that governs both overhead and statistical accuracy is sample rate:

Higher frequency narrows the confidence interval on each frame’s width but linearly increases per-sample CPU work and storage volume. The relationship is statistical: the number of samples for a frame is rate × duration × CPUs × share, and the relative error on a frame’s width falls roughly as 1/sqrt(samples). So going from 19 to 100 Hz (5.3×) only halves the error (sqrt(5.3) ≈ 2.3), for 5× the cost — textbook diminishing returns. The sane posture: 19 Hz fleet-wide, with the ability to crank a single namespace to 100 Hz during an investigation. Resist 100 Hz everywhere; you pay overhead on every node to sharpen estimates you mostly aren’t reading.

Storage volume scales with sample rate, fleet size, stack depth, and label cardinality:

Driving cost optimization from CPU attribution

The most defensible ROI of continuous profiling is turning CPU-share into money. Fleet-wide attribution answers a question you cannot get any other way: which function, across all services, burns the most aggregate CPU-seconds. Query the aggregate flame graph across the whole cluster for a representative week, sort leaf functions by total self-CPU-seconds, and the top of that list is your optimization backlog ranked by cost.

A representative top-of-list from a real cluster review looks like this:

Top CPU consumers, cluster-wide, last 7d (by self CPU-seconds)
  1. compress/flate.(*compressor).deflate        18.4%   ~46 cores
  2. encoding/json.(*encodeState).string          9.1%   ~23 cores
  3. regexp.(*Regexp).doExecute                    6.7%   ~17 cores  <- compiled per-request
  4. crypto/sha256.block                           4.2%   ~11 cores
  5. runtime.mallocgc                              3.9%   ~10 cores  <- allocation churn

Item 3 — a regex recompiled on every request instead of once at init — is a five-line fix that returns ~17 cores across the fleet. You only see it because the profiler aggregated the same hot leaf across every service that imported the offending library. Item 5 (runtime.mallocgc) is a pointer to the allocation profile: heavy GC means an allocation-churn problem to chase in alloc_space.

A repeatable FinOps loop turns this into a standing practice:

The kinds of waste that reliably surface as fat cross-service frames — each translating directly to deprovisionable cores:

Architecture at a glance

There is no single diagram that captures continuous profiling honestly, because the system is a pipeline with a fan-in at the node and a fan-out at the query. Walk it as a flow instead.

At the bottom sits the node kernel. A perf event (software CPU clock at, say, 19 Hz per CPU) fires an interrupt; an eBPF program attached to that event runs in the interrupted task’s context, captures the user stack (unwinding via frame pointers where present, otherwise via a flattened DWARF/.eh_frame table loaded into BPF maps) and the kernel stack (unwound by the kernel’s ORC), and increments a counter in a BPF hash map keyed by (pid, user_stack_id, kernel_stack_id). This is the fan-in: thousands of interrupts per second per node collapse into a small set of distinct, pre-counted stacks. Crucially, hostPID lets the agent map each pid back to the container and binary on the host so the sample can be labeled namespace/pod/container/node.

Every few seconds the agent (Alloy pyroscope.ebpf, Parca Agent, or the OTel eBPF profiler — one per node, a DaemonSet) reads the map, symbolizes addresses (in-binary .gopclntab for Go; per-runtime interpreter walk for Python/JVM/Node/.NET; build-ID → debuginfod / Parca debug-info store for stripped native), and emits a pprof (or OTLP-profiles) payload of samples — each a stack, a value (cpu nanoseconds, or alloc_space bytes on a memory profile), and labels. Where an application SDK is present, samples taken inside a span also carry span_id/trace_id labels — the seam that later enables span-scoped flame graphs.

Those payloads flow to the storage backend: Parca’s FrostDB columnar store, or Pyroscope backed by S3/GCS (the retention bill lives here, governed by max_query_lookback), or an OTLP-profiles backend reached through the OpenTelemetry Collector. This is the fan-out point. On top of storage sit the consumers: a Grafana/Parca UI rendering a flame graph for {namespace="payments"} over a chosen window; a differential view diffing a before-deploy baseline against after (pprof -diff_base) to surface a red regression frame; a trace UI that, on click of a slow span, pulls the span-scoped flame graph via the span_id label; and a FinOps review querying the cluster-wide aggregate to rank self-CPU leaves into cores. The whole method is: sample and count in the kernel, symbolize by build-ID, store as pprof, and query the aggregate — turning “where in the code did the CPU go, last Tuesday at 03:00, in this exact span” from a reproduction exercise into a lookup.

Real-world scenario

Meridian Pay runs its settlement platform on three EKS clusters in eu-west-1: about 600 pods, a roughly even split of Go and Python services, fronted by an internal gateway, handling ~2,000 settlement requests per second at peak. The platform team is six engineers; the monthly compute bill for these clusters is about ₹34 lakh (~USD 41k). They already ran the Grafana eBPF profiler (Alloy pyroscope.ebpf) as a DaemonSet at 19 Hz, writing to a Pyroscope cluster on S3 with 30-day retention — so profiling was a standing dataset, not something they’d bolt on during an incident.

The incident began after a planned migration of the settlement service from m6i (amd64) to m7g (Graviton, arm64) nodes to cut cost. Within a day, p99 latency on settlement crept up ~12% with no code change and no obvious metric to blame — CPU was up a little, but uniformly across pods, so dashboards showed nothing actionable. The constraints were brutal: they could not reproduce it in staging (which still ran amd64), and they refused to add manual profiling hooks to a PCI-scoped service under an active change freeze. A traditional shop would have been stuck bisecting infrastructure.

Because the profiler was always-on, the investigation was a query, not a deployment. They pulled the cluster-wide differential flame graph, amd64 baseline (the week before the migration) versus arm64 (after), for the settlement service, and diffed with go tool pprof:

go tool pprof -http=:8080 \
  -diff_base=settlement-amd64.pb.gz \
  settlement-arm64.pb.gz

One frame lit up red across both Go and Python pods: a checksum routine in a shared vendored crypto library that fell back to a generic, byte-at-a-time implementation on arm64 because the build hadn’t been compiled with the ARM CRC32 intrinsics enabled. The diff made it unmistakable — the same leaf, fatter on arm64, in services that otherwise shared no code. On amd64 the library had used a hardware CRC path; on arm64, without the right build flags, it silently used the slow path. The uniform CPU rise was this one routine, spread everywhere the library was imported.

The fix was a build flag, not a rewrite — rebuild the dependency with the architecture’s CRC/LSE extensions enabled:

# arm64 build picks up the hardware CRC + LSE atomics path
ENV GOARCH=arm64 GOARM64=v8.1
RUN go build -trimpath -o /out/settlement ./cmd/settlement

p99 returned to baseline within a deploy. They verified with the reverse diff (arm64-before vs arm64-after) — the red frame went blue — and the Graviton migration’s cost savings (~18% on those nodes) were finally realized without the latency penalty. The lesson written on the wall: “An always-on profiler turns a vague, unreproducible, cross-language regression into a single red frame — no code change to a frozen service, no staging repro.” That capability is the entire reason the stack exists.

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

Continuous eBPF profiling is powerful but not free of trade-offs. Weigh it honestly.

The model is right for large, polyglot fleets where CPU is real money and reproduction is expensive — payments, ad tech, streaming, big SaaS. It is overkill for a handful of low-traffic services where a one-off perf/pprof session suffices, and it is the wrong tool as a sole latency instrument for I/O-bound services (add off-CPU or lean on tracing). The disadvantages are all manageable — but only if you know they exist, which is the point of the troubleshooting section.

Hands-on lab

Stand up Grafana Pyroscope, run the eBPF profiler against a workload, read a flame graph, then diff two windows — all on a local kind/minikube cluster (free-tier-friendly; delete at the end). Where an eBPF agent can’t get privileges locally (some Docker Desktop kernels), a fallback using the Pyroscope Go SDK is included so the lab still works end to end.

Step 1 — Namespace and Pyroscope (single-binary mode).

kubectl create namespace monitoring
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
# Single-binary Pyroscope with local filesystem storage (lab only)
helm install pyroscope grafana/pyroscope -n monitoring \
  --set pyroscope.persistence.enabled=false
kubectl -n monitoring rollout status statefulset/pyroscope

Expected: the pyroscope StatefulSet reaches Ready. Port-forward the UI:

kubectl -n monitoring port-forward svc/pyroscope 4040:4040 &
# Open http://localhost:4040 — the Pyroscope UI loads (empty until data arrives).

Step 2 — Deploy a CPU-burning sample workload. A tiny Go service with a deliberately hot function (a per-request regex compile) makes the flame graph obvious:

kubectl -n monitoring create deployment hotpath \
  --image=grafana/pyroscope-rideshare-go:latest
kubectl -n monitoring rollout status deployment/hotpath

(The Grafana rideshare demo images are SDK-instrumented and push to Pyroscope directly, which also exercises the SDK path used in Step 5.)

Step 3 — Deploy the Alloy eBPF profiler DaemonSet. Apply the config and DaemonSet from the deployment section (ConfigMap + DaemonSet). Point pyroscope.write at http://pyroscope.monitoring.svc.cluster.local:4040:

kubectl -n monitoring create configmap alloy-profiler-config \
  --from-file=config.alloy=./alloy-config.alloy
kubectl -n monitoring apply -f ./alloy-daemonset.yaml
kubectl -n monitoring rollout status ds/alloy-profiler

Expected: one alloy-profiler pod per node, READY equal to node count.

Step 4 — Confirm the eBPF program loaded (no silent permission failure).

kubectl -n monitoring logs ds/alloy-profiler | grep -iE "ebpf|perf_event|loaded|unwind"
# Look for lines indicating perf events armed and (if native targets) unwind tables built.
# A "permission denied" / "operation not permitted" here means missing CAP_BPF/CAP_PERFMON
# or a kernel too old — on Docker Desktop, skip to Step 5's SDK fallback.

Step 5 — (Fallback / always works) SDK path. If the eBPF agent can’t get privileges locally, the SDK-instrumented hotpath/rideshare pods from Step 2 still push profiles. Either way, generate load if the demo doesn’t self-load:

kubectl -n monitoring port-forward deploy/hotpath 5000:5000 &
# hit the endpoint in a loop to create CPU work
for i in $(seq 1 2000); do curl -s localhost:5000/ >/dev/null; done

Step 6 — Read a flame graph. In the UI (http://localhost:4040): pick the process_cpu profile type and the workload’s app/service, set the time range to the last 15 minutes. Expected: a flame graph with a wide leaf at the hot function (the per-request compile / rideshare hot path). Confirm you see function names, not hex — symbolization is working.

Step 7 — Diff two windows. Change the workload (e.g. scale it, or in a real app deploy a fix), then use the UI’s comparison / diff view: set a baseline range (before) and a comparison range (after). Expected: frames that got busier are red, quieter are blue. This is the regression-hunting workflow in miniature.

Validation checklist. You deployed a profiling store, ran an eBPF DaemonSet (or the SDK fallback), confirmed the program loaded, read a flame graph with real symbols, and diffed two windows — the full loop.

Cleanup.

kubectl delete namespace monitoring
# If you created a kind cluster for this: kind delete cluster

Cost note. Entirely local — the only cost is your laptop’s CPU for the duration. In a real cluster, the ongoing cost is the object-store retention and the ~1% agent overhead, both quantified in Cost & sizing below.

Common mistakes & troubleshooting

This is the part you bookmark: the failure modes that make a profiler produce nothing, produce garbage, or — worst — produce a plausible-looking flame graph that lies. First as a scannable table, then the highest-impact entries expanded.

The expanded form for the entries that bite hardest:

1. Flame graph is all [unknown] / hex under a recognizable native function. Root cause: the binary is stripped and there is no debug info for its build-ID anywhere the symbolizer can reach. Confirm: the UI shows bare addresses at the leaves; extract the binary’s build-ID (readelf -n <binary> | grep -i build-id if you can reach the node/image) and confirm no debuginfod/store entry exists for it. Fix: in CI, parca-debuginfo upload the .debug ELF for that build-ID, or set DEBUGINFOD_URLS to a server that has it. For managed runtimes, enable the runtime’s perf map. This is a symbolization fix — never “restart the agent.”

2. Stacks are shallow — just the leaf, nothing below it. Root cause: the binary omitted frame pointers (release -O2) and the agent isn’t using DWARF unwinding, so the walk stops at the first FP-less frame. Confirm: agent logs indicate frame-pointer-only mode; the binary was built without -fno-omit-frame-pointer. Fix: turn on the agent’s DWARF/.eh_frame unwinding (--dwarf-unwinding=enabled on Parca; on by default in modern Alloy/OTel profilers), and for your services build with frame pointers so the cheap path also works.

3. No samples at all; agent is running but the store is empty. Root cause: the eBPF program failed to load — missing capabilities or a too-old kernel — and the agent kept running without profiling. Confirm: kubectl logs ds/alloy-profiler | grep -iE "permission|not permitted|perf_event|bpf" shows an EPERM/verifier/perf_event_open failure. Fix: grant CAP_BPF+CAP_PERFMON (and often CAP_SYS_PTRACE for reading other processes’ maps) or run privileged: true; ensure the kernel is 5.4+ (5.10+ for stable DWARF unwinding) and BTF is present.

4. Samples arrive but can’t be attributed to the right pod/container. Root cause: hostPID: true is missing, so the PID the perf event reports (host namespace) doesn’t match anything in the agent’s PID namespace. Confirm: the DaemonSet spec lacks hostPID; stacks are labeled with empty or wrong container/pod. Fix: set hostPID: true. This is the single most common “it runs but the data is useless” mistake.

6. A slow service with a nearly-empty CPU flame graph. Root cause: the latency is off-CPU — threads blocked on locks, disk, network, or epoll — which a CPU sampler cannot see by construction. Confirm: CPU utilization is low during the slow period; a trace of a slow request shows time in waits, not compute. Fix: enable off-CPU profiling (eBPF sched_switch) if your agent supports it, or use the language’s block/mutex profile, or read the trace. Do not keep widening the CPU profile — the answer isn’t there.

7. “I found the leak” but memory keeps climbing after the fix. Root cause: you optimized the top of the allocation profile (alloc_space, total bytes allocated — churn) when the problem is retention (inuse_space). Confirm: the panel’s sample_type is alloc_space/alloc_objects; the fixed site allocated a lot but held nothing. Fix: switch to inuse_space/inuse_objects and diff it over time — the site whose in-use bytes grow across the window is the leak.

Best practices

• 19 Hz fleet-wide; 100 Hz only for a scoped, short investigation. You pay overhead on every node for resolution you mostly don’t read. Keep a documented, one-command path to bump a single namespace during an incident.
• Set hostPID: true on the DaemonSet. Without it, samples are unattributable. This is the number-one deployment mistake.
• Run kernel 5.10+ on all nodes for stable DWARF-based unwinding. 5.4 is the floor; 5.10+ is where the frame-pointer-less path is comfortable.
• Keep frame pointers on your own binaries (-fno-omit-frame-pointer, ~1% cost) or ship separated .debug files. It makes both unwinding and symbolization bulletproof for the code you control.
• Make symbolization a CI step, not a runtime accident. Upload debug info per build-ID right after the build (parca-debuginfo upload) or run an internal debuginfod. Verify zero [unknown] on hot paths as an acceptance check.
• Profile labeled targets, not literally every process. default_target=false plus relabeling keeps volume and cardinality bounded to what you’ll actually query.
• Keep labels bounded; never make trace_id a storage series label. Carry it on the sample for span lookup. Unbounded series dimensions explode the storage bill.
• Document the diff workflow (pprof -diff_base) and tie it to deploy timestamps. A regression hunt should be “diff the window before this deploy against after,” reflexively.
• Wire span-scoped labels (span_id/trace_id) for latency-critical services first. The eBPF agent gives you whole-fleet coverage; the SDK labels give you the “why is this span slow” superpower where it matters most.
• Choose the profile type deliberately. CPU for compute cost, alloc_space for GC pressure, inuse_space diffed over time for leaks, off-CPU for wait-bound latency. Label your dashboards with the sample_type so nobody misreads alloc as retained.
• Run a monthly fleet-wide CPU-attribution review feeding a cost-optimization backlog ranked by cores-returned. This is the ROI that pays for the whole stack.
• Set an explicit object-store retention window (max_query_lookback) and tier old data. Don’t let profiling retention silently grow unbounded.
• Teach the three traps to everyone who reads a flame graph: x-axis is share not time, CPU profiles are blind off-CPU, and memory profiles default to alloc not in-use.

Security notes

• Justify the privilege. The agent runs privileged / hostPID and reads every process’s memory maps — it is a high-value target. Restrict it to a dedicated namespace with a PodSecurity exception, its own least-privilege ServiceAccount, and image provenance you trust (pin the digest, scan it).
• Prefer scoped capabilities over full privileged where the kernel allows. CAP_BPF + CAP_PERFMON (+ CAP_SYS_PTRACE) is a materially smaller blast radius than privileged: true; use it on kernels/distros that support it.
• Profiles can leak sensitive information. Function names, file paths, and (rarely, via labels or symbolized strings) argument-shaped data can appear in flame graphs. Treat the profiling backend as sensitive — put it behind auth/RBAC, don’t expose the UI publicly, and be deliberate about which labels you attach.
• Debug info is source-revealing. A debuginfod server or uploaded .debug files expose function/line structure of your binaries. Keep the internal debuginfod/Parca store internal and access-controlled; don’t point production agents at an untrusted public server for your build-IDs.
• Multi-tenancy and label hygiene. In a shared profiling DB (Pyroscope tenants), enforce tenant isolation and cap per-tenant ingest so one noisy team can’t starve or bill-bomb others. Avoid putting request payloads or PII into pprof labels — labels are stored and queryable.
• Network path. Agent→backend traffic (pprof/OTLP) should be TLS and, ideally, stay in-cluster or on a private path; it carries a detailed picture of what your code is doing.

The security controls that also improve reliability — they pull the same direction here:

Cost & sizing

Two costs dominate: the agent overhead (compute you pay on every node) and the storage/retention (object-store bytes plus query capacity). Both are tunable, and the default posture is cheap.

• Agent overhead is the sample rate’s price. At 19 Hz fleet-wide, budget ~0.2–0.5% CPU per node for the profiler — negligible against the fleet, and far cheaper than the cores a single fleet-wide CPU review typically recovers. Bumping to 100 Hz costs ~2–3%; do it only on a scoped namespace, briefly.
• Storage is a function of sample rate × profiled processes × retention, softened hard by compression and symbol dedup. For a ~600-pod fleet at 19 Hz with 30-day retention, object-store cost is typically a few thousand rupees a month (order ₹3,000–8,000, dominated by S3/GCS storage-GB and requests) — trivially less than the compute it helps you optimize.
• Query capacity (Pyroscope queriers/compactor, or the Parca server) is the other side; size it to your read pattern (broad monthly cost reviews want more max_query_length headroom than day-to-day flame-graph reads).
• The ROI asymmetry is the whole point. A single fleet-wide review that returns tens of cores (as in the scenarios above) pays for years of profiling overhead and storage. Continuous profiling is one of the rare observability tools that reduces the total bill.

Rough monthly picture for the 600-pod, 3-cluster fleet in the scenario:

The sizing decision rule: start at 19 Hz, profile labeled targets only, 30-day retention on object storage, symbolize in CI. Scale query compute to your read pattern, and treat the whole thing as cost-saving infrastructure whose bill is dwarfed by the compute it lets you reclaim. Sample-rate escalation is the one knob that can turn cheap into expensive — keep it scoped.

Interview & exam questions

1. Why is a flame graph’s x-axis “share, not time,” and what does that forbid? A sampling profiler interrupts the CPU at a fixed rate and counts stacks; a frame’s width is the fraction of samples it appears in, an unbiased estimate of CPU-time share. It is not a timeline — left-to-right order is not sequence, and you cannot read a single request’s duration off it. Duration-of-one-request is a trace question; time-share-over-a-window is the profile’s answer.

2. Explain frame-pointer vs DWARF vs ORC unwinding and when each is used. Frame-pointer unwinding follows the rbp chain — cheap, but only works if the binary kept frame pointers (release builds usually omit them). DWARF/.eh_frame CFI is a per-PC table (present even in stripped binaries) that an agent flattens into BPF maps to unwind frame-pointer-less binaries in-kernel. ORC is the kernel’s own compact unwind format used for the kernel portion of a stack. eBPF agents use FP where available, DWARF-CFI otherwise for user stacks, and rely on ORC for kernel stacks.

3. A flame graph is full of [unknown]/hex frames under a native function. Broken agent? No — that is a symbolization failure: there is no debug info for that binary’s build-ID reachable by the symbolizer. Fix the build-ID (upload the .debug ELF in CI via parca-debuginfo upload, or set DEBUGINFOD_URLS, or enable the runtime’s perf map for JIT frames), not the agent. The agent unwound fine; it just can’t name the addresses.

4. Why does an eBPF profiler aggregate stacks in the kernel, and why does that matter? It increments a counter in a BPF hash map keyed by (pid, user_stack_id, kernel_stack_id) instead of shipping one event per sample to user space. This moves only distinct, pre-counted stacks across the kernel boundary, which is what keeps overhead near ~1% at high sample rates and fleet scale — the opposite of perf record, which copies every sample.

5. Difference between alloc_space and inuse_space, and the trap. alloc_space is total bytes allocated over the window — the driver of GC pressure/churn. inuse_space is live/retained memory right now — the leak signal. A function can dominate alloc_space (allocates lots of short-lived objects) yet contribute nothing to inuse_space. Concluding “leak” from an allocation profile is the classic mistake; confirm with inuse_space diffed over time.

6. A service is slow but its CPU flame graph is nearly empty. What’s happening? The cost is off-CPU — the threads are blocked on locks, disk, network, or epoll, and a CPU sampler by construction cannot see time when nothing is on-CPU. Switch to off-CPU profiling (eBPF sched_switch), a mutex/block profile, or read the trace. Widening the CPU profile won’t help; the latency isn’t in compute.

7. What is span-scoped profiling and how is it implemented? It’s a flame graph of just one span’s CPU time. Implementation: the SDK/runtime puts the active span_id/trace_id into the pprof labels (Go’s pprof.Do/pprof.Labels), so every CPU sample taken while that span is active carries it. Clicking a slow span in the trace UI then pulls the samples with that span_id. Caveat: trace_id is unbounded, so it’s carried on the sample for lookup, not used as a top-level storage series label.

8. Why 19 Hz fleet-wide instead of 100 Hz everywhere? Overhead and storage scale linearly with rate, but statistical error falls only as 1/sqrt(samples) — going 19→100 Hz (5.3×) roughly halves the error for 5× the cost, textbook diminishing returns. So run 19 Hz everywhere (cheap, good enough for hot-path and aggregate analysis) and crank a single namespace to 100 Hz only for a scoped, short investigation.

9. What does hostPID: true do for the profiler and what breaks without it? The perf event reports the host-namespace PID; hostPID: true lets the agent see the host PID namespace so it can map each sampled PID to the right container and binary. Without it, samples arrive but are unattributable — labeled with wrong or empty container/pod. It’s the most common “it runs but the data is useless” misconfiguration.

10. Why is pprof the format that ties this ecosystem together? pprof (profile.proto) is a compressed protobuf modeling a profile as samples (a stack + values + labels) over deduplicated location/function/string tables. Parca, Pyroscope, the Go runtime, and go tool pprof all speak it (OTLP profiles carry the same model), so a profile from any source interoperates — you can pull any of them into go tool pprof -diff_base to diff a regression.

11. How do you use continuous profiling to cut cloud cost? Query the cluster-wide aggregate CPU flame graph for a representative week, sort self-CPU leaves, convert share to cores (share × provisioned cores), and triage the top by cores-returned-vs-effort. Cross-service fat frames (a per-request regex compile, software crypto, allocation churn) are each deprovisionable cores. Fix, verify with a before/after diff, then lower replicas/node size. It’s the rare observability tool that reduces the bill.

12. What kernel and privilege requirements does an eBPF profiler have? Kernel 5.4+ (5.10+ comfortable for stable DWARF unwinding), BTF present for CO-RE (one agent across kernels), and either CAP_BPF+CAP_PERFMON(+CAP_SYS_PTRACE) or privileged: true to load programs, arm perf events, and read other processes’ maps — plus hostPID: true for attribution. Managed-cloud VMs often lack PMU access, so agents default to the software CPU clock.

These map to no single vendor exam but align with the CNCF observability body of knowledge (the CNCF OTel Certified Associate touches the profiling signal and correlation), and to SRE/platform-engineering interviews where performance analysis, eBPF, and cost optimization come up. The trace-correlation and OTLP-profiles material overlaps with OpenTelemetry certification content.

Quick check

1. You’re told a wide frame at the bottom of a flame graph is “the bottleneck.” Why is that usually wrong, and where should you look instead?
2. A stripped Rust service’s flame graph shows hex addresses at the leaves. Is the agent broken? What’s the fix?
3. Your Go service spends 30% of CPU in the garbage collector. Which profile type do you open, and which value do you sort by?
4. A service’s p99 is 400 ms but its CPU flame graph is almost empty. What kind of profiling (or signal) do you switch to, and why?
5. You want a flame graph of exactly one slow span. What has to be attached to the samples, and what’s the cardinality caveat?

Answers

1. The bottom frame is the entry point (main/thread root); it’s 100% wide because every sample descends through it, which says nothing. The cost is in its children — follow the width upward to the hot leaf and optimize that.
2. Not broken — it’s a symbolization failure: no debug info for that binary’s build-ID is reachable. Fix by uploading the .debug ELF for the build-ID in CI (parca-debuginfo upload) or pointing the symbolizer at a debuginfod server that has it. The agent unwound correctly; it just can’t name the addresses.
3. Open the allocation profile and sort by alloc_space (total bytes allocated) — GC pressure is driven by allocation churn, so the top allocation sites are what to attack (pool/reuse buffers). runtime.mallocgc prominent in the CPU profile is the pointer that sends you there.
4. Switch to off-CPU profiling (eBPF sched_switch) or read the trace — the latency is off-CPU (locks/IO/epoll), which a CPU sampler cannot see because nothing is on-CPU during the wait. Widening the CPU profile won’t reveal it.
5. The active span_id (and trace_id) must be attached to each CPU sample as pprof labels (via pprof.Do/the SDK). Caveat: trace_id is unbounded cardinality, so it’s carried on the sample for lookup — not used as a top-level storage series dimension you group by, or you explode the series count.

Glossary

• Continuous profiling — always-on, low-overhead, fleet-wide sampling of process stacks, stored keyed by time and labels so any past window is queryable.
• Sampling profiler — a profiler that interrupts the CPU at a fixed rate and records the current stack; frame width estimates CPU-time share, not time.
• perf event — a kernel counter (software CPU clock or hardware cpu-cycles) configured to fire at a frequency, driving the sampler.
• eBPF — sandboxed, verified programs run in the kernel; here, attached to a perf event to capture and aggregate stacks in-kernel.
• Stack unwinding — reconstructing the chain of callers from a leaf address back to the thread root.
• Frame pointer — the rbp-based frame chain; cheap to unwind but omitted by most release builds.
• DWARF CFI (.eh_frame) — per-PC call-frame information (present even when stripped) that agents flatten into BPF maps to unwind frame-pointer-less binaries.
• ORC — the Linux kernel’s own compact unwind format, used for kernel-side stacks.
• Symbolization — mapping an address to function + file:line, via in-binary tables, per-runtime unwinders, or build-ID → debug info.
• build-ID — a content hash in the ELF NT_GNU_BUILD_ID note that keys the matching debug info for a binary.
• debuginfod — an HTTP service that serves debug info (and sources) by build-ID.
• pprof — the profile.proto sample format (stack + values + labels over dedup’d tables) shared by Go, Parca, Pyroscope, and go tool pprof.
• Flame graph — stacked rectangles of aggregated samples; width = CPU share, y = stack depth, top edge = where the CPU was.
• Icicle graph — a flame graph drawn root-at-top; a top-down orientation.
• Differential (diff) flame graph — two windows compared, frames colored by change in share (red = worse, blue = better); the regression-hunting view.
• On-CPU profile — where cycles burn; blind to off-CPU time.
• Off-CPU profile — time threads spend descheduled (locks, IO, epoll), captured via eBPF sched_switch.
• alloc_space / alloc_objects — total bytes/objects allocated over a window (GC-pressure signal).
• inuse_space / inuse_objects — live/retained memory now (leak signal).
• Span-scoped profiling — CPU samples labeled with the active span_id/trace_id so you can render a flame graph of one span.
• Parca / Parca Agent — a pprof-native profiling server (FrostDB, server-side symbolization) and its eBPF DaemonSet.
• Grafana Pyroscope — an object-store-backed, horizontally scalable profiling database; fed by SDKs or Alloy’s pyroscope.ebpf.
• OpenTelemetry eBPF profiler — the vendor-neutral, multi-runtime eBPF profiler emitting the OTLP profiles signal.

Next steps

You can now deploy an eBPF profiler, symbolize correctly, read and diff flame graphs, correlate with traces, and turn CPU-share into cost savings. Build outward:

• Next: eBPF Zero-Code Instrumentation with Grafana Beyla & OpenTelemetry — the same eBPF, applied to RED metrics and spans, so your zero-touch telemetry and profiling share a foundation.
• Related: Distributed Tracing with Tempo & Jaeger: Exemplars & Correlation — the traces you click into for span-scoped flame graphs.
• Related: OpenTelemetry Metrics: Exemplars & Prometheus Trace Correlation — exemplar links that jump a metric spike straight to the trace and then the profile.
• Related: OpenTelemetry Collector Pipelines in Production — where the OTLP profiles signal is routed alongside metrics, logs, and traces.
• Related: Cilium Hubble: Network Flow Observability & Service Map — eBPF at the network layer, completing the eBPF observability picture.