Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code

Most Grafana dashboards die the same way. Someone drags forty panels onto a screen, every metric the exporter emits gets its own graph, the layout grows by accretion as people add “just one more” panel, and six months later it is the dashboard nobody opens during an incident because it answers no question. A dashboard is a tool for reasoning under pressure, not a museum of telemetry. The test is simple: when the pager fires at 03:00, does this dashboard get the on-call to a hypothesis in under a minute, or does it make them squint at thirty sparklines while the SLO burns? Almost every dashboard fails that test, for reasons that have nothing to do with Grafana and everything to do with method.

This article is the discipline that produces dashboards that pass. You will structure services with RED (Rate, Errors, Duration) and resources with USE (Utilization, Saturation, Errors) so every board shares a grammar a stranger can read; parameterize with template variables (every type, chained, multi-value with All, and $__rate_interval so rate() is correct at every zoom); pick the right panel type per question; reshape data with transformations; turn numbers into verdicts with thresholds; scale one board across a fleet with repeated rows and panels; and put the whole thing under version control as dashboards-as-code — file provisioning, the JSON model, Grafana git sync, the Terraform provider, and Grizzly — so it survives the person who built it and a stray “Save” can never delete it.

Everything here is concrete: real PromQL against real exporters (node_exporter, cAdvisor, a histogram-instrumented HTTP service), real provisioning YAML, Terraform HCL, JSON-model fields, and the gotchas a senior engineer learns the hard way — why [5m] returns gaps when you zoom out, why instance="$instance" silently breaks the instant a user multi-selects, why a leftover "id" collides across instances, and why terraform apply can succeed everywhere yet leave half your clusters with empty panels.

What problem this solves

The pain lands at exactly the wrong moment. A wall-of-vanity-panels board is fine in a planning meeting and useless in an incident, because under load the eye cannot scan thirty graphs and find the anomaly — the on-call ends up doing archaeology, clicking panel to panel to build the mental model the dashboard should have handed them. That is the difference between a five-minute incident and a two-hour one, and it is a design failure, not a skill failure.

What breaks is more than slow incidents. It is dashboard sprawl (one hardcoded board per service, dozens of near-identical JSON files drifting apart, thresholds that mean different things on different boards, a “golden” dashboard someone edited in the UI weeks ago that nobody can reproduce because the change lives only in Grafana’s database); alerts with no context (a latency spike with no deploy annotation, no trace link, no firing-alert overlay); and the bus-factor-of-one board the original author left behind as a black box. Who hits it: everyone running Grafana on Prometheus, Loki, Tempo, or a cloud metrics source. It bites hardest where there is no method, where boards are built by clicking (unversioned, irreproducible), and where one Grafana serves many teams (governance becomes an afterthought that turns into a fire). The fix is to start from a method, parameterize ruthlessly, and treat the JSON model as source code.

To frame the whole field before the deep dive, here is the spine of dashboard engineering — each decision, the question it answers, and the failure it prevents:

Learning objectives

By the end of this article you can:

• Structure any service dashboard with the RED method and any resource dashboard with the USE method, and explain why the discipline matters more than the exact queries.
• Write PromQL for Rate, Errors, Duration and for Utilization, Saturation, Errors against real exporters, using $__rate_interval so the queries stay correct at every time range.
• Define every template-variable type Grafana offers — query, custom, constant, interval, datasource, textbox, ad-hoc — chain them so choices narrow as you drill, and configure multi-value and the All option safely with regex matchers.
• Pick the right panel type for each question (time series, stat, gauge, bar gauge, table, heatmap, state timeline, logs, node graph) and know when each is the wrong choice.
• Apply thresholds, value mappings, and field overrides to turn raw series into a glance-readable verdict, and reshape data with transformations instead of contorting PromQL.
• Scale one dashboard across a fleet with repeated rows and panels driven by a multi-value variable, and add data links, drilldowns, exemplars, and annotations that turn a symptom into a one-click jump to the cause.
• Put dashboards under version control as code via file provisioning, the JSON model, Grafana git sync, the Terraform provider, and Grizzly — and pick the right delivery mechanism for who owns the dashboard.

Prerequisites & where this fits

You should be comfortable with Prometheus as a metrics backend — the data model (metric name plus labels), rate() over a counter, and the idea of a histogram with _bucket/_sum/_count series. You should know your way around the Grafana UI enough to add a panel and pick a data source, and be able to run kubectl or docker for the hands-on lab. Basic familiarity with Terraform (a provider, a resource, plan/apply) helps for the dashboards-as-code section but is not assumed.

This sits in the Observability track, downstream of the metrics fundamentals and upstream of alerting. If your PromQL is shaky, read PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work first — every panel here is a PromQL query, and getting rate() and histogram_quantile() right is the foundation. This article focuses on the dashboard side; the deeper provisioning-and-alerting-as-code story lives in Grafana as Code: Provisioning Dashboards, Folders, and Unified Alerting with Terraform, and the exemplar click-through wiring is in Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation. A good dashboard is the front end of an observability stack; the back ends are Prometheus/Mimir for metrics, Loki for logs, and Tempo for traces.

A quick map of where each concern lives, so you know which layer to reach for:

Core concepts

Five ideas make every later decision obvious.

A panel answers a question; a dashboard answers a sequence of them. The unit of design is not the metric, it is the question. “Is the service healthy right now?” is one panel (an error-ratio stat with a threshold). “If not, is it a uniform slowdown or a long tail?” is another (a latency panel with p50/p90/p99). A dashboard is a deliberate path through those questions, top to bottom, so the reader’s eye moves from “is something wrong” to “where” to “what changed.” A metric that answers no question on the current board does not belong on it.

A method decides which panels exist so you don’t have to. Left to taste, two engineers build two different dashboards for the same service and neither can read the other’s. RED and USE remove the taste: for every service, draw exactly three panels (Rate, Errors, Duration); for every resource backing it, draw exactly three (Utilization, Saturation, Errors). The method is a grammar — once every dashboard shares it, a stranger can read any of them, and you stop debating which metrics to add because the method already decided.

Template variables turn one dashboard into N dashboards. Hardcoding service="checkout" into every panel means one JSON file per service and dozens of near-duplicates to maintain. A template variable is a placeholder ($service) the panels interpolate at render time; switch its value in a dropdown and every panel repoints. A query variable populates its dropdown from live label values in Prometheus, so newly deployed services appear automatically. That is the difference between a dashboard you maintain and a template that maintains itself.

The rate window must scale with the time range. rate() needs a lookback window wide enough to contain at least two scrape samples, or it returns nothing; too wide and it smears over the very spike you are chasing. A hardcoded [5m] is right at one zoom level and wrong at every other. $__rate_interval is a Grafana macro that computes a safe window from the panel’s scrape interval and the current time range, so the same query is correct whether you are looking at the last ten minutes or the last week. Hardcoded windows are the single most common reason a panel “works on my screen” and shows gaps on someone else’s.

A dashboard built by clicking is a dashboard one Save away from gone. Grafana stores dashboards as a JSON model in its database. If that JSON exists only in the running instance, it is unversioned, unreviewable, undeletable-by-accident, and irreproducible — lose the database, lose the dashboards. Dashboards-as-code means the JSON is the source of truth, lives in Git, and is delivered to Grafana by an automated mechanism (file provisioning, git sync, Terraform, Grizzly). The dashboard becomes a reviewed artifact like any other code, and “who changed this and why” has an answer.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

The RED method for services

RED, popularized by Tom Wilkie, applies to anything that serves requests — HTTP services, gRPC endpoints, queue consumers, anything with an inbound request you can count. Three signals:

• Rate — requests per second.
• Errors — failed requests per second, or (better) the error ratio.
• Duration — the latency distribution, viewed at percentiles.

Why these three and not more: they are the symptoms a user feels. A user does not feel your goroutine count; they feel slow (Duration), broken (Errors), or — at the aggregate level — they generate the Rate that everything else scales with. RED is a symptom-based method, which is exactly what you want at the top of an incident: start from what is wrong for the user, then drill into resources (USE) to find why.

Assume a Prometheus counter http_requests_total with a code label, and a histogram http_request_duration_seconds with the usual _bucket/_sum/_count series. The three core panels, scoped to a $service variable:

# RATE — requests per second, summed across all instances of the service
sum(rate(http_requests_total{service="$service"}[$__rate_interval]))

# ERRORS — 5xx as a fraction of all requests (the error RATIO, not a raw count)
sum(rate(http_requests_total{service="$service", code=~"5.."}[$__rate_interval]))
/
sum(rate(http_requests_total{service="$service"}[$__rate_interval]))

# DURATION — p99 latency from a classic histogram
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{service="$service"}[$__rate_interval]))
)

A few rules that separate a RED panel that works from one that lies:

Prefer the error ratio over the raw error rate. A raw “5 errors/sec” means nothing without the denominator — 5/sec out of 10,000 is healthy, out of 50 is an outage. The ratio is self-normalizing and maps directly onto an SLO (“error budget is 0.1% — alert at 0.1% error ratio”). Set the panel’s unit to Percent (0.0–1.0) so Grafana renders 0.012 as 1.2%.

Plot p50, p90, and p99 as separate series on the Duration panel. The gap between them is the diagnosis. If p50 and p99 rise together, you have a uniform slowdown — something systemic (a slow dependency hitting every request, CPU saturation). If p50 is flat and p99 explodes, you have a long-tail problem hitting a subset — a slow shard, a GC pause, a single bad instance. Those are two completely different investigations, and a single “average latency” line hides the distinction entirely. Averages are a lie for latency; never plot _sum / _count as your primary latency view.

Aggregate with sum by (le) for histogram_quantile. The quantile function needs the per-bucket rate grouped by the le (less-than-or-equal) bucket boundary. Forgetting the by (le) or aggregating away le gives nonsense. If you run native histograms (the newer Prometheus format), the query simplifies — histogram_quantile(0.99, sum(rate(http_request_duration_seconds[$__rate_interval]))) with no by (le) — but the data source and Prometheus must support them.

The RED panel set, mapped to question, panel type, and the gotcha each one hides:

A complete RED row is therefore three time-series panels plus, optionally, a stat panel that distills the current error ratio into one big number with a threshold — the “is it on fire” glance before you read any graph.

The USE method for resources

USE, from Brendan Gregg, applies to resources that have finite capacity — CPU, memory, disk, network interfaces, connection pools, thread pools. Three signals:

• Utilization — the fraction of time (or capacity) the resource was busy.
• Saturation — the degree of queued, un-serviced work waiting on the resource.
• Errors — error events on the resource itself (disk errors, dropped packets, failed allocations).

Where RED is symptom-based (what the user feels), USE is cause-based (where the bottleneck is). The two are a pair: RED tells you the service is slow; USE tells you the CPU is saturated, the disk is at 100% utilization, or the connection pool is exhausted. Saturation is the signal people forget and the one that predicts pain — a resource at 70% utilization with a growing run queue is in more trouble than one at 95% utilization with zero queue.

For a Kubernetes node using node_exporter and cAdvisor, scoped to an $instance variable:

# UTILIZATION — CPU busy fraction (1 minus idle)
1 - avg(rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[$__rate_interval]))

# UTILIZATION — memory used fraction
1 - (
  node_memory_MemAvailable_bytes{instance=~"$instance"}
  /
  node_memory_MemTotal_bytes{instance=~"$instance"}
)

# SATURATION — run-queue pressure: load average relative to CPU count
node_load1{instance=~"$instance"}
/
count(count by (cpu) (node_cpu_seconds_total{instance=~"$instance"}))

# SATURATION — disk I/O time fraction (the disk was busy this long)
rate(node_disk_io_time_seconds_total{instance=~"$instance"}[$__rate_interval])

# ERRORS — network receive errors per second
rate(node_network_receive_errs_total{instance=~"$instance"}[$__rate_interval])

For a container (pod) using cAdvisor, the USE signals shift to the cgroup metrics:

# UTILIZATION — pod CPU as a fraction of its limit
sum(rate(container_cpu_usage_seconds_total{pod="$pod", container!=""}[$__rate_interval]))
/
sum(kube_pod_container_resource_limits{pod="$pod", resource="cpu"})

# SATURATION — CPU throttling: the smoking gun of a too-low CPU limit
rate(container_cpu_cfs_throttled_periods_total{pod="$pod"}[$__rate_interval])
/
rate(container_cpu_cfs_periods_total{pod="$pod"}[$__rate_interval])

# UTILIZATION — memory working set vs limit (the OOMKill predictor)
container_memory_working_set_bytes{pod="$pod", container!=""}
/
kube_pod_container_resource_limits{pod="$pod", resource="memory"}

The discipline matters more than the exact queries. For every resource backing a service, draw the three USE panels; the method decides which metrics, so you never debate it. The USE signal set per common resource, with the metric and the panel that surfaces it:

Putting RED and USE together gives the canonical incident dashboard shape: a RED row at the top (is the service healthy?), then a USE row per backing resource (if not, where is the bottleneck?). The reader’s eye flows from symptom to cause, top to bottom, exactly as the investigation should.

Template variables: every type, chained and multi-value

A template variable is a $name placeholder that panels interpolate at render time. Defined under Dashboard settings → Variables, each has a type. Knowing all of them — not just query variables — is what lets you build dashboards that adapt to data, to the user, and to the time range.

The variable types

Grafana offers several variable types, each solving a different problem:

A query variable for service pulls live values from Prometheus with the label_values helper:

# In a Query-type variable named "service":
label_values(http_requests_total, service)

A custom variable for percentile lets the reader pick which quantile the Duration panel shows, without editing the query:

# Custom variable "percentile", values:
0.5 : p50, 0.9 : p90, 0.99 : p99

The value : label syntax shows a friendly label in the dropdown (p99) while interpolating the real value (0.99) into histogram_quantile($percentile, ...).

A datasource variable named ds (type Datasource, filtered to Prometheus) lets one dashboard work against any Prometheus — dev, staging, prod, or a per-cluster instance — by setting each panel’s data source to ${ds}. This is the cleanest way to ship one dashboard that works across thirty clusters without thirty copies.

Chained variables — choices that narrow as you drill

Make a variable’s query reference another variable, and Grafana re-runs it whenever the parent changes. An instance variable that depends on the selected service:

# In the "instance" variable — note the $service reference:
label_values(http_requests_total{service="$service"}, instance)

Add namespace → service → instance chaining and the dropdowns cascade: pick a namespace, the service list narrows to that namespace, pick a service, the instance list narrows to that service. This is what makes a single dashboard navigable across a large fleet — you are not scrolling a thousand-item instance list, you are drilling.

# namespace (top of the chain)
label_values(kube_pod_info, namespace)

# service — narrowed by namespace
label_values(http_requests_total{namespace="$namespace"}, service)

# instance — narrowed by service
label_values(http_requests_total{namespace="$namespace", service="$service"}, instance)

Multi-value and the All option — and the regex trap

Two toggles on a query variable change how it interpolates:

• Multi-value lets the user select several values at once.
• Include All option adds an All entry that means “every value.”

The trap: the moment a variable can hold more than one value, an equality matcher breaks. instance="$instance" works for a single selection but produces instance="a|b|c" for a multi-selection — which matches a literal label value a|b|c that does not exist, so the panel goes empty. The fix is to use a regex matcher everywhere a variable can be multi-valued: instance=~"$instance". Grafana interpolates a multi-value variable as a regex alternation (a|b|c), and =~ treats it correctly.

The All option has a matching subtlety. By default, All interpolates as every concrete value joined by |, which is correct with =~. But on a high-cardinality label this produces a huge regex. Set a Custom all value of .* and use it with =~, so All interpolates to the pattern .* (match everything) rather than enumerating thousands of values — far cheaper for Prometheus to evaluate.

The variable settings that make dashboards pleasant to live with, and what each one prevents:

There is also a deprecated detail worth knowing: when All is selected and you have not set a custom all value, some panels and older setups need the matcher to handle the literal string. Always prefer =~ plus a .* custom all value — it is the combination that never surprises you.

Variable interpolation formats

When a variable is interpolated into something other than a PromQL matcher — a URL, a legend, a panel title — the format matters. Grafana supports format modifiers: ${var:regex}, ${var:pipe}, ${var:csv}, ${var:json}, ${var:percentencode}. For a data link URL, ${service:percentencode} URL-encodes the value; for a legend, $service raw is fine. The common bug is interpolating a multi-value variable into a place that expects one value — guard those with a format modifier or scope the link to a single-value variable.

The rate window: why $__rate_interval and not [5m]

This deserves its own section because it is the single most common reason a panel “works on my machine” and shows gaps on someone else’s. rate() computes per-second change over a lookback window. That window must contain at least two scrape samples, or rate() returns nothing for that step. With a 15-second scrape, a 5-second window is too small and returns gaps; with a 60-second scrape, even a 1-minute window is borderline.

But the right window also depends on the time range you are viewing. When you zoom out to a week, Grafana asks Prometheus for fewer, wider steps; a 5-minute window now smears over hours and hides the spike. When you zoom in to ten minutes, the same 5-minute window is fine. A hardcoded window cannot be right for both.

$__rate_interval is Grafana’s answer: it computes a window from the panel’s configured scrape interval and the current step, guaranteeing at least four samples in the window while scaling up as you zoom out. The rule of thumb baked in is roughly max(4 × scrape_interval, $__interval + scrape_interval). You do not compute it; you just write [$__rate_interval] and it is correct at every zoom level.

The macros Grafana exposes, what each means, and when to use which:

The discipline: $__rate_interval for rate() in interactive panels; literal windows only in recording rules and alert expressions (where the evaluation interval is fixed and known). If you find yourself typing [5m] into a dashboard panel, stop — that is the hardcoded-window bug in the making.

Panel types: pick the visualization that answers the question

Grafana has many panel types and most dashboards over-use the time-series graph. The right panel makes the answer obvious; the wrong one buries it. Choose by the question, not by habit.

Two choices people get wrong most often:

Stat vs Time series for “current health.” The error ratio over time belongs on a time-series panel — but the current error ratio, the thing the on-call glances at first, belongs on a Stat panel: one huge number, a threshold coloring it green/amber/red, and a sparkline for trend. A stat panel turns “read the right edge of the graph and compare to the axis” into “the number is red.” Put the stat at the top-left; it is the first thing the eye lands on.

Heatmap vs percentile lines for latency. Three percentile lines (p50/p90/p99) are a great summary, but they discard the shape of the distribution. A heatmap of the histogram buckets shows the full distribution over time — you can see a bimodal latency (two bands, e.g. cache hits vs misses) that percentile lines flatten into one misleading curve. Use percentile lines as the default and add a heatmap when you need to see the distribution’s shape.

Thresholds, value mappings, and field overrides

Raw series are noise until you give them meaning. Three features in the panel’s Field config do the heavy lifting.

Thresholds turn a number into a verdict by coloring the panel at value boundaries. On the error-ratio panel, set the unit to Percent (0.0–1.0) and add thresholds derived from your SLO — for example green below 1%, amber at 1%, red at 5%. Now a glance tells you the state without reading the axis. Thresholds can color the line, the area, the background, or a threshold band; for a Stat panel, color the background so the whole tile goes red.

Value mappings convert numbers (or ranges, or special values) to text and color. A panel showing the up metric returns 1 or 0; map 1 → "UP" (green) and 0 → "DOWN" (red). Kubernetes pod-phase enums, HTTP status classes, and boolean health all become legible the same way. Value mappings also handle Null and NaN — map Null → "no data" with a gray background so a gap reads as “no data,” not “zero.”

Field overrides apply config to specific series matched by name, regex, or query. On a Duration panel with p50/p90/p99, override the p99 series to render thicker and in red, p50 thin and gray — so the eye immediately weights the tail. Overrides also fix units per series (one panel showing both bytes and a ratio), set custom display names, and hide a noisy series without removing it from the query.

The field-config tools and when each applies:

A small but high-value pattern: set min: 0, max: 1 on any “fraction” gauge (utilization, error ratio) so the gauge is always scaled to the meaningful 0–100% range, not auto-scaled to the data’s current min/max — auto-scaling makes a quiet 2% utilization look full.

Transformations: reshape data without re-querying

Transformations reshape the result of a query, in the panel, after Prometheus returns it — no second query, no PromQL contortion. They run in order, top to bottom, and each operates on the output of the previous. The ones you reach for constantly:

The decision rule: reshape in PromQL when it changes what Prometheus computes; transform in the panel when it only changes presentation. Computing an error ratio is genuinely a PromQL operation (you want Prometheus to divide the rates). But renaming instance to “Host,” dropping a series from the legend, sorting a table, or turning labels into columns is presentation — do it with a transformation so the query stays a clean, reusable expression. Over-using transformations to do real math that PromQL should do makes panels slow and opaque; under-using them leads to ugly tables and duplicate queries.

A worked example — a “Top 10 slowest endpoints right now” table from a single histogram query:

# One query: p99 per route
histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{service="$service"}[$__rate_interval]))
)

Then, in the Transform tab, chain: Reduce (calculation: Last, to get the current p99 per route) → Labels to fields (so route becomes a column) → Organize fields (rename, reorder) → Sort by (the p99 value, descending) → Limit (10). The result is a clean, sorted top-10 table built from one query, with zero extra PromQL.

Repeating rows and panels: one dashboard, a whole fleet

You want to show the same RED panels for every service, or the same USE panels for every instance — without hand-placing N copies. Repeating does this: bind a panel or a row to a multi-value variable, and Grafana renders one copy per selected value at render time.

Repeat a panel: in the panel’s options, set Repeat by to a multi-value variable (e.g. instance). Select three instances and the panel renders three times, each scoped to one instance, the variable resolving to a single value inside each copy. Set the max per row so they tile neatly.

Repeat a row: a row can repeat by a variable, rendering the entire row of panels once per value. This is how you build “a USE row per node” — pick five nodes, get five identical USE rows, each labeled with its node. Row repeating is the heavier hammer; use it when a group of panels should multiply, panel repeating when a single panel should.

The mechanics and trade-offs:

The discipline that keeps repeating safe: cap the variable. A repeat bound to an unbounded multi-value variable with All selected on a thousand-instance label will try to render a thousand panels and melt the browser. Use a Limit on the variable, encourage the reader to select a handful, or scope the repeat to a naturally small dimension (regions, not instances). Repeating is a force multiplier; an uncapped one is a footgun.

Inside a repeated panel, reference the loop value with the variable name — Grafana scopes it to the single value for that copy, so instance="$instance" (or =~) inside a panel repeated by instance correctly targets just that one. Title the panel $instance — CPU so each copy self-labels.

Links and drilldowns: from symptom to cause in one click

This is where dashboards stop being passive read-outs and become investigation tools. Four mechanisms get the on-call from “something is wrong here” to “here is exactly why” without leaving Grafana to go hunting.

Data links

A data link is a templated URL attached to a panel (or a specific field) that interpolates series labels and the clicked time range. On the Duration panel, a link to the logs for that exact service and window turns “latency is high” into “here are the logs from the slow window.” The link uses Explore’s URL schema:

/explore?left={"datasource":"loki","queries":[{"expr":"{service=\"${__field.labels.service}\"}"}],"range":{"from":"${__from}","to":"${__to}"}}

Grafana substitutes ${__field.labels.service} from the hovered series and ${__from}/${__to} from the time range. The exact Explore URL schema shifts between major versions, so build one link in the UI, then copy what Grafana generates rather than hand-writing the JSON from memory. The interpolation tokens you have available:

Exemplars — latency spike to the exact trace

Exemplars are sampled trace IDs that Prometheus attaches to histogram observations when your instrumentation emits them. With exemplar storage enabled in Prometheus and the app emitting exemplars (via OpenTelemetry or the Prometheus client), Grafana renders them as diamonds on the latency graph. Click one and jump straight to the trace in Tempo or Jaeger — the single most powerful drilldown in the stack, because it links an aggregate (p99 is high) to a concrete instance (this exact slow request).

Wire it in the Prometheus data source. In provisioned YAML:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      exemplarTraceIdDestinations:
        - name: trace_id            # must match the label your exemplars carry
          datasourceUid: tempo      # must match the uid of your tracing data source

The datasourceUid must match the uid of your tracing data source, and name must match the label your exemplars carry (commonly trace_id). Then enable the Exemplars toggle on the panel’s query. Now a p99 spike is one click from the request that caused it. The full wiring — instrumentation, storage, destinations — is covered in Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation and the tracing side in Distributed Tracing End-to-End: Context Propagation, Tempo, and Correlating Traces with Metrics and Logs.

Annotations — overlaying events on the time axis

Annotations mark events as vertical lines or regions on every panel. The most valuable annotation is deploys: if your CI emits a Prometheus metric or writes a marker, an annotation query overlays a line at each release, so “latency jumped at 14:32” immediately reads as “latency jumped right after the 14:31 deploy.” Add the annotation query in dashboard settings, backed by your deploy metric, and every panel gets the markers.

Alert-state annotations shade the window an alert was firing directly on the relevant graph, so the firing window visibly lines up with the error climb — removing any doubt about which symptom the alert is catching.

Panel and dashboard links

Beyond data links, panels and dashboards carry links (top-right of the panel/dashboard) to related boards — the service dashboard links to the node dashboard for its cluster, which links to the cost dashboard. Combined with variable pass-through (carry $service into the linked dashboard’s URL), this builds a navigable web of dashboards instead of thirty disconnected boards.

Dashboards as code

A dashboard built by clicking is unversioned, unreviewable, and one accidental “Save” or one lost database from gone. Treat the JSON model as the source of truth, put it in Git, and deliver it to Grafana by an automated mechanism. There are several mechanisms; pick by who owns the dashboard and how the rest of your infrastructure is managed.

The delivery mechanisms, side by side:

File-based provisioning

For dashboards the platform team owns, drop the JSON on disk and point Grafana at it. A provisioning config tells Grafana where to look:

# /etc/grafana/provisioning/dashboards/platform.yaml
apiVersion: 1
providers:
  - name: platform
    orgId: 1
    folder: Platform
    type: file
    disableDeletion: true
    editable: false
    allowUiUpdates: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

editable: false and allowUiUpdates: false make the dashboard read-only in the UI, forcing every change through the repo. foldersFromFilesStructure: true mirrors your directory layout into Grafana folders, keeping the filesystem and UI in sync. Grafana watches the path and reloads on change — no restart needed. The provisioning options that matter:

Strip the top-level "id" field from exported JSON before committing it. A stale numeric id from the source instance can collide on the target. Keep the "uid" — it is the stable, portable identifier Grafana uses to match dashboards across instances. Setting a deterministic uid is what makes provisioning idempotent.

The JSON model — the fields that matter

You rarely hand-write the whole JSON, but you must know the load-bearing fields to review a diff and to debug a provisioning failure. The structure:

{
  "uid": "service-red",
  "title": "Service · RED",
  "schemaVersion": 39,
  "version": 7,
  "tags": ["red", "service"],
  "templating": { "list": [ /* variables */ ] },
  "panels": [ /* each panel: type, gridPos, targets, fieldConfig */ ],
  "annotations": { "list": [ /* deploy markers, alert state */ ] },
  "time": { "from": "now-6h", "to": "now" },
  "refresh": "30s"
}

The fields you must understand:

Grafana git sync

Newer Grafana supports native git sync — connecting a dashboard folder to a Git repository so changes flow both ways: edit in the UI and Grafana commits to the repo; push to the repo and Grafana pulls. This is the middle path between “locked, files-only” provisioning and “UI is the source of truth.” It gives application teams the comfort of editing visually while still getting Git history, review, and rollback. The trade-off is that two-way sync needs clear conventions about who edits where, or you get merge conflicts between UI edits and repo commits.

The Terraform provider

For dashboards that live alongside infrastructure, or when you want folders, permissions, data sources, and alerts managed in one place, use the Grafana Terraform provider:

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = "~> 3.0"
    }
  }
}

provider "grafana" {
  url  = "https://grafana.example.com"
  auth = var.grafana_service_account_token   # service account, NOT a legacy API key
}

resource "grafana_folder" "platform" {
  title = "Platform"
}

resource "grafana_dashboard" "service_red" {
  folder      = grafana_folder.platform.id
  config_json = file("${path.module}/dashboards/service-red.json")
}

# Folder permissions managed as code — access control in the same review process
resource "grafana_folder_permission" "platform" {
  folder_uid = grafana_folder.platform.uid
  permissions {
    role       = "Viewer"
    permission = "View"
  }
  permissions {
    team_id    = grafana_team.platform.id
    permission = "Edit"
  }
}

Authenticate with a service account token, not a legacy API key — Grafana deprecated API keys in favor of service accounts. Keep the dashboard JSON in its own file and reference it with file() so designers can still edit visually in a scratch instance, export, and commit the result. The provider diffs the JSON on every terraform plan, so drift from someone editing in the UI shows up as a plan change you can revert. The deeper end-to-end Terraform-and-alerting workflow is in Grafana as Code: Provisioning Dashboards, Folders, and Unified Alerting with Terraform.

Grizzly (grr)

Grizzly is a kubectl-style CLI that manages Grafana resources (dashboards, folders, datasources, alert rules) from YAML/JSON files. The workflow is familiar to anyone who uses Kubernetes:

# Pull current state into files
grr pull dashboards/

# See what would change before applying
grr diff dashboards/service-red.json

# Apply the desired state
grr apply dashboards/service-red.json

# Watch a file and live-apply on save (great for authoring)
grr watch ./dashboards dashboards/service-red.json

Grizzly shines in GitOps pipelines: the repo holds desired state, CI runs grr diff on a PR and grr apply on merge. It pairs well with Jsonnet/grafonnet to generate the dashboard JSON from code, so fifty similar dashboards come from one parameterized template rather than fifty hand-maintained files. The tooling-vs-ownership decision:

Library panels and folders for governance

Across many teams, copy-paste is the enemy of consistency. Library panels let you define a panel once — the canonical RED error-ratio panel with the right thresholds and units — and reuse it across dashboards. Edit the library panel and every dashboard embedding it updates. That is how you roll out a threshold change to fifty dashboards without touching fifty files.

Folders are the unit of permission. Grant a team edit rights on its folder and viewer rights elsewhere. Combined with provisioning, you get a clean model: the platform team owns provisioned, read-only golden dashboards in shared folders, while product teams build freely in their own folders. Folder permissions are themselves manageable through Terraform, so access control lives in the same review process as everything else. The governance building blocks:

Architecture at a glance

Picture the full path a dashboard sits on, left to right, because every design decision maps onto a hop in it. On the far left are your applications, instrumented to emit two kinds of telemetry: counters and histograms for metrics (the http_requests_total counter and the http_request_duration_seconds histogram that feed RED), and spans for traces, each span carrying a trace_id that also rides along as an exemplar on the matching histogram observation.

Those metrics are scraped by Prometheus (or written to Mimir for scale), which stores the time series plus the exemplars. The spans flow to Tempo. This is the data layer — the source of truth for every number a panel shows. Critically, the dashboard never invents data; it only asks questions of this layer, which is why a panel’s correctness depends on the backend’s version and feature flags (native histograms, exemplar storage) as much as on the PromQL.

In the middle sits Grafana, and inside Grafana the dashboard itself is a layered object. At the base are data sources (the Prometheus, Tempo, and Loki connections, each with a stable uid). On top sit template variables — $namespace → $service → $instance chained, multi-value where it helps — which scope every query. Then the panels: a RED row (Rate, Errors, Duration time-series panels plus an error-ratio Stat), and USE rows that repeat per node or pod from a multi-value variable. Each panel carries field config (units, thresholds, value mappings) that turns its raw numbers into a colored verdict, and links (data links to Loki logs, exemplar diamonds to Tempo traces) that turn a symptom into a one-click jump. Annotations — deploy markers, alert-firing windows — overlay every panel from queries against the same backend.

On the right is the human at 03:00 and, equally important, the delivery pipeline that put the dashboard there. The dashboard’s JSON model lives in Git, not just in Grafana’s database. A delivery mechanism — file provisioning, native git sync, the Terraform provider, or Grizzly — reconciles that JSON into Grafana, so the running dashboard is a reproducible artifact, version-controlled, reviewed, and impossible to lose to a stray Save. Read the whole picture as a loop: instrumentation feeds the backend, the backend feeds the panels, the panels (via thresholds and links) feed the human’s decision, and the JSON-in-Git feeds the running dashboard — close every link in that loop and you have a dashboard that gets used and survives.

Real-world scenario

A payments platform team — call them Lumio — ran one Grafana per cluster across 32 EKS clusters, all driven by the Terraform provider, with a single “golden” RED dashboard stamped into every cluster’s Platform folder. The dashboard was clean: a RED row per service, USE rows repeating per node, exemplars wired to Tempo, deploy annotations from their CI. By every measure it was a well-engineered, dashboards-as-code setup. Then they shipped what looked like a trivial improvement and learned that provisioning makes dashboards reproducible, not correct.

The change: switch the Duration panels from classic-histogram histogram_quantile(0.99, sum by (le) (rate(..._bucket[$__rate_interval]))) to the simpler native-histogram form histogram_quantile(0.99, sum(rate(...[$__rate_interval]))). They edited the JSON, opened a PR, terraform plan showed the dashboard JSON changing identically across all 32 clusters, and terraform apply succeeded everywhere. Green across the board. They closed the ticket.

Within an hour, the on-call for half the fleet reported empty Duration panels. Not errors — just no data. The RED Rate and Errors panels were fine; only Duration was blank, and only on roughly sixteen clusters. The instinct was “the query is wrong,” but the query was identical to the sixteen clusters where it worked. The difference was not in Grafana at all: native histograms require --enable-feature=native-histograms on the Prometheus side, and only the clusters already on the newer Prometheus release had the flag. On the older clusters, the _bucket series still existed (classic histogram) but there was no native histogram for the simplified query to read, so histogram_quantile over sum(rate(...)) returned nothing. The terraform plan diffed clean because the dashboard JSON was identical; the data backend was not.

The fix had two parts. First, the immediate one: revert the sixteen lagging clusters to the classic-histogram JSON. They did this with a Terraform for_each over a capability map — a per-cluster table of which Prometheus features were live — so each cluster got the dashboard variant its backend could actually serve, classic or native. Second, the durable lesson: they added a backend capability gate to CI, querying each cluster’s Prometheus build-info and feature flags before applying, and failing the apply for any cluster whose backend could not serve the new query.

# CI gate: fail the apply if a target cluster's Prometheus lacks the required feature.
for ctx in $(kubectl config get-contexts -o name); do
  url="https://prometheus.${ctx}.internal"
  # Does this Prometheus expose native histograms? (probe a known native series)
  has_native=$(curl -s "${url}/api/v1/query" \
    --data-urlencode 'query=http_request_duration_seconds' \
    | jq '.data.result | map(select(.histogram != null)) | length')
  if [ "${has_native:-0}" -eq 0 ]; then
    echo "SKIP-NATIVE: ${ctx} has no native histograms; pin classic-histogram dashboard"
  fi
done

The takeaway Lumio wrote into their runbook: a dashboard’s PromQL has an implicit contract with the data source’s version and feature flags, and version-skew across a fleet will bite even when every plan is green. Dashboards-as-code gives you reproducibility — the same JSON everywhere — but reproducibility is not correctness when the backends differ. They kept the capability map permanently, and every fleet-wide dashboard change since has been gated on what each cluster’s backend can actually serve.

Advantages and disadvantages

Engineering dashboards with method and code is not free; it trades up-front rigor for long-term sanity. The honest two-column view:

When each side matters: the advantages dominate at scale — many teams, many services, many clusters, where consistency and reproducibility are survival. The disadvantages bite hardest for a single small team with one dashboard, where the ceremony of Terraform and capability maps can outweigh the benefit; there, file provisioning of a couple of hand-built boards is plenty. Match the rigor to the blast radius: a personal scratch dashboard does not need a pipeline, but a board stamped into 32 clusters absolutely does, and Lumio’s empty-panel hour is the cost of skipping the capability gate.

Hands-on lab

Build a real RED dashboard against a local Prometheus, parameterize it, then provision it as code — entirely on your laptop with Docker. No cloud spend. Roughly 30–40 minutes.

Step 1 — Stand up Prometheus + Grafana + a metrics target. Create docker-compose.yml:

services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml"]
  node-exporter:
    image: prom/node-exporter:latest
    ports: ["9100:9100"]
  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./provisioning:/etc/grafana/provisioning
      - ./dashboards:/var/lib/grafana/dashboards

And prometheus.yml:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: prometheus
    static_configs: [{ targets: ["localhost:9090"] }]
  - job_name: node
    static_configs: [{ targets: ["node-exporter:9100"] }]

mkdir -p provisioning/dashboards provisioning/datasources dashboards
docker compose up -d
# Expected: three containers running; Grafana on http://localhost:3000
docker compose ps

Step 2 — Provision the Prometheus data source as code. Create provisioning/datasources/prom.yaml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST

docker compose restart grafana
# Expected: Grafana > Connections > Data sources shows "Prometheus", uid "prometheus"

Step 3 — Build a USE panel by hand (we have node_exporter, so USE is the natural fit). In the UI (http://localhost:3000), create a new dashboard, add a Time series panel, data source Prometheus, query:

1 - avg(rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval]))

Set the unit to Percent (0.0–1.0), add thresholds (green < 0.7, amber 0.7, red 0.9). Expected: a CPU-utilization line, colored by threshold. Note you typed $__rate_interval, not [5m].

Step 4 — Add a template variable. Dashboard settings → Variables → New → type Query, name instance, data source Prometheus, query:

label_values(node_cpu_seconds_total, instance)

Turn on Multi-value and Include All option, set Custom all value to .*. Now change the panel query to use it with a regex matcher:

1 - avg(rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[$__rate_interval]))

Expected: an instance dropdown appears; selecting values repoints the panel. Confirm =~ works with “All” (it interpolates to .*).

Step 5 — Repeat the panel per instance. Panel options → Repeat options → Repeat by instance, max per row 2. Select two instances (if you only have one, add a second node-exporter target to see it). Expected: one panel per selected instance, each scoped to its own.

Step 6 — Export the JSON model and clean it. Dashboard settings → JSON Model → copy it into dashboards/use-node.json. Then strip the id and set a deterministic uid:

# Remove the per-instance numeric id; pin a stable uid
jq 'del(.id) | .uid = "use-node"' dashboards/use-node.json > dashboards/use-node.tmp \
  && mv dashboards/use-node.tmp dashboards/use-node.json
jq '{uid, title, id}' dashboards/use-node.json
# Expected: uid "use-node", a title, and id is null (stripped)

Step 7 — Provision it as code. Create provisioning/dashboards/local.yaml:

apiVersion: 1
providers:
  - name: local
    orgId: 1
    folder: Local
    type: file
    disableDeletion: true
    editable: false
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

docker compose restart grafana
# Validate the JSON first
jq empty dashboards/use-node.json && echo "valid JSON"
# Expected: a "Local" folder containing the dashboard, now READ-ONLY (no Save in UI)

Validation checklist. You built a USE panel with $__rate_interval (not a hardcoded window), parameterized it with a multi-value query variable using a =~ matcher and a .* custom-all-value, repeated it per instance, exported and cleaned the JSON (stripped id, set uid), and provisioned it as a locked, code-owned dashboard. What each step proves:

Teardown.

docker compose down -v
# Removes containers and volumes; nothing left running, no charges.

Common mistakes & troubleshooting

The failures that turn a good dashboard into a misleading one — symptom, root cause, how to confirm, and the fix. Read the table at speed, then the expanded notes for the ones that bite hardest.

The expanded reasoning for the entries that bite hardest:

1 & 2 — the two interpolation footguns. Hardcoded rate windows and equality-on-multi-value are the most common dashboard bugs in existence. The tell for #1 is “works at the default zoom, gaps at others” — always $__rate_interval. The tell for #2 is “works when I pick one, breaks when I pick two” — always =~ where a variable can be multi-valued. Make both a review checklist item; they are mechanical to catch.

4 — the id/uid distinction. Exported JSON carries a numeric "id" that is local to the source Grafana’s database. Import it elsewhere and the id either collides with an unrelated dashboard or causes a confusing replace. The "uid" is the portable key Grafana matches on. The fix is a one-liner (jq 'del(.id)') and a deterministic uid; bake it into your export pipeline so a human never forgets.

8 & 14 — the backend contract. A dashboard’s queries assume the backend can serve them. Native histograms, exemplar storage, and even specific recording rules are backend features; a terraform plan that diffs only the dashboard JSON cannot see whether the backend supports them. Across a fleet, this surfaces as “the same dashboard works here and not there.” The fix is a capability map and a CI gate that checks the backend before applying — reproducibility is not correctness.

11 — exemplars need three things. Diamonds appear only when (a) Prometheus exemplar storage is enabled, (b) the application actually emits exemplars, and © the panel’s Exemplars toggle is on and the data source has exemplarTraceIdDestinations. Miss any one and you get a silent no-op. Instrument first, then expect the link.

Best practices

• Start from a method, always. Every service gets RED; every backing resource gets USE. If a panel does not answer a RED or USE question for the thing on screen, justify it or cut it. The method is what makes a stranger able to read your board.
• Lead with a verdict, not a graph. Put a Stat panel (current error ratio, thresholded) top-left so the first glance is “green or red,” then graphs below for “why.” The eye should reach a hypothesis before it reads an axis.
• $__rate_interval in every panel rate(); literal windows only in recording rules/alerts. This single rule eliminates the most common “works on my screen” bug.
• =~ on every multi-value variable, with a .* custom-all-value. Equality matchers are a time bomb that goes off the instant someone multi-selects.
• Plot percentiles, never the average, for latency — and plot p50/p90/p99 together so the gap reveals uniform-vs-tail. Add a heatmap when distribution shape matters.
• Chain variables so dropdowns narrow as you drill (namespace → service → instance), and filter/sort them so they are scannable, not thousand-item walls.
• Repeat panels/rows for fleets, but cap the variable. One templated board beats forty hand-placed ones; an uncapped repeat melts the browser.
• Wire drilldowns: exemplars to traces, data links to logs, annotations for deploys. A symptom should be one click from its cause; build the link in the UI and copy the generated URL.
• Treat the JSON model as source code. It lives in Git, it is reviewed, and a delivery mechanism reconciles it — never “it only exists in the running Grafana.”
• Strip id, set a deterministic uid in your export pipeline, so dashboards are portable and provisioning is idempotent.
• Lock golden dashboards (editable:false, allowUiUpdates:false) and give teams their own folders for free-build — platform owns the golden boards, teams own theirs.
• Extract shared panels into library panels so a threshold change rolls out everywhere at once, and manage folder permissions as code.
• Gate fleet-wide dashboard changes on backend capability. A clean plan is not correctness when Prometheus versions/features differ across clusters.

Security notes

• Authenticate IaC with service accounts, not API keys. Grafana deprecated legacy API keys; the Terraform provider and any CI automation should use a service-account token scoped to exactly the org and role they need (often a dedicated “provisioner” account with editor rights on specific folders, not Admin).
• Folder permissions are your authorization model. Grant teams edit on their folders and viewer elsewhere, and manage those grants as code (grafana_folder_permission) so access control is reviewed like everything else. Avoid org-wide Editor; scope to folders.
• Don’t leak topology in panels or links. A panel title, a data-source name, or a data-link URL can expose internal hostnames, cluster names, or service maps. Keep public/shared dashboards free of internal addressing; a drilldown URL is still a URL someone can read.
• Lock the data sources’ query access. A data source with broad query permissions lets any dashboard editor run arbitrary PromQL/LogQL against your backend. Use Grafana’s data-source permissions and, where supported, query-time tenant scoping (e.g. Mimir/Loki multi-tenancy) so a dashboard can only see its own tenant.
• Treat the JSON repo as sensitive-adjacent. Dashboards rarely hold secrets, but they hold structure — what services exist, what they depend on, what the thresholds (and thus SLOs) are. Review who can read the dashboards-as-code repo.
• Beware embedded credentials in data-source provisioning. Provisioned data sources can carry tokens/passwords in secureJsonData; keep those out of the committed YAML (inject via env/secret store), never in plain jsonData.
• Public/snapshot dashboards leak by design. If you enable public dashboards or share snapshots, remember the data travels with them; scope to non-sensitive boards and prefer authenticated access.

The security knobs that also keep dashboards reproducible:

Cost & sizing

Grafana OSS itself is free; the cost of a dashboard practice is the load it puts on the backends and the human time it saves or wastes — and the second dwarfs the first. The drivers:

• Query load on Prometheus/Mimir is the real “cost” of a dashboard. A board with thirty heavy panels on a 5-second refresh, opened on ten screens, can generate hundreds of expensive queries per minute — histogram_quantile over high-cardinality buckets is not cheap. Raise the refresh to 30s–1m for most boards, and back heavy panels with recording rules (precompute the p99, the error ratio) so the dashboard reads a cheap pre-aggregated series instead of computing it live. See Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir.
• Cardinality is the silent bill. A variable that enumerates a thousand instances, a repeat that renders hundreds of panels, an All that expands to a giant regex — each multiplies query cost. Chaining and capping variables is a cost control, not just a UX nicety; the relabeling/limits side is in Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus.
• Grafana Cloud vs self-hosted. Self-hosting Grafana OSS is essentially free software on a small VM (a single Grafana serves many users on modest resources). Grafana Cloud prices on active users, metrics series, logs, and traces ingested — predictable but metered; the dashboard side is rarely the dominant line, the ingested series are. A small self-hosted Grafana fits comfortably in a few hundred rupees a month of compute; the backend storage is the larger figure.
• The human-time ROI is the whole point. A dashboard that gets the on-call to a hypothesis in 60 seconds instead of an hour, across dozens of incidents a year, is worth more than any infrastructure line. Conversely, a wall-of-panels board that adds query load and slows diagnosis is negative ROI twice over.

A rough monthly picture for a self-hosted setup serving one team: Grafana on a small VM (~₹500–1,500), with the dominant cost being the Prometheus/Mimir storage behind it (sized by series count, often the larger figure). The cost levers and what each buys:

Interview & exam questions

1. What is the RED method and what is each letter? RED (Tom Wilkie) is a symptom-based method for services: Rate (requests per second), Errors (failed requests, ideally as a ratio of total), and Duration (the latency distribution at percentiles). It defines exactly which three panels a service dashboard needs, so the board reflects what a user feels — too slow, too many failures, under what load.

2. What is the USE method and how does it relate to RED? USE (Brendan Gregg) is a cause-based method for resources with finite capacity: Utilization (busy fraction), Saturation (queued, un-serviced work), Errors (resource error events). RED and USE are a pair — RED (symptom) tells you the service is slow; USE (cause) tells you which resource is the bottleneck. The canonical incident board is a RED row over USE rows.

3. Why use $__rate_interval instead of rate(...[5m])? A rate() window must hold at least two scrape samples and must scale with the viewed time range — a fixed [5m] returns gaps when you zoom out (the step grows past it) and smears spikes when slightly wrong. $__rate_interval is a Grafana macro that computes a safe window from the scrape interval and current step, so the same query is correct at every zoom level. Use literal windows only in recording rules and alerts.

4. Why does instance="$instance" break when a variable is multi-value, and what’s the fix? A multi-value variable interpolates as a regex alternation (a|b|c). An equality matcher then looks for a literal label value a|b|c, which doesn’t exist, so the panel goes empty. Use a regex matcher — instance=~"$instance" — which treats the alternation correctly for both single and multiple selections. Pair it with a .* custom all value so “All” interpolates cheaply.

5. How do you read a latency panel, and why not plot the average? Plot p50/p90/p99 from histogram_quantile(q, sum by (le) (rate(..._bucket[$__rate_interval]))). The gap is the diagnosis: p50 and p99 rising together is a uniform slowdown (systemic); p50 flat with p99 exploding is a long tail (a subset of requests). The average (_sum/_count) hides this entirely — a few very slow requests barely move the mean, so the average lies about user pain.

6. Name four template-variable types and a use for each. Query (label values → $service), Custom (a hand-written list → $percentile), Datasource (flip between Prometheus instances → $ds), Interval (let the reader pick a resolution), with Constant, Textbox, and Ad-hoc filters rounding out the set. Chaining (a query variable referencing another) makes dropdowns narrow as you drill.

7. How do you cover every instance of a service with one panel? Make instance a multi-value query variable and set the panel (or its row) to Repeat by instance. Grafana renders one copy per selected value, each scoped to a single instance. Cap the variable so an uncapped All on a huge label doesn’t render hundreds of panels and freeze the browser.

8. What’s the difference between id and uid in a dashboard JSON, and why does it matter for provisioning? id is a numeric identifier local to one Grafana’s database; uid is a stable, portable string. When you provision exported JSON, a leftover id can collide on the target instance, while the uid is what Grafana matches dashboards on across instances. Strip id, set a deterministic uid, and provisioning becomes idempotent.

9. Compare file provisioning, the Terraform provider, and Grizzly for delivering dashboards. File provisioning loads JSON from a config dir at reload — one-way, ideal for locked platform-owned golden boards. The Terraform provider manages dashboards via the API as grafana_dashboard resources — best when dashboards live with the surrounding IaC, and plan surfaces UI drift. Grizzly is a kubectl-style diff/apply CLI — best in GitOps/CI pipelines, often paired with grafonnet to generate many boards from one template.

10. How do exemplars work and what do you need to enable them? Exemplars are sampled trace IDs Prometheus attaches to histogram observations; Grafana renders them as clickable diamonds on a latency panel that jump to the trace. You need three things: exemplar storage enabled in Prometheus, the app emitting exemplars (OpenTelemetry/Prometheus client), and the data source configured with exemplarTraceIdDestinations (label → tracing data-source uid) plus the panel’s Exemplars toggle on.

11. A fleet-wide dashboard change passed terraform apply everywhere but a panel is empty on half the clusters. What happened? The dashboard JSON is identical (so plan was clean), but the backend differs — e.g. a native-histogram query needs --enable-feature=native-histograms, present only on newer Prometheus. Dashboards-as-code gives reproducibility, not correctness, when backends are skewed. Fix with a per-cluster capability map and a CI gate that checks backend features before applying.

12. When do you reshape in PromQL versus a transformation? Reshape in PromQL when it changes what Prometheus computes (an error ratio is genuinely a division of rates you want the backend to do). Use a transformation when it only changes presentation (rename a column, drop a series from the legend, sort a table, turn labels into columns). Over-using transformations for real math makes panels slow and opaque; under-using them yields ugly tables and duplicate queries.

These map to the Grafana Certified Associate (dashboards, variables, panels, provisioning), to PCA — Prometheus Certified Associate (PromQL, histograms, rate), and to general SRE interview ground (RED/USE, SLO-driven thresholds, drilldowns). A compact mapping:

Quick check

1. A panel shows clean data at the default zoom but gaps when you select “Last 7 days.” What is almost certainly wrong, and the fix?
2. You enable multi-value on the instance variable; now picking two instances makes the panel go empty. Why, and what one change fixes every affected query?
3. Your latency panel is a single smooth line and an incident’s tail-latency spike doesn’t show. What should you be plotting instead?
4. You exported a dashboard’s JSON and provisioning it elsewhere either fails or replaces an unrelated board. Which field is the culprit and what do you do?
5. terraform apply of a dashboard change succeeded across all clusters, but one cluster’s Duration panel is empty. Name the most likely cause and the durable fix.

Answers

1. A hardcoded rate() window (e.g. [5m]): at a wide range Grafana’s step grows past the window so rate() returns nothing per step → gaps. Fix: use [$__rate_interval] in the panel, which scales the window to the scrape interval and current step.
2. A multi-value variable interpolates as a regex alternation a|b|c; an equality matcher instance="$instance" then matches a literal a|b|c that doesn’t exist → empty. Fix: change the matcher to instance=~"$instance" (regex) — and set a .* custom all value for “All.”
3. Percentiles — p50/p90/p99 via histogram_quantile(q, sum by (le) (rate(..._bucket[$__rate_interval]))), plotted together so the p50-to-p99 gap distinguishes a uniform slowdown from a long tail. The average (_sum/_count) hides tail latency because a few slow requests barely move the mean.
4. The top-level "id" — a numeric id local to the source Grafana’s database that collides on the target. Strip it (jq 'del(.id)') and keep/set a deterministic uid, which is the portable key Grafana matches on; this also makes provisioning idempotent.
5. Backend feature skew — the new query (e.g. native-histogram) needs a Prometheus feature flag present only on some clusters; the JSON is identical so plan is clean, but the backend can’t serve it. Durable fix: a per-cluster capability map with for_each to pin the right dashboard variant, and a CI gate that checks backend features before applying.

Glossary

• RED method — Rate, Errors, Duration: the three symptom-based panels for any request-serving service.
• USE method — Utilization, Saturation, Errors: the three cause-based panels for any finite-capacity resource.
• Panel — a single visualization answering one question; the unit of dashboard design.
• Dashboard — an ordered set of panels that walks the reader from “is something wrong” to “where” to “what changed.”
• Template variable — a $name placeholder panels interpolate at render time; turns one dashboard into many.
• Query variable — a template variable whose dropdown is populated from a data-source query (commonly label_values(...)).
• Chained variable — a variable whose query references another variable, so its options narrow when the parent changes.
• Multi-value / All — variable options to select several values, or every value; require a regex (=~) matcher and a .* custom all value.
• $__rate_interval — a Grafana macro computing a safe rate() window from the scrape interval and current step; correct at every zoom.
• histogram_quantile — a PromQL function computing a percentile from histogram buckets; needs sum by (le) over the bucket rate.
• Native histogram — Prometheus’s newer high-resolution histogram format; requires the native-histograms feature flag and changes the quantile query shape.
• Threshold — a value boundary that colors a panel, turning a number into a green/amber/red verdict.
• Value mapping — a rule mapping numbers/ranges/special values to text and color (e.g. 1 → UP).
• Field override — config applied to specific series matched by name/regex (e.g. make p99 red and thick).
• Transformation — a post-query reshape of the data in the panel (rename, filter, group, join, reduce, sort).
• Repeat — rendering one panel or row per value of a multi-value variable, so one board covers a fleet.
• Data link — a templated URL from a panel/series interpolating labels and the time range, for one-click drilldown.
• Exemplar — a sampled trace ID attached to a histogram observation, rendered as a clickable diamond linking to a trace.
• Annotation — an event overlaid on every panel’s time axis (deploy markers, alert-firing windows).
• JSON model — the serialized definition of a dashboard; the source of truth for dashboards-as-code.
• uid — a stable, portable dashboard identifier; the key Grafana matches on across instances (unlike the local id).
• Provisioning — loading dashboards/data sources from files in Grafana’s config directory at startup/reload.
• Library panel — a panel defined once and reused across dashboards; editing it updates every embedder.
• Grizzly (grr) — a kubectl-style CLI (diff/apply/watch) managing Grafana resources from files.

Next steps

You can now build dashboards that get used and survive the person who built them. Build outward:

• Foundation: PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work — every panel here is a PromQL query; get rate() and histogram_quantile() deeply right.
• Next: Grafana as Code: Provisioning Dashboards, Folders, and Unified Alerting with Terraform — the full dashboards-and-alerting-as-code workflow, end to end.
• Related: Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation — make the latency-spike-to-trace drilldown actually work.
• Related: Distributed Tracing End-to-End: Context Propagation, Tempo, and Correlating Traces with Metrics and Logs — the tracing backend your exemplars jump into.
• Related: Grafana Loki Deep Dive: LogQL, Label Cardinality, and Chunk Storage Tuning — the logs your data links open to.
• Related: Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir — precompute heavy panels so dashboards stay fast and cheap.
• Related: SLOs and Error Budgets in Practice: Defining SLIs and Building Multi-Window Burn-Rate Alerts — turn the RED error-ratio thresholds on your board into real SLOs and burn-rate alerts.