PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work

Most PromQL bugs are not syntax errors. A syntax error fails loudly — the expression browser turns red, you fix it in thirty seconds. The dangerous bug is the one that parses cleanly, returns a plausible number, ships to a dashboard, and quietly lies for six months. avg(histogram_quantile(0.99, ...)) across forty pods is valid PromQL — and wrong in a way that consistently understates your tail latency, with nothing in the engine to tell you. rate(my_gauge[5m]) is valid PromQL — and nonsense, because rate assumes monotonicity and a gauge has none. The engine hands you a float, you graph it, and the graph is fiction.

This is a working engineer’s tour of the parts of PromQL — Prometheus’s query language for time-series data — that bite people who already know the basics: the data model (what a sample is, why metric type is metadata, not enforcement), the instant-vs-range-vector distinction that underpins everything, the rate/irate/increase family and the counter-reset and extrapolation machinery inside it, aggregation with by/without, classic and native histograms (the le buckets, histogram_quantile, the bucket-boundary interpolation error, why you can never average a percentile), recording rules, subqueries, vector matching and label joins (on, ignoring, group_left, label_replace), the time-shifting modifiers offset and @, and a catalogue of the specific mistakes that produce confident, wrong numbers. Every concept comes with a real query; the high-stakes ones come with a promtool test rules snippet that pins the expected value so a future “cleanup” cannot silently break it.

By the end you will stop guessing. You will know, before a query reaches a dashboard or an alert, whether it computes what you think — which metric type each function assumes, which way a type-mismatch error points, why an increase over an hour returns a non-integer, why a panel is blank, why your p99 is pinned to exactly 10 for a week, and why summing counters before rating them turns one pod’s restart into a fleet-wide negative spike. Knowing which within ninety seconds is what separates a five-minute fix from a half-day at the wrong graph.

What problem this solves

PromQL is deceptively forgiving. The engine evaluates almost any well-typed expression and returns a vector, a vector renders on a graph, and a graph looks authoritative. That forgiveness is the entire problem: the language does not enforce the semantic contracts that make a number true. It does not stop you calling rate() on a gauge, averaging quantiles, or summing counters across instances before rating them, and it does not warn you that your range window is narrower than your scrape interval so the function silently returns nothing. Each produces output — just not output that means what you assumed.

What breaks without this knowledge is subtle and expensive. An on-call engineer builds a latency panel that averages per-pod p99s; during a failover two pods get cold caches and their real p99 jumps to 8 seconds, but the averaged panel barely moves past 1.2 seconds and no alert fires while customers time out. A capacity dashboard charts a rate with a 1-minute window against a 60-second scrape, flickers between a value and a gap, and gets deleted because nobody trusts it. An SRE writes a burn-rate alert on a counter, gets the window-to-scrape ratio wrong, and it flaps on every missed scrape. None are syntax errors; all passed review because the query looked right and returned a number.

Who hits this: everyone who graduates from copy-pasting example queries to writing their own — especially anyone building alerting rules (where a wrong number wakes someone or fails to), SLO dashboards (where averaged percentiles are endemic), and multi-instance aggregations (where rate-then-sum order and by/without decide correctness). The fix is internalising a handful of semantic rules, knowing the metric type before you choose a function, and validating the dangerous queries against synthetic data with promtool before they reach production.

To frame the field, here is every class of PromQL mistake this article disarms, its symptom, and the section that fixes it:

Learning objectives

By the end of this article you can:

• Read the data model precisely — what a sample is, why __name__ makes the metric name just another label, and why metric type is metadata, not enforcement — and predict which function each of the four types (counter, gauge, histogram, summary) supports.
• Distinguish an instant vector from a range vector at a glance, know which functions consume which, decode both directions of the type-mismatch error, and reason about how the window interacts with the graph step and scrape interval.
• Apply the rate / irate / increase family correctly — choose by use case, explain counter-reset detection and the extrapolation that makes increase non-integer, and size the window to survive a missed scrape.
• Write aggregation that is correct and intentional — by vs without deliberately, rate() before sum, and topk/count_values/quantile knowing their per-timestamp semantics.
• Build histogram queries that tell the truth — rate the buckets, preserve le, call histogram_quantile once, lay out buckets around SLO thresholds, ratio buckets for “fraction under N”, use native histograms where available — and articulate the bucket-boundary error and the averaged-percentile trap.
• Compose cross-metric math with vector matching — on/ignoring, group_left/group_right for many-to-one and enrichment joins, and label_replace/label_join to synthesise and rewrite labels.
• Use recording rules, subqueries, and the offset/@ modifiers correctly, and decide when a hot subquery must become a recording rule.
• Validate any non-trivial query before it lands with promtool check and promtool test rules, pinning expected values so a refactor cannot reintroduce a lie.

Prerequisites & where this fits

You should have Prometheus scraping at least one target, be able to open the expression browser at /graph, recognise the exposition format (http_requests_total{job="api",code="200"} 4823), know that Prometheus pulls on a scrape interval (commonly 15s or 30s), and have written a trivial query like up. Familiarity with HTTP status codes, what a percentile is, and labels as key-value dimensions helps. No prior promtool experience is required — the lab introduces it from zero.

This sits in the Observability — Metrics track and is the language layer everything else consumes: upstream, instrumentation produces the series you query; downstream, queries power dashboards and alerts. It pairs most tightly with Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir (recording rules, operationalised) and Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus (the label sets you query). Your queries become panels in Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code and alerts routed by Designing Alertmanager Routing Trees: Grouping, Inhibition, Silences, and Dedup, and the SLO math here underlies SLOs and Error Budgets in Practice: Defining SLIs and Building Multi-Window Burn-Rate Alerts.

A quick map of where each PromQL concept lives, so you know what owns what when a number looks wrong:

Core concepts

Five mental models make every later diagnosis obvious. The rest of the article is consequences.

A series is a named, labelled stream of (timestamp, float64) samples — and the name is just a label. A series is uniquely identified by a metric name plus labels, and the name is itself sugar for the reserved __name__ label, so these two are exactly the same selector:

http_requests_total{job="api", code="200"}
{__name__="http_requests_total", job="api", code="200"}

Every series is an append-only stream of (timestamp, float64) samples at roughly the scrape interval. Everything PromQL does is select some of those streams and transform them. There is no other primitive.

The metric type is metadata you must honour in your head — the engine does not enforce it. There are four types — counter, gauge, histogram, summary — and the type lives only in the metric’s # TYPE line and in your understanding. PromQL will happily call rate() on a gauge or avg on a counter and return a number; get the type wrong and you get a value, just not a true one. This single fact is the root of more wrong dashboards than any other.

Counter metrics conventionally end in _total, but that suffix is a hint to humans and changes nothing. The behaviour you rely on — monotonicity plus reset detection — is asserted by the exporter, not the name.

Instant vector vs range vector is the type system you must feel in your fingers. An instant vector is one sample per series at the evaluation timestamp (http_requests_total); a range vector is a set of samples per series over a lookback window written with a duration in brackets (http_requests_total[5m]). Functions like rate, increase, and the _over_time family consume a range vector and return an instant vector. You cannot graph a range vector or pass an instant vector to rate() — the single most common error after the basics, and the error message tells you which way you got it backwards.

Counters are read through rate, never raw. A raw counter value (4823) is meaningless — it only grows and resets. Critically, rate and increase detect counter resets: if the value drops between two samples, the engine assumes a restart and treats the drop as a reset (counting the post-reset rise as continued increase) rather than computing a negative rate. That is why you must never compute rates by hand with subtraction, and why the order of rate and sum matters so much later.

Histograms are a family of series, and percentiles are not averageable. A classic histogram is cumulative _bucket counters (each labelled with an upper bound le) plus _sum and _count; for a percentile you rate() the buckets, aggregate while preserving le, then call histogram_quantile once. Two hard truths the field gets wrong constantly: accuracy is bounded by your bucket layout (interpolation is within the matched bucket), and you cannot average percentiles — it’s meaningless and understates the tail.

The vocabulary in one table

Before the deep sections, pin down every moving part — the mental model side by side (the glossary repeats these for lookup):

The selector and matcher reference

Before any function, you select series. A selector is an optional metric name plus zero or more label matchers in braces. The four matcher operators are the foundation of every query — get a =~ anchor wrong and you silently match nothing or everything.

Three rules that cause real bugs: regexes are fully anchored (=~"api" will not match api-prod — use .*api.*); =~"" matches the empty value and series missing the label; and at least one matcher must restrict, so bare {} is a parse error while {__name__=~"node_cpu_.*"} regexes over metric names.

A few selectors that show the range of what is possible:

http_requests_total{job="api", code=~"5.."}      # all 5xx for the api job
{__name__=~"node_.+", instance="10.0.1.4:9100"}  # every node_* metric (name regex)
http_requests_total{canary=""}                   # series with NO 'canary' label
http_requests_total{method=~"GET|POST"}          # anchored: exactly GET or POST, not GETX

The classic empty-result mystery is the anchoring rule: someone writes {handler=~"/api/users"} expecting /api/users/42, gets nothing, and concludes “the metric isn’t there.” It is — the regex is anchored. Always reach for .* when you mean “contains.”

Instant vectors versus range vectors

This is the distinction that unlocks PromQL, and the one that produces the most beginner-to-intermediate errors. Restating it precisely:

http_requests_total              # instant vector: one value per series, at eval time
http_requests_total[5m]          # range vector: many values per series, last 5 minutes

You can graph an instant vector. You cannot graph a range vector — it has no single value to plot per step. The only thing you can do with a range vector is feed it to a function that reduces it back to an instant vector. Those functions are a closed, learnable set:

The type-mismatch error is your friend because it names the direction of your mistake. Memorise both:

How the window interacts with the graph step

When you render a range query over time, Prometheus evaluates your instant-vector expression at each step across the dashboard’s time range; the [5m] is the lookback applied at each of those steps independently. Two rules keep this honest:

With a 30-second scrape, [2m] gives roughly four samples per window — enough to survive one missed scrape and still compute a rate. A window exactly equal to the scrape interval frequently catches only one sample (depending on alignment), so rate() flickers between a value and nothing — always give the window slack. The most common “my panel is blank” cause is a window narrower than the scrape interval, or a series too new to have two samples in the window.

The rate family in depth

Counters only go up until a restart sets them back to 0. Three functions turn that sawtooth into something meaningful, and choosing the wrong one — or the wrong window — is a daily source of misleading graphs.

# Per-second average request rate over 5 minutes (the default for graphs and alerts)
rate(http_requests_total{job="api"}[5m])

# Instantaneous rate from the last two samples (live, twitchy)
irate(http_requests_total{job="api"}[1m])

# Absolute count of requests in the last hour (NOT per-second)
increase(http_requests_total{job="api"}[1h])

The three functions compared, with the trade-off that decides which you pick:

Counter-reset detection — what actually happens

When rate/increase find a sample lower than its predecessor, they treat it as a reset: the drop is read as the counter resetting to 0 and the post-reset rise counts as continued increase. So a pod restarting mid-window produces a sensible rate — as long as you rate the per-instance series before aggregating. sum first, and the per-series reset information is gone: a restart looks like the aggregate going backwards.

Extrapolation — why increase returns non-integers

Both rate and increase extrapolate to the exact edges of the window. Window boundaries rarely align with scrape timestamps, so the engine extends the observed slope to the start and end — which is why increase(http_requests_total[1h]) can return 1342.8 even though you counted whole requests. That is expected, not a bug; treat it as an estimate. Two more behaviours follow from the same machinery: if extrapolation near the start of a series would project below zero, Prometheus clamps it to the series start so it never invents negative history; and stale markers end a series cleanly, so gaps don’t produce phantom rates.

Real queries you’ll write with this family:

# 5xx error rate per second, smoothed — the canonical alerting input
sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))

# Requests per second by route, dropping the noisy instance label
sum without (instance) (rate(http_requests_total[5m]))

Aggregation operators and by / without

Aggregation operators collapse many series into fewer. The grouping clause decides which labels survive — and that decision is your dashboard’s grouping and your alert’s identity.

# Total request rate per service — keep only `job`, collapse everything else
sum by (job) (rate(http_requests_total[5m]))

# Same intent, but keep every label except the noisy per-pod `instance`
sum without (instance) (rate(http_requests_total[5m]))

The full set of aggregation operators, what they do, and the parameter (if any) they take:

by versus without — the choice has a real robustness consequence as labels evolve:

without is often the more robust default: add a region label upstream and by (job) silently drops it (panels unchanged) while without (instance) carries it through (panels gain detail). Prefer without for exploratory dashboards; by for a single authoritative number where extra labels would split the series unexpectedly.

The order rule: rate before sum, always

The rule the counter-reset section set up: always rate() the per-instance counters first, then sum:

# CORRECT — reset detection happens per instance, then we add the rates
sum(rate(http_requests_total[5m]))

# WRONG — summing counters across instances destroys per-instance reset
# detection; one pod restarting looks like the aggregate going backwards
rate(sum(http_requests_total)[5m])     # also a type error without a subquery

topk, bottomk, and per-timestamp evaluation

topk(k, ...) and bottomk(k, ...) are evaluated independently at every timestamp, so the membership of the top 5 can change across a graph — a pod in the top 5 at 10:00 may drop out at 10:05, producing lines that start and stop. Correct, but surprising.

# Top 5 CPU-hungry pods, recomputed every step
topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))

# Distribution of HTTP status codes by count of series (creates a new label — watch cardinality)
count_values("code", http_requests_total)

A frequent confusion: quantile(0.99, http_request_duration_seconds) is an aggregation operator computing the 99th percentile of the per-series values at this instant — not your request latency p99, which comes from histogram_quantile over buckets (next section). Mixing them up produces a number that looks like a latency percentile and is not.

Histograms done right

A classic Prometheus histogram is not one series but a family: cumulative _bucket counters (each labelled with an upper bound le), plus _sum and _count. “Cumulative” is the key word: the le="0.5" bucket counts every observation ≤ 0.5, le="1" counts everything ≤ 1 (including the ≤ 0.5 ones), and so on up to the mandatory le="+Inf" bucket that counts everything.

Computing a percentile correctly

To get a percentile you rate() the buckets first, then apply histogram_quantile, preserving le through the aggregation:

histogram_quantile(
  0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

The order is fixed and falls out of the types: (1) rate(..._bucket[5m]) — buckets are counters, so you need per-second rates; (2) sum by (le) — combine instances while preserving le (drop it and the function has nothing to interpolate over); (3) histogram_quantile(0.95, …) — compute the percentile once, on the whole fleet.

The bucket-boundary interpolation error

histogram_quantile does linear interpolation within the matched bucket. If your p99 lands in a bucket spanning [1s, 10s], the answer is a guess in that range — and if the true p99 is 9.9s, the interpolation is wildly off. Two failure modes follow:

The design rule: lay out buckets around your SLO thresholds, not on a generic exponential scale. For “p99 under 300ms,” use buckets like 0.1, 0.2, 0.25, 0.3, 0.35, 0.5, 1 — dense around 0.3 — not the default 0.005, … , 10. Quantisation error is bounded by the width of the bucket your percentile lands in, so make it narrow where it matters.

Why you can never average a percentile

Averaging per-instance p95s across pods is meaningless and consistently understates tail latency: a percentile is a property of a distribution, and the mean of several percentiles is not the percentile of the combined distribution. Always aggregate the buckets and compute the quantile once.

Fraction-under-threshold without quantiles

For an SLO-style “what fraction of requests were under 300ms” question, skip quantiles and ratio the buckets — exact, no interpolation, as long as a bucket boundary sits at your threshold:

# Fraction of requests faster than 300ms (requires an le="0.3" bucket)
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

This is the basis of SLO compliance math: numerator is “good” requests (under the objective), denominator is all requests. It is exact, cheap, and sidesteps the bucket-boundary error — which is why latency SLOs should be defined on a bucket boundary you control.

Native (exponential) histograms

Recent Prometheus supports native histograms (also called exponential or sparse histograms), which store buckets far more efficiently with automatic exponential boundaries, sidestepping manual le layout. The mental model is the same — histogram_quantile works on them too — but the wire format, storage, and a few functions differ.

# Native histogram: no le, no sum by (le) — rate the single series and ask for the quantile
histogram_quantile(0.95, rate(http_request_duration_seconds[5m]))

# Fraction of requests under 300ms on a native histogram
histogram_fraction(0, 0.3, rate(http_request_duration_seconds[5m]))

If you control instrumentation on a recent version, evaluate native histograms — the le-bucket pain and the cardinality cost of many _bucket series largely go away. Until then, classic histograms with carefully placed buckets are the workhorse, and everything here about preserving le and never averaging percentiles applies.

Vector matching and label joins

Binary operators between two instant vectors match series by their full label set by default. When the label sets differ, tell PromQL how to match with on (only these labels) or ignoring (everything except these), and for unequal cardinality add group_left/group_right.

A canonical ratio — error rate as a fraction of total — where both sides already share labels:

# Both sides carry identical `job` labels, so default matching works
sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))

The matching keywords and what each one does:

Many-to-one matching

The interesting case is many-to-one, where one side has extra labels — say per-code error fractions against a per-job total (left has job and code, right only job):

sum by (job, code) (rate(http_requests_total[5m]))
/ on (job) group_left
sum by (job) (rate(http_requests_total[5m]))

on (job) matches by job; group_left declares the left side as the “many” (it may have extra labels — here code), and the result keeps the left side’s full labelling. Use group_right for the mirror image. The decision table:

Enrichment joins with an info metric

group_left also does enrichment — pulling labels from a metadata “info” metric (like kube_pod_info, which is always 1) onto your real metric. The arithmetic is just a vehicle for the label copy:

# Attach the `node` label from kube_pod_info onto each pod's CPU rate
sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))
* on (pod) group_left(node)
kube_pod_info

Here group_left(node) copies the node label off kube_pod_info onto each CPU series; multiplying by kube_pod_info (value 1) leaves the numbers unchanged — the point is the label join. This pattern — multiply by an info metric to attach its labels — is how you slice app metrics by infrastructure dimensions you did not export directly.

Set operators

Beyond arithmetic, three logical operators combine vectors by label-set membership rather than by value: and keeps left series that have a match on the right (“only where both exist”), or is a union (left series plus right series with no left match), and unless keeps left series with no match on the right (exclusion).

# Request rate only for pods that are currently Running
sum by (pod) (rate(http_requests_total[5m]))
and on (pod)
kube_pod_status_phase{phase="Running"} == 1

If a binary operation returns “many-to-many matching not allowed,” you have duplicate series on the side you think is unique. Either the join key isn’t unique (add the missing label to on/ignoring) or aggregate that side to one series per match group first. This is almost always a label-cardinality surprise, not a bug in PromQL.

Synthesising and rewriting labels: label_replace and label_join

label_replace adds or rewrites a label using a regex capture; label_join concatenates several labels into one. Both are essential for normalising labels before a join or for shaping output for a dashboard.

# Create a `device_simple` label by stripping the partition number from `device`
label_replace(
  node_filesystem_avail_bytes,
  "device_simple", "$1", "device", "(/dev/[a-z]+)[0-9]*"
)

# Build a `target` label by joining job and instance with a colon
label_join(up, "target", ":", "job", "instance")

A common real use: the two sides of a join carry the same value under different label names (pod vs pod_name). label_replace renames one so the on clause lines up — without it, the join silently matches nothing.

Subqueries, offset, and the @ modifier

offset — shifting the lookback in time

offset shifts a vector’s evaluation back by a fixed duration (or forward, with a negative offset) — perfect for week-over-week comparisons:

# This week's rate versus the same 5-minute window 7 days ago
sum(rate(http_requests_total[5m]))
/
sum(rate(http_requests_total[5m] offset 7d))

@ — pinning to an absolute timestamp

The @ modifier pins a vector’s evaluation to a fixed Unix timestamp (or start()/end() of the query range), regardless of the step rendered. This is how you draw a flat reference line — “the value at deploy time” — across a whole graph:

# Ratio of current rate to the rate at a fixed instant (a deploy timestamp)
sum(rate(http_requests_total[5m]))
/
sum(rate(http_requests_total[5m] @ 1700000000))

Subqueries — a range query over an instant expression

A subquery lets you run a range function over the result of an instant expression, with the syntax [outer_range:resolution]. This is how you ask “what was the max 5-minute rate over the last hour”:

# Highest 5-minute request rate seen in the last hour, sampled every minute
max_over_time(
  sum(rate(http_requests_total[5m]))[1h:1m]
)

The [1h:1m] evaluates the inner expression every minute across a 1-hour window, then max_over_time reduces those 60 points. Subqueries are powerful but expensive — they materialise many intermediate points at query time on every dashboard refresh. The cost characteristics:

The rule of thumb: a subquery is fine for ad-hoc investigation, but anything you run repeatedly (a panel, an alert) should have its inner expression promoted to a recording rule so the engine isn’t recomputing it on every evaluation.

Recording rules

A recording rule pre-computes an expression on the server at a fixed interval and stores the result as a new series — turning an expensive, repeated query into a cheap read, and giving dashboards and alerts a single materialised series to share so they can never drift apart.

# rules/recording.yml
groups:
  - name: http_rates
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

Follow the level:metric:operations naming convention to encode meaning and prevent collisions: the prefix is the aggregation level / grouping labels (job:, instance:, cluster:), the middle is the underlying metric (http_requests), and the suffix is what was done (:rate5m, :p99_5m, :sum) — as in the job:http_requests:rate5m rule above.

Why and when to record, versus leaving a query inline:

Three operational rules that bite if ignored:

The single biggest correctness win: an alert and its panel both read job:http_request_duration_seconds:p99_5m instead of each re-deriving the quantile, so a “cleanup” that reintroduces an averaged percentile on the dashboard can’t make the two disagree — they’re the same series.

Architecture at a glance

PromQL has no diagram here — its “architecture” is a small, strict evaluation pipeline you can hold entirely in your head, and seeing it as a pipeline is what makes wrong queries obvious. Trace a single query, histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))), from the inside out, and you have the whole model.

At the base sits the TSDB: append-only series, each a name plus labels, each holding (timestamp, float64) samples. A selector picks series by name and matchers — here, every http_request_duration_seconds_bucket across all instances and le boundaries. Attaching [5m] promotes that instant vector (one sample per series) to a range vector (five minutes of samples per series), the only thing rate consumes: rate(..._bucket[5m]) applies reset detection and edge extrapolation and emits a per-second rate, collapsing back to an instant vector per (instance, le). Then sum by (le) is an aggregation step — collapsing every label except le into one fleet-wide rate per boundary, with le preserved so the bucket structure survives. Finally, histogram_quantile(0.99, ...) reads those cumulative bucket rates and linearly interpolates within the matched bucket, emitting an instant vector with no le — a percentile, ready to graph or threshold.

Read it as a strict layering — TSDB → selector → range vector → rate → aggregation (sum by le) → histogram_quantile → instant-vector result — and every error in this article maps to a layer: a type mismatch crosses the range/instant boundary wrong; a dropped le is an aggregation that destroyed structure the next layer needed; a rate-after-sum is two layers in the wrong order. When a number looks wrong, walk the pipeline from the TSDB outward and ask at each layer, “what type do I have, and is it what the next layer needs?” That walk is the debugging method.

Real-world scenario

Meridian Pay runs a checkout API on Kubernetes — roughly 40 pods behind a service, scraped every 30 seconds by a single Prometheus, with a hard SLO of p99 checkout latency under 1 second. The latency panel and the paging alert were both inherited from a previous team and had “worked” for a year.

The panel used avg(histogram_quantile(0.99, rate(checkout_duration_seconds_bucket[5m]))) — a quantile computed per pod, then averaged across the forty pods — and the alert read the same expression. On an ordinary Tuesday a cache tier degraded and two pods that lost their warm cache saw real p99 climb to 8 seconds while the other thirty-eight stayed near 200ms. Customers on those two pods timed out at the payment step. The dashboard crept from 210ms to about 540ms; the alert, thresholded at 1s on that same averaged number, never fired. The SLO board stayed green through a customer-facing incident, and the on-call engineer only found out from a support escalation forty minutes in.

The breakthrough was walking the query pipeline. The engineer pulled the per-pod p99 with histogram_quantile(0.99, sum by (pod, le) (rate(checkout_duration_seconds_bucket[5m]))) and immediately saw two pods pinned at 8s while the rest sat at 200ms. The averaged panel had diluted two catastrophic tails across thirty-eight healthy ones — the lie of averaged percentiles. Worse, the buckets were a generic exponential layout (…, 1, 2.5, 5, 10) with nothing between 0.5 and 1, so even the correct fleet p99 had a coarse interpolation step right at the SLO boundary — it could swing 200ms from interpolation noise alone.

The fix landed in two parts. First, the query: replace the averaged-quantile panel and alert with the correct aggregate, computed once over the fleet’s buckets:

# Correct fleet-wide p99 — aggregate the buckets (keep le), quantile once
histogram_quantile(
  0.99,
  sum by (le) (rate(checkout_duration_seconds_bucket[5m]))
)

That was promoted to a recording rule, job:checkout_duration_seconds:p99_5m, so the alert and dashboard read the same materialised series and could never drift apart again. Second, the buckets: instrumentation placed explicit boundaries around the 1s SLO — …, 0.5, 0.75, 0.9, 1, 1.5, 2, … — so interpolation near the threshold became tight. They also added a per-pod outlier panel (max by (pod) (histogram_quantile(0.99, sum by (pod, le) (rate(checkout_duration_seconds_bucket[5m]))))) to catch the “two bad pods” case a fleet aggregate can still mask if the fleet is large enough.

Finally, they locked it with a promtool test: a synthetic two-pod skew with an expected fleet p99 that the averaged form gets wrong, so a future “simplification” back to avg(...) fails CI. The next degradation — three weeks later, same cache tier — paged within the for: 5m window with p99 jumping cleanly to 8s on both alert and panel. Same data, same Prometheus, correct math. The lesson on the wall: “You cannot average a percentile, and a generic bucket layout hides the truth at exactly the threshold you care about.”

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

PromQL’s design — a forgiving, type-light, vector-based language over a labelled TSDB — both makes it extraordinarily expressive and makes the silent-wrong-number class of bug possible. Weigh it honestly:

The model is right for metrics-driven observability where you want flexible, label-based querying without predefining a schema and are willing to internalise a handful of semantic rules. It bites hardest on teams that treat PromQL as “SQL-ish” and assume the engine catches their mistakes (it won’t), on latency SLOs built by averaging percentiles, and on high-cardinality joins written without checking uniqueness. Every disadvantage here is manageable — by knowing the metric type, respecting the range-vector boundary, preserving le, recording hot queries, and testing with promtool — which is the entire point of this article.

Hands-on lab

This lab runs entirely locally with a single Prometheus binary and promtool — no cloud account, no cost. You will run the core query families, reproduce the averaged-percentile bug against synthetic series, and pin the correct answer with a rule unit test. It is the fastest way to feel the semantics rather than read them.

Step 1 — Get Prometheus and promtool. Use local binaries (both ship in the same tarball) or a container that mounts the working dir:

prometheus --version && promtool --version          # local binaries
# or: docker run --rm -it -v "$PWD":/work -w /work -p 9090:9090 prom/prometheus:latest --version

Expected: a version banner for both.

Step 2 — Write a minimal config and a rules file. Create prometheus.yml and rules.yml in the working directory:

# prometheus.yml — scrape ourselves so 'up' exists; load the rules
global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
  - rules.yml
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

# rules.yml — one recording rule and one alert to test later
groups:
  - name: lab
    interval: 15s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - alert: HighErrorRate
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
            /
          sum by (job) (rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "5xx ratio above 5% for {{ $labels.job }}"

Step 3 — Validate config and rules before running anything. This is the habit the whole article argues for:

promtool check config prometheus.yml
promtool check rules rules.yml

Expected: SUCCESS per rule. If you mistyped the alert expr, promtool names exactly which rule and why — fix it here, not in production.

Step 4 — Run Prometheus and open the expression browser at http://localhost:9090/graph:

prometheus --config.file=prometheus.yml

Run a few queries to feel the instant-vs-range distinction:

up                          # instant vector — 1 per target, value 1
prometheus_http_requests_total              # raw counter (don't graph this)
rate(prometheus_http_requests_total[5m])    # per-second rate — graph this
prometheus_http_requests_total[2m]          # range vector — Table tab only, can't graph

Expected: up is 1; the raw counter only grows; rate(...) is graphable; graphing the [2m] range vector errors with “invalid expression type range vector for range query” — the type-mismatch lesson, live.

Step 5 — Reproduce the averaged-percentile bug with a unit test. promtool test rules evaluates against synthetic series you declare, so you can construct the two-pod skew deterministically — no live data needed. Create histogram_test.yml:

# histogram_test.yml — prove avg-of-quantiles understates the tail
rule_files: []          # we test ad-hoc expressions, not a rules file here
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # Pod A: everything fast — all observations land in the le="0.5" bucket
      - series: 'lat_bucket{pod="a", le="0.5"}'
        values: '0+60x10'
      - series: 'lat_bucket{pod="a", le="1"}'
        values: '0+60x10'
      - series: 'lat_bucket{pod="a", le="+Inf"}'
        values: '0+60x10'
      # Pod B: everything slow — observations only land in le="+Inf"
      - series: 'lat_bucket{pod="b", le="0.5"}'
        values: '0+0x10'
      - series: 'lat_bucket{pod="b", le="1"}'
        values: '0+0x10'
      - series: 'lat_bucket{pod="b", le="+Inf"}'
        values: '0+60x10'
    promql_expr_test:
      # CORRECT: aggregate buckets (keep le), quantile once → reflects pod B's slow tail
      - expr: histogram_quantile(0.99, sum by (le) (rate(lat_bucket[5m])))
        eval_time: 10m
        exp_samples:
          - labels: '{}'
            value: 1            # p99 sits in the +Inf-bounded region: returns last finite le boundary (1)

promtool test rules histogram_test.yml

Expected: SUCCESS. The point is constructive: with half the traffic slow (pod B only ever increments +Inf), the correct fleet quantile reflects the slow tail. Average per-pod quantiles instead and pod A’s fast p99 drags the number down — the bug from the scenario. (Prove it by adding a second promql_expr_test with the averaged expression and watching the value come out lower.)

Step 6 — Pin the rate math so the number can never silently change. Add a second test that fixes the extrapolation result — feed a counter that increments by 10 every minute and assert the 5-minute rate:

# rate_test.yml — feed 0,10,20,... and pin the per-second rate
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", code="500"}'
        values: '0+10x10'
    promql_expr_test:
      - expr: rate(http_requests_total{code="500"}[5m])
        eval_time: 10m
        exp_samples:
          - labels: '{job="api", code="500"}'
            value: 0.16666666666666666     # ~10 increments per 60s

Run with promtool test rules rate_test.yml. If 0.1666… surprises you, good — that’s the per-second math (10 per 60s), now locked against regression.

Validation checklist. You ran promtool check before a server existed, met the instant-vs-range boundary error, built a deterministic histogram skew, and pinned the rate math. Each step maps to a core lesson:

Teardown. Stop Prometheus with Ctrl-C (or stop the container) and remove the small YAML files (and data/ if a local binary created one):

rm -f prometheus.yml rules.yml histogram_test.yml rate_test.yml; rm -rf data/

Common mistakes & troubleshooting

This is the part you bookmark. First as a scannable table you can read while a panel is lying to you, then the worst offenders expanded with the exact confirming query.

The three worst offenders, expanded, because they cause the most damage and the table can’t carry the full reasoning:

Averaged percentiles (#1) — the tail that never fires. The panel and alert compute histogram_quantile per instance and then avg across instances. A percentile is a property of a distribution; averaging several is meaningless and consistently understates the tail — two bad pods in forty dilute to nothing. Confirm by pulling the per-instance quantile and looking for buried outliers: histogram_quantile(0.99, sum by (instance, le) (rate(http_request_duration_seconds_bucket[5m]))). Fix by aggregating the buckets and computing the quantile once — histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) — promoted to a recording rule the alert and dashboard share.

Rate-after-sum (#3) — the phantom spike on restart. sum() applied before rate() merges counters across instances, so when one resets to 0 the aggregate dips and rate over it sees a drop — the per-instance reset detection that would have handled it is gone. Confirm by graphing sum(rate(x[5m])) against rate(sum(x)[5m]) across a known restart; the second shows the spurious spike. Fix: always rate() first, then sum. There is no correct use of rate(sum(...)) for counters.

Pinned p99 (#6) — the truth above the top bucket. The 99th percentile falls in the last finite bucket below +Inf; because +Inf has no upper bound to interpolate toward, histogram_quantile returns the largest finite le, and your real p99 sits above it, invisible. Confirm with count by (le) (http_request_duration_seconds_bucket); if the pinned value equals the top finite le, that’s it. Fix: re-instrument with higher finite buckets above your real tail so the percentile lands between two finite boundaries.

Best practices

• Know the metric type before you choose a function. Counter → rate/increase; gauge → read raw / deriv / delta; histogram → histogram_quantile over buckets. The engine won’t stop you; you must. When unsure, curl .../metrics | grep '# TYPE'.
• rate() before sum(), never the reverse. Per-instance reset detection only works if you rate the per-series counters first. rate(sum(...)) has no correct use for counters.
• Aggregate buckets and call histogram_quantile once; never average percentiles. Preserve le in every aggregation feeding the quantile. Averaging per-instance quantiles silently understates the tail.
• Lay out histogram buckets around your SLO thresholds. The quantisation error is bounded by the bucket your percentile lands in. Put boundaries densely where your objective is, and add finite buckets above your real tail so p99 isn’t pinned to the top bucket.
• Size range windows to at least ~4× the scrape interval. A window ≤ scrape interval flickers or returns empty. On Grafana, prefer $__rate_interval, which derives a safe window from the scrape interval and step.
• Anchor your regexes deliberately. =~ is fully anchored — use .* when you mean “contains.” A silently empty result is usually an over-tight anchored match.
• Promote hot or repeated queries to recording rules, and make alerts and their panels read the same recorded series. Expensive expressions (quantiles, subqueries, big aggregations) precomputed once and shared mean the panel and the pager can never disagree after a “cleanup.”
• Add a for: clause and consider multi-window burn-rate. A bare threshold flaps on a single missed scrape; for: and multi-window logic absorb transient blips.
• Check join-key uniqueness before composing cross-metric math. A non-unique “one” side throws many-to-many mid-incident. Aggregate or extend on() first.
• Prefer without for exploratory dashboards, by for authoritative single numbers. without carries new labels through; by pins the grouping.
• Validate with promtool in CI. promtool check config/rules plus promtool test rules with synthetic series, pinning the expected value, so a refactor can’t reintroduce a silent lie.
• Use the expression browser’s Table tab to inspect intermediate vectors. When a query is wrong, evaluate it inside-out and check the type and labels at each layer.

A compact “which function for which question” reference to keep beside you:

Security notes

PromQL is a read language, but the queries and the endpoints they run on have real security implications:

• Treat the query API as sensitive read access. /api/v1/query and /graph can expose every label and value Prometheus holds — internal hostnames, namespaces, traffic patterns. Put Prometheus behind authentication and network restrictions; it ships with no auth of its own.
• Unbounded queries are a denial-of-service vector. {__name__=~".+"} or a deeply nested subquery can pin CPU and memory. Enforce --query.max-samples, --query.timeout, and --query.max-concurrency, and consider a query-fronting proxy that rejects abusive expressions.
• Don’t leak topology through labels you query and display. A secret-ish label (a tenant key, an internal IP) is an exposure on every panel and alert annotation that includes it. Drop or hash such labels at relabeling time, upstream of the query.
• Recording and alerting rules run with the server’s full data access. A rule file is code — review it in PRs; a careless rule can record an enormous high-cardinality series and exhaust storage.
• Scope remote-read/remote-write endpoints. When queries fan out to long-term storage (Thanos, Mimir), authenticate and authorise per tenant so one tenant’s query can’t read another’s series.
• Alert annotations can exfiltrate data. A {{ $labels.<x> }} or {{ $value }} sends that value to Slack/email/PagerDuty — don’t template sensitive labels into messages that leave your trust boundary.

The security-relevant query controls and what each protects against:

Cost & sizing

PromQL is free, but the shape of your queries and the series they create drive real cost in CPU, memory, and storage:

• Cardinality is the dominant cost driver, and your queries shape it. Every unique label combination is a series; count_values, careless by groupings, and high-cardinality labels (user IDs, full URLs) multiply series and storage. A recorded quantile grouped by (job, le) is cheap; by (job, instance, path, le) can be thousands of times larger.
• Recording rules trade storage for query speed — spend it deliberately. Each rule adds a series evaluated every interval. Record the expensive, frequently-read aggregates (RED/USE panels, SLO quantiles); don’t reflexively record everything.
• Subqueries and big range windows cost CPU at query time, on every refresh. A [1h:1m] subquery materialises ~60 inner points per evaluation; on a 20-panel dashboard refreshing every 10s that’s a lot of recomputation. Promote to recording rules.
• Range-window size affects samples scanned. rate(x[5m]) scans far fewer than rate(x[1h]). For alerting, the smallest window that’s still ≥4× scrape is cheapest and most responsive.
• Native histograms reduce series count dramatically. A classic histogram with 12 buckets across 40 pods is 480+ _bucket series; a native histogram collapses each pod’s into one, cutting cardinality and storage for the same fidelity.

Rough resource intuition for a single Prometheus (these scale with your data, treat as starting points):

In INR terms, the cost of PromQL is mostly the cost of the Prometheus (and long-term storage) it runs on: a well-tuned VM handling ~1–2M active series might be a few thousand rupees a month, an untuned one drowning in high-cardinality series several times that and still slow. The cheapest optimisation is almost always fewer, lower-cardinality series and recording the hot aggregates — both shaped by how you write PromQL.

Interview & exam questions

1. Instant vector vs range vector, and which functions consume which? An instant vector is one sample per series at the eval timestamp (http_requests_total); a range vector is a set of samples per series over a lookback window (http_requests_total[5m]). rate, irate, increase, and the _over_time family consume a range vector and return an instant vector; graph instant vectors, not range vectors, and passing the wrong one gives the “expected type range vector … got instant vector” error.

2. Why rate() before sum(), and what breaks if reversed? rate detects counter resets per series — a drop is treated as a restart, not a negative rate — but sum first merges counters, so a restart makes the aggregate dip and rate(sum(...)) produces a spurious spike. Always sum(rate(x[5m])); rate(sum(...)) has no correct use for counters.

3. Why can’t you average percentiles, and how do you get a fleet-wide p99? A percentile is a property of a distribution; the mean of several instances’ percentiles isn’t the percentile of the combined distribution and consistently understates the tail. Aggregate the buckets (preserving le) and compute the quantile once: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))).

4. What does histogram_quantile do, and what bounds its accuracy? It does linear interpolation within the bucket where the quantile falls, using the cumulative _bucket counters, so accuracy is bounded by that bucket’s width. If the quantile lands in the last finite bucket below +Inf it returns that finite le (no upper bound to interpolate toward), so p99 can pin to a round number.

5. Why might a rate() panel be blank or flicker, and how do you fix it? rate needs at least two samples in the window; a window ≤ scrape interval (or a brand-new series) frequently catches zero or one and returns nothing or flickers. Fix by widening the window to roughly 4× the scrape interval, or use Grafana’s $__rate_interval.

6. Explain on, ignoring, and group_left. Binary operators match by the full label set by default; on (labels) restricts matching to those labels, ignoring (labels) matches on all except them. group_left/group_right handle many-to-one matching — declaring which side may have extra labels — and group_left(extra) also copies extra labels from the “one” side (the enrichment pattern with info metrics like kube_pod_info).

7. Is a non-integer increase a bug? No — rate and increase extrapolate to the exact window boundaries, which rarely align with scrape timestamps, so increase(x[1h]) legitimately returns fractional values like 1342.8. It’s an estimate by design; don’t floor it and claim exactness.

8. When should a query become a recording rule? When it’s expensive (quantiles over many buckets, large aggregations, subqueries) and run repeatedly (a panel or alert). Recording computes it once per interval into a series the alert and dashboard share — so they never diverge — and removes per-refresh recomputation, though each rule adds a series and write load.

9. What is a subquery and what’s its cost? A subquery expr[outer_range:resolution] runs a range function over an instant expression’s result — e.g. max_over_time(sum(rate(x[5m]))[1h:1m]). It materialises many intermediate points at query time on every refresh, so it’s expensive; promote the inner expression to a recording rule for anything run repeatedly.

10. How do offset and @ differ? offset 7d shifts the lookback by a relative duration from each step (week-over-week). @ <unix_ts> pins evaluation to an absolute timestamp regardless of the step (a flat reference line at an event like a deploy), and @ start()/@ end() pin to the query range’s edges.

11. rate() gives a plausible number but the graph behaves oddly — how do you check it’s even a counter? Inspect the type, which lives only in metadata: curl .../api/v1/metadata | jq '.data["<metric>"]' or curl .../metrics | grep '# TYPE <metric>'. If it says gauge, rate is wrong — use deriv/delta or read it raw; the engine never enforces this.

12. quantile(0.99, ...) vs histogram_quantile(0.99, ...)? quantile(0.99, latency_seconds) is an aggregation operator computing the 99th percentile across series values at this instant — not a latency percentile. histogram_quantile(0.99, sum by (le) (rate(..._bucket[5m]))) computes the 99th percentile of the observed distribution from buckets; for request latency p99 you want the latter.

These map to the Prometheus Certified Associate (PCA) — PromQL, the data model, recording and alerting rules, instrumentation — and to SRE/observability interview loops generally. The histogram and SLO math underpins burn-rate alerting questions. A compact mapping for revision:

Quick check

1. You write rate(node_memory_MemAvailable_bytes[5m]) and get a number on the graph. Why is this query meaningless, and what should you use instead?
2. A latency panel reads avg(histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))). What is wrong with it, and write the correct fleet-wide p99.
3. Your rate(http_requests_total[15s]) panel is blank even though traffic is flowing and the scrape interval is 15s. What’s happening and what’s the one-line fix?
4. An aggregate rate shows a sharp negative spike exactly when one pod restarted. What’s the query mistake and the correct form?
5. {handler=~"/api/users"} returns nothing, but you can see /api/users/42 requests in another tool. Why, and how do you fix the matcher?

Answers

1. node_memory_MemAvailable_bytes is a gauge (it moves up and down), and rate assumes a monotonic counter, so it computes a per-second “rate” of a non-monotonic value — nonsense. To see how available memory is changing, read it raw, or use deriv(node_memory_MemAvailable_bytes[1h]) for a slope, or delta(...[1h]) for the change over a window.
2. It averages percentiles — computing p99 per instance and then averaging, which is mathematically meaningless and understates the tail (a few slow pods get diluted). The correct form aggregates the buckets and computes the quantile once: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))).
3. The range window ([15s]) equals the scrape interval, so it frequently catches only one sample, and rate needs at least two — hence empty/flickering. Widen the window to at least ~4× the scrape interval: rate(http_requests_total[1m]) (or use $__rate_interval in Grafana).
4. The query summed counters before rating them (rate(sum(...)) or sum then rate), which destroyed per-instance counter-reset detection, so the restarting pod’s reset looked like the aggregate going backwards. The correct form rates first: sum(rate(http_requests_total[5m])).
5. PromQL regex matchers are fully anchored, so =~"/api/users" matches only the exact string, not /api/users/42. Use a substring pattern: {handler=~"/api/users.*"} (or ".*\/api\/users.*" if it can appear mid-string).

Glossary

• Sample — a single (timestamp, float64) data point; the only primitive PromQL operates on.
• Series — a metric name plus labels identifying one stream of samples; the name is the reserved __name__ label.
• Instant vector — one sample per series at the eval timestamp; graphable, and the operand of binary operators.
• Range vector — a set of samples per series over a lookback window ([5m]); only valid as input to range functions like rate.
• Counter — a cumulative metric that only increases until a reset to 0; read via rate/irate/increase, never raw.
• Gauge — a metric that goes up and down; read raw or via delta/deriv/predict_linear, never rate.
• Histogram — a family of cumulative _bucket{le} counters plus _sum and _count; queried with histogram_quantile over rated buckets.
• le — the bucket-boundary label on histogram buckets; must be preserved through every aggregation feeding histogram_quantile.
• rate / irate / increase — per-second average / instantaneous (last two samples) / total-over-window functions for counters, all reset-aware and extrapolated.
• histogram_quantile(φ, v) — interpolates the φ-quantile from cumulative bucket rates by linear interpolation within the matched bucket.
• Native histogram — an exponential, high-resolution histogram stored as a single series, removing manual le layout and reducing cardinality.
• by / without — aggregation clauses choosing which labels survive (by keeps only the listed; without keeps all except the listed).
• Vector matching — how binary operators pair series: by full label set, or restricted with on/ignoring, with group_left/group_right for many-to-one.
• offset / @ — time-shift modifiers; offset shifts the lookback by a relative duration, @ pins evaluation to an absolute timestamp.
• Subquery — expr[outer_range:resolution]; runs a range function over an instant expression’s result, at query time (expensive).
• Recording rule — a server-side rule that pre-computes an expression every interval and stores the result as a new series for cheap, shared reads.
• promtool — the Prometheus CLI for check config/check rules and test rules (unit-testing rules against synthetic series).

Next steps

You can now read any PromQL query as a pipeline and tell whether it computes what you think. Build outward:

• Next: Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir — operationalise the recording rules introduced here and push aggregates to durable storage.
• Related: Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus — shape the label sets your queries depend on, and keep series count (and cost) in check.
• Related: Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code — turn these queries into panels people trust, with $__rate_interval and template variables.
• Related: SLOs and Error Budgets in Practice: Defining SLIs and Building Multi-Window Burn-Rate Alerts — apply the histogram and ratio math to error budgets and multi-window burn-rate alerting.
• Related: Designing Alertmanager Routing Trees: Grouping, Inhibition, Silences, and Dedup — route the alerts your correct queries now fire reliably.
• Related: Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation — connect the metrics you query here to the traces that explain them.