SLOs and Error Budgets in Practice: Defining SLIs and Building Multi-Window Burn-Rate Alerts

Most alerting catalogs are a museum of every incident the team has ever had: one rule per scar, each with a threshold someone picked at 3am and nobody has touched since. The result is a pager that fires when nothing is wrong and stays silent when something is. Service Level Objectives (SLOs) replace the museum with a single question — are we spending reliability faster than we agreed users would tolerate? — and burn-rate alerts answer it with arithmetic instead of vibes. The method is the multi-window multi-burn-rate (MWMBR) design from Google’s SRE Workbook — the closest thing alerting has to a solved problem. What is rare is seeing every number in it derived rather than cargo-culted, which is what this article does.

You will build the whole system from first principles. First the Service Level Indicator (SLI) — the good/valid event ratio that makes user experience measurable — across availability, latency, and quality. Then the SLO target and the error budget that falls out of it, with the nines-to-downtime math. Then burn rate, defined precisely, and the proof that every single-threshold alert is either too noisy or too slow. From there we derive the canonical four-rung ladder — 14.4x over 1h+5m (page), 6x over 6h+30m (page), 3x over 1d+2h (ticket), 1x over 3d+6h (ticket) — showing where 14.4 comes from, why the short window is one-twelfth of the long, and proving that the budget consumed by the time each alert fires is constant regardless of outage severity.

Everything ships as real code: recording rules for all seven windows, the four alert rules, promtool unit tests, low-traffic guards, a budget-wired release gate, and the Grafana panels the weekly review needs. This is the conceptual deep-dive behind the tooling in SLOs as Code: Authoring SLIs with OpenSLO and Generating Burn-Rate Alerts via Sloth and Pyrra — read this to understand every number those generators emit, and that to stop writing the YAML by hand.

What problem this solves

Threshold alerting fails for a structural reason, not a tuning reason: a static threshold on an instantaneous signal cannot distinguish how much damage an event does to a monthly promise. A 90-second deploy blip and a three-day slow leak can produce the same 5-minute error ratio, but one costs 0.07% of the month’s budget and the other costs all of it. Any threshold tight enough to catch the leak pages constantly on blips; any threshold loose enough to ignore blips sleeps through the leak. Teams respond by adding more rules, and the museum grows.

The damage is concrete. On-call acknowledges-and-ignores because 90% of pages need no action, so the page that matters gets the same 20-minute shrug. Postmortems discover the SLO was missed with zero alerts fired, because every rule watched a cause (CPU, queue depth, pod restarts) and the failure arrived through one nobody had scarred on yet. Severity debates burn incident minutes because there is no shared currency for harm. And nobody dares delete a rule, because nobody remembers which incident it commemorates.

Who hits this: every team grown past a handful of services, every platform/SRE group inheriting 40 teams’ ad-hoc rules, and every organization whose reliability conversations with product are vibes-based (“it feels flaky”) rather than budget-based (“we spent 140% of last month’s budget; the roadmap pauses”). The fix: one indicator per user journey, one target, one budget, and exactly four alerts whose thresholds are arithmetic, not archaeology.

Learning objectives

By the end of this article you can:

• Define SLIs as good/valid ratios for availability, latency, and quality; choose the measurement point (edge, white-box, synthetic, RUM); specify exactly which traffic counts as valid.
• Do the nines math fluently: convert any target and window into allowed downtime and bad requests, and explain why request-based budgets beat time-based ones.
• Define burn rate precisely, convert between burn rate, error ratio, and time-to-exhaustion, and state the maximum burn rate for a given target.
• Prove — via precision/recall/detection/reset — why single-threshold and for:-duration alerts fail, reproducing the SRE Workbook’s six-iteration progression.
• Derive all sixteen numbers in the four-rung ladder, and recompute them for a 28-day window or a looser SLO where 14.4x is mathematically impossible.
• Compute budget-consumed-at-detection and show it is invariant to outage severity — the property that makes the ladder trustworthy.
• Implement it all in Prometheus: seven recording-rule windows, four alerts, division-by-zero and low-traffic guards, promtool tests, Alertmanager routing with inhibition.
• Operate the budget: write a policy with freeze/release gates, wire a CI deploy gate to budget remaining, and build the dashboard the weekly review runs on.

Prerequisites & where this fits

You should be fluent in PromQL basics — rate(), sum by (), histogram _bucket/_count conventions — at the level of PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work, know why recording rules and pre-aggregation matter (Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir), and have seen an Alertmanager config (Designing Alertmanager Routing Trees: Grouping, Inhibition, Silences, and Dedup is the routing companion).

This article sits at the top of the observability track — the layer where telemetry turns into decisions: who gets paged, what ships, what freezes. Downstream sit the operational practices: Building an On-Call Practice: PagerDuty Escalation, Alert Routing, and Actionable Runbooks for what happens after the page, and SLOs as Code: Authoring SLIs with OpenSLO and Generating Burn-Rate Alerts via Sloth and Pyrra for generating at scale what this article derives by hand. Nothing here needs Kubernetes, a mesh, or a vendor — one Prometheus and one HTTP counter are enough.

Core concepts

Five mental models carry the entire article.

An SLI is a ratio of good events to valid events — nothing else. Not a gauge, not a percentile estimate, not a health check: SLI = good_events / valid_events, a dimensionless number between 0 and 1 computed from two counters. Everything downstream inherits its meaning from those two nouns; if they are wrong (health checks in the denominator, client mistakes in the numerator), every derived number lies with perfect arithmetic confidence.

An SLO is a promise over a window; the error budget is the promise’s complement made spendable. “99.9% of valid requests succeed over a rolling 30 days” defines a target and a horizon. The error budget is 1 − SLO — for 99.9%, a 0.1% allowance: 43.2 minutes of total downtime or one bad request in a thousand. Reframing “percentage to defend” as “quantity to spend” is the entire cultural trick: budgets can be spent on velocity, tracked, and run out.

Burn rate is a speed, not a level. A burn rate of 1 consumes budget at exactly the pace that exhausts it at the window’s end — break-even. Burn 14.4 means the month’s budget gone in about two days. Alerting on burn rate asks the only question the pager should ask: at this speed, when do we run out? — which maps directly to how fast a human must respond.

Every alerting design is a four-way trade: precision, recall, detection time, reset time. You cannot maximize all four. Single-threshold alerts sacrifice precision or recall entirely; in the MWMBR construction, multiple burn rates buy recall across fast and slow failures, long windows buy precision, and the short window bounds reset time so a resolved incident stops paging.

The budget is a governance instrument, not a monitoring artifact. Alerts are the small half. The error-budget policy — the pre-agreed, product-signed answer to “what happens when the budget is gone?” — converts the number into freeze decisions, release gates, and prioritization. Without it, an exhausted budget is a sad dashboard; with it, a brake the whole organization respects.

Choosing SLIs: measure what users experience

A Service Level Indicator starts as a sentence about users, not a metric. The SLI specification is the user-facing statement — “checkout requests succeed quickly” — and the SLI implementation is the measurable proxy: which metric, from which observation point, with which events counted as good and valid. Skip the specification and you get an indicator that moves when infrastructure is unhappy rather than when users are. The test for a good SLI is brutal: it moves when users suffer and stays flat when they do not — if you can construct a scenario that breaks that, fix the indicator before arguing about targets.

The SLI menu

Nearly every service maps onto a small menu: request-driven services get availability, latency, and quality; pipelines get freshness, correctness, and coverage; storage gets durability. All reduce to good/valid:

The quality SLI is the one experts underuse. Systems that degrade gracefully — stale cache, a fallback model, partial search results — look perfectly available while quietly serving worse answers. If your architecture has fallbacks, instrument them as a first-class label and give quality its own SLO; availability will never see the degradation:

Where to measure: the observation point decides what you can see

The same SLI specification implemented at different points tells different truths. Measure as close to the user as you can while still owning the signal — and know each point’s blind spots, because the worst is fatal: a hard-down service often emits no metrics, so a purely white-box SLI can be silent during the exact outage class it exists to catch.

In production, layer them: the SLO of record computes from the edge, white-box metrics explain why it moved, and a synthetic probe plus missing-data alerts cover the “so dead it stopped reporting” hole.

What counts as valid: the exclusion table you must write down

Every SLI needs an explicit validity policy, decided once and encoded in the query. The default failure is silent inclusion: health checks inflate the denominator, client errors burn your budget for someone else’s bug, retries triple-count a single user failure.

Encode exclusions with labels at instrumentation time (path!="/healthz", a synthetic="true" label on probes) rather than fragile regex archaeology later — watching the cardinality budget as you do (Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus).

Latency SLIs: thresholds, not averages, and never quantiles

Latency SLIs must be count-based threshold ratios: “the fraction of requests completing within 500 ms”. The two seductive alternatives are both wrong: averages hide the tail where users suffer (a 100 ms mean coexists happily with 5% of users waiting 8 seconds), and histogram_quantile() is an estimate that interpolates inside buckets — an SLO built on it moves when someone changes bucket layout. The count of events at or under a boundary is exact:

# Latency SLI: fraction of requests completing within 500ms over 1h (exact, count-based)
sum by (job) (rate(http_request_duration_seconds_bucket{le="0.5"}[1h]))
  /
sum by (job) (rate(http_request_duration_seconds_count[1h]))

The non-negotiable consequence: the threshold must sit exactly on an le bucket boundary, so latency SLOs are chosen at instrumentation design time. If the histogram has le="0.25" and le="1" but you promised 500 ms, you cannot compute your SLI — re-bucket or change the promise. Prometheus native histograms relax this with histogram_fraction(0, 0.5, rate(http_request_duration_seconds[1h])), estimating the fraction below an arbitrary bound from exponential buckets; a real improvement, but bucket-edge discipline remains the safe default on classic histograms.

Serious services carry two latency objectives, because one threshold cannot protect both the median feel and the tail:

Each row is a separate SLO with its own budget and ladder — never blend them into one number. Finally, decide the valid set: the common convention is successful requests only in the latency denominator, so a fast 500 is not good latency and a slow 500 is not punished twice (availability already caught it). Whichever you pick, write it down — the conventions diverge exactly during incidents.

SLO targets and the arithmetic of nines

Every target is a downtime number in disguise. Translate nines to minutes without a calculator, because every “why not 99.99?” conversation is actually about four minutes a month:

Read it with two lenses. Per-window downtime is the operations constraint: at 99.99% one 5-minute incident blows the month. Per-week downtime is the human constraint: 99.99% is a minute a week, so no human-in-the-loop response can be part of meeting it — the system must survive failures automatically, and the pager exists only for the automation’s failures.

Request-based budgets: the honest unit

Time-based budgets (“43 minutes of downtime”) are intuition; request-based budgets are what the SLI actually computes, and they are better where it matters. A partial outage failing 20% of requests for an hour costs exactly 20% of that hour’s traffic in budget — time-based accounting forces an arbitrary “was it down?” call. Request weighting also discounts 3am automatically: low-traffic outages burn little budget, matching real user harm.

The first row is the trap: a 1,000-request-per-month service at 99.95% has a budget of half a request — one failure blows the month. That is a target-definition problem, not an alerting problem; the low-traffic section gives the escape hatches. The last row is the other lesson: at real traffic, budgets are large enough that burn rate — not individual failures — is the only meaningful signal.

The window: rolling 28/30 days, not calendar months

The window is part of the promise, and there are only a few defensible choices:

Rolling windows are strictly better for operating (yesterday’s outage ages out gradually instead of vanishing at month end); calendar windows are sometimes required for contracting. Run rolling internally, translate to calendar for the SLA report, and never operate on the calendar number.

Setting the target: baseline, not aspiration

The correct first target is discovered, not declared. Measure the SLI untouched for four to six weeks, then set the target at or slightly below what the service already achieves — an SLO violated on day one is either ignored or triggers a freeze nobody signed up for. Ratchet deliberately: tighten only when users demonstrably need it and the budget has gone chronically unspent (more than half unspent for a quarter means the target taxes velocity — tighten it, or spend the surplus on riskier rollouts). Each additional nine roughly multiplies engineering cost — redundancy, rollout caution, on-call rigor, dependency discipline — while the marginal user barely notices; set the target where users would change behavior or churn, not at the prettiest number.

Two structural checks before ratifying. Dependency ceiling: your SLO is capped by hard dependencies — three serial hard dependencies at 99.95% each cap you at 0.9995³ ≈ 99.85% regardless of your own code; promising 99.9% requires fewer hard dependencies or degradation that converts them to soft ones. SLA headroom: the internal SLO must be strictly tighter than any external SLA — the SLO breach is the early warning. A sane stack:

Error budgets and burn rate, precisely

Now the definitions that everything else compiles down to. Fix the notation once:

β  (error budget fraction)  = 1 − SLO                      e.g. 99.9% → β = 0.001
T  (SLO window)             = 30 d = 720 h = 43,200 min
E  (observed error ratio)   = bad events / valid events, over some lookback
B  (burn rate)              = E / β                        dimensionless speed

Allowed bad events per window                 = β × valid events
Time to exhaustion from a full budget         = T / B
Budget fraction consumed in time P at burn B  = B × P / T
Maximum possible burn rate                    = 1 / β      (E cannot exceed 1)

Burn rate 1 is break-even: the budget lands at exactly zero as the window closes. Burn rate 2 exhausts in 15 days. The table below is worth memorizing down the left column, because it is the severity scale your pager will speak from now on:

Two properties of that ceiling matter. First, the maximum burn rate is 1/β — 100 at 99%, 1,000 at 99.9%, 10,000 at 99.99% — because the error ratio cannot exceed 1; for loose SLOs (90%, 95%) some canonical multipliers are unreachable and the ladder must be adjusted, as shown later. Second, burn rate is traffic-independent: 1.44% errors is 14.4x at 10 rps or 10,000 rps — which lets one alert design serve every service, with a low-traffic caveat handled below.

And because burn rates compare against your budget, the same multiplier means a different error ratio per target. This matrix is what your alert literals will encode:

Finally, the intuition-builder: what do common incident shapes actually cost? Assume a 99.9% target, whose budget is 43.2 bad-minutes — one bad-minute being 100% of traffic failing for one minute, or equivalently 1% of traffic failing for 100 minutes:

Sit with the last two rows: the headline outage that gets the war room costs less budget than a 1% error rate humming for a day, and barely a tenth of a 0.2% leak over a month. Threshold alerting catches the first row and is structurally blind to the last two — loud cheap incidents, silent expensive ones. That asymmetry is the entire case for burn-rate alerting.

Why single-threshold alerts fail

Every alerting strategy should be judged on four dimensions, and being explicit about them is what turns alert design from folklore into engineering:

Now score the naive alert every team writes first — page when the 5-minute error ratio exceeds the SLO threshold (0.001 for 99.9%):

The alert is simultaneously too sensitive and not sensitive enough, and no threshold fixes both rows: lower it and the blips get worse; raise it and the leaks get worse. The problem is structural — an instantaneous ratio carries no information about duration, and damage is the product of both.

The Google SRE Workbook’s “Alerting on SLOs” chapter works through six successive designs, each fixing the previous one’s defect. Knowing the progression matters because you will meet all six in the wild, usually defended by someone who stopped at iteration 2 or 3:

Two numbers in that table are worth pinning. Iteration 2’s precision gain: a 36-hour window at threshold represents 36/720 = 5% of the budget, so every firing is significant — but the same integration keeps the alert red for most of 36 hours after recovery. Iteration 4’s detection figure: a total outage crosses 14.4 × 0.001 on a 1h window after 3600s × 0.0144 ≈ 52 seconds, while a 6x burn (budget gone in 5 days) never touches the single threshold at all.

Iteration 3’s for: clause deserves its own autopsy, because it is the most common defect in hand-rolled SLO alerts. A for: duration requires the condition true at every evaluation; a single good evaluation resets the pending timer to zero. An error ratio oscillating between 0.09% and 0.30% every few minutes — a genuine ~2x sustained burn — never accumulates one contiguous hour above threshold and never fires. A long-window rate integrates instead of gating: the same oscillation averages ~0.2% and alerts. That is the precise reason “just add for: 1h” is not a substitute for a long window. Recall demands several thresholds (iteration 5); bounded reset demands the second window (iteration 6) — the design we now derive completely.

The four-alert ladder: deriving every number

The MWMBR ladder is four alert rules per SLO. Each rung fires when the burn rate exceeds a threshold over a long window (the burn is sustained and significant) and a short window (it is still happening right now). The canonical ladder for a 30-day SLO, every derived quantity in one place:

None of these numbers is arbitrary. The procedure: choose how much budget an event may consume before a human hears about it, and over what horizon; the burn rate follows. Burn B sustained for window W consumes B × W / T of the budget, so “fraction f gone in window W” forces B = f × T / W:

So 14.4 is not a magic constant — it is 2% × 720h / 1h, the burn rate that torches two percent of a 30-day budget in sixty minutes, equivalently the pace that exhausts everything in 50 hours. The 2%/5%/10% choices encode a defensible escalation philosophy: several page-worthy events fit in a month before a quarter of the budget is gone, and slow leaks surface as tickets with 90% of the budget still intact. Rung 4’s threshold of exactly 1 is the anchor: burning at precisely the sustainable pace earns a ticket — and anything below 1x is within budget and should never alert at all. That last clause is what finally lets you delete the museum.

Why the short window, and why W/12

With only the long window, reset is terrible: after a 20-minute total outage, a 1-hour window still averages 33% errors at recovery and stays above 1.44% until the outage almost fully exits the window — about 59 more minutes of firing while everything is healthy. The short window, required simultaneously by and, collapses on recovery within its own length. The workbook’s short = long/12 (5m, 30m, 2h, 6h) is a tested compromise: short enough to bound reset tightly, long enough that one scrape hiccup does not gate a real page. Detection is essentially unaffected: during a genuine burn the short window crosses far earlier (for a total outage, the 5m window crosses 1.44% in ~4 seconds), so the long window remains the pacing element.

One nuance: after a big outage the budget really is spent. The MWMBR position is that burn alerts report active burning only; the “we spent X%” state belongs to dashboards and a separate budget-remaining alert. Mixing the two turns the pager into a grudge-holder.

Detection time: severity-proportional by construction

For a sustained error ratio E, the long window W with threshold B crosses after t = W × B × β / E (capped at W; never fires if E ≤ B × β). Detection scales inversely with severity — the property no fixed for: duration can give you:

(Add the for: duration and one evaluation interval for the real-world number; the notification-latency table later itemizes that overhead.) Two observations hide in this matrix. First, rungs 3 and 4 have identical detection times wherever both fire, because W × B is 72 hours for both (24×3 = 72×1). They are not redundant: rung 4’s lower threshold catches the 1x–3x burns rung 3 never sees, while rung 3 turns a 3x–6x burn into a ticket after 12–24 hours instead of waiting on the 3-day window. Second, during a severe outage all four rungs eventually fire — which is why Alertmanager inhibition (pages suppress tickets for the same SLO) is part of the design, not a nicety.

Budget consumed at detection: the invariance that makes the ladder trustworthy

Here is the property that justifies trusting four rules over forty. At the moment the long window crosses its threshold, E × t = W × B × β by definition. The fraction of total budget consumed is E × t / (β × T) = W × B / T — the error ratio E cancels out. However bad the incident, by the time rung 1 fires you have spent 2% of the month’s budget; rung 2, 5%; rungs 3 and 4, 10%. A total collapse and a barely-over-threshold burn cost the same fraction at detection — faster burns are simply detected proportionally sooner.

This is the answer to “why these thresholds?”: the ladder guarantees no failure mode — fast crash or slow leak — consumes more than ~10% of the budget before a human is told, and no page fires for anything that consumed less than ~2%.

Coverage: no gaps between the rungs

Read as intervals on the burn-rate axis, the ladder is gapless from 1x upward:

Recomputing the ladder: 28-day windows and loose SLOs

The multipliers encode “f% in W” for T = 720 h; change the window and the faithful derivation shifts:

Generators like Sloth derive multipliers from the declared window automatically — one good reason to let them.

The other adjustment is forced, not stylistic: the error-ratio threshold is B × β, and a ratio cannot exceed 1, so the ladder breaks for loose SLOs where 14.4 × β ≥ 1:

If you run 90–95% targets, also revisit the SLI — very loose targets often mean the indicator includes traffic it should not, or the service needs a quality SLI instead. The method is portable: Google Cloud Monitoring exposes native burn-rate alerting on SLOs, Datadog SLO monitors implement the same long/short-window design, and Grafana’s SLO tooling and Pyrra generate exactly the rules we are about to write by hand.

Building it in Prometheus: recording rules, alerts, and tests

The implementation has three layers: recording rules that pre-compute the error ratio over every window the ladder references, four alert rules comparing those series against B × β, and unit tests proving the arithmetic before it pages anyone. Pre-computation is not optional at fleet scale — alert expressions must stay O(1) lookups over recorded series, not repeated 3-day range scans over raw counters (Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir covers why long-window queries hurt).

The window inventory

Four rungs × two windows = seven distinct lookbacks (6h appears twice — rung 2’s long and rung 4’s short):

Recording rules for every window

The naming convention is the standard level:metric:operations. The ratio of two rate()s over the same window is dimensionless (per-second normalization cancels), so each series is the error fraction over that window:

# slo-recording-rules.yaml — availability SLI for every ladder window
groups:
  - name: slo-availability-ratios-fast
    interval: 30s
    rules:
      - record: job:slo_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))
        labels:
          slo: http-availability
      - record: job:slo_errors:ratio_rate30m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[30m]))
          /
          sum by (job) (rate(http_requests_total[30m]))
        labels:
          slo: http-availability
      - record: job:slo_errors:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total[1h]))
        labels:
          slo: http-availability

  - name: slo-availability-ratios-slow
    interval: 2m
    rules:
      - record: job:slo_errors:ratio_rate2h
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[2h]))
          /
          sum by (job) (rate(http_requests_total[2h]))
        labels:
          slo: http-availability
      - record: job:slo_errors:ratio_rate6h
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[6h]))
          /
          sum by (job) (rate(http_requests_total[6h]))
        labels:
          slo: http-availability
      - record: job:slo_errors:ratio_rate1d
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[1d]))
          /
          sum by (job) (rate(http_requests_total[1d]))
        labels:
          slo: http-availability
      - record: job:slo_errors:ratio_rate3d
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[3d]))
          /
          sum by (job) (rate(http_requests_total[3d]))
        labels:
          slo: http-availability

Three notes separate working systems from broken ones. Carry every routing label through by (...) — anything you route, group, silence, or inhibit on (team, tenant, env) must appear in the aggregation; Prometheus discards the rest here and nothing downstream can resurrect them. Exclusions live here too: if health checks share the counter, {code=~"5..", path!~"/healthz|/readyz"} goes on both numerator and denominator. The numerator must be a strict subset of the denominator — same metric and selectors plus the badness condition — or ratios above 1 and phantom burns follow.

The latency SLI records the same shape from histogram counters — the error ratio is one minus the good fraction (one window shown; repeat for the rest):

  - name: slo-latency-ratios-fast
    interval: 30s
    rules:
      - record: job:slo_latency_errors:ratio_rate1h
        expr: |
          1 - (
            sum by (job) (rate(http_request_duration_seconds_bucket{le="0.5"}[1h]))
            /
            sum by (job) (rate(http_request_duration_seconds_count[1h]))
          )
        labels:
          slo: http-latency-500ms

The four alert rules

Each alert compares long and short windows against the same B × β threshold. The literals are for β = 0.001 (99.9%); the slo label rides in from the recorded series, so four rules cover every job emitting the SLI:

# slo-burn-alerts.yaml — the four-rung MWMBR ladder, 99.9% / 30d (β = 0.001)
groups:
  - name: slo-availability-burn-alerts
    interval: 30s
    rules:
      # Rung 1 — 14.4x over 1h AND 5m. 2% of budget gone. Exhaustion in ~50h.
      - alert: ErrorBudgetFastBurn
        expr: |
          job:slo_errors:ratio_rate1h{slo="http-availability"} > (14.4 * 0.001)
          and
          job:slo_errors:ratio_rate5m{slo="http-availability"} > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          long_window: 1h
        annotations:
          summary: "Fast error-budget burn on {{ $labels.job }} (>14.4x)"
          description: "1h error ratio {{ $value | humanizePercentage }} — >14.4x burn on 1h and 5m windows. Budget gone in <50h at this pace; ~2% spent in the last hour."
          runbook_url: "https://runbooks.example.com/slo/fast-burn"

      # Rung 2 — 6x over 6h AND 30m. 5% of budget gone. Exhaustion in ~5d.
      - alert: ErrorBudgetSlowBurn
        expr: |
          job:slo_errors:ratio_rate6h{slo="http-availability"} > (6 * 0.001)
          and
          job:slo_errors:ratio_rate30m{slo="http-availability"} > (6 * 0.001)
        for: 5m
        labels:
          severity: page
          long_window: 6h
        annotations:
          summary: "Sustained error-budget burn on {{ $labels.job }} (>6x)"
          description: "6h error ratio {{ $value | humanizePercentage }} — >6x burn on 6h and 30m windows. Exhaustion in ~5 days at this pace."
          runbook_url: "https://runbooks.example.com/slo/slow-burn"

      # Rung 3 — 3x over 1d AND 2h. 10% of budget gone. Exhaustion in ~10d.
      - alert: ErrorBudgetHighBurnTicket
        expr: |
          job:slo_errors:ratio_rate1d{slo="http-availability"} > (3 * 0.001)
          and
          job:slo_errors:ratio_rate2h{slo="http-availability"} > (3 * 0.001)
        for: 15m
        labels:
          severity: ticket
          long_window: 1d
        annotations:
          summary: "Elevated error-budget burn on {{ $labels.job }} (>3x)"
          description: "24h error ratio {{ $value | humanizePercentage }} — >3x burn; ~10% of the 30d budget gone in a day. Investigate in business hours."

      # Rung 4 — 1x over 3d AND 6h. On-pace burn with zero margin. 10% gone.
      - alert: ErrorBudgetElevatedBurnTicket
        expr: |
          job:slo_errors:ratio_rate3d{slo="http-availability"} > (1 * 0.001)
          and
          job:slo_errors:ratio_rate6h{slo="http-availability"} > (1 * 0.001)
        for: 1h
        labels:
          severity: ticket
          long_window: 3d
        annotations:
          summary: "Error budget burning at or above sustainable pace on {{ $labels.job }}"
          description: "3d error ratio {{ $value | humanizePercentage }} — >=1x burn: budget exhausts exactly at window end, zero margin."

A subtlety: with and, the resulting sample (and $value) comes from the left-hand side — put the long window on the left so annotations report the sustained ratio, not the twitchy short one. The for: durations stay short: the short window already gates flakes, and every minute of for: adds a minute to every detection time above.

Timing budget: what the pager actually adds

The window math gives the crossing time; the pipeline adds latency on top. Budget it explicitly:

So the honest rung-1 story for a total outage is “~52 s of math plus ~4 min of pipeline”: a page in about five minutes, roughly 9–12% of the budget spent (each minute of total outage on 99.9% burns ~2.3%). If that is unacceptable, tune for:, group interval, and group_wait — in that order — before touching the ladder.

Guards: division by zero, missing series, and absent traffic

The ratio expression has well-defined but surprising edge behavior; every row here has caused a real 3am mystery somewhere:

The third and fourth rows are the fatal pair: a full outage that stops traffic or kills the scrape target can make burn alerts physically unable to fire. Mitigations are mechanical. First, zero-fill the numerator with the labeled-zero trick so “no errors yet” produces 0 instead of an empty vector:

      - record: job:slo_errors:ratio_rate5m
        expr: |
          (
            sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
            or
            sum by (job) (rate(http_requests_total[5m])) * 0
          )
          /
          sum by (job) (rate(http_requests_total[5m]))
        labels:
          slo: http-availability

Second, alert on the SLI disappearing — absent() needs a concrete selector, so this is one small rule per critical service, not a wildcard:

      - alert: SLOSignalAbsent
        expr: absent(job:slo_errors:ratio_rate5m{job="checkout"})
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "SLO signal for checkout has disappeared"
          description: "No availability SLI samples for 10m — scrape failure or total outage. Burn alerts are blind while this fires."

Third, the observation-point lesson again: an edge-sourced SLI keeps counting when every backend is dead (requests become 5xx at the edge), converting “no data” into an honest 100% error ratio — the strongest argument for measuring the SLO of record at the edge.

for:, keep_firing_for:, and evaluation cadence

Unit tests: prove the arithmetic before it pages anyone

promtool runs rules against synthetic series, turning the derivation into CI-checkable assertions. The notation 0+60x90 means “start at 0, increment by 60 per interval, 90 times” — a counter growing 60/min:

# slo-tests.yaml
rule_files:
  - slo-recording-rules.yaml
  - slo-burn-alerts.yaml
evaluation_interval: 1m
tests:
  # 20% error ratio => burn 200x at 99.9% => rung 1 must fire
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout", code="200"}'
        values: '0+240x90'   # 240 good/min
      - series: 'http_requests_total{job="checkout", code="500"}'
        values: '0+60x90'    # 60 bad/min  => 60/300 = 20% errors
    promql_expr_test:
      - expr: job:slo_errors:ratio_rate5m{job="checkout"}
        eval_time: 10m
        exp_samples:
          - labels: 'job:slo_errors:ratio_rate5m{job="checkout", slo="http-availability"}'
            value: 0.2
    alert_rule_test:
      - eval_time: 20m
        alertname: ErrorBudgetFastBurn
        exp_alerts:
          - exp_labels:
              job: checkout
              slo: http-availability
              severity: page
              long_window: 1h

  # Zero errors => nothing may fire
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout", code="200"}'
        values: '0+300x120'
      - series: 'http_requests_total{job="checkout", code="500"}'
        values: '0+0x120'
    alert_rule_test:
      - eval_time: 60m
        alertname: ErrorBudgetFastBurn
        exp_alerts: []

promtool check rules slo-recording-rules.yaml slo-burn-alerts.yaml
promtool test rules slo-tests.yaml
# Expected:
#   SUCCESS

Extend the suite with the cases that catch real regressions: a 0.5% trickle firing only ticket rungs, a recovered outage where the short window must resolve the page, and a label-propagation assertion (team present on the fired alert). This file is the closest thing alerting has to a proof of correctness — treat it as mandatory.

Shipping the rules: files, CRDs, and generators

On plain Prometheus, the two YAML files go into rule_files: and CI runs the promtool pair on every PR. On Kubernetes with the prometheus-operator, wrap the same groups in a PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-slo-burn
  namespace: monitoring
  labels:
    release: kube-prometheus-stack   # must match the operator's ruleSelector
spec:
  groups:
    # paste the recording + alerting groups from above

Hand-maintaining seven windows × N SLOs is the toil generators exist for. In Sloth, the article so far compresses to a spec; sloth generate -i slo.yaml -o rules.yaml emits the recording rules and the same four-rung ladder you now know how to verify:

version: "prometheus/v1"
service: "checkout"
labels:
  team: payments
slos:
  - name: "requests-availability"
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="checkout",code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="checkout"}[{{.window}}]))
    alerting:
      name: CheckoutErrorBudgetBurn
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

Pyrra does the same from a ServiceLevelObjective CRD with a built-in UI; the tooling comparison and OpenSLO portability live in SLOs as Code: Authoring SLIs with OpenSLO and Generating Burn-Rate Alerts via Sloth and Pyrra. The point of deriving everything by hand is that you can now audit what any generator emits.

Routing: one incident, one page

During a severe burn all four rungs fire; without coordination the on-call gets four notifications for one problem. Group by SLO and service, route by severity, let pages inhibit tickets for the same {slo, job} pair:

# alertmanager.yaml (fragment)
route:
  receiver: default
  group_by: ['slo', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers: [ severity = page ]
      receiver: pagerduty
    - matchers: [ severity = ticket ]
      receiver: jira

inhibit_rules:
  - source_matchers: [ severity = page ]
    target_matchers: [ severity = ticket ]
    equal: ['slo', 'job']

The equal: list is the part people get wrong: inhibition applies only when those labels match between source and target — what stops a checkout page from silencing a search ticket. Validate routing with amtool config routes test --config.file=alertmanager.yaml severity=page slo=http-availability job=checkout; Designing Alertmanager Routing Trees: Grouping, Inhibition, Silences, and Dedup covers the full tree design, silences, and mute-time patterns.

When traffic is low: adapting the ladder

Burn rate is a ratio, and ratios go feral when the denominator is small. The canonical ladder assumes the short windows contain hundreds of events; below that, single failures swing the ratio across page thresholds. Run the numbers for a 99.9% SLO (rung-1 threshold 1.44%):

At one request per minute, a single 500 makes the 5m ratio 20% and the 1h ratio 1.67% — both over threshold, page fired, human woken, for one request a retry probably rescued. The adaptations, in order of preference:

The gate is one and per page rung — here requiring at least 1 rps averaged over the hour (3,600 events) before a fast page is eligible:

      - alert: ErrorBudgetFastBurn
        expr: |
          (
            job:slo_errors:ratio_rate1h{slo="http-availability"} > (14.4 * 0.001)
            and
            job:slo_errors:ratio_rate5m{slo="http-availability"} > (14.4 * 0.001)
          )
          and on (job)
          sum by (job) (rate(http_requests_total[1h])) > 1
        for: 2m
        labels:
          severity: page

Do not gate the ticket rungs — over 1–3 days even small services accumulate enough events, and tickets are cheap. And remember what the gate means: below the floor you have chosen not to page, so the synthetic probe and SLOSignalAbsent are all that stand between you and a silent weekend outage.

Error-budget policy: making the number bite

Alerts spend seconds of your month; the error-budget policy governs the rest. It is a short, written, pre-agreed document — signed by engineering and product before the first bad month, because a policy negotiated during a freeze is not a policy, it is a hostage situation. Its core is a trigger table:

Beyond the table, the document needs only: covered SLOs, a named arbiter, the exception process (who can override a freeze and what they must write down), maintenance accounting, and review cadence. Google’s SRE books also institutionalized the counterintuitive half: a budget should be spent — their cautionary tale is a service so reliable consumers assumed it never failed, until it did. Teams chronically under budget take planned risk (faster rollouts, failure injection) so nobody calibrates to accidental over-delivery.

The enforcement that actually changes behavior is the release gate: CI queries budget remaining before a prod deploy and fails closed. With the 30d recording rule in place (job:slo_errors:ratio_rate30d, evaluated every 5m), the gate is a dozen lines:

#!/usr/bin/env bash
# deploy-gate.sh — block prod deploys when error budget remaining < 20%
set -euo pipefail
PROM="${PROM:-http://prometheus.monitoring.svc:9090}"
SVC="${1:?usage: deploy-gate.sh <job>}"

REMAINING=$(curl -sfG "$PROM/api/v1/query" \
  --data-urlencode "query=1 - (job:slo_errors:ratio_rate30d{job=\"${SVC}\",slo=\"http-availability\"} / 0.001)" \
  | jq -r '.data.result[0].value[1] // empty')

if [[ -z "$REMAINING" ]]; then
  echo "GATE: no SLO data for ${SVC} — failing closed"; exit 1
fi
if (( $(echo "$REMAINING < 0.20" | bc -l) )); then
  echo "GATE: budget remaining ${REMAINING} < 0.20 — blocked; see policy for exceptions"
  exit 1
fi
echo "GATE: budget remaining ${REMAINING} — proceed"

Three design choices matter: it fails closed on missing data (an SLO you cannot compute is not a green light), the 20% threshold comes from the policy table rather than the script author’s mood, and the failure message points at the exception process instead of daring engineers to bypass it. Make it a required prod-pipeline step and the budget stops being advisory.

Dashboards and reporting

The ladder answers “is it burning now?”; the dashboard answers “how much is left and where did it go?” — the question sprint planning and the weekly review actually ask. One dashboard per SLO, six panels, no vanity graphs:

The budget-remaining number deserves care: there are two ways to compute it, and they disagree exactly when it matters. The ratio of sums is the correct, traffic-weighted spend:

# Budget remaining, request-weighted (correct): 1 = untouched, 0 = exhausted, negative = overspent
1 - (
  (
    sum by (job) (increase(http_requests_total{code=~"5.."}[30d]))
    /
    sum by (job) (increase(http_requests_total[30d]))
  ) / 0.001
)

The tempting shortcut — avg_over_time(job:slo_errors:ratio_rate5m[30d]) — is an average of ratios: every 5-minute slice weighs the same, so a 3am blip on 10 rps counts as much as a peak-hour incident on 2,000 rps. Use it only as a cheap approximation, never for compliance reporting. The honest query has an infrastructure cost instead: increase(...[30d]) needs 30+ days of counter history, and Prometheus defaults to 15 days of retention (--storage.tsdb.retention.time). Raise retention to ~45d, or push the SLI series to long-term storage — Thanos in Production: Global Query View, Deduplication, and Object-Storage Downsampling and Running Grafana Mimir: Multi-Tenant, Horizontally Scalable Prometheus Storage are the standard answers. Build the dashboard as code with slo as a template variable so one dashboard serves every service — Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code has the provisioning patterns.

Architecture at a glance

Picture the system as a pipeline with a feedback loop, flowing left to right. At the far left sits the SLI source of record — ingress/load-balancer counters (http_requests_total by code, duration histograms), chosen at the edge so a dead backend still produces countable 5xx rather than silence, with health-check and synthetic traffic excluded by label. Prometheus scrapes those counters every 15 seconds, and a recording-rule fan-out turns two raw counters into eight pre-computed error-ratio series — the 5m, 30m, 1h, 2h, 6h, 1d, and 3d windows the ladder consumes, plus a 30d series for budget accounting — each carrying the job, team, and slo labels every downstream decision routes on.

The middle stage is the four-rung MWMBR ladder: four alert rules, each and-ing a long and short window against the same burn × budget literal — 14.4x over 1h+5m and 6x over 6h+30m marked severity: page, 3x over 1d+2h and 1x over 3d+6h marked severity: ticket — with absent() guards and minimum-traffic gates watching the silent-failure modes. Alertmanager groups whatever fires by {slo, job} so one incident is one notification thread, routes pages to PagerDuty and tickets to the backlog, and lets pages inhibit tickets for the same SLO. The right-hand edge is the human loop: on-call answers pages via the runbook link in the annotation, the budget dashboard (SLI, budget remaining, three burn-rate stats, burn-down vs ideal) feeds the weekly review, and the error-budget policy closes the loop into planning — release gates query the 30d budget before deploys, freezes trigger at exhaustion, and the target is re-ratified quarterly from the same data. Nothing here is bespoke: two counters in, four alerts and one governance number out.

Real-world scenario

PayStream, a payments platform out of Chennai, runs 42 microservices across two regions. Its authorize API averages 1,100 rps (about 2.85 billion requests per rolling 30 days) and carries the revenue. Before the migration, alerting was 118 static rules accreted over four years; on-call averaged 31 pages a week, roughly 9% actionable. Two incidents in one quarter — a missed slow burn that torched a month’s reliability, and a 40-minute severity argument during a real outage — earned the team a mandate to rebuild on SLOs.

They ratified 14 SLOs in six weeks, starting with authorize: 99.95% availability over rolling 30 days (β = 0.0005; budget = 1.425M bad requests, or 21.6 full-outage minutes), plus a 99% ≤ 1s latency SLO, each with the standard four-rung ladder (rung-1 threshold 14.4 × 0.0005 = 0.72%). Week two delivered the classic bruise: a checkout burn paged the platform rotation because recording rules aggregated with sum by (job) and dropped the team label the Alertmanager routes matched on. The fix — sum by (job, team) everywhere, team in every equal: list, and a promtool test asserting the label survives onto the fired alert — turned a silent label drop into a CI failure forever after.

Then the ladder earned its keep twice in one month. Incident A, fast burn: at 09:12 a canary deploy (10% of fleet) shipped a serialization bug failing 70% of the canary’s requests — 7% overall error ratio, burn 140. The 1h window crossed 0.72% at 09:18 (t = 3600 × 0.0072/0.07 ≈ 6.2 min), for: 2m plus pipeline latency delivered the page at 09:21, rollback landed at 09:31. Total spend: 19 minutes at 7% ≈ 88,000 bad requests — 6.2% of the monthly budget, only ~2.9% of it gone when the page arrived, almost exactly what the math promises. Incident B, the one the museum would have missed: a downstream’s TLS certificate rotation churned connection pools, producing a steady 0.5% error ratio — burn 10, invisible to the old “ratio > 1% for 5m” rule. Rung 2 (6x over 6h+30m) paged after ~2h10m of sustained burn; the pool settings were fixed that afternoon with 5% of the budget spent instead of a third by week’s end.

A quarter in, the numbers that convinced the skeptics: pages fell from 31 to 4 per week with actionability up from 9% to ~78%; the budget dashboard entered sprint planning; and the one freeze the policy triggered (budget at −8% after a botched database migration) postponed a risky cutover by two weeks on a pre-agreed rule instead of a meeting war. The lesson on PayStream’s wall: the ladder is arithmetic, but the labels are architecture — derive the numbers once, and spend review time on what survives aggregation.

Advantages and disadvantages

The honest summary: MWMBR is the best-known general solution for paging on reliability promises, and deliberately nothing else. It does not replace debug dashboards, direct alarms for irreversible failures (disk full, cert expiry), or the tracing you need once the page arrives — Wiring OpenTelemetry Metrics and Exemplars for Click-Through Trace Correlation is the natural next hop from a burning SLI panel.

Hands-on lab

Everything runs locally with Docker inside an hour; cloud cost ₹0. You will unit-test the ladder, then watch it fire and resolve against live synthetic traffic.

1. Layout. Create the workspace and save the three YAML files from the implementation section — slo-recording-rules.yaml, slo-burn-alerts.yaml, slo-tests.yaml — into rules/:

mkdir -p slo-lab/rules && cd slo-lab
# save the recording rules, alert rules, and test file from this article into rules/

2. Static validation and unit tests (no server needed; this alone is CI-grade):

docker run --rm -v "$PWD/rules:/work" --entrypoint promtool \
  prom/prometheus:v2.53.0 check rules /work/slo-recording-rules.yaml /work/slo-burn-alerts.yaml
docker run --rm -v "$PWD/rules:/work" --entrypoint promtool \
  prom/prometheus:v2.53.0 test rules /work/slo-tests.yaml

Expected output ends with SUCCESS for both. If the test run fails, read the printed diff of expected vs actual alerts — exactly how you will debug threshold changes for the rest of your career.

3. The synthetic service. Save as exporter.py — http_requests_total growing at ~50 req/s with a tunable error rate:

#!/usr/bin/env python3
"""Synthetic traffic: /metrics for Prometheus, /set?rate=0.2 to change error rate."""
import http.server, threading, time, urllib.parse

RPS = 50
error_rate = 0.0005          # healthy: 0.05% errors = 0.5x burn at 99.9%
counters = {"200": 0.0, "500": 0.0}

def tick():
    while True:
        counters["500"] += RPS * error_rate
        counters["200"] += RPS * (1 - error_rate)
        time.sleep(1)

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        global error_rate
        if self.path.startswith("/set"):
            q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query)
            error_rate = float(q["rate"][0])
            body = f"error_rate={error_rate}\n"
        else:
            body = "".join(f'http_requests_total{{code="{c}"}} {v:.0f}\n'
                           for c, v in counters.items())
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(body.encode())
    def log_message(self, *args):
        pass

threading.Thread(target=tick, daemon=True).start()
http.server.HTTPServer(("0.0.0.0", 9100), Handler).serve_forever()

4. Prometheus config. Save as prometheus.yml — the scrape config supplies the job="checkout" label:

global:
  scrape_interval: 15s
  evaluation_interval: 30s
rule_files:
  - /etc/prometheus/rules/slo-recording-rules.yaml
  - /etc/prometheus/rules/slo-burn-alerts.yaml
scrape_configs:
  - job_name: checkout
    static_configs:
      - targets: ["exporter:9100"]

5. Compose and start. Save as docker-compose.yml, then docker compose up -d:

services:
  exporter:
    image: python:3.12-alpine
    command: ["python", "/lab/exporter.py"]
    volumes: ["./exporter.py:/lab/exporter.py:ro"]
    ports: ["9100:9100"]
  prometheus:
    image: prom/prometheus:v2.53.0
    command: ["--config.file=/etc/prometheus/prometheus.yml"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules:/etc/prometheus/rules:ro
    ports: ["9090:9090"]

6. Verify the healthy baseline (~2 minutes for windows to populate):

curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=job:slo_errors:ratio_rate5m / 0.001' | jq -r '.data.result[0].value[1]'
# Expected: ~0.5  (burn rate 0.5x — healthy, under-budget)
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts | length'
# Expected: 0

7. Ignite a fast burn. Push the error rate to 20% (burn 200x at 99.9%):

curl 'localhost:9100/set?rate=0.2'
watch -n 15 "curl -s localhost:9090/api/v1/alerts | jq '[.data.alerts[] | {alert: .labels.alertname, state: .state}]'"

8. Recover and watch the reset property. Restore health; the page must resolve within minutes — the whole point of the short window:

curl 'localhost:9100/set?rate=0.0005'
# In ~5-7 min the 5m window falls below 1.44% and ErrorBudgetFastBurn resolves,
# though the 1h window is still contaminated. Single-window designs stay red ~1h here.

9. Teardown.

docker compose down && cd .. && rm -rf slo-lab

You have validated the ladder three ways — statically (check rules), deterministically (test rules), and live (injection and recovery) — the exact promotion path SLO rules should follow in a real pipeline.

Common mistakes & troubleshooting

The playbook, in the order the symptoms usually present:

Rows 3, 5, and 6 account for most real-world MWMBR grief — the blind total outage, the dropped routing label, the small denominator — and all three are testable in CI, which is why the promtool suite and label assertions are non-negotiable.

Best practices

• Measure the SLO of record at the edge; keep white-box SLIs for diagnosis. A dead backend must become countable 5xx, not silence.
• Write the validity table (4xx, 429, health checks, synthetics, retries) into the SLO doc and encode it in selectors at instrumentation time.
• Set targets from measured baselines, ratchet deliberately, re-ratify quarterly. An SLO violated on day one is theater; a budget never spent taxes velocity.
• Deploy all four rungs — the ticket rungs catch the expensive failures. Page rungs alone is half a system.
• Long window on the left of and, for: short (the short window already gates flakes), keep_firing_for small but non-zero on pages.
• Pre-compute every window with recording rules; alert expressions must be O(1) lookups, never 3d range queries over raw counters.
• Carry every routing dimension (team, env, tenant) through every by() and equal: list; assert label propagation in promtool tests.
• Guard the blind spots: absent() per critical service, min-traffic gates on page rungs, synthetic probe floor for low traffic and off-hours.
• Unit-test the ladder in CI: one burn scenario per rung, one healthy, one recovery, one label assertion — promtool check + test on every PR.
• Group and inhibit in Alertmanager: group_by: ['slo','job'], pages suppress tickets on matching equal: — one incident, one thread.
• Put budget remaining in sprint planning and wire the release gate to it; alerts without the policy loop revert to threshold culture within two quarters.
• Generate at scale (Sloth/Pyrra) beyond a handful of SLOs — but only after you can derive the numbers yourself, or you cannot audit the generator.

Security notes

The SLO pipeline is control-plane infrastructure: it decides who gets woken and whether deploys proceed, which makes it worth attacking and worth hardening.

One non-obvious point: the absence of monitoring data is a security signal too. The absent() guards deployed for reliability double as tamper detection — an SLI that stops reporting deserves the same page whether the cause is a crash or someone quietly disabling a scrape.

Cost & sizing

SLO alerting itself is nearly free; the bill hides in retention and cardinality. Cost drivers for a self-hosted stack (INR, pay-as-you-go; reserved/spot cuts 30–60%):

The ladder adds well under 5% to an existing Prometheus footprint; if adopting SLOs blows up the bill, the problem is source-metric cardinality — fix it with Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus. The negative cost matters too: PayStream-style page reduction (31 → 4/week) is an on-call staffing line item that dwarfs the infrastructure delta.

Interview & exam questions

Q1. Define burn rate and explain why alerting on it beats alerting on the raw error ratio. Burn rate is the observed error ratio over the budget fraction: B = E / (1 − SLO) — budget-consumption speed as a multiple of the exactly-sustainable pace. Alerting on it makes severity proportional to time-to-exhaustion (T/B): outages and slow leaks alert when they threaten the promise, not when they cross an arbitrary instantaneous level.

Q2. Derive 14.4. Design statement: “page when 2% of a 30-day budget is consumed in one hour.” Burn = fraction × SLO window ÷ alert window = 0.02 × 720h / 1h = 14.4; equivalently, exhaustion in 720/14.4 = 50 hours.

Q3. What does the short window add, and why is it one-twelfth of the long window? The and-ed short window requires the burn to be happening right now, bounding reset time: after recovery the alert resolves within the short window, not the long one (5 minutes vs ~1 hour for rung 1). W/12 is the workbook’s compromise — short enough for fast reset, long enough not to gate on a scrape artifact. Detection is unaffected: the short window crosses before the long one during a real burn.

Q4. How much budget is consumed when each rung fires, and why is it severity-independent? At the long-window crossing, E × t = W × B × β; dividing by the total budget β × T gives W × B / T — E cancels. Rung 1: 2%, rung 2: 5%, rungs 3–4: 10% (plus for: and pipeline slop). Worse outages are detected proportionally sooner, so spend-at-detection is constant.

Q5. Rungs 3 and 4 have the same W × B product (72h). Why keep both? Same product means identical detection times where both fire and the same 10% spend-at-detection — but different thresholds: rung 3 (3x) tickets 3–6x burns within a day; rung 4 (1x) catches the 1–3x burns rung 3 never sees, down to exactly on-pace.

Q6. Your service does 5 requests/minute. What breaks, and what do you do? The short windows contain a handful of events, so one failure exceeds page thresholds. Fixes in order: min-traffic gates on page rungs, error-count floors, longer windows, synthetic traffic, composite SLOs, looser target/longer window, or ticket-only ladder — always backstopped by a probe-based SLI and absent() alerts.

Q7. Why is error ratio > threshold for: 1h not equivalent to a 1h-window burn alert? for: requires the condition true at every evaluation — one good scrape resets the pending timer, so an oscillating 2x burn never fires. A 1h rate() integrates: the same oscillation averages above threshold and alerts. for: also makes detection severity-independent, inverting the entire point.

Q8. What is the maximum burn rate, and when does the canonical ladder become invalid? 1/β, since the error ratio cannot exceed 1. At 90% (β = 0.1), 14.4x implies a 144% error ratio — impossible; re-derive with multipliers below 1/β (e.g., fast page at 6x = 60%).

Q9. Ratio-of-sums vs average-of-ratios for 30-day budget reporting? Ratio of sums (increase(errors[30d]) / increase(total[30d])) weights by traffic and matches the request-based budget — correct for compliance, but needs 30d+ retention or long-term storage. Averaging the 5m ratio over 30d weights all time slices equally, over-counting quiet hours — a cheap approximation only.

Q10. Why must burn alerts be paired with absent() and edge measurement? A hard-down service stops producing the metrics the SLI divides — 0/0 is NaN, missing scrapes are empty vectors, comparisons against both are false, so the worst outage class can never fire the burn alert. Edge measurement converts dead backends into countable 5xx; absent() pages when the SLI itself disappears.

Q11. What belongs in an error-budget policy? A trigger table (budget thresholds → mandatory actions → owners → exit criteria), scope, a named arbiter, the exception process, maintenance accounting, and review cadence — signed by engineering and product before it is needed. Enforcement: release gates on budget remaining, freezes at exhaustion, escalation for chronic exhaustion.

Q12. How do you prove the ladder correct before production? promtool test rules with synthetic series: assert each rung fires at its designed burn, healthy input fires nothing, recovery resolves pages within the short window, and routing labels propagate to fired alerts. Plus promtool check rules and amtool config routes test in CI.

Quick check

1. A 99.95% SLO over 30 days — how many minutes of full outage is the budget, and what burn rate exhausts it in exactly 5 days?
2. Which rung catches a sustained 0.4% error ratio on a 99.9% SLO, and roughly when?
3. Why does the fast page put job:slo_errors:ratio_rate1h on the left of and?
4. Your budget dashboard uses avg_over_time(ratio_rate5m[30d]). Name the bias.
5. A single failed request pages your 1-req/min service. Which two guards fix it without deleting the ladder?

Answers

1. 0.0005 × 43,200 = 21.6 minutes; exhaustion in 5 days needs B = 30/5 = 6 — exactly why rung 2’s threshold is 6x.
2. 0.4% is burn 4 — above rung 3’s 3x (0.3%) but below rung 2’s 6x: rung 3 tickets at t = 24h × 0.003/0.004 = 18h (rung 4 ties at 18h, but rung 3’s for: is shorter).
3. With set operators the result (and $value) comes from the left-hand side — you want the sustained long-window ratio reported, not the twitchy short one.
4. Average-of-ratios weights all time slices equally regardless of traffic; the request-weighted truth is the ratio of 30d increase() sums.
5. A minimum-traffic gate (and sum(rate(total[1h])) > X) on the page rungs, plus a synthetic-probe floor and absent() coverage backstopping the gate’s silence.

Glossary

• SLI (Service Level Indicator): the measured ratio of good to valid events for a user journey; the raw material of everything else.
• SLO (Service Level Objective): the internal target for an SLI over a window, e.g. 99.9% over rolling 30 days.
• SLA (Service Level Agreement): the external, contractual, penalty-bearing promise — always looser than the SLO protecting it.
• Error budget: 1 − SLO; allowed badness per window, expressible as a ratio, minutes, or requests.
• Burn rate: error ratio ÷ budget fraction; budget-spend speed as a multiple of the exactly-sustainable pace.
• MWMBR: multi-window multi-burn-rate alerting — several burn thresholds, each requiring a long and short window to agree.
• Long window: the lookback proving a burn is sustained and significant (1h/6h/1d/3d canonically).
• Short window: the lookback proving the burn is still active; long/12; bounds reset time.
• Detection time: failure start → alert firing; W × B × β / E for sustained error ratio E.
• Reset time: recovery → alert resolving; bounded by the short window in MWMBR designs.
• Precision / recall: fired alerts that were real threats / real threats that fired — the two axes every design trades.
• Recording rule: a Prometheus rule pre-computing an expression into a new series; here, the error ratio per window.
• keep_firing_for: rule field (Prometheus ≥ 2.42) holding an alert firing briefly after its condition clears, against flapping.
• Error-budget policy: the pre-agreed trigger table of consequences (reviews, gates, freezes) as budget depletes.
• Release gate: a CI/CD step querying budget remaining, blocking deploys below a policy threshold, failing closed on missing data.

Next steps

• Automate the derivation: SLOs as Code: Authoring SLIs with OpenSLO and Generating Burn-Rate Alerts via Sloth and Pyrra generates these exact rules from declarative specs.
• Design the notification layer: Designing Alertmanager Routing Trees: Grouping, Inhibition, Silences, and Dedup.
• Make long windows and 30d budget queries cheap and durable: Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir.
• Build the dashboards the review runs on: Engineering Grafana Dashboards That Get Used: RED, USE, Template Variables, and Provisioning-as-Code.
• Close the human loop after the page: Building an On-Call Practice: PagerDuty Escalation, Alert Routing, and Actionable Runbooks.