Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir

A single Prometheus is one of the best engineering deals in infrastructure: scrape, store, alert, and query from one static binary with no external dependencies. That deal expires the moment your active series count, retention requirement, or dashboard concurrency outgrows one machine’s RAM and disk — and at that point most teams reach for a horizontally scalable backend without ever understanding the machinery between their Prometheus and that backend. The result is predictable: remote-write queues that silently fall hours behind, recording rules that cost more than they save, WALs that eat the disk during a receiver outage, and a “long-term storage” bill that nobody can explain.

This article is the missing operations manual for that middle layer. You will go deep on recording rules — the level:metric:operations naming discipline, rule-group evaluation mechanics, dependent rules, and the honest cost/benefit arithmetic of each rule — and deeper still on remote_write: the WAL-tailing mechanism underneath it, every queue_config parameter with its default and failure mode, retry and backoff semantics, write_relabel_configs filtering, metadata/exemplar/native-histogram forwarding, and what the Remote-Write 2.0 protocol changes on the wire. Then the receiving side: Thanos Receive, Grafana Mimir, VictoriaMetrics, and Cortex compared as remote-write backends, with retention and downsampling strategy per backend, HA-pair deduplication, capacity planning with worked numbers, and the alert rules that watch the shipping pipeline itself.

The framing assumption throughout: you already run Prometheus seriously and it hurts. Every section is written against real configuration keys, real metric names, and real failure modes — the things you only otherwise learn at 02:00 when prometheus_remote_storage_samples_pending is a straight line up and the on-call dashboard is thirty minutes stale.

What problem this solves

Prometheus is deliberately a vertically scaled, single-node system. It keeps an in-memory index of every active series (unique label-set), holds roughly the last two hours of samples in the head block, and persists immutable 2-hour blocks to local disk. There is no clustering, no replication, and no built-in horizontal query path. That design is why it is so reliable — and why it hits three hard walls:

Without this layer done right, teams suffer in specific, recurring ways. The on-call engineer restarts a 200 GB-WAL Prometheus and waits 25 minutes for replay while flying blind. A deploy adds one label and doubles series; the remote-write queue can no longer drain and the WAL grows until the disk fills — taking scraping down with it, which is how a storage problem becomes a visibility outage. Somebody sets max_shards: 200 after reading a blog post, and the receiving Mimir cluster’s distributors saturate, returning 429s to every tenant, converting one team’s mistake into everyone’s incident. A year later, finance asks why object storage costs ₹4 lakh a month and nobody can say which tenant, which metric, or which retention tier is responsible.

Who hits this: any platform team past roughly 1–2 million active series or more than two or three Kubernetes clusters, anyone with retention requirements past 30 days, and every team standing up a central observability platform for multiple product teams. The knowledge here is the difference between a metrics platform with a predictable cost curve and an unbounded science experiment.

Learning objectives

By the end of this article you can:

• Name and apply the level:metric:operations recording-rule convention, structure rule groups so dependent rules evaluate in the right order, and compute the cost/benefit of any individual rule in series and CPU terms.
• Explain the remote-write pipeline end to end — appender → WAL → WAL watcher → queue manager → shards → HTTP — including exactly when samples are dropped versus retried, and how long a receiver outage is survivable.
• Tune every queue_config parameter (capacity, min_shards/max_shards, max_samples_per_send, batch_send_deadline, min_backoff/max_backoff, retry_on_http_429, sample_age_limit) from measured ingest rate and round-trip latency, not folklore.
• Filter the remote stream with write_relabel_configs, forward exemplars and native histograms deliberately, and state what Remote-Write 2.0 changes (string interning, inline metadata, created timestamps, stats headers).
• Choose between Thanos Receive, Mimir, VictoriaMetrics, and Cortex as a remote-write receiver using an operational comparison — tenancy, dedup mechanism, downsampling, storage medium, license — rather than feature folklore.
• Decide between agent mode, full server + remote-write, and federation for each cluster topology, and defend the decision.
• Do the capacity math: active series → samples/s → bytes/s → GB/month → INR/month, plus receiver component sizing, with worked numbers you can adapt.
• Run HA Prometheus pairs into each backend without duplicate series, and monitor the pipeline itself with prometheus_remote_storage_* metrics and production-grade alert rules.

Prerequisites & where this fits

You should be fluent in PromQL (rate, sum by, histogram_quantile) — PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work is the companion — and comfortable with Prometheus configuration: scrape_configs, relabeling, external_labels. You need a working mental model of series cardinality; Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus covers the source-side controls this article assumes. Kubernetes familiarity helps for the receiver deployments but nothing here requires it — everything in the lab runs in plain Docker.

This article sits at the centre of the observability track and hands off to three sibling deep-dives on the receiving side:

Core concepts

Six mental models carry every later section.

1. Samples flow through the WAL, and remote-write is a WAL tail. Every scraped sample is appended to the in-memory head and to the write-ahead log (WAL) — 128 MB segment files under wal/. Since Prometheus 2.8, remote-write does not intercept scrapes; a WAL watcher tails those segment files from disk and feeds a per-endpoint queue. This is the single most important fact in this article, because it explains everything else: why a receiver outage does not immediately lose data (the WAL is the buffer), why the buffer is finite (the WAL is truncated roughly every two hours), why remote-write survives a Prometheus restart (it checkpoints its read position), and why a fallen-behind queue cannot recover data older than the last truncation (remote-write never reads from compacted blocks — WAL only).

2. An active series is the unit of memory; a sample is the unit of throughput. Memory on the sender and the receiver scales with active series (unique label-sets currently receiving samples). Network, CPU, and storage scale with samples per second, which is simply active_series / scrape_interval. Keep the two axes separate in your head: relabeling attacks series; scrape interval and filtering attack samples.

3. Recording rules move cost from query time to write time. A rule evaluates an expression every interval and appends the result as a new series — permanently, into the TSDB and the remote stream. You pay evaluation CPU plus stored-series cost forever; you save every dashboard refresh and alert evaluation that would have recomputed it. The trade is only profitable when the rule aggregates labels away and the output is read often. A rule that keeps all input labels is pure overhead.

4. The receiver owns durability, tenancy, and the long read path. Once samples cross the remote-write HTTP boundary, everything — replication, object storage, retention, compaction, deduplication, per-tenant limits, long-range query — belongs to the backend. Your Prometheus becomes a shipping agent with a short local window (or, in agent mode, no queryable window at all).

5. Backpressure travels backwards. Receiver slow → shards saturate → queue backs up → WAL watcher falls behind → WAL grows on disk → disk fills → scraping and rule evaluation stop. Every incident in this pipeline is a variation of that chain, and every monitoring section later maps a metric to a link in it.

6. HA is two independent scrapers, deduplicated at the read or ingest side. Prometheus HA is not clustering — it is two identical servers scraping the same targets, distinguished by an external label, with the duplicate stream collapsed by the receiver (Mimir/Cortex HA tracker at ingest; Thanos replica-label dedup at query; VictoriaMetrics dedup at storage). The pair doubles sender-side cost; whether it doubles receiver-side cost depends on the backend, and that difference shows up directly in the bill.

The moving parts, in one table:

One more distinction that saves arguments in design reviews — what stays local versus what ships:

Recording rules: precompute what dashboards read

A recording rule is a contract: “this expression is important enough to evaluate every interval and store forever.” Treat each one as a small piece of infrastructure with a name, an owner, a cost, and a test.

The level:metric:operations naming convention

The community convention — used by the kubernetes-mixin, node-exporter mixin, and every serious rule corpus — encodes what a recorded series is into its name:

level : metric : operations

Worked examples, from cheap to layered:

The convention is not cosmetic. It makes collisions with scraped metrics impossible (scraped names never contain :), makes the aggregation level machine-parsable, and lets a reviewer spot a broken rule from the name alone — a job:... rule whose expression forgets sum by (job) is visibly wrong. Enforce it in CI with a regex gate: every record: must match ^[a-z_]+:[a-z0-9_]+:[a-z0-9_]+$.

Rule groups: intervals, ordering, and the fields that matter

Rules live in files referenced by rule_files: in prometheus.yml, organized into groups. The group is the unit of scheduling, ordering, and failure isolation:

Evaluation mechanics you must know cold:

• Rules within a group run sequentially, sharing one evaluation timestamp. Rule N can reference the output of rule N-1 from the same tick — the earlier result is committed before the later rule runs. This is what makes dependent rule chains deterministic.
• Groups run concurrently and independently. A rule that reads another group’s output sees that group’s previous evaluation — up to one interval stale. Never split a chain (e.g., a rate and the ratio built on it) across groups unless a one-interval skew is acceptable.
• A slow group self-destructs quietly. If evaluation takes longer than interval, the next tick is skipped and prometheus_rule_group_iterations_missed_total increments. Nothing crashes; your recorded series just silently gain gaps, which then break the rate() calls that read them.
• Groups are the unit of parallelism. Fifty rules in one group evaluate serially; five groups of ten evaluate five-wide. Split hot groups both for latency and for blast-radius isolation — one failing rule marks the group’s evaluation failed, not the file’s.

A production-shaped rule file, with a dependent chain done correctly:

# rules/http-aggregations.yml
groups:
  - name: http_requests_aggregation
    interval: 30s
    limit: 5000
    rules:
      # Layer 1: collapse instance/pod. This is the expensive evaluation.
      - record: job_method_code:http_requests:rate5m
        expr: sum by (job, method, code) (rate(http_requests_total[5m]))

      # Layer 2: reads Layer 1 FROM THIS SAME TICK - same group, later rule.
      - record: job:http_requests:rate5m
        expr: sum by (job) (job_method_code:http_requests:rate5m)

      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (job_method_code:http_requests:rate5m{code=~"5.."})
            /
          sum by (job) (job_method_code:http_requests:rate5m)

  - name: http_latency_aggregation
    interval: 30s
    rules:
      # Keep le - histogram_quantile needs it. Drop everything else.
      - record: job_le:http_request_duration_seconds_bucket:rate5m
        expr: sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))

      - record: job:http_request_duration_seconds:p99_rate5m
        expr: histogram_quantile(0.99, job_le:http_request_duration_seconds_bucket:rate5m)

Layering matters for cost: layer 1 touches the raw high-cardinality data once; every further rollup reads the already-small recorded series. Ten dashboards and six alerts then read layer 2 for near-zero cost.

The cost/benefit ledger for a single rule

Every rule has four costs and one benefit. Make the arithmetic explicit before merging:

The heuristics that fall out of the ledger:

One trap deserves its own paragraph: recording quantiles is lossy. histogram_quantile(0.99, ...) per job is fine to record for that job-level view only — you can never re-aggregate recorded p99s across jobs (the average of p99s is not the p99). Record the bucket rates at each level you need, and compute quantiles at read time. The same logic applies to recorded avg — record sum and count separately so downstream rollups stay correct.

Testing rules with promtool

Rules are code; test them like code. promtool ships in the Prometheus tarball and container image and covers syntax, semantics, and behaviour:

A behaviour test uses the expanding notation value+incrementxCount to synthesize counters, then asserts on the recorded output:

# tests/http-aggregations.test.yml
rule_files:
  - ../rules/http-aggregations.yml

evaluation_interval: 30s

tests:
  - interval: 30s
    input_series:
      # A counter increasing 60/min per pod: rate = 2/s each.
      - series: 'http_requests_total{job="checkout", method="GET", code="200", pod="a"}'
        values: '0+60x20'
      - series: 'http_requests_total{job="checkout", method="GET", code="500", pod="a"}'
        values: '0+6x20'
    promql_expr_test:
      - expr: job:http_requests:rate5m
        eval_time: 5m
        exp_samples:
          - labels: 'job:http_requests:rate5m{job="checkout"}'
            value: 2.2   # (60+6)/30 = 2.2 samples/sec
      - expr: job:http_request_errors:ratio_rate5m
        eval_time: 5m
        exp_samples:
          - labels: 'job:http_request_errors:ratio_rate5m{job="checkout"}'
            value: 0.09090909090909091   # 6/66

$ promtool test rules tests/http-aggregations.test.yml
Unit Testing:  tests/http-aggregations.test.yml
  SUCCESS

The discipline that makes this pay: assert exact float values (they are deterministic given synthetic inputs), test the dependent chain end to end (the ratio rule above implicitly tests its layer-1 input), and add a regression test whenever a rule change breaks a dashboard — the test file becomes the executable history of every subtle rate window and label-matching bug you have shipped.

Rule anti-patterns and their fixes

Inside remote_write: WAL, watcher, shards

Everything about remote-write tuning becomes obvious once you can narrate the pipeline. Here is the path of one sample, in order, with the failure mode at each hop:

Four consequences of this design that you must internalize:

Ordering is per-shard and strict. A shard never sends batch N+1 until batch N succeeds or is dropped. That is why receivers can enforce out-of-order limits, and why one slow batch head-of-line-blocks everything hashed to that shard. Retries are not reordering — they are the same batch again.

The buffer is the WAL, and it is time-boxed. In-memory shard buffers are tiny (seconds); the real outage tolerance is how far back the WAL reaches — roughly the last two to three hours under default head/truncation behaviour. A receiver outage shorter than that is fully recovered, in order, on reconnect. Longer than that, the watcher’s position is truncated away and the gap is unrecoverable from the sender: remote-write never reads from persisted blocks. Plan receiver maintenance windows against this number and size the WAL volume for growth during outages (ingest bytes/s × outage seconds, plus normal WAL).

Restarts resume, they do not re-send history. The watcher persists progress; on restart it resumes from the last checkpointed position. But a config change to write_relabel_configs or a new remote endpoint starts from the current WAL position — a brand-new endpoint does not receive the WAL’s past two hours; it starts from approximately now.

Sample loss is explicit and enumerable. Exactly four ways a sample legitimately fails to arrive: (a) filtered by write_relabel_configs (intentional, counted in samples_dropped_total); (b) receiver returned a non-retryable 4xx (counted in samples_failed_total); © sample_age_limit exceeded during a long backlog (dropped as too old); (d) WAL truncated past the watcher during an extended outage. Everything else is retried forever within the WAL window.

The remote_write block, field by field

Every top-level field, its default, and when you touch it:

Tuning queue_config from first principles

queue_config is where most remote-write incidents are created and most are fixed. Every parameter, with the current defaults:

The retry semantics behind those knobs — what the sender does per HTTP response:

Note the asymmetry: 4xx loses data by design (re-sending would fail identically), 5xx never loses data within the WAL window. When a receiver misclassifies overload as 400 instead of 429/503 — or an ingress rewrites 503 into 404 — the sender throws samples away. Verify with curl -v what status codes your actual edge returns under pressure.

Sizing shards: the arithmetic

A single shard’s throughput is bounded by round-trip time:

per-shard throughput ≈ max_samples_per_send / round_trip_time
required shards      ≈ ingest samples/s ÷ per-shard throughput  (×1.5 headroom)

Worked table at max_samples_per_send: 2000:

Prometheus recomputes desired shards every ~10 s from observed in-rate, out-rate, and send latency, and reshards within [min_shards, max_shards]. The failure signature to memorize: prometheus_remote_storage_shards_desired pinned above prometheus_remote_storage_shards_max while samples_pending climbs — genuinely under-sharded. The counterfeit version: shards_desired explodes because latency exploded (receiver dying); more shards would just be more clients hammering a drowning backend. Always read sent_batch_duration_seconds before touching max_shards.

Sender-side memory cost of the queue is real but modest: buffered samples cost roughly 100–200 bytes each (labels are interned references). At max_shards: 50 × capacity: 10000 that is ~0.5M samples ≈ 50–100 MB — but the WAL-watcher series cache adds memory proportional to active series per remote endpoint, which is why five remote_write endpoints on a 4M-series Prometheus is a memory problem independent of shard counts.

A production-tuned block for a high-volume, same-region Mimir endpoint:

remote_write:
  - name: central-mimir
    url: https://mimir.internal.example.com/api/v1/push
    remote_timeout: 30s
    headers:
      X-Scope-OrgID: team-platform
    queue_config:
      min_shards: 4              # skip ramp-up after restarts
      max_shards: 100            # sized: 300k samples/s ÷ (2000/0.1s) = 15; ceiling 100 for incident headroom
      capacity: 10000
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 30s           # give a recovering receiver room to breathe
      retry_on_http_429: true    # Mimir rate limits MUST be retried, not dropped
      sample_age_limit: 0s       # full catch-up after outages; set 15m if freshness > completeness

Symptom → knob decision table

Filtering, metadata, exemplars, and the 2.0 protocol

write_relabel_configs: the per-endpoint cost lever

write_relabel_configs runs relabeling on every sample after the WAL watcher and before queueing — the same engine as scrape-time relabeling, applied to the outbound stream of one endpoint. It sees series labels (including __name__) but not scrape metadata. All actions work; these are the ones that matter here:

Three worked patterns:

remote_write:
  # Pattern 1 - allow-list: ship ONLY recorded aggregates + SLO inputs.
  # The cheapest long-term-storage bill is the series you never send.
  - name: lts-aggregates-only
    url: https://mimir.internal.example.com/api/v1/push
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "([a-z_]+:[a-z0-9_]+:[a-z0-9_]+|up|probe_success)"   # ':' = recorded rules only
        action: keep

  # Pattern 2 - deny-list: full stream minus known-noisy families and churn labels.
  - name: lts-full-minus-noise
    url: https://receive.internal.example.com/api/v1/receive
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_(gc|memstats)_.*|apiserver_request_slo_duration_seconds_bucket"
        action: drop
      - regex: "pod_template_hash|controller_revision_hash"
        action: labeldrop

  # Pattern 3 - hashmod sharding: split one giant stream across two receivers.
  - name: shard-a
    url: https://receive-a.internal/api/v1/receive
    write_relabel_configs:
      - source_labels: [__name__]
        modulus: 2
        target_label: __tmp_shard
        action: hashmod
      - source_labels: [__tmp_shard]
        regex: "0"
        action: keep
      - regex: "__tmp_shard"
        action: labeldrop

Two traps. First, labelkeep/aggressive labeldrop can collapse two distinct series into one label-set — the receiver then sees interleaved duplicate/out-of-order samples for “the same” series and returns 400s. Always ask: after this drop, is the remaining label-set still unique per source series? Second, dropping instance/pod at the remote-write layer to “save cardinality” while keeping per-instance metric names achieves exactly that collapse; the correct tool for de-instancing is a recording rule that sum bys, not a label drop.

Metadata, exemplars, native histograms

Remote-Write 1.0 vs 2.0 on the wire

Remote-Write 2.0 (spec finalized 2024; sender support arriving through Prometheus 2.5x and standard in 3.x) is a new protobuf message negotiated over the same HTTP+snappy transport. Selected per endpoint via protobuf_message:

Operational guidance: treat RW 2.0 as a per-endpoint rollout. Flip one low-risk endpoint, confirm the receiver’s written-samples headers match sent counts, watch receiver CPU (interning usually reduces it), then migrate. Keep 1.0 config in version control for rollback — a receiver behind an old ingress that strips content-type parameters will fail opaquely.

Receiver backends: Thanos Receive, Mimir, VictoriaMetrics, Cortex

All four accept Prometheus remote-write and serve PromQL back. They differ in architecture, tenancy, dedup, downsampling, storage medium, and license — differences that dominate your operations for years. Endpoints and tenancy first:

The operational comparison — this is the table to argue from in design reviews:

Capability details that decide real projects:

And the decision table:

Minimal receiver-side configuration sketches — enough to recognize the shape of each system:

# Thanos Receive (hashring member): quorum-replicated, multi-tenant TSDBs
# thanos receive \
#   --receive.replication-factor=3 \
#   --receive.hashrings-file=/etc/thanos/hashrings.json \
#   --receive.hashrings-algorithm=ketama \
#   --receive.tenant-header=THANOS-TENANT \
#   --tsdb.retention=2d \
#   --objstore.config-file=/etc/thanos/objstore.yaml

# Mimir per-tenant guardrails (runtime overrides file)
overrides:
  team-platform:
    ingestion_rate: 250000            # samples/s
    ingestion_burst_size: 500000
    max_global_series_per_user: 3000000
    out_of_order_time_window: 5m
    compactor_blocks_retention_period: 13m   # 13 months
  team-payments:
    ingestion_rate: 100000
    max_global_series_per_user: 1000000
    compactor_blocks_retention_period: 90d

# VictoriaMetrics single-node: one flag each for retention and HA dedup
/victoria-metrics-prod \
  -retentionPeriod=12 \
  -dedup.minScrapeInterval=30s \
  -storageDataPath=/var/lib/victoria-metrics

Agent mode, and federation vs remote-write

Agent mode: Prometheus as a pure shipper

Agent mode (--agent in Prometheus 3.x; --enable-feature=agent in 2.x) strips Prometheus down to scrape + WAL + remote-write. There is no head-block query path, no local blocks, no rules, no alerting — the data directory (data-agent/) holds only the WAL, truncated as soon as samples are confirmed shipped.

When agent mode is right — and when it quietly ruins your week:

The last two rows are the classic mistakes: teams flip a fleet to agent mode and keep rule_files in the config (silently ignored — validate with promtool check config which warns), or discover during their first backend outage that no cluster can answer “is prod up?” locally. Decide alerting topology first, mode second.

Federation: the legacy global view, and where it still fits

Federation is a pull pattern: a central Prometheus scrapes /federate on leaf servers, copying whatever series match the match[] selectors at that instant.

scrape_configs:
  - job_name: federate-clusters
    scrape_interval: 30s
    honor_labels: true            # keep the leaf's job/instance labels
    metrics_path: /federate
    params:
      "match[]":
        - '{__name__=~"job:.*"}'  # recorded aggregates ONLY - never ".+"
    static_configs:
      - targets:
          - prom-eu1.internal:9090
          - prom-us1.internal:9090

Head-to-head, because interviews and design docs keep asking:

The one-line rule: federate aggregates, remote-write everything you must keep. A federation job whose match[] is {__name__=~".+"} is an incident in waiting — it doubles your biggest Prometheus into your central one, at 1/6 the resolution, with no delivery guarantees.

Long-term retention and downsampling strategies

Retention strategy is per-backend because the mechanisms differ fundamentally. First, the sample-count arithmetic that motivates downsampling — one series, one year, by resolution:

A 90-day dashboard over raw 15 s data touches 240× more samples than over 1 h resolution. That is why long-range queries against downsample-capable backends feel instant and against raw-only backends need aggressive query splitting/caching (Mimir’s answer) or brute efficiency (VictoriaMetrics’ answer).

Thanos: real downsampling, with a catch

The Thanos Compactor produces 5-minute resolution for blocks older than ~40 hours and 1-hour resolution for blocks older than ~10 days. Each downsampled series stores five aggregates (count, sum, min, max, counter) so that rate, avg, min, max all remain answerable. Retention is set per resolution:

thanos compact \
  --objstore.config-file=/etc/thanos/objstore.yaml \
  --data-dir=/var/thanos/compact \
  --retention.resolution-raw=30d \
  --retention.resolution-5m=180d \
  --retention.resolution-1h=2y \
  --wait

The catch experts know: downsampling is a query-speed feature, not primarily a storage saver. Because each downsampled series carries five aggregates, 5m data is nowhere near 1/20th of raw size — and for the window where you retain both raw and 5m, total storage increases. Never set --retention.resolution-raw below ~40 h (the 5m downsampling input) or 5m retention below ~10 d (the 1h input), or the compactor cannot produce the next tier. Queriers pick resolution automatically via max_source_resolution (--query.auto-downsampling).

Mimir, Cortex, VictoriaMetrics: retention without downsampling

The tiering strategy that actually works

Design retention as explicit tiers with named audiences, then map tiers onto your backend’s mechanism:

The most cost-effective trick in this table is the last column of “Cold”: on raw-only backends, run a second remote_write endpoint carrying only recorded aggregates (the keep on [a-z_]+:.*:.* pattern from the filtering section) into a long-retention tenant, and keep the full-fidelity tenant at 30–90 days. You get two-year trend queries over series that cost hundreds, not millions.

Capacity planning: the series and samples math

All sizing flows from two numbers you can measure today: active series and scrape interval. Everything else is multiplication. A worked fleet — 3 Kubernetes clusters, 60 nodes total, ~2,400 pods, typical exporters (assumptions stated so you can swap in your own):

Now the derived numbers:

Do not trust the bandwidth constant — measure yours in one query, because label verbosity dominates it:

# Actual compressed bytes per sample on the wire, per remote endpoint
rate(prometheus_remote_storage_bytes_total[15m])
  / rate(prometheus_remote_storage_samples_total[15m])

Sender (Prometheus) sizing rules of thumb — verify against your own ratio (process_resident_memory_bytes / prometheus_tsdb_head_series, commonly 3–8 KB/series including index, churn, and query headroom):

Receiver sizing (published rules of thumb — treat as starting points, then load-test; series-in-memory at the receiver = active series × replication factor):

Three planning rules that prevent the classic failures. Plan for churn, not just count: a deploy that restarts every pod creates a full generation of new series while the old ones linger in the head for hours — head series can be 1.5–2× steady-state during rollout storms, and receiver per-tenant series limits must absorb that or you get 429/per_user_series_limit drops precisely during deploys. Plan the catch-up path: after a 2-hour receiver outage, the backlog is 2 h × samples/s; at 30% spare shard throughput the drain takes ~6.6 h — pre-provision max_shards and receiver ingest for ~1.5–2× steady state. Alert on growth, not thresholds: predict_linear(prometheus_tsdb_head_series[6h], 86400 * 7) against your RAM budget turns capacity from an incident into a ticket.

HA pairs and deduplication at the receiver

Prometheus HA is brutally simple: run two identical replicas scraping the same targets. Each sees every sample (modulo scrape phase), each remote-writes independently, and the receiver — not the pair — is responsible for storing one copy. The pair is identified by external_labels:

# prometheus-a.yml                      # prometheus-b.yml differs ONLY in replica
global:
  external_labels:
    cluster: eu1-prod
    __replica__: prom-a                 # prom-b on the sibling

Two label conventions exist: Mimir/Cortex expect cluster + __replica__ (the __replica__ label is stripped on accept, so stored series are replica-free); Thanos convention is a visible replica or prometheus_replica label that survives storage and is collapsed at query time. Per backend:

The corresponding cost profile is worth spelling out: Mimir/Cortex HA tracking discards the duplicate stream at the edge — you pay double sender bandwidth but single storage. Thanos Receive stores both replicas (double storage before compaction-level dedup) and pays query-time dedup CPU. VictoriaMetrics ingests both but persists one sample per interval — near-single storage, no election machinery. For Mimir, the config that makes HA work:

# mimir.yaml (relevant fragment)
limits:
  accept_ha_samples: true
  ha_cluster_label: cluster
  ha_replica_label: __replica__
distributor:
  ha_tracker:
    enable_ha_tracker: true
    kvstore:
      store: memberlist

Two sharp edges. First, never mix conventions: if the replica label is not exactly what the tracker expects, both streams are accepted as distinct series — instant 2× series bill and rate() sawtooth artifacts in every dashboard. Second, do not put a load balancer that merges HA streams round-robin into one connection pool without the tracker enabled — interleaved timestamps from two replicas for the same series produce out-of-order 400s that the sender permanently drops. The confirm-and-fix for both lives in the troubleshooting table.

Monitoring the shipping pipeline itself

The pipeline that carries your telemetry needs its own telemetry, and it already exists — Prometheus instruments remote-write exhaustively. The metrics that matter, grouped by the question they answer (all carry remote_name and url labels):

The alert pack — these five rules, largely aligned with the upstream prometheus-mixin, catch every failure class in this article before users do:

groups:
  - name: remote-write-pipeline
    rules:
      - alert: PrometheusRemoteWriteBehind
        expr: |
          (
            max_over_time(prometheus_remote_storage_highest_timestamp_in_seconds[5m])
            - ignoring(remote_name, url) group_right
            max_over_time(prometheus_remote_storage_queue_highest_sent_timestamp_seconds[5m])
          ) > 120
        for: 15m
        labels: {severity: critical}
        annotations:
          summary: "Remote write to {{ $labels.url }} is {{ $value | humanize }}s behind"

      - alert: PrometheusRemoteWriteFailures
        expr: |
          (
            rate(prometheus_remote_storage_samples_failed_total[5m])
            /
            (rate(prometheus_remote_storage_samples_failed_total[5m])
             + rate(prometheus_remote_storage_samples_total[5m]))
          ) * 100 > 1
        for: 15m
        labels: {severity: critical}
        annotations:
          summary: "{{ printf \"%.1f\" $value }}% of samples to {{ $labels.url }} failing (permanent loss)"

      - alert: PrometheusRemoteWriteDesiredShards
        expr: |
          prometheus_remote_storage_shards_desired
            > prometheus_remote_storage_shards_max
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "Queue {{ $labels.remote_name }} wants more shards than max_shards allows"

      - alert: PrometheusRuleEvaluationProblems
        expr: |
          increase(prometheus_rule_evaluation_failures_total[5m]) > 0
            or increase(prometheus_rule_group_iterations_missed_total[5m]) > 0
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "Recording/alerting rules failing or missing evaluations on {{ $labels.instance }}"

      - alert: PrometheusWALGrowing
        expr: |
          predict_linear(prometheus_tsdb_wal_storage_size_bytes[30m], 6*3600)
            > 0.8 * (node_filesystem_size_bytes{mountpoint="/prometheus"})
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "WAL on {{ $labels.instance }} will breach 80% of the volume within 6h (remote-write backlog?)"

Watch the receiver side with its own counters and close the loop per tenant — on Mimir/Cortex, cortex_discarded_samples_total broken down by reason (rate_limited, per_user_series_limit, out_of_order, too_old, label_value_too_long…) is the single most valuable receiver metric: it tells each tenant why their data vanished. Thanos Receive exposes thanos_receive_replications_total / ..._forward_requests_total failures; VictoriaMetrics exposes vm_rows_ignored_total with reason labels. Route these alerts through a routing tree that separates platform pages from tenant notifications — Designing Alertmanager Routing Trees covers that layer.

Architecture at a glance

Picture the reference topology as three zones left to right. Zone one, per cluster: an HA pair of Prometheus servers (or agents, where local alerting is not required), each scraping the same targets through the same relabeling, stamped with external_labels: {cluster, __replica__}. Inside each server, samples flow appender → WAL → WAL watcher → per-endpoint queue → 4–100 shards, with write_relabel_configs deciding, per endpoint, whether the full stream or only job:* recorded aggregates leave the building. Recording rules evaluate locally in dependency-ordered groups, so the expensive aggregations are computed once, next to the raw data, and their outputs join the outbound stream. Cluster-scoped alerts also evaluate here, because the pair must be able to page even when the WAN or the backend is down.

Zone two, the receiving edge: a load balancer terminating TLS in front of the backend’s write path — Mimir distributors, a Thanos Receive hashring, or vminsert nodes — where tenancy headers are enforced, per-tenant rate and series limits are applied, and HA deduplication (Mimir’s tracker electing one replica per cluster) collapses the pair back into one logical stream. Ingesters/receivers hold the recent window in memory, replicate it (RF=3, zone-aware), and cut 2-hour blocks. Zone three, durable storage and read: object storage holds compacted blocks; the compactor merges, enforces per-tenant retention, and — on Thanos — downsamples to 5m/1h tiers; store-gateways serve historical blocks; queriers and a caching query-frontend fan PromQL across recent + historical; and a backend ruler evaluates the global rules no single cluster could. Grafana sees exactly one datasource. Every arrow in that picture is monitored from the sending side by prometheus_remote_storage_* and from the receiving side by per-tenant discard counters — the two vantage points you triangulate from during every incident in this pipeline.

Real-world scenario

NordPay, a payments processor, ran six EKS clusters with HA Prometheus pairs (2.1 M active series per cluster average, 15 s scrape on SLO-critical jobs, 60 s elsewhere) remote-writing to a central Mimir on S3 — roughly 420 k samples/s at the receiver after HA dedup, 13-month retention for the payments tenant. Two incidents in one quarter taught them everything in this article.

Incident one: the silent 30-minute blindfold. A routine Mimir upgrade rolled ingesters slowly; distributor p99 push latency rose from 90 ms to 700 ms for 40 minutes. Each Prometheus computed higher desired shards, hit the default max_shards: 50, and fell behind — samples_pending climbed to 14 M per replica and dashboards went ~25 minutes stale. Nobody was paged: the team had alerts on Mimir’s own health (all green — it was up, just slow) but nothing on sender-side lag. The postmortem added the PrometheusRemoteWriteBehind alert (gap > 120 s for 15 m) and re-derived shard math: at 300 ms degraded RTT they needed ~63 shards for 210 k samples/s per replica; they set max_shards: 120, min_shards: 8, and — critically — verified Mimir distributors had 2× ingest headroom so the extra shards helped rather than harmed. Catch-up after the fix drained a 40-minute backlog in 11 minutes.

Incident two: the ₹3.1 lakh label. A checkout-service deploy added card_bin (six digits, ~1,400 observed values) to http_requests_total and its 14-bucket duration histogram. Series for that job went from 38 k to 2.9 M over two hours. Mimir’s per_user_series_limit (3 M for the tenant) began rejecting with err-mimir-max-series-per-user; cortex_discarded_samples_total{reason="per_user_series_limit"} spiked — and because those rejections are HTTP 400s, the senders permanently dropped the excess: the team lost 90 minutes of partial checkout telemetry during a payment-provider brownout they were simultaneously debugging. Containment was a sender-side emergency filter (write_relabel_configs dropping card_bin via labeldrop — safe here because the underlying series remained unique per pod), then a proper fix at the source with metric_relabel_configs, plus a CI gate rejecting any exporter change that adds labels to SLO-critical metric families without a cardinality budget line. Finance saw the counterfactual afterwards: had the limit not stopped it, 2.9 M extra series × 13-month retention ≈ 5.6 TB-months ≈ ₹3.1 lakh/quarter in storage and ingester memory they never intended to buy. The limit was the guardrail working — the missing piece had been alerting on the guardrail, so now cortex_discarded_samples_total by tenant/reason pages the owning team, not the platform team.

The durable outcomes: sender lag and receiver discards are both paged with tenant attribution; max_shards is derived from measured RTT quarterly; every tenant has a series budget with predict_linear growth alerts; and the payments tenant ships full fidelity for 90 days but only job:* recorded aggregates to the 13-month tenant — cutting the long tier’s series count by 97% with zero dashboard changes.

Advantages and disadvantages

When do the disadvantages win? Below roughly one million active series and 30-day retention with a single team, a well-sized single Prometheus (or one per cluster plus thin federation of aggregates) is genuinely better: fewer moving parts, zero staleness, no bill surprise. The stack in this article earns its complexity at multi-cluster, multi-team, or multi-quarter-retention scale — adopt it because you measured the walls approaching, not because the architecture diagram looks mature.

Hands-on lab

Build the whole pipeline on your laptop with Docker: an edge Prometheus in agent mode scraping node metrics and remote-writing — with tuned queue, filtering, and HA-style external labels — into a central Prometheus acting as a remote-write receiver that evaluates the recording rules. You will test rules with promtool, watch the pipeline metrics live, and break the receiver on purpose to watch the WAL buffer save you. Cost: zero; everything is local.

1. Create the workspace.

mkdir -p ~/rw-lab/{edge,central,tests} && cd ~/rw-lab

2. Central config — receiver + recording rules. The --web.enable-remote-write-receiver flag turns any Prometheus into a spec-compliant remote-write endpoint at /api/v1/write — perfect for labs and small hub-and-spoke setups.

# central/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 30s
rule_files:
  - /etc/prometheus/rules.yml
scrape_configs: []        # receives everything via remote-write

# central/rules.yml
groups:
  - name: node_cpu_aggregation
    interval: 30s
    limit: 500
    rules:
      - record: instance_mode:node_cpu_seconds:rate1m
        expr: sum by (instance, mode) (rate(node_cpu_seconds_total[1m]))
      - record: instance:node_cpu_utilisation:ratio_rate1m
        expr: |
          1 - sum by (instance) (instance_mode:node_cpu_seconds:rate1m{mode="idle"})
              / sum by (instance) (instance_mode:node_cpu_seconds:rate1m)

3. Edge config — agent scraping + tuned remote_write.

# edge/prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    cluster: lab
    __replica__: edge-a
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]
remote_write:
  - name: central
    url: http://central:9090/api/v1/write
    queue_config:
      min_shards: 1
      max_shards: 10
      max_samples_per_send: 500
      batch_send_deadline: 5s
      retry_on_http_429: true
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_(gc|memstats)_.*"
        action: drop

4. Compose file.

# docker-compose.yml
services:
  node-exporter:
    image: quay.io/prometheus/node-exporter:v1.8.2
  central:
    image: prom/prometheus:v3.4.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.enable-remote-write-receiver
      - --storage.tsdb.retention.time=2d
    volumes: ["./central:/etc/prometheus"]
    ports: ["9090:9090"]
  edge:
    image: prom/prometheus:v3.4.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --agent
    volumes: ["./edge:/etc/prometheus"]
    ports: ["9091:9090"]

5. Test the rules BEFORE running anything.

# tests/rules.test.yml
rule_files:
  - ../central/rules.yml
evaluation_interval: 30s
tests:
  - interval: 15s
    input_series:
      - series: 'node_cpu_seconds_total{instance="n1", mode="idle"}'
        values: '0+12x40'      # 0.8 cores idle
      - series: 'node_cpu_seconds_total{instance="n1", mode="user"}'
        values: '0+3x40'       # 0.2 cores user
    promql_expr_test:
      - expr: instance:node_cpu_utilisation:ratio_rate1m
        eval_time: 5m
        exp_samples:
          - labels: 'instance:node_cpu_utilisation:ratio_rate1m{instance="n1"}'
            value: 0.2

docker run --rm -v "$PWD:/lab" --entrypoint promtool prom/prometheus:v3.4.1 \
  test rules /lab/tests/rules.test.yml
# Expected:
#   Unit Testing:  /lab/tests/rules.test.yml
#     SUCCESS
docker run --rm -v "$PWD/central:/c" --entrypoint promtool prom/prometheus:v3.4.1 \
  check rules /c/rules.yml
# Expected: SUCCESS: 2 rules found

6. Launch and validate the flow.

docker compose up -d && sleep 45

# Samples arriving at central, carrying the edge's external labels:
curl -s 'http://localhost:9090/api/v1/query?query=up{cluster="lab"}' | python3 -m json.tool | grep -E '"cluster"|"__replica__"|"value"'
# Expected: "cluster": "lab", "__replica__": "edge-a", value [..., "1"]

# Recording rules evaluating ON RECEIVED DATA at the central:
curl -s 'http://localhost:9090/api/v1/query?query=instance:node_cpu_utilisation:ratio_rate1m' \
  | python3 -m json.tool | grep -c '"instance"'
# Expected: 1 (one series, a sane 0.0-1.0 value)

# Pipeline health from the edge's own /metrics (agent mode still self-instruments):
curl -s http://localhost:9091/metrics | grep -E \
 'remote_storage_(samples_total|samples_pending|samples_failed_total|shards) '
# Expected: samples_total climbing, pending near 0, failed 0, shards 1

7. Break it — watch the WAL buffer work.

docker compose stop central && sleep 180   # 3-minute receiver outage
curl -s http://localhost:9091/metrics | grep -E 'samples_(pending|retried_total|failed_total) '
# Expected: pending in the thousands, retried_total climbing, failed_total STILL 0

docker compose start central && sleep 60
curl -s http://localhost:9091/metrics | grep -E 'samples_pending '
# Expected: pending back near 0 - the backlog drained in order, nothing lost.
# Confirm no gap: the range query over the outage window has continuous values.
curl -s 'http://localhost:9090/api/v1/query_range?query=up{cluster="lab"}&start='"$(date -u -v-10M +%s)"'&end='"$(date -u +%s)"'&step=15'

8. Teardown.

docker compose down -v && cd ~ && rm -rf ~/rw-lab

What you proved: rules tested offline, evaluated centrally on remote-written data (the agent-mode edge could not have run them), filtering applied per endpoint, external labels flowing end to end, and — the money lesson — a receiver outage covered entirely by the WAL with samples_failed_total at zero. Swap step 4’s central for a Mimir or Thanos Receive container later and only the url, a tenant header, and the dedup config change.