Thanos in Production: Global Query View, Deduplication, and Object-Storage Downsampling

Prometheus is superb at being a single node and hopeless at being a fleet. It was designed that way on purpose: one process scrapes its targets, writes a local time-series database (TSDB), evaluates its own rules, and answers PromQL against the data on its own disk. That model is fast, simple, and operationally bulletproof for one cluster. It falls apart the moment you need three things production almost always demands eventually — high availability (run two replicas so a node loss doesn’t blind you), long retention (keep a year of data for capacity planning and compliance, not two weeks), and a global view (one query surface across regions, clusters, and teams). A single Prometheus gives you none of these without bolting something on, and the naive bolt-ons (federation, a giant local disk, a load balancer in front of two replicas) each break in their own ugly way.

Thanos is the bolt-on that actually works, and it works by not changing Prometheus at all. Your Prometheus instances keep scraping and alerting exactly as they do today. A thin kit of single-purpose, mostly-stateless binaries then turns their local TSDB blocks into a globally queryable, deduplicated, downsampled, near-infinite-retention store backed by cheap object storage (S3, Azure Blob, GCS). The Sidecar ships sealed blocks to a bucket and serves the recent window; the Store Gateway serves everything historical from the bucket; the Querier fans out across all of them and presents one PromQL endpoint that transparently merges HA replicas and picks the cheapest data resolution for the time range you asked for; the Compactor is the bucket’s janitor that compacts, downsamples, and enforces retention. This article assembles that stack the way it actually runs at scale — every component and its flags, the deduplication penalty algorithm that makes HA pairs collapse cleanly, the object-storage block layout you must verify by hand, the 5m/1h downsampling tiers and the single-compactor rule that protects them, the caching that makes object storage fast enough to query, how Thanos compares to Cortex and Grafana Mimir, real objstore.config YAML for all three clouds, and the operational runbook for the day a block goes bad.

By the end you will be able to design and operate a multi-region Thanos deployment without the two failure modes that bankrupt teams: a silently halted Compactor that keeps serving raw blocks until your object-storage bill explodes, and a missing replica label that doubles every rate() and fires alerts on phantom data. You will know which component is stateful and which is disposable, where every byte of memory goes, what each cache buys you, and exactly which thanos tools bucket command to run when the Querier stops returning data. This is the level of detail that separates a Thanos deployment that survives its first month-end reporting spike from one that pages you at 02:00 with an S3 invoice attached.

What problem this solves

A single Prometheus is a single point of failure, a single retention budget, and a single query island. Three distinct pains follow, and most teams hit all three within a year of going past one cluster.

The HA pain. To survive a node loss you run two identical Prometheus replicas scraping the same targets. Now you have two copies of every series, slightly offset in time because the scrapes don’t align to the millisecond. Put a naive load balancer in front and your dashboards flicker between replicas; every panel can show a step where one replica had a gap. Worse, if you point Grafana at “both” by unioning them, every counter rate() is computed across doubled samples and reads roughly 2× the truth, and alerts fire on series that don’t exist. There is no clean way to merge two HA Prometheus replicas with stock Prometheus — the merge has to understand that two series differing only in a replica label are the same logical series, and stitch across each other’s gaps. That merge is exactly what Thanos’s Querier does.

The retention pain. Prometheus stores everything on local disk at full resolution forever-until-you-delete-it. Keep 15 days and a busy cluster is already tens of gigabytes; keep 13 months for compliance and you are provisioning terabytes of fast SSD per replica, per cluster, and you still lose it all if the disk dies because local TSDB has no durability story. Querying a year of 15-second-resolution data is also brutally slow and memory-hungry — you scan billions of samples to draw a quarterly trend nobody needs at second granularity. Thanos moves old blocks to object storage (durable, cheap, effectively infinite) and downsamples them to 5-minute and 1-hour resolutions so a year-long query reads a few thousand points instead of billions.

The global-view pain. You have Prometheus in us-east-1, eu-west-1, and ap-southeast-2, plus a dozen Kubernetes clusters each with their own. A single SLO dashboard spanning all of them, or a fleet-wide “total error rate across every region” query, is impossible with isolated Prometheis. Prometheus federation (one Prometheus scraping /federate from others) sort-of works for small aggregates but doesn’t scale, drops resolution, and creates a new single point of failure. Thanos’s Querier connects to every Prometheus (via its Sidecar) and every historical block (via the Store Gateway) and presents one PromQL endpoint over the whole fleet — the global view you actually wanted.

Who hits this: anyone running more than one Prometheus and caring about it. It bites hardest on teams that adopted Prometheus per-cluster (now they have twenty query islands), teams under compliance retention mandates (13-month, 7-year), and cost-sensitive teams whose Prometheus disk bill is climbing linearly while object storage would be a tenth the price. The alternative bolt-ons — bigger disks, federation, Cortex’s older blocks-less architecture — each trade one pain for another; Thanos’s bet is that object storage + a stateless read fan-out solves all three at once, and for most teams that bet pays off.

To frame the whole field before the deep dive, here is what each problem looks like with stock Prometheus, the naive fix people try first, and what Thanos does instead:

Learning objectives

By the end of this article you can:

• Name every Thanos component (Sidecar, Store Gateway, Querier, Compactor, Receive, Ruler, Query Frontend), state which is stateful and which is disposable, and assemble a minimal HA topology and a multi-region one.
• Configure the Sidecar to ship TSDB blocks to S3, Azure Blob, or GCS, including the two mandatory Prometheus flags that disable local compaction, and verify the bucket layout by hand.
• Build the global query surface with the Querier, configure deduplication with --query.replica-label, and explain the penalty deduplication algorithm that stitches HA replicas across gaps.
• Operate the Compactor safely: understand vertical/horizontal compaction, configure 5m and 1h downsampling, set retention tiers so raw outlives the downsample window, and enforce the single-compactor-per-bucket rule.
• Make object-storage queries fast with an index cache, a caching bucket (chunk cache), and Query Frontend result caching — and size memcached for the working set.
• Shard the Store Gateway by block and the Compactor by tenant/group so the read and compaction paths scale horizontally past a single instance.
• Decide between keeping rules on Prometheus versus the Thanos Ruler, and run the Ruler as an HA pair correctly.
• Choose between Thanos, Cortex, and Grafana Mimir for a given scale and operating model, and explain the architectural trade-offs.
• Diagnose the canonical Thanos failures — a halted Compactor, overlapping blocks, doubled metrics, OOM-ing Store Gateways, stale uploads — with the exact thanos tools bucket command and metric to confirm each.

Prerequisites & where this fits

You should already run Prometheus comfortably: you know what a TSDB block is (a 2-hour immutable directory of an index plus chunk files plus a meta.json), what external labels are and why every Prometheus needs unique ones, how remote_write works at a high level, and how to write PromQL with rate(), histogram_quantile(), and aggregations. You should be fluent with one object store (S3/Blob/GCS) and its IAM model (IRSA on EKS, workload identity on GKE, managed identity on AKS), and comfortable on Kubernetes since that is where Thanos almost always runs (Helm chart, or the thanos-io/thanos images as Deployments/StatefulSets). Familiarity with gRPC helps because every Thanos component speaks the StoreAPI over gRPC.

This sits squarely in the metrics long-term-storage and global-query layer of an observability stack. Upstream of it is everything that produces metrics and the discipline to keep them affordable: Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus matters even more once those series live in object storage for a year, and Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir is the broader framing this article drills into. The query language you run against the global view is the same one covered in PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work. Its closest architectural sibling and main alternative is Running Grafana Mimir: Multi-Tenant, Horizontally Scalable Prometheus Storage, which shares Thanos’s block format and object-storage model but assembles it as a single multi-tenant system rather than a kit. Rules and alerts that ride on top connect to Designing Alertmanager Routing Trees: Grouping, Inhibition, Silences, and Dedup.

A quick map of who owns what during a Thanos incident, so you escalate to the right place:

Core concepts

Six mental models make every later decision in this article obvious. Internalize these and the flags stop being arbitrary.

Thanos is a kit, not a monolith. It is a set of single-purpose binaries (all the same thanos image, different subcommands) that each do one job and speak a common gRPC interface, the StoreAPI. There is no central server. The Querier is the only thing that knows about all the others, and it knows them only as a list of StoreAPI endpoints to fan out to. This is the opposite of Cortex/Mimir, which package similar capabilities as one multi-component-but-unified system. The kit model means you compose exactly the pieces you need and scale each independently — but it also means you are responsible for wiring them correctly, and the single most dangerous wiring mistake (two Compactors) is one the kit will not stop you from making.

Object storage is the source of truth; everything else is a cache or a fan-out. Once a block is in the bucket, the Querier and Store Gateway can be deleted and recreated with zero data loss — they hold only caches and in-memory indexes derived from the bucket. The bucket is durable (11 nines on S3), cheap, and the one thing you must back up and protect with lifecycle policies and least-privilege IAM. The corollary: a healthy bucket plus the config to read it is your disaster-recovery plan for the read path. The only components holding unique state are Prometheus (recent, un-shipped data), the Receive (its WAL of pushed data), the Ruler (its own small TSDB of rule results), and the Compactor’s scratch disk (recreatable).

The StoreAPI is the universal seam. Every read backend — Sidecar (recent data from Prometheus), Store Gateway (historical from the bucket), Ruler (its rule blocks), Receive (its pushed data), even another Querier — exposes the same gRPC StoreAPI. The Querier doesn’t care what’s behind an endpoint; it asks each “what series and time ranges do you have, and give me these samples,” then merges. This uniformity is why the architecture is so composable: you add capacity or a new data source by adding another StoreAPI endpoint to the Querier’s list.

External labels identify a Prometheus; one replica label identifies its HA twin. Every Prometheus stamps its external labels (e.g. cluster, region, replica) into every block’s meta.json. The Querier uses these to know which blocks are genuinely distinct (different cluster/region — keep both, that’s the global view) versus which are HA replicas of each other (identical labels except the replica label — merge them, that’s deduplication). Getting external labels right is the entire foundation: identical except one replica label for an HA pair, fully distinct for genuinely separate Prometheis. Botch this and you either double your data (forgot the replica label) or accidentally merge unrelated clusters (made them too similar).

Downsampling is lossless for aggregations. A 5-minute or 1-hour downsampled point is not “every 5th sample” — it stores five aggregates per original series (count, sum, min, max, and the counter value), so rate(), increase(), histogram_quantile(), avg_over_time(), max_over_time() all stay correct at coarse resolution. The Querier auto-selects the resolution: a one-hour dashboard with a 15-second step reads raw blocks; a one-year dashboard with an hour step reads cheap 1h blocks. You never lose the ability to answer the question; you only lose the ability to zoom into 15-second detail on data older than your raw retention — which is exactly the trade you want.

The Compactor is a singleton and the most dangerous component. Exactly one Compactor may operate on a bucket at a time. Two will race, each trying to compact and downsample the same source blocks, and produce overlapping blocks the system cannot reconcile — at which point the Compactor halts and refuses to proceed, downsampling stops, and (silently) every long-range query falls back to scanning raw blocks until your bill explodes. The Compactor is also the only component whose mistakes are written permanently into the bucket. Treat it like a database primary: one of it, never autoscaled, alerted on thanos_compact_halted.

The vocabulary in one table

Pin down every moving part before the deep sections. The glossary at the end repeats these for lookup; this is the mental model side by side.

Component topology — every binary and what it does

Thanos is assembled from these binaries. Know each before you deploy any of them. The Querier fans out to everything that speaks StoreAPI; the rest are either read backends, the bucket janitor, or optional ingestion.

The mental model in one breath: Sidecar and Store Gateway are read backends, the Querier (fronted by Query Frontend) is the read frontend, the Compactor is the bucket janitor, and Ruler/Receive are optional ingestion. A minimal HA deployment is two Prometheus replicas each with a Sidecar, one Store Gateway, one Querier, and exactly one Compactor. A multi-region deployment runs that per region and adds a “global” Querier on top that fans out to each region’s Querier (Queriers can be StoreAPI endpoints for other Queriers — the architecture is recursive).

Sidecar vs Receive — two ways data enters Thanos

There are two ingestion models, and choosing wrong is a common early mistake. The Sidecar model is pull-based and the default: Prometheus scrapes as normal, the Sidecar uploads its blocks. The Receive model is push-based: agents remote_write to a Thanos Receive hashring, which holds the data in a WAL and ships blocks itself — there is no Prometheus at all on the receiving side. Use Receive when the data source can’t run a Sidecar (a managed/hosted Prometheus you don’t control, IoT/edge agents, multi-tenant SaaS where tenants push to you) or when you want a centralized write path. Use the Sidecar everywhere else — it’s simpler, has no extra stateful tier to operate, and keeps Prometheus’s own alerting local and resilient.

Shipping TSDB blocks to object storage

Every Sidecar, Store Gateway, Compactor, and Ruler reads the same objstore.config. Define it once and mount it everywhere. For AWS S3:

# objstore-s3.yaml
type: S3
config:
  bucket: "acme-thanos-metrics"
  endpoint: "s3.us-east-1.amazonaws.com"
  region: "us-east-1"
  # Prefer IRSA / instance profile over static keys.
  # access_key / secret_key only if you truly cannot use a role.
  sse_config:
    type: "SSE-S3"          # or SSE-KMS with a kms_key_id
  # Tune for cost and throughput on busy buckets:
  put_user_metadata: {}
  http_config:
    idle_conn_timeout: 90s
    max_idle_conns: 100

Azure Blob and GCS use the same --objstore.config-file flag with a different type:

# objstore-azure.yaml
type: AZURE
config:
  storage_account: "acmethanos"
  container: "metrics"
  # Use a managed identity (AKS workload identity / pod identity);
  # leave storage_account_key empty so the SDK uses the identity.
  endpoint: "blob.core.windows.net"
  max_retries: 3

# objstore-gcs.yaml
type: GCS
config:
  bucket: "acme-thanos-metrics"
  # service_account omitted -> uses Workload Identity / ADC.
  # Provide the JSON only if you cannot use Workload Identity.

Thanos supports a range of object stores through the same provider interface. The common ones and their type:

The two mandatory Prometheus flags

The Sidecar ships blocks only when you point Prometheus at it and disable Prometheus’s own compaction. Two flag groups on Prometheus are non-negotiable:

prometheus \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.min-block-duration=2h \
  --storage.tsdb.max-block-duration=2h \
  --web.enable-lifecycle \
  --web.enable-admin-api          # lets the Sidecar trigger snapshots if needed

Setting min and max block duration both to 2h disables Prometheus’s local compaction. This is mandatory: the Thanos Compactor must own all compaction in the bucket, and it cannot reconcile blocks that Prometheus has already merged locally (Prometheus would create 8h/1d blocks that overlap with what the Compactor builds, and the Compactor halts). If you take one thing from this article: --storage.tsdb.min-block-duration must equal --storage.tsdb.max-block-duration and both must be 2h. Then run the Sidecar:

thanos sidecar \
  --tsdb.path=/prometheus \
  --prometheus.url=http://localhost:9090 \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --objstore.config-file=/etc/thanos/objstore-s3.yaml

The Sidecar uploads a block roughly every two hours, after Prometheus seals its head block to disk. Recent (un-shipped) data is still queryable because the Sidecar also serves the local TSDB over StoreAPI — the Querier sees the recent ~2h window from the Sidecar and everything older from the Store Gateway, with no gap and no double-counting (the Sidecar stops serving a time range once the Store Gateway can serve it from the uploaded block).

The Prometheus flags that matter for Thanos, and what breaks if you get them wrong:

External labels are the deduplication key

Each Prometheus in an HA pair must carry identical external labels except for one replica label:

# prometheus-a.yml
global:
  external_labels:
    cluster: "prod-use1"
    region: "us-east-1"
    replica: "a"     # the ONLY difference vs the partner

The partner (prometheus-b.yml) is identical with replica: "b". These labels are stamped into every block’s meta.json and are how the Querier later knows two series are replicas of each other rather than genuinely distinct. Keep Prometheus’s own local retention short (a day or two) since the bucket now holds history — there’s no reason to hoard months on the scrape node’s expensive disk.

A reference for designing your external-label scheme — what to include, why, and the trap:

Verify the bucket layout

After a Sidecar has run for a few hours, inspect what landed. Do not trust that it worked — check:

thanos tools bucket ls \
  --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --output=table

Each row is a block ULID with its time range, sample count, resolution, compaction level, and external labels. The on-disk layout under the bucket is one directory per block:

acme-thanos-metrics/
  01J8X.../              # block ULID (lexicographically time-sortable)
    meta.json            # time range, labels, downsample resolution, compaction level
    index                # the block's inverted index (postings + series)
    chunks/000001        # compressed sample chunks (512MB segments)
    chunks/000002
  debug/                 # marker files (deletion / no-compact) live nearby

The meta.json is the source of truth. The two fields you watch evolve as the Compactor works:

Watch those fields: thanos.downsample.resolution: 0 means raw; compaction.level: 1 means a fresh 2h block straight from a Sidecar; a block older than a couple of days that still shows resolution: 0 and level: 1 means the Compactor isn’t running — which is the single most expensive silent failure in Thanos.

Querier deduplication across HA pairs

The Querier connects to every StoreAPI endpoint and presents one PromQL surface. Point it at your Sidecars and Store Gateway:

thanos query \
  --http-address=0.0.0.0:9090 \
  --grpc-address=0.0.0.0:10903 \
  --query.replica-label=replica \
  --endpoint=dns+thanos-sidecar.monitoring.svc:10901 \
  --endpoint=dns+thanos-store.monitoring.svc:10901 \
  --query.auto-downsampling

Using dns+ with a headless Kubernetes Service makes the Querier discover all Sidecar pods behind that name — you do not list each replica by hand. The critical flag is --query.replica-label=replica. It tells the Querier that series differing only in the replica label are the same logical series. With --query.auto-downsampling, the Querier picks the coarsest resolution whose step still satisfies the query, so wide dashboards automatically read cheap downsampled blocks.

With deduplication enabled (the default in the UI, or dedup=true on the API), the Querier merges the two replica time series, preferring whichever has continuous data and stitching across gaps when one replica was down. Disable it with dedup=false and you see both replicas side by side — useful when debugging why one Prometheus missed scrapes:

# Deduplicated (one clean series)
curl -s 'http://thanos-query:9090/api/v1/query?query=up&dedup=true'

# Raw, both replicas visible
curl -s 'http://thanos-query:9090/api/v1/query?query=up&dedup=false'

The penalty deduplication algorithm

The hard part of deduplication is how to merge two replicas whose samples are offset by a few seconds and which each have occasional gaps. A naive “just take series A, fall back to B” produces jagged rate() results at every switchover. Thanos uses a penalty-based deduplication algorithm: it iterates the merged stream and, when it must switch from one replica to the other (because the current one has a gap), it applies a time penalty before accepting the other replica’s samples, choosing the path that yields the most continuous series and avoids double-counting samples that are really the same scrape seen twice. This is why dedup is correct for counters: the algorithm understands counter resets and won’t fabricate a spike when stitching A’s data to B’s.

You can supply multiple replica labels (for example replica plus a prometheus_replica) by repeating the flag; the Querier treats any combination of those labels as identifying replicas. For genuinely separate Prometheus instances — different clusters, different external labels — the Querier keeps them distinct and you query the union, which is exactly the “global view” you wanted.

A reference for the deduplication knobs and behaviors:

A common production mistake: forgetting --query.replica-label. Without it, every counter rate() doubles, every gauge shows two slightly offset lines, and alerts fire on phantom data. If your graphs suddenly show pairs of nearly-identical lines, this flag is missing. Confirm instantly with the count(up) test: dedup=true should return half of dedup=false for an HA pair.

Compactor: compaction, downsampling, retention

The Compactor is the only component stateful against the bucket and the most dangerous to misconfigure. It does three jobs: compaction (merging 2h blocks into 8h, then daily, then larger blocks — fewer, bigger blocks are cheaper to query), downsampling (computing 5m and 1h aggregates so long-range queries stay cheap), and retention (deleting blocks past their per-resolution tier limit).

thanos compact \
  --data-dir=/var/thanos/compact \
  --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --http-address=0.0.0.0:10912 \
  --retention.resolution-raw=30d \
  --retention.resolution-5m=120d \
  --retention.resolution-1h=730d \
  --compact.concurrency=2 \
  --downsample.concurrency=2 \
  --delete-delay=48h \
  --wait

The retention model is the heart of cost control. Thanos keeps three resolutions of the same data, each on its own retention tier:

Downsampling does not throw away accuracy for aggregations — because each downsampled point stores the five aggregates, rate(), histogram_quantile(), and avg_over_time() remain correct at coarse resolution. The Querier auto-selects the right resolution for the query’s step. A 6-month dashboard never touches raw blocks; it reads cheap 1h aggregates.

The downsampling timing rule

Downsampling only happens after blocks are compacted to the required source level. A 5m downsample needs source blocks covering at least 40 hours; a 1h downsample needs source blocks covering roughly 10 days. Brand-new data is always served raw. The critical consequence: if you set --retention.resolution-raw shorter than the downsampling window, you delete data before it can be downsampled and create permanent gaps. Raw retention must comfortably outlive the time it takes to compact-then-downsample (give it at least a few days of headroom past the 5m window’s source requirement).

A safe ordering for the three retentions is raw < 5m < 1h, with raw long enough (e.g. 30 days) that downsampling has finished well before raw is deleted. Setting raw to, say, 90d “to be safe” is a different trap: it’s too long, so nobody notices when downsampling silently stops because raw data is still there to (expensively) serve.

Avoiding compactor halts — the single-compactor rule

The Compactor halts and refuses to proceed when it finds overlapping blocks it cannot reconcile — almost always caused by running two Compactors, or by a Sidecar shipping blocks from a Prometheus that did its own local compaction. The fix is prevention: exactly one Compactor per bucket (or per disjoint group, if sharded), and min/max-block-duration=2h on every Prometheus. A halt is not a crash; the process keeps running and serving metrics but stops doing its job, which is why it’s so dangerous — downsampling quietly stops while raw data keeps (expensively) serving wide queries.

If a halt happens, do not delete blocks blindly. Diagnose first:

# List overlapping blocks the compactor is choking on
thanos tools bucket verify \
  --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --issues=overlapped_blocks

# See the whole bucket as a timeline to spot the overlap visually
thanos tools bucket web \
  --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --http-address=0.0.0.0:8080

Genuine accidental overlaps can be repaired offline with thanos tools bucket rewrite (rewrites a block dropping the overlapping series/time range) or, as a last resort, by marking the bad block for deletion with thanos tools bucket mark --marker=deletion-mark.json. Run the Compactor with a generous --data-dir on fast disk — it downloads blocks to merge them, and a daily compaction of a busy bucket needs tens of GB of scratch space.

The Compactor flags you actually tune, with defaults and the gotcha for each:

Sharding the Compactor at scale

A single Compactor eventually can’t keep up with a multi-terabyte bucket. Shard it by compaction group (the set of blocks sharing the same external-label set) using a relabel config, and run one Compactor per shard — each owning a disjoint set of groups. The rule still holds within a group: exactly one Compactor per group, never two. Sharding is horizontal scale for compaction; it is not permission to run two Compactors over the same blocks.

# compact-shard-tenant-a.yaml  (one Compactor keeps only tenant=a's groups)
- action: keep
  source_labels: ["tenant"]
  regex: "a"

Pass it with --selector.relabel-config-file=/etc/thanos/compact-shard-tenant-a.yaml. Run a second Compactor with a keep on tenant=b, and so on. Each Compactor now owns a disjoint slice of the bucket and they never collide.

Store Gateway: index caching, chunk caching, and sharding

The Store Gateway answers all historical queries by reading blocks from object storage. Object storage is high-latency (tens of milliseconds per GET, charged per request and per byte), so caching is what makes it usable. Configure three layers: the Store Gateway’s in-memory index header (always present — the postings/series index of every block it owns), an external index cache (postings and series lookups, shared across replicas), and a caching bucket / chunk cache (the actual sample data and object-existence checks).

thanos store \
  --data-dir=/var/thanos/store \
  --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --index-cache-size=2GB \
  --chunk-pool-size=4GB \
  --store.grpc.series-sample-limit=50000000

For anything beyond a single instance, move the index cache to memcached so it survives restarts and is shared across Store Gateway replicas:

# index-cache-memcached.yaml
type: MEMCACHED
config:
  addresses: ["dns+memcached.monitoring.svc:11211"]
  max_item_size: "16MiB"
  max_async_concurrency: 20
  timeout: 500ms

Pass it with --index-cache.config-file=/etc/thanos/index-cache-memcached.yaml. A separate caching-bucket config (--store.caching-bucket.config-file) caches chunk subranges and the existence of objects, cutting GET requests to the object store dramatically — which matters because the store charges per request and per byte.

# caching-bucket.yaml
type: MEMCACHED
config:
  addresses: ["dns+memcached-chunks.monitoring.svc:11211"]
  max_item_size: "1MiB"
chunk_subrange_size: 16000
max_chunks_get_range_requests: 3
chunk_object_attrs_ttl: 24h
chunk_subrange_ttl: 24h

The three caches, what each holds, where it lives, and what you lose without it:

Sharding the Store Gateway

A single Store Gateway eventually cannot hold the index headers for a multi-terabyte bucket in memory (index header size is proportional to total series across all blocks it owns). Shard by block using relabeling on the block’s identity, so each replica owns a disjoint slice:

# store-shard-0.yaml  (one file per shard)
- action: hashmod
  source_labels: ["__block_id"]
  target_label: shard
  modulus: 3
- action: keep
  source_labels: ["shard"]
  regex: "0"

Run three Store Gateways, each with its own --selector.relabel-config-file keeping shard 0, 1, and 2. The Querier fans out to all three and merges. This is horizontal scale for the read path: no single Store Gateway has to know about every block, so index-header memory is divided across the shards. You can also shard by time (one Store for recent blocks, one for old) so the hot recent shard gets more replicas and cache.

Query Frontend: splitting, caching, and parallelizing

In front of the Querier sits an optional but highly recommended Query Frontend. It does three things that make wide and repeated queries dramatically faster: it splits a long-range query into per-day sub-queries it can run in parallel; it caches results (in memcached/Redis) so a dashboard refresh over the same window returns instantly; and it applies retries and limits so one slow shard doesn’t stall everything. It speaks the Prometheus HTTP API, so Grafana points at the Query Frontend, which points at the Querier.

thanos query-frontend \
  --http-address=0.0.0.0:9090 \
  --query-frontend.downstream-url=http://thanos-query:9090 \
  --query-range.split-interval=24h \
  --query-range.max-retries-per-request=3 \
  --labels.split-interval=24h \
  --query-frontend.compress-responses \
  --cache-compression-type=snappy \
  --query-range.response-cache-config-file=/etc/thanos/frontend-cache.yaml

# frontend-cache.yaml
type: MEMCACHED
config:
  addresses: ["dns+memcached-frontend.monitoring.svc:11211"]
  max_item_size: "5MiB"
  max_async_concurrency: 10

What each Query Frontend capability buys you:

Thanos Ruler vs Prometheus rules

You now have two places rules can live, and the distinction is about what data the rule needs to see.

Keep alerting and recording rules on Prometheus when they only need local, recent data: per-target up alerts, fast burn-rate SLO alerts on a single cluster, host-level recording rules. Local evaluation is faster, survives a Querier outage, and is the resilient default. This is the right home for the vast majority of rules.

Use Thanos Ruler only for rules that genuinely need the global, deduplicated view — cross-cluster aggregations, fleet-wide SLOs spanning regions, or recording rules over data that no single Prometheus holds:

thanos rule \
  --data-dir=/var/thanos/rule \
  --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --query=dns+thanos-query.monitoring.svc:9090 \
  --rule-file=/etc/thanos/rules/*.yaml \
  --alertmanagers.url=dns+alertmanager.monitoring.svc:9093 \
  --label='ruler_cluster="global"' \
  --label='replica="r0"' \
  --grpc-address=0.0.0.0:10901 \
  --eval-interval=1m \
  --resend-delay=1m

The Ruler writes its rule-result blocks to the bucket (so recording rules become long-term data) and serves them over StoreAPI, so the Querier sees them too. Run the Ruler itself as an HA pair (distinct replica labels) and let the Querier deduplicate its output blocks, exactly like Prometheus pairs.

The trap with Ruler: it evaluates rules over the network against the Querier. If the Querier is slow or a Store Gateway is degraded, rule evaluation lags and alerts are delayed. Never move your critical, low-latency alerts to Ruler — keep those on Prometheus where they evaluate against the local TSDB and fire even if the entire Thanos read path is down.

Capacity planning, query limits, and where memory goes

Thanos’s main scaling failure mode is a single expensive query reading millions of samples through the Store Gateway and exhausting memory, OOM-killing the Store Gateway and taking down all historical queries for everyone. Cap query blast radius on the Querier and the Store Gateway:

# On the Querier
thanos query \
  --query.max-concurrent=20 \
  --query.timeout=2m \
  --store.response-timeout=30s \
  --query.max-concurrent-select=4 \
  --query.partial-response          # return partial data if a store is down

# On the Store Gateway
thanos store \
  --store.grpc.series-sample-limit=50000000 \
  --store.grpc.series-max-concurrency=20 \
  --store.grpc.touched-series-limit=50000000

--store.grpc.series-sample-limit is the safety valve: a runaway query that would touch more than 50M samples is rejected with an error instead of OOM-killing the Store Gateway. Size it against your real dashboards — too low and legitimate long-range panels fail; too high and one bad query is a denial of service.

Where the memory actually goes, per component, so you size pods correctly:

A worked rule of thumb: provision Store Gateway memory for index headers (roughly proportional to total active series, on the order of a few KB per series across all blocks) plus your chunk pool plus burst query memory, and size memcached to hold the working set of postings for your common dashboards. When index-header memory exceeds what one pod can hold, shard — don’t just keep growing the pod.

Thanos vs Cortex vs Mimir

Thanos is not the only way to give Prometheus global scale and long retention. The other two serious options share its block format and object-storage model but package it differently. The choice is mostly about operating model and multi-tenancy, not raw capability.

Cortex is the original “Prometheus-as-a-service” — a horizontally scalable, multi-tenant system you remote_write into. It started with a chunks-based storage engine and later adopted the same blocks-on-object-storage model Thanos pioneered. It is push-based (no Sidecar), strongly multi-tenant, and operationally heavier (more components, a ring, ingesters with WALs).

Grafana Mimir is a fork/evolution of Cortex by Grafana Labs, re-engineered for much higher scale (billions of active series), simpler operations (a “monolithic” single-binary mode plus microservices mode), and strong multi-tenancy out of the box. It uses the Thanos/Prometheus block format and object storage, adds its own query engine optimizations, and is the modern choice when you want a turnkey, massively scalable, multi-tenant metrics backend. See Running Grafana Mimir: Multi-Tenant, Horizontally Scalable Prometheus Storage for the deep dive.

The decision in one table:

The honest heuristic: if you already run Prometheus per-cluster and want to keep that model while gaining a global view, dedup, and long-term storage with the fewest new moving parts, choose Thanos. If you want a single, central, multi-tenant write-target that hundreds of Prometheis push into and you’re optimizing for the highest possible scale and operational simplicity at that scale, choose Mimir. Choose Cortex mainly if you already operate it. Mimir and Thanos are far more alive than Cortex in 2026; for a green-field “central metrics backend” decision it is usually Thanos-vs-Mimir.

Architecture at a glance

There is no diagram here; the topology is simple enough to hold in your head once you see the data flow described, and the value is in the flow, not a picture. Trace a metric from scrape to a year-old query.

A metric is born when a Prometheus replica scrapes a target and writes it to its local TSDB head block. Two such replicas (replica: a and replica: b) scrape the same targets for HA — identical external labels except that one replica label. Every two hours each Prometheus seals a 2h block to disk, and its co-located Sidecar uploads that block to the object-storage bucket and, in the meantime, serves the still-on-disk recent data over the gRPC StoreAPI. So at any instant the freshest ~2 hours live on Prometheus disk (served by the Sidecar) and everything older lives in the bucket.

In the bucket, the lone Compactor continuously does three things: it merges 2h blocks into 8h then daily blocks (fewer, bigger, cheaper to scan); it downsamples blocks older than ~40h into 5m-resolution copies and blocks older than ~10 days into 1h-resolution copies (each downsampled point keeping count/sum/min/max/counter so aggregations stay exact); and it deletes blocks past their per-resolution retention tier (raw after 30 days, 5m after 120, 1h after two years). The Store Gateway reads those bucket blocks and serves them over StoreAPI, holding each owned block’s index header in memory and leaning on an index cache (postings) and a caching bucket (chunks, in memcached) so object storage is fast enough to query.

On the read path, the Querier is the single PromQL endpoint. It fans out over StoreAPI to every Sidecar (recent data) and every Store Gateway shard (historical data), asks each what it has, and merges. Crucially it deduplicates: series that differ only in the replica label are collapsed into one continuous series by the penalty algorithm, so HA pairs read as a single clean line and rate() is correct. It also auto-selects resolution: a one-hour panel pulls raw blocks; a one-year panel pulls cheap 1h blocks, never touching raw. A Query Frontend in front of the Querier splits wide queries by day, runs them in parallel, and caches results so dashboard refreshes are instant. A Thanos Ruler (optional) evaluates cross-cluster rules against the Querier and writes its results back to the bucket as blocks. Grafana points at the Query Frontend and sees one global, deduplicated, multi-resolution, year-deep PromQL surface — while each Prometheus keeps scraping and alerting locally, untouched. The whole right-hand side (Querier, Store Gateway, Query Frontend) is stateless or cache-only: lose any of it and recreate it from the bucket with zero data loss.

The single path to remember, left to right: target → Prometheus(×2) → Sidecar → bucket → Compactor (compact/downsample/retain) → Store Gateway → Querier (dedup + resolution-select) → Query Frontend → Grafana. The Sidecar bridges live data in; the bucket is the durable truth; the Compactor keeps it cheap; the Querier makes it one global view.

Real-world scenario

Northwind Payments ran Prometheus HA pairs in three regions — us-east-1, eu-west-1, and ap-southeast-2 — six Prometheus instances total, each on a 1.5 TB local SSD holding 15 days of data. They needed two things they couldn’t get from isolated Prometheis: a single global SLO dashboard for the payments API across all regions, and 13-month retention for PCI compliance audits. Monthly object-storage budget was set at roughly ₹40,000; the platform team was three engineers.

They stood up Thanos the textbook way: a Sidecar beside each of the six Prometheis shipping to a single S3 bucket, one Store Gateway, one Querier with --query.replica-label=replica, a Query Frontend, and a Compactor with --retention.resolution-raw=90d --retention.resolution-5m=400d --retention.resolution-1h=0 (1h kept forever for compliance). For three weeks it was perfect — the global dashboard worked, dedup collapsed the HA pairs, Grafana was snappy.

Then month-end hit and the pager exploded. Finance pulled a 13-month, 1-second-step revenue-by-region dashboard. The Querier happily tried to serve it — from raw blocks — because the Compactor had silently halted seventeen days earlier. Store Gateways OOM-killed one after another as each wide query tried to load billions of raw samples; the global dashboard went blank; and the S3 bill for the month was already 4× budget from the GET volume of weeks of wide queries scanning raw data that should have been 1h downsampled.

The breakthrough was the first metric they should have been alerting on. curl -s http://thanos-compact:10912/metrics | grep thanos_compact_halted returned 1. The Compactor had halted on overlapping blocks. thanos tools bucket verify --issues=overlapped_blocks showed the culprit: a fourth-region migration two months earlier had left a second Compactor running against the same bucket — two janitors racing, producing overlaps neither could reconcile. And because raw retention was set to a generous 90d, nobody noticed downsampling had stopped: raw data was still there to (expensively) serve, masking the halt completely.

The fix was disciplined and ran in three moves. First, scale every Compactor to zero, confirm none are running, then start exactly one with a --selector.relabel-config-file so even a future stray copy would own a disjoint group. Second, repair the overlap offline:

# Confirm the overlap and identify the bad blocks
thanos tools bucket verify --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --issues=overlapped_blocks
# Mark the duplicate (migration-era) blocks for deletion, with the standard delay
thanos tools bucket mark --objstore.config-file=/etc/thanos/objstore-s3.yaml \
  --id=01J... --marker=deletion-mark.json --details="duplicate from 2nd compactor"

Third, cap query blast radius so one dashboard could never DoS the bucket again, and fix the retention so a halt couldn’t hide:

thanos store \
  --store.grpc.series-sample-limit=50000000 \
  --index-cache.config-file=/etc/thanos/index-cache-memcached.yaml \
  --store.caching-bucket.config-file=/etc/thanos/caching-bucket.yaml
# Retention re-tiered: raw shortened so a halt becomes visible fast
#   --retention.resolution-raw=30d  (was 90d)
#   --retention.resolution-5m=400d  --retention.resolution-1h=0

With one Compactor, downsampling caught up over the next 48 hours; the 13-month dashboard dropped from raw to 1h resolution; S3 GET volume fell by over 90%; and the sample limit ensured a single broad query now failed fast with an error instead of cascading into fleet-wide OOM. The bill returned to budget the following month. Two alerts went up the same day — thanos_compact_halted > 0 and time() - thanos_objstore_bucket_last_successful_upload_time{} > 7200 — and a runbook line went on the wall: “A halted Compactor is invisible until it bankrupts you. Shorten raw retention so the gap shows, and alert on the halt directly.”

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

The “kit of stateless binaries over a durable bucket” model both enables global Prometheus and demands discipline. Weigh it honestly.

The model is right when you already run Prometheus and want a global, deduplicated, long-retention view with the fewest new stateful tiers, and your team can hold the operational discipline (one Compactor, replica labels, alerts on halts and stale uploads). It bites hardest on teams that treat the Compactor casually (autoscaling it, running two by accident), forget the replica label, or skip caching and then blame object storage for being slow. Every disadvantage is manageable — but only if you know it exists, which is the entire point of this article.

Hands-on lab

Stand up a complete, single-node Thanos stack locally with Docker Compose and MinIO as the object store — no cloud account, no charges. You’ll run two Prometheus replicas (an HA pair), two Sidecars, a Store Gateway, a Compactor, and a Querier, then prove deduplication and the bucket layout. Everything tears down with one command.

Step 1 — Workspace and MinIO bucket config.

mkdir -p thanos-lab && cd thanos-lab
cat > objstore.yaml <<'EOF'
type: S3
config:
  bucket: "thanos"
  endpoint: "minio:9000"
  access_key: "minioadmin"
  secret_key: "minioadmin"
  insecure: true            # MinIO over plain HTTP for the lab only
EOF

Step 2 — Two Prometheus configs (the HA pair). Identical except the replica label, and both with local compaction disabled via the block-duration flags (set in the compose command, below).

cat > prometheus-a.yml <<'EOF'
global:
  scrape_interval: 15s
  external_labels:
    cluster: "lab"
    replica: "a"
scrape_configs:
  - job_name: "self"
    static_configs: [{ targets: ["prometheus-a:9090"] }]
  - job_name: "node"
    static_configs: [{ targets: ["node-exporter:9100"] }]
EOF
sed 's/replica: "a"/replica: "b"/; s/prometheus-a:9090/prometheus-b:9090/' \
  prometheus-a.yml > prometheus-b.yml

Step 3 — Docker Compose for the whole stack.

# docker-compose.yml
version: "3"
services:
  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    environment: { MINIO_ROOT_USER: minioadmin, MINIO_ROOT_PASSWORD: minioadmin }
    ports: ["9001:9001"]
  createbucket:
    image: minio/mc
    depends_on: [minio]
    entrypoint: >
      /bin/sh -c "until /usr/bin/mc alias set m http://minio:9000 minioadmin minioadmin;
      do sleep 1; done; /usr/bin/mc mb -p m/thanos; exit 0;"
  node-exporter:
    image: prom/node-exporter

  prometheus-a:
    image: prom/prometheus:v2.53.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.min-block-duration=2h
      - --storage.tsdb.max-block-duration=2h
      - --web.enable-lifecycle
    volumes: ["./prometheus-a.yml:/etc/prometheus/prometheus.yml"]
  prometheus-b:
    image: prom/prometheus:v2.53.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.min-block-duration=2h
      - --storage.tsdb.max-block-duration=2h
      - --web.enable-lifecycle
    volumes: ["./prometheus-b.yml:/etc/prometheus/prometheus.yml"]

  sidecar-a:
    image: quay.io/thanos/thanos:v0.36.0
    command:
      - sidecar
      - --tsdb.path=/prometheus
      - --prometheus.url=http://prometheus-a:9090
      - --grpc-address=0.0.0.0:10901
      - --objstore.config-file=/etc/thanos/objstore.yaml
    volumes: ["./objstore.yaml:/etc/thanos/objstore.yaml"]
  sidecar-b:
    image: quay.io/thanos/thanos:v0.36.0
    command:
      - sidecar
      - --tsdb.path=/prometheus
      - --prometheus.url=http://prometheus-b:9090
      - --grpc-address=0.0.0.0:10901
      - --objstore.config-file=/etc/thanos/objstore.yaml
    volumes: ["./objstore.yaml:/etc/thanos/objstore.yaml"]

  store:
    image: quay.io/thanos/thanos:v0.36.0
    command:
      - store
      - --data-dir=/var/thanos/store
      - --grpc-address=0.0.0.0:10901
      - --objstore.config-file=/etc/thanos/objstore.yaml
    volumes: ["./objstore.yaml:/etc/thanos/objstore.yaml"]

  compact:
    image: quay.io/thanos/thanos:v0.36.0
    command:
      - compact
      - --data-dir=/var/thanos/compact
      - --http-address=0.0.0.0:10912
      - --objstore.config-file=/etc/thanos/objstore.yaml
      - --retention.resolution-raw=30d
      - --retention.resolution-5m=120d
      - --retention.resolution-1h=730d
      - --wait
    volumes: ["./objstore.yaml:/etc/thanos/objstore.yaml"]

  query:
    image: quay.io/thanos/thanos:v0.36.0
    command:
      - query
      - --http-address=0.0.0.0:9090
      - --query.replica-label=replica
      - --endpoint=sidecar-a:10901
      - --endpoint=sidecar-b:10901
      - --endpoint=store:10901
    ports: ["9090:9090"]

Step 4 — Bring it up.

docker compose up -d
docker compose ps          # all services Up; createbucket exits 0 after making the bucket

Expected: MinIO, two Prometheis, two Sidecars, Store, Compactor, and Querier all running.

Step 5 — Open the Querier and confirm it sees every backend. Browse to http://localhost:9090 (the Thanos Querier UI). Under Stores, you should see sidecar-a, sidecar-b, and store, all healthy. Or via the API:

curl -s http://localhost:9090/api/v1/stores | \
  python3 -c 'import sys,json; [print(s["name"], s.get("lastError")) for s in json.load(sys.stdin)["data"]["sidecar"]+json.load(open("/dev/stdin"))]' 2>/dev/null \
  || curl -s http://localhost:9090/api/v1/stores

Expected: each store entry has lastError: null and a sane minTime/maxTime.

Step 6 — Prove deduplication. Run the count(up) test — with dedup the HA pair collapses, without it you see double:

# Deduplicated: counts each target once
curl -s 'http://localhost:9090/api/v1/query?query=count(up)&dedup=true' \
  | python3 -c 'import sys,json;print("dedup=true :", json.load(sys.stdin)["data"]["result"][0]["value"][1])'

# Not deduplicated: counts each target twice (once per replica)
curl -s 'http://localhost:9090/api/v1/query?query=count(up)&dedup=false' \
  | python3 -c 'import sys,json;print("dedup=false:", json.load(sys.stdin)["data"]["result"][0]["value"][1])'

Expected: dedup=false is exactly 2× dedup=true. That doubling-collapse is deduplication working. (If they’re equal, your --query.replica-label is missing.)

Step 7 — Watch the bucket fill (after ~2h, or simulate). A fresh lab has no shipped blocks yet (Prometheus seals a block every 2h). To see the layout without waiting, list whatever exists and inspect the structure:

docker compose run --rm store \
  tools bucket ls --objstore.config-file=/etc/thanos/objstore.yaml --output=table
# After a block ships, each row shows ULID, time range, samples, resolution(0=raw), level(1)

You can also open the MinIO console at http://localhost:9001 (minioadmin/minioadmin) and browse the thanos bucket to see the <ULID>/meta.json, index, and chunks/ layout once a block lands.

Validation checklist. You ran a full HA Thanos stack, the Querier discovered all three StoreAPI backends, the count(up) test proved deduplication collapses the replica pair 2:1, and you inspected the object-storage block layout. What each step proved:

Teardown (removes everything, including volumes).

docker compose down -v
cd .. && rm -rf thanos-lab

Cost note. Entirely local — MinIO and Thanos in Docker cost nothing. In a real cloud, the equivalent is a few rupees of object storage per GB-month plus per-request charges, which is why production caching matters.

Common mistakes & troubleshooting

This is the runbook. First as a scannable table you keep open mid-incident, then the entries that bite hardest expanded with the full confirm-command detail.

The expanded form for the entries that cause the most pain:

1. Every rate() reads ~2×; dashboards show pairs of nearly-identical lines. Root cause: The Querier isn’t deduplicating because --query.replica-label is missing or names the wrong label, so HA replicas are treated as distinct series. Confirm: Run count(up) twice — dedup=true and dedup=false. For a clean HA pair, dedup=false should be exactly the replica factor (2×) of dedup=true. If they’re equal, dedup isn’t happening. Fix: Add --query.replica-label=replica (matching your external_labels.replica) to every Querier. Repeat the flag for multiple replica labels.

2. Long-range queries are slow and the object-storage bill is climbing. Root cause: The Compactor halted (usually on overlapping blocks), so downsampling stopped; the Querier falls back to scanning raw blocks for wide queries, multiplying GET requests and bytes. Confirm: curl -s http://thanos-compact:10912/metrics | grep thanos_compact_halted returns 1; thanos tools bucket ls shows old blocks still at resolution: 0. Fix: Run exactly one Compactor; repair the overlap (verify → mark/rewrite); shorten raw retention so a future halt is visible fast; alert on thanos_compact_halted > 0.

3. The Compactor halts on startup and won’t proceed. Root cause: Overlapping blocks it can’t reconcile — almost always a second Compactor racing it, or a Prometheus that did local compaction (block-duration flags not both 2h). Confirm: thanos tools bucket verify --objstore.config-file=… --issues=overlapped_blocks lists the overlapping ULIDs; thanos tools bucket web shows them on a timeline. Fix: Ensure one Compactor per bucket/group; fix the Prometheus block-duration flags; mark the accidental duplicate blocks for deletion (thanos tools bucket mark --marker=deletion-mark.json) or repair with thanos tools bucket rewrite. Never delete blindly.

4. Store Gateways get OOM-killed during wide queries. Root cause: Index-header memory (proportional to total series across owned blocks) plus the chunk pool plus burst query memory exceeds the pod limit; a single huge query tips it over. Confirm: Pod shows OOMKilled; container_memory_working_set_bytes rides the limit; the query touched a huge series count. Fix: Shard the Store Gateway by block (hashmod on __block_id) so index-header memory divides across shards; raise pod memory; set --store.grpc.series-sample-limit to reject runaway queries before they OOM.

7. New blocks never appear in the bucket; recent data eventually gaps. Root cause: The Sidecar can’t upload — bucket IAM/role missing the right permissions, wrong objstore.config, SSE/KMS denial, or a network path blocked. Confirm: time() - thanos_objstore_bucket_last_successful_upload_time grows without bound; the Sidecar logs show upload errors (access denied, no such bucket, timeout). Fix: Grant the Sidecar’s identity write access to the bucket (and KMS key if SSE-KMS); verify the objstore.config bucket/endpoint/region; check egress to the object store. Alert on stale thanos_objstore_bucket_last_successful_upload_time.

10. Permanent gaps appear in 5m/1h data for older time ranges. Root cause: Raw blocks were deleted (retention) before the Compactor could downsample them — --retention.resolution-raw was shorter than the downsample window needs. Confirm: Long-range panels show holes that line up with where raw retention cut off; the 5m/1h blocks for that span never got created. Fix: Lengthen raw retention so it comfortably outlives the downsample source requirement (5m needs ≥40h source blocks, 1h needs ≥10d). The gaps already created can’t be recovered — only future data is protected.

Best practices

• Exactly one Compactor per bucket (or per disjoint group). Never autoscale it, never run a second copy “to be safe.” Enforce it in your deployment (a single-replica StatefulSet, no HPA) and alert on thanos_compact_halted > 0. This is the number-one rule in Thanos.
• Set --storage.tsdb.min-block-duration equal to --max-block-duration (2h) on every Prometheus. This disables local compaction so the Thanos Compactor owns the bucket. Audit it across the fleet — one misconfigured Prometheus halts the Compactor for everyone.
• Always run the Querier with --query.replica-label and verify dedup with count(up). Make the dedup=true vs dedup=false 2:1 check part of your post-deploy validation.
• Make raw retention outlive the downsampling window, but not by much. Long enough that 5m/1h downsampling always finishes first (no gaps); short enough that a halted Compactor becomes visible quickly (a long raw retention masks the halt).
• Use external caches (memcached/Redis), not in-process ones. An external index cache and caching bucket survive Store Gateway restarts and are shared across replicas — cold caches after every rollout are a self-inflicted latency wound.
• Cap query blast radius. Set --store.grpc.series-sample-limit, --query.max-concurrent, and --query.timeout so one runaway dashboard fails fast with an error instead of OOM-killing the Store Gateway and taking down all historical queries.
• Keep critical, low-latency alerts on Prometheus, not the Ruler. Local evaluation fires even if the entire Thanos read path is down. Reserve the Ruler for genuinely cross-cluster rules, and run it as an HA pair.
• Shard before you grow the pod. When Store Gateway index-header memory or the Compactor’s throughput exceeds one instance, shard (by block / by group) rather than provisioning an ever-bigger pod.
• Put a Query Frontend in front of the Querier with result caching. It splits wide queries, parallelizes them, and makes repeated dashboard refreshes instant — a large, cheap win for Grafana-heavy teams.
• Use dns+ service discovery for StoreAPI endpoints. Point the Querier at headless Services (dns+thanos-store.monitoring.svc:10901) so it discovers all pods automatically; don’t hardcode replica addresses.
• Verify the bucket, don’t trust it. Run thanos tools bucket ls and check meta.json resolution/level after deploy and periodically; confirm old blocks are downsampled, not stuck at resolution: 0, level: 1.
• Pin and protect the bucket. Least-privilege IAM per component (Sidecar: write; Store: read; Compactor: read+write+delete), SSE encryption, and a lifecycle policy that does not delete blocks out from under Thanos (let the Compactor own retention).

The leading-indicator alerts to wire before your first incident:

Security notes

• Least-privilege per component. Give each component only the bucket permissions it needs: the Sidecar writes (PutObject); the Store Gateway reads (GetObject/ListBucket); the Compactor reads, writes, and deletes; the Querier and Query Frontend touch the bucket not at all. Use IRSA (EKS), workload identity (GKE), or managed identity (AKS) — never static keys baked into config.
• Encrypt at rest and in transit. Enable SSE (SSE-S3 or SSE-KMS with a customer-managed key) in objstore.config; the bucket holds a year of potentially sensitive metrics. Run StoreAPI gRPC with TLS (--grpc-server-tls-cert/--grpc-server-tls-key and client TLS on the Querier) so components don’t fan out in cleartext, especially across regions.
• Don’t expose the Querier or bucket tools to the internet. The Querier is a powerful read surface over your entire fleet; put it behind auth (an OAuth2 proxy / your ingress auth) and network policy. thanos tools bucket web and the admin endpoints must never be publicly reachable.
• Authenticate and isolate tenants if multi-tenant. Thanos’s multi-tenancy is label-based, not enforced isolation — if tenants must not see each other’s data, enforce it at the query gateway (per-tenant Queriers with hardcoded label selectors, or choose Mimir whose tenancy is first-class). Don’t rely on label hygiene alone for a security boundary.
• Protect the bucket lifecycle. A cloud-side lifecycle rule that deletes objects will silently destroy blocks Thanos still needs (and can cause Compactor halts). Let the Compactor own retention via its --retention.* flags; if you must have a lifecycle backstop, set it far longer than your 1h retention tier.
• Audit and pin component images. Run a known Thanos version (pin the image digest), watch the changelog for breaking flag changes, and scan the image. A surprise upgrade that changes default behavior on the Compactor is a bucket-wide risk.
• Secure the Ruler’s Alertmanager path. The Ruler sends alerts to Alertmanager over the network — use TLS and authentication so alert traffic (which can reveal topology) isn’t exposed.

The security controls that also prevent outages — secure and resilient pull the same direction:

Cost & sizing

The bill drivers, and how each interacts with the architecture:

• Object-storage capacity is the cheap part and the reason Thanos exists. A year of metrics that would be terabytes of SSD on Prometheus is a fraction of the cost in object storage — and downsampling shrinks the long-tail dramatically (1h blocks are tiny versus raw). Capacity cost scales with active series × retention, but the 5m/1h tiers keep the year-deep portion small.
• Object-storage requests are the part that surprises people. Every query through the Store Gateway is GET requests against the bucket, charged per request and per byte transferred. A halted Compactor (serving raw to wide queries) or missing caches turn this into the dominant line on the bill. Caching is a cost control, not just a latency control — the caching bucket cuts GETs by an order of magnitude.
• Compute is the Store Gateways (memory-heavy, for index headers and chunk pools), the Querier/Frontend (CPU and merge memory), the Compactor (one box, CPU + scratch disk), and memcached (sized to the working set). The Store Gateway is usually the largest single line; shard it rather than running one giant pod.
• Downsampling pays for itself fast. A 13-month dashboard reading 1h blocks reads a few thousand points; the same dashboard reading raw reads billions, costing both query memory and GET charges. The Compactor’s CPU to downsample is trivial next to what raw wide-queries cost.
• Egress matters in multi-region: if a Querier in one region reads a bucket in another, cross-region transfer charges apply. Keep Store Gateways co-located with their bucket’s region and fan out at the Querier layer.

A rough monthly picture for a mid-size fleet (six Prometheis, ~2M active series, 13-month 1h retention) on a major cloud: object-storage capacity is often surprisingly modest (a few thousand rupees), while requests can dwarf it without caching (tens of thousands) and drop back near capacity with a well-sized caching bucket. Compute (a few Store Gateway pods, a Querier/Frontend, one Compactor, memcached) is the steady line. The cost drivers and what each buys:

Interview & exam questions

1. Why can’t you just put a load balancer in front of two HA Prometheus replicas, and what does Thanos do instead? A load balancer routes a query to one replica, so you get whichever replica’s gaps and offsets, and you can’t get a merged continuous series; unioning both doubles every counter. Thanos’s Querier instead treats series differing only in the replica external label as the same logical series and merges them with the penalty deduplication algorithm, stitching across each replica’s gaps without double-counting — one correct, continuous series.

2. What is the single-compactor rule and why does violating it corrupt the bucket? Exactly one Compactor may operate on a bucket (or disjoint group) at a time. Two Compactors race to compact and downsample the same source blocks and produce overlapping blocks that neither can reconcile; the Compactor then halts, downsampling stops, and wide queries silently fall back to scanning raw blocks. Because the Compactor’s output is written permanently into the bucket, the damage persists until you repair the overlap offline.

3. Why must --storage.tsdb.min-block-duration equal --max-block-duration on Prometheus when using Thanos? Setting them both to 2h disables Prometheus’s local compaction. The Thanos Compactor must own all compaction in the bucket; if Prometheus also compacts (creating 8h/1d blocks), the Sidecar ships blocks that overlap with what the Compactor builds, and the Compactor halts on the unreconcilable overlap.

4. Explain downsampling and why it’s lossless for aggregations. The Compactor computes 5m and 1h downsampled blocks where each point stores five aggregates — count, sum, min, max, and the counter value — not just a sampled value. So rate(), increase(), histogram_quantile(), and *_over_time() remain correct at coarse resolution. The Querier auto-selects resolution by query step, so a year-long dashboard reads cheap 1h points while a one-hour dashboard reads raw — you only lose the ability to zoom into second-level detail on old data, which is the intended trade.

5. What happens if raw retention is set shorter than the downsampling window? Raw blocks get deleted before the Compactor can downsample them (5m needs ≥40h source blocks, 1h needs ≥10 days), creating permanent gaps in the 5m/1h data for those time ranges. The gaps can’t be recovered. Raw retention must comfortably outlive the downsample source requirement.

6. A user reports every rate() graph shows two nearly-identical lines and reads about double. What’s wrong and how do you confirm it? The Querier is missing --query.replica-label (or it names the wrong label), so HA replicas aren’t deduplicated and appear as distinct doubled series. Confirm with the count(up) test: dedup=false should be exactly 2× dedup=true for an HA pair; if they’re equal, dedup isn’t happening. Fix by adding --query.replica-label=replica.

7. Which Thanos components are stateful, and what’s the disaster-recovery implication? Only Prometheus (recent un-shipped data), the Receive (its WAL), the Ruler (its small TSDB), and the Compactor’s scratch disk hold unique state; the object-storage bucket is the durable source of truth. The Querier, Store Gateway, and Query Frontend are stateless/cache-only, so DR for the read path is simply “recreate them and point at the bucket” — no data loss.

8. When do you keep rules on Prometheus versus moving them to the Thanos Ruler? Keep rules on Prometheus when they only need local recent data (per-target up alerts, single-cluster SLOs, host recording rules) — local evaluation is faster and fires even if the Thanos read path is down. Use the Ruler only for rules needing the global deduplicated view (cross-cluster aggregates, fleet-wide SLOs); the trade-off is that the Ruler evaluates over the network against the Querier, so it lags if the read path is degraded — never put critical low-latency alerts there.

9. How do you scale the Store Gateway when it starts OOM-ing on a large bucket? Its memory is dominated by index headers proportional to the total series across the blocks it owns. Shard the Store Gateway by block (relabel hashmod on __block_id) and run several, each owning a disjoint slice so index-header memory divides across shards; the Querier fans out to all of them and merges. Shard before growing one giant pod, and back the index/chunk caches with external memcached.

10. Thanos vs Mimir — when would you pick each? Pick Thanos when you already run Prometheus per-cluster and want to keep that model while gaining a global view, deduplication, and long-term storage with the fewest new stateful tiers (the Sidecar model adds none). Pick Mimir when you want a single, central, turnkey, strongly multi-tenant write-target that hundreds of Prometheis push into, optimized for the highest scale and operational simplicity at that scale. Both use the same block format and object storage, so it’s an operating-model choice, not a capability gap.

11. What’s the difference between the Sidecar and Receive ingestion models? The Sidecar is pull-based: Prometheus scrapes normally and the Sidecar ships its blocks and serves recent data — no new stateful tier. Receive is push-based: agents remote_write to a Receive hashring that holds a WAL and ships blocks itself, with no Prometheus on the receiving side. Use Receive for sources that can’t run a Sidecar (hosted/edge/multi-tenant push); use the Sidecar everywhere else because it’s simpler and keeps local alerting resilient.

12. How do you prevent one expensive query from taking down all historical queries? Cap blast radius: --store.grpc.series-sample-limit on the Store Gateway rejects a query that would touch too many samples before it OOMs the pod; --query.max-concurrent and --query.timeout on the Querier bound concurrency and runtime; --query.partial-response returns partial data if a store is down. Without these, one wide dashboard can OOM a Store Gateway and blank every historical panel for everyone.

These map to vendor-neutral observability and SRE knowledge and to the Prometheus Certified Associate (PCA) scope (Prometheus fundamentals, TSDB, federation, remote storage), plus the platform-engineering portions of Kubernetes certifications where running Prometheus/Thanos on Kubernetes is in scope. A compact mapping for revision:

Quick check

1. You run an HA Prometheus pair and your rate() panels read about double. What flag is almost certainly missing, and how do you confirm it in one query?
2. Why must --storage.tsdb.min-block-duration equal --max-block-duration (2h) on every Prometheus feeding Thanos?
3. True or false: a 1h downsampled point throws away accuracy, so you shouldn’t use it for rate() over old data.
4. Your object-storage bill is climbing and long-range queries are slow. What’s the single metric you check first, and what does a value of 1 mean?
5. Which Thanos components can you delete and recreate with zero data loss, and why?

Answers

1. --query.replica-label (e.g. =replica) is missing on the Querier, so HA replicas aren’t deduplicated. Confirm with count(up): run it with dedup=false and dedup=true — for an HA pair dedup=false should be exactly 2× dedup=true. If they’re equal, dedup isn’t happening.
2. Setting both to 2h disables Prometheus’s local compaction, so the Thanos Compactor owns all compaction in the bucket. If Prometheus compacts locally it ships 8h/1d blocks that overlap what the Compactor builds, and the Compactor halts on the unreconcilable overlap.
3. False. A downsampled point stores count, sum, min, max, and the counter value, so rate(), histogram_quantile(), and *_over_time() stay correct at coarse resolution. You only lose the ability to zoom into second-level detail on old data.
4. thanos_compact_halted (from the Compactor’s /metrics). A value of 1 means the Compactor has halted — almost always on overlapping blocks — so downsampling stopped and wide queries fall back to scanning expensive raw blocks. Confirm overlaps with thanos tools bucket verify --issues=overlapped_blocks.
5. The Querier, Store Gateway, and Query Frontend — they hold only caches and in-memory indexes derived from the bucket, which is the durable source of truth. Recreate them and point at the bucket; nothing is lost. (Prometheus, Receive, Ruler, and the Compactor’s scratch disk hold the only unique state.)

Glossary

• Block — an immutable ~2-hour TSDB directory containing an index, chunks/, and a meta.json; the unit Thanos ships, compacts, downsamples, and retains.
• meta.json — a block’s metadata: time range (minTime/maxTime), external labels, compaction.level, and thanos.downsample.resolution; the source of truth when debugging.
• StoreAPI — the common gRPC interface every read backend exposes (Sidecar, Store Gateway, Ruler, Receive, even another Querier); the seam the Querier fans out over.
• External labels — per-Prometheus labels (cluster, region, replica) stamped into every block; identify a Prometheus and its HA replica.
• Replica label — the one external label that differs between HA twins (e.g. replica: a/b); the key the Querier deduplicates on.
• Sidecar — runs next to each Prometheus; serves its recent TSDB over StoreAPI and ships sealed 2h blocks to the bucket.
• Store Gateway — serves historical blocks from the bucket over StoreAPI; holds index headers in memory and is cache-only (disposable).
• Querier — the stateless PromQL frontend that fans out to every StoreAPI backend, deduplicates HA replicas, and auto-selects resolution.
• Query Frontend — sits before the Querier; splits long queries, runs them in parallel, caches results, and retries.
• Compactor — the singleton bucket janitor: compacts blocks, downsamples to 5m/1h, and enforces per-resolution retention.
• Compaction — merging many small 2h blocks into fewer, larger blocks (8h, 1d, …) that are cheaper to query.
• Downsampling — computing 5m and 1h aggregate blocks (count/sum/min/max/counter per point) so long-range queries are cheap and still correct for aggregations.
• Penalty (deduplication) algorithm — how the Querier merges HA replicas across gaps without double-counting, keeping counters correct.
• Receive — an optional push target for remote_write; holds a WAL and ships blocks itself, for sources that can’t run a Sidecar.
• Ruler — evaluates recording/alerting rules over the global deduplicated view (via the Querier) and writes its results to the bucket as blocks.
• objstore.config — the YAML (type + config) defining the bucket and credentials, shared by every bucket-touching component.
• Index cache — an external (memcached) cache of resolved postings/series lookups, shared across Store Gateway replicas and surviving restarts.
• Caching bucket — an external cache of chunk subranges and object-existence checks that cuts GET requests to object storage.
• Halt — the Compactor’s safe-stop state when it finds unreconcilable overlapping blocks; downsampling stops while metrics keep serving, making it dangerously silent.
• ULID — the lexicographically time-sortable identifier that names each block directory in the bucket.

Next steps

You can now design and operate a global, deduplicated, downsampled Thanos stack and survive its first month-end spike. Build outward:

• Next: Running Grafana Mimir: Multi-Tenant, Horizontally Scalable Prometheus Storage — the main alternative; see how the same block-and-bucket model is packaged as a turnkey multi-tenant system.
• Related: Scaling Prometheus: Recording Rules, Remote-Write, and Long-Term Storage with Thanos and Mimir — the broader long-term-storage framing this article drills into.
• Related: Taming Metric Cardinality: Relabeling, Limits, and Cost Governance in Prometheus — control series count before it lives in object storage for a year.
• Related: PromQL in Anger: Rate, Histograms, and Aggregation Patterns That Actually Work — the query language you run against the global Thanos surface.
• Related: Designing Alertmanager Routing Trees: Grouping, Inhibition, Silences, and Dedup — where the alerts from Prometheus and the Thanos Ruler converge.