Cosmos DB for NoSQL: Partition Key Design, RU Optimization, and Hot Partition Repair

Most Cosmos DB cost and latency incidents trace back to one decision made early and never revisited: the partition key. Get it right and the container scales horizontally and predictably to any throughput you can pay for. Get it wrong and you hit a wall no amount of RU/s can buy past, because a single physical partition tops out at 10,000 RU/s regardless of what you provision on the container. The cruel part is that the symptom — HTTP 429 under load while the container sits at 30% utilization — looks like an under-provisioning problem, so the reflex is to throw RU/s at it, which does nothing and burns money. This is a working guide to choosing the key, measuring and shrinking RU consumption, tuning the indexing policy, detecting a hot partition with partition-scoped metrics, and repairing a container that is already skewed in production.

Azure Cosmos DB for NoSQL is the globally distributed, horizontally partitioned document database where you trade a fixed schema and joins for predictable single-digit-millisecond latency at any scale — if your partitioning is sound. The whole model rests on one mechanism: Cosmos hashes your partition key, maps each key value to a logical partition, and packs logical partitions onto physical partitions it provisions behind the scenes. Every performance property — and every failure — is downstream of how evenly that hash spreads your traffic. This article treats the partition key, the Request Unit (RU), the indexing policy and the throughput mode as one coupled system, because in production they are.

By the end you will stop guessing. When 429s spike you will know within ninety seconds whether you face a genuinely under-provisioned container, a single hot logical partition saturating one physical partition’s 10,000 RU/s, a cross-partition fan-out query billing you the sum of every partition, an index write-tax from indexing properties you never query, or an autoscale break-even you got wrong. Because this is a reference you will return to mid-incident, the partition limits, RU costs, indexing knobs, throughput modes and the hot-partition playbook are all laid out as scannable tables — read the prose once, then keep the tables open when the dashboard is red.

What problem this solves

Cosmos DB hides enormous machinery so you can write a document and read it back in single-digit milliseconds anywhere on earth. That abstraction is a gift until your partitioning is wrong, then it becomes a wall you cannot climb with the throughput slider. The bare 429 Too Many Requests tells you almost nothing about which of five distinct causes you hit, and the container-level “Total Request Units” chart actively lies — it shows healthy average utilization while one physical partition is on fire.

What breaks without this knowledge: an on-call engineer doubles the provisioned RU/s (masking nothing — the hot partition is still capped at 10,000 RU/s), or migrates to a “bigger” account (no such thing helps a single saturated key), or files a support ticket and waits while checkout writes fail during a sale. Meanwhile the actual cause — a partition key like /merchantId that worked for hundreds of balanced tenants until one whale arrived, or a query that omits the key and fans out to every partition, or an indexing policy that indexes a 40-field document on every write — sits there, perfectly diagnosable, ignored.

Who hits this: every team running Cosmos DB at scale. It bites hardest on multi-tenant SaaS (power-law tenant distributions blow past a single tenant’s 20 GB / 10,000 RU/s ceiling), event/telemetry ingestion (monotonic /date keys create an append hot spot on the “current” partition), write-heavy workloads (default full-property indexing inflates every write), and anyone who picked a low-cardinality key like /status or /region early and cannot change it in place. The fix is almost never “more RU/s” — it’s “spread the key, align the query, trim the index, and migrate if the key itself is wrong.”

To frame the whole field before the deep dive, here is every symptom class this article covers, the question it forces, and the one place to look first:

Learning objectives

By the end of this article you can:

• Distinguish a logical partition from a physical partition and explain the 20 GB and 10,000 RU/s ceilings that derive every hot-partition incident.
• Evaluate any candidate partition key against cardinality, read alignment and write spread, and pick the right one (or a synthetic / hierarchical key) instead of the obvious-but-wrong one.
• Measure real RU cost from x-ms-request-charge / Data Explorer Query Stats instead of guessing, and read what drives a read, a write, a query and a stronger consistency level up.
• Tune an indexing policy to exclude /* and include only queried paths, add composite indexes for filter + ORDER BY, and cut write RU 30–50% on wide documents.
• Detect a hot partition with NormalizedRUConsumption (Max) split by PhysicalPartitionId plus 429-by-PartitionKeyRangeId, and confirm it with the three-signal triad.
• Choose between manual and autoscale throughput from the ~66%-utilization break-even, and explain why neither saves you from a single saturated key.
• Repair a skewed container with hierarchical partition keys, synthetic keys, or a change-feed migration to a correctly-keyed new container — with no maintenance window.
• Map the moving parts to DP-420 and AZ-204 exam objectives and answer the partition/RU questions cleanly.

Prerequisites & where this fits

You should already understand the Cosmos DB basics: an account holds databases, which hold containers (the unit of partitioning and throughput), which hold items (JSON documents). You should know how to run az in Cloud Shell, read JSON output, and that Cosmos exposes multiple APIs (NoSQL, MongoDB, Cassandra, Gremlin, Table) — this article is the NoSQL (formerly SQL/Core) API, though the partitioning mechanics apply broadly. Familiarity with JSON, basic SQL-like query syntax, and HTTP status codes helps.

This sits in the Data platform track. It assumes the modeling fundamentals (the Database Selection 101: SQL vs NoSQL — When to Use What decision is upstream of it) and the non-relational concepts from DP-900: Non-Relational Data and Analytics on Azure. It pairs tightly with Cosmos DB Multi-Region Writes & Conflict Resolution (global distribution layered on top of the partitioning you design here) and with Azure Monitor & Application Insights for Observability, because the hot-partition detection in this article lives in Azure Monitor metrics and Log Analytics. If you ingest a firehose into Cosmos, Event Hubs, Kafka Capture & Stream Analytics is usually the upstream.

A quick map of which layer owns what during a throughput incident, so you reason about the right tier fast:

Core concepts

Five mental models make every later diagnosis obvious.

There are two layers of partitioning, and conflating them is the root mistake. A logical partition is the set of all items sharing one partition key value; a physical partition is the compute-and-storage unit Cosmos provisions and onto which it hashes logical partitions. You choose the key (and thus the logical partitioning); Cosmos derives the physical partition count. Every ceiling lives on one of these two layers, and “I gave it more RU/s and it still throttles” is always a confusion between them.

The two numbers to internalize: 20 GB and 10,000 RU/s. A logical partition is hard-capped at 20 GB of storage (raw data plus index) — a ceiling you cannot raise. A physical partition serves up to 10,000 RU/s of throughput and up to 50 GB of storage. Because a logical partition never spans more than one physical partition, a single hot key value can never exceed 10,000 RU/s, no matter what you provision on the container. Internalize this one rule and most incidents explain themselves.

The physical partition count is derived, not chosen. Cosmos takes the maximum of two requirements — throughput and storage — and provisions that many physical partitions:

physical partitions = ceil( max(
    provisioned_RU / 10000,
    total_storage_GB / 50
))

Two consequences explain most throughput tickets: (1) provisioning 100,000 RU/s on a container with one hot key does nothing for that key, because it cannot be split across physical partitions; and (2) throughput is distributed evenly across physical partitions — provision 60,000 RU/s across 6 physical partitions and each gets exactly 10,000 RU/s, even if 5 are idle and 1 is on fire.

The Request Unit is the universal currency. A Request Unit (RU) is Cosmos’s normalized cost for throughput: a 1 KB point read by id costs roughly 1 RU; writes, queries, larger documents and stronger consistency cost more. You provision RU/s (per second), and every operation debits the bucket. Stop estimating the moment you can read the real cost: every response carries x-ms-request-charge and Data Explorer shows it in Query Stats. The single highest-leverage RU optimization after the partition key is the indexing policy — because writes pay to maintain the index.

You cannot change a partition key in place. The partition key is effectively permanent — you migrate to a new container, never alter it on an existing one. This makes the choice the decision to over-invest in, and it makes every real repair a data movement (synthetic key, hierarchical key, or change-feed migration). Plan the escape hatch up front; you will eventually need it.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

The RU & partition limits reference

Before the per-topic detail, here is the lookup table you scan first: the hard numbers that bound every Cosmos design. The non-obvious ones are the per-logical-partition 20 GB ceiling (independent of physical partition size) and the fact that throughput is per container but spent per physical partition.

Three reading notes that save the most time:

Logical vs physical partitions, and the 20 GB ceiling

Cosmos DB has two layers of partitioning, and conflating them is the root of most design mistakes.

A logical partition is the set of all items sharing one partition key value. If your key is /tenantId, every document for tenant-42 lives in one logical partition. Its hard constraints:

• 20 GB of storage (raw data plus index). This is a ceiling you cannot raise.
• All items with that key value are co-located — which is what makes single-partition queries and transactional batch operations cheap.

A physical partition is the actual compute-and-storage unit Cosmos provisions behind the scenes. Cosmos hashes the partition key value and maps each logical partition onto exactly one physical partition. Its constraints:

• Up to ~50 GB of storage per physical partition (newer accounts support larger; treat 50 GB as the planning number).
• Up to 10,000 RU/s of throughput per physical partition.

The number of physical partitions is derived, not chosen — the maximum of the two requirements shown in the formula above. Two consequences explain most “I gave it 50,000 RU/s and it’s still throttling” tickets:

1. A single hot logical partition cannot exceed 10,000 RU/s, because it cannot be split across physical partitions. Provisioning 100,000 RU/s on the container does nothing for one key value receiving all the traffic.
2. Throughput is distributed evenly across physical partitions. If you provision 60,000 RU/s and Cosmos created 6 physical partitions, each gets 10,000 RU/s — even if 5 are idle and 1 is on fire.

The single most important number to internalize: 10,000 RU/s per physical partition, and a logical partition never spans more than one physical partition. Every hot-partition incident is some violation of this rule.

The two layers side by side, every property that differs:

What forces Cosmos to add physical partitions (a split), and what it means for you:

Choosing a partition key

The partition key is effectively permanent — you can only migrate to a new container, never change it in place — so this is the decision to over-invest in. Evaluate every candidate against three properties.

Cardinality. You want many distinct values so Cosmos can spread data across many logical (and therefore physical) partitions. /userId in a system with millions of users is excellent. /country is terrible: a few hundred values, wildly skewed toward your largest markets, each capped at 20 GB and 10,000 RU/s.

Access pattern alignment. The key should match how you read. If 90% of queries filter by customerId, partitioning on /customerId turns those into single-partition queries that touch one physical partition for a fraction of a fan-out’s cost. A query that omits the partition key becomes a cross-partition query, which fans out to every physical partition and bills you for the sum.

Write distribution. Hot logical partitions are usually write problems. Avoid keys that funnel writes:

• Monotonic keys like /date or an incrementing ID concentrate every new write into the “current” partition — the classic append hot spot.
• Status-like keys (/status with values active/closed) skew because most live traffic hits one value.

The heuristic I apply, in order of preference:

When no single field is both high-cardinality and read-aligned, build a synthetic key by concatenating fields, or reach for hierarchical partition keys (covered below). The scoring rubric I score candidates on, so the choice is defensible in review:

The anti-patterns, named, with what actually goes wrong:

Estimating and measuring RU/s

A Request Unit is Cosmos DB’s normalized currency for throughput: a 1 KB point read by id costs roughly 1 RU. Writes, queries, and larger documents cost more. Two activities matter — estimating up front, and measuring in production.

Measure, do not guess. Every response carries the real cost in the x-ms-request-charge header. Stop estimating the moment you can issue a real query against real data.

# Read the request charge for a query using the REST surface via az rest is awkward;
# in practice you read the header from your SDK. With the .NET SDK:
#   response.RequestCharge  ->  double, RUs consumed
# With the Python SDK, the charge is on the client after the call:
#   client.client_connection.last_response_headers['x-ms-request-charge']

In the Data Explorer Query Stats tab, every query shows its Request Charge and Retrieved document count. A query reporting 2.8 RU is fine; one reporting 850 RU on a small container is doing a cross-partition scan or fighting the indexing policy.

For sizing before you have data, the official Cosmos DB capacity calculator translates item size, read/write rates, and consistency level into a baseline RU/s. Rules of thumb worth carrying:

• A 1 KB point read is ~1 RU; a 1 KB create is ~5 RU at default indexing.
• Stronger consistency costs more on reads: Strong and Bounded Staleness reads cost roughly 2× the equivalent Session/Eventual read.
• Indexing every property inflates write cost. Writes pay to maintain the index; trimming it is the highest-leverage write optimization.

When throttled, Cosmos returns HTTP 429 with an x-ms-retry-after-ms header. The SDKs retry automatically up to a configurable limit, but sustained 429s mean you are either under-provisioned overall or — far more often — hammering one physical partition. The per-operation RU costs worth memorizing as a baseline (default indexing, 1 KB item unless noted):

How consistency level and item size move the read cost — both are levers you set:

The 429 retry behavior, and the knobs that govern it:

Hierarchical partition keys for skewed tenants

Multi-tenant systems almost always want to partition by /tenantId for query locality, but real tenant distributions are power-law: a handful of tenants generate most of the data and traffic. A single big tenant blows past 20 GB or saturates its 10,000 RU/s, and /tenantId traps you.

Hierarchical partition keys (also called subpartitioning) solve this by letting you define up to three levels. Cosmos uses the full path to place items, but can still route a query that supplies only a prefix to the right physical partitions.

Define the hierarchy at container creation:

az cosmosdb sql container create \
  --account-name cosmos-platform-prod \
  --resource-group rg-data-platform \
  --database-name events \
  --name telemetry \
  --partition-key-path "/tenantId" "/deviceId" "/sessionId" \
  --partition-key-version 2 \
  --throughput 10000

Now the effective partitioning is tenantId -> deviceId -> sessionId. A whale tenant’s data is spread across many deviceId sub-partitions and is no longer confined to a single logical partition or its 20 GB / 10,000 RU/s ceiling. Crucially, queries keep their efficiency depending on how much of the prefix they supply:

-- Single physical partition: full key supplied
SELECT * FROM c WHERE c.tenantId = 'acme' AND c.deviceId = 'dev-9' AND c.sessionId = 's-1'

-- Targeted subset: prefix supplied, Cosmos routes to the relevant physical partitions
SELECT * FROM c WHERE c.tenantId = 'acme'

-- Full cross-partition fan-out: prefix NOT supplied
SELECT * FROM c WHERE c.deviceId = 'dev-9'

The middle query is the payoff: you get tenant-scoped reads without ever creating a 20 GB-capped, throughput-capped logical partition for acme. Note that hierarchical keys must be enabled at creation time with partition key version 2; you cannot retrofit them onto an existing single-key container without migrating. In Bicep:

resource telemetry 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2024-05-15' = {
  name: 'telemetry'
  parent: eventsDb
  properties: {
    resource: {
      id: 'telemetry'
      partitionKey: {
        paths: [ '/tenantId', '/deviceId', '/sessionId' ]
        kind: 'MultiHash'      // hierarchical
        version: 2
      }
    }
    options: { throughput: 10000 }
  }
}

How much of the prefix you supply determines the cost — this is the whole reason hierarchical keys beat synthetic keys for multi-tenant reads:

Hierarchical vs synthetic vs single key for the multi-tenant case, decided:

Indexing policy tuning

By default Cosmos DB indexes every property of every document — ad hoc queries are fast on day one, writes are needlessly expensive forever. On write-heavy containers this is the single biggest RU lever after the partition key.

The strategy: index only what you filter, sort, or join on; exclude the rest. Path precedence is resolved by longest match, so the robust pattern is exclude everything, then include the specific paths you query.

{
  "indexingMode": "consistent",
  "automatic": true,
  "includedPaths": [
    { "path": "/customerId/?" },
    { "path": "/status/?" },
    { "path": "/createdOn/?" }
  ],
  "excludedPaths": [
    { "path": "/*" },
    { "path": "/_etag/?" }
  ],
  "compositeIndexes": [
    [
      { "path": "/customerId", "order": "ascending" },
      { "path": "/createdOn", "order": "descending" }
    ]
  ]
}

Two things to understand precisely:

• The /? suffix means “index the scalar value at this path.” The /* wildcard under excludedPaths excludes everything beneath root, which combined with the explicit includedPaths gives a tight allowlist.
• Composite indexes are required for efficient queries that filter on one property and ORDER BY another, or ORDER BY two properties. WHERE c.customerId = @id ORDER BY c.createdOn DESC is far cheaper — or only possible without a full scan — with the composite index above. Property order and sort direction must match the query (or be its exact reverse).

Apply a policy update with the CLI; index transformation runs online in the background:

az cosmosdb sql container update \
  --account-name cosmos-platform-prod \
  --resource-group rg-data-platform \
  --database-name orders \
  --name orders \
  --idx @indexing-policy.json

Trimming a wide-open policy down to a handful of indexed paths routinely cuts create/upsert cost by 30–50% on documents with many properties, because the write no longer maintains dozens of index entries it will never serve a query from. The indexing-policy knobs, end to end:

Which index a query needs — match the query shape to the index type:

What the write actually pays to maintain — why trimming matters on wide documents:

Detecting hot partitions

A hot partition is invisible at the container level — average RU consumption looks healthy while one physical partition sits at 100% throwing 429s. You detect it with partition-scoped metrics, not aggregates.

The key metric is Normalized RU Consumption: the percentage of provisioned RU/s used by the hottest partition in each window. Pinned near 100% while container-level utilization sits at 30% means a hot partition by definition.

In Azure Monitor / Metrics, chart it like this:

// Azure Monitor metric, split by physical partition.
// Metric: NormalizedRUConsumption
// Aggregation: Max
// Split (filter) by: PhysicalPartitionId
//
// In the Metrics blade:
//   Metric        = Normalized RU Consumption
//   Aggregation   = Max
//   Apply splitting on dimension "PhysicalPartitionId"

For log-based analysis, query the throttled requests in Log Analytics if diagnostic settings are routing DataPlaneRequests:

CDBDataPlaneRequests
| where TimeGenerated > ago(1h)
| where StatusCode == 429
| summarize Throttled = count() by PartitionKeyRangeId, bin(TimeGenerated, 5m)
| order by Throttled desc

A single PartitionKeyRangeId dominating the 429 count is the signature of a hot partition. Cross-reference it with PartitionKeyStatistics (available via the SDK’s GetPartitionKeyRangesAsync and storage metrics) to see which key values carry the most data. The triad to confirm a hot partition:

1. Normalized RU Consumption (Max) near 100% on one PhysicalPartitionId.
2. 429s concentrated on one PartitionKeyRangeId.
3. Container-level RU utilization comfortably below provisioned.

The signals and exactly where each lives — open these in order during an incident:

Reading the metric combinations — the decision table for the dashboard:

Remediation: re-partitioning, synthetic keys, migration

You cannot change a partition key in place. Every real fix moves data to a better-keyed container, but the right approach depends on the failure mode.

Synthetic / composite keys address low cardinality. If you were forced onto /status or /region, redefine the key as a computed field on each document that combines a high-cardinality value with the natural one:

# Stamp a synthetic partition key on write to spread load.
# Combine a meaningful prefix with a bucketed suffix for high cardinality.
import hashlib

def synthetic_pk(tenant_id: str, entity_id: str, buckets: int = 100) -> str:
    suffix = int(hashlib.sha256(entity_id.encode()).hexdigest(), 16) % buckets
    return f"{tenant_id}-{suffix:03d}"

doc["pk"] = synthetic_pk(doc["tenantId"], doc["id"])
# Container partition key path is "/pk".
# Reads for a tenant must now fan across the 100 buckets, so prefer this only
# when point reads dominate, or use hierarchical keys instead for query locality.

The trade-off is explicit: synthetic suffixes spread writes well but turn tenant-scoped reads into a fan-out across the buckets. When you need both write spread and read locality, hierarchical partition keys are the better tool — the default for the multi-tenant case.

Container migration is the path when the key itself is wrong. There is no in-place repartition; you create a new container with the correct key (or hierarchy and indexing policy) and copy the data:

• Change feed is the production-safe mechanism. Stand up the new container, run an Azure Function or self-hosted change-feed processor to drain the source’s change feed into the destination, then cut writes over once it has caught up — a live, resumable backfill with no maintenance window.
• For one-shot bulk copies, the Azure Cosmos DB Spark connector or desktop Data Migration tool moves data quickly, but you still need the change feed to capture writes that land during the copy.

Always provision the destination with high RU/s during the backfill (bulk ingestion is throughput-bound) and dial it back once steady-state. The remediation options matched to the failure mode:

The migration mechanisms compared, so you pick the right tool:

The cutover runbook as a checklist of phases:

Autoscale vs manual throughput

The throughput mode shapes both your bill and your resilience to spikes.

Manual throughput pins a fixed RU/s. You pay for that ceiling 24/7 whether you use it or not — correct only for steady, predictable workloads you can size tightly.

Autoscale sets a maximum and instantly scales between 10% and 100% of it based on load, billing per hour for the highest RU/s reached that hour. Autoscale costs 1.5× the manual rate per RU, so the break-even is roughly 66% average utilization: below that, autoscale is cheaper because you avoid paying for idle headroom; above it, a well-sized manual setting wins.

# Create a container with autoscale: max 40,000 RU/s, floor is automatically 4,000 (10%)
az cosmosdb sql container create \
  --account-name cosmos-platform-prod \
  --resource-group rg-data-platform \
  --database-name orders \
  --name orders \
  --partition-key-path "/customerId" \
  --max-throughput 40000

# Convert an existing manual container to autoscale
az cosmosdb sql container throughput migrate \
  --account-name cosmos-platform-prod \
  --resource-group rg-data-platform \
  --database-name orders \
  --name orders \
  --throughput-type autoscale

Two operational nuances:

• Autoscale does not save you from a hot partition. The 10,000 RU/s per-physical-partition cap applies to autoscale exactly as to manual. Autoscale absorbs aggregate spikes; it does nothing for a single saturated key.
• Burst capacity lets a physical partition temporarily exceed its provisioned share by drawing on idle RU/s accumulated over the prior 5 minutes (up to ~3,000 RU/s). It smooths short bursts on otherwise-cool partitions, but it is a buffer, not a fix for sustained skew.

Manual vs autoscale, every axis that decides it:

Throughput provisioning scope — where you attach RU/s changes everything:

The decision table for picking a mode:

Architecture at a glance

The diagram traces a request as it actually flows through Cosmos DB, then maps each throughput failure onto the exact hop where it bites. Read it left to right. An app with the SDK issues a query or write in direct mode (ports 10250–10256) and reads back the real x-ms-request-charge. The request reaches the gateway / control plane, which hashes the partition key and consults its address cache to map the logical key value to a physical partition (PKRange) — badge 2 lands here, because a query that omits the partition key cannot be routed and instead fans out to every partition. In the physical partitions zone you can see the whole disease: one hot PKRange pinned at 100% Normalized RU and returning 429 with retry-after, sitting right next to a cool PKRange below 30% with idle headroom it cannot lend. Badge 1 marks the hot partition; badge 3 marks the trap of provisioning more RU/s, which splits evenly and never rescues the one saturated key.

The index + throughput zone shows the two container-level levers — the indexing policy (exclude /*, include only queried paths, add composites) carries badge 4, the write-RU tax; and autoscale (max 40k, 10–100%, 1.5× the manual rate) which absorbs aggregate spikes but not a hot key. Finally the repartition path is the escape hatch you design up front: because there is no in-place key change (badge 5), you stand up a new container with the right key and a lean index, drain the source’s change feed with a Function at high backfill RU/s, and cut writes over behind a flag. The whole method is on the diagram: localize the symptom to a hop, read the badge, run the named confirm, apply the fix — and notice that “more RU/s” only ever helps the one case (badge 3’s opposite: all partitions genuinely hot).

Real-world scenario

Lumio Commerce, a SaaS marketplace platform, runs its order-management service on Azure Cosmos DB for NoSQL: a transactions container partitioned on /merchantId — reasonable, since nearly every query is merchant-scoped (WHERE c.merchantId = @id AND c.createdOn > @since). It is provisioned at autoscale max 50,000 RU/s in Central India, holds ~600 GB across thousands of mid-size merchants, and costs about ₹95,000/month. The platform team is five engineers; the design held up beautifully for two years.

The incident began on a Friday. Lumio had onboarded a marketplace customer — a single large retailer — whose Black Friday traffic was roughly 40× their next-largest merchant. At 18:02 the order-service dashboard lit up with HTTP 429 on checkout writes: about 9% failing, climbing to 28% by 18:15. The on-call engineer’s reflex: raise the autoscale max from 50,000 to 100,000 RU/s. The 429 rate did not move. Second reflex: open a support ticket assuming a platform issue. Forty minutes in, checkout revenue for the whale merchant was visibly dropping and the bridge was full.

The breakthrough came from the right metric. Container-level Total Request Units showed overall utilization at ~22% — the container was nowhere near its ceiling. But NormalizedRUConsumption with Max aggregation, split by PhysicalPartitionId, showed exactly one physical partition pinned at 100%, and CDBDataPlaneRequests in Log Analytics showed the 429s concentrated on a single PartitionKeyRangeId. That was the whole story: all of the whale merchant’s traffic hashed to one logical partition (merchantId = 'bigretail'), which lives on one physical partition, which is capped at 10,000 RU/s — and no amount of container-level RU/s can split one key value across partitions. The 100,000 RU/s did nothing because the constraint was per-partition, not aggregate.

The constraint was unmovable in place: you cannot change a partition key on an existing container, a single logical partition cannot be split, and they could not take a maintenance window during the holiday peak. The fix was a migration to hierarchical partition keys, /merchantId then /orderId. They created a new container with partition key version 2, set a tight indexing policy (excluding the large lineItems blob they never filtered on — a 30-field document trimmed to four indexed paths), provisioned 80,000 RU/s for the backfill, and drained the source’s change feed into it with an Azure Function so the copy was live and resumable. They cut writes over behind a feature flag once the processor caught up, then dropped to autoscale max 50,000.

az cosmosdb sql container create \
  --account-name cosmos-orders-prod \
  --resource-group rg-orders \
  --database-name commerce \
  --name transactions_v2 \
  --partition-key-path "/merchantId" "/orderId" \
  --partition-key-version 2 \
  --idx @lean-indexing.json \
  --max-throughput 80000

The whale merchant’s orders now spread across thousands of orderId sub-partitions instead of one logical partition; the per-partition ceiling stopped binding, and merchant-scoped reads stayed single-partition because queries still supplied the /merchantId prefix. The next sale ran at the same load with zero sustained 429s, checkout write p99 fell from seconds-of-retry to ~12 ms, and steady-state RU spend actually dropped because the lean index cut write cost on a container doing millions of order writes a day — Lumio landed at ₹88,000/month, below where they started. The lesson on the wall: “A 429 with the container at 22% is a partition problem, not a provisioning problem. Split the key by PhysicalPartitionId before you touch the RU slider.”

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

The hash-partitioned, RU-metered model both causes this class of problem and makes it diagnosable. Weigh it honestly:

The model is right for high-scale, low-latency, globally distributed document workloads where you can design the access pattern up front and key to it. It bites hardest on skewed multi-tenant data (whale tenants), monotonic ingestion (append hot spots), and write-heavy containers left on the default index. Every disadvantage is manageable — but only if you know it exists before you pick the key, which is the point of this article.

Hands-on lab

Create a container, measure real RU cost, reproduce an expensive cross-partition query, fix it with the partition key and a trimmed index, and tear it down — all free-tier-friendly (Cosmos DB offers a free tier: the first 1,000 RU/s and 25 GB are free per account). Run in Cloud Shell (Bash).

Step 1 — Variables and resource group.

RG=rg-cosmos-lab
LOC=centralindia
ACCT=cosmoslab$RANDOM     # globally-unique account name
DB=shop
CONT=orders
az group create -n $RG -l $LOC -o table

Step 2 — Create a free-tier account (first 1000 RU/s + 25 GB free).

az cosmosdb create -n $ACCT -g $RG \
  --default-consistency-level Session \
  --enable-free-tier true -o table
az cosmosdb sql database create -a $ACCT -g $RG -n $DB -o table

Expected: an account row and a database. Free tier means this lab costs ₹0 if you stay under the free RU/s and delete promptly.

Step 3 — Create a container keyed on /customerId with 400 RU/s.

az cosmosdb sql container create -a $ACCT -g $RG -d $DB -n $CONT \
  --partition-key-path "/customerId" --throughput 400 -o table

Step 4 — Insert a few items and read the request charge. In Data Explorer (portal), open the container, New Item, and insert:

{ "id": "o-1", "customerId": "cust-7", "status": "active", "createdOn": "2026-06-01", "total": 4200 }

Open the Query Stats tab and run a single-partition query — note the Request Charge (single digits):

SELECT * FROM c WHERE c.customerId = "cust-7"

Step 5 — Reproduce an expensive cross-partition query. Run a query that omits the partition key and watch the charge climb (it fans out):

SELECT * FROM c WHERE c.status = "active"

Compare the two Request Charges in Query Stats: the keyed query touches one partition; the status query fans out. On a small lab container the gap is modest, but the mechanism is the point — at scale this is the difference between 3 RU and 800 RU.

Step 6 — Trim the indexing policy and confirm. Replace the container’s index policy (Data Explorer → Settings → Indexing Policy) with an allowlist:

{
  "indexingMode": "consistent",
  "automatic": true,
  "includedPaths": [ { "path": "/customerId/?" }, { "path": "/status/?" } ],
  "excludedPaths": [ { "path": "/*" } ]
}

Save (transformation runs online). Confirm the key and throughput from the CLI:

az cosmosdb sql container show -a $ACCT -g $RG -d $DB -n $CONT \
  --query "resource.{pk:partitionKey.paths, indexMode:indexingPolicy.indexingMode}" -o json
az cosmosdb sql container throughput show -a $ACCT -g $RG -d $DB -n $CONT \
  --query "resource.throughput" -o tsv

Expected: pk is ["/customerId"], indexMode is consistent, throughput 400.

Validation checklist. You created a keyed container on free tier, read the real RU charge from Query Stats, saw a keyed query stay single-partition while a non-keyed one fanned out, and trimmed the index to an allowlist. No application code required — exactly the point. The lab steps mapped to what each proves:

Cleanup (avoid lingering charges).

az group delete -n $RG --yes --no-wait

Cost note. On free tier this lab is ₹0 if you stay within 1,000 RU/s and delete the resource group. Without free tier, a 400 RU/s container is a few rupees per hour; deleting the group stops everything.

Common mistakes & troubleshooting

This is the playbook — the part you bookmark. First as a scannable table you can read when the dashboard is red, then the same entries with the full confirm-command detail underneath.

The expanded form, with the full reasoning for the entries that bite hardest:

1. 429 under load while container utilization sits below 50%. Root cause: A hot logical partition — one key value taking the traffic — saturating its single physical partition’s 10,000 RU/s cap. Confirm: Metrics → NormalizedRUConsumption, aggregation Max, split on dimension PhysicalPartitionId: one partition near 100% while others idle. Corroborate with 429s concentrated on one PartitionKeyRangeId in CDBDataPlaneRequests. Fix: Spread the key — hierarchical partition keys (version 2) for multi-tenant locality, or a synthetic bucketed key if point reads dominate. Raising provisioned RU/s does nothing for a single key.

2. A query reports hundreds of RU on a small container. Root cause: A cross-partition query (the partition key is not in the WHERE) fanning out to every physical partition, or a missing index forcing a scan. Confirm: Data Explorer → Query Stats → Request Charge (hundreds) and Retrieved document count; check whether the query supplies the partition key and whether the filtered/sorted path is indexed. Fix: Add the partition key (or the hierarchical prefix) to the filter; index the filtered path; project fewer fields. If the access pattern fundamentally omits the key, the key is wrong.

3. You raised provisioned RU/s and it still throttles on one key. Root cause: Throughput is distributed evenly across physical partitions; a single logical partition can never exceed 10,000 RU/s, so container-level RU/s is irrelevant to one hot key. Confirm: Container Total Request Units far below provisioned while one PartitionKeyRangeId dominates 429s. Fix: Re-key (hierarchical/synthetic) and migrate; do not keep buying RU/s.

4. Create/upsert RU is unexpectedly high. Root cause: The default indexing policy indexes every property, so each write maintains dozens of index entries — most never serve a query. Confirm: Container → Settings → Indexing Policy shows includedPaths of /*; write x-ms-request-charge is high on wide documents. Fix: Exclude /* and include only queried paths; the transform runs online. Expect 30–50% lower write RU on 40-field documents.

5. An ORDER BY query is very expensive or fails outright. Root cause: No composite index for a query that filters on one path and sorts on another (or sorts on two paths). Confirm: Query Stats shows high RU; the indexing policy has no compositeIndexes entry matching the query’s paths and directions. Fix: Add a composite index (filterPath ASC, sortPath DESC) matching the query (or its exact reverse).

6. Writes for one key value start failing around 20 GB. Root cause: That logical partition hit the 20 GB ceiling — a hard limit per key value you cannot raise. Confirm: PartitionKeyStatistics (SDK / storage metrics) shows one key value’s SizeInKB near 20 GB. Fix: Re-key to spread that value across more logical partitions (hierarchical or synthetic) via migration.

7. You cannot change the partition key. Root cause: The partition key is permanent on an existing container by design. Confirm: az cosmosdb sql container show --query "resource.partitionKey" returns the fixed key; there is no update path for it. Fix: Create a new container with the correct key and drain the change feed into it; cut over behind a flag.

8. The autoscale bill is higher than the load seems to justify. Root cause: Autoscale costs 1.5× the manual rate, so below ~66% average utilization you pay a premium for elasticity you may not need. Confirm: Metrics show low average utilization while throughput type is autoscale. Fix: For steady workloads above ~66% util, switch to manual at a tightly sized RU/s.

9. A monotonic key creates an append hot spot. Root cause: /date or an incrementing id funnels every new write into the “current” partition. Confirm: NRU and 429s concentrate on the newest partition only. Fix: Use a high-cardinality key and a range index for time queries, or a bucketed synthetic key that spreads the write across N buckets.

10. Tenant reads got slower after a hot-partition “fix”. Root cause: A synthetic bucketed key spread writes but turned tenant-scoped reads into a fan-out across the buckets. Confirm: Reads that were single-partition now touch many partitions; per-query RU rose. Fix: Use hierarchical partition keys (prefix routing keeps tenant reads local) instead of bucketing when you need read locality.

11. The SDK shows no 429 but latency spikes under load. Root cause: The SDK is silently retrying 429s with backoff (default up to 9 attempts), so callers see latency instead of errors. Confirm: x-ms-request-charge looks fine but retry/latency telemetry is high; check CDBDataPlaneRequests for the underlying 429s. Fix: Treat it as a hot partition (re-key); tune retry options so the masking is visible in your metrics.

12. Reads cost twice what you expected. Root cause: Strong or Bounded Staleness consistency costs roughly 2× the RU of Session/Eventual on reads. Confirm: Check the account’s default consistency level; compare RU for the same read at different levels. Fix: Use Session (the default) or Eventual where the workload tolerates it; reserve Strong for the operations that truly need it.

Best practices

• Pick the key for cardinality, read alignment, and write spread — in that order of scrutiny. The obvious key (/tenantId, /status, /date) is often the wrong one; profile your top five queries before you commit.
• Default to hierarchical partition keys for multi-tenant data. Power-law tenant distributions are the norm, not the exception; /tenantId → /entityId spreads whales while keeping prefix reads local. Set it at creation — you cannot retrofit it.
• Measure RU, never estimate past first data. Read x-ms-request-charge / Data Explorer Query Stats on every hot query; a query in the hundreds of RU is a design bug, not a fact of life.
• Always put the partition key in the WHERE clause of high-volume queries. Omitting it fans out to every partition and bills the sum. Align the key to the read so this is natural.
• Exclude /* and include only queried paths. Default full-property indexing is the biggest write tax; trim it and add composite indexes for filter + ORDER BY. This is the highest-leverage write optimization after the key.
• Detect hot partitions with NormalizedRUConsumption (Max) split by PhysicalPartitionId. Container-level utilization lies; the per-partition Max is the only honest signal.
• Choose throughput mode by the ~66% break-even. Autoscale below it (avoid paying for idle), manual above it (avoid the 1.5× premium). Re-evaluate as the load shape changes.
• Never answer a hot-partition 429 with more RU/s. Throughput splits evenly; one key caps at 10,000 RU/s regardless. Re-key instead.
• Design the migration escape hatch up front. Document a change-feed re-partition path so that when the key needs to change, it is a runbook, not a research project.
• Provision high RU/s for backfills, then dial back. Bulk ingestion is throughput-bound; size the destination generously during a migration and reduce it at steady state.
• Keep items small and consistency as weak as correctness allows. RU scales with bytes processed and roughly doubles for Strong reads; both are levers you control.

The metrics and alerts worth wiring before the next incident — leading indicators, not the lagging “writes failing”:

Security notes

• Use managed identity and RBAC, not keys, for the data plane. Cosmos supports Microsoft Entra ID authentication with data-plane RBAC roles (Cosmos DB Built-in Data Reader / Data Contributor). Assign the app’s managed identity a least-privilege role instead of distributing the account’s primary keys, which grant full control and cannot be scoped.
• Disable key-based auth where you can. With Entra auth in place, set disableLocalAuth: true so the powerful primary/secondary keys cannot be used at all — eliminating the highest-value secret to leak.
• Lock the network path. Use Private Endpoints so the account is reachable only from your VNet, and disable public network access. See Azure Private Endpoint vs Service Endpoint for the routing choice; combine with IP firewall rules for any remaining public access.
• Store any remaining secrets in Key Vault. If you must use connection strings (e.g. for a legacy SDK), keep them in Azure Key Vault referenced by managed identity, never in app settings or code. See Azure Key Vault: Secrets, Keys & Certificates.
• Encryption is on by default; bring your own key if required. Data is encrypted at rest with service-managed keys; for regulatory needs configure customer-managed keys (CMK) via Key Vault.
• Scope data-plane RBAC to the right resource. Entra data-plane roles can be scoped to an account, database, or container — grant a service access only to the containers it needs, not the whole account.
• Audit with diagnostic logs. Route DataPlaneRequests and control-plane logs to Log Analytics; the same CDBDataPlaneRequests table you use for hot-partition detection is your access audit trail.

The security controls and what each one buys you — secure and resilient pull together here:

Cost & sizing

The bill drivers and how they interact with the design:

• Provisioned RU/s dominates the bill — you pay per 100 RU/s per hour regardless of how much you use (manual) or up to the hourly peak (autoscale). Right-sizing throughput and trimming per-operation RU (lean index, small items, weaker consistency) are the two levers that move the number.
• Manual vs autoscale is the ~66% break-even. Autoscale’s 1.5× rate is worth it for spiky/idle workloads (you avoid paying for headroom); above ~66% average utilization a tightly sized manual setting is cheaper. Getting this wrong is a common, silent overspend.
• Storage is billed per GB-month (data + index), so a lean index also shrinks storage cost, not just write RU. Excluding a large unqueried blob subtree cuts both.
• The free tier gives the first 1,000 RU/s and 25 GB free per account — enough for dev/test and small production. Serverless mode bills per RU consumed with no provisioning floor, ideal for intermittent workloads.
• Multi-region and stronger consistency add cost: each additional region multiplies provisioned RU/s (and write regions for multi-write), and Strong/Bounded Staleness reads cost ~2× — design distribution and consistency deliberately. See Cosmos DB Multi-Region Writes & Conflict Resolution.

A rough monthly picture for a single-region production container, before any multi-region multiplier:

Sizing heuristics worth carrying:

Interview & exam questions

1. A container is throwing 429 while its overall RU utilization is only 25%. What’s happening and how do you confirm? A hot logical partition — one key value taking the traffic — has saturated its single physical partition’s 10,000 RU/s cap; container-level utilization is low because the other partitions are idle. Confirm with NormalizedRUConsumption, aggregation Max, split by PhysicalPartitionId (one near 100%), and 429s concentrated on one PartitionKeyRangeId. The fix is to re-key (hierarchical/synthetic), not to add RU/s.

2. Why does provisioning more RU/s not fix a hot partition? Throughput is distributed evenly across physical partitions, and a single logical partition can never span more than one physical partition — so it is capped at 10,000 RU/s no matter what the container is provisioned to. Adding RU/s helps only when all partitions are genuinely hot (true under-provisioning).

3. What are the two hard ceilings every partition design must respect? A logical partition caps at 20 GB of storage (a single key value cannot exceed it), and a physical partition serves at most 10,000 RU/s. Because a logical partition lives on exactly one physical partition, a single key value is bounded by both — the root of every hot-partition incident.

4. How do you choose a partition key? Evaluate candidates on cardinality (many distinct values to spread across partitions), read alignment (the key appears in your highest-volume query filters, avoiding cross-partition fan-out), and write spread (no monotonic or status-like funnelling). The best key is high-cardinality and in the filter of most reads; when none exists, use a synthetic or hierarchical key.

5. What is a cross-partition query and why is it expensive? A query that does not include the partition key in its filter; Cosmos cannot route it to one partition, so it fans out to every physical partition and bills you the sum of their charges. Confirm via Query Stats Request Charge. Fix by adding the partition key (or hierarchical prefix) to the WHERE.

6. When do hierarchical partition keys help, and what’s the catch? They help multi-tenant / skewed workloads: defining up to three levels (e.g. /tenantId → /deviceId) spreads a whale tenant across many sub-partitions while a query supplying the prefix still routes to the right partitions (read locality preserved). The catch: they must be set at container creation with partition key version 2 — you cannot retrofit them without migrating.

7. How do you cut write RU without changing the partition key? Trim the indexing policy: exclude /* and include only the paths you filter, sort, or join on, then add composite indexes for filter + ORDER BY. On wide documents this cuts create/upsert RU by 30–50% because the write stops maintaining index entries no query uses. The transform runs online.

8. Manual vs autoscale — how do you decide? Autoscale costs 1.5× the manual rate but scales 10–100% of a max and bills the hourly peak. The break-even is roughly 66% average utilization: below it, autoscale is cheaper (no paying for idle headroom); above it, a tightly sized manual setting wins. Neither rescues a single hot partition.

9. You must change a partition key that’s wrong. What’s the production-safe path? There is no in-place repartition. Create a new container with the correct key (and a lean index), provision high RU/s for the backfill, drain the source’s change feed with an Azure Function (live, resumable, no maintenance window), and cut writes over behind a feature flag once the processor catches up. Then dial throughput back.

10. What does the x-ms-request-charge header tell you, and why does it matter? It reports the exact RU cost of that operation. It matters because you should measure, not estimate — a query reporting hundreds of RU is doing a cross-partition fan-out or fighting the index, which you can fix; a single-partition point read should be ~1 RU. Reading it on your top queries is the fastest cost optimization.

11. How does consistency level affect cost? Strong and Bounded Staleness reads cost roughly 2× the RU of Session or Eventual reads (and Strong constrains multi-region write topologies). Use the weakest level your correctness allows — Session (the default) suits most workloads — and reserve Strong for the operations that truly need it.

12. What is burst capacity and when does it save you? Burst capacity lets a physical partition temporarily exceed its provisioned share by drawing on idle RU/s accumulated over the prior ~5 minutes (up to ~3,000 RU/s). It smooths short bursts on otherwise-cool partitions — it is a buffer, not a fix for a sustained hot partition or chronic under-provisioning.

These map to DP-420 (Designing and Implementing Cloud-Native Applications Using Microsoft Azure Cosmos DB) — partitioning, throughput, indexing, change feed, consistency — and to AZ-204 (Developer Associate) — develop solutions that use Azure Cosmos DB (partition keys, request units, consistency). A compact cert-mapping for revision:

Quick check

1. A container returns 429 while its container-level RU utilization is 25%. What is the cause, and which metric (with what aggregation and split) confirms it?
2. You provision 100,000 RU/s and one key still throttles. Why doesn’t the extra throughput help?
3. True or false: you can change a container’s partition key in place as long as you do it during a maintenance window.
4. A query reports 850 RU on a small container. Name the two most likely causes and the one place you’d look to confirm.
5. Your multi-tenant app keys on /tenantId and one whale tenant just blew past 10,000 RU/s. What is the recommended fix, and what’s the one constraint on applying it?

Answers

1. A hot logical partition — one key value taking the traffic — has saturated its single physical partition’s 10,000 RU/s cap; the container looks underused because the other partitions are idle. Confirm with NormalizedRUConsumption, aggregation Max, split by PhysicalPartitionId (one near 100%), corroborated by 429s on one PartitionKeyRangeId.
2. Throughput is distributed evenly across physical partitions, and a single logical partition lives on exactly one physical partition, capped at 10,000 RU/s. Container-level RU/s never raises that per-partition ceiling for one key value — only a better key spreads the load.
3. False. The partition key is permanent on an existing container; there is no in-place change, maintenance window or not. You create a new container with the correct key and migrate (change feed), then cut over.
4. Cross-partition fan-out (the query omits the partition key, so it bills the sum of all partitions) or a missing index (forcing a scan). Confirm in Data Explorer → Query Stats → Request Charge and Retrieved document count, and check whether the query supplies the PK and whether the filtered path is indexed.
5. Migrate to hierarchical partition keys (e.g. /tenantId → /orderId) so the whale spreads across many sub-partitions while prefix queries keep tenant read locality. The constraint: hierarchical keys must be set at container creation with partition key version 2 — you cannot retrofit them, so it requires a change-feed migration to a new container.

Glossary

• Logical partition — the set of all items sharing one partition key value; hard-capped at 20 GB of storage and bounded by its physical partition’s throughput.
• Physical partition (PKRange) — the compute-and-storage unit Cosmos provisions and onto which it hashes logical partitions; serves up to 10,000 RU/s and ~50 GB. Count is derived, not chosen.
• Partition key — the property (/path) Cosmos hashes to place each item; effectively permanent on a container.
• Request Unit (RU) — Cosmos’s normalized currency for throughput; a 1 KB point read by id is ~1 RU, writes and queries cost more.
• x-ms-request-charge — the response header reporting the exact RU cost of an operation; the source of truth for cost.
• Cross-partition query — a query whose filter omits the partition key; it fans out to every physical partition and bills the sum.
• Hierarchical partition key (subpartitioning) — up to three key levels (version 2) that spread a skewed value across sub-partitions while prefix queries stay local; set at creation only.
• Synthetic key — a computed partition key (e.g. value + hashed bucket) stamped on each item to raise cardinality; spreads writes but can lose read locality.
• Indexing policy — the container JSON declaring which paths are indexed (includedPaths/excludedPaths) and any composite indexes; over-broad indexing inflates write RU.
• Composite index — a multi-path index required for efficient filter + ORDER BY or multi-ORDER BY queries; order and direction must match the query.
• NormalizedRUConsumption — the Azure Monitor metric giving the percentage of provisioned RU/s used by the hottest partition; the best hot-partition signal when read with Max aggregation split by PhysicalPartitionId.
• PartitionKeyRangeId — the identifier of a physical partition’s key range as seen in data-plane logs; a single one dominating 429s signals a hot partition.
• Autoscale — throughput mode scaling instantly between 10% and 100% of a configured max, billed at the hourly peak and 1.5× the manual rate.
• Burst capacity — temporary headroom letting a physical partition exceed its share by drawing on idle RU/s accrued over ~5 minutes (up to ~3,000 RU/s); a buffer, not a fix.
• Change feed — the ordered log of inserts/updates per container; the production-safe mechanism for live, resumable re-partition migrations.
• HTTP 429 (Too Many Requests) — the throttling response carrying x-ms-retry-after-ms; sustained 429 means under-provisioning or, more often, a hot partition.

Next steps

You can now design a partition key, measure and shrink RU, detect a hot partition, and repair a skewed container. Build outward:

• Next: Cosmos DB Multi-Region Writes & Conflict Resolution — layer global distribution and multi-write conflict handling on top of the partitioning you designed here.
• Related: Database Selection 101: SQL vs NoSQL — When to Use What — the decision upstream of ever choosing Cosmos DB at all.
• Related: Azure Monitor & Application Insights for Observability — go deep on the metrics and KQL that power the hot-partition detection in this article.
• Related: Event Hubs, Kafka Capture & Stream Analytics: Throughput & Scaling — the ingestion firehose that usually feeds a Cosmos container.
• Related: Troubleshooting Azure SQL Database: Connectivity, Timeouts, Throttling & Blocking — the relational counterpart when a workload argues for SQL over NoSQL.