Cosmos DB 429s Explained: Diagnosing RU Throttling, Hot Partitions and Retry Tuning

Your API latency dashboard spikes, a queue backs up, and the logs fill with one status code: 429 Request rate too large. Azure Cosmos DB — Microsoft’s globally distributed, horizontally partitioned NoSQL database — is telling you something precise, but easy to misread. A 429 is not an error in the “something is broken” sense; it is flow control: “you asked for more work in this one-second window than the throughput you provisioned, so I’m rejecting the surplus and telling you exactly how long to wait.” The maddening part is that the same 429 can mean two opposite things — you genuinely under-provisioned, or you provisioned plenty but one partition is taking all the heat — and the fix for each is the opposite of the other.

This is the diagnostic playbook. We treat “request rate too large” not as one bug but as a small family of root causes — a true throughput shortfall, a hot partition, a single expensive query, a write amplified by indexing, or a client that retried badly — each with a signal you confirm with a metric or header, and a specific fix. You will learn the two numbers that end most of these arguments in seconds: the Request Charge (how many Request Units an operation cost) and the Retry-After the server hands back with every 429. You will learn why throwing more RU/s at a hot partition burns money without fixing anything, why the SDK already retried before you ever saw the exception, and how to tell — from Normalized RU Consumption alone — whether your problem is the whole container or one slice of it.

By the end you will stop guessing: when 429s appear you will know whether to raise throughput, switch to autoscale, fix a partition key, cap a query, or tune a retry policy — confirming which within minutes using metrics and headers that were there all along. Every diagnosis comes with the exact metric, header or az command that proves it, plus CLI and Bicep for the fix. Because you will reach for this mid-incident, the playbook, headers, metrics and limits are all laid out as scannable tables.

What problem this solves

Cosmos DB sells you throughput in Request Units per second (RU/s), not CPU or IOPS. Every operation — point read, query, insert, replace — costs a number of RUs, and you provision a budget per second. Exceed it within a one-second window and Cosmos DB rate-limits the surplus with HTTP 429. This is good design: a hard, predictable cost ceiling that protects the service from a runaway client. But it means “the database is throwing errors” is almost never the right framing — it’s doing exactly what you paid for; the question is whether you budgeted correctly and spent the budget evenly.

What breaks without this understanding: a team sees 429s, panics, and doubles the provisioned RU/s — sometimes several times — watching the bill climb while the 429s barely move, because the real cause was a hot partition where most traffic hit one key. Or they disable the SDK’s retries to “see the errors faster” and turn graceful, invisible backoff into a flood of user-facing 500s. Or they never read the x-ms-request-charge header and so never learn that one cross-partition ORDER BY query costs 380 RUs while the point read next to it costs 1 — and that the query, called in a loop, is the entire problem.

Who hits this: nearly everyone running Cosmos DB at volume. It bites hardest on write-heavy workloads (writes cost far more RUs than reads, and indexing amplifies them), multi-tenant designs where one tenant becomes a hot partition, batch/ingest jobs that hammer a container in bursts, and teams who set a fixed RU/s once and never revisited it as traffic grew or skewed. The fix is rarely “buy more RU/s” — it is “find out whether you’re short on budget or spending it unevenly, and which operation is expensive, then fix that.”

To frame the field before the deep dive, here is every flavour of 429 this article covers, the question it forces, and the signal to check first:

Learning objectives

By the end of this article you can:

• Explain the Request Unit model precisely — what an RU is, why a point read is the 1-RU baseline, and why writes and queries cost multiples of it.
• Read a 429 correctly: distinguish a true throughput shortfall from a hot partition from a single expensive operation, and name the metric or header that proves each.
• Use the x-ms-request-charge and x-ms-retry-after-ms response headers (and the SDK diagnostics) to find the exact cost and required wait of any operation.
• Diagnose a hot partition from Normalized RU Consumption split by PartitionKeyRangeId, and explain why adding RU/s does not fix it.
• Choose between manual and autoscale throughput, and between database-level (shared) and dedicated container throughput — and say what each fixes and what it costs.
• Tune the SDK retry policy (MaxRetryAttemptsOnRateLimitedRequests, MaxRetryWaitTimeOnRateLimitedRequests) and explain why disabling retries is almost always wrong.
• Reduce RU consumption at the source: tighten the indexing policy, prefer point reads over queries, page large result sets, and pick a partition key with high cardinality and even access.
• Right-size throughput against an INR/USD bill and the free-tier and serverless options, and know when each is the correct mode.

Prerequisites & where this fits

You should already understand the Cosmos DB basics: a Cosmos account holds databases, which hold containers (the rough equivalent of tables/collections), which hold items (JSON documents). You should know that Cosmos DB is horizontally partitioned — you choose a partition key, and Cosmos DB groups items by it and spreads them across physical partitions — and that you provision throughput (RU/s) at either the database or container level. Comfort running az in Cloud Shell, reading JSON, and basic HTTP status-code literacy are assumed. Examples use the API for NoSQL (the native API); the RU model and 429 behaviour apply across the other APIs too, though some surface 429 as a wire-protocol-specific code.

This sits in the Observability & Troubleshooting track for data services. It assumes you have already chosen Cosmos DB and an API — that decision is upstream, in Choosing a Cosmos DB API: NoSQL vs MongoDB vs Cassandra vs Gremlin vs Table Decoded. It pairs tightly with Azure Monitor and Application Insights: Full-Stack Observability, because every diagnosis here lands in a metric or a logged request charge. If your 429s come from an event-driven pipeline, Cosmos DB Change Feed in Action: Event-Driven Pipelines with Azure Functions and the Processor is the workload that often produces them. And the shape of this problem — a 429 that means “slow down, here’s how long” — is the same one you meet in Taming Azure OpenAI 429s: TPM vs RPM Limits, Quota Math, Retry-After Headers, and Backoff That Works; the retry discipline transfers directly.

A quick map of who confirms what during a 429 incident, so you look in the right place fast:

Core concepts

Five mental models make every later diagnosis obvious.

A Request Unit is a normalized currency for work, and a 429 is the overdraft notice. Cosmos DB doesn’t bill you in CPU, memory or IOPS; it abstracts all of them into one unit — the Request Unit (RU) — and prices every operation in RUs. The reference point: a point read of a 1 KB item costs 1 RU; everything else is measured against that. You provision a budget — RU/s — and may spend up to it each second. Spend over and the surplus operations get 429 Request rate too large, with a header telling you how long to back off. A 429 is not a crash; it is the database refusing to overspend the budget you set.

Throughput is enforced per physical partition, not just per container. This is the most misunderstood fact about Cosmos DB, and the reason “add more RU/s” so often fails. Provision 10,000 RU/s on a container with 5 physical partitions and Cosmos DB does not give any one partition the full 10,000 — it splits the budget roughly evenly, ~2,000 RU/s each. If your access pattern sends most traffic to one partition key (and thus one physical partition), that partition saturates at its ~2,000 RU/s share and 429s while the container as a whole sits at 20% utilization. The container isn’t over budget; one slice is. This is a hot partition, and you can’t buy your way out of it — you fix the access pattern or the partition key.

Every response carries the two numbers that end the argument. Cosmos DB returns the cost of every operation in x-ms-request-charge (the RUs consumed) and, on a 429, the required wait in x-ms-retry-after-ms. The SDK surfaces both. You almost never need a profiler — log the request charge of your hot code paths once and you’ll instantly see which operation is costly. A point read at 1 RU sitting next to a cross-partition query at 350 RU tells you exactly where to spend your attention.

The SDK already retried before you saw the error. By default the Cosmos DB SDK automatically retries rate-limited (429) requests, honouring the server’s Retry-After, up to a configured attempt count and cumulative wait (defaults around 9 retries and 30 seconds). So a 429 that reaches your code as an exception is one the SDK gave up on after exhausting its budget — sustained throttling, not a blip. A burst the SDK absorbed shows up in the 429 metric but never as an app error. “429s in the portal but no errors in my app” is healthy; “429s surfacing as exceptions” means throttling outlasted the retry window — a real shortfall or hot partition.

Writes and queries are expensive; point reads are cheap. RU cost tracks how much data the engine touches and indexes. A point read (by id + partition key) is cheapest — 1 RU for ~1 KB. A query costs more, scaling with items scanned, index use, cross-partition fan-out, and sorting/aggregation. A write (create/replace/upsert) is far costlier because Cosmos DB indexes every write by default — a 1 KB create is ~5–10 RUs and grows with document size and indexed paths. This hierarchy — read ≪ write < query, cross-partition sorted queries priciest — tells you where to look first when RU/s vanish.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

The 429 response: headers, substatus and what the SDK does

Before the per-symptom anatomy, understand the envelope of a 429 — because the answer to “why” is usually printed right on it. A throttled response is an HTTP 429 with a body that names the substatus, and three headers that matter.

The substatus is the part people miss. A plain provisioned-throughput 429 carries substatus 3200 (“RU/s for the container is exceeded”). Others point elsewhere — throttling tied to the partition key reaching its storage or RU limit, or the rarer metadata/control-plane throttling (substatus 1007/1016) when you hammer create-container/offer-update operations rather than data operations. So substatus 3200 is the everyday “you spent over budget” case; a 429 on a management call (creating containers in a loop, changing throughput repeatedly) is a different animal not fixed by raising data-plane RU/s.

Here is the default SDK retry behaviour you are almost certainly relying on without realising it. When a 429 comes back, the .NET/Java/JavaScript/Python SDKs catch it, read x-ms-retry-after-ms, wait, and retry — repeating until they hit either the attempt cap or the cumulative-wait cap, at which point they finally throw the 429 to you.

The reading rule that saves the most confusion: 429s in the metric are not the same as 429s in your logs. The metric counts every throttled response, including ones the SDK silently retried and succeeded; your app only sees an exception when the SDK exhausts its retry budget. A small steady 429 count with zero surfaced errors is a healthy system riding its budget; surfaced 429 exceptions mean sustained throttling that outlasted ~30 seconds of retries.

Anatomy of a true throughput shortfall

The simplest 429: you provisioned 4,000 RU/s, real demand is 6,000 RU/s, and the container is genuinely over budget across the board. Scan the matrix, then read the detail:

Confirm a shortfall with Normalized RU Consumption

The metric that ends most arguments is Normalized RU Consumption: the percentage of available RU/s being used, reported as the maximum across all partitions in the time grain. If this sits near 100% across the whole container, evenly, you have a true shortfall — the budget is genuinely too small. (If it’s near 100% but only on one partition, that’s a hot partition — next section.) Pull it for the container:

# Normalized RU Consumption for a container over the last hour (max %, by container)
ACCT=cosmos-shop-prod
RG=rg-shop-prod
RID=$(az cosmosdb show -n $ACCT -g $RG --query id -o tsv)
az monitor metrics list --resource "$RID" \
  --metric NormalizedRUConsumption \
  --dimension CollectionName \
  --interval PT1M --aggregation Maximum -o table

In the portal: the Cosmos account → Metrics → Normalized RU Consumption, with a filter on your database/container and a split by PartitionKeyRangeId. The shape tells you everything: a flat ceiling at 100% across all ranges = shortfall; a ceiling on one range with the rest idle = hot partition.

Fix a shortfall: raise RU/s or switch to autoscale

If the budget is honestly too small, raise it. With manual throughput you set a fixed number:

# Raise a container's manual throughput to 8000 RU/s (API for NoSQL)
az cosmosdb sql container throughput update \
  --account-name $ACCT --resource-group $RG \
  --database-name shop --name orders \
  --throughput 8000

But if the load is spiky — peaks well above average — fixed RU/s is the wrong tool: you either over-provision for the peak (paying for idle headroom all day) or under-provision and 429 at the peak. Autoscale solves exactly this: set a maximum RU/s and Cosmos DB scales the container instantly between 10% and 100% of it, billing per hour for the highest level reached.

# Convert a container to autoscale with a max of 10000 RU/s (scales 1000–10000)
az cosmosdb sql container throughput migrate \
  --account-name $ACCT --resource-group $RG \
  --database-name shop --name orders \
  --throughput-type autoscale

az cosmosdb sql container throughput update \
  --account-name $ACCT --resource-group $RG \
  --database-name shop --name orders \
  --max-throughput 10000

resource orders 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2024-11-15' = {
  parent: shopDb
  name: 'orders'
  properties: {
    resource: {
      id: 'orders'
      partitionKey: { paths: ['/customerId'], kind: 'Hash' }
    }
    options: {
      // Autoscale: scales between 1,000 and 10,000 RU/s automatically
      autoscaleSettings: { maxThroughput: 10000 }
    }
  }
}

Manual vs autoscale is a genuine trade-off, not a default. Here is the decision laid out:

The rule: steady and predictable → manual (cheaper per RU); spiky, bursty or unknown → autoscale (you stop firefighting 429s at peaks and pay only for what you use, at a modest premium). A flat 24/7 workload on autoscale wastes the premium; a spiky workload on manual either 429s or wastes money on idle headroom.

The shared-database noisy-neighbour shortfall

If you provisioned throughput at the database level (shared across all containers in that database), the RU/s is pooled and split among them. One busy container can consume the lion’s share and starve its siblings, which then 429 even though they are quiet — a noisy-neighbour shortfall. Confirm by comparing per-container RU against the shared offer; fix by moving the busy container to its own dedicated throughput so it can’t crowd the others (a sizing trade-off covered in Cost & sizing).

Anatomy of a hot partition

The expensive misdiagnosis. The container has plenty of RU/s, but one physical partition is saturated while the others idle — so you get 429s at, say, 25% overall utilization. Adding RU/s here is pouring money down a drain. Scan, then read the matching detail:

Confirm a hot partition with per-partition Normalized RU

The proof is the same metric as before — Normalized RU Consumption — but split by PartitionKeyRangeId. A hot partition shows one range pinned near 100% while the rest sit low. That single chart is the difference between “raise RU/s” (wrong here) and “fix the key” (right):

# Split Normalized RU Consumption by physical partition — one range hot = hot partition
az monitor metrics list --resource "$RID" \
  --metric NormalizedRUConsumption \
  --dimension PartitionKeyRangeId \
  --interval PT1M --aggregation Maximum -o table

In the portal, the Insights → Throughput and the Metrics blade both let you split by PartitionKeyRangeId. If one line hugs 100% and the others are flat near the floor, stop thinking about throughput — you have a data-distribution problem. To find which partition-key value is hot, log the partition key on your hottest operations (or use the SDK diagnostics, which name the contacted partition range) and look for the value that dominates.

Why more RU/s doesn’t help — the even-split rule

Recall the core concept: provisioned RU/s is divided roughly evenly across physical partitions. If 80% of traffic hits one partition that holds ~20% of the budget, doubling the container RU/s also only doubles that partition’s share — and you’ve doubled the cost of the whole container to give the hot partition a little more headroom it will quickly exhaust again. The skew, not the budget, is the bug.

Fix a hot partition: the partition key is the lever

You cannot change a container’s partition key in place — you choose it at creation. So the fix is to create a new container with a better key and migrate, or design it right the first time. A good partition key has three properties:

When a natural key is too skewed, build a synthetic / composite key — concatenate a high-cardinality field, or append a hashed suffix (tenantId_<0..N>) to spread one hot tenant across N logical partitions while still fanning-in queries. For a date-bucketed write hotspot (/date = 2026-06-24, so every write today hits one partition), append a bucket suffix (2026-06-24_07). Define the key in Bicep at creation:

resource events 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2024-11-15' = {
  parent: shopDb
  name: 'events'
  properties: {
    resource: {
      id: 'events'
      // High-cardinality, evenly-accessed key — NOT /date or /status
      partitionKey: { paths: ['/deviceId'], kind: 'Hash' }
    }
    options: { autoscaleSettings: { maxThroughput: 10000 } }
  }
}

A hard limit that turns a hot partition into a wall: a single logical partition (all items with the same partition-key value) is capped at 20 GB of storage and at the RU/s a single physical partition can serve. A partition key whose one value grows past 20 GB will eventually fail writes outright, not just 429 — another reason low-cardinality keys are dangerous.

Anatomy of an expensive operation

Sometimes the container and partitions are fine, but one operation costs a fortune in RUs, and called frequently it drains the budget. The fix is to make that operation cheaper, not to buy more throughput. The hierarchy of cost:

Confirm with the Request Charge header

You never have to guess an operation’s cost — it’s on every response. Log RequestCharge for your hot paths:

// .NET — read the RU cost of any operation off the response
ItemResponse<Order> resp = await container.ReadItemAsync<Order>(
    id, new PartitionKey(customerId));
logger.LogInformation("Point read cost {RU} RU", resp.RequestCharge);

FeedIterator<Order> it = container.GetItemQueryIterator<Order>(
    "SELECT * FROM c WHERE c.status = 'OPEN' ORDER BY c.createdAt DESC");
double total = 0;
while (it.HasMoreResults)
{
    FeedResponse<Order> page = await it.ReadNextAsync();
    total += page.RequestCharge;   // sum charge across pages
}
logger.LogInformation("Query cost {RU} RU total", total);

When a query is mysteriously expensive, enable the query metrics / diagnostics — the SDK can return the per-query breakdown (retrieved document count, output document count, index hit ratio, query engine time). A low index hit ratio or a huge retrieved vs output gap means the query is scanning far more than it returns — usually a missing index or a filter the index can’t serve.

Fix expensive operations

Three levers, in order of impact:

1. Prefer point reads. A ReadItem by id + partition key is 1 RU; the same fetch as SELECT * FROM c WHERE c.id = @id costs more because it goes through the query engine. If you have both keys, never query for a single item — read it.
2. Make queries single-partition. Include the partition key in the WHERE clause so the query targets one partition instead of fanning out. A cross-partition query’s cost scales with partition count; a single-partition one doesn’t.
3. Page large results and project narrowly. SELECT only the fields you need (not SELECT *), set a sane MaxItemCount, and consume the continuation token to page rather than pulling thousands of items at once. A query returning 5,000 items in one shot is a 429 waiting to happen.

To read the diagnostics: RequestCharge answers “how many RUs?”; the contacted partition ranges show whether it fanned out (more than one range = cross-partition); and the query metrics’ index hit ratio (well below 1.0) and retrieved-vs-output gap (retrieved ≫ output) mean it’s scanning — usually a missing index or filter.

Anatomy of write amplification and indexing cost

Writes are where RU budgets quietly evaporate, because Cosmos DB indexes every property of every document by default. That automatic indexing is wonderful for ad-hoc queries and brutal for write throughput: every indexed path on every write costs RUs. A workload that’s 70% writes against a document with 50 indexed fields can spend several times the RU/s a read-heavy version would. Four factors drive the cost up: the number of indexed paths (each one adds RU per write — trim the policy), document size (larger items cost more to write and index — keep them lean and split rarely-read blobs out), Replace vs Patch (Replace rewrites and reindexes the whole item — use Patch for small changes), and upsert churn (frequent upserts reindex repeatedly — batch them, and avoid hot-path upserts where a patch fits).

Confirm and fix indexing cost

Read the write’s RequestCharge (as above) — if a create costs 20+ RUs on a small document, indexing is the suspect. The fix is a custom indexing policy that indexes only the paths you actually query, and excludes large or never-filtered properties:

{
  "indexingMode": "consistent",
  "includedPaths": [
    { "path": "/customerId/?" },
    { "path": "/status/?" },
    { "path": "/createdAt/?" }
  ],
  "excludedPaths": [
    { "path": "/*" }
  ]
}

This indexes only customerId, status and createdAt and excludes the rest — cutting write RU cost sharply while keeping your real queries fast. Apply it via az or Bicep:

resource orders 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2024-11-15' = {
  parent: shopDb
  name: 'orders'
  properties: {
    resource: {
      id: 'orders'
      partitionKey: { paths: ['/customerId'], kind: 'Hash' }
      indexingPolicy: {
        indexingMode: 'consistent'
        includedPaths: [ { path: '/customerId/?' }, { path: '/status/?' }, { path: '/createdAt/?' } ]
        excludedPaths: [ { path: '/*' } ]
      }
    }
    options: { autoscaleSettings: { maxThroughput: 10000 } }
  }
}

The trade-off is explicit: excluded paths can’t be efficiently filtered or sorted on without a scan, so index exactly the fields your queries use — no more (wasting write RUs), no less (forcing query scans). For bulk ingest, the SDK’s bulk mode (AllowBulkExecution = true) batches operations to use throughput far more efficiently than one-at-a-time writes, sharply reducing 429s during loads.

Architecture at a glance

There is no single diagram for a 429 because the whole point is that the same status code maps to different failures depending on one fork: is the whole container over budget, or is one partition over budget? Hold this mental model. A request enters the SDK, which sends it to the Cosmos DB gateway/replica owning the relevant physical partition. That partition has its own slice of the container’s provisioned RU/s — roughly the total divided by the number of physical partitions. The engine charges the operation some number of RUs (1 for a small point read, hundreds for a cross-partition sorted query) and debits that partition’s per-second budget. If the partition’s slice is exhausted in this one-second window, it returns 429 with a Retry-After; the SDK catches it, waits, and retries, up to its retry budget.

Now overlay the failure shapes. In a true shortfall, every partition’s slice saturates at once — Normalized RU Consumption is pinned near 100% across all PartitionKeyRangeIds evenly — so the container needs more RU/s or autoscale. In a hot partition, one slice saturates while the rest idle — 100% on a single range, low elsewhere — so extra container RU/s won’t help; you fix the key or access pattern. The third axis is cost per operation: even a well-provisioned, evenly-keyed container 429s if one operation (a fan-out ORDER BY, an over-indexed write) is so expensive a handful blow the per-second slice. So the diagnostic is a three-question funnel: (1) Is the metric high evenly (shortfall) or on one range (hot partition)? (2) What does x-ms-request-charge on the hot path say (is one operation pathologically expensive)? (3) Are 429s surfacing as errors, or is the SDK absorbing them (retry tuning)? Answer those in order and the fix is never ambiguous.

Real-world scenario

Mandara Foods runs an order-management API on Cosmos DB (API for NoSQL): a single orders container at 10,000 RU/s manual, partitioned by /storeId, in Central India, behind Azure Functions. They have ~400 stores, but three flagship metros drive roughly 65% of all orders. Monthly Cosmos spend is about ₹52,000; the platform team is three engineers.

The incident began on a festival-sale morning. From 10:00 the Functions logs filled with 429 Request rate too large, order writes failed back to the queue, and “order failed, please retry” rates climbed. The on-call reflex: raise throughput, 10,000 → 20,000 RU/s. The 429s dropped briefly, then returned within twenty minutes. Second move: 20,000 → 30,000 RU/s. The bill projection tripled, the 429s persisted, and the manager was on the bridge asking why a 3× increase hadn’t fixed it.

The breakthrough was one chart: Metrics → Normalized RU Consumption split by PartitionKeyRangeId. One partition range was pinned at 100% while every other sat under 20% — the whole container was at ~30% utilization. A hot partition: the three flagship stores’ orders, keyed by /storeId, concentrated onto a few physical partitions, and the busiest store’s logical partition was saturating its per-partition slice. Doubling and tripling container RU/s only nudged that one slice’s share — which is why each increase bought twenty minutes then failed again. Two more facts fell out of the diagnostics: per-write RequestCharge was ~18 RU for a ~2 KB order (the policy indexed all 40-odd fields, including a free-text notes blob nobody queried), and a dashboard query ran every few seconds was a cross-partition ORDER BY createdAt at ~240 RU per call.

The fix landed in three parts. That morning: dropped RU/s back to 12,000 (the tripling was pure waste) and switched to autoscale (max 20,000) so genuine peaks were absorbed without babysitting — that couldn’t save the hot partition but stopped the other partitions ever being the bottleneck. That week: a new container keyed on a synthetic composite key (storeId + a 0–9 hash suffix from the order id), spreading even flagship stores across ten logical partitions each, migrated via the change feed; the indexing policy trimmed to the six queried fields (write charge ~18 → ~6 RU); and the dashboard query rewritten to filter by storeId (single-partition, ~12 RU instead of 240). Result: the next sale ran at higher volume with zero surfaced 429 exceptions, Normalized RU even across partitions peaking ~70%, settling at autoscale max 16,000 averaging ₹41,000/month — below the original flat bill. The note on the wall: “A 429 isn’t ‘buy more RU/s’ — it’s ‘open the per-partition chart first.’”

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

The RU-and-429 model — pay for a throughput budget, get rate-limited past it — is what makes Cosmos DB predictable and is also what trips teams up. Weigh it honestly:

The model is right for workloads that want predictable cost and elastic scale and are willing to design a good partition key and watch a couple of metrics. It bites hardest on teams who treat Cosmos DB like a relational database that should silently absorb any load, never look at request charges, and reach for “raise RU/s” as the universal fix. Every disadvantage here is manageable — but only if you know which of the five 429 flavours you face.

Hands-on lab

Provoke a real 429 on a tiny container, read the headers and metric that diagnose it, then fix it two ways — all on the free tier / minimum throughput, deleted at the end. Run in Cloud Shell (Bash), with a small script to generate load.

Step 1 — Variables and a free-tier account.

RG=rg-cosmos-lab
LOC=eastus
ACCT=cosmoslab$RANDOM   # globally-unique
az group create -n $RG -l $LOC -o table

# Free-tier account: first 1000 RU/s + 25 GB free (one per subscription)
az cosmosdb create -n $ACCT -g $RG \
  --enable-free-tier true \
  --default-consistency-level Session -o table

Step 2 — A database and a deliberately tiny-throughput container. Provision the container at the minimum 400 RU/s so you can saturate it easily.

az cosmosdb sql database create -a $ACCT -g $RG -n labdb -o table

az cosmosdb sql container create -a $ACCT -g $RG \
  --database-name labdb --name items \
  --partition-key-path "/pk" \
  --throughput 400 -o table

Expected: a container with throughput = 400 and partition key /pk.

Step 3 — Hammer it and watch 429s appear. Use a short loop of writes far exceeding 400 RU/s. Each ~1 KB create costs ~5–6 RU, so a few hundred rapid writes will overrun the budget:

ENDPOINT=$(az cosmosdb show -n $ACCT -g $RG --query documentEndpoint -o tsv)
KEY=$(az cosmosdb keys list -n $ACCT -g $RG --query primaryMasterKey -o tsv)

# Generate burst load with a small Python script (uses the Cosmos SDK)
pip install azure-cosmos --quiet
python3 - <<'PY'
import os, time
from azure.cosmos import CosmosClient, exceptions
c = CosmosClient(os.environ["ENDPOINT"], os.environ["KEY"])
cont = c.get_database_client("labdb").get_container_client("items")
throttled = 0
for i in range(800):
    try:
        r = cont.create_item({"id": str(i), "pk": "hot", "blob": "x"*900})
        # RU charge is on the response headers
        ru = cont.client_connection.last_response_headers.get("x-ms-request-charge")
    except exceptions.CosmosHttpResponseError as e:
        if e.status_code == 429:
            throttled += 1
            wait = e.headers.get("x-ms-retry-after-ms")
            if throttled <= 3:
                print(f"429! retry-after-ms={wait}")
print(f"Total 429s: {throttled}")
PY

You’ll see 429! lines with a retry-after-ms value and a non-zero total. Note we wrote every item to the same pk = "hot" — a deliberate hot partition.

Step 4 — Confirm it in the metric. Pull Normalized RU Consumption and watch it pin at 100%:

RID=$(az cosmosdb show -n $ACCT -g $RG --query id -o tsv)
az monitor metrics list --resource "$RID" \
  --metric NormalizedRUConsumption \
  --interval PT1M --aggregation Maximum -o table

Expected: values at or near 100 during the burst — the container was genuinely over its 400 RU/s budget.

Step 5 — Fix it two ways and re-test. First, raise throughput (fixes a true shortfall):

az cosmosdb sql container throughput update -a $ACCT -g $RG \
  --database-name labdb --name items --throughput 1000

Re-run Step 3 — far fewer (or zero) 429s, because 1,000 RU/s now covers the burst. Then switch to autoscale to absorb future spikes without over-provisioning:

az cosmosdb sql container throughput migrate -a $ACCT -g $RG \
  --database-name labdb --name items --throughput-type autoscale
az cosmosdb sql container throughput update -a $ACCT -g $RG \
  --database-name labdb --name items --max-throughput 4000

Step 6 — Teardown. Delete everything so the lab costs nothing:

az group delete -n $RG --yes --no-wait

What you proved end to end: a 429 carries a retry-after-ms, the burst pinned Normalized RU Consumption at 100%, raising RU/s cleared a true shortfall, and autoscale future-proofs the spike — while writing everything to one pk showed how a hot partition concentrates load.

Common mistakes & troubleshooting

This is the centerpiece. Most 429 incidents are one of the rows below. Find your symptom, run the exact confirm step, apply the fix. The first table is the broad playbook spanning basic and advanced failures; the tables after it drill into the error/substatus codes and the per-symptom detail.

The 429 substatus / status reference

The status and substatus tell you which kind of throttle you hit. Keep this open beside the playbook:

The reading note that saves the most time: 429 substatus 3200 is the everyday case — “the container is over its RU/s” — and your very next move is always the per-partition Normalized RU split, because that one chart forks you between “raise RU/s” and “fix the partition key.” Do not raise throughput before you’ve looked at that split.

Per-symptom detail

Hot partition (row 2) — the most expensive misdiagnosis. The trap: the container shows low utilization, so it looks like there’s headroom, yet you’re 429ing. Split Normalized RU by PartitionKeyRangeId — one line at 100%, the rest near the floor. Cause: a key with too few distinct values, or one value (a tenant, store, “today” bucket) taking most traffic. Fix it with a better key (high-cardinality, evenly accessed) on a new container you migrate into, since the key is immutable. Raising RU/s here is the wrong move and the most common one.

Write amplification (row 3). Over-indexing is usually why writes 429. A small document costing 15–25 RU to create points straight at the indexing policy. Fix: a custom indexing policy including only queried paths, plus Patch instead of Replace for small updates so you don’t reindex the whole item.

Expensive query (row 4). A single query in a loop drains a budget. Sum its RequestCharge across pages — hundreds of RU per call is the signal — and read the query metrics (low index hit ratio, retrieved ≫ output). Fix: include the partition key in the filter (single-partition, not fan-out), SELECT only needed fields, page with MaxItemCount + continuation token, and add a composite index for multi-field ORDER BY.

Surfaced 429 exceptions (row 6) and disabled retries (row 10). 429 in the metric with no app errors is healthy — the SDK absorbed it. 429 exceptions in logs mean throttling outlasted the retry budget (~9 retries / ~30 s): a genuine shortfall or hot partition — fix the cause. For tolerant batch jobs you may raise MaxRetryAttemptsOnRateLimitedRequests / MaxRetryWaitTimeOnRateLimitedRequests to ride out longer bursts, but never to mask a chronic shortfall. The inverse mistake is setting attempts to 0 to “see the real errors,” turning every transient 429 into a user-facing failure — re-enable retries and log metrics for visibility instead.

Bulk import (row 9) and control-plane throttling (row 12). A migration or nightly ingest that pins the container at 100% is fixed by the SDK’s bulk mode (AllowBulkExecution = true), throttling the job’s concurrency, and temporarily raising RU/s for the load’s duration. A different animal is metadata throttling (substatus 1007/1016) from creating containers or changing throughput in a tight loop — not a data-plane problem at all, and raising RU/s does nothing; back off management operations instead (provision once, space out throughput changes).

Where to look first, in one rule: on any unexplained 429, open Normalized RU Consumption split by partition before anything else (even = shortfall, one range = hot partition); if 429s track one write or query, read that operation’s RequestCharge next; if 429s surface as exceptions, check the SDK retry settings against the metric.

Best practices

• Open the per-partition chart before touching throughput. On any 429, split Normalized RU Consumption by PartitionKeyRangeId first. Even saturation → raise RU/s; one hot range → fix the key. This single habit prevents the most expensive mistake.
• Choose the partition key for cardinality, even access, and query alignment — in that priority. High cardinality and even distribution prevent hot partitions; query alignment makes your common queries single-partition and cheap. Never key on a low-cardinality field (status, country) or a value you know will be skewed.
• Log RequestCharge on every hot code path. You can’t optimise what you don’t measure, and the cost is already on every response. Surface it in your telemetry so an expensive operation announces itself.
• Prefer point reads; make queries single-partition. A point read is 1 RU; always include the partition key in query WHERE clauses to avoid fan-out; SELECT only the fields you need and page large results.
• Trim the indexing policy to what you query. Automatic indexing of every path is convenient and expensive on writes. Include only filtered/sorted paths, exclude the rest — especially large free-text blobs.
• Use autoscale for spiky or unknown load, manual for steady predictable load. Don’t pay the autoscale premium on a flat workload; don’t fight 429s at peaks on a flat manual setting.
• Keep the SDK’s automatic retries on, and honour Retry-After. Tune the attempt and wait caps for batch tolerance; never disable retries to “see errors” — log metrics instead.
• Alert on Normalized RU Consumption, not just on 429 count. A metric alert at, say, 80% sustained gives you warning before throttling; pair it with an action group. See How to Create Your First Metric Alert and Action Group for Email, SMS and Push.
• Use bulk mode for ingest and Patch for partial updates. Bulk batches operations to use throughput efficiently; Patch avoids reindexing the whole item on a small change.
• Re-baseline throughput as traffic grows and skews. A setting that was right last quarter may be wrong now — trend the metric over weeks, not just during incidents.
• Cache hot, read-heavy lookups outside Cosmos DB. Offloading repeated point reads to a cache cuts RU consumption directly — see Azure Cache for Redis 101: Tiers, Cache-Aside Patterns and Connecting Your First App.
• Treat serverless as the default for low or unpredictable traffic. No provisioning means no 429-from-shortfall; switch to provisioned only when sustained volume makes per-RU billing costlier.

Security notes

Throttling and security intersect more than people expect. Rate-limiting is itself a protection — Cosmos DB’s per-second budget caps the blast radius of a runaway or hostile client; a buggy loop can’t run up an unbounded bill or starve the backend. Lean into it: provision deliberately so the ceiling is a known number, not a surprise.

Lock down access with identity, not keys, wherever you can. Cosmos DB supports Microsoft Entra ID (AAD) RBAC for the data plane, so an app can authenticate with a managed identity instead of a primary master key in config. Master keys are all-or-nothing (full account control) and a leaked one is catastrophic; prefer least-privilege data-plane roles (via the az cosmosdb sql role family) scoped to the containers an app needs. Where you must use keys, store them in Key Vault, rotate via the secondary-key swap, and never commit them — a connection string in git is a breach.

Isolate the network: put the account behind a private endpoint so it’s reachable only from your VNet, disable public network access, and use firewall rules to restrict source IPs if a private endpoint isn’t feasible. Encryption is on by default — at rest and in transit (TLS); for stricter needs use customer-managed keys (CMK). None of this changes the RU model, but it ensures the only clients spending your RU budget are the ones you authorised.

Cost & sizing

What drives the Cosmos DB bill is simple to state and easy to get wrong: provisioned throughput (RU/s) and stored data (GB), plus egress and any backup/restore. Throughput dominates for active workloads. The levers:

Three sizing rules that matter:

• Manual vs autoscale is a cost decision, not just a 429 decision. With a high peak-to-average ratio (spiky), autoscale is usually cheaper overall despite the per-RU premium, because manual must be sized for the peak 24/7. Flat load → manual wins. Estimate from Total Request Units over a representative week.
• Shared (database) throughput packs small containers cheaply. Up to 25 containers can share one database-level offer (minimum ~400 RU/s), far cheaper than 25 separate minimums. Move a container to dedicated throughput only when it’s busy enough to be a noisy neighbour — the lever for multi-container, low-traffic designs.
• Free tier and serverless cover dev and small prod. The free tier gives the first 1,000 RU/s and 25 GB free per account (one per subscription) — enough for a small production workload at zero throughput cost. Serverless has no provisioning and no shortfall-429, ideal for intermittent workloads, but bills per RU and caps lower, so high sustained traffic favours provisioned. The four capacity modes side by side:

The Mandara Foods lesson was financial as much as technical: panic-tripling RU/s pushed a ₹52,000 bill toward ₹150,000 and still failed, while the real fix landed at ₹41,000/month — below the original. Right-sizing Cosmos DB is about spending the budget evenly and cheaply per operation, not buying more of it.

Interview & exam questions

Q1. What does an HTTP 429 from Cosmos DB mean, and is it an error? It means Request rate too large — operations in a one-second window exceeded the provisioned RU/s for the container (or one partition). It is not a failure but flow control: the server rejects the surplus and returns a Retry-After. The SDK retries automatically by default, so transient 429s are usually invisible to the app. Maps to DP-420.

Q2. A container at 10,000 RU/s is 429ing at ~25% overall utilization. What’s happening? A hot partition: one physical partition is saturated while others idle, because the partition key concentrates traffic onto one slice. Confirm by splitting Normalized RU Consumption by PartitionKeyRangeId. Raising RU/s won’t fix it (the budget splits evenly); fix the partition key (high cardinality, even access) on a new container and migrate.

Q3. What is a Request Unit, and what’s the reference cost? An RU is Cosmos DB’s normalized currency for the work (CPU/memory/IOPS) of an operation. The baseline is a point read of a 1 KB item = 1 RU. Writes cost several RUs (indexing amplifies them), and queries scale with items scanned, cross-partition fan-out, and sorting.

Q4. How do you find the RU cost of a specific operation? Read the x-ms-request-charge response header (surfaced as RequestCharge in the SDK) on every response; sum it across pages for a query. For deeper analysis, enable the query metrics/diagnostics to see index hit ratio and retrieved-vs-output counts.

Q5. When would you choose autoscale over manual throughput? When load is spiky, bursty or unpredictable. Autoscale scales instantly between 10% and 100% of a configured max and bills per hour at the highest level reached — so you avoid 429s at peaks and don’t pay for idle headroom. For steady, predictable load, manual is cheaper per RU.

Q6. Why do writes cost more RUs than reads, and how do you reduce write cost? Because Cosmos DB indexes every property by default, every indexed path adds RU per write. Reduce it with a custom indexing policy (include only queried paths, exclude the rest), smaller documents, and Patch instead of Replace for partial updates.

Q7. What’s the difference between a 429 in the metric and a 429 exception in your app logs? The metric counts every throttled response, including ones the SDK silently retried and succeeded. Your app only sees a 429 exception when the SDK exhausts its retry budget (~9 attempts / ~30 s). Metric-only 429s are healthy; surfaced exceptions mean sustained throttling — a real shortfall or hot partition.

Q8. What properties make a good partition key? High cardinality (many distinct values → even spread across partitions), even access distribution (no single value dominates traffic), and query alignment (the key is the field you filter on most, so queries are single-partition and cheap). Bad keys are low-cardinality or known to be skewed.

Q9. Is disabling SDK retries on 429 a good way to “see errors faster”? No. The SDK’s automatic retries honour the server’s Retry-After and hide transient throttling from users. Disabling them turns invisible backoff into user-facing failures. For visibility, log the 429 metric and request charges instead; keep retries on and tune the caps for batch tolerance.

Q10. What’s the storage limit on a single logical partition, and why does it matter? A logical partition (all items with the same partition-key value) is capped at 20 GB. A low-cardinality key whose one value keeps growing will eventually fail writes outright — not just 429 — which is another reason to choose a high-cardinality key.

Q11. You raised RU/s 3× and the 429s barely moved. What does that tell you? That it is almost certainly not a true throughput shortfall but a hot partition (or a single pathologically expensive operation). A real shortfall clears proportionally when you add RU/s; a hot partition doesn’t, because the extra budget still splits evenly and the hot slice gets only its share. Open the per-partition metric.

Quick check

1. A 429 with substatus 3200 means what, and what’s your very next diagnostic move?
2. You provisioned 10,000 RU/s across 5 physical partitions. Roughly how much RU/s can a single hot partition use, and why does doubling the container’s RU/s not fix a hot partition?
3. Which response header tells you the RU cost of an operation, and which tells you how long to wait after a 429?
4. Why do writes typically cost more RUs than reads, and name one way to cut write cost.
5. You see 429s in the Azure Monitor metric but zero 429 exceptions in your application logs. Is anything wrong?

Answers

1. It means the container’s provisioned RU/s was exceeded in a one-second window. Your next move is to open Normalized RU Consumption split by PartitionKeyRangeId — even saturation across ranges = true shortfall (raise RU/s); one range hot = hot partition (fix the key). Never raise throughput before checking that split.
2. About 2,000 RU/s (the 10,000 is split roughly evenly across the 5 partitions). Doubling to 20,000 only raises each partition’s share to ~4,000 — the hot partition still saturates and you’ve doubled the cost — because throughput is enforced per partition, not pooled for one slice. The fix is the partition key, not the budget.
3. x-ms-request-charge gives the RU cost (on every response); x-ms-retry-after-ms gives the required wait (on 429 responses). The SDK reads both automatically.
4. Because Cosmos DB indexes every property by default, so each indexed path adds RU on every write. Cut it by trimming the indexing policy to only queried paths (and/or using Patch for partial updates, smaller documents).
5. No — that’s healthy. The metric counts every throttled response including ones the SDK silently retried and succeeded; your app only sees an exception when the SDK exhausts its retry budget. A small steady 429 count with no surfaced errors means the system is riding its budget normally.

Glossary

• Request Unit (RU) — Cosmos DB’s normalized currency for the cost of an operation, abstracting CPU/memory/IOPS. A 1 KB point read = 1 RU.
• RU/s (throughput) — The provisioned budget of Request Units per second, set at the container or database level. Exceeding it causes 429s.
• 429 Request rate too large — The HTTP status returned when operations exceed the available RU/s in a one-second window; flow control, not a failure.
• Retry-After (x-ms-retry-after-ms) — The header on a 429 telling the client how many milliseconds to wait before retrying; the SDK honours it automatically.
• Request Charge (x-ms-request-charge) — The header on every response giving the RUs that operation consumed.
• Substatus — A sub-reason code on a response; 3200 = container RU/s exceeded (the everyday 429); 1007/1016 = control-plane/metadata throttling.
• Partition key — The JSON field by which items are grouped and distributed across partitions; immutable once a container is created.
• Logical partition — All items sharing one partition-key value; capped at 20 GB of storage.
• Physical partition — A backend shard that holds many logical partitions and receives an even share of the container’s RU/s.
• PartitionKeyRangeId — The metric dimension identifying a physical partition; splitting Normalized RU by it reveals a hot partition.
• Hot partition — A physical partition saturating its RU/s share while others idle, caused by a skewed partition key; not fixable by adding RU/s.
• Normalized RU Consumption — The Azure Monitor metric showing the percentage of available RU/s used (max across partitions) — the single best 429 diagnostic.
• Autoscale — A throughput mode that scales instantly between 10% and 100% of a configured maximum, billed per hour at the peak level reached.
• Serverless — A capacity mode with no provisioning that bills per RU consumed; no shortfall-429, but lower ceilings and per-RU cost.
• Indexing policy — The configuration of which JSON paths are indexed; automatic indexing of all paths drives write RU cost.

Next steps

• Decide which API and consistency level fit before tuning throughput — start with Choosing a Cosmos DB API: NoSQL vs MongoDB vs Cassandra vs Gremlin vs Table Decoded.
• Wire request charges and 429 metrics into dashboards and alerts with Azure Monitor and Application Insights: Full-Stack Observability and a metric alert from How to Create Your First Metric Alert and Action Group for Email, SMS and Push.
• Offload hot, read-heavy lookups to cut RU consumption with Azure Cache for Redis 101: Tiers, Cache-Aside Patterns and Connecting Your First App.
• See the same 429-and-Retry-After pattern in a different service in Taming Azure OpenAI 429s: TPM vs RPM Limits, Quota Math, Retry-After Headers, and Backoff That Works.
• Build an event-driven pipeline (a common source of 429-inducing load) with Cosmos DB Change Feed in Action: Event-Driven Pipelines with Azure Functions and the Processor.