Tokens, Context Windows, and Cost: How Azure OpenAI Billing Actually Works Before You Burn Your Budget

Your first Azure OpenAI proof-of-concept costs almost nothing — a few hundred rupees, a delighted demo, a green light to “productionise it.” Three months later the bill is a five-figure surprise and nobody can explain why, because nobody can answer the one question every Azure OpenAI cost rests on: what exactly are we paying for, per call? The answer is tokens — not requests, not characters, not API calls — and until you can look at a prompt, roughly count its tokens, estimate the reply, and add the two, the bill feels like weather. This article makes it something you control.

Azure OpenAI Service is Microsoft’s hosted version of OpenAI’s models (the GPT-4o family, GPT-4o mini, GPT-3.5 Turbo, embeddings, and more), running in your Azure subscription with enterprise networking, identity, and regional data residency. It bills almost entirely on tokens — the sub-word chunks a model reads and writes — and meters input tokens (everything you send: system prompt, conversation history, retrieved documents, the question) separately from output tokens (everything generated), usually at a higher rate for output. Two more concepts decide the bill: the context window (the hard ceiling on how many tokens one request may contain, input plus output together) and the deployment model — flexible pay-as-you-go (Standard) versus reserved Provisioned Throughput Units (PTU).

By the end you will read a prompt and see the money in it: why a chatbot that “remembers the whole conversation” gets quadratically more expensive every turn, why retrieval-augmented generation (RAG) can quietly 10× your input bill, when to reach for GPT-4o mini over GPT-4o, what prompt caching does to the input rate, and how to turn “about 50,000 questions a day” into a defensible monthly figure before finance asks. Mental model first; the pricing tables, decision grids, and a token-counting lab follow.

What problem this solves

Most teams ship an Azure OpenAI feature without anyone owning the unit economics, because the SDK call is trivially easy — client.chat.completions.create(...) and you have a working assistant. The cost machinery stays invisible right up until one of three things happens: traffic grows, the prompt grows, or the conversation grows — and all three multiply the same hidden quantity, tokens per call. A team that never built the mental model can’t see the cliff coming.

What breaks is predictable and expensive. Someone stuffs the entire knowledge base into every prompt “to be safe,” paying input on 40,000 tokens to answer a 12-token question. A chatbot keeps the full transcript in the prompt, so by turn 20 every message re-sends the previous nineteen. A team picks GPT-4o for a task GPT-4o mini would nail at a fifteenth of the price. Another leaves max_tokens huge “just in case,” and a chatty model bills 4,000 output tokens where 300 would do. None of these are code bugs — they are bugs in the understanding, and they only show up on the invoice.

Who hits this: every team adopting generative AI on Azure — a startup’s support bot, an enterprise RAG copilot, a data team’s batch summarisation over millions of records. The pattern is identical: the POC is cheap because volume and prompt size are tiny; production is expensive because both grew; the gap is entirely tokens × price × calls. Compute that product on a napkin and Azure OpenAI becomes a line item you can forecast, optimise, and defend.

To frame the whole field before the deep dive, here is what actually drives an Azure OpenAI bill, the lever that controls each driver, and the single most common way each one quietly explodes:

Learning objectives

By the end of this article you can:

• Explain what a token is, estimate token counts from text (the ~4-characters / ~0.75-words-per-token rule of thumb), and tell why “tokens” and not “words” or “API calls” is the unit of billing.
• Describe the context window as a hard per-request ceiling on input + output tokens combined, and predict the two failure modes when you exceed it (truncation or an error).
• Break any Azure OpenAI charge into its parts — input tokens, output tokens, cached input, and (for embeddings) input-only — and identify which the system prompt, history, and RAG context fall into.
• Compare GPT-4o, GPT-4o mini, and GPT-3.5 Turbo on price, capability, and context size, and pick the right one for a given task instead of defaulting to the flagship.
• Explain why multi-turn chat cost grows faster than linearly, and apply history windowing and summarisation to bound it.
• Decide between pay-as-you-go (Standard) and Provisioned Throughput Units (PTU) using a load-shape and break-even argument, and say what prompt caching and Batch change about the rate.
• Estimate a monthly Azure OpenAI bill in INR and USD from request volume and average tokens per call, and set quota / TPM limits and budget alerts so a runaway loop can’t bankrupt the month.

Prerequisites & where this fits

You should be comfortable calling a large language model over an API — sending a prompt, getting a completion — and reading JSON. You do not need machine-learning theory; this is an economics and operations article, not a modelling one. Running az in Cloud Shell helps for the deployment and quota commands, and we define RAG (retrieval-augmented generation) inline.

This sits at the very front of the AI/ML on Azure track — the cost-literacy foundation everything else assumes. New to the models? AI-900: Generative AI & Azure OpenAI Fundamentals is the on-ramp. Once the token economics here make sense, the architecture follow-ons do too: An Enterprise Landing Zone for Azure OpenAI: Networking, Quotas, and Gateways wraps quotas and private networking around a deployment, Azure AI Search for RAG: Vector Indexing, Hybrid Search, Semantic Ranking, and Indexer Pipelines is where your RAG input tokens come from, and An LLM Gateway for Cost, Safety and Observability is how large orgs meter this spend centrally. The broader discipline lives in Azure FinOps and Cost Management: Controlling Cloud Spend at Scale.

A quick map of who in a team owns which token lever, so the conversation has the right people in the room:

Core concepts

Six mental models make every later pricing decision obvious. Internalise these and the tables that follow are just lookups.

A token is a chunk of text, not a word. Models read tokens — sub-word units from a tokenizer. Common short words are one token (the, and); longer or rarer words split (tokenization → token + ization); spaces and punctuation count; non-English text and code tokenize less efficiently. The rule of thumb for English prose is ~4 characters per token, or ~0.75 words per token (~750 words ≈ 1,000 tokens). You pay per token, so this is your napkin-math currency.

You pay for input and output separately, and output usually costs more. Every chat call has two metered halves: prompt (input) tokens — everything you send, concatenated: system prompt, the conversation history you include, retrieved documents, the latest message — and completion (output) tokens the model generates. Azure prices these at different per-1,000-token (per-1K) rates, and output is typically several times more expensive than input. A short question that triggers a long answer is output-dominated; a huge RAG prompt yielding one line is input-dominated. Which half dominates tells you which lever to pull.

The context window is a hard ceiling on input + output combined. Each model’s context window is the maximum tokens a single request can involve, counting prompt and completion together (e.g. 128K for the GPT-4o family). It is a capacity limit, not a billing one, and it bites two ways. Reserved output space eats into it: if the window is 128K and your prompt is 127K, the model has ~1K tokens to answer in. And exceeding it doesn’t silently work — you get an error, or your own code truncates history/context to fit and loses information. The window caps what fits in one call; tokens cap what you pay.

Conversation memory is re-sent text — that’s why chat gets expensive. A model is stateless; to make it “remember,” your code re-sends prior turns in the prompt every message. Turn 2 re-sends turn 1; turn 10 re-sends turns 1–9; turn n pays input on all n−1 prior turns plus the new one. Cumulative input grows with the square of the turn count. The fix: don’t send history forever — window it (last k turns) or summarise older turns.

RAG is “paste documents into the prompt,” and documents are big. Retrieval-augmented generation retrieves relevant chunks from a search index and pastes them into the prompt as context before the question. Those chunks are input tokens. Eight chunks of 800 tokens adds ~6,400 input tokens to every call. RAG is often the single largest line on a copilot’s bill; the lever is retrieval discipline — fewer, smaller, better-ranked chunks beat dumping the corpus.

The deployment type changes the pricing model, not just the price. Standard (pay-as-you-go) charges per token — pay for exactly what you use, no commitment, ideal for variable or low volume. Provisioned Throughput Units (PTU) reserve dedicated capacity for a fixed price regardless of tokens consumed — predictable cost and latency at high steady volume, but you pay whether or not you use it. Choosing wrong wastes money: PTUs for spiky low traffic, or pay-as-you-go for massive steady traffic where reserved would be cheaper and faster.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

What a token really is, and how to count one

You cannot budget what you cannot count. A tokenizer splits text using a fixed vocabulary learned during training (the GPT-4o family uses o200k_base; GPT-3.5/4 used cl100k_base). The rules that surprise people:

• A leading space is usually part of the token — " cat" and "cat" can tokenize differently.
• Common words are one token; rare or long words split into several.
• Numbers, punctuation, and emoji cost tokens; a long number can be several.
• Whitespace and newlines count — pretty-printed JSON costs more than minified.
• Non-English text and source code use more tokens per character, so ~4-chars is an English-prose approximation, not a law.

The practical rules of thumb for English prose: ~4 characters per token, ~0.75 words per token (so ~750 words ≈ 1,000 tokens), or equivalently ~1.3 tokens per word; a typical A4 page (~500 words) is ~650–700 tokens. Minifying JSON/HTML before you paste it can cut 10–30% of tokens. The only exact count comes from a tokenizer library (tiktoken) or the API’s returned usage. A few examples to calibrate your eye (approximate — verify with tiktoken when it matters):

The only number you should bill against is the API’s own accounting. Every chat completion response includes a usage object with the exact split — ground truth, and you should log it:

{
  "usage": {
    "prompt_tokens": 1287,
    "completion_tokens": 142,
    "total_tokens": 1429
  }
}

prompt_tokens is the input charge; completion_tokens the output charge; total_tokens their sum, which must stay under the context window. Logging usage per call is the highest-leverage observability move here — it turns the bill into a sum you can group by user, feature, and model.

The anatomy of a billed call

The “prompt” you pay input on is not just the user’s question — it’s the concatenation of several pieces your code assembled. Knowing the parts tells you where to cut. Everything on the input side of a single call, and how each behaves:

Read that as a priority list. The question is almost never the cost; the system prompt is fixed overhead; the things that grow — and wreck budgets — are conversation history and RAG context on the input side, and an uncapped completion on the output side. Three components, three levers.

A worked decomposition. Suppose a support copilot call looks like this:

The user typed 40 tokens, but you billed input on 6,490 — and 81% is history + RAG, the two growable components. Halve RAG to 3 well-ranked chunks and window history to the last 4 turns and input falls to ~3,500 tokens — a ~46% cut on the larger half, with no change to the model or the experience. That is the whole optimisation game in one example.

The pricing model: per-1K tokens, input vs output

The pay-as-you-go meter is simple: (input tokens ÷ 1,000 × input rate) + (output tokens ÷ 1,000 × output rate), summed over every call. The two rates differ per model and output is the pricier one. (Some materials quote per-1M tokens; that’s the same number ÷ 1,000 — be consistent.)

Exact rates change by time and region, so treat numbers here as illustrative for the arithmetic and confirm live figures on the pricing page before committing a budget. The durable lesson is the shape — output costs multiples of input, and the mini/3.5 models are an order of magnitude cheaper than the flagship:

The key relationship is input vs output asymmetry. Because output is the expensive half, answer verbosity is often a bigger lever than prompt size. A RAG call with a 6,000-token prompt and a 100-token answer may cost less than a 500-token-prompt call that writes a 2,000-token essay. Reason about both halves; never optimise the prompt while ignoring max_tokens.

The same call costed three ways at illustrative rates (input ₹0.40/1K, output ₹1.60/1K — placeholders to show the method):

The third call costs more than the second despite a twelve-times-smaller prompt — output is 4× the rate and there’s ten times more of it. That’s why “just set max_tokens” is one of the highest-ROI changes available.

Choosing a model: GPT-4o vs GPT-4o mini vs GPT-3.5 Turbo

The biggest lever after “send fewer tokens” is which model you send them to. Flagship and small models differ by more than 10× for the same token count, and much production traffic — classification, extraction, short Q&A, routing — doesn’t need the flagship. The skill is routing: hard reasoning to the big model, everything else to a small one.

A capability-and-cost comparison (relative cost shows the gap, not a quote; capabilities and context sizes are the durable facts to confirm on the model page):

The decision in practice is a routing table — match the task, not the team’s habit, to the model:

A pragmatic pattern is a two-tier cascade: try GPT-4o mini first; if a cheap confidence check or validator says the answer is weak, escalate the same prompt to GPT-4o. Most traffic resolves on the cheap tier; you pay flagship rates only for the hard minority. Routing this centrally is what An LLM Gateway for Cost, Safety and Observability describes.

Why multi-turn chat gets expensive: the quadratic trap

This surprises nearly everyone and is the most common “the bill grew and nothing changed” cause. Because the model is stateless, “memory” is re-sending the transcript. If each turn adds roughly the same tokens, the input on turn n includes all prior turns, so cumulative input across N turns grows with N², not N.

Concretely — each exchange adds ~400 tokens, and you re-send the full history every turn:

By turn 40 a single conversation has cost 820 units of input where a stateless one-shot would cost 40 — a 20× multiplier hidden inside “let the user keep chatting.” A few long power-user sessions can dominate the bill. Mitigations, by how much the model must remember:

The rule: never re-send unbounded history in production. A sliding window of the last 6–10 turns plus a periodic running summary keeps cost flat with good-enough recall. “Append to the messages array forever” is the most expensive default in the SDK.

Prompt caching, Batch, and embeddings: the rate-changers

Three features change the rate, not the token count, each fitting a workload shape.

Prompt caching discounts the input tokens of a repeated prompt prefix. If many calls share a large identical opening — a big system prompt, fixed few-shot examples, a long instruction block — the platform serves that prefix from cache at a reduced input rate after the first call. The constraint: the cached part must be the prefix and byte-identical, so put static content at the very top and variable bits (question, chunks) at the bottom. A 2,000-token preamble hit thousands of times an hour is a material saving for zero behavioural change.

Batch runs requests asynchronously within a completion window (up to ~24 h) at a discounted rate (~half of Standard) against a separate, larger quota. It fits anything not latency-sensitive — overnight ticket summarisation, backlog classification, bulk embeddings. You trade immediacy for roughly half the price: superb for back-office work, useless for interactive chat.

Embeddings turn text into vectors for RAG and semantic search; they bill input-only (no completion) at a tiny fraction of chat. The cost that matters is volume: embedding a million documents is a real one-time bill, and re-embedding on every change adds up. Embedding queries at request time is negligible; embedding the corpus is the line to size.

How the rate-changers compare and when to reach for each:

Pay-as-you-go vs Provisioned Throughput Units (PTU)

The deployment type is a strategic decision, not a setting. Standard (pay-as-you-go) charges per token with no commitment — you pay for exactly what you consume and nothing when idle. Provisioned Throughput Units (PTU) reserve dedicated capacity for a flat price; you get predictable, lower latency and a fixed bill regardless of tokens pushed through — wonderful at high steady load, wasteful at low or spiky load. Reservations (1-month/1-year) discount PTU capacity further for committed users.

The trade-off, side by side:

A simple decision rule, expressed as a table you can apply without a spreadsheet:

The break-even is a crossover: PTU has high fixed cost and ~zero marginal cost up to its capacity; PAYG has zero fixed cost and constant marginal cost. Below the crossover, PAYG wins; above it, PTU. The trap is buying PTUs on optimistic forecasts and running them at 20% utilisation. Earn your way into PTUs with real PAYG data, then size the reservation to your steady-state floor, not your peak.

Architecture at a glance

There is no system diagram here because the “architecture” that matters is a cost flow you carry in your head. Picture a request moving through five stages, watching the token meter at each.

Stage one — assembly: your code concatenates the system prompt, few-shot examples, the conversation history it chose to include, the RAG context retrieved from a search index, and the user’s question. Everything here is input tokens, and this is where 80–90% of the bill is decided — history and RAG are the components that grow. Stage two — the call: the prompt goes to the model deployment (pay-as-you-go or PTU), which enforces your TPM/RPM quota — the valve that throttles a runaway loop. Stage three — generation: the model produces a completion, bounded by max_tokens — the pricier output tokens. Stage four — accounting: the response returns a usage object with the exact prompt_tokens / completion_tokens / total_tokens split, the ground truth you log. Stage five — control: logged usage feeds dashboards, per-feature attribution, and budget alerts, closing the loop.

Money is spent at assembly and generation but understood at accounting. Every optimisation here targets one of three points: shrink what stage one assembles (trim the system prompt, window history, retrieve fewer chunks, cache the prefix), cap what stage three generates (max_tokens, brevity, cheaper model), and instrument stage four so the other two are decisions, not accidents. Narrate a request through those five stages and you understand Azure OpenAI billing better than most teams running it.

Real-world scenario

Saanvi Logistics built an internal “shipment copilot” — a RAG chat assistant over their policy and tariff documents on Azure OpenAI in Central India. The POC cost about ₹3,000 across a month of light use and got an enthusiastic rollout to all 600 operations staff. The team is three engineers; nobody built a token budget because the POC bill was a rounding error.

Two months later the Azure OpenAI line hit ₹2,40,000 and was still climbing. Finance asked why? and the team had no answer — they had never logged the usage object. The first fix was diagnostic, not optimisation: log prompt_tokens and completion_tokens per call, tagged with user and feature, into Application Insights (the pattern from Azure Monitor and Application Insights: Full-Stack Observability). Within a day the breakdown was damning.

Three things ate the budget, all token-mechanics, none a code bug. RAG with no discipline: the retriever returned the top 15 chunks of ~900 tokens, so every question carried ~13,500 input tokens — to answer questions a human could answer from one paragraph. Unbounded history: the chat kept the entire session transcript, so power users running 30–40-turn sessions each paid the quadratic tax, and a dozen of them dominated daily spend. The flagship for everything: every call, including “what’s the cutoff for same-day dispatch?”, went to GPT-4o where mini would have answered indistinguishably at a fraction of the rate.

The fixes were fast. RAG dropped from 15 chunks to 4 well-ranked chunks (with semantic re-ranking), cutting average input ~70%. Full-history became a sliding window of the last 6 turns plus a running summary, flattening the quadratic curve. They routed by task: a cheap mini classifier sent factual lookups to GPT-4o mini and reserved GPT-4o for multi-document synthesis — ~15% of traffic. They enabled prompt caching on the 1,800-token static prefix and set max_tokens to 400.

The bill fell from ₹2,40,000 to about ₹41,000 — an 83% cut — with no measurable quality drop, and every rupee now attributable on a dashboard. They set a budget alert at ₹60,000 so a regression pages them instead of surprising finance. The wiki lesson: “The model is stateless and the meter is per token. If you don’t decide what goes into the prompt, the defaults decide for you — expensively.”

The cost story as a before/after, because the order of fixes is the lesson:

Advantages and disadvantages

The per-token model makes Azure OpenAI both wonderfully accessible and quietly dangerous. Weigh it honestly:

The model is right for almost everyone if you instrument it: log usage, set max_tokens, route models, bound history, budget-alert. It bites hardest on teams that treat the SDK call as the whole story and never look at the meter — most teams, for the first few months, until the invoice teaches the lesson.

Hands-on lab

Two parts: count tokens locally (free, no Azure spend), then deploy a model and read the real usage from one call. Run in Cloud Shell (Bash) or any machine with Python and the Azure CLI.

Part A — count tokens with tiktoken (free, offline).

Step 1 — install the tokenizer.

pip install --quiet tiktoken

Step 2 — count tokens for some sample text and verify the rules of thumb.

# save as count.py, then: python count.py
import tiktoken
enc = tiktoken.get_encoding("o200k_base")  # GPT-4o-family tokenizer

samples = [
    "Hello, world!",
    "Azure OpenAI bills you per token, not per word.",
    "tokenization",
    '{"id":1,"name":"widget","price":9.99}',
]
for s in samples:
    n = len(enc.encode(s))
    print(f"{n:>4} tokens | {len(s):>4} chars | ratio {len(s)/n:0.1f} chars/token | {s}")

Expected: short strings land near ~4 chars/token; tokenization splits into 2–3 tokens; the JSON shows a lower chars-per-token ratio (structure costs tokens). You’ve now felt why code and JSON tokenize less efficiently than prose.

Step 3 — estimate a document’s tokens before paying to send it.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
doc = open("some_policy.txt").read()   # any text file you have
tokens = len(enc.encode(doc))
print(f"{tokens} tokens  (~{tokens/1000:0.1f}K, ~{tokens*4} chars, ~{int(tokens*0.75)} words)")
print(f"At 4 chunks of 800 tokens, this doc is ~{-(-tokens//800)} chunks of RAG context")

This is the calculation you do before deciding how many chunks of a document to retrieve.

Part B — deploy a model and read the real usage (small spend).

Step 4 — create the resource and a deployment.

RG=rg-aoai-lab
LOC=eastus            # pick a region where your model/quota is available
ACCT=aoai-lab-$RANDOM
az group create -n $RG -l $LOC -o table

az cognitiveservices account create -n $ACCT -g $RG -l $LOC \
  --kind OpenAI --sku S0 --yes -o table

# Deploy a cheap model (GPT-4o mini) at a small capacity (TPM in thousands)
az cognitiveservices account deployment create -n $ACCT -g $RG \
  --deployment-name gpt4o-mini \
  --model-name gpt-4o-mini --model-version "2024-07-18" --model-format OpenAI \
  --sku-name Standard --sku-capacity 10 -o table

Expected: an account row, then a deployment named gpt4o-mini. The --sku-capacity 10 caps throughput to ~10K TPM — your blast-radius limit for the lab.

Step 5 — make one call and print the usage block.

ENDPOINT=$(az cognitiveservices account show -n $ACCT -g $RG --query properties.endpoint -o tsv)
KEY=$(az cognitiveservices account keys list -n $ACCT -g $RG --query key1 -o tsv)

curl -s "$ENDPOINT/openai/deployments/gpt4o-mini/chat/completions?api-version=2024-08-01-preview" \
  -H "Content-Type: application/json" -H "api-key: $KEY" \
  -d '{
        "messages": [
          {"role":"system","content":"You are a terse assistant. Answer in one sentence."},
          {"role":"user","content":"In one sentence, what is a token in an LLM?"}
        ],
        "max_tokens": 60
      }' | python -c "import sys,json; d=json.load(sys.stdin); print(json.dumps(d['usage'], indent=2))"

Expected output is the ground-truth meter, e.g.:

{
  "prompt_tokens": 33,
  "completion_tokens": 24,
  "total_tokens": 57
}

Step 6 — prove max_tokens caps output (and cost). Re-run Step 5 with "max_tokens": 5 and watch completion_tokens drop to ~5 and the answer cut off. Then ask for “three detailed paragraphs” without a cap and watch it balloon. You’ve now observed the most important output-cost lever directly.

Validation checklist. You counted tokens offline (proving a token ≠ a word), sized a document for RAG, deployed a capacity-capped model (quota as a blast-radius cap), read the exact billable prompt_tokens/completion_tokens split per call, and watched max_tokens control the expensive half — the whole token-economics loop in one sitting.

Cleanup (avoid lingering charges).

az group delete -n $RG --yes --no-wait

Cost note. A few chat calls on GPT-4o mini cost a tiny fraction of a rupee; Part A is entirely free (offline tokenizer). Deleting the resource group removes the account and deployment so nothing meters afterwards.

Common mistakes & troubleshooting

The “incidents” in a cost article are budget surprises and limit errors. First the scannable table, then the detail on the ones that bite hardest. Each is symptom → root cause → how to confirm → fix.

1. The bill grew but “nothing changed.” Almost always unbounded history: as sessions lengthen, each turn re-sends more transcript, so average prompt_tokens drifts up under identical traffic. Confirm: plot logged prompt_tokens against turn-number-in-session — a rising line is the smoking gun. Fix: a sliding window (last 6–10 turns) plus a running summary flattens the quadratic curve.

2. Every call carries a huge prompt. RAG without discipline — too many or too-large chunks, or no re-ranking so you pad with irrelevant context. Confirm: log prompt_tokens and the retrieved chunk count/size; thousands of tokens for a one-paragraph answer is the tell. Fix: fewer chunks (3–5 re-ranked beat 15 raw), smaller chunk size, semantic re-ranking.

5. 429 Too Many Requests. You exceeded the deployment’s TPM/RPM quota — a rate limit, not billing. Confirm: the 429 and retry-after header; az cognitiveservices account deployment show ... --query sku.capacity. Fix: exponential backoff/retry, raise capacity if quota allows, spread across regions/deployments, consider PTU for a justified steady rate.

6. Context length exceeded. Your prompt tokens plus reserved max_tokens exceed the context window — output space counts against the same ceiling. Confirm: the API error names the requested count and the limit. Fix: trim the prompt, lower max_tokens, or move to a larger-context model — but treat a too-long prompt as a cost smell, not just a limit to raise.

8. Prompt caching isn’t biting. Caching discounts only an identical prefix. Per-request content (question, chunks, a timestamp) placed before the static block makes the prefix differ every call. Confirm: no cached-token discount despite a large repeated system prompt. Fix: order static-first — system prompt and examples at the top, variable content at the bottom.

Best practices

Crisp, production-grade rules — one decision each:

• Log the usage object on every call, tagged with user, feature, and model. You can’t optimise or defend a bill you can’t break down — this is the prerequisite for everything else.
• Always set max_tokens — the cheapest control on the expensive half. Pick the real maximum the feature needs, not a giant “just in case” number.
• Never re-send unbounded history. A sliding window of the last k turns plus a running summary; the append-forever pattern is quadratic and the most expensive default.
• Default to the cheapest adequate model. Route trivial tasks (classify, extract, route, short Q&A) to GPT-4o mini; reserve GPT-4o for genuine reasoning via a two-tier cascade.
• Discipline RAG retrieval. Fewer, smaller, re-ranked chunks beat dumping the corpus — it’s the biggest controllable input line, so measure it.
• Put static content first and cache it — a fixed system/few-shot block at the prompt’s head becomes a prompt-caching discount for free.
• Use Batch for anything non-interactive — overnight summarisation, backlog classification, bulk embedding on the ~50%-cheaper async path.
• Size embeddings by corpus, not query. Embed only changed documents; re-embedding the whole corpus on every edit is a silent recurring cost.
• Set TPM/RPM quotas deliberately as a blast-radius cap, with retry/backoff so a 429 degrades gracefully.
• Earn your way into PTUs. Run on PAYG until you have steady-state data, then size a reservation to your floor — never to a peak.
• Create a cost budget with alerts from day one, so a runaway loop pages you instead of surprising finance.
• Forecast in tokens, never requests or words — “calls/day × avg total tokens/call × rate” is the only estimate that survives the invoice.

Security notes

Cost and security overlap — a leaked key or open endpoint is also a spend incident, because someone else can run up your token bill.

• Use Microsoft Entra ID (managed identity), not API keys, where possible. Keys are bearer credentials anyone can spend until you rotate them; Entra ID with RBAC is identity-bound and auditable. Patterns in Azure Key Vault: Secrets, Keys and Certificates Done Right.
• If you must use keys, store them in Key Vault and rotate — never in source or config. A key in git is a direct line to your budget.
• Lock the endpoint down with Private Endpoints / network rules. An internet-reachable deployment is an invitation to token theft. Blueprint in An Enterprise Landing Zone for Azure OpenAI: Networking, Quotas, and Gateways.
• Cap blast radius with per-deployment quotas (TPM/RPM) — if a key leaks or a bug loops, the quota throttles damage to a known ceiling.
• Front shared access with a gateway for centralised auth, per-team budgets, rate limits, and logging — the role of An LLM Gateway for Cost, Safety and Observability in a larger org.
• Mind data residency and content logging. Choose the region for compliance, and don’t debug-log full prompts — that’s a privacy and storage concern.

Cost & sizing

Turn a volume estimate into a budget number with one formula:

Monthly cost ≈ (calls/day × 30) × ( (avg input tokens ÷ 1,000 × input rate) + (avg output tokens ÷ 1,000 × output rate) )

Fill in four quantities: calls/day, avg input tokens, avg output tokens, and the per-1K rates for your model (confirm live on the pricing page). A worked estimate at illustrative rates (input ₹0.40/1K, output ₹1.60/1K — placeholders) for three shapes:

Treat the rupee totals as method demonstrations — plug in real rates and your token averages from logged usage for a defensible figure. The relationships are the lesson: the RAG-heavy uncapped variant costs ~8× the lean one for the same traffic, entirely from bigger input and uncapped output; Batch roughly halves the rate for deferrable work.

What drives the bill, ranked, with the lever and rough magnitude of each:

To stop a surprise, set a budget with alerts on the resource group on day one:

# Anomaly/threshold alerts: create a budget on the resource group that holds the AOAI account
az consumption budget create \
  --budget-name aoai-monthly \
  --amount 60000 --time-grain Monthly \
  --category Cost \
  --resource-group rg-aoai-lab \
  --start-date 2026-06-01 --end-date 2027-06-01

// A monthly cost budget with email alerts at 80% and 100% of ₹60,000
resource budget 'Microsoft.Consumption/budgets@2023-11-01' = {
  name: 'aoai-monthly'
  properties: {
    category: 'Cost'
    amount: 60000
    timeGrain: 'Monthly'
    timePeriod: { startDate: '2026-06-01', endDate: '2027-06-01' }
    notifications: {
      warning80: {
        enabled: true
        operator: 'GreaterThanOrEqualTo'
        threshold: 80
        contactEmails: [ 'finops@example.com' ]
      }
      actual100: {
        enabled: true
        operator: 'GreaterThanOrEqualTo'
        threshold: 100
        contactEmails: [ 'finops@example.com' ]
      }
    }
  }
}

And cap the rate (worst-case spend velocity) by sizing deployment capacity — --sku-capacity is in thousands of TPM:

# Lower capacity = a smaller blast radius if something loops
az cognitiveservices account deployment create -n $ACCT -g $RG \
  --deployment-name gpt4o-mini --model-name gpt-4o-mini \
  --model-version "2024-07-18" --model-format OpenAI \
  --sku-name Standard --sku-capacity 20

The broader practice — tagging, showback, anomaly detection across the estate — lives in Azure FinOps and Cost Management: Controlling Cloud Spend at Scale.

Interview & exam questions

Useful for AI-900 / AI-102 prep and for any architect screening on GenAI economics. Question, then a model answer.

1. What is a token, and why does Azure OpenAI bill on tokens rather than words or API calls? A token is a sub-word chunk produced by the model’s tokenizer (~4 characters of English prose on average). Models process text as tokens, so compute — and therefore cost — scales with token count, not word count or request count. A single call can range from tens to over a hundred thousand tokens, so billing per call would be meaningless; per-token is the natural unit.

2. Explain the difference between input and output tokens and which is more expensive. Input (prompt) tokens are everything you send — system prompt, history, RAG context, the question. Output (completion) tokens are what the model generates. Output is typically several times more expensive per token because generation is more compute-intensive than reading, which is why capping max_tokens is a high-leverage cost control.

3. What is a context window, and what happens if you exceed it? It’s the maximum number of tokens a single request may contain, counting input and output together (e.g. ~128K for the GPT-4o family). Exceeding it doesn’t silently work: you get an error for an over-long request, or your own code must truncate history/context to fit, losing information. Reserved output space also counts against the window.

4. Why does multi-turn chat cost grow faster than linearly with the number of turns? Because the model is stateless, “memory” means re-sending prior turns as input each call. Turn n re-sends turns 1…n−1, so cumulative input across a conversation grows roughly with the square of the turn count. Windowing history or summarising older turns bounds it back to near-linear or constant per turn.

5. When would you choose GPT-4o mini over GPT-4o? For high-volume, lower-complexity tasks — classification, field extraction, routing, summarisation, short factual Q&A over RAG — where mini is accurate enough at a small fraction of the cost. Reserve GPT-4o for genuine multi-step reasoning, nuanced writing, or hard synthesis. A two-tier cascade (mini first, escalate on low confidence) captures most of the savings.

6. Compare pay-as-you-go (Standard) and Provisioned Throughput Units (PTU). Standard bills per token with no commitment — ideal for variable, low, or unknown volume, and ~free when idle. PTU reserves dedicated capacity for a flat price with lower, predictable latency — cheaper per token and better for SLAs at high steady volume, but you pay for idle capacity. Choose PAYG until real usage data justifies sizing a PTU reservation to your steady-state floor.

7. What does prompt caching do, and what’s the constraint? It discounts the input cost of a repeated prompt prefix served from cache. The constraint is that the cached portion must be an identical prefix across calls, so static content (system prompt, few-shot examples) must come first and variable content (question, retrieved chunks) last, or nothing caches.

8. How does RAG affect your token bill, and how do you control it? RAG pastes retrieved document chunks into the prompt as input tokens, frequently making it the largest input contributor. Control it with retrieval discipline: fewer chunks, smaller chunks, and semantic re-ranking so the few you send are the most relevant — often 3–5 good chunks beat 15 raw ones at a fraction of the input cost.

9. You get a 429 Too Many Requests. What does it mean and how do you respond? You exceeded the deployment’s tokens-per-minute or requests-per-minute quota — a rate limit, not a billing error. Respond with exponential backoff/retry, raise the deployment capacity if quota allows, distribute load across deployments/regions, and consider PTU if the sustained rate justifies reserved capacity.

10. How would you estimate the monthly cost of a new Azure OpenAI feature? Multiply calls/day × 30 by the per-call cost, where per-call cost = (avg input tokens ÷ 1,000 × input rate) + (avg output tokens ÷ 1,000 × output rate) for the chosen model. Get the token averages from logged usage on a pilot, never from word or request counts, and confirm current per-1K rates on the pricing page.

11. What’s the cheapest single change to reduce output cost, and why? Setting max_tokens. Output is the pricier half of every call, and many models default to verbose answers; capping output length directly bounds the expensive component with no model or infrastructure change.

12. Why are embeddings cheap, and where does embedding cost actually accumulate? Embeddings bill input-only (no generated output) and at a tiny per-token rate, so any single embedding is negligible. Cost accumulates with volume — embedding a large corpus, and especially re-embedding the whole corpus on every content change. Embed only changed documents and batch the rest.

Quick check

1. A user types a 12-token question, but the call bills 6,000 input tokens. Name the two prompt components most likely responsible.
2. The context window is 128K tokens and your prompt is already 127K tokens. Roughly how much room does the model have to answer, and what happens if your answer needs more?
3. Why does turn 30 of a chat cost far more than turn 3, even if each message is the same length?
4. You have a trivial “classify this email” task running on GPT-4o. What’s the first change you’d make and roughly what magnitude of saving might it yield?
5. Your team is about to buy PTUs for a brand-new feature with unknown, spiky traffic. What’s the risk, and what should you do first?

Answers

1. Conversation history (re-sent prior turns) and RAG context (retrieved document chunks) — the two input components that grow; the user’s question is almost never the cost.
2. Only about 1K tokens of output room, since input + output share the 128K ceiling. If the answer needs more, you get a context-length error or must trim the prompt — reserved output space counts against the window.
3. The model is stateless, so “memory” is re-sent transcript; turn 30 re-sends ~29 prior turns as input, while turn 3 re-sends ~2. Cumulative input grows roughly with the square of the turn count — the quadratic trap.
4. Route it to GPT-4o mini (or 3.5) instead of the flagship; for trivial classification this can be on the order of 10–20× cheaper per token with no quality loss.
5. The risk is paying full price for mostly-idle reserved capacity. Run on pay-as-you-go first to gather real steady-state usage, then size a PTU reservation to your floor — don’t commit on an optimistic forecast.

Glossary

• Token — A sub-word unit (~4 chars of English prose) a model reads and writes; the unit Azure OpenAI bills on.
• Tokenizer — Splits text into tokens via a fixed vocabulary (o200k_base for GPT-4o). tiktoken counts them exactly.
• Input / prompt tokens — Everything you send: system prompt, few-shot examples, history, RAG context, the question.
• Output / completion tokens — Everything the model generates; billed at a higher per-token rate than input.
• Context window — Max tokens (input + output combined) per request; a capacity ceiling, not a billing meter.
• max_tokens — Per-call cap on output length; the cheapest control over the expensive half.
• System prompt — Instructions prepended to every call; billed as input each time, so recurring overhead.
• Conversation history — Prior turns re-sent so a stateless model appears to remember; grows quadratically if unbounded.
• RAG (retrieval-augmented generation) — Retrieving relevant chunks from a search index and pasting them into the prompt as input context.
• Embeddings — Vectors representing text, used to build search/RAG indexes; billed input-only and very cheaply.
• Prompt caching — Discounted input rate for a repeated, identical prompt prefix; requires static-first ordering.
• Batch — Async processing within a ~24-h window at ~half the Standard rate, for non-urgent bulk work.
• Standard (pay-as-you-go) — Per-token billing, no commitment; ideal for variable, low, or unknown volume.
• Provisioned Throughput Unit (PTU) — Reserved capacity at a flat price; predictable at high steady volume, but you pay for idle.
• TPM / RPM — Tokens- and requests-per-minute quota on a deployment; a rate limit capping blast radius (exceed → 429).
• usage object — Per-call response field with exact prompt_tokens, completion_tokens, total_tokens — the ground truth to log.

Next steps

• New to the models? AI-900: Generative AI & Azure OpenAI Fundamentals grounds the concepts behind the economics.
• Productionising a deployment? An Enterprise Landing Zone for Azure OpenAI: Networking, Quotas, and Gateways wraps quotas, private networking, and governance around it.
• Building RAG? Azure AI Search for RAG: Vector Indexing, Hybrid Search, Semantic Ranking, and Indexer Pipelines is the retrieval side of the bill.
• Governing spend across teams? An LLM Gateway for Cost, Safety and Observability centralises metering, routing, and per-team budgets.
• Wider cost discipline: Azure FinOps and Cost Management: Controlling Cloud Spend at Scale puts token spend alongside the rest of your estate.