Azure Document Intelligence Misreads: Low-Confidence Fields, Broken Table Extraction, and Custom Model Drift

The pipeline ran clean for three weeks, then finance flagged it: an invoice for ₹4,18,500 was booked as ₹4,185.00 because the decimal moved, and a supplier name came back as Acme lndia Pvt Ltd with a lowercase L where a capital I should be. Nobody changed the code. The model is the same model. But the supplier started sending scanned PDFs instead of digital ones, and that one change quietly demolished your extraction accuracy. This is the most common production reality of Azure AI Document Intelligence — the managed OCR-plus-structure service (the artist formerly known as Form Recognizer) that turns invoices, receipts, IDs and your own custom forms into typed JSON — and it is maddening because the service almost never errors. It returns a 200 OK with a confident-looking payload that happens to be wrong.

That is the whole problem with debugging document AI: there is rarely an exception to catch. A misread is a successful API call with a bad value inside it, often carrying a confidence score the caller never checked. A broken table is a tables array that parsed — it just put the total in the wrong column. A drifted custom model still returns fields; they’re just blank or shifted on the new document layout the model never saw. The failures hide inside well-formed responses, which is why teams ship them straight to a database and find the damage weeks later in a reconciliation report.

This is the diagnostic playbook for those silent failures. We treat low-confidence fields, broken table extraction and custom-model drift not as three bugs but as three symptom classes, each confirmed with a specific check — the raw JSON, the confidence and spans, the Document Intelligence Studio overlay, the training labels, or the API version you pinned. By the end you stop trusting the green checkmark: when a value looks wrong you will know whether you face a low-DPI scan, a model probing the wrong region, a prebuilt-invoice field that doesn’t exist for your locale, a table whose header merged into the body, or a custom model trained on five clean samples now meeting the real world. Knowing which, in minutes rather than days, is the difference between a contained data-quality issue and a quarter of corrupted records.

What problem this solves

Document Intelligence hides enormous machinery — OCR, layout analysis, key-value association, table reconstruction, and for custom models a trained extractor — so you can POST a PDF and get back typed fields. That abstraction is a gift until the values are subtly wrong, and then it becomes an opaque wall. The service deliberately returns a structured result rather than throwing, because for most callers a best-effort answer with a confidence number is more useful than a hard failure. But that design means the burden of judging correctness is on you, and if you don’t read the confidence, inspect the spans, or look at the bounding regions, you have no idea the answer is wrong until a human downstream catches it.

What breaks without this knowledge: a team pipes fields.InvoiceTotal.content straight into accounts payable with no threshold, so a 0.42-confidence misread posts as fact. Another trains a custom model on a dozen pristine samples, sees 0.98 on the test set, ships it — then the first real batch arrives rotated, at 150 DPI, with a logo the model now treats as an anchor, and extraction collapses. A third expects a Tax field its region’s receipts don’t print, gets null, and assumes the API is broken. None of these throw; all are diagnosable in the raw response; all are ignored because the response said 200.

Who hits this: anyone automating document workflows — AP automation, KYC/onboarding, claims, contract intake, expense management. It bites hardest on teams feeding scanned or photographed documents (DPI, skew and noise dominate accuracy), using custom models trained on too few or too clean samples (drift is near-universal), that never set a confidence threshold (so every misread ships), or that pinned an old API version and don’t realise newer models extract tables and fields differently. The fix is almost never “the model is bad” — it’s “find the stage degrading the answer and feed it, configure it, or threshold it correctly.”

To frame the whole field before the deep dive, here is every symptom class this article covers, the question it forces, and the one place to look first:

Learning objectives

By the end of this article you can:

• Map any Document Intelligence misread to a stage in the analysis pipeline — OCR/read → layout → field extraction → your post-processing — and name the most likely root cause at each stage.
• Read a raw analyzeResult: locate a field’s confidence, its content, its spans into the read results, and its boundingRegions, and use them to prove whether a value is right, wrong, or merely uncertain.
• Diagnose a low-confidence field as a low-DPI/skewed/noisy input, a handwriting vs print mismatch, an unsupported language, or a genuinely ambiguous source — and confirm which.
• Diagnose broken table extraction as merged/spanning cells, missing ruling lines, a header that fell into the body, a multi-page table split, or a borderless table the layout model under-segmented.
• Diagnose custom-model drift by comparing production confidence against the training set, recognise overfitting to too-few/too-clean samples, and decide between a template and a neural custom model.
• Set and enforce a confidence threshold with a human-in-the-loop fallback, so a low-confidence value never ships unreviewed.
• Drive the core diagnostic tools fluently: Document Intelligence Studio, the raw REST analyzeResult JSON, the az cognitiveservices CLI, and the operation-status polling flow.

Prerequisites & where this fits

You should already understand the basics: Document Intelligence is one of the Azure AI Services (the multi-service family formerly branded Cognitive Services), provisioned as a resource with an endpoint and keys (or, better, a managed identity). You should know how to run az in Cloud Shell, read JSON, and that the service exposes prebuilt models (invoice, receipt, ID, business card, layout, read) and lets you train custom models on your own labelled documents. Familiarity with REST APIs, HTTP status codes and async long-running operations (POST to start, GET to poll) helps, because the analyze flow is asynchronous.

This sits in the AI/ML & Observability track. It assumes you have a resource provisioned and a document to analyse; the resource-model and identity fundamentals live in Azure AI Foundry: Hub & Project Resource Model Explained, and securing the keys with identity is covered in Managed Identity: System-Assigned vs User-Assigned Patterns. If you are pushing extracted fields into a search index for retrieval, Azure AI Search: Create Your First Index, Indexer & Skillset is the natural next hop — Document Intelligence is a common skill inside that skillset. For storing the source documents, Azure Blob Storage Fundamentals is where they live, and it is the input to batch analysis.

A quick map of who owns what when a misread lands, so you escalate to the right person fast:

Core concepts

Five mental models make every later diagnosis obvious.

A misread is a successful call with a bad value, not an error. Document Intelligence is best-effort: it almost always returns 200 OK with an analyzeResult, even when it read garbage. The signal of correctness is not the status code — it is the confidence on each field, line and word, and whether the field’s bounding region sits over the pixels you expected. If your code only checks for HTTP errors, you are blind to every misread the service ever makes. The first discipline of this whole topic is: read the confidence, look at the location.

The pipeline has stages, and a misread belongs to exactly one. Every analysis flows OCR/read (turn pixels into words with positions) → layout (group words into lines, paragraphs, tables, selection marks, and detect document regions) → field extraction (map content to typed fields — done by a prebuilt model or your custom model) → your post-processing (threshold, map, validate, store). A wrong character is an OCR problem (fix the input). A total in the wrong column is a layout/table problem. A blank InvoiceTotal on a doc that clearly has one is a field-extraction problem (model or drift). Localising the failure to a stage tells you which knob to turn — and stops you from “retraining the model” when the real fix was a higher-DPI scan.

Prebuilt and custom models fail differently. Prebuilt models (prebuilt-invoice, prebuilt-receipt, prebuilt-read, prebuilt-layout, prebuilt-idDocument, and the general prebuilt-document/key-value model) ship with a fixed, documented field schema and are trained by Microsoft on huge corpora — they fail by not having a field you want (it’s not in their schema or not in your locale) or by mis-mapping on an unusual layout. Custom models you train on your own labelled documents fail by drift: they learned the layout of your training set and degrade on anything that differs (new vendor template, different DPI, rotation). Knowing which kind you’re calling tells you whether the fix is “accept this field doesn’t exist” or “expand the training set.”

Confidence is per-element and bounded 0 to 1 — and your threshold is a product decision. Each field, line and word carries a confidence in [0.0, 1.0]. It expresses the model’s certainty, not a guarantee of correctness — a confident misread is possible (and is its own symptom class). The right pattern is a threshold with a human-in-the-loop fallback: above the threshold, auto-process; below it, route to a person. There is no universal “good” number — a 0.80 cut-off that’s fine for a free-text comment field is reckless for a payment amount. Treating confidence as a binary “it worked” rather than a tunable risk dial is the root cause of most shipped misreads.

Every config — model, API version, page range, locale — changes the answer. The result depends on which model id you call, which API version you pinned (newer versions extract tables and add fields the old one didn’t — the service has moved from the v2.x Form Recognizer API to the v3.x / 2024-11-30 GA Document Intelligence API, with prebuilt-layout gaining far better table handling along the way), the page range you analysed, and the locale/language you hinted. A pipeline that “suddenly” extracts differently usually had its model id or API version changed, or started receiving a document type the chosen model wasn’t built for.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

The model and API-version reference

Before the per-symptom anatomy, here is the lookup table you scan first: which model produces which output, and what each is for. Half of all “the API is broken” tickets are someone calling the wrong model for their document — asking prebuilt-receipt to extract invoice line items, or prebuilt-read (pure OCR, no fields) and wondering where the typed fields went.

Three reading notes that save the most time:

Reading a result: confidence, content, spans, and regions

You cannot debug what you don’t look at, and the field value alone is not enough. Every diagnosis in this article starts by opening the raw analyzeResult and reading four things about the suspicious element: its content (what was recognised), its confidence (how sure), its spans (where in the flat text stream it came from), and its boundingRegions (which page and what polygon — i.e. where on the page). A value that’s wrong-but-confident has the wrong bounding region; a value that’s uncertain has a low confidence; a value that’s missing simply isn’t in the fields object.

The call is asynchronous: you POST to ...:analyze and get a 202 with an Operation-Location header, then GET that URL until status is succeeded (the lab below shows the full POST). Poll and inspect the suspicious field in one slice:

ENDPOINT="https://di-shop-prod.cognitiveservices.azure.com"; KEY="<key>"   # prefer a managed-identity token
# Poll the operation, then read the field the way you must ALWAYS inspect it — value + confidence + region:
curl -s "$ENDPOINT/documentintelligence/documentModels/prebuilt-invoice/analyzeResults/<operationId>?api-version=2024-11-30" \
  -H "Ocp-Apim-Subscription-Key: $KEY" | jq '.status, .analyzeResult.documents[0].fields.InvoiceTotal'

That jq slice returns the field as value and confidence and region together:

{
  "type": "currency",
  "valueCurrency": { "amount": 4185.00, "currencyCode": "INR" },
  "content": "₹4,185.00",
  "confidence": 0.412,
  "spans": [ { "offset": 1842, "length": 9 } ],
  "boundingRegions": [ { "pageNumber": 1, "polygon": [6.1,8.9, 7.0,8.9, 7.0,9.1, 6.1,9.1] } ]
}

Three things to read off this immediately: confidence is 0.412 (well below any sane auto-accept threshold — this should have gone to review, not the ledger); the content is ₹4,185.00 where the source said ₹4,18,500 (the Indian digit grouping confused the parse); and the boundingRegions polygon tells you exactly which pixels to go look at to confirm. The fields you must learn to read, and what each tells you:

The fastest way to see all of this is Document Intelligence Studio (https://documentintelligence.ai.azure.com): upload the document, run the model, and it overlays the bounding boxes on the page with confidence colour-coding. When a value is wrong, the overlay shows you in one glance whether the box is over the right text (an OCR misread) or over the wrong region entirely (a field-mapping or drift problem). Keep Studio open beside the JSON during any investigation.

Symptom 1 — Low-confidence fields

A low-confidence field is the service saying “I read something here, but I’m not sure.” It is the honest failure mode — the value may still be right, but the model is flagging risk, and if you’re not reading confidence you’re throwing that signal away. Five distinct causes. Scan the matrix, then read the detail for whichever row matches:

Cause 1 — Low-DPI, blurry or noisy scans (the dominant cause)

OCR accuracy is dominated by input quality, and resolution is the single biggest lever. A document scanned or photographed at 150 DPI, compressed hard, or full of speckle noise gives the read model fuzzy glyphs, and it returns plausible-but-uncertain characters: 5 read as S, 0 as O, 1 as l, a moved decimal. The field comes back with low word-level confidence because the model genuinely could not see clearly.

Confirm. Look at per-word confidence on the page, not just the field, to see whether the whole image is degraded or just one field:

# Lowest-confidence words on page 1 — a broadly low floor means the SCAN is the problem
curl -s "$ENDPOINT/documentintelligence/documentModels/prebuilt-invoice/analyzeResults/<operationId>?api-version=2024-11-30" \
  -H "Ocp-Apim-Subscription-Key: $KEY" \
  | jq '[.analyzeResult.pages[0].words[] | {content, confidence}] | sort_by(.confidence) | .[0:10]'

In Studio, the page overlay highlights low-confidence words — if most of the page lights up, it’s the image, not the model. If you have to squint to read the pixels at the bounding region, so did the model.

Fix. Fix the input. Source at 300 DPI or higher (Microsoft’s recommended OCR minimum is around 300 DPI; tiny fonts benefit from more). Where you can’t control the source, preprocess — de-noise, raise contrast, and keep the file within the service’s input limits.

Causes 2–5 — skew, handwriting, language, and genuine ambiguity

The other four low-confidence causes share the same diagnostic move — read the right field of the result — so handle them together:

• Skewed/rotated pages (Cause 2). Heavy skew warps the baseline and degrades OCR. Confirm: jq '.analyzeResult.pages[] | {pageNumber, angle, unit}' — a few degrees is auto-corrected, beyond ~10° hurts. Fix: de-skew/auto-rotate before upload; don’t rely on the service for badly tilted scans.
• Handwriting vs print (Cause 3). Handwritten fields legitimately score lower; the model flags them with isHandwritten style spans. Confirm: jq '.analyzeResult.styles[] | select(.isHandwritten == true) | {confidence, spans}'. Fix: set a separate, lower bar for handwriting and route it to review more readily — the lower confidence is correct signalling, not a model defect.
• Unsupported/wrong language (Cause 4). A model forced onto a language it doesn’t support returns a low or nonsensical whole document. Read OCR supports a broad set; prebuilt field models support fewer locales. Confirm: run prebuilt-read (it detects languages) and compare against the model’s supported list. Fix: use a supported model/locale; prebuilt-read covers the widest OCR set.
• Genuinely ambiguous source (Cause 5). A smudged total or overlapping stamp is ambiguous to a human too; low confidence is correct, and there is no model fix. Confirm: look at the pixels at the bounding region. Fix: this is exactly what the human-in-the-loop path is for — and the real fix is upstream, a better source document.

Symptom 2 — Confidently wrong values

The nastier sibling of low confidence: a field comes back with high confidence and the wrong value. The model isn’t flagging risk because it isn’t uncertain — it read something cleanly, just not what you wanted. This is where teams that “trust high confidence” get burned, because a threshold alone won’t catch it. Four causes:

The wrong-region check is the anchor for all four. When confidence is high but the value is wrong, the field read clean text from the wrong place, so the value is irrelevant — the location is the bug. In Studio, click the field and see where its box sits; in JSON, read boundingRegions[].polygon and compare to where the right value lives. The fix is a custom model with the correct field labelled, and/or a post-extraction validation that rejects the impossible. The four variants:

• Wrong region (Cause 1). Grabbed the ship-to total instead of the grand total, or a custom model anchored on a logo that moved. Fix: correct labels + a business rule (total = subtotal + tax).
• Multiple candidates (Cause 2). Two totals/dates on the page; the model picked the salient one wrongly. Confirm: both values appear in content; the box points at the chosen one. Fix: enforce total ≥ subtotal; custom-label the exact field.
• Wrong model (Cause 3). An invoice sent to prebuilt-receipt, or Vendor A’s custom model run on Vendor B. Confirm: jq '.analyzeResult.modelId' ≠ intended. Fix: route to the right model; a custom classifier dispatches mixed batches.
• Locale/format misparse (Cause 4). content is right (07/04/2026) but the typed value parsed in the wrong locale (4 Jul vs 7 Apr). Confirm: jq '.fields.InvoiceDate | {content, valueDate}' — compare the two. Fix: apply the known locale; re-derive from content when they disagree.

Symptom 3 — Broken table extraction

Tables are the hardest thing OCR does, because a table is spatial structure inferred from text positions, not something explicitly tagged in a scanned PDF. When extraction breaks, the tables array still parses — it just has the wrong shape: a header merged into row one, a total in the wrong column, a two-page table split into two tables, or a borderless table under-segmented. Five causes. Remember: tables come from prebuilt-layout (or any model that includes layout) — prebuilt-read returns no tables at all.

Reading a table the right way

The crucial discipline: never address table data by raw row/column number alone — read kind and the span fields. Each cell carries rowIndex, columnIndex, rowSpan, columnSpan, and kind (columnHeader, rowHeader, stubHead, or content). Code that assumes “header is row 0, amount is column 3” shatters the moment a cell spans two columns or the header detection shifts.

# Dump a table's cells with their structure — the only honest way to see what was reconstructed
curl -s ".../analyzeResults/<operationId>?api-version=2024-11-30" -H "Ocp-Apim-Subscription-Key: $KEY" \
  | jq '.analyzeResult.tables[0] | {rowCount, columnCount,
        cells: [.cells[] | {r:.rowIndex, c:.columnIndex, rs:.rowSpan, cs:.columnSpan, kind, content}]}'

The cell anatomy you must understand to debug any table:

The five ways tables break, and the fix for each:

• Merged/spanning cells (Cause 1). A cell with columnSpan/rowSpan > 1 makes the grid non-rectangular, so fixed-index code lands amounts in the wrong field. Fix: read the span fields and map by the header cell’s text (kind == "columnHeader"), never a hardcoded index.
• Borderless under-segmentation (Cause 2). With no ruling lines the layout model infers columns from alignment; tight spacing bleeds columns together and columnCount comes back too low. Fix: use api-version=2024-11-30, whose prebuilt-layout handles borderless tables far better than the old v2.x API; or a custom model for a critical fixed layout.
• Header fell into the body (Cause 3). Detection failed, row 0 holds data and no cell is columnHeader, so “skip row 0” drops a real row. Confirm: no cell has kind == "columnHeader". Fix: find headers by kind, not position.
• Multi-page split (Cause 4). A table crossing a page break returns as two tables entries (each page segmented independently). Confirm: two entries, matching columnCount, adjacent pageNumbers. Fix: stitch the rows in code when column structure matches.
• No tables at all (Cause 5). An empty tables array usually means you called prebuilt-read (OCR only, no tables). Fix: switch to prebuilt-layout — this one-line mistake accounts for a surprising share of “tables return nothing” tickets.

Symptom 4 — Custom-model drift and training problems

A custom model that scored beautifully at training and now returns blank or wrong fields in production hasn’t “broken” — it’s meeting documents that differ from what it learned. This is the most preventable failure class and the one teams most consistently walk into, because the test-set score feels like a guarantee. Five causes:

Cause 1 — Too few training samples

A template custom model can train on as few as five documents, and that’s the trap: five near-identical samples produce a model that memorised that exact layout. It scores ~0.98 on a held-out copy of the same layout and falls apart on real variety. The high training score is overfitting, not quality.

Confirm. Look at how many documents — and how much variety — the model was trained on, versus the variety arriving in production. A handful of samples against dozens of real-world variants is the diagnosis.

# List your custom models and inspect one's build details
az cognitiveservices account list --query "[?kind=='FormRecognizer'].{name:name, rg:resourceGroup}" -o table
# Model details (created date, description, doc types) via REST:
curl -s "$ENDPOINT/documentintelligence/documentModels/<your-model-id>?api-version=2024-11-30" \
  -H "Ocp-Apim-Subscription-Key: $KEY" | jq '{modelId, createdDateTime, description, docTypes: (.docTypes|keys)}'

Fix. Add training samples that span the real variation: every vendor template, both digital and scanned, the DPI range you actually receive, the rotations you see. Quantity matters less than representative coverage.

Cause 2 — trained too clean. You trained on pristine digital PDFs because they were easy to gather, but production is phone-photographed receipts at 150 DPI — the model never learned noise it never saw, so confidence collapses on real inputs. Confirm: compare training input quality (DPI, scanned vs digital, skew) to production. Fix: make the training set look like production.

Cause 3 — Template model meeting layout variation

Custom models build in two modes. A template model is fast and cheap, excellent on a fixed layout but layout-rigid — a new vendor’s arrangement returns blank fields. A neural model tolerates layout variation by learning the meaning of fields, at the cost of longer training. Calling a template model “drifted” when you fed it a new layout is a category error. Fix: rebuild as neural for varying layouts; or one template model per layout plus a classifier that routes. Choose by how much your layouts vary:

Cause 4 — a new production variant. A field that worked for months suddenly returns null because a supplier changed their template or a new vendor was onboarded — drift is often event-driven, not gradual. Confirm: per-field confidence drops sharply for that variant while old ones still work; bucket production confidence by vendor to find it. Fix: add the variant to training and retrain; monitor per-field confidence so you catch the next one the day it arrives, not in a quarterly audit.

Cause 5 — inconsistent/sparse labelling. If you labelled a field in some training docs but missed it in others, or inconsistently (sometimes including the currency symbol, sometimes not), the model learns a muddled signal and the field is flaky from day one — this looks like drift but was a training defect. Confirm: review the labels in Studio — is every occurrence labelled, consistently? Fix: re-label every occurrence with the same boundaries and retrain. Consistent labels matter as much as sample count.

Architecture at a glance

The system that turns a scanned PDF into validated fields is a left-to-right pipeline, and every misread lives at a specific stage on it. Documents land in Blob Storage (the input zone) and are submitted to the Document Intelligence resource over its *.cognitiveservices.azure.com endpoint, authenticated by a managed identity rather than a static key. Inside the service, analysis runs as a chain: the read/OCR stage turns pixels into positioned words (this is where DPI and skew bite — badge 1), the layout stage assembles tables and selection marks (where merged cells and borderless tables break — badge 2), and the field-extraction stage — a prebuilt model or your custom model — maps content to typed fields (where confidence drops and custom models drift — badge 3). The result is an async operation: you POST, receive an Operation-Location, and poll until succeeded.

The right-hand zone is the part teams forget exists and the part that actually protects production: your post-processing. Here you read each field’s confidence against a threshold; anything above it auto-flows to the downstream store, while anything below it — or anything that fails a business-rule validation (total = subtotal + tax) — is routed to a human-in-the-loop review queue (badge 4). Telemetry from every stage flows to Azure Monitor, where per-field confidence over time is the leading indicator that a custom model is drifting (badge 5) long before a human notices bad data. Follow the arrows left to right and you can place any symptom in this article onto exactly one hop — and that placement is the diagnosis.

Real-world scenario

Lumio Retail Finance (the same fictional outfit from our App Service playbook, now automating accounts payable) processed roughly 8,000 supplier invoices a month: invoices arrived by email, landed in a Blob container, and a Function called prebuilt-invoice (pinned at an old v2.1 default from the original build) to push vendor, date, total and line items straight into their ERP. For a quarter it was a quiet success — ~92% auto-posted, the rest kicked to a clerk.

Then reconciliation flagged a cluster of wrong totals, all from one large supplier. First instinct: “the model regressed” — and a support ticket. The real investigation took twenty minutes once they opened the raw analyzeResult instead of consuming InvoiceTotal.content blindly. Three findings stacked up. That supplier had switched from digital PDFs to scanned copies at ~150 DPI; per-word confidence sat around 0.5–0.6 and the Indian digit grouping (₹4,18,500) was misread with a moved decimal — low-DPI OCR (Symptom 1, Cause 1). The supplier’s line-item table was borderless, and the old API under-segmented it, merging the rate and amount columns (Symptom 3, Cause 2). And most damning: the pipeline had no confidence threshold at all — every value, including the 0.41-confidence total, posted to the ERP as fact.

The fix was three changes, none of them “a better model.” A confidence threshold of 0.85 on monetary fields with a human-in-the-loop queue below it (the highest-leverage change — it would have caught every bad total on day one); an upgrade to api-version=2024-11-30, whose prebuilt-layout segmented the borderless table correctly; and working with the supplier to send digital PDFs again, or scan at 300 DPI. Within a week the wrong-total cluster was gone, auto-post actually rose to 94% (the new API extracted more fields cleanly), and the review queue held exactly the documents that needed a human — the low-DPI stragglers — not a random sample of everything. The lesson: the model was never the bug. The bug was trusting a 200 OK without reading the confidence, and pinning an API version that pre-dated the table improvements they needed.

Advantages and disadvantages

Document Intelligence is the right tool for a large class of problems and the wrong tool for a few — knowing which keeps you out of the failure modes above.

Where each matters: the prebuilt models are unbeatable when your documents are standard invoices/receipts/IDs — don’t train a custom model for those. Custom models earn their keep on your proprietary forms, but only if you respect the training discipline (representative, well-labelled samples). The confidence-as-risk-dial advantage is only realised if you use it — a pipeline with no threshold throws away the service’s best safety feature. And the “improves each version” point cuts both ways: it’s an advantage if you track versions and a silent liability if you pin once and forget.

Hands-on lab

A free-tier-friendly walk-through: provision a Document Intelligence resource, analyse a document with a prebuilt model, read the confidence and bounding regions from the raw result, and implement a threshold check. You need an Azure subscription and the Azure CLI.

1. Create a resource (free F0 tier for the lab). F0 allows a low request rate and is perfect for experimenting.

RG="rg-di-lab"; LOC="eastus"; NAME="di-lab-$RANDOM"
az group create -n $RG -l $LOC
az cognitiveservices account create \
  --name $NAME --resource-group $RG --location $LOC \
  --kind FormRecognizer --sku F0 \
  --custom-domain $NAME --yes

2. Get the endpoint and a key. (In production you’d use a managed identity; for the lab a key is fine.)

ENDPOINT=$(az cognitiveservices account show -n $NAME -g $RG --query properties.endpoint -o tsv)
KEY=$(az cognitiveservices account keys list -n $NAME -g $RG --query key1 -o tsv)
echo "$ENDPOINT"

3. Analyse a sample invoice with prebuilt-invoice. Use any invoice URL (or a Microsoft sample). The POST returns 202 with an Operation-Location header.

OP=$(curl -s -D - -o /dev/null -X POST \
  "$ENDPOINT/documentintelligence/documentModels/prebuilt-invoice:analyze?api-version=2024-11-30" \
  -H "Ocp-Apim-Subscription-Key: $KEY" -H "Content-Type: application/json" \
  -d '{"urlSource":"https://raw.githubusercontent.com/Azure-Samples/cognitive-services-REST-api-samples/master/curl/form-recognizer/sample-invoice.pdf"}' \
  | tr -d '\r' | awk '/[Oo]peration-[Ll]ocation:/ {print $2}')
echo "$OP"

4. Poll until succeeded, then read totals with confidence.

# Repeat until status is "succeeded"
curl -s "$OP" -H "Ocp-Apim-Subscription-Key: $KEY" | jq '.status'

# Then read the field with its confidence and region
curl -s "$OP" -H "Ocp-Apim-Subscription-Key: $KEY" \
  | jq '.analyzeResult.documents[0].fields.InvoiceTotal | {content, confidence, boundingRegions}'

Expected output: a JSON object with a content like "$1,098.23", a confidence near 0.9x, and a boundingRegions polygon. That is the shape you must always inspect — value, confidence, and location together.

5. Implement the threshold check (the actual point of the lab). Pull every field and split into auto-accept vs needs-review at 0.85:

curl -s "$OP" -H "Ocp-Apim-Subscription-Key: $KEY" \
  | jq '.analyzeResult.documents[0].fields
        | to_entries
        | map({field:.key, content:.value.content, confidence:.value.confidence,
               decision: (if (.value.confidence // 0) >= 0.85 then "auto" else "review" end)})'

This is the pattern your production code implements: fields at or above the threshold flow downstream; the rest go to a human queue.

6. (Optional) Inspect tables with prebuilt-layout. Re-run the POST against prebuilt-layout:analyze, poll, then dump tables[0].cells to see rowIndex/columnIndex/kind — the structure you debug table breaks with.

7. Teardown. Delete the resource group so nothing lingers.

az group delete -n $RG --yes --no-wait

For repeatability, the resource as Bicep (use a managed identity in real deployments):

resource di 'Microsoft.CognitiveServices/accounts@2024-10-01' = {
  name: 'di-shop-prod'
  location: location
  kind: 'FormRecognizer'
  sku: { name: 'S0' }            // S0 standard for production; F0 free for the lab
  identity: { type: 'SystemAssigned' }   // identity over keys
  properties: {
    customSubDomainName: 'di-shop-prod'
    publicNetworkAccess: 'Disabled'      // pair with a private endpoint
  }
}

Common mistakes & troubleshooting

This is the centerpiece — the playbook you keep open while a misread is live. Each row is a real failure: symptom → root cause → how to confirm (exact command / Studio path) → fix. Scan to your symptom, then read the per-symptom detail below for the ones that bite hardest.

The error/status codes you’ll see, separated from the (more common) silent misreads:

Three points from that table deserve emphasis because teams get them wrong most often. Row 1 is the one that matters most: a wrong value ships for weeks only because the pipeline consumed fields.<F>.content with no confidence threshold — pull the historical result and the confidence was below any sane cut-off the whole time. A threshold with a human-in-the-loop queue (start ~0.80–0.85 for money/identity fields, tune from the review queue’s hit rate) is the single highest-leverage change in this article. For broken tables (rows 10–13), the universal fix is to address cells by kind and header text, never by raw index, and to upgrade to api-version=2024-11-30 for far better borderless handling. For drift (rows 15–18), the operational fix is to monitor per-field confidence by vendor so you catch the next new variant the day it arrives — drift is event-driven, and a quarterly reconciliation is the most expensive way to find it.

Best practices

• Always set a confidence threshold with a human-in-the-loop fallback. This is the single most important rule. Auto-accept above it, route below it to a person. A pipeline with no threshold throws away the service’s best safety feature and ships every misread.
• Never trust a 200 as “correct.” A successful call is best-effort. Read confidence and, for anything that matters, verify the boundingRegions sit over the right pixels.
• Address tables by kind and header text, not by raw row/column index. Merged cells and shifted headers break index-based code instantly; map values to their columnHeader cell.
• Pick the right model for the document. prebuilt-read for OCR only, prebuilt-layout for tables/structure, a field prebuilt for known doc types, a custom model only for your proprietary forms. Half of “broken” tickets are the wrong model id.
• Pin and track the API version. Use the current GA (2024-11-30); review when you upgrade, because newer versions extract more fields and far better tables — but don’t let a pinned-once version silently miss those gains.
• Train custom models on production-representative data. Cover every template, both scanned and digital, the real DPI range and rotations. Variety beats volume; a handful of clean samples will overfit.
• Label consistently — every occurrence, same boundaries. Inconsistent labels create fields that are flaky from day one and masquerade as drift.
• Prefer neural custom models when layouts vary, template models only for genuinely fixed layouts; add a classifier to route mixed batches.
• Validate with business rules. Total = subtotal + tax; dates within range; amounts non-negative. Rules catch confident misreads that a threshold alone cannot.
• Monitor per-field confidence over time. Drift is often event-driven (a new vendor template). Alert on a confidence drop per field/vendor so you catch it the day it starts, not in a quarterly audit.
• Fix accuracy at the source first. Higher-DPI scans, de-skewing and digital originals beat any downstream tweak. The cheapest accuracy gain is a better input.
• Use managed identity, not keys. Authenticate to the endpoint with the app’s identity and a least-privilege data-plane role; keep keys out of code and config.

The signals worth wiring before the next bad-data incident — leading indicators, not a quarterly reconciliation:

Security notes

• Authenticate with managed identity, not keys. Disable local (key) auth where possible and grant the app’s system- or user-assigned managed identity the Cognitive Services User data-plane role. Keys in code or config are the most common leak vector for AI Services.
• Lock down the network. Set publicNetworkAccess: Disabled and reach the resource over a private endpoint from your VNet, so the endpoint isn’t internet-exposed. Restrict the Blob container holding source documents the same way.
• Mind the data you send. Invoices, IDs and forms are often PII/PHI. Know your data residency: pick a region that meets your compliance needs, and prefer private networking so documents never traverse the public internet. If you must redact, do it before submission.
• Scope SAS tightly when using urlSource. If you pass a SAS URL instead of identity-based access, make it read-only, short-lived, and IP-scoped — a broad SAS on a document store is an exfiltration path.
• Secure the human-in-the-loop queue. The review tool sees the same sensitive documents; apply the same access control, auditing and least privilege there as on the pipeline.
• Don’t log raw content. Extracted fields can contain account numbers and personal data; keep them out of plaintext logs and traces, and scrub before sending telemetry to Monitor.
• Pin and scan custom-model training data. The labelled training set is sensitive (real customer documents); store it in a locked-down container with auditing, and purge what you no longer need.

The security knobs that also reduce these incidents — secure and reliable pull the same direction:

Cost & sizing

The bill is driven by pages analysed, not API calls, and which model you use:

• Pricing is per page (per 1,000 pages), and custom/prebuilt field models cost more per page than plain prebuilt-read OCR. So routing every document through an expensive field model when you only need text is overspend — use prebuilt-read for OCR-only needs and the heavier models only where you need fields/tables.
• The free F0 tier is for development and small volumes — a low monthly page allowance and a tight request rate (the source of 429s). It is not a production tier; move to S0 (standard) for any real throughput, where you pay per page with no hard daily cap.
• Custom model training and storage have their own (modest) costs, but the bigger cost lever is re-training discipline: retraining repeatedly on poorly-chosen samples wastes effort more than money. Train representative once, monitor, and retrain on real drift.
• The human-in-the-loop queue has a labour cost, and your threshold tunes it directly: too high a threshold floods the queue (people reviewing fine documents); too low ships misreads (expensive downstream corrections). Tune the threshold to the cheapest total of (review labour + correction cost).
• Adjacent costs: Blob storage for source documents (cheap, but lifecycle-manage old ones), egress if processing cross-region (keep the resource and blobs co-located), and Monitor ingestion for the confidence telemetry.

A rough monthly picture for Lumio’s ~8,000 invoices/month (multi-page, so more analysed pages): an S0 resource billed per page for prebuilt-invoice, plus negligible Blob storage, plus the labour of reviewing the ~6–8% below threshold. The dominant cost was never the API — it was the clerk-hours, which the right threshold minimised by sending only the genuinely-ambiguous documents to review. The cost drivers and what each one buys:

Interview & exam questions

1. Why is a Document Intelligence misread usually not an exception, and what does that imply for your code? The service is best-effort and returns 200 OK with an analyzeResult even when it read wrongly — a misread is a successful call with a bad value inside. The implication is that checking only for HTTP errors is blind to every misread; your code must read each field’s confidence and, where it matters, verify the boundingRegions sit over the right pixels.

2. What is a confidence threshold and why is a human-in-the-loop fallback essential? A confidence threshold is your cut-off for auto-accepting a field versus routing it to a person. Above it you auto-process; below it a human reviews. It’s essential because confidence is a risk dial, not a guarantee — without the fallback, every low-confidence value (and any misread) ships unreviewed straight into downstream systems.

3. Difference between prebuilt-read and prebuilt-layout? prebuilt-read is OCR only — text, lines, words, language — with no tables and no typed fields. prebuilt-layout adds tables, selection marks and document structure. If you call read and wonder where the tables went, that’s why; switch to layout for any structure.

4. A field comes back with high confidence but the wrong value. What’s happening and how do you confirm? The model read clean text from the wrong location — it bound the field to the wrong region (e.g. subtotal instead of total, or a custom model anchored on something that moved). Confirm with the field’s boundingRegions.polygon (or click it in Studio) and see the box is over the wrong area. Fix with a correctly-labelled custom model and/or a business-rule validation.

5. Why might table extraction put values in the wrong column, and how do you make consuming code robust? Merged/spanning cells (columnSpan/rowSpan > 1) shift the grid so fixed-index code reads the wrong column, and header detection can fail so row 0 holds data. Make code robust by reading each cell’s kind (columnHeader vs content) and span fields, and mapping values to their header text rather than a hardcoded column index.

6. What is custom-model drift and what most commonly causes it? Drift is a custom model returning blank or wrong fields in production despite a high training score, because production documents differ from the training set. The most common causes are too few or too-clean training samples (overfitting), a template model meeting a new layout, and a new document variant (a vendor changing their template). The fix is representative training data and, for varying layouts, a neural model.

7. Template vs neural custom models — when do you pick each? A template model is fast/cheap, trains on as few as ~5 docs, and is excellent on a single fixed layout but rigid (blank on a new layout). A neural model tolerates layout variation by learning field meaning, at the cost of longer training. Pick template for consistent fixed forms, neural for varying layouts of the same document type.

8. A custom model scored 0.98 on its test set but performs badly in production. What likely went wrong? It overfit to a small, non-representative training set — a handful of clean, near-identical samples. The 0.98 reflects memorising that exact layout/quality, not generalisation. Fix by training on production-representative documents (multiple templates, scanned and digital, real DPI range) and monitoring per-field confidence.

9. How do you handle a table that spans two pages? A multi-page table often returns as two separate tables entries, one per page. Detect it by matching columnCount and adjacent pageNumbers in the cells’ bounding regions, then stitch the rows together in code, validating that the column headers match.

10. You suddenly see 429 TooManyRequests. What is it and how do you respond? You exceeded the tier’s request rate — the F0 free tier is especially small. The response includes a Retry-After header. Respond by honouring Retry-After, adding exponential backoff and a submission queue, and moving from F0 to S0 for real throughput.

11. The model returns null for a field you expected. Is the API broken? Usually no — either the field isn’t in that model’s schema (e.g. asking a receipt model for an invoice-only field), or your locale’s documents don’t print that field. Confirm against the model’s documented field list; if it’s not there or not on the document, null is correct behaviour, not a bug.

12. How do you authenticate to Document Intelligence securely, and what role is needed? Use the application’s managed identity (system- or user-assigned) rather than a static key, disable local auth where possible, and grant the identity the Cognitive Services User data-plane role. Pair with a private endpoint so the resource isn’t internet-exposed. This keeps keys out of code and the endpoint off the public internet.

These map to AI-102 (Azure AI Engineer Associate) — plan and manage an Azure AI solution; implement knowledge mining and document intelligence (Document Intelligence/Form Recognizer, prebuilt and custom models, confidence handling) — and touch AZ-204 where it’s wired into an app, and AI-900 for the foundational “what is Document Intelligence” concepts. A compact cert mapping:

Quick check

1. A field returns 200 OK with content that’s clearly wrong. Why didn’t the API throw, and what’s the one property you check first to catch this in code?
2. Your prebuilt-layout table puts the amount in the wrong column. Name the two cell properties you must read to make your consuming code robust (instead of using a fixed column index).
3. True or false: a custom model that scores 0.98 on its test set is safe to ship to production.
4. A field that extracted correctly for months suddenly returns null for one supplier’s invoices. What’s the most likely cause and the fix?
5. You’re getting 429 TooManyRequests. What header tells you when to retry, and what’s the production fix beyond backoff?

Answers

1. The service is best-effort and returns a successful response with a confident-looking value even when it read wrongly — a misread is a 200, not an error. Check the field’s confidence first (and verify boundingRegions for important fields); a threshold on confidence is what catches it.
2. Read the cell’s kind (columnHeader vs content) and its columnSpan/rowSpan. Map values to their header text found via kind, not by a hardcoded column index, so merged/spanning cells don’t shift your mapping.
3. False. A high test score on a small, non-representative set usually means overfitting — the model memorised that layout/quality. It can still fail on real-world variety. Train on production-representative data and monitor per-field confidence.
4. A new document variant — that supplier changed their invoice template, which the custom model never saw, so its fields don’t map and confidence collapses. Fix by adding the new variant to the training set and retraining, and monitor per-field/per-vendor confidence to catch the next one immediately.
5. The Retry-After header tells you how long to wait. Beyond honouring it with exponential backoff, the production fix is to move off F0 to S0 (and queue submissions), because the free tier’s request rate is the constraint.

Glossary

• Azure AI Document Intelligence — the managed service (formerly Form Recognizer) that extracts text, tables, structure and typed fields from documents via prebuilt and custom models.
• analyzeResult — the JSON envelope returned by an analysis, containing pages, tables, documents/fields, styles and paragraphs; the source of truth for debugging.
• Confidence — a per-field/line/word certainty score in [0.0, 1.0]; the model’s built-in risk signal, not a guarantee of correctness.
• Confidence threshold — your chosen cut-off above which a field auto-processes and below which it routes to human review.
• Human-in-the-loop (HITL) — the review path where a person validates below-threshold or rule-failing extractions before they’re accepted.
• content — the recognised text of an element; what you usually consume, and what can be confidently wrong.
• spans — offset + length into the flat content text stream, letting you trace a field back to its recognised source text.
• boundingRegions — the page number and polygon describing where on the page an element sits; proves the model read the right location.
• Prebuilt model — a Microsoft-trained, fixed-schema extractor (prebuilt-invoice, prebuilt-receipt, prebuilt-read, prebuilt-layout, prebuilt-idDocument, etc.).
• Custom model — a model you train on your own labelled documents to extract your specific fields.
• Template model — a custom build mode that is fast and layout-rigid; excellent on one fixed layout, blank on new ones.
• Neural model — a custom build mode that tolerates layout variation by learning field meaning; slower to train, more robust.
• Custom classifier — a model that classifies a document’s type so you can route it to the right extractor.
• prebuilt-read — the OCR-only model: text, lines, words, language — no tables, no typed fields.
• prebuilt-layout — the structure model: text plus tables, selection marks and document layout.
• Selection mark — a checkbox/radio detected as selected or unselected in the layout output.
• Drift — degradation of a custom model’s accuracy when production documents differ from the training set (new template, DPI, rotation).
• DPI (dots per inch) — scan resolution; the dominant lever on OCR accuracy (target ≥ 300 DPI).
• API version — the dated service contract (api-version, e.g. 2024-11-30) that determines which fields and table improvements you get.
• Document Intelligence Studio — the web tool (documentintelligence.ai.azure.com) for running models, labelling custom-model data, and viewing bounding-box overlays with confidence.

Next steps

You can now localise any Document Intelligence misread to a pipeline stage and fix it. Build outward:

• Next: Azure AI Search: Create Your First Index, Indexer & Skillset — Document Intelligence is a common skill inside a skillset; feed your extracted fields into retrieval.
• Related: Azure AI Foundry: Hub & Project Resource Model Explained — where the AI Services resource model and identity live.
• Related: Managed Identity: System-Assigned vs User-Assigned Patterns — authenticate to the endpoint without keys.
• Related: Azure Blob Storage Fundamentals — where source documents live and the input to batch analysis.
• Related: Azure AI Language Service: PII, Sentiment, Entity NER & CLU — the text-analytics companion for the content you’ve extracted.