A deployment that ran clean in staging throws AuthorizationFailed the moment it touches production storage, and the only thing that changed was the subscription. Or a Function that read a Key Vault secret for six months suddenly logs ManagedIdentityCredential authentication failed at 03:00 with no code change. Or a brand-new VM extension can’t reach 169.254.169.254 at all and the deploy hangs for two minutes before failing. These are the three faces of the same problem: managed identity — the Entra ID identity Azure attaches to a resource so it can call other Azure services without a secret in your code — looks like magic until the token doesn’t come back, and then it becomes the most confusing failure in the platform because two completely different layers can each say “no” and they say it with different status codes.
The first layer is authentication: can the resource even get a token? That is a conversation between your code, the local IMDS (Instance Metadata Service) endpoint at 169.254.169.254 (on VMs and VMSS) or the App Service / Functions token endpoint (IDENTITY_ENDPOINT), and Entra ID. When this fails you get a 401 Unauthorized, a timeout, or a ManagedIdentityCredential error — no token was issued. The second layer is authorization: you got a perfectly valid token, presented it to (say) Storage or Key Vault, and that service checked its RBAC assignments or access policy and decided your identity isn’t allowed. That is a 403 Forbidden / AuthorizationFailed — the token was fine, the permission wasn’t. Telling these two apart in the first ninety seconds is the entire game, because the fixes live in opposite places: a 401/IMDS problem is fixed on the source (enable the identity, pass the right client-id, fix the endpoint), while a 403 is fixed on the target (grant the role at the right scope and wait for it to propagate).
This is the diagnostic playbook for both. You will learn to read the token-acquisition path — your code → DefaultAzureCredential / ManagedIdentityCredential → IMDS or IDENTITY_ENDPOINT → Entra ID → a JWT scoped to a resource → the target’s RBAC check — and to localise any failure to one hop, using the tools that tell the truth: curl against IMDS, az account get-access-token, the decoded JWT claims (aud, oid, appid), and az role assignment list. Every diagnosis comes with the exact command to confirm it and the precise fix, in both az CLI and Bicep. Because this is a reference you reach for mid-incident, the playbook, error strings, environment variables and credential behaviours are all laid out as scannable tables. Read the prose once; keep the tables open at 03:00.
What problem this solves
Managed identity removes the single worst thing in cloud security: a long-lived secret sitting in config that someone eventually leaks to git. In exchange you get an identity that is implicit — there is no connection string, no client secret, nothing you can echo to see what’s wrong. When it works you forget it exists. When it breaks, the abstraction becomes a wall: the token grab happens inside an SDK credential class, against a link-local IP that isn’t routable, validated by a directory you can’t see into, and the error surfaces three layers away as a generic 401 or 403 with a message that names neither the identity nor the missing role clearly.
What breaks without this knowledge: an engineer sees AuthorizationFailed and assumes the code is wrong, or sees ManagedIdentityCredential authentication failed and assumes the identity wasn’t assigned, and the two get conflated constantly. People grant Owner to make a 403 go away (a real and common over-grant), or recreate the identity to fix a 401 that was actually an audience mismatch, or chase a phantom network problem when the real issue is that DefaultAzureCredential silently fell through managed identity to a stale developer login. Meanwhile the actual cause — a user-assigned identity whose client_id was never passed, a role assigned at the wrong scope, an RBAC change that simply hasn’t propagated yet, or an aud claim for the wrong resource — sits there, perfectly diagnosable, ignored.
Who hits this: essentially everyone running Azure compute that talks to Azure data and secrets. It bites hardest on App Service and Functions apps reading Key Vault references (boot-time token failures that look like random crashes), VMs and VMSS with extensions or agents calling IMDS (network and retry issues), anyone using user-assigned managed identity (the client_id ambiguity is near-universal the first time), multi-identity resources (which identity did the SDK actually pick?), and freshly-deployed infrastructure where the role assignment is correct but Entra ID replication lag means it isn’t live yet. The fix is almost never “grant Owner” — it’s “find which of the two layers said no, and make exactly that layer say yes.”
To frame the whole field before the deep dive, here is every symptom class this article covers, the question it forces, and the one place to look first:
| Symptom class | Which layer is failing | First question to ask | First place to look | Most common single cause |
|---|---|---|---|---|
| 401 / no token | Authentication (token never issued) | Did the identity even get a token? | az account get-access-token on the resource; IDENTITY/IMDS reachable? |
Identity not enabled, or wrong client-id passed |
| IMDS timeout / unreachable | Authentication (endpoint path) | Can the host reach 169.254.169.254? |
curl IMDS with Metadata:true; check NIC/proxy/UDR |
Proxy or firewall hijacking link-local; container hop limit |
| IMDS 400 Bad Request | Authentication (request shape) | Is the resource/client_id parameter right? |
The IMDS response body (it names the problem) | Missing resource=, or ambiguous identity not disambiguated |
| 403 / AuthorizationFailed | Authorization (token was valid) | Got a token, but is the identity granted? | az role assignment list --assignee <oid> at the target scope |
No role, wrong scope, or not yet propagated |
Wrong audience (aud) |
Authentication → rejected by target | Was the token minted for the right resource? | Decode the JWT aud claim |
Asked for the wrong scope/resource URI |
| Wrong identity picked | Authentication (ambiguity) | On a multi-identity resource, which one ran? | Decode oid/appid from the token |
DefaultAzureCredential picked system-assigned, not your UAMI |
Learning objectives
By the end of this article you can:
- Split any managed-identity failure into authentication (no token: 401/IMDS) versus authorization (token rejected: 403/
AuthorizationFailed) within ninety seconds, and name the likely cause for each. - Drive the token endpoints by hand —
curlagainst IMDS on VMs and the App ServiceIDENTITY_ENDPOINT/IDENTITY_HEADERpair — and read the response body, which usually names the problem. - Decode a JWT’s
aud,oidandappidclaims to prove which identity got the token and which resource it was minted for. - Diagnose the user-assigned
client_idambiguity: why a multi-identity resource needsclient_id/object_id/mi_res_idpassed explicitly, and howDefaultAzureCredentialandManagedIdentityCredentialdiffer. - Fix a 403 by granting the right RBAC role at the right scope (data-plane, not
Owner), and account for Entra propagation delay so you don’t “fix” something already correct. - Confirm the most common boot-time failure — an App Service / Functions Key Vault reference that can’t resolve — and trace it to identity, role, firewall, or a bad secret URI.
- Read the canonical error-string, environment-variable and credential-behaviour tables and reach for the right diagnostic per symptom.
Prerequisites & where this fits
You should already understand what a managed identity is: an Entra ID service principal that Azure creates and lifecycle-manages for a resource, in two flavours — system-assigned (1:1 with the resource, dies with it) and user-assigned (a standalone resource you attach to one or many compute resources). If those two are fuzzy, read Managed Identities Demystified: System vs User-Assigned and When to Use Each first — this article assumes you know the difference and focuses entirely on why the token fails. You should be able to run az in Cloud Shell, read JSON output, and understand HTTP status codes at a basic level.
You should also know that RBAC (role-based access control) is how most Azure services authorise a token: a role definition (a set of allowed actions) is bound to a principal (your identity) at a scope (management group / subscription / resource group / resource). If AuthorizationFailed and “scope” aren’t second nature, Azure RBAC Fundamentals: Roles, Scopes and Assignments Without the Confusion is the upstream primer, and Troubleshooting AuthorizationFailed: Reading Effective Azure RBAC Permissions is its dedicated troubleshooting companion — this article overlaps it only where RBAC is the cause of a managed-identity 403.
This sits in the Identity & Troubleshooting track. It is the authentication-and-authorization layer beneath nearly every other Azure integration. It pairs tightly with Azure Key Vault: Secrets, Keys and Certificates Done Right and Key Vault 403 Forbidden: Untangling Firewall, RBAC, Purge Protection, and Soft-Delete Recovery, because Key Vault is the single most common target a managed identity calls. For the broader token model (what a JWT is, what scope and aud mean across OAuth2 flows), OIDC and OAuth2 on Entra ID: Choosing the Right Flow (Auth Code, PKCE, Client Credentials) is the conceptual backstop, and App Registrations vs Enterprise Applications: The Service Principal Model Explained explains the service-principal object the identity actually is.
A quick map of who confirms what during an incident, so you call the right person fast:
| Layer | What lives here | Who usually owns it | Failure classes it can cause |
|---|---|---|---|
| Your code / SDK credential | DefaultAzureCredential, client_id config |
App / dev team | Wrong identity picked, fell through to dev login |
| Local token endpoint (IMDS / IDENTITY) | 169.254.169.254, IDENTITY_ENDPOINT |
Platform + host networking | 401, timeout, IMDS 400 |
| Identity assignment on the resource | System/user-assigned enabled & attached | App + platform | 400 “identity not found”, no token |
| Entra ID | Issues the JWT, replicates assignments | Microsoft (directory) | Propagation lag, disabled principal |
| Target service RBAC / policy | Role assignments, access policies, scope | Resource owner | 403 / AuthorizationFailed |
| Target service network | Firewall, private endpoint on the target | Network team | Looks like 403/timeout, is really network |
Core concepts
Five mental models make every later diagnosis obvious.
Authentication and authorization are different layers that fail differently. Authentication is “get a token”; authorization is “present it and be allowed.” A 401 (or an IMDS timeout, or ManagedIdentityCredential authentication failed) means authentication failed — no token was issued, so there’s nothing to check permissions against. A 403 / AuthorizationFailed means authentication succeeded — you hold a valid token — and the target refused it on permission grounds. This is the first fork in every decision tree: did I get a token at all? If no → fix the source. If yes but rejected → fix the target’s RBAC.
The token comes from a local endpoint, not from your app calling Entra directly. Your code never holds an Entra client secret. The SDK credential asks a local endpoint for a token, and that endpoint (trusted by the platform) talks to Entra on your behalf. On a VM or VMSS it is IMDS — the Instance Metadata Service at the link-local 169.254.169.254, reached at /metadata/identity/oauth2/token with the header Metadata: true. On App Service and Functions it is the IDENTITY_ENDPOINT (an http://127.0.0.1:<port>/... URL injected as an env var) authenticated with the secret in IDENTITY_HEADER. Knowing which endpoint your workload uses tells you exactly what to curl and what can break it.
A token is for one resource, and the aud claim proves it. A managed-identity token is requested for a specific target resource — https://storage.azure.com, https://vault.azure.net, https://management.azure.com, etc. (or the equivalent .default scope in newer SDKs). Entra mints a JWT whose aud (audience) claim is that resource. Ask for the wrong resource and you get a perfectly valid token the real target rejects on audience grounds — a surprising share of “it returns 401/403 but I clearly have a token” incidents. The fix is to request the right scope, not to touch RBAC.
A user-assigned identity is ambiguous unless you name it. A resource can have one system-assigned identity and any number of user-assigned identities at once. When more than one is present, the token endpoint cannot guess which you mean — you must pass a disambiguator: the identity’s client_id (most common), its object_id (the principal’s oid), or its full mi_res_id (the ARM resource id). Omit it on a multi-identity resource and you get an IMDS 400 (“specify the client_id”) — or, worse, the SDK silently picks the wrong one. This single ambiguity is the most common user-assigned failure there is.
RBAC propagation is eventually consistent — “correct” and “live” aren’t the same instant. A new role assignment is written to Entra and then replicated to the enforcement points, and the target caches assignments too. So a freshly-granted role can be correct in the portal yet still 403 for several minutes (commonly up to ~5, longer for cross-tenant/cross-region targets); a removed role can keep working briefly. “I just granted it and it still fails” is usually lag, not a mistake — az role assignment list shows the assignment exists, the call still 403s, and a wait-and-retry clears it.
The vocabulary in one table
Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:
| Concept | One-line definition | Where it lives | Why it matters to 401/403 |
|---|---|---|---|
| System-assigned identity | Identity tied 1:1 to one resource | On the resource (identity.type) |
Simple; only one, so no client_id needed |
| User-assigned identity (UAMI) | Standalone identity attached to many resources | Its own ARM resource | Multiple → must pass client_id |
| IMDS | Local metadata/token endpoint on VMs/VMSS | 169.254.169.254 (link-local) |
Token source for VMs; network-fragile |
| IDENTITY_ENDPOINT | App Service/Functions token endpoint | Injected env var (127.0.0.1) |
Token source for PaaS apps |
aud (audience) |
The resource a token is minted for | Inside the JWT | Wrong aud → valid token, still rejected |
oid (object id) |
The principal id of the identity | Inside the JWT / principalId |
The thing RBAC assigns roles to |
appid / client_id |
The app/client id of the identity | Inside the JWT / identity props | Disambiguates which UAMI; in logs |
| Role assignment | Role bound to a principal at a scope | On the target / scope | Missing/wrong scope → 403 |
| Scope | Where a role applies (RG, sub, resource) | The assignment | Too narrow/broad → 403 or over-grant |
DefaultAzureCredential |
SDK chain that tries several auth sources | In your code | Can fall through to dev login / wrong identity |
ManagedIdentityCredential |
SDK credential that uses only MI | In your code | Explicit; fails loudly if MI absent |
| Propagation delay | Lag before a role assignment is enforced | Entra ID → enforcement | New grant 403s for minutes |
The error and status-code reference
Before the per-symptom anatomy, here is the lookup table you scan first: every error string and status code you realistically see from managed-identity token acquisition, what it actually means, the likely cause, how to confirm it, and the first fix. The two that trip people most are the ManagedIdentityCredential authentication failed wrapper (an authentication failure that can mean five different things underneath) and the difference between a 401 from the token endpoint and a 403 from the target.
| Error / code | Where it surfaces | Likely cause | How to confirm | First fix |
|---|---|---|---|---|
AuthorizationFailed (403) |
ARM / target service response | Token valid, but no RBAC role at this scope | az role assignment list --assignee <oid> --scope <id> |
Grant the least-privilege role at the right scope |
| 403 Forbidden (Key Vault / Storage) | Data-plane call | No data-plane role, or target firewall blocked you | Decode token aud; check role + target networking |
Grant data-plane role; allow the caller on the target |
| 401 Unauthorized | Token endpoint or target | No token issued, expired, or wrong aud |
az account get-access-token; decode aud/exp |
Re-acquire; request the correct resource scope |
ManagedIdentityCredential authentication failed |
SDK (Azure.Identity) | Identity not enabled, endpoint unreachable, wrong client-id, IMDS timeout | Inner exception + curl the endpoint |
Enable identity / pass client_id / fix network |
| IMDS 400 Bad Request | curl/SDK to IMDS |
Missing resource=, or ambiguous identity not specified |
IMDS response body names it | Add resource=; pass client_id |
| IMDS timeout / connection refused | curl/SDK to IMDS |
Proxy/firewall hijack of link-local, UDR, container hop limit | curl -s -m 5 169.254.169.254/... hangs |
Bypass proxy for 169.254.169.254; fix routing |
No managed identity endpoint found |
SDK | No MI available in this environment | Running off-Azure, or identity not assigned | Run on Azure with an identity, or use another credential |
Identity not found (400) |
IMDS / ARM | Named client_id/mi_res_id not attached to this resource |
az <res> identity show lists attached UAMIs |
Attach the identity, or pass the right id |
DefaultAzureCredential failed to retrieve a token |
SDK | Every credential in the chain failed | Read the per-credential reasons in the message | Fix the intended credential; exclude the rest |
InvalidAuthenticationTokenAudience |
Target service | Token aud doesn’t match this service |
Decode aud; compare to the service’s resource URI |
Request the correct resource/scope |
403 AccountIsDisabled / principal disabled |
Target / Entra | The identity’s service principal is disabled | az ad sp show --id <appid> accountEnabled |
Re-enable the principal; recreate the identity |
Caller is not authorized + RoleAssignment |
Granting RBAC itself | You lack rights to grant the role | Your own role at that scope | Get User Access Administrator/Owner to grant |
Three reading notes that save the most time:
| Distinction | The trap | How to tell them apart |
|---|---|---|
| 401 (auth) vs 403 (authz) | Treating “no permission” and “no token” as the same bug | 401 / IMDS error = no token issued → fix the source; 403 / AuthorizationFailed = token valid → fix the target’s RBAC |
AuthorizationFailed (ARM) vs data-plane 403 (Key Vault/Storage) |
Granting a control-plane role and expecting data-plane access | AuthorizationFailed is the ARM management plane; a Key Vault/Storage 403 needs a data role (e.g. Key Vault Secrets User, Storage Blob Data Reader) — Owner ≠ Storage Blob Data Reader for data unless data actions are included |
| Real 403 vs propagation lag | “I granted it and it still fails” → re-granting again | If az role assignment list already shows the role, it’s lag, not a missing grant — wait and retry, don’t grant twice |
Anatomy of authentication: when no token is issued (401 / IMDS)
This whole class means the same thing: you never got a token. Five distinct causes. Scan the matrix, then read the detail for whichever row matches:
| # | Auth failure cause | Tell-tale signal | Confirm with | Real fix | Band-aid that masks it |
|---|---|---|---|---|---|
| 1 | Identity not enabled / not attached | “No managed identity endpoint”, or IMDS “identity not found” | az <res> identity show is empty |
Enable system- or attach user-assigned | None — it never works |
| 2 | Ambiguous identity (multi-UAMI) | IMDS 400 “specify the client_id” | curl IMDS without client_id 400s |
Pass client_id/object_id/mi_res_id |
Remove all but one identity (often wrong) |
| 3 | Wrong client_id passed |
400 “identity not found” for that id | az <res> identity show lacks that id |
Pass an attached identity’s id | Hard-code another id (still wrong) |
| 4 | IMDS unreachable (network) | curl to 169.254.169.254 times out |
curl -s -m 5 hangs/refused |
Bypass proxy for link-local; fix UDR/hop | Add retries (masks intermittency) |
| 5 | Wrong audience / resource | Token issued, target says 401/InvalidAudience |
Decode aud; ≠ target resource URI |
Request the correct resource scope | Touch RBAC (it’s not the problem) |
Cause 1 — The identity isn’t enabled or isn’t attached
The token endpoint can only mint a token for an identity the resource actually has — if you never enabled system-assigned and never attached a user-assigned identity, there’s nothing to authenticate as. On a VM the SDK reports No managed identity endpoint found; on App Service the IDENTITY_ENDPOINT env var simply isn’t injected.
Confirm. Ask the resource what identities it has:
# What identities does this resource carry? Empty/None = nothing to authenticate as.
az webapp identity show --name app-orders-prod --resource-group rg-orders-prod -o json
# For a VM:
az vm identity show --name vm-batch-prod --resource-group rg-batch-prod -o json
A null / empty response (or "type": "None") is the smoking gun. On App Service you can also check from inside the app whether the endpoint exists:
# In the app's Kudu/SSH console — if IDENTITY_ENDPOINT is unset, no MI is wired up
printenv | grep -i IDENTITY
Fix. Enable a system-assigned identity, or attach a user-assigned one:
# System-assigned on a web app
az webapp identity assign --name app-orders-prod --resource-group rg-orders-prod
# Or attach an existing user-assigned identity
az webapp identity assign --name app-orders-prod --resource-group rg-orders-prod \
--identities /subscriptions/<sub>/resourceGroups/rg-identity/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-orders
resource site 'Microsoft.Web/sites@2023-12-01' = {
name: 'app-orders-prod'
location: location
identity: {
type: 'SystemAssigned' // or 'UserAssigned', or 'SystemAssigned, UserAssigned'
// userAssignedIdentities: { '${uami.id}': {} } // when attaching a UAMI
}
properties: { serverFarmId: plan.id }
}
Enabling the identity creates the service principal, but it has no permissions yet — that’s a separate step (the 403 section below). Enabling identity fixes authentication; it does nothing for authorization.
Cause 2 — The resource has multiple identities and you didn’t say which
The number-one user-assigned failure. The moment a resource has more than one identity — system-assigned plus a user-assigned, or two user-assigned — the token endpoint cannot infer your intent. IMDS returns a 400 whose body is explicit:
{"error":"invalid_request","error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"}
Confirm. List the attached identities; if there’s more than one (or one UAMI and system-assigned), ambiguity is your cause:
az webapp identity show --name app-orders-prod --resource-group rg-orders-prod \
--query "{type:type, userAssigned:userAssignedIdentities}" -o json
Fix. Pass a disambiguator. Get the clientId of the identity you intend:
az identity show --name id-orders --resource-group rg-identity \
--query "{clientId:clientId, principalId:principalId, id:id}" -o json
Then in code, name it on the credential — never leave it to chance on a multi-identity resource:
# Raw IMDS, disambiguated by client_id (VM/VMSS)
curl -s -H "Metadata:true" \
"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net&client_id=<clientId>"
// Azure.Identity — pin the exact user-assigned identity by client id
var credential = new ManagedIdentityCredential(clientId: "<clientId>");
// or via DefaultAzureCredential options:
var cred2 = new DefaultAzureCredential(
new DefaultAzureCredentialOptions { ManagedIdentityClientId = "<clientId>" });
The three ways to name a user-assigned identity, and when to use which:
| Disambiguator | IMDS query param | SDK property | Use when |
|---|---|---|---|
| Client ID (appId) | client_id=<guid> |
ManagedIdentityClientId |
Default choice; stable, human-trackable |
| Object ID (principal oid) | object_id=<guid> |
ManagedIdentityObjectId (where supported) |
You only have the principal id to hand |
| ARM resource ID | mi_res_id=<resourceId> |
ManagedIdentityResourceId |
IaC passes the full resource id naturally |
The pattern that sidesteps the whole problem: set AZURE_CLIENT_ID on the resource to the intended UAMI’s client id. DefaultAzureCredential reads it automatically, so the code stays identity-agnostic and the IaC controls which identity is used.
Cause 3 — You passed a client_id that isn’t attached
The mirror image of Cause 2: you did pass a client_id (or mi_res_id), but it isn’t attached to this resource — a copy-paste from another environment, a UAMI recreated with a new client id, or IaC drift. IMDS returns a 400 / “Identity not found,” wrapped by the SDK as ManagedIdentityCredential authentication failed.
Confirm. Compare the id you’re passing against what’s attached:
# What client ids are actually attached? Your passed id must be in this list.
az vm identity show --name vm-batch-prod --resource-group rg-batch-prod \
--query "userAssignedIdentities" -o json
# Resolve a UAMI name to its clientId to compare:
az identity show -n id-batch -g rg-identity --query clientId -o tsv
Fix. Either attach the identity you’re naming, or change the code/config to name one that is attached. Treat the client id as configuration (an app setting / Bicep parameter), never a literal buried in code, so an environment swap can’t leave a stale id behind.
Cause 4 — IMDS is unreachable (the network path is broken)
On VMs and VMSS the token comes from 169.254.169.254, a link-local address that must resolve on the host’s own NIC. Anything that intercepts or reroutes outbound traffic can break it (the breakers table below enumerates them), and the symptom is always the same: the IMDS call times out and the SDK throws after its retry budget.
Confirm. curl IMDS directly from the host with a short timeout:
# From the VM/host. A healthy IMDS answers fast; a hang/refusal is the network.
curl -s -m 5 -H "Metadata:true" \
"http://169.254.169.254/metadata/instance?api-version=2021-02-01" | head -c 200
echo
# If that hangs, check whether a proxy is hijacking it:
env | grep -i proxy
If HTTP_PROXY/HTTPS_PROXY is set without 169.254.169.254 (or 169.254.169.254/16) in NO_PROXY, the proxy is almost certainly the culprit.
Fix. Make the link-local endpoint bypass the proxy and stay single-hop:
# Always exclude IMDS (and the Azure DNS VIP) from the proxy
export NO_PROXY="169.254.169.254,169.254.0.0/16,168.63.129.16,$NO_PROXY"
The common IMDS-network breakers and their fixes:
| Breaker | Why it kills IMDS | How to confirm | Fix |
|---|---|---|---|
| HTTP(S) proxy for all traffic | Proxy tries to relay a link-local request | `env | grep -i proxy; NO_PROXY` lacks the IP |
0.0.0.0/0 UDR to an appliance |
Default route swallows link-local | Effective routes show next-hop = appliance | Keep link-local local; don’t reroute it |
| Container default hop limit (1) | Single-hop packet dropped across bridge | IMDS works on host, fails in container | Raise the hop limit / use host networking |
| NSG/firewall on the host | Egress rule blocks 169.254/16 | Effective security rules | Allow link-local egress |
| Wrong API version / missing header | IMDS rejects the request shape | Body says bad request, not a timeout | Send Metadata:true + valid api-version |
Note that App Service / Functions do not use 169.254.169.254 — they use the injected IDENTITY_ENDPOINT on loopback, so none of these network breakers apply there. Diagnosing a PaaS token failure by curling IMDS is a classic dead end.
Cause 5 — The token is for the wrong resource (audience mismatch)
You got a token — but for the wrong aud. Ask for https://management.azure.com and present it to Key Vault, and the vault rejects it (InvalidAuthenticationTokenAudience / 401) even though the token is genuine and the role is assigned. People stare at RBAC for an hour, but RBAC was never the problem — the audience was.
Confirm. Get a token for the resource you think you’re using and decode its aud:
# Ask explicitly for the Key Vault resource, then read the audience claim
az account get-access-token --resource https://vault.azure.net \
--query accessToken -o tsv | cut -d. -f2 | base64 -d 2>/dev/null | python3 -c "import sys,json;print(json.load(sys.stdin)['aud'])"
If the aud you see in a failing call’s token isn’t the target’s resource URI, that’s the bug. The canonical resource URIs (and their .default scope equivalents) you must match:
| Target service | Resource URI (resource=) |
.default scope (newer SDKs) |
|---|---|---|
| Azure Resource Manager (ARM) | https://management.azure.com |
https://management.azure.com/.default |
| Key Vault | https://vault.azure.net |
https://vault.azure.net/.default |
| Storage (blob/queue/table/file) | https://storage.azure.com |
https://storage.azure.com/.default |
| Azure SQL Database | https://database.windows.net |
https://database.windows.net/.default |
| Microsoft Graph | https://graph.microsoft.com |
https://graph.microsoft.com/.default |
| Cosmos DB (AAD data plane) | https://cosmos.azure.com |
https://cosmos.azure.com/.default |
| Service Bus / Event Hubs | https://servicebus.azure.net |
https://servicebus.azure.net/.default |
Fix. Request the correct resource. With the SDK clients (SecretClient, BlobServiceClient, etc.) this is automatic — the client knows its own audience — which is why hand-rolled token requests cause most audience bugs. If you’re calling a REST endpoint by hand, set --resource / the scope to the exact URI above. Don’t touch RBAC for an audience mismatch; the role is fine.
Anatomy of authorization: when the token is valid but rejected (403)
This class means the opposite of the last one: you got a real token, and the target said no on permission grounds. The fix is always on the target side. Five causes — scan, then read the matching detail:
| # | 403 cause | Tell-tale signal | Confirm with | Real fix |
|---|---|---|---|---|
| 1 | No role assigned at all | AuthorizationFailed/403 on first use |
az role assignment list --assignee <oid> empty |
Grant least-privilege role at right scope |
| 2 | Role at the wrong scope | Works on resource A, 403 on resource B | List assignments; scope ≠ target’s id | Assign at the scope that covers the target |
| 3 | Control-plane role, data-plane call | Owner yet Storage/KV still 403 |
Decode aud is data plane; role lacks dataActions |
Grant a data role (Blob Data Reader, KV Secrets User) |
| 4 | Assignment not yet propagated | New grant 403s for a few minutes | Assignment exists but call still 403 | Wait/retry; don’t re-grant |
| 5 | Target network blocked the caller | “403”/timeout, but role is correct | Target firewall / private endpoint config | Allow the caller on the target, not RBAC |
Cause 1 — No role is assigned (the identity has zero permissions)
Enabling a managed identity grants it nothing. A freshly-enabled identity authenticates fine but is authorised for nothing, so the first call returns 403 / AuthorizationFailed. This is the most common 403: the identity exists, the token is valid, and no one ever ran the role assignment.
Confirm. Ask what the identity is allowed to do. Get its principalId (the oid), then list assignments:
# The identity's principal (object) id — RBAC assigns to THIS
PRINCIPAL=$(az webapp identity show -n app-orders-prod -g rg-orders-prod --query principalId -o tsv)
# Everything this identity is granted, anywhere it can see
az role assignment list --assignee "$PRINCIPAL" --all \
--query "[].{role:roleDefinitionName, scope:scope}" -o table
An empty result means no permissions — exactly the cause.
Fix. Grant the least-privilege role at the narrowest scope that works (the target resource, not the subscription):
# Example: read blobs in one storage account (data plane), scoped to that account
az role assignment create --assignee "$PRINCIPAL" \
--role "Storage Blob Data Reader" \
--scope $(az storage account show -n stordersprod -g rg-orders-prod --query id -o tsv)
// Grant the app's identity a data role on exactly one storage account
resource blobRead 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(storage.id, site.id, 'blob-data-reader')
scope: storage
properties: {
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions',
'2a2b9908-6ea1-4ae2-8e65-a410df84e7d1') // Storage Blob Data Reader
principalId: site.identity.principalId
principalType: 'ServicePrincipal' // set this — avoids replication-race failures
}
}
Setting principalType: 'ServicePrincipal' on the assignment matters in IaC: without it, ARM may try to validate a principal that hasn’t replicated yet and fail the deployment with a transient error.
Cause 2 — The role is assigned, but at the wrong scope
RBAC is hierarchical — a role on a resource group covers everything in it, a role on one resource covers only that one. A common 403 is an assignment at too narrow a scope: granted on storage account A but the code now reads account B, or granted on a resource group that doesn’t contain the target. The identity has the role “somewhere,” just not where it’s calling.
Confirm. List the assignments and compare each scope to the target’s resource id:
TARGET=$(az storage account show -n storbillingprod -g rg-billing --query id -o tsv)
az role assignment list --assignee "$PRINCIPAL" --all \
--query "[?contains('$TARGET', scope)].{role:roleDefinitionName, scope:scope}" -o table
# Empty => no assignment covers this target's scope
Fix. Assign the role at a scope that covers the target — the target resource itself (the safe, least-privilege default), or a resource group / subscription above it only if the identity legitimately needs many resources. Prefer the narrowest scope that satisfies the real need; don’t jump to subscription scope to dodge a per-resource grant, since every level up widens the blast radius.
Cause 3 — A control-plane role on a data-plane call (the Owner-but-403 trap)
This one fools senior engineers. Azure splits control-plane (managing the resource via ARM) from data-plane (reading the data inside it: blobs, secrets, messages). Owner and Contributor are control-plane roles — they let you manage a storage account or Key Vault, but on most data services they do not grant access to the data. So an identity with Owner on a storage account still gets 403 reading a blob with Entra auth, because blob data needs Storage Blob Data Reader. Same for Key Vault RBAC (Key Vault Secrets User), Service Bus (Azure Service Bus Data Receiver), and others.
Confirm. The token’s aud is a data-plane resource (e.g. https://storage.azure.com), the call 403s, yet the identity has Owner/Contributor. List the data roles specifically:
# Does the identity hold a DATA role on the storage account (not just Owner/Contributor)?
az role assignment list --assignee "$PRINCIPAL" \
--scope $(az storage account show -n stordersprod -g rg-orders-prod --query id -o tsv) \
--query "[].roleDefinitionName" -o tsv
# Look for 'Storage Blob Data *' — 'Owner' alone will still 403 on blob data.
Fix. Grant the matching data-plane role. The control-plane → data-plane role pairs you actually need:
| Target | Control-plane role (NOT enough for data) | Data-plane role you need |
|---|---|---|
| Storage blobs | Owner / Contributor | Storage Blob Data Reader / …Contributor |
| Storage queues | Owner / Contributor | Storage Queue Data Reader / …Message Processor |
| Key Vault (RBAC mode) | Owner / Contributor | Key Vault Secrets User (secrets) / Key Vault Crypto User (keys) |
| Service Bus | Owner / Contributor | Azure Service Bus Data Receiver / …Sender |
| Event Hubs | Owner / Contributor | Azure Event Hubs Data Receiver / …Sender |
| Cosmos DB (data plane) | Owner / Contributor | A Cosmos DB SQL role (data-plane RBAC, assigned separately) |
Note the Key Vault wrinkle: if the vault is on the access-policy model (not RBAC), role assignments do nothing — you grant an access policy instead. Checking which model the vault uses is step one; Key Vault RBAC vs Access Policies: Choosing and Migrating the Permission Model covers the fork, and Key Vault 403 Forbidden: Untangling Firewall, RBAC, Purge Protection, and Soft-Delete Recovery is the full vault-side playbook.
Cause 4 — The assignment is correct but hasn’t propagated
You granted the role, the portal shows it, and the call still 403s. Before you grant it again (creating a duplicate) or escalate to Owner, check whether this is simply propagation lag. New role assignments take time to replicate to enforcement points and for the target’s cache to refresh — commonly a couple of minutes, sometimes up to ~5, occasionally longer for cross-tenant or globally-distributed targets.
Confirm. The assignment already exists and the call still fails — that combination is lag, not a missing grant:
# If this returns the role yet the app still 403s, it's propagation, not permission
az role assignment list --assignee "$PRINCIPAL" \
--scope $(az keyvault show -n kv-orders --query id -o tsv) \
--query "[].{role:roleDefinitionName, created:createdOn}" -o table
Fix. Wait and retry — don’t re-grant. Make the dependency call resilient with a brief backoff, which covers both first-boot propagation (commonly up to ~5 minutes, longer for cross-tenant/cross-region targets) and routine token-cache refresh. In CI/CD, if a deploy assigns a role and immediately uses it, insert a short wait or retry loop rather than failing on the first attempt. The flip side: a removed role can stay cached as valid briefly, so don’t rely on instant revocation for security.
Cause 5 — The target’s network blocked you (it isn’t really RBAC)
A 403 (or a timeout that surfaces as one) can come from the target’s firewall, not its RBAC. If the storage account, Key Vault, or SQL server is on selected networks or behind a private endpoint, a call from a disallowed IP/subnet is refused at the network layer while the role is perfectly correct — and you burn time in RBAC because the message looks permission-shaped.
Confirm. Check the target’s networking and whether the caller’s egress is allowed:
# Is the target restricted to selected networks?
az storage account show -n stordersprod -g rg-orders-prod \
--query "{default:networkRuleSet.defaultAction, ips:networkRuleSet.ipRules, vnets:networkRuleSet.virtualNetworkRules}" -o json
If defaultAction is Deny and the caller’s subnet/IP isn’t in the rules (or there’s no private endpoint + DNS), that’s the cause — not RBAC. Fix: allow the caller on the target (a service/private endpoint from the app’s integration subnet, or the egress IP in the firewall), and ensure private DNS resolves. The detailed network-vs-RBAC fork for storage specifically lives in the storage-403 playbook; here the rule is simply: if the role is correct, suspect the target’s firewall next.
Decoding the token: the claims that end the argument
When the two layers disagree with you, the JWT itself settles it. A managed-identity access token is a JWT with three dot-separated parts; the middle part is base64url JSON you can read. The claims that matter for these incidents:
| Claim | Meaning | Use it to answer |
|---|---|---|
aud |
Audience — the resource the token is for | “Is this token even for the service I’m calling?” (audience bugs) |
oid |
Object id of the principal | “Which identity is this? Does this oid have the role?” |
appid |
App/client id of the identity | “On a multi-identity resource, which one did the SDK pick?” |
iss |
Issuer (your tenant’s Entra endpoint) | “Is this my tenant?” (cross-tenant mix-ups) |
tid |
Tenant id | “Right directory?” |
exp |
Expiry (unix time) | “Is the 401 just an expired token?” |
roles / xms_mirid |
App roles / the MI’s resource id (when present) | “Is this the user-assigned identity I think it is?” |
Get a token and decode it in one line (works in Cloud Shell):
# Pretty-print the claims of a token for the Key Vault audience
az account get-access-token --resource https://vault.azure.net --query accessToken -o tsv \
| cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool
From a workload, the same trick on a token from IMDS / IDENTITY_ENDPOINT proves which identity actually ran — decode appid/oid and compare to the UAMI you intended. This single check resolves most “wrong identity picked” mysteries: if the appid is the system-assigned principal when you expected your UAMI, your DefaultAzureCredential didn’t get the client_id you thought it did. In practice you reach for three of these claims: aud (proves an audience mismatch — it won’t equal the target’s resource URI), appid/oid (proves the wrong identity ran on a multi-identity resource), and exp (proves a sporadic 401 is just an expired token).
DefaultAzureCredential vs ManagedIdentityCredential: the silent fall-through
Most code uses DefaultAzureCredential — a convenience chain that tries multiple credential sources in order and uses the first that yields a token: environment variables, then managed identity, then developer sign-ins (Azure CLI, Azure Developer CLI, Visual Studio, etc.). On Azure this is great — it picks up the managed identity automatically. The trap is the fall-through: if managed identity fails for any reason (not enabled, wrong client_id, IMDS blip), the chain doesn’t stop — it quietly moves on to the next source. Locally that’s a feature (it uses your az login); in production it can mask a broken managed identity, or — if a stale developer login is somehow present — succeed as the wrong principal, producing a baffling 403 against a token nobody expected.
Confirm. Read the credential’s diagnostic message: when DefaultAzureCredential fails it lists every credential it tried and why each failed. If managed identity’s line says “not enabled” / “client_id not found” while another source succeeded, you’ve found the fall-through. In code you can also force the question:
// In production, prefer the explicit credential so failures are LOUD, not silent
var credential = new ManagedIdentityCredential(clientId: Environment.GetEnvironmentVariable("AZURE_CLIENT_ID"));
// If you keep DefaultAzureCredential, disable the dev paths in prod:
var prodCred = new DefaultAzureCredential(new DefaultAzureCredentialOptions {
ExcludeAzureCliCredential = true,
ExcludeVisualStudioCredential = true,
ExcludeInteractiveBrowserCredential = true,
ManagedIdentityClientId = Environment.GetEnvironmentVariable("AZURE_CLIENT_ID")
});
The two credentials compared, so you pick deliberately:
| Aspect | DefaultAzureCredential |
ManagedIdentityCredential |
|---|---|---|
| What it tries | Env → MI → dev logins (CLI, VS, azd…) | Only managed identity |
| Local development | Works via your az login |
Fails (no MI off-Azure) |
| Production failure mode | Falls through silently; can mask MI failure | Fails loudly with a clear MI error |
| Multi-identity disambiguation | ManagedIdentityClientId / AZURE_CLIENT_ID |
clientId constructor arg |
| Best for | One codebase across laptop + Azure | Prod paths where you want no fallback surprises |
| The footgun | Wrong principal / hidden MI break | None (that’s the point) |
Practical rule: keep DefaultAzureCredential for code that must run both on a laptop and on Azure, but always pin the client id (via AZURE_CLIENT_ID or the option) and exclude the interactive/dev credentials in production, so a deployed app can never silently authenticate as a human. For pure production services, ManagedIdentityCredential is the honest choice — it fails fast and tells you exactly what’s wrong.
Architecture at a glance
There is no diagram for this topic — the mental model is a single, short pipeline you can hold in your head, and the whole article is built around walking it hop by hop. Picture the token’s journey left to right.
Your code asks a credential (DefaultAzureCredential or ManagedIdentityCredential) for a token for a specific resource. The credential calls a local token endpoint — on a VM/VMSS that’s IMDS at 169.254.169.254/metadata/identity/oauth2/token with Metadata: true; on App Service/Functions it’s the injected IDENTITY_ENDPOINT on loopback, authenticated by IDENTITY_HEADER. That endpoint, trusted by the platform, asks Entra ID to mint a token for the resource’s managed identity. Entra returns a JWT whose aud is the requested resource and whose oid/appid identify the principal. Your code then presents that JWT to the target service (Key Vault, Storage, SQL, ARM…), which performs an RBAC / access-policy check at the relevant scope and either serves the request or returns 403.
Now map the failures onto that line. Everything up to and including “Entra returns a JWT” is authentication — break any hop here and you get no token (identity not enabled, multi-identity ambiguity or wrong client_id, IMDS unreachable, or a wrong-resource request). Everything after the JWT is authorization — the token is fine, but the target’s RBAC check at that scope says no (no role, wrong scope, a control-plane role on a data-plane call, an assignment not yet propagated, or the target’s firewall). The question that splits the pipeline in half is the one the whole article turns on: did I get a token at all? If no, you’re left of the JWT — fix the source. If yes but rejected, you’re right of it — fix the target.
Real-world scenario
Finovo Payments runs a reconciliation service on Azure: a .NET 8 app on App Service (P1v3) in Central India that reads a settlement file from Blob Storage and pulls a database connection string from Key Vault (RBAC mode). It was migrated off a hard-coded connection string to a user-assigned managed identity (id-finovo-recon) shared by three services, because the platform team standardised on one UAMI per domain. The team is five engineers; the app had run on the old secret-based config for a year without incident.
The cutover went out on a Tuesday. Within a minute the app crash-looped: Application Insights showed ManagedIdentityCredential authentication failed at startup, and the Key Vault reference in app settings showed a red “failed to resolve” in the Environment variables blade. The on-call engineer’s first reflex was the usual one for a 403 — “the identity needs the role” — so they granted Key Vault Secrets User to the UAMI and redeployed. Still failing, identically. Second reflex: grant Owner on the resource group “to be safe.” Still failing. Twenty minutes in, with Owner granted and the app still dead, it was clearly not a permissions problem at all.
The breakthrough was decoding what the app was actually doing. The team SSH’d into the app and ran printenv | grep -i IDENTITY — the endpoint was wired up, so managed identity was available. Then they tried to get a token by hand against IDENTITY_ENDPOINT and got a 400 whose body said “Multiple user assigned identities exist, please specify the client_id.” That was the real cause: the app had both a leftover system-assigned identity (enabled months earlier during an experiment and never removed) and the new id-finovo-recon user-assigned identity attached. With two identities present, the token endpoint refused to guess, the SDK couldn’t get a token, and the Key Vault reference — which had no way to specify a client id — failed at boot. Every 403-shaped fix had been treating an authentication problem (no token) as an authorization problem (no role). The granted roles, including the panic Owner, had done nothing because the app never held a token to present.
The fix had two parts. Immediately: tell the platform which identity to use by setting the keyVaultReferenceIdentity property on the web app to the resource id of id-finovo-recon, so Key Vault references resolve via that specific UAMI; and set AZURE_CLIENT_ID to its client id so the app’s own DefaultAzureCredential pins the same identity. The app came up immediately. Within the week: remove the stray system-assigned identity entirely (it was unused and was the source of the ambiguity), and revert the Owner over-grant back to a single Key Vault Secrets User on the vault plus Storage Blob Data Reader on the one storage account — least privilege restored. The post-incident note on the wall: “ManagedIdentityCredential failed is a no-token error, not a no-permission error. Get the token by hand before you touch RBAC.”
The incident as a timeline, because the order of moves is the lesson:
| Time | Symptom | Action taken | Effect | What it should have been |
|---|---|---|---|---|
| T+0 | App crash-loops; ManagedIdentityCredential failed |
(cutover deployed) | — | Ask: did it get a token at all? |
| T+4 | Same error every boot | Grant Key Vault Secrets User to UAMI |
No change | Authentication failing, not authz |
| T+11 | Still failing | Grant Owner on the RG “to be safe” |
No change; over-grant created | Don’t escalate blind |
| T+18 | Still dead with Owner |
SSH in; token request by hand → 400 “specify client_id” | Root cause found | This was the breakthrough |
| T+24 | Root cause: two identities attached | Set keyVaultReferenceIdentity + AZURE_CLIENT_ID to the UAMI |
App comes up | Correct fix |
| +1 week | Hardened | Remove stray system-assigned identity; revert Owner to least-privilege data roles |
Ambiguity gone; least privilege | The real fix is identity hygiene |
Advantages and disadvantages
Managed identity both causes this class of confusion and is overwhelmingly worth it. Weigh it honestly:
| Advantages (why you use managed identity) | Disadvantages (why it bites) |
|---|---|
| No secret in code or config — the biggest credential-leak class simply disappears | The identity is implicit; there’s nothing to echo, so failures are opaque |
| Azure lifecycle-manages the credential; rotation is automatic and invisible | Two layers (auth vs authz) fail with different codes that get conflated constantly |
| RBAC gives precise, scoped, auditable access per identity | Control-plane vs data-plane roles trip even seniors (Owner ≠ blob data access) |
| Works uniformly across VMs, App Service, Functions, AKS, Container Apps | The token source differs (IMDS vs IDENTITY_ENDPOINT) — wrong endpoint = dead-end diagnosis |
| One user-assigned identity can be shared and pre-granted across many resources | Multiple identities create the client_id ambiguity — the #1 user-assigned failure |
DefaultAzureCredential lets one codebase run on a laptop and in prod |
…and silently falls through, masking a broken MI or using the wrong principal |
| Role assignments are declarative IaC you can review | Assignments are eventually consistent — a correct grant 403s for minutes |
The model is right for essentially every Azure workload that calls Azure services — there is no good reason to keep storage keys or SQL passwords in config when managed identity exists. It bites hardest on multi-identity resources (ambiguity), teams that reach for Owner instead of data-plane roles, hand-rolled token requests (audience and disambiguation bugs the SDKs handle for you), and freshly-deployed infrastructure that uses a role the instant it’s granted (propagation lag). Every disadvantage is manageable — pin the client id, grant the right data role at the narrowest scope, prefer the SDK clients, and build in retry — but only if you know the two-layer model, which is the whole point of this article.
Hands-on lab
Reproduce both failure layers on one VM — first a 401 / no-token path (raw IMDS), then a 403 (valid token, no role) against Key Vault — and fix each. Free-tier-friendly: a B1s VM and a standard Key Vault cost a few rupees for the hour; delete at the end. Run in Cloud Shell (Bash).
Step 1 — Variables and resource group.
RG=rg-mi-lab
LOC=centralindia
VM=vm-mi-lab
KV=kvmilab$RANDOM # vault names are globally unique
az group create -n $RG -l $LOC -o table
Step 2 — Create a small Linux VM (no identity yet — that’s deliberate).
az vm create -n $VM -g $RG --image Ubuntu2204 --size Standard_B1s \
--admin-username azureuser --generate-ssh-keys -o table
Step 3 — From inside the VM, prove there is NO token yet (the authentication failure). SSH in (az vm ssh or the public IP), then:
# No identity assigned → IMDS returns an error, not a token
curl -s -H "Metadata:true" \
"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net"
Expected: an error body (no access_token) — there is no identity to mint for. This is the 401/no-token class, on purpose.
Step 4 — Enable a system-assigned identity, then get a token (authentication fixed).
# From Cloud Shell:
az vm identity assign -n $VM -g $RG -o table
Back on the VM, re-run the IMDS call from Step 3. Expected now: a JSON body containing access_token, expires_on, and resource: https://vault.azure.net. You now have a valid token — authentication works.
Step 5 — Create a Key Vault with a secret, then prove the token is REJECTED (the authorization failure).
# From Cloud Shell — RBAC-mode vault:
az keyvault create -n $KV -g $RG -l $LOC --enable-rbac-authorization true -o table
# Grant YOURSELF data access so you can write a secret:
ME=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --assignee "$ME" --role "Key Vault Secrets Officer" \
--scope $(az keyvault show -n $KV --query id -o tsv)
az keyvault secret set --vault-name $KV -n db-conn --value "Server=...;Pwd=demo" -o table
On the VM, use the token to read the secret over REST:
TOKEN=$(curl -s -H "Metadata:true" \
"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net" \
| python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
curl -s -H "Authorization: Bearer $TOKEN" \
"https://$KV.vault.azure.net/secrets/db-conn?api-version=7.4"
Expected: a 403 Forbidden body. The token is valid (you minted it in Step 4), but the VM’s identity has no role on the vault — the pure authorization failure, with a valid token in hand. Decode the token to prove it’s real:
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | python3 -c "import sys,json;d=json.load(sys.stdin);print('aud=',d['aud'],'oid=',d['oid'])"
Step 6 — Grant the data role, then succeed.
# From Cloud Shell — grant the VM's identity read on secrets, scoped to the vault
PRINCIPAL=$(az vm identity show -n $VM -g $RG --query principalId -o tsv)
az role assignment create --assignee "$PRINCIPAL" --role "Key Vault Secrets User" \
--scope $(az keyvault show -n $KV --query id -o tsv)
// The IaC form of the same grant
resource kvSecretsUser 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(kv.id, vm.id, 'kv-secrets-user')
scope: kv
properties: {
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions',
'4633458b-17de-408a-b874-0445c86b69e6') // Key Vault Secrets User
principalId: vm.identity.principalId
principalType: 'ServicePrincipal'
}
}
Wait ~1–3 minutes for propagation (Cause 4!), then re-run the Step 5 secret read on the VM. Expected: the secret value returns. You’ve now traversed both layers — fixed authentication (Step 4) and authorization (Step 6) — and felt the propagation delay first-hand.
Validation checklist. You proved a no-token failure before identity was enabled (Step 3, the 401 class), fixed it by enabling the identity (Step 4, authentication only), then proved a valid token can still be 403’d with no role (Step 5, the authorization layer), and fixed that with a scoped data-plane grant (Step 6) — observing propagation lag in between. That is the whole article in six commands: enabling the identity fixes authentication and grants zero permissions; the role grant fixes authorization.
Cleanup (avoid lingering charges).
az group delete -n $RG --yes --no-wait
Cost note. A B1s VM plus a standard Key Vault for an hour is well under ₹50; deleting the resource group stops everything. (Key Vault charges per-operation pennies; the VM is the only meaningful cost.)
Common mistakes & troubleshooting
This is the playbook — the part you bookmark. First as a scannable table you can read at 03:00, then the same entries with full confirm-command detail for the ones that bite hardest. The single most important column is the one that tells you which layer you’re in: a 401/IMDS row is fixed on the source, a 403 row on the target.
| # | Symptom | Root cause | Confirm (exact cmd / portal path) | Fix |
|---|---|---|---|---|
| 1 | ManagedIdentityCredential authentication failed at startup; no token |
Identity not enabled / not attached | az <res> identity show empty; `printenv |
grep IDENTITY` unset |
| 2 | IMDS / token endpoint returns 400 “specify the client_id” | Multiple identities; SDK can’t choose | az <res> identity show lists >1 identity |
Pass client_id / set AZURE_CLIENT_ID |
| 3 | 400 / “Identity not found” for a client_id you passed |
That id isn’t attached to this resource | az <res> identity show --query userAssignedIdentities lacks it |
Attach it, or pass an attached id |
| 4 | IMDS call times out on a VM | Proxy/UDR hijacking link-local; container hop limit | curl -m 5 169.254.169.254/... hangs; `env |
grep -i proxy` |
| 5 | Valid token, but target returns 401 / InvalidAuthenticationTokenAudience |
Token minted for the wrong resource (aud) |
Decode aud; compare to the target’s resource URI |
Request the correct resource scope (let SDK clients do it) |
| 6 | AuthorizationFailed / 403 on first use |
No RBAC role assigned at all | az role assignment list --assignee <oid> --all empty |
Grant least-privilege role at the target scope |
| 7 | 403 on resource B though it works on resource A | Role assigned at the wrong / too-narrow scope | List assignments; no scope covers B’s id |
Assign at a scope covering the target |
| 8 | Owner granted, yet Storage/Key Vault still 403 |
Control-plane role on a data-plane call | Token aud is data plane; role lacks dataActions |
Grant a data role (Storage Blob Data Reader, Key Vault Secrets User) |
| 9 | Just granted the role; still 403 for a few minutes | Entra propagation lag | az role assignment list shows the role already |
Wait and retry; do NOT re-grant or escalate |
| 10 | App settings show Key Vault reference “failed to resolve”; app crash-loops | Identity/role/firewall/URI on the KV reference | Environment variables blade red error; az webapp config appsettings list --query "[?contains(value,'KeyVault')]" |
Enable identity; grant Key Vault Secrets User; set keyVaultReferenceIdentity; fix firewall/URI |
| 11 | Worked locally, 403 in production with an unexpected principal | DefaultAzureCredential fell through to a dev login / wrong identity |
Decode appid/oid; read DAC failure message |
Pin AZURE_CLIENT_ID; exclude dev credentials in prod |
| 12 | 403 though role is correct and propagated | Target’s firewall/private endpoint blocked the caller | Target networkRuleSet.defaultAction = Deny; caller IP/subnet not allowed |
Allow the caller on the target; fix private DNS |
| 13 | Curling IMDS on App Service times out / refused | App Service doesn’t use 169.254.169.254 |
`printenv | grep IDENTITY_ENDPOINT` (it’s loopback) |
| 14 | 403 from Cosmos DB data plane despite an Azure RBAC role | Cosmos data plane uses its own role system | az cosmosdb sql role assignment list empty |
Assign a Cosmos DB SQL role (data-plane RBAC) |
| 15 | Token works for one service, 403 for another in the same app | Per-service role missing; each target is independent | az role assignment list --assignee <oid> --all lacks the second target |
Grant the role on the second target too |
The expanded form, with the full reasoning for the two entries that fool senior engineers most and need more than the table can hold:
Owner granted, yet Storage/Key Vault still 403 (row 8).
Root cause: Owner/Contributor are control-plane roles; most data services need a separate data-plane role. The identity can manage the resource but not read its data.
Confirm: the token aud is a data plane (https://storage.azure.com, https://vault.azure.net), the call 403s, and az role assignment list --assignee <oid> --scope <target> shows only Owner/Contributor, no *Data* role.
Fix: grant the matching data role (Storage Blob Data Reader, Key Vault Secrets User, Azure Service Bus Data Receiver, …). For access-policy-mode Key Vaults, grant an access policy, not a role.
Key Vault reference “failed to resolve”; app crash-loops (row 10).
Root cause: a Key Vault reference in app settings (@Microsoft.KeyVault(SecretUri=...)) resolves at startup via the app’s managed identity; if the identity is missing/ambiguous, lacks Key Vault Secrets User, the vault firewall blocks the app, or the URI is wrong, it resolves to empty and the app crashes — looking like a random restart loop.
Confirm: Environment variables blade shows the reference with a red error and reason; az webapp config appsettings list --query "[?contains(value,'KeyVault')]"; az webapp identity show.
Fix: enable the identity; on a multi-identity app set keyVaultReferenceIdentity to the intended UAMI’s resource id; grant Key Vault Secrets User; allow the app on the vault’s firewall; verify the secret URI. (The boot-time crash-loop angle is also covered from the App Service side in Troubleshooting Azure App Service: 502/503 Errors, Cold Starts & Restart Loops.)
Best practices
- Always answer one question first: did I get a token at all?
az account get-access-token(or a raw IMDS/IDENTITY_ENDPOINTcall) splits every incident into authentication (no token → fix the source) vs authorization (token → fix the target). Don’t touch RBAC until you’ve confirmed a token exists. - Pin the client id on any multi-identity resource. Set
AZURE_CLIENT_ID(andkeyVaultReferenceIdentityfor KV references) so the credential and the platform never have to guess. Treat the client id as configuration, never a literal in code. - Grant data-plane roles for data, control-plane for management.
Owner/Contributordo not read blobs, secrets, or messages on most services — reach forStorage Blob Data Reader,Key Vault Secrets User,Azure Service Bus Data Receiver, etc. - Assign at the narrowest scope that works — the target resource first, a resource group only if the identity genuinely spans it. Never grant subscription-wide to dodge a per-resource assignment.
- Never grant
Ownerto clear a 403. It’s the most common over-grant in Azure. If a role looks correct but still 403s, suspect propagation lag or audience/network — not insufficient privilege. - Set
principalType: 'ServicePrincipal'in IaC role assignments so ARM doesn’t fail validating a just-created principal that hasn’t replicated. - Build retry with backoff into first-use dependency calls, and wait in grant-then-use pipelines, to absorb propagation delay without failures.
- In production, prefer
ManagedIdentityCredentialor a hardenedDefaultAzureCredentialwith the dev/interactive credentials excluded, so a deployed app can never silently authenticate as a human. - Use the SDK clients (
SecretClient,BlobServiceClient, …) instead of hand-rolling token requests — they pick the correctaudand handle caching, eliminating most audience and disambiguation bugs. - Remove identities you don’t use. A stray system-assigned identity is the usual source of the multi-identity ambiguity; one identity per purpose keeps token acquisition deterministic.
- Decode the token when the layers disagree with you.
aud,oid, andappidend most “but it should work” arguments in seconds. - Know your token source. VMs/VMSS use IMDS (
169.254.169.254); App Service/Functions useIDENTITY_ENDPOINT. Diagnosing the wrong one is a guaranteed dead end.
Security notes
Managed identity is a security win precisely because it removes the long-lived secret, but its safety depends on disciplined authorization. Least privilege is the whole point: an identity that can be granted exactly Storage Blob Data Reader on one account is squandered the moment someone gives it Owner on the subscription to silence a 403. Grant the smallest role at the smallest scope, and periodically audit with az role assignment list --assignee <oid> --all — permissions accrete over incidents and rarely get pruned.
Prefer user-assigned identities for anything shared or long-lived: they survive resource recreation, can be pre-granted before the workload exists (avoiding the first-deploy propagation race), and make the blast radius explicit. Reserve system-assigned for single-purpose resources, and don’t leave a stray system-assigned identity enabled alongside a UAMI — it causes the ambiguity and is an extra principal to govern.
Protect the token source, not just the target. On VMs the IMDS endpoint is reachable by any process on the host, so a compromised process can mint tokens for the VM’s identity — another reason to scope that identity tightly and avoid attaching a powerful shared UAMI to a VM running untrusted code. Lock the targets down at the network layer too: a Key Vault or storage account on selected networks or behind a private endpoint refuses even a valid token from an unexpected network, giving defence in depth beyond RBAC. Finally, prefer the RBAC model on Key Vault over legacy access policies for consistent, auditable authorization, and keep the secrets the identity reads in the vault rather than in app settings.
Cost & sizing
Managed identity itself is free — there is no charge to enable a system-assigned identity, create a user-assigned identity, or acquire tokens. The cost conversation here is therefore about the failure modes, not the feature:
| Cost driver | What it is | Rough figure | How to control it |
|---|---|---|---|
| Managed identity (system or user-assigned) | The identity and token issuance | Free | n/a — use it liberally |
| Key Vault operations | Per-secret/key/cert transaction the identity makes | ~₹0.25 / 10,000 secret ops (region-dependent) | Cache resolved secrets; don’t fetch per request |
| IMDS / token endpoint calls | Local token acquisition + Entra issuance | Free | Cache tokens (SDKs do; ~24h lifetime) |
| Over-granted roles (security cost) | Owner granted to fix a 403 |
No $ — but large breach blast radius | Least-privilege; audit and prune |
| Incident time (the real cost) | Hours lost conflating 401 vs 403 | Engineer-hours | This playbook; decode tokens early |
| Lab resources | The hands-on VM + vault | < ₹50 / hour | Delete the resource group when done |
The sizing guidance is about identity hygiene, not capacity: a single shared user-assigned identity per domain (pre-granted to its targets) is cheaper to operate than dozens of system-assigned identities each needing their own role assignments. The one genuine micro-cost to watch is Key Vault transaction volume — an app that resolves a secret on every request instead of caching it can rack up millions of operations; cache the value (or use Key Vault references / App Configuration, which resolve once at boot). Managed identity itself has no free-tier limit; the free-tier consideration is the targets it calls, each priced on its own meter.
Interview & exam questions
Q1. What is the difference between a 401 and a 403 when a managed identity calls a service, and where does the fix live for each?
A 401 (or an IMDS timeout / ManagedIdentityCredential authentication failed) is an authentication failure — no token was issued — fixed on the source (enable the identity, pass the right client_id, fix the endpoint/network). A 403 / AuthorizationFailed is an authorization failure — a valid token was rejected — fixed on the target by granting the right RBAC role at the right scope. The first diagnostic is “did I get a token at all?”
Q2. A resource has a system-assigned identity and two user-assigned identities. A token request to IMDS returns 400 “specify the client_id.” Why, and how do you fix it?
With more than one identity attached, the token endpoint cannot infer which to use, so it 400s. Fix it by passing a disambiguator — the intended identity’s client_id (or object_id/mi_res_id) on the request — or by setting AZURE_CLIENT_ID so DefaultAzureCredential resolves deterministically. Best long-term fix: remove identities the resource doesn’t need.
Q3. An identity has Owner on a storage account but still gets 403 reading a blob with Entra auth. Explain.
Owner is a control-plane role — it manages the account but does not include the data actions to read blob data. Blob data access needs a data-plane role such as Storage Blob Data Reader. Control-plane and data-plane authorization are separate; Owner ≠ data access on most data services.
Q4. You grant a role and the call still 403s a minute later. What’s happening and what should you do?
Entra propagation lag — the assignment is correct but hasn’t replicated to all enforcement points / the target’s cache yet (commonly up to ~5 minutes). Confirm with az role assignment list (the role is already there), then wait and retry. Do not re-grant or escalate to Owner; build retry into first-use calls and pipelines.
Q5. How do you prove which identity a token belongs to and which resource it’s for?
Decode the JWT: the aud claim is the resource the token was minted for, and oid/appid identify the principal (so on a multi-identity resource you can see which identity the SDK actually used). One base64 -d of the token’s middle segment ends most “it should work” arguments.
Q6. Why can DefaultAzureCredential succeed in a way that masks a broken managed identity?
It’s a chain: it tries env vars, then managed identity, then developer logins. If managed identity fails it silently falls through to the next source — so locally it uses your az login, and in production it can hide an MI failure or authenticate as the wrong principal. In production, pin AZURE_CLIENT_ID and exclude the interactive/CLI credentials, or use ManagedIdentityCredential which fails loudly.
Q7. On a VM, IMDS calls time out. List three causes and the confirming check.
An all-traffic HTTP proxy without 169.254.169.254 in NO_PROXY (confirm: env | grep -i proxy); a UDR sending 0.0.0.0/0 to an appliance that swallows link-local (confirm: effective routes); and, in a container on the VM, a default IP hop limit of 1 dropping the single-hop packet (confirm: IMDS works on the host but not the container). App Service/Functions are unaffected — they don’t use IMDS.
Q8. What’s the difference between the token source on a VM and on App Service?
A VM/VMSS uses IMDS at the link-local 169.254.169.254/metadata/identity/oauth2/token with Metadata: true. App Service/Functions use an injected IDENTITY_ENDPOINT on loopback (127.0.0.1), authenticated with the secret in IDENTITY_HEADER. Diagnosing a PaaS token failure by curling IMDS is a dead end.
Q9. An App Service Key Vault reference shows “failed to resolve” and the app crash-loops. Walk the diagnosis.
The reference resolves at startup via the app’s managed identity. Check: is an identity enabled (az webapp identity show)? On a multi-identity app, is keyVaultReferenceIdentity set to the intended UAMI? Does that identity have Key Vault Secrets User on the vault? Does the vault firewall allow the app? Is the secret URI correct and the secret enabled? Any one failing yields an empty value and a crash.
Q10. What does setting principalType: 'ServicePrincipal' on a Bicep role assignment prevent?
It tells ARM the principal is a service principal, so it skips a directory lookup that can fail when the managed identity’s principal was just created and hasn’t replicated — preventing a transient “principal not found” deployment error in infrastructure that creates an identity and grants it a role in the same deployment.
Q11. The role is correct and propagated, yet the call still 403s. What’s left to check?
The target’s network: if the storage account / Key Vault / SQL server is on selected networks or behind a private endpoint, a token-bearing call from a disallowed IP/subnet is refused at the network layer despite valid RBAC. Confirm the target’s networkRuleSet.defaultAction and whether the caller’s egress is allowed; fix it on the target (service/private endpoint, firewall allow, private DNS).
Cert mapping: this material maps to AZ-104 (managed identities, RBAC, Key Vault access), AZ-204 (securing app resources with managed identity and Key Vault), AZ-500 (identity and access, least privilege, network isolation of targets), and the identity portions of AZ-305.
Quick check
- You see
ManagedIdentityCredential authentication failedat startup. Is this an authentication or an authorization problem, and which side do you fix? - An IMDS token request returns 400 “specify the client_id.” What single condition on the resource causes this?
- An identity has
Owneron a storage account but 403s reading a blob. What role does it actually need? - You granted
Key Vault Secrets Userand the call still 403s 60 seconds later. What’s the most likely cause and the correct action? - Which claim in a decoded token tells you whether the token was minted for the right service?
Answers
- Authentication (no token was issued) — fix the source: enable the identity, pass the right
client_id, or fix the token endpoint/network. RBAC is irrelevant until a token exists. - The resource has more than one identity attached (e.g. system-assigned plus a user-assigned, or two UAMIs), so the endpoint can’t choose. Pass
client_id/object_id/mi_res_idor setAZURE_CLIENT_ID. - A data-plane role —
Storage Blob Data Reader(or…Contributor).Owneris control-plane and doesn’t grant blob data access. - Entra propagation lag — the assignment is correct but not yet enforced. Wait and retry (up to ~5 minutes); confirm with
az role assignment listthat the role already exists. Don’t re-grant or escalate. - The
aud(audience) claim — it must match the target service’s resource URI; if it doesn’t, you have a valid token the real target will reject.
Glossary
- Managed identity — an Entra ID identity Azure creates and lifecycle-manages for a resource so it can authenticate to other Azure services without a stored secret.
- System-assigned identity — a managed identity tied 1:1 to a single resource; created and deleted with it.
- User-assigned identity (UAMI) — a standalone Azure resource representing an identity that can be attached to one or many compute resources.
- IMDS (Instance Metadata Service) — the local endpoint at the link-local address
169.254.169.254on VMs/VMSS that supplies instance metadata and managed-identity tokens. - IDENTITY_ENDPOINT / IDENTITY_HEADER — the loopback token endpoint and its auth secret, injected as environment variables on App Service and Functions (the PaaS equivalent of IMDS).
- Authentication — proving identity and obtaining a token; failure here yields a 401 / IMDS error / no token.
- Authorization — being permitted to perform an action with a valid token; failure here yields a 403 /
AuthorizationFailed. aud(audience) — the JWT claim naming the resource a token was minted for; a mismatch means a valid token the real target rejects.oid(object id) /principalId— the identity’s principal id; the thing RBAC role assignments are bound to. (appid/client_idis the separate application id used to disambiguate which UAMI to use.)- RBAC role assignment — a binding of a role definition to a principal at a scope, which is how most services authorise a token.
- Scope — where a role applies: management group, subscription, resource group, or a single resource.
- Control plane vs data plane — managing a resource (ARM, via
Owner/Contributor) versus reading the data inside it (via data roles likeStorage Blob Data Reader); separate authorizations. DefaultAzureCredential— an SDK credential chain that tries several sources (env, managed identity, dev logins) and uses the first that yields a token.ManagedIdentityCredential— an SDK credential that uses only managed identity and fails loudly if it isn’t available.- Propagation delay — the eventual-consistency lag between creating a role assignment and it being enforced everywhere (commonly up to ~5 minutes).
keyVaultReferenceIdentity— the App Service property selecting which identity resolves Key Vault references on a multi-identity app.
Next steps
- Ground the identity model itself in Managed Identities Demystified: System vs User-Assigned and When to Use Each — the upstream primer this troubleshooting guide assumes.
- Master the authorization layer with Azure RBAC Fundamentals: Roles, Scopes and Assignments Without the Confusion and its troubleshooting companion Troubleshooting AuthorizationFailed: Reading Effective Azure RBAC Permissions.
- Go deep on the most common target with Azure Key Vault: Secrets, Keys and Certificates Done Right and the vault-side 403 playbook, Key Vault 403 Forbidden: Untangling Firewall, RBAC, Purge Protection, and Soft-Delete Recovery.
- Understand the token model beneath it all in OIDC and OAuth2 on Entra ID: Choosing the Right Flow (Auth Code, PKCE, Client Credentials) and the principal object in App Registrations vs Enterprise Applications: The Service Principal Model Explained.
- Wire up the telemetry that makes these failures visible with Azure Monitor and Application Insights: Full-Stack Observability.