Identity Azure

Managed Identities Deep Dive: User-Assigned Identities, Federated Credentials, and RBAC Patterns for Azure Workloads

Almost every “the pipeline can’t reach Key Vault” incident I get pulled into ends the same way: a system-assigned identity that got recreated on a redeploy, taking every role assignment with it. Managed identities are the right answer to “stop putting secrets in app settings” — an Azure-managed service principal whose credential the platform rotates and injects, so no key ever lands in your code or your CI. But the defaults push teams toward the brittle pattern, and the failure modes are subtle: a 403 that “exists” in your IaC, a federated credential whose subject is one character off, a cached token that predates the role you just granted, a tenant that already holds 999 role assignments. None of those throw an obvious “access denied” — they throw silence, then an incident.

This is the architecture I deploy instead, and the mechanics underneath it so you can debug when it bites. User-assigned identities (UAMIs) as first-class, reusable resources whose principalId survives any redeploy. The Instance Metadata Service (IMDS) token flow — what 169.254.169.254 actually does, why the audience matters, and how the SDK caches the JWT for up to 24 hours. Federated identity credentials (FICs) on a UAMI so external workloads — GitHub Actions, AKS, GKE, another cloud — present their own OIDC token and never hold an Azure secret. RBAC scoped tightly enough that a compromised workload can’t pivot, and the real ceilings (the ~4000 role-assignments-per-subscription limit, the older ~2000/3000 stages, Azure AD replication lag of up to ~30 minutes) that turn “it works in dev” into “it fails at scale.” It assumes you can create role assignments (Owner, User Access Administrator, or Role Based Access Control Administrator at the relevant scope) and have a basic grasp of Entra service principals and OAuth.

By the end you will design identity as a decoupled, governed resource; reason about every token request from DefaultAzureCredential down to the raw IMDS HTTP call; stand up secretless federation for CI and Kubernetes; scope RBAC at the resource, not the subscription; and diagnose the four failures that account for nearly every managed-identity ticket — “no managed identity found,” AADSTS70021, the 403-right-after-granting, and the cross-tenant trust that silently doesn’t apply.

What problem this solves

Secrets in configuration are the single most durable security debt in a cloud estate. A connection string in an app setting, a service-principal client secret in a CI variable, a storage account key checked into a Helm values file — each is a long-lived credential that must be stored, rotated, audited, and revoked, and each leaks eventually: into logs, into a fork, into a screen-share, into a breach dump. Managed identities exist to delete that whole category: the workload proves who it is by virtue of where it runs (an Azure VM, a Function, an AKS pod) or by presenting a short-lived token its own platform vouches for, and Entra hands back an Azure access token. Nobody holds a secret because there is no secret to hold.

What breaks without this, in production terms: a team uses a service principal with a client secret for everything, the secret hits its expiry on a Saturday, and every pipeline and app that referenced it fails at once with AADSTS7000215: Invalid client secret provided. Or the secret doesn’t expire and instead lives for three years in a variable group four people can read, until one leaves and offboarding misses it. Or a developer, blocked by a managed-identity setup that “didn’t work,” falls back to a storage account key in code “just to ship.” The pain is the recurring, audit-flagged toil of managing credentials that should not exist.

Who hits the managed-identity failures specifically: anyone running PaaS or containers who adopts identity but trips the defaults. System-assigned users hit the recreate-loses-roles trap; multi-identity apps hit the “which identity did you mean?” ambiguity; CI/CD teams hit AADSTS70021 on their first FIC subject; platform teams at scale hit the role-assignment ceiling and replication lag; cross-tenant designs hit the “managed identities can’t cross tenants the way you assumed” wall. Every one is diagnosable in minutes once you know the mechanism — and a multi-hour mystery if you don’t.

To frame the field before the deep dive, here is every managed-identity failure class this article covers, what it really is, and where to look first:

Failure class What is actually happening First question to ask First place to look
“No managed identity found” Identity not attached, or wrong client ID requested Is an identity attached, and did you pin its client ID? az webapp identity show / IMDS response body
403 right after granting a role RBAC replication lag, or a cached token with stale claims Did it work after ~5–30 min or a restart? az role assignment list + token iat claim
AADSTS70021 (no matching FIC) Presented subject/issuer doesn’t match the FIC Does the token’s sub/iss match byte-for-byte? Decoded OIDC token claims vs FIC config
Roles vanished after redeploy System-assigned SP recreated; assignments orphaned Was the parent resource recreated? principalId before vs after deploy
Hit the assignment ceiling ~4000 role assignments per subscription reached Are you assigning per-resource at scale? az role assignment list --all | wc -l
Cross-tenant access denied MI tenant boundary; SP not in target tenant Is this a single MI reaching across tenants? Tenant ID of the token vs the target resource

Learning objectives

By the end of this article you can:

Prerequisites & where this fits

You should already understand Entra service principals and app registrations at a basic level: a service principal (SP) is the local representation of an identity in a tenant that can hold role assignments, and a managed identity is a special, Azure-managed SP. You should know what an OAuth access token and a JWT are (header, claims, signature), what an OIDC issuer is, and how Azure RBAC layers role definition + principal + scope into a role assignment. Comfort with az in Cloud Shell, reading JSON output, and basic Bicep/Terraform is assumed.

This sits at the centre of the Identity & platform-security track. It builds on token fundamentals from OIDC and OAuth2 on Entra ID: Choosing the Right Flow (Auth Code, PKCE, Client Credentials) and the app-registration side of federation in Building a Secure OIDC Confidential Client in Entra ID: App Registrations, Secrets, and Workload Identity Federation. It is the secretless backbone under Secretless CI/CD: Workload Identity Federation for GitHub Actions and AKS and the concrete GitHub path in GitHub Actions to Azure with OIDC: Passwordless Deploys via Federated Credentials. Downstream it unlocks Azure Key Vault: Secrets, Keys and Certificates Done Right and pairs with AKS Cluster Identity: Managed Identity vs Service Principal and Why It Matters for Day-2.

A quick map of which layer owns which decision, so you involve the right person:

Layer What lives here Who usually owns it Managed-identity decisions it drives
Entra tenant SPs, app registrations, FICs, token issuance Identity team FIC subjects, multi-tenant, replication behaviour
Subscription / RG Role assignments, scope, assignment ceiling Platform / landing-zone team Scope strategy, the ~4000 limit, governance
Compute (VM/Func/AKS) Identity attachment, IMDS, AZURE_CLIENT_ID App / platform team System vs user-assigned, which ID is default
Data plane (KV/Storage/SQL) The resource the workload accesses Data / app owner Which data-plane role; vault RBAC mode; SQL DB user
External CI / cluster The OIDC issuer presenting a token DevOps / app team Issuer URL, ServiceAccount/repo subject

Core concepts

Six mental models make every later diagnosis obvious.

A managed identity is a service principal Azure babysits. Creating a managed identity creates an SP in your tenant with an auto-managed credential (a certificate the platform rotates roughly every ~45–90 days, which you never see or touch). Everything true of an SP is true here: it has an object ID (the principalId, what RBAC references) and, for a UAMI, a client ID (the clientId/appId, what your code requests a token as). The only difference from a normal SP is that you don’t manage the credential and, for system-assigned, you don’t manage the lifecycle either.

System-assigned is bound to a resource; user-assigned is a standalone resource. A system-assigned identity is enabled on a parent (a VM, a Function App) and shares that parent’s life — delete the parent, the SP is deleted, its role assignments orphaned. A user-assigned identity is its own Microsoft.ManagedIdentity/userAssignedIdentities resource you create independently and attach to one or many compute resources; it outlives any of them. This single distinction dictates blast radius, reuse, and whether you can do federation at all (FICs need a UAMI or an app registration — never system-assigned).

Tokens come from IMDS, not from your code calling Entra. Code on an Azure resource asks the Instance Metadata Service at the link-local 169.254.169.254 for a token, with a Metadata: true header. The platform — which knows the resource’s identity — fetches a token from Entra on the workload’s behalf and returns it. Your code never sees a credential; it only ever sees the resulting bearer token scoped to the resource/audience it asked for. For federation, the flow inverts: the external workload presents its OIDC token to Entra’s token endpoint, Entra validates it against a FIC, and returns an Azure token — same result, no IMDS, no secret.

The token is scoped to one audience and cached. Every token names an audience (aud) — https://vault.azure.net for Key Vault, https://storage.azure.com for Storage, https://management.azure.com for ARM. A token for Key Vault is useless against Storage. SDKs cache the token in memory until shortly before expires_on, and managed-identity tokens can live up to ~24 hours, so a long-running process may hold a token whose claims predate a role change. This is the reason “I granted the role but it still 403s.”

RBAC is definition + principal + scope, and scope is hierarchical. A role assignment grants a role definition (a set of Actions/DataActions) to a principal (here, the identity’s principalId) at a scope (management group → subscription → resource group → resource). Permissions inherit downward: a grant at the resource group applies to every resource in it. Least privilege means scoping to the individual resource and choosing the data-plane role, not a management role that also exposes keys.

Federation replaces a secret with a trusted token exchange. A federated identity credential is a trust record on a UAMI (or app registration) that says “accept tokens from this issuer whose subject is this, for this audience.” The audience for Entra’s exchange is always api://AzureADTokenExchange. When a GitHub job, a Kubernetes pod, or another cloud’s workload presents an OIDC token matching the FIC, Entra issues an Azure token for the identity. No client secret exists anywhere in the path — the workload’s own identity, vouched for by its own issuer, is the credential.

The vocabulary in one table

Pin down every moving part before the deep sections; the glossary repeats these for lookup.

Concept One-line definition Where it lives Why it matters
Managed identity An Entra SP whose credential Azure manages Your tenant The secretless principal you grant roles to
System-assigned Identity tied to one parent resource’s lifecycle On the resource Simple, but recreated → roles orphaned
User-assigned (UAMI) Standalone identity resource, attach to many Its own resource Stable principalId; required for FICs
principalId (object ID) The SP’s object ID Identity resource What every RBAC assignment references
clientId (appId) The app/client ID of the identity Identity resource (UAMI) What your code requests a token as
IMDS Link-local token endpoint at 169.254.169.254 Every Azure compute instance Issues tokens to in-Azure workloads
Audience (resource) The target service the token is for Token claim Wrong audience → token rejected by the API
Federated identity credential Trust record for an external OIDC issuer On a UAMI / app reg Lets external workloads in with no secret
Issuer / subject The OIDC iss / sub the FIC matches External IdP token Mismatch → AADSTS70021
Role assignment definition + principal + scope grant Subscription/RG/resource The actual permission
Data-plane role A *Data* role granting resource-internal access Role definition Access without exposing account keys
Replication lag Delay before a new SP/role is visible everywhere Azure AD / ARM The 403-right-after-granting cause

System-assigned vs user-assigned: lifecycle and blast radius

Both create an SP Azure manages — no credential ever lands in your code. The difference is lifecycle, and lifecycle dictates blast radius, reuse, and what capabilities you unlock.

Property System-assigned User-assigned (UAMI)
Lifecycle Tied to the parent; deleted with it Standalone resource; independent lifecycle
Resource type A property (identity.type) on the parent Microsoft.ManagedIdentity/userAssignedIdentities
Reuse across resources One identity, one resource One identity, many resources
principalId stability across redeploy No — SP recreated, assignments lost Yes — identity and principalId persist
Federated identity credentials Not supported Supported (the decisive capability)
clientId to pin in code Not applicable (only one identity) Yes — required when ≥2 identities attach
Max attached per resource 1 (system) 1 system + many user-assigned simultaneously
Cleanup model Self-cleaning with the parent Can orphan if ungoverned
Best for Throwaway/single-tenant-resource simple cases The production default; anything federated or shared

A system-assigned identity feels simpler, but its principalId is regenerated whenever the resource is recreated — and role assignments reference the principal ID, not the resource. Recreate a Function App and every Key Vault Secrets User grant silently evaporates: a clean 403 at runtime, because the assignment in your IaC “exists” but points at a principal that’s gone. This is the single most common managed-identity incident, and it is invisible in a terraform plan because the assignment resource itself didn’t change — only the principal it points to did.

User-assigned identities decouple the identity from its consumers. Create it once, in its own resource group, then attach it to the VM, AKS pod, or Function App. Redeploy the compute all day; the principalId and its grants stay put.

# Create a user-assigned identity as a standalone, reusable resource
az identity create \
  --name id-payments-prod \
  --resource-group rg-identity-prod \
  --location eastus2

# Capture the three values you will reference everywhere
CLIENT_ID=$(az identity show -n id-payments-prod -g rg-identity-prod --query clientId -o tsv)
PRINCIPAL_ID=$(az identity show -n id-payments-prod -g rg-identity-prod --query principalId -o tsv)
RESOURCE_ID=$(az identity show -n id-payments-prod -g rg-identity-prod --query id -o tsv)

The same thing in Bicep, where the identity is a first-class module output other deployments consume:

resource uami 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'id-payments-prod'
  location: location
}

output principalId string = uami.properties.principalId  // for RBAC assignments
output clientId string    = uami.properties.clientId     // for AZURE_CLIENT_ID
output resourceId string  = uami.id                       // for attaching to compute

Blast-radius rule I enforce: one identity per workload (a deployable unit with a single owner), not per resource and not spanning environments. Sharing across prod and non-prod means a non-prod compromise carries prod roles; per-resource means ten identities for one app and no reuse. Per-workload is the sweet spot.

When system-assigned is actually the right call

System-assigned isn’t wrong — it’s right for a narrow set of cases. The decision table:

Situation Pick Why
A single, long-lived resource never recreated in place System-assigned Simplest; self-cleaning; no extra resource to govern
Anything in a CI/CD pipeline that recreates compute User-assigned Survives recreation; roles don’t orphan
Workload needs federation (GitHub/k8s/other cloud) User-assigned FICs are UAMI-only
Several resources need the same identity User-assigned One identity, one set of grants, shared
You want to pre-create grants before the compute exists User-assigned Identity (and roles) can exist first
A throwaway lab VM you’ll delete tomorrow System-assigned Nothing to clean up; no governance burden
Many resources, each genuinely needing distinct access System-assigned (each) Per-resource isolation, accepting orphan risk

Identity attributes you’ll reference constantly

Three identifiers, three different uses — confusing them is a common source of “it doesn’t work”:

Identifier Also called Used for Where you get it
principalId Object ID, OID RBAC role assignments; SQL FROM EXTERNAL PROVIDER az identity show --query principalId
clientId App ID, appId Token requests (AZURE_CLIENT_ID, IMDS client_id) az identity show --query clientId
id Resource ID Attaching the identity to compute az identity show --query id
tenantId Directory ID Cross-tenant reasoning; token tid claim az account show --query tenantId

How the token flow works: IMDS, the metadata endpoint, and caching

When code on an Azure resource asks for a token, it does not call Entra directly. It calls the Azure Instance Metadata Service (IMDS) at the non-routable 169.254.169.254, passing a Metadata: true header (which defeats SSRF attacks that can’t set arbitrary headers). The platform injects the identity’s credential into that local endpoint; your code never sees a secret. Two query parameters carry the request:

Parameter Purpose Example Notes
api-version IMDS contract version 2018-02-01 Required; pin a known-good version
resource The audience/target service https://vault.azure.net Wrong value → token the API rejects
client_id Disambiguate which UAMI the identity’s clientId Needed when ≥2 identities attached
principal_id Alternative disambiguator the principalId Use instead of client_id if preferred
mi_res_id Alternative disambiguator the identity resource ID Use instead of client_id/principal_id

The audiences you’ll actually request, by target service:

Target service resource / audience Token consumer
Azure Key Vault https://vault.azure.net Secrets/keys/certs data plane
Azure Storage (blob/queue/table) https://storage.azure.com Storage data plane (OAuth)
Azure Resource Manager https://management.azure.com Control-plane / management APIs
Microsoft Graph https://graph.microsoft.com Directory/Graph APIs
Azure SQL Database https://database.windows.net SQL token auth
Azure Service Bus / Event Hubs https://servicebus.azure.net Messaging data plane
Azure Cosmos DB (RBAC) https://cosmos.azure.com Cosmos data-plane RBAC
# IMDS token request, pinned to a specific user-assigned identity
curl -s -H "Metadata: true" \
  "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net&client_id=${CLIENT_ID}"

The endpoint returns a JSON object — a JWT in access_token plus metadata:

Field Meaning Use
access_token The bearer JWT Send as Authorization: Bearer <token>
expires_on Unix epoch expiry Refresh before this; drives caching
expires_in Seconds until expiry Convenience form of the above
resource The audience granted Confirm it matches what you asked
token_type Bearer Always Bearer for these
client_id Which identity issued it Confirms disambiguation worked

In real code you never hand-roll this. DefaultAzureCredential walks a chain of credential sources so the same code runs locally (your dev login) and in Azure (managed identity), with no branching:

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

# Pin the user-assigned identity explicitly — never rely on the default
# when more than one identity could be attached.
credential = DefaultAzureCredential(
    managed_identity_client_id="00000000-0000-0000-0000-000000000000"  # the UAMI clientId
)
client = SecretClient(vault_url="https://kv-payments.vault.azure.net", credential=credential)
secret = client.get_secret("db-connection-string")

The credential chain, in order, and the footguns at each link:

Order Credential in the chain Reads from Footgun
1 Environment AZURE_CLIENT_ID/TENANT_ID/secret or cert Stale env vars silently win over MI
2 Workload Identity Federated token file (AKS) Needs the webhook + annotations set
3 Managed Identity IMDS Ambiguous if ≥2 UAMIs and no client ID
4 Azure CLI az login cache Works locally, never in prod
5 Azure PowerShell / azd / interactive Local dev sessions Dev-only; not for workloads

The footgun worth repeating: with multiple identities attached, DefaultAzureCredential with no managed_identity_client_id is ambiguous and fails — always pin the client ID. For a workload that is only ever managed-identity, ManagedIdentityCredential(client_id=...) is more explicit and skips the chain.

Caching behaviour you have to plan for

The SDK caches the access token in memory until close to expiry, so the next call is cheap — but those tokens can live up to ~24 hours. After granting a role, a long-lived process may keep using a cached token that predates the change, so it needs a refresh or restart before it sees the new access. Short-lived workers and request-scoped credentials sidestep this entirely. The caching reality:

Behaviour Detail Consequence
Cache scope In-memory, per credential object Not shared across processes/restarts
Refresh trigger Near expires_on (typically a few min before) New token re-reads current roles
Max token lifetime Up to ~24 h for MI tokens Stale claims can persist that long
New role visibility Only on next token issuance Restart or wait for refresh to see it
Revoked role Same — cached token still works until expiry Revocation isn’t instant for tokens

The practical rule: treat a managed-identity token like a short-lived lease you don’t control the renewal of. If you need a permission change to take effect now, restart the process (or recreate the credential) rather than waiting on the cache.

Assigning least-privilege RBAC and scoping to specific resources

The identity is worthless until it has a role at a scope. The mistake I see most is granting at subscription or resource-group level “to keep it simple.” Scope to the individual resource unless the workload genuinely needs the group.

The scope hierarchy and what each level means for blast radius:

Scope level Example scope string Grants access to Use when
Management group /providers/Microsoft.Management/managementGroups/mg-corp Every subscription under it Almost never for a workload identity
Subscription /subscriptions/<sub-id> Every resource in the subscription Rarely; platform automation only
Resource group /subscriptions/<sub>/resourceGroups/rg-x Every resource in the RG When the workload truly spans the RG
Resource …/providers/Microsoft.KeyVault/vaults/kv-x That one resource Default for least privilege
Sub-resource …/vaults/kv-x/... (where supported) A slice of the resource Tightest, where the service supports it
# Least privilege: data-plane read on ONE Key Vault, not the resource group
KV_ID=$(az keyvault show -n kv-payments -g rg-payments --query id -o tsv)

az role assignment create \
  --assignee-object-id "$PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$KV_ID"

--assignee-principal-type ServicePrincipal is not cosmetic: without it, az resolves the principal by querying Graph, which fails for a just-created identity that hasn’t replicated yet (Cannot find principal … in the directory). Always pass the object ID plus the explicit type to skip the Graph lookup and dodge the replication race.

Pick the purpose-built data-plane role, never a management role.

Workload needs Correct role Audience Avoid (and why)
Read Key Vault secrets Key Vault Secrets User vault.azure.net Contributor/Key Vault Administrator (too broad)
Manage Key Vault keys (crypto) Key Vault Crypto User vault.azure.net Key Vault Administrator (full control)
Read/write blobs Storage Blob Data Contributor storage.azure.com Contributor (exposes account keys)
Read-only blobs Storage Blob Data Reader storage.azure.com Storage Account Contributor (mgmt + keys)
Read queue messages Storage Queue Data Reader storage.azure.com Contributor
Query Azure SQL (none — contained DB user, see below) database.windows.net server-level Azure RBAC (doesn’t exist for data)
Cosmos data-plane a Cosmos SQL-role (data-plane) cosmos.azure.com Cosmos DB Account Contributor (mgmt only)
Call Microsoft Graph Graph app role (e.g. User.Read.All) graph.microsoft.com Directory roles (over-privileged)

Contributor on a storage account lets the identity list and regenerate account keys — a full bypass of the data-plane RBAC model. Reach for *Data* roles every time; they grant blob/queue/table access without ever exposing account keys.

Custom roles when built-ins don’t fit

When no built-in role is tight enough, a custom role with exactly the Actions/DataActions you need is the least-privilege answer. The shape:

{
  "Name": "Payments Blob Reader (custom)",
  "Description": "Read blobs in the payments container only",
  "Actions": [],
  "DataActions": [
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read"
  ],
  "NotDataActions": [],
  "AssignableScopes": [
    "/subscriptions/<sub-id>/resourceGroups/rg-payments"
  ]
}

Custom-role considerations that bite at scale: put data-plane operations in DataActions, not Actions (a data op in Actions does nothing); set AssignableScopes wide enough to assign where needed but no wider; respect the ~5000 custom roles per tenant limit (don’t generate one per workload); prefer enumerating actions over */read wildcards that over-grant; and remember definition changes replicate with the same lag as assignments.

Federated identity credentials: trust external issuers without secrets

This is the capability that changes the architecture. A federated identity credential (FIC) lets a user-assigned identity trust tokens from an external OIDC issuer — GitHub Actions, a GKE or self-hosted Kubernetes cluster, GitLab, another cloud, any compliant IdP. The workload presents its own OIDC token, Entra validates the issuer/subject against the FIC and returns an Entra token for your identity. No Azure secret ever leaves Azure. FICs work on user-assigned identities and app registrations — never system-assigned.

The four FIC fields and what each must be:

FIC field What it is Must equal Gotcha
issuer The external IdP’s OIDC issuer URL The token’s iss claim, exactly Trailing slash / wrong region URL fails
subject The workload’s identity claim The token’s sub claim, exactly Branch vs environment vs PR differ
audiences Who the token is for api://AzureADTokenExchange Always this value for Entra exchange
name A label for the FIC (any unique name) Cosmetic; for your own bookkeeping

GitHub Actions

az identity federated-credential create \
  --name github-deploy-main \
  --identity-name id-payments-prod \
  --resource-group rg-identity-prod \
  --issuer "https://token.actions.githubusercontent.com" \
  --subject "repo:my-org/payments-service:ref:refs/heads/main" \
  --audiences "api://AzureADTokenExchange"

The subject is the exact claim GitHub stamps into its OIDC token, and it differs by trigger:

GitHub trigger subject format Example
A branch repo:ORG/REPO:ref:refs/heads/BRANCH repo:my-org/payments:ref:refs/heads/main
A tag repo:ORG/REPO:ref:refs/tags/TAG repo:my-org/payments:ref:refs/tags/v1.4.2
An environment repo:ORG/REPO:environment:NAME repo:my-org/payments:environment:production
A pull request repo:ORG/REPO:pull_request repo:my-org/payments:pull_request

Get this string wrong and you get AADSTS70021: No matching federated identity record found — the single most common FIC failure. Match it character-for-character. The matching GitHub Actions job — note there is no client secret anywhere:

permissions:
  id-token: write   # REQUIRED to mint the OIDC token; without it azure/login fails
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}     # the identity's clientId
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}

AKS Workload Identity

AKS Workload Identity is the modern path; the older aad-pod-identity is deprecated and being retired. AKS Workload Identity is itself a federated credential — the cluster is the OIDC issuer and a Kubernetes ServiceAccount is the subject. Enable the OIDC issuer and the workload-identity add-on, then federate a ServiceAccount to the UAMI:

# Enable OIDC issuer + workload identity on the cluster
az aks update -n aks-prod -g rg-aks \
  --enable-oidc-issuer --enable-workload-identity

# Federate the in-cluster ServiceAccount to the identity
ISSUER=$(az aks show -n aks-prod -g rg-aks --query oidcIssuerProfile.issuerUrl -o tsv)
az identity federated-credential create \
  --name aks-payments-sa \
  --identity-name id-payments-prod -g rg-identity-prod \
  --issuer "$ISSUER" \
  --subject "system:serviceaccount:payments:payments-sa" \
  --audiences "api://AzureADTokenExchange"

The ServiceAccount and Pod that consume it — the annotation binds the SA to the identity’s client ID, the label opts the pod into the mutating webhook that injects the token:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-sa
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "<CLIENT_ID>"   # the UAMI clientId
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: payments, namespace: payments }
spec:
  template:
    metadata:
      labels:
        azure.workload.identity/use: "true"   # opt the pod into the webhook
    spec:
      serviceAccountName: payments-sa
      containers:
        - name: app
          image: myregistry.azurecr.io/payments:1.4.2

How AKS Workload Identity replaces the legacy approaches:

Approach Mechanism Status Why it changed
aad-pod-identity NMI pod intercepts IMDS per node Deprecated Race conditions, node-level scope, throttling
Pod-managed identity (preview) Earlier add-on form Superseded Replaced by federation model
Workload Identity OIDC federation, SA → UAMI via FIC Current Standards-based, per-pod, no IMDS interception

External Kubernetes, GKE, and other clouds

Any Kubernetes cluster that exposes its OIDC issuer can federate — point the FIC at the issuer URL and ServiceAccount subject. For GKE:

# Trust a specific ServiceAccount on an external (e.g. GKE) cluster
az identity federated-credential create \
  --name gke-batch-runner \
  --identity-name id-payments-prod \
  --resource-group rg-identity-prod \
  --issuer "https://container.googleapis.com/v1/projects/PROJECT/locations/LOCATION/clusters/CLUSTER" \
  --subject "system:serviceaccount:payments:batch-runner-sa" \
  --audiences "api://AzureADTokenExchange"

The subject formats across common issuers — this is where most cross-platform FICs fail:

Issuer / platform issuer shape subject shape
GitHub Actions https://token.actions.githubusercontent.com repo:ORG/REPO:...
AKS https://<region>.oic.prod-aks.azure.com/<tenant>/<guid>/ system:serviceaccount:NS:SA
GKE https://container.googleapis.com/v1/projects/.../clusters/... system:serviceaccount:NS:SA
GitLab CI https://gitlab.com (or self-managed URL) project_path:GROUP/PROJECT:ref_type:branch:ref:BRANCH
Generic OIDC IdP the IdP’s discovery issuer whatever sub the IdP stamps

The audience for Entra token exchange is always api://AzureADTokenExchange. There is a hard limit of 20 federated credentials per identity, so model subjects deliberately — a branch and an environment, not every ephemeral PR. The FIC limits and behaviours:

Limit / behaviour Value Implication
FICs per UAMI / app registration 20 Model subjects; don’t enumerate every branch
Audience for Entra exchange api://AzureADTokenExchange Fixed; not your API’s audience
Wildcard subjects Not supported (exact match) One FIC per distinct subject pattern
FIC propagation Subject to replication lag New FIC may need minutes to take effect
Issuer requirements Public OIDC discovery + JWKS reachable Air-gapped/private-only issuers can’t federate

Federation vs a managed-identity secret — there is no secret

The whole point bears a side-by-side. What you would have done with a service-principal secret, and what federation does instead:

Concern SP client secret Federated identity credential
Credential stored in CI/cluster Yes (a secret) No — nothing to store
Rotation You rotate on expiry None — short-lived tokens, auto
Leak blast radius Anyone with the secret is you A token is minutes-lived and audience-bound
Expiry outage risk Secret expires → outage No secret → no expiry outage
Trust model “Knows the secret” “Is this workload, per its issuer”
Revocation Rotate/delete the secret Delete the FIC; issuer stops being trusted

Using user-assigned identities across AKS, Functions, VMs, and App Service

The attach mechanism differs per service; the identity is the same. VMs and VMSS attach by resource ID:

az vm identity assign \
  --name vm-batch-01 -g rg-payments \
  --identities "$RESOURCE_ID"

App Service and Functions — assign, then tell the runtime which client ID is the default so the SDK doesn’t guess:

az functionapp identity assign \
  --name func-payments -g rg-payments \
  --identities "$RESOURCE_ID"

# Make this identity the default for the runtime's token requests
az functionapp config appsettings set \
  --name func-payments -g rg-payments \
  --settings "AZURE_CLIENT_ID=${CLIENT_ID}"

The per-service attach matrix and the gotcha each one carries:

Service How to attach How to pick the identity in code Gotcha
Virtual Machine / VMSS az vm/vmss identity assign --identities <id> IMDS client_id param Multiple UAMIs → must disambiguate
App Service / Functions az webapp/functionapp identity assign AZURE_CLIENT_ID app setting Default identity not auto-chosen
AKS pod FIC + SA annotation + pod label SA annotation client-id Needs OIDC issuer + add-on enabled
Container Apps az containerapp identity assign AZURE_CLIENT_ID env var System vs user-assigned per app
Logic Apps (Standard) Identity blade / ARM Connection auth = managed identity Some connectors lag MI support
Data Factory / Synapse Managed identity on the factory Linked-service auth type System-assigned by default; UAMI optional

The Bicep for attaching a UAMI to a web app — note the userAssignedIdentities map keyed by resource ID:

resource site 'Microsoft.Web/sites@2023-12-01' = {
  name: 'func-payments'
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${uami.id}': {}   // attach the UAMI by its resource ID
    }
  }
  properties: {
    serverFarmId: plan.id
    siteConfig: {
      appSettings: [
        { name: 'AZURE_CLIENT_ID', value: uami.properties.clientId }  // default for the SDK
      ]
    }
  }
}

One mental model covers all three of VM/Function/AKS: the identity is a stable principal with grants; the only variation is the plumbing that makes a token reachable from the workload — IMDS for in-Azure compute, the federated-token file for AKS.

Accessing Key Vault, Storage, and SQL with managed identity

Key Vault needs the data-plane role (above) and the vault set to Azure RBAC authorization. A vault still on the legacy access-policy model ignores your role assignment entirely — a silent failure that sends people in circles:

az keyvault update -n kv-payments -g rg-payments \
  --enable-rbac-authorization true

The two Key Vault authorization models, and why mixing them up is a classic 403:

Model How access is granted Your role assignment is… Migration note
Azure RBAC (recommended) RBAC roles at vault/secret scope Honoured --enable-rbac-authorization true
Access policies (legacy) Per-principal policy on the vault Ignored Add a policy or migrate to RBAC

Storage uses the OAuth path automatically once the identity holds a *Data* role — the SDK requests a https://storage.azure.com token, no connection string or account key. If you still pass a connection string, the SDK uses that and your RBAC grant is moot, which masks a missing role until you remove the string.

Azure SQL trips people up: there is no Azure RBAC role for the SQL data plane. Create a contained database user mapped to the identity and grant roles in T-SQL — connect as an Entra admin and run:

-- Map the user-assigned identity into the database by its display name
CREATE USER [id-payments-prod] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [id-payments-prod];
ALTER ROLE db_datawriter ADD MEMBER [id-payments-prod];

The app connects with Authentication=Active Directory Managed Identity;User Id=<CLIENT_ID>; — the driver pulls the token from IMDS. How the three data services differ in their authorization plumbing:

Service Azure RBAC for data? Extra step required Token audience
Key Vault Yes (*Secrets/Crypto/Certificates User) Set vault to RBAC mode vault.azure.net
Storage (blob/queue/table) Yes (*Data* roles) Don’t also pass a connection string storage.azure.com
Azure SQL No Contained DB user (FROM EXTERNAL PROVIDER) database.windows.net
Cosmos DB Data-plane RBAC (separate role system) Assign a Cosmos SQL role cosmos.azure.com

Cross-tenant realities and the assignment ceilings

Two topics catch architects off guard at scale: what managed identities can and cannot do across tenants, and the ceilings (role-assignment count, replication lag, FIC count) that make a working dev setup fail in a large estate.

Cross-tenant: what a managed identity can and can’t do

A managed identity lives in one tenant — the tenant of the subscription that hosts it. Its SP exists only there, so its RBAC assignments only apply to resources in subscriptions associated with that tenant. You cannot take one managed identity and grant it a role on a resource in a different tenant the way you would within a tenant. The options when you genuinely need cross-tenant access:

Need Right approach Why not “just a managed identity”
One workload, resources in its own tenant Managed identity The normal case; nothing special
App accessed by users/apps in many tenants Multi-tenant app registration MI is single-tenant; app reg can be multi-tenant
CI in tenant A deploying to tenant B FIC on a UAMI in tenant B Federate the CET token into B’s identity
Service in tenant A calling an API in tenant B App reg + admin consent in B Cross-tenant API access is an app-consent problem
B2B/guest access to resources Entra External ID / cross-tenant access settings Identity collaboration, not a workload identity

The clean pattern for “CI in one place, deploy to many tenants” is federation into a UAMI in each target tenant: the CI’s OIDC token is the same, but each tenant has its own UAMI with a FIC trusting that issuer/subject and its own scoped roles. No secret crosses any boundary, and each tenant’s blast radius is its own. For app-to-app cross-tenant calls, you’re in multi-tenant app registration + admin consent territory — covered in Building a Secure OIDC Confidential Client in Entra ID and the B2B side in Securing B2B Collaboration with Entra External ID.

The role-assignment ceiling (the 1000/2000/4000 trap)

There is a hard cap on role assignments per subscription. It has been raised over time — historically ~2000, then ~3000, and currently ~4000 (with a portion reserved). Teams that assign per-resource at scale (one app × one role × one Key Vault × N environments, multiplied across hundreds of services) hit it, and the failure is abrupt: RoleAssignmentLimitExceeded, and no new assignments can be created in the subscription until you delete some. The strategies to stay under it:

Strategy How it reduces assignment count Trade-off
Assign at RG scope, not per-resource One assignment covers many resources Looser scope; weigh blast radius
Use a group as the principal One assignment to a group of identities Indirection; group governance needed
Consolidate identities per workload Fewer principals → fewer assignments The per-workload rule already helps
Split into more subscriptions Each subscription has its own ceiling More subscriptions to manage
Prune stale assignments Remove orphaned/decommissioned grants Needs the inventory discipline below
Custom role spanning resources One role assigned once vs many built-ins Custom-role maintenance

The honest tension: least privilege wants per-resource scope, the ceiling pushes toward broader scope or grouping. The resolution is usually groups of identities (assign a role once to an Entra group, add identities to the group) plus subscription segmentation — not abandoning resource scope wholesale. Check where you stand:

# How many role assignments exist in the current subscription (vs the ~4000 ceiling)
az role assignment list --all --query "length(@)"

Azure AD replication lag (the 403-right-after-granting)

Azure AD is geo-distributed and eventually consistent. A newly created SP, a new role assignment, or a new FIC is written to one replica and propagates to others over seconds to — worst case — up to ~30 minutes. During that window, a token issued from a replica that hasn’t caught up won’t carry the new permission, and you get a 403 that “shouldn’t” happen. The lag sources and how to handle each:

What’s lagging Typical delay Symptom Mitigation
New SP visible to RBAC seconds–minutes Cannot find principal in directory on assign Pass --assignee-object-id + --assignee-principal-type
New role assignment effective seconds–~30 min 403 despite the grant existing Retry with backoff; don’t fail the deploy hard
New FIC effective seconds–minutes AADSTS70021 transiently after create Brief wait before first federated run
Token cache vs new role up to ~24 h 403 long after the grant Restart the process / new credential
Custom-role definition change seconds–minutes New action not yet allowed Retry; verify with az role definition list

The defensive coding rule: never treat a 403 immediately after a grant as fatal. Retry with exponential backoff for the first several minutes, and only after that treat it as a real authorization problem. In CI, put a short wait (or a retry loop) between “create role assignment” and “use it.”

Architecture at a glance

This article ships no diagram — the architecture is best held as a mental model you can reconstruct, because the moving parts differ per workload while the shape stays constant. Build the picture in three planes.

The identity plane (left). A user-assigned identity sits as its own resource in a dedicated rg-identity-prod. It has a stable principalId (the object RBAC points at) and a clientId (what code requests a token as). Hanging off it are up to 20 federated identity credentials, each a trust record naming an external issuer and subject. Nothing here ever holds a secret; the platform manages the identity’s own credential invisibly.

The compute plane (centre). Workloads attach to that identity by different plumbing but reach the same principal. An Azure VM or Function asks IMDS at 169.254.169.254 (with Metadata: true, the audience, and client_id) and gets a bearer JWT. An AKS pod doesn’t use IMDS — the workload-identity webhook projects a federated ServiceAccount token into the pod, which the SDK exchanges at Entra’s token endpoint (api://AzureADTokenExchange) for an Azure token. A GitHub Actions job or another cloud’s workload does the same exchange from outside Azure, presenting its own OIDC token. Three different left-hand entries, one identity, one resulting token.

The resource plane (right). The Azure token, scoped to one audience (vault.azure.net, storage.azure.com, …), is presented to the target service, which checks RBAC: does this principalId hold a role at this scope? If yes — and the Key Vault is in RBAC mode, the storage account isn’t bypassed by a connection string, or the SQL contained user exists — the call succeeds. If the role was granted seconds ago and the replica is behind, or the cached token predates the grant, it 403s until consistency catches up.

Read the whole thing left to right and the diagnostic method falls out: who is the principal (identity plane), how did it get a token (compute plane), and does that principal hold the right role at the right scope for this audience (resource plane)? Every managed-identity failure lands in exactly one of those three planes — attachment/disambiguation in the compute plane, FIC subject in the identity plane, RBAC scope and mode in the resource plane — and naming the plane is most of the fix.

Real-world scenario

Northwind Logistics runs a multi-tenant SaaS on AKS: about 140 microservices across three clusters in Central India and East US 2, deployed by GitHub Actions, reading secrets from Key Vault and data from Azure SQL and Blob Storage. The platform team is six engineers; the original design gave every service its own system-assigned identity with Key Vault and Storage role assignments declared in Terraform, and CI authenticated with a single service-principal client secret stored as a GitHub org secret.

Two incidents, six weeks apart, forced a redesign. The first: a blue/green cluster migration. Standing up new node pools recreated several workloads, regenerating their system-assigned principalIds. Terraform showed every azurerm_role_assignment as present — terraform plan was clean — but runtime tokens 403’d against Key Vault because the assignments pointed at principal IDs that no longer existed. Eleven services were down for ninety minutes, and the on-call wrote: “the IaC lied — the assignment existed, the principal didn’t.”

The second: the CI client secret expired at 23:00 on a Sunday. Every pipeline failed with AADSTS7000215: Invalid client secret provided. The secret had a two-year life and the reminder had been set by someone who’d since left. No deploys — including the hotfix pipeline — until someone with Application Administrator minted a new secret at 01:40.

The redesign addressed both at the root. Every workload moved to a per-service user-assigned identity in a dedicated rg-identity-prod, federated to each cluster’s OIDC issuer via AKS Workload Identity, with role assignments keyed off the stable principal_id — cluster rebuilds no longer touched identities or grants. CI moved to federated identity credentials — a UAMI per target subscription with a FIC trusting repo:northwind/<service>:environment:production, and azure/login@v2 with no secret anywhere. The expiry-outage class was deleted because there was no longer a secret to expire.

resource "azurerm_user_assigned_identity" "svc" {
  name                = "id-${var.service_name}-${var.env}"
  resource_group_name = azurerm_resource_group.identity.name
  location            = var.location
}

resource "azurerm_federated_identity_credential" "aks" {
  name                = "aks-${var.service_name}"
  resource_group_name = azurerm_resource_group.identity.name
  parent_id           = azurerm_user_assigned_identity.svc.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = var.aks_oidc_issuer_url
  subject             = "system:serviceaccount:${var.namespace}:${var.service_name}-sa"
}

# Stable principal_id — survives cluster and pod recreation
resource "azurerm_role_assignment" "kv" {
  scope                = azurerm_key_vault.svc.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.svc.principal_id
}

Then the ceiling bit. At 140 services × (Key Vault + Blob + a couple of others) × three environments, the production subscription’s role-assignment count crossed ~3,900 and a routine terraform apply failed with RoleAssignmentLimitExceeded. The fix wasn’t to loosen scope blindly: they moved the read-only blob grants to an Entra group (grp-blob-readers) assigned once at the storage-account scope, added each service’s identity to the group, and split the estate across two subscriptions by environment. Assignment count per subscription dropped under 1,500 with room to grow, and least-privilege resource scope stayed intact for the sensitive Key Vault grants. The lesson on the wall: “identity is a first-class resource — decouple it, federate it, and count your assignments before the platform counts them for you.”

The redesign as a before/after, because the shape of the change is the lesson:

Dimension Before After
Identity type System-assigned per service User-assigned per service
Survives cluster rebuild No (roles orphaned) Yes (stable principalId)
CI auth Single SP client secret Per-subscription FIC, no secret
Secret-expiry outage risk High (two-year secret) Eliminated
Blob read grants Per-service, per-resource One Entra group, assigned once
Assignment count (prod sub) ~3,900 (hit the ceiling) <1,500 (split + grouped)
Blast radius of one compromise That service’s roles Unchanged — still just its roles

Advantages and disadvantages

The managed-identity model deletes the secret problem but introduces its own subtleties. Weigh it honestly:

Advantages (why this model wins) Disadvantages (why it bites)
No secret to store, rotate, leak, or expire — the entire credential-management category disappears The “no secret” magic hides the token flow; debugging requires understanding IMDS/FIC you can’t see
The platform rotates the identity’s credential invisibly (~45–90 days) You have no control over that credential; you can’t pre-stage or inspect it
User-assigned identities survive any redeploy — roles never orphan System-assigned (the default-feeling choice) does orphan roles, abruptly
Federation lets external workloads in with zero Azure secrets subject/issuer must match byte-for-byte or AADSTS70021; easy to get wrong
RBAC + data-plane roles give precise, auditable access Scope and role choice are easy to over-grant; Contributor quietly exposes keys
One identity reusable across many resources Ungoverned UAMIs become orphans nobody owns
Tokens are short-lived and audience-bound — small blast radius Token caching (up to ~24 h) delays role changes taking effect
Works identically across VM/Function/AKS/CI via one model Per-service attach plumbing differs; multi-identity needs explicit client_id
Scales to large estates The ~4000-assignment ceiling and replication lag are real walls at scale

The model is right for essentially every workload that today holds a secret — which is most of them. It bites hardest on teams that adopt it with defaults (system-assigned, no client_id pinning, no FIC-subject discipline) and on platform teams operating at a scale where the ceiling and replication behaviour stop being theoretical. Every disadvantage is manageable — but only if you know it exists, which is the entire point of this article.

Hands-on lab

Create a user-assigned identity, attach it to a VM, fetch a token from IMDS by hand, read a Key Vault secret, then add a GitHub Actions federated credential — all minimal-cost (a B1s VM and a standard Key Vault; delete at the end). Run in Cloud Shell (Bash).

Step 1 — Variables and resource group.

RG=rg-mi-lab
LOC=eastus2
SUFFIX=$RANDOM
az group create -n $RG -l $LOC -o table

Step 2 — Create the user-assigned identity and capture its IDs.

az identity create -n id-lab -g $RG -l $LOC -o table
CLIENT_ID=$(az identity show -n id-lab -g $RG --query clientId -o tsv)
PRINCIPAL_ID=$(az identity show -n id-lab -g $RG --query principalId -o tsv)
RESOURCE_ID=$(az identity show -n id-lab -g $RG --query id -o tsv)
echo "client=$CLIENT_ID  principal=$PRINCIPAL_ID"

Expected: a UAMI row, and two GUIDs printed. Note they differ — clientIdprincipalId.

Step 3 — Create a Key Vault in RBAC mode and a secret.

KV=kv-lab-$SUFFIX
az keyvault create -n $KV -g $RG -l $LOC --enable-rbac-authorization true -o table
# You need a data-plane role yourself to write the secret:
ME=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --assignee-object-id "$ME" --assignee-principal-type User \
  --role "Key Vault Secrets Officer" --scope $(az keyvault show -n $KV -g $RG --query id -o tsv)
sleep 30   # let the role replicate before writing
az keyvault secret set --vault-name $KV --name db-conn --value "Server=demo;" -o none

Expected: a vault with enableRbacAuthorization: true; the secret set without error after the wait.

Step 4 — Grant the identity read access to the vault.

az role assignment create \
  --assignee-object-id "$PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope $(az keyvault show -n $KV -g $RG --query id -o tsv)

Note the explicit --assignee-principal-type ServicePrincipal — it avoids the Graph lookup that fails on a fresh identity.

Step 5 — Create a VM, attach the identity.

az vm create -n vm-lab -g $RG --image Ubuntu2204 --size Standard_B1s \
  --assign-identity "$RESOURCE_ID" --admin-username azureuser \
  --generate-ssh-keys -o table

Expected: a VM with the UAMI attached (identity.type includes UserAssigned).

Step 6 — From inside the VM, fetch a token from IMDS and read the secret. SSH in (az ssh vm or the public IP), then:

# Raw IMDS token request for the Key Vault audience, pinned to our identity
TOKEN=$(curl -s -H "Metadata: true" \
  "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net&client_id=<CLIENT_ID>" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")

# Use the bearer token directly against the Key Vault data plane
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<KV>.vault.azure.net/secrets/db-conn?api-version=7.4" | python3 -m json.tool

Expected: the secret JSON with "value": "Server=demo;". No secret, key, or password was used to read it — only the identity the VM is.

Step 7 — Add a GitHub Actions federated credential (no secret).

az identity federated-credential create \
  --name gh-main --identity-name id-lab -g $RG \
  --issuer "https://token.actions.githubusercontent.com" \
  --subject "repo:my-org/my-repo:ref:refs/heads/main" \
  --audiences "api://AzureADTokenExchange"
az identity federated-credential list --identity-name id-lab -g $RG \
  --query "[].{name:name, subject:subject}" -o table

Expected: one FIC listed with the exact subject you set. A workflow on my-org/my-repo’s main branch could now azure/login as this identity with no client secret.

Validation checklist. You created a standalone identity, granted it a data-plane role scoped to one vault, fetched a real token from IMDS by hand (proving the flow), read a secret with only that token, and federated GitHub Actions — the full arc, no secret anywhere. The steps mapped to what each proves:

Step What you did What it proves
2 Create UAMI, capture IDs clientIdprincipalId; three distinct identifiers
3 Vault in RBAC mode + sleep 30 RBAC mode is required; replication lag is real
4 Grant with explicit principal type The flag that dodges the Graph-lookup race
6 Raw IMDS fetch + bearer call The token flow, demystified end to end
7 Create a FIC Secretless external trust in one command

Cleanup (avoid lingering charges).

az group delete -n $RG --yes --no-wait

Cost note. A Standard_B1s VM and a standard Key Vault for an hour are a few rupees; deleting the resource group stops everything. Key Vault operations are billed per 10,000 transactions — a handful here is effectively free.

Common mistakes & troubleshooting

This is the playbook — bookmark it. First as a scannable table, then the full reasoning for the entries that bite hardest.

# Symptom Root cause Confirm (exact cmd / portal path) Fix
1 ManagedIdentityCredential authentication unavailable / “no managed identity” No identity attached, or wrong/ambiguous client_id az webapp identity show; raw IMDS call returns an error Attach identity; pin AZURE_CLIENT_ID/client_id
2 403 immediately after granting a role RBAC replication lag, or cached token with stale claims Re-test after ~5–30 min or restart; az role assignment list --assignee Retry with backoff; restart the process
3 AADSTS70021: No matching federated identity record found FIC subject/issuer ≠ the token’s claims Decode the OIDC token; compare sub/iss to the FIC Match subject/issuer byte-for-byte
4 Roles vanished after a redeploy System-assigned SP recreated; assignments orphaned Compare principalId before vs after deploy Move to user-assigned; key roles off stable principalId
5 RoleAssignmentLimitExceeded on assign Hit the ~4000 assignments-per-subscription ceiling az role assignment list --all --query "length(@)" Group principals; broaden scope; split subscriptions; prune
6 Key Vault 403 with a valid role Vault still on access policies, not RBAC az keyvault show --query properties.enableRbacAuthorization --enable-rbac-authorization true (or add a policy)
7 Cannot find principal in the directory on az role assignment create New SP hasn’t replicated; az did a Graph lookup The assignee was passed by name, not object ID Pass --assignee-object-id + --assignee-principal-type
8 Storage call uses keys despite a *Data* role A connection string/account key is still configured Check app settings for a connection string Remove the string; force the OAuth (token) path
9 DefaultAzureCredential ambiguous / picks the wrong identity ≥2 UAMIs attached, no client_id pinned az vm identity show lists multiple UAMIs Set managed_identity_client_id / AZURE_CLIENT_ID
10 AKS pod gets no token; SDK falls back Webhook label/annotation missing or add-on off kubectl describe sa; check pod label azure.workload.identity/use Add SA annotation + pod label; enable add-on/issuer
11 Cross-tenant access denied MI is single-tenant; SP not in the target tenant Compare token tid to the resource’s tenant Use a UAMI+FIC in the target tenant, or a multi-tenant app reg
12 Azure SQL login fails for the identity No contained DB user mapped Query sys.database_principals for the name CREATE USER [..] FROM EXTERNAL PROVIDER + roles
13 FIC works for main but fails for a PR/env Subject format differs per trigger Decode the job’s token sub Add a second FIC with the correct subject form
14 App still 403s long after the grant Long-lived process holding a cached token Process uptime exceeds the grant time Restart the process / recreate the credential

The expanded form for the entries that cost the most time:

1. ManagedIdentityCredential authentication unavailable / “no managed identity found.” Root cause: either no identity is attached to the compute, or multiple are attached and the SDK can’t tell which to use, or the requested client_id doesn’t match any attached identity. Confirm: az webapp identity show -n <app> -g <rg> (or az vm identity show) — is anything attached? Then a raw IMDS call: if it returns an error body about the identity, that’s your signal. Fix: attach the identity (… identity assign --identities <id>), and pin AZURE_CLIENT_ID (apps) or the IMDS client_id param (raw) to the exact UAMI.

2. 403 immediately after granting a role. Root cause: Azure AD replication lag (the grant hasn’t propagated to the replica issuing your token — up to ~30 min worst case) or a cached token that predates the grant (up to ~24 h). Confirm: az role assignment list --assignee "$PRINCIPAL_ID" --query "[].{role:roleDefinitionName,scope:scope}" shows the grant exists; the call still 403s. Re-test after a few minutes or after restarting the process. Fix: retry with exponential backoff for the first several minutes; restart the workload (or recreate the credential) to force a fresh token. Never hard-fail a deploy on a 403 that’s seconds old.

3. AADSTS70021: No matching federated identity record found. Root cause: the presented OIDC token’s sub/iss doesn’t match any FIC on the identity — almost always a subject typo or the wrong subject format for the trigger (branch vs environment vs PR). Confirm: decode the token the workload presents (GitHub: print the OIDC token claims in a debug step; AKS: kubectl create token <sa> --duration=10m then decode) and compare sub/iss to az identity federated-credential list. Fix: make the FIC subject/issuer match the claims exactly, or add a second FIC for the other subject form.

4. Roles vanished after a redeploy. Root cause: a system-assigned identity’s SP was recreated when its parent resource was recreated, regenerating the principalId; existing assignments now point at a principal that no longer exists. Confirm: capture principalId before and after the deploy — if it changed, that’s it. The assignment resource still “exists” in IaC, which is the trap. Fix: move to a user-assigned identity (stable principalId) and key all role assignments off it; this is the single highest-leverage change in the whole topic.

6. Key Vault 403 with a valid role. Root cause: the vault is on the legacy access-policy model, which ignores RBAC role assignments entirely. Confirm: az keyvault show -n <kv> -g <rg> --query properties.enableRbacAuthorization returns false. Fix: az keyvault update --enable-rbac-authorization true (validate nothing relied on access policies first), or add an access policy if you’re staying on that model. See Key Vault RBAC vs Access Policies: Choosing and Migrating the Permission Model.

11. Cross-tenant access denied. Root cause: a managed identity is single-tenant; its SP exists only in its home tenant, so it can’t be granted a role on a resource in a different tenant. Confirm: decode the token’s tid (tenant) claim and compare to the target resource’s tenant. Fix: federate the source’s OIDC token into a UAMI created in the target tenant (each tenant gets its own identity + FIC + scoped roles), or for app-to-app access use a multi-tenant app registration with admin consent in the target tenant.

Best practices

Security notes

The security controls that also prevent the incidents in this article — secure and resilient pull the same way:

Control Mechanism Secures against Also prevents
User-assigned over system-assigned Stable principalId Orphaned grants exploited later Redeploy role-loss outages
Data-plane roles only *Data* role definitions Account-key exposure via Contributor Accidental over-privilege
Tight FIC subjects Exact repo:.../environment:... Unintended workloads assuming the identity AADSTS70021 from sloppy subjects
Key Vault RBAC mode enableRbacAuthorization Policy/RBAC confusion bypassing intent The silent KV-403 class
FIC-write least privilege RBAC on federated-credential write Rogue trust records Untracked federation sprawl
Resource Graph inventory Scheduled KQL audit Orphaned/over-scoped identities Hitting the assignment ceiling blind

Cost & sizing

Managed identities are, directly, free — no charge to create a system- or user-assigned identity, attach it, fetch tokens from IMDS, or create federated identity credentials. The cost story is indirect: what they let you stop paying for, and the small charges on the resources they access.

A rough picture for a mid-size estate: identities and FICs themselves are ₹0; the Key Vault those 140 services read might run ₹500–2,000/month in transactions if uncached (far less if cached); governance tooling is ₹0 in Azure cost and a few engineer-hours a month. The genuine line on the spreadsheet is the absence of a secret-rotation outage. The cost factors:

Factor What you pay Rough INR What it buys / costs you
Managed identity (system or user) Nothing ₹0 The secretless principal
Federated identity credentials Nothing ₹0 Secretless external trust
IMDS / token fetches Nothing ₹0 Tokens on demand
Key Vault transactions (uncached) Per 10k operations ₹500–2,000/mo at scale Cache to cut both cost and latency
Governance (inventory, tagging) Engineer time ₹0 Azure cost Avoids orphan/ceiling incidents
Avoided secret-expiry outage (negative cost) saves hours of senior time The real ROI

Interview & exam questions

1. What is the practical difference between a system-assigned and a user-assigned managed identity? A system-assigned identity is tied to one parent resource’s lifecycle (deleted with it, principalId regenerated if the resource is recreated); a user-assigned identity is a standalone resource you attach to many resources, with a stable principalId that survives redeploys — and, critically, only user-assigned identities support federated identity credentials. The user-assigned model is the production default because role assignments don’t orphan on recreation.

2. Walk through how an Azure VM gets a token without holding a secret. Code on the VM calls the Instance Metadata Service at 169.254.169.254/metadata/identity/oauth2/token with a Metadata: true header and a resource (audience) parameter. The platform, which knows the VM’s identity, fetches a token from Entra on the VM’s behalf and returns a bearer JWT plus expires_on. The VM never sees a credential — only the resulting token, scoped to the audience it asked for.

3. Why might code with multiple attached identities fail with “no managed identity found,” and how do you fix it? With more than one user-assigned identity attached, IMDS (and DefaultAzureCredential with no managed_identity_client_id) can’t tell which identity you mean, so the request is ambiguous and fails. Fix it by pinning the identity’s clientId — via the IMDS client_id parameter, the AZURE_CLIENT_ID app setting, or ManagedIdentityCredential(client_id=...).

4. What is a federated identity credential and what problem does it solve? A FIC is a trust record on a user-assigned identity (or app registration) that tells Entra to accept tokens from an external OIDC issuer with a specific subject, for the audience api://AzureADTokenExchange. It lets external workloads — GitHub Actions, AKS, GKE, other clouds — present their own OIDC token and receive an Azure token, with no Azure secret stored anywhere. It eliminates the client-secret-in-CI class of problem, including expiry outages and leaks.

5. A GitHub Actions deploy fails with AADSTS70021. What’s wrong and how do you confirm? The OIDC token GitHub presented doesn’t match any FIC subject/issuer on the identity — usually a subject typo or the wrong subject format (a branch subject repo:org/repo:ref:refs/heads/main when the job ran for an environment or PR). Confirm by decoding the job’s OIDC token and comparing its sub/iss to az identity federated-credential list. Fix by matching the FIC exactly, or adding a FIC for the correct subject form.

6. You grant a managed identity Key Vault Secrets User and it still 403s. List the possible causes. (a) The vault is on access-policy mode, which ignores RBAC — set --enable-rbac-authorization true; (b) replication lag — the grant hasn’t propagated yet (retry for a few minutes); © a cached token that predates the grant — restart the process; (d) the wrong scope (granted on the RG but the vault is elsewhere, or vice versa); (e) the wrong audience requested. Check the vault’s auth mode and the assignment’s scope first.

7. Why does scoping RBAC to Contributor on a storage account violate least privilege even for “just reading blobs”? Contributor (a management-plane role) lets the identity list and regenerate account keys, which is a full bypass of the data-plane RBAC model — anyone with the key has total access. The least-privilege choice is a data-plane role (Storage Blob Data Reader/Contributor) that grants blob access without ever exposing account keys.

8. Explain the role-assignment ceiling and how to design around it. A subscription caps role assignments (~4000 currently; historically ~2000/3000). Per-resource assignment at scale (services × roles × resources × environments) hits it, and the failure (RoleAssignmentLimitExceeded) blocks all new assignments. Design around it by assigning roles to Entra groups of identities (one assignment per group), broadening scope where blast radius allows, pruning orphaned assignments, and segmenting the estate across subscriptions (each has its own ceiling).

9. How do managed identities behave across tenants, and what do you use when you need cross-tenant access? A managed identity is single-tenant — its SP exists only in its home tenant, so you can’t grant it a role on a resource in another tenant. For “CI in one place deploying to many tenants,” federate the CI’s OIDC token into a separate UAMI in each target tenant. For app-to-app cross-tenant access, use a multi-tenant app registration with admin consent in the target tenant; for user collaboration, B2B/External ID.

10. What replaced aad-pod-identity for AKS, and how does it work? AKS Workload Identity replaced the deprecated aad-pod-identity. The cluster exposes an OIDC issuer; a Kubernetes ServiceAccount is federated to a UAMI via a FIC (system:serviceaccount:NS:SA as the subject). A mutating webhook projects a federated token into pods labelled azure.workload.identity/use: "true", and the SDK exchanges it for an Azure token. It’s per-pod, standards-based, and doesn’t intercept IMDS like the old NMI did.

11. Why can a permission change take up to a day to take effect for a running workload? Managed-identity access tokens are cached in memory by the SDK and can live up to ~24 hours. A long-lived process keeps using its cached token — which carries the claims/roles from when it was issued — until that token expires and a new one is fetched. To force the change immediately, restart the process or recreate the credential object.

12. Why pass --assignee-principal-type ServicePrincipal when assigning a role to a just-created identity? Without it, az role assignment create resolves the assignee by querying Microsoft Graph, which fails for an SP that hasn’t replicated yet (Cannot find principal in the directory). Passing the object ID plus the explicit principal type skips the Graph lookup, avoiding the replication race that otherwise breaks first-time IaC runs.

These map to AZ-500 (Security Engineer)manage identity and access, managed identities, RBAC, Key Vault access — AZ-204 (Developer)implement secure cloud solutions, managed identities and Key Vault references — and SC-300 (Identity and Access Administrator)implement workload identities and federation. The AKS/CI angle touches AZ-400 (secretless pipelines). A compact cert mapping:

Question theme Primary cert Objective area
System vs user-assigned; lifecycle AZ-204 / AZ-500 Implement managed identities
IMDS token flow; caching AZ-204 Secure cloud solutions; tokens
Federated identity credentials SC-300 / AZ-400 Workload identity federation
RBAC scope, data-plane roles AZ-500 Manage access to Azure resources
Assignment ceiling, replication lag AZ-500 / AZ-104 RBAC at scale; governance
AKS Workload Identity AZ-400 / CKA-adjacent Secretless pipelines & clusters
Cross-tenant, multi-tenant apps SC-300 External identities & app registration

Quick check

  1. You recreate a Function App via your pipeline and its Key Vault calls suddenly 403, even though the role assignment still exists in Terraform. What kind of identity is in use, and what’s the one-line fix?
  2. Your code on a VM with two user-assigned identities throws “no managed identity found.” What did you forget, and where do you set it?
  3. A GitHub Actions job federating to Azure fails with AADSTS70021. Name the two FIC fields you compare against the token, and the audience value a FIC must use.
  4. You granted Key Vault Secrets User 30 seconds ago and still get 403. Give two distinct causes and the fix for each.
  5. Your platform subscription suddenly refuses new role assignments with RoleAssignmentLimitExceeded. What ceiling did you hit, and name two ways to design around it without abandoning least privilege.

Answers

  1. It’s a system-assigned identity — recreating the parent regenerated its principalId, orphaning the assignment (which still “exists” but points at a gone principal). Fix: move to a user-assigned identity and key the role assignment off its stable principalId.
  2. You forgot to pin the identity’s clientId — with two UAMIs attached, the token request is ambiguous. Set AZURE_CLIENT_ID (app setting / env var) or pass managed_identity_client_id to the credential / client_id to the IMDS call.
  3. Compare the FIC’s subject (to the token’s sub) and issuer (to the token’s iss); the FIC’s audiences must be api://AzureADTokenExchange. AADSTS70021 almost always means the subject doesn’t match the trigger’s format (branch vs environment vs PR).
  4. (a) Replication lag — the grant hasn’t propagated; retry with backoff for a few minutes. (b) The vault is in access-policy mode, ignoring RBAC; set --enable-rbac-authorization true. (A third: a cached token predating the grant — restart the process.)
  5. You hit the role-assignment ceiling (~4000 per subscription, historically lower). Design around it by assigning roles to an Entra group of identities (one assignment for many) and segmenting across subscriptions (each has its own ceiling); also prune orphaned assignments. Resource-scope the sensitive grants and group only the low-sensitivity ones.

Glossary

Next steps

You can now design managed identity as a decoupled, federated, least-privilege resource and debug it across all three planes. Build outward:

Entra IDManaged IdentityAzure RBACFederated Identity CredentialsWorkload Identity FederationIMDSAKS Workload IdentityBicep
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments

Keep Reading