The Secrets Store CSI Driver lets a pod mount Key Vault secrets, keys, and certificates as files on a tmpfs volume, with no secret material written to etcd by default. On AKS it ships as a first-party add-on (azure-keyvault-secrets-provider): a managed DaemonSet, an Azure provider plugin, and platform-owned lifecycle rather than a Helm chart you babysit.
The interesting engineering is not the mount; it is the identity backing the mount and the rotation semantics once secrets change underneath running pods. This walkthrough wires the add-on to Microsoft Entra Workload ID (federated, no client secrets), authors a SecretProviderClass, syncs the mounted objects into a native Kubernetes Secret for env-var consumption, and turns on auto-rotation - with a clear-eyed view of what actually propagates and what does not.
Throughout, I prefer workload identity over the add-on’s auto-created managed identity. The add-on identity is a node-scoped, cluster-wide credential; workload identity scopes access to a single service account - what you want for least privilege and clean auditing.
1. Enable the add-on, and choose your identity model
The add-on installs the Secrets Store CSI Driver plus the Azure provider. On an existing cluster:
export RESOURCE_GROUP=rg-platform
export CLUSTER_NAME=aks-platform
az aks enable-addons \
--addons azure-keyvault-secrets-provider \
--name $CLUSTER_NAME \
--resource-group $RESOURCE_GROUP
Enabling the add-on always creates a user-assigned managed identity named azurekeyvaultsecretsprovider-<cluster> in the node resource group (MC_...) and assigns it to the node VMSS. You cannot prevent its creation, but you do not have to use it. Two ways to authenticate to Key Vault:
| Model | Credential scope | When to use |
|---|---|---|
| Add-on managed identity | Node-level, shared by every pod on the cluster | Quick demos, single-tenant clusters where all workloads share secrets |
| Workload ID (recommended) | Per-Kubernetes-service-account | Multi-team clusters, least privilege, per-app Key Vault RBAC |
Workload ID requires the OIDC issuer and the workload-identity webhook. If you are creating the cluster fresh, include both flags:
az aks create \
--name $CLUSTER_NAME \
--resource-group $RESOURCE_GROUP \
--enable-addons azure-keyvault-secrets-provider \
--enable-oidc-issuer \
--enable-workload-identity \
--generate-ssh-keys
On an existing cluster, enable them in place (idempotent, triggers a control-plane reconcile):
az aks update \
--name $CLUSTER_NAME \
--resource-group $RESOURCE_GROUP \
--enable-oidc-issuer \
--enable-workload-identity
Confirm the add-on landed and grab the auto-created identity details (useful even if you do not use it, for inventory):
az aks show \
--name $CLUSTER_NAME \
--resource-group $RESOURCE_GROUP \
--query addonProfiles.azureKeyvaultSecretsProvider
{
"config": { "enableSecretRotation": "false", "rotationPollInterval": "2m" },
"enabled": true,
"identity": {
"clientId": "00001111-aaaa-2222-bbbb-3333cccc4444",
"objectId": "aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb",
"resourceId": ".../userAssignedIdentities/azurekeyvaultsecretsprovider-aksplatform"
}
}
The driver and provider run as DaemonSets in kube-system:
kubectl get pods -n kube-system \
-l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)' -o wide
You should see aks-secrets-store-csi-driver-* (3/3 containers) and aks-secrets-store-provider-azure-* (1/1), one of each per node.
2. Federate a managed identity to a Kubernetes service account
The heart of the passwordless model: create a user-assigned identity, grant it data-plane RBAC on the vault, then bind it to a service account via a federated credential. No client secret is ever issued.
export UAMI=id-app-secrets
export KEYVAULT_NAME=kv-platform-prod
export SA_NAME=app-sa
export SA_NAMESPACE=payments
# 1. Create the workload identity
az identity create --name $UAMI --resource-group $RESOURCE_GROUP
export USER_ASSIGNED_CLIENT_ID=$(az identity show \
--resource-group $RESOURCE_GROUP --name $UAMI --query clientId -o tsv)
export IDENTITY_TENANT=$(az aks show \
--name $CLUSTER_NAME --resource-group $RESOURCE_GROUP \
--query identity.tenantId -o tsv)
Grant data-plane access. With an RBAC-enabled vault, the role depends on the object type you mount - this trips people up constantly, because it does not follow the intuitive crypto/secret split:
Object type in SecretProviderClass |
Required built-in role |
|---|---|
secret |
Key Vault Secrets User |
key |
Key Vault Certificate User |
cert |
Key Vault Certificate User |
Per the AKS docs, both key and cert object types require Key Vault Certificate User (the provider retrieves them through the certificate path, not a crypto operation). Because a Key Vault certificate is internally backed by both a key and a secret, mounting a full cert chain via objectType: secret against a certificate needs Key Vault Secrets User. Assign only what you mount:
export KEYVAULT_SCOPE=$(az keyvault show --name $KEYVAULT_NAME --query id -o tsv)
az role assignment create \
--role "Key Vault Secrets User" \
--assignee $USER_ASSIGNED_CLIENT_ID \
--scope $KEYVAULT_SCOPE
Now the federation. Get the cluster’s OIDC issuer URL, create the service account annotated with the identity’s client ID, then bind the federated credential to the system:serviceaccount:<ns>:<name> subject:
export AKS_OIDC_ISSUER=$(az aks show \
--resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
--query oidcIssuerProfile.issuerUrl -o tsv)
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: ${SA_NAME}
namespace: ${SA_NAMESPACE}
annotations:
azure.workload.identity/client-id: ${USER_ASSIGNED_CLIENT_ID}
EOF
az identity federated-credential create \
--name fic-app-secrets \
--identity-name $UAMI \
--resource-group $RESOURCE_GROUP \
--issuer ${AKS_OIDC_ISSUER} \
--subject system:serviceaccount:${SA_NAMESPACE}:${SA_NAME} \
--audience api://AzureADTokenExchange
The --subject must match the namespace and service account name exactly. A typo here produces the single most common failure mode - AADSTS70021: No matching federated identity record found - at pod startup, not at apply time.
3. Author the SecretProviderClass
The SecretProviderClass (SPC) is a namespaced CRD that tells the Azure provider which vault to hit, how to authenticate, and which objects to fetch. For workload identity, the two load-bearing settings are usePodIdentity: "false" and clientID set to the workload identity’s client ID - that combination is what signals the provider to use the projected service account token.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: app-kv
namespace: payments
spec:
provider: azure
parameters:
usePodIdentity: "false"
clientID: "00001111-aaaa-2222-bbbb-3333cccc4444" # workload identity clientId
keyvaultName: "kv-platform-prod"
cloudName: "" # defaults to AzurePublicCloud
tenantId: "ffffffff-1111-2222-3333-444444444444"
objects: |
array:
- |
objectName: db-connection-string
objectType: secret
objectVersion: "" # empty = latest
- |
objectName: signing-key
objectType: key
objectVersion: ""
- |
objectName: tls-app
objectType: secret # full PEM chain + private key
objectAlias: tls-app.pem
A few things worth internalizing:
objectsis a YAML string containing another YAML document. The|block scalar and the- |per-item delimiters are required; the provider parses this nested document itself.objectVersion: ""resolves to the latest version. Leave it empty unless you must freeze a version - pinning defeats rotation.objectAliascontrols the filename on the mount. Without it, the file is named afterobjectName. This matters in the next step, because sync keys off the mounted filename.
For certificates, recall the Key Vault object model: objectType: key returns the public key (PEM), objectType: cert returns the certificate only (PEM, no chain), and objectType: secret returns the private key plus certificate as a full PEM chain. Ingress controllers want the last one.
4. Mount as a volume, and the inline-volume startup requirement
The driver only fetches secrets when a pod mounts a CSI volume referencing the SPC - no pod, no secret, by design. The sharp consequence: the volume must mount successfully for the pod to start. If the identity is misconfigured or the object does not exist, the pod stays in ContainerCreating and you read the reason from events.
apiVersion: v1
kind: Pod
metadata:
name: payments-api
namespace: payments
labels:
azure.workload.identity/use: "true" # required: opt the pod into the webhook
spec:
serviceAccountName: app-sa # must match the federated subject
containers:
- name: api
image: ghcr.io/acme/payments-api:1.8.2
volumeMounts:
- name: kv
mountPath: /mnt/secrets-store
readOnly: true
volumes:
- name: kv
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "app-kv"
Two non-negotiables here: the pod label azure.workload.identity/use: "true" (this makes the webhook inject the projected token and the AZURE_* env vars) and serviceAccountName matching the federated subject from step 2. Miss the label and the token is never projected; the provider fails with an auth error.
The secrets land as files:
kubectl exec -n payments payments-api -- ls /mnt/secrets-store/
# db-connection-string signing-key tls-app.pem
5. Sync mounted objects into a native Kubernetes Secret
Files on a volume suit apps that read from disk, but most apps want env vars, and env vars come from a Kubernetes Secret. The driver mirrors mounted content into a real Secret via the secretObjects block. Add it to the same SPC:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: app-kv
namespace: payments
spec:
provider: azure
secretObjects:
- secretName: app-db
type: Opaque
data:
- objectName: db-connection-string # must match the mounted FILENAME
key: DB_CONNECTION_STRING # key inside the K8s Secret
parameters:
# ... unchanged from step 3 ...
The critical rule: objectName under secretObjects.data must match the mounted filename, which is objectAlias if you set one, otherwise objectName. People wire this to the Key Vault object name and get an empty Secret. Now consume it as an env var:
env:
- name: DB_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: app-db
key: DB_CONNECTION_STRING
Two lifecycle facts that are easy to miss and cause incidents:
- The synced
Secretonly exists after at least one pod mounts the volume. It is created on first mount, not on SPC apply. Anything that reads the Secret before that first pod (a Helm pre-install hook, another Deployment) gets a not-found. - The synced
Secretis garbage-collected when the last consuming pod is deleted. The driver owns its lifecycle. Do not point unrelated workloads at it expecting it to persist independently.
6. Enable auto-rotation, tune the poll interval, and understand propagation
Rotation is off by default. It is an add-on-level setting, not an SPC field. Enable it and optionally widen the poll interval:
az aks addon update \
--resource-group $RESOURCE_GROUP \
--name $CLUSTER_NAME \
--addon azure-keyvault-secrets-provider \
--enable-secret-rotation \
--rotation-poll-interval 5m
The default poll interval is 2 minutes. The driver polls every interval, and on change updates both the mounted file content and the synced Kubernetes Secret. Now the part nobody reads carefully - what actually reaches the application:
| Consumption pattern | Picks up rotation automatically? | What you must do |
|---|---|---|
| App reads the mounted file | Yes, on next poll | App must re-read the file (watch for change or re-open per request) |
| App reads the synced Secret as a volume | Yes, on next poll | Mounted Secret volume contents update in place |
| App reads the synced Secret as an env var | No | Env vars are injected once at container start; restart the pod |
That last row is the trap. Environment variables are a point-in-time snapshot taken at container start. Rotating the secret updates the Kubernetes Secret, but a running container’s env block never changes. To close the loop, run something like Reloader, which watches synced Secrets and triggers a rolling restart:
metadata:
annotations:
reloader.stakater.com/auto: "true"
There is also a known Kubernetes limitation orthogonal to all of this: a Secret or ConfigMap mounted via subPath does not receive updates - that is a kubelet behavior, not a driver bug. Mount the whole volume, not a subPath, if you want in-place updates.
Set realistic expectations on lag. Worst-case time from a Key Vault write to a file update is roughly one poll interval plus the kubelet’s atomic-write sync (usually under a minute). With env vars and Reloader, add the rollout time. Do not assume sub-second rotation; design for “new value is live within a poll interval, connections re-established on next use.”
7. TLS certificate consumption for ingress
A common goal is terminating TLS at an ingress controller with a cert that lives in Key Vault and rotates automatically. The pattern: mount objectType: secret against the certificate name (yielding the full PEM chain plus private key), sync it into a kubernetes.io/tls Secret, and point the Ingress at that Secret.
spec:
provider: azure
secretObjects:
- secretName: app-tls
type: kubernetes.io/tls
data:
- objectName: tls-app.pem # the mounted filename (objectAlias above)
key: tls.crt
- objectName: tls-app.pem
key: tls.key
parameters:
objects: |
array:
- |
objectName: app-cert
objectType: secret # full chain + key
objectAlias: tls-app.pem
A
kubernetes.io/tlsSecret requires bothtls.crtandtls.key. The provider splits the PEM bundle when you map both keys to the same mounted object, becauseobjectType: secretagainst a certificate returns the concatenated private key and cert chain. Keep a pod mounting the SPC alive so the synced TLS Secret is never garbage-collected out from under the ingress controller - this is why teams run a tinypause-style keeper pod alongside the controller.
Verify
End to end, in order:
# 1. Files are present on the mount
kubectl exec -n payments payments-api -- ls /mnt/secrets-store/
# 2. The synced K8s Secret exists and has the expected keys
kubectl get secret app-db -n payments -o jsonpath='{.data}' | jq 'keys'
# 3. Rotation is actually enabled at the add-on level
az aks show -n $CLUSTER_NAME -g $RESOURCE_GROUP \
--query addonProfiles.azureKeyvaultSecretsProvider.config
# 4. Prove rotation: write a new value, wait one poll interval, re-read the file
az keyvault secret set --vault-name $KEYVAULT_NAME \
--name db-connection-string --value "rotated-$(date +%s)"
sleep 320 # poll interval + sync slack
kubectl exec -n payments payments-api -- cat /mnt/secrets-store/db-connection-string
To watch rotation reconciles directly, the driver and provider export Prometheus metrics on localhost (not exposed off-pod by default):
kubectl port-forward -n kube-system ds/aks-secrets-store-csi-driver 8095:8095 &
curl -s localhost:8095/metrics | grep -E 'total_rotation_reconcile|total_sync_k8s_secret'
Enterprise scenario
A payments platform team ran a 40-node AKS cluster shared by eight product squads. They had standardized early on the add-on’s auto-created managed identity, granting it Key Vault Secrets User on a single shared vault. It worked - and it became a finding in their PCI assessment. Because the credential was node-scoped, every pod on every node could read every secret in the vault. The blast radius of one compromised pod was the entire secret store, and the audit log could not attribute a secret get to a workload, since all reads came from one identity.
The constraint: they could not split into per-team clusters (cost and operational load), nor take a maintenance window long enough to re-platform. They needed per-squad isolation on the existing cluster, with auditable, attributable Key Vault access.
The fix was a migration to workload identity, one squad at a time, with no cluster downtime. They enabled the OIDC issuer and workload-identity webhook in place (az aks update --enable-oidc-issuer --enable-workload-identity - a control-plane reconcile; pods kept running). Each squad got its own user-assigned identity, its own vault, and a federated credential bound to that squad’s service account:
# Per squad: scope the identity to exactly its service account + its vault
az identity create --name id-squad-payments --resource-group rg-platform
CID=$(az identity show -g rg-platform -n id-squad-payments --query clientId -o tsv)
az role assignment create --role "Key Vault Secrets User" \
--assignee "$CID" \
--scope "$(az keyvault show -n kv-squad-payments --query id -o tsv)"
az identity federated-credential create \
--name fic-payments --identity-name id-squad-payments --resource-group rg-platform \
--issuer "$AKS_OIDC_ISSUER" \
--subject system:serviceaccount:payments:payments-sa \
--audience api://AzureADTokenExchange
The migration was reversible per workload: they kept the add-on identity’s role assignment until each squad’s pods rolled over to the new SPC (usePodIdentity: "false", clientID set to the squad’s identity) and verified mounts, then revoked the shared assignment last. Because rotation was already on, no app-side change was needed on the data path - only the identity backing the mount changed. The finding closed: Key Vault diagnostic logs now attributed every SecretGet to a named per-squad identity, and a compromised pod could reach exactly one squad’s secrets.