VM Extension Provisioning Failed: Debugging Custom Script, DSC and Azure Monitor Agent Errors

Your Terraform plan applies cleanly, the VM boots, and then the deployment hangs for forty minutes before failing with one of the most unhelpful strings in Azure: “VM has reported a failure when processing extension ‘CustomScript’. Error message: ‘Enable failed: …’”. The portal shows the VM as Running, the extension as Provisioning failed, your pipeline as red — and nothing tells you whether your script had a bug, the Azure VM Guest Agent never came up, the handler timed out pulling a file, or the VM has no outbound route to fetch what it needs. A VM extension is a small agent-driven payload — install software (CustomScript), apply configuration (DSC), or stream telemetry (AzureMonitorWindowsAgent / AzureMonitorLinuxAgent) — that Azure injects into a running VM through the guest agent. When it fails, the abstraction that made az vm extension set a one-liner becomes an opaque wall.

This is the diagnostic playbook for that wall. We treat “extension provisioning failed” not as one bug but as a layered failure: the Azure fabric asks the guest agent (waagent on Linux, WindowsAzureGuestAgent on Windows) to run a handler, the handler runs your payload, and any of those three layers can be the liar. You will learn to read the real signal — the handler status file, the per-extension log, the substatus and exit code returned to ARM — and localise a failure to exactly one layer: a dead guest agent, a handler that couldn’t download its package, a script that exited non-zero, a DSC configuration that threw mid-apply, or an Azure Monitor Agent that installed fine but has no managed identity or Data Collection Rule to talk to. Every diagnosis comes with the exact file to open or command to run, plus the fix in az CLI and Bicep.

By the end you will stop re-running the pipeline and hoping. You will know whether to look on the box (agent and handler logs at /var/log/azure/... or C:\WindowsAzure\Logs\...), in the resource’s instanceView, or in your own script’s stderr — and that an extension stuck at Creating/Transitioning for forty minutes is a different problem from one that returned Failed in ninety seconds. Knowing which layer within a few minutes turns a half-day stall into a five-minute fix.

What problem this solves

VM extensions are the seam between provisioning a VM and configuring a VM. Terraform, Bicep, ARM and az vm all lean on them: install the monitoring agent, run a bootstrap script, join a domain, apply DSC, onboard to Azure Arc. The seam is powerful because it runs inside the guest after boot — and fragile for the same reason. The control plane (ARM) only sees a provisioning state and a terse status bubbled up from the guest; the truth lives on the VM, in agent logs the control plane never shows you. The information you need is real and captured, but it is on the box, and if you don’t know which log maps to which layer you burn an hour clicking Refresh.

What breaks without this knowledge: an engineer re-runs the deployment (which sometimes “works” because a transient download succeeded the second time, teaching the wrong lesson), deletes and re-creates the VM (expensive and still failing), or files a ticket and waits — while the real cause (a script that exits 1 on a curl to a blocked endpoint, a guest agent that’s Not Ready, a DSC node reboot-looping, or an AMA with no DCR association shipping zero data) sits there, perfectly diagnosable, ignored. Worse, a failed extension can block the whole VM deployment — leaving the VM in a Failed state that downstream resources depend on, or hanging terraform apply.

Who hits this: anyone bootstrapping VMs as code. It bites hardest on Custom Script Extension (your script’s bugs plus network/download issues), DSC (long applies, reboots, the WMF/PowerShell version maze), and the Azure Monitor Agent migration (the agent installs cleanly but silently sends nothing because the identity or DCR is wrong). The fix is almost never “re-run it” — it’s “find the layer that’s lying and read its status file.”

To frame the whole field before the deep dive, here is every failure class this article covers, the layer it lives in, and the one place to look first:

Learning objectives

By the end of this article you can:

• Explain the guest agent → handler → payload chain and name which layer owns each failure mode.
• Tell a stuck extension (no terminal status, agent/handler problem) apart from a failed one (terminal Failed with a substatus and exit code) — and confirm which with instanceView.
• Read the on-box signal: the handler status file, the per-extension log directory, and the exit code/substatus that ARM surfaces — for both Linux (/var/log/azure/...) and Windows (C:\WindowsAzure\Logs\...).
• Diagnose Custom Script Extension failures as a script bug (non-zero exit), a download failure (storage auth, no managed identity, blocked egress), a timeout, or a protected-settings mistake — and confirm each.
• Diagnose DSC failures as a compilation error, a module/WMF mismatch, a reboot loop, or a node never reaching Compliant — and read Get-DscConfigurationStatus.
• Diagnose Azure Monitor Agent problems as install failure, missing managed identity, no DCR association, or blocked egress to the workspace/ingestion endpoint — and verify data is actually flowing with KQL.
• Drive the core tools fluently: az vm extension, az vm get-instance-view, az vm run-command, the on-box agent logs, and the resource’s instanceView JSON.

Prerequisites & where this fits

You should already know how to create a VM and read JSON from az (Cloud Shell or a local CLI), and understand that an extension is a child resource of the VM (Microsoft.Compute/virtualMachines/extensions) that runs after the OS boots, injected through the VM Guest Agent — the service Azure pre-installs in marketplace images that talks to the fabric, downloads handler packages, runs them, and reports status. Familiarity with az vm run-command (a sibling mechanism that runs a command via the same agent, invaluable for diagnosis) helps. If VMs are new, start with Your First Azure Virtual Machine: A Step-by-Step Deployment in Portal, CLI and PowerShell.

This sits in the Compute & Troubleshooting track and pairs with two neighbours. When the failure turns out to be connectivity (the agent or handler can’t reach a fabric/storage/ingestion endpoint), cross into Cannot RDP or SSH into Your Azure VM? A Beginner’s Connectivity Troubleshooting Checklist and Diagnosing Connectivity with Network Watcher: Connection Monitor, Connection Troubleshoot and Next Hop. When it’s identity (Custom Script can’t read a blob, or AMA has no identity), cross into Managed Identities Demystified: System vs User-Assigned and When to Use Each. For the AMA half, the data side lives in Azure Monitor and Application Insights: Full-Stack Observability and No Logs Showing Up? Troubleshooting Empty Log Analytics Tables and Ingestion Gaps.

A quick map of who confirms what during an incident, so you open the right log fast:

Core concepts

Five mental models make every later diagnosis obvious.

An extension is a goal-state contract executed by an agent. You declare “this VM should have extension X with these settings”; ARM records that as the goal state; the guest agent polls the fabric, sees the goal, downloads the handler package, runs it, and reports a status back to ARM. The portal’s Provisioning succeeded/failed is the top of a chain whose real detail lives in the on-box agent and handler logs. Read only the portal and you read the summary of a summary.

“Stuck” and “failed” are different bugs. A failed extension returned a terminal status — ARM has a substatus and (for script-like handlers) an exit code; the payload ran and reported something. A stuck extension (sitting at Creating/Transitioning for tens of minutes) usually means a layer below your payload never finished: the guest agent is Not Ready, can’t reach the fabric, or the handler is mid-retry. The first fork in every diagnosis is “terminal status, or hung?” — they send you to different logs.

The guest agent is the single point all extensions go through. On Linux it is waagent (the walinuxagent service); on Windows it is WindowsAzureGuestAgent plus WaAppAgent. It must be installed, running, and Ready before any extension can succeed. A Not Ready agent fails or stalls every extension at once — the tell: if all extensions on a VM are stuck, suspect the agent, not your script. The agent needs outbound access to the Azure fabric (host 168.63.129.16) to fetch goal state and handler packages.

The handler captures your payload’s output and exit code — that’s your real log. For Custom Script and DSC, the handler runs your script/config and records stdout, stderr, and the exit code into a status file. Exit code 0 = success; non-zero = failed, and that code is what bubbles up to ARM. So the most important thing your bootstrap script can do is log loudly and return honest exit codes — a script that does real work but exit 0s on error makes the extension “succeed” while the VM is broken, the worst class because nothing alerts.

Some extensions install fine and still do nothing. The Azure Monitor Agent (AMA) is the canonical case: the extension provisions successfully (binary installed and running) yet ships zero data, because it has no managed identity to authenticate, no Data Collection Rule (DCR) associated to tell it what to collect and where, or the ingestion endpoint is unreachable. “Provisioning succeeded” for AMA means “installed,” not “flowing.” You confirm flowing with a KQL query, never the provisioning state.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

The status and exit-code reference

Before the per-handler anatomy, here is the lookup table you scan first: the states and codes you realistically see, what each means on this platform, the likely cause, how to confirm it, and the first fix. The non-obvious ones are exit code 51 (CSE “not supported”/skipped), the difference between a provisioning state and the handler status, and that a successful provisioning state can still mean a broken outcome.

Three reading notes that save the most time:

Anatomy: the guest agent and the request pipeline

Every extension failure is, ultimately, a failure somewhere on this path. The request a deployment makes: (1) you declare the extension → ARM stores the goal state; (2) the guest agent polls the fabric over 168.63.129.16, sees the goal, and downloads the handler package; (3) the handler runs its enable operation, executing your payload; (4) the handler writes a status file, the agent reports status to ARM, and ARM flips the extension to Succeeded or Failed. Any hop can break, and the symptom tells you which:

Confirm the guest agent first — it gates everything

Before touching your script, prove the agent is healthy. The instanceView carries the agent’s own status:

# The single most useful command: full runtime status of the VM, agent, and every extension
az vm get-instance-view --name vm-app01 --resource-group rg-app \
  --query "{provisioningState:provisioningState, agent:vmAgent.statuses[].displayStatus, ext:instanceView.extensions}" -o json

The vmAgent.statuses block should show Ready. If it shows Not Ready (or the agent version is conspicuously old), every extension on this VM is suspect for the same reason. On the box:

# Linux — is walinuxagent running, and is it talking to the fabric?
systemctl status walinuxagent      # or 'waagent' on some distros
sudo tail -n 100 /var/log/waagent.log
# Can the VM reach the Azure fabric wire server at all?
curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://168.63.129.16/  # any response = reachable

# Windows — are the two agent services running, and is the fabric reachable?
Get-Service WindowsAzureGuestAgent, RdAgent | Format-Table Name, Status
Test-NetConnection -ComputerName 168.63.129.16 -Port 80
Get-Content 'C:\WindowsAzure\Logs\WaAppAgent.log' -Tail 100

The agent’s behaviour and where it logs, by OS — know your row before you go log-hunting:

The on-box layout you actually open

When the agent is Ready and one extension still failed, the truth is in two places: the handler log (what it did) and the status file (what it reported). The status file is JSON and mirrors what ARM saw:

# Linux — read the most recent handler status file (it's the source of the ARM message)
sudo cat /var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-*/status/*.status | python3 -m json.tool

That JSON has a status (“success”/“error”/“transitioning”), a code, and a formattedMessage — the same string the portal shows, but untruncated. When in doubt, the status file is more honest than the portal.

Anatomy of a Custom Script Extension failure

The Custom Script Extension (CSE) downloads file(s) and runs a command on the VM — the workhorse for bootstrap. Its failures cluster into five causes. Scan the matrix, then read the detail for whichever row matches:

Cause 1 — Your script exited non-zero

The most common CSE failure is the honest one: your script ran and returned a non-zero exit code, and the handler bubbled it up as the failure. The substatus often includes the tail of your stderr. Read the substatus from ARM, then the full stdout/stderr on the box:

# What ARM saw (the substatus carries your script's error text, often truncated)
az vm get-instance-view -n vm-app01 -g rg-app \
  --query "instanceView.extensions[?contains(name,'CustomScript')].substatuses[].message" -o tsv

# On the Linux box — the handler captures your script's real output here
sudo cat /var/lib/waagent/custom-script/download/0/stdout
sudo cat /var/lib/waagent/custom-script/download/0/stderr
# (older/newer layouts: /var/log/azure/custom-script/handler.log and the status dir above)

# On the Windows box
Get-Content 'C:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension\*\CustomScriptHandler.log' -Tail 200

Fix. Fix the throwing line. The deeper discipline: make the script idempotent (CSE may run more than once — on re-deploy, on a model change), log loudly, and set -euo pipefail (bash) so a failing command actually fails the script instead of silently continuing. The single biggest own-goal is a script that does apt-get install ... || true and then exit 0, hiding the real failure.

Cause 2 & 3 — The script (or its inputs) failed to download

CSE fetches the script from a storage blob (fileUris) before running it. If the blob is private and the VM can’t authenticate — or has no outbound route to storage — the download fails before your script ever runs. People stare at the script logic when the script never executed. The handler/agent log shows the attempt and HTTP code: a 403 means auth (private blob, missing identity, expired SAS); a timeout/refused means egress/routing.

sudo grep -iE "download|fileUris|403|denied|timed out" /var/log/waagent.log /var/log/azure/custom-script/*.log

Fix — auth. The secret-free way: give the VM a managed identity, grant it Storage Blob Data Reader on the blob, and reference the identity in the extension — no SAS tokens:

# 1) Give the VM a system-assigned identity and grant it read on the storage account
az vm identity assign -n vm-app01 -g rg-app
PRINCIPAL=$(az vm show -n vm-app01 -g rg-app --query identity.principalId -o tsv)
SA_ID=$(az storage account show -n stbootstrap -g rg-app --query id -o tsv)
az role assignment create --assignee "$PRINCIPAL" \
  --role "Storage Blob Data Reader" --scope "$SA_ID"

# 2) CSE pulls the blob using that identity (no SAS)
az vm extension set -n CustomScript --publisher Microsoft.Azure.Extensions \
  --vm-name vm-app01 -g rg-app \
  --settings '{"fileUris":["https://stbootstrap.blob.core.windows.net/scripts/bootstrap.sh"]}' \
  --protected-settings '{"managedIdentity":{},"commandToExecute":"bash bootstrap.sh"}'

resource cse 'Microsoft.Compute/virtualMachines/extensions@2024-07-01' = {
  parent: vm
  name: 'CustomScript'
  location: location
  properties: {
    publisher: 'Microsoft.Azure.Extensions'
    type: 'CustomScript'
    typeHandlerVersion: '2.1'
    autoUpgradeMinorVersion: true
    settings: {
      fileUris: [ 'https://stbootstrap.blob.core.windows.net/scripts/bootstrap.sh' ]
    }
    protectedSettings: {
      // Use the VM's managed identity to read the private blob — no SAS token in config
      managedIdentity: {}
      commandToExecute: 'bash bootstrap.sh'
    }
  }
}

If you understand why a managed identity is the right answer here (and when a user-assigned one is better for many VMs sharing the same bootstrap storage), Managed Identities Demystified: System vs User-Assigned and When to Use Each is the companion read.

Fix — egress. If it’s a connectivity failure, the VM has no route to storage or the agent’s CDN — an NSG/UDR/firewall problem, diagnosed like any other VM egress issue (Diagnosing Connectivity with Network Watcher: Connection Monitor, Connection Troubleshoot and Next Hop). Prefer a private blob + managed identity (403 if the role is missing, timeout if the route is); a SAS token works but is a credential to scope and expire; a public blob dodges auth but is world-readable (avoid); an inline commandToExecute with no download sidesteps the issue entirely.

Cause 4 — The script ran too long and timed out

A bootstrap that compiles software, pulls large images, or waits on a slow dependency can exceed the handler’s execution budget — the extension fails with a timeout even though the script was “working.” The handler log shows a long gap between the script’s start line and a “timed out” message; your own logging stops mid-way.

Fix. Don’t “raise the timeout” — make the bootstrap fast and the long work asynchronous: kick the slow part into the background (nohup/systemd unit/scheduled task), let the script return quickly, and have the background job report its own health. Keep the script idempotent so a retry doesn’t double-install. For genuinely long first-boot setup, bake it into a custom image rather than running CSE every time.

Cause 5 — Protected settings and secrets

CSE has two settings blocks: settings (plaintext, visible in the resource) and protectedSettings (encrypted, never returned). The commandToExecute and anything secret (SAS tokens, passwords) belong in protectedSettings. Putting a secret in settings leaks it; malformed JSON in either block rejects the extension.

Confirm. The status message complains about settings/JSON, or a secret you expected is empty/echoed. (Protected settings are write-only — you can’t read them back, which is the point.)

Fix. Keep the schema correct: non-secret config (fileUris, flags, anything you’d commit to git) in settings; everything sensitive (commandToExecute, SAS tokens, storageAccountKey, passwords, the managedIdentity block) in protectedSettings; both valid JSON. The CSE knobs you actually set, with their defaults and gotchas:

A practical re-run note: CSE only re-runs when its configuration changes. If you fixed the script in storage but the extension config is byte-identical, the extension won’t re-execute. Bump the timestamp integer (or change any setting) to force it.

Anatomy of a DSC extension failure

The PowerShell DSC extension (Microsoft.Powershell.DSC, Windows) applies a Desired State Configuration to the VM and enforces it. DSC failures are richer (and slower) than CSE because DSC compiles, applies, and can reboot and resume. Five causes:

Confirm: read the DSC configuration status, not just the extension

DSC keeps its own rich status separate from the extension’s. On the box, this is the most informative single command:

Get-DscConfigurationStatus | Format-List *                                  # per-resource pass/fail + why
Get-DscConfigurationStatus -All | Select StartDate, Status, RebootRequested  # history: applies + reboots

Status = Failure with a ResourcesNotInDesiredState list tells you which resource threw — far more useful than the extension’s one-line “configuration failed.” The handler log carries the rest:

Get-Content 'C:\WindowsAzure\Logs\Plugins\Microsoft.Powershell.DSC\*\*.log' -Tail 200

Causes 1–3 — Compile, modules, and WMF

DSC must compile your configuration to a MOF and have every DSC resource module present at the right version. Classic failures: a configuration that compiles on your laptop but not on the VM (different module versions), or a module the config Import-DscResources that was never packaged with it. DSC on Azure VMs targets WMF 5.1; resources written for a newer PowerShell can misbehave. The handler log shows the compile error or “could not find module X”; $PSVersionTable on the box shows the version.

Fix. Package modules with the configuration (supply a .zip containing the config and its modules) and pin module versions so the VM uses exactly what you tested. Compile locally first to catch MOF errors before they cost a 30-minute apply:

# Apply a DSC configuration packaged (config + modules) into a zip in blob storage
az vm extension set -n DSC --publisher Microsoft.Powershell.DSC \
  --vm-name vm-web01 -g rg-app \
  --settings '{
    "configuration": { "url": "https://stbootstrap.blob.core.windows.net/dsc/IIS.zip",
                       "script": "IIS.ps1", "function": "IISInstall" },
    "configurationArguments": { "WebsiteName": "shop" }
  }' \
  --protected-settings '{"configurationUrlSasToken":"?<sas>"}'

resource dsc 'Microsoft.Compute/virtualMachines/extensions@2024-07-01' = {
  parent: vm
  name: 'DSC'
  location: location
  properties: {
    publisher: 'Microsoft.Powershell.DSC'
    type: 'DSC'
    typeHandlerVersion: '2.83'      // pin the handler; do not float on a config you didn't test
    autoUpgradeMinorVersion: false
    settings: {
      configuration: {
        url: 'https://stbootstrap.blob.core.windows.net/dsc/IIS.zip'
        script: 'IIS.ps1'
        function: 'IISInstall'
      }
      configurationArguments: { WebsiteName: 'shop' }
    }
    protectedSettings: {
      configurationUrlSasToken: '?${dscSasToken}'   // secret → protected block
    }
  }
}

Causes 4–5 — A resource throws, or the node reboot-loops

A single DSC resource throwing (a path that doesn’t exist yet, a feature needing a reboot first, a service not installed) fails the whole apply. The nastier variant is a reboot loop: the config requests a reboot (RebootNodeIfNeeded), the VM reboots, DSC re-applies, finds it still not in desired state, reboots again — forever; the extension never reaches terminal success. Get-DscConfigurationStatus -All shows repeated applies with RebootRequested = True.

Fix. Make the configuration converge: ensure each resource’s preconditions are met in order (use DependsOn), and that a reboot actually resolves the “not in desired state” condition (a config that always reports drift will always reboot). The DSC settings that govern this:

DSC is powerful but heavy; for new bootstrap work, many teams now prefer a CSE that runs a small idempotent script, or bake state into a custom image — DSC shines when you need ongoing enforcement, not one-time setup.

Anatomy of an Azure Monitor Agent (AMA) failure

The Azure Monitor Agent (AzureMonitorWindowsAgent / AzureMonitorLinuxAgent) replaces the retired Log Analytics agent (MMA/OMS). Its failures are special: the extension usually provisions successfully yet no data arrives, because AMA is config-driven by Data Collection Rules and authenticates with a managed identity. “Succeeded” is necessary but nowhere near sufficient. Four causes:

Cause 1 — The install itself failed

Less common, but real: the AMA extension fails to install (unsupported distro/version, a missing dependency, an OS too old) — a genuine Failed provisioning state with an install error in the substatus. Read the substatus and the AMA install log:

az vm get-instance-view -n vm-app01 -g rg-app \
  --query "instanceView.extensions[?contains(name,'AzureMonitor')].substatuses[].message" -o tsv
# Linux AMA logs:
sudo cat /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/*.log | tail -100

Fix. Check the agent’s supported OS matrix and prerequisites; use a supported image and a current handler version.

Cause 2 — No managed identity (AMA can’t authenticate)

AMA does not use a workspace key (that was the old MMA model) — it authenticates with the VM’s managed identity to reach the ingestion endpoint. No identity → no authenticated channel → no data, even though the agent is installed and running.

az vm identity show -n vm-app01 -g rg-app -o json   # empty/null = no identity = AMA can't auth

Fix. Assign an identity (system-assigned is simplest for one VM; user-assigned is cleaner at fleet scale):

az vm identity assign -n vm-app01 -g rg-app

resource vm 'Microsoft.Compute/virtualMachines@2024-07-01' = {
  name: 'vm-app01'
  location: location
  identity: { type: 'SystemAssigned' }   // AMA needs an identity to authenticate to the ingestion endpoint
  properties: { /* ... */ }
}

Cause 3 — No Data Collection Rule associated (the #1 “no data” cause)

The canonical AMA trap. The agent installs, has an identity, is perfectly healthy — and collects nothing, because a Data Collection Rule is what tells AMA what to collect (perf counters, syslog, Windows events) and where to send it. An agent with no DCR association is a radio with no station tuned in. Unlike the old agent, AMA has no implicit “send everything to this workspace” default.

Confirm. List the DCR associations for the VM — an empty list is the smoking gun:

VM_ID=$(az vm show -n vm-app01 -g rg-app --query id -o tsv)
az monitor data-collection rule association list --resource "$VM_ID" -o table
# No rows = no DCR = AMA collects nothing, regardless of "Provisioning succeeded"

In the workspace, the Heartbeat table is the fastest proof of life — if AMA were configured and flowing, every agent sends a heartbeat:

Heartbeat
| where TimeGenerated > ago(30m)
| where Computer == "vm-app01"
| summarize LastSeen = max(TimeGenerated) by Computer, Category

No Heartbeat rows for the VM in the last 30 minutes ≈ the agent isn’t authenticated and configured to that workspace.

Fix. Create (or reuse) a DCR and associate it with the VM:

# Associate an existing DCR with the VM (the step everyone forgets)
az monitor data-collection rule association create \
  --name "vm-app01-dcr-assoc" \
  --resource "$VM_ID" \
  --rule-id "/subscriptions/<sub>/resourceGroups/rg-obs/providers/Microsoft.Insights/dataCollectionRules/dcr-linux-perf-syslog"

resource dcrAssoc 'Microsoft.Insights/dataCollectionRuleAssociations@2022-06-01' = {
  name: 'vm-app01-dcr-assoc'
  scope: vm
  properties: {
    dataCollectionRuleId: dcr.id   // without this, AMA installs but ships nothing
  }
}

The AMA “installed but no data” checklist — work it top to bottom:

Cause 4 — Egress to the workspace/ingestion endpoint is blocked

Identity present, DCR associated, agent healthy — and still no data. Now it’s network: AMA can’t reach the ingestion endpoint (or, in a Private Link setup, the Data Collection Endpoint / AMPLS) because an NSG, UDR, or firewall blocks the egress.

Confirm. The AMA log shows connection/timeout errors to the ingestion host; from the box, test outbound to the relevant endpoints (and 168.63.129.16 for agent health). The deep version of this diagnosis is the ingestion-gap playbook: No Logs Showing Up? Troubleshooting Empty Log Analytics Tables and Ingestion Gaps.

Fix. Open the required egress, or wire a Data Collection Endpoint + Private Link for locked-down VNets. The pattern across all four AMA causes is the same ladder — install → identity → DCR association → egress — and you climb it in order, stopping at the first rung that fails. For how AMA, DCRs and workspaces fit into observability, Azure Monitor and Application Insights: Full-Stack Observability is the upstream read.

Architecture at a glance

Hold this mental model and every diagnosis becomes a question of which box on the path failed. Picture the flow left to right. Your deployment tool (Bicep, Terraform, az vm extension set) writes a goal state into ARM: “VM-app01 should have these extensions.” ARM hands the goal to the Azure fabric. Inside the VM — the pivot of the whole picture — sits the VM Guest Agent (waagent on Linux, WindowsAzureGuestAgent on Windows), the one component every extension passes through. It polls the fabric over 168.63.129.16, notices the goal, and downloads the handler package for each extension from an Azure CDN/storage endpoint.

From the agent, the path fans out into one handler per extension, and this is where the three families diverge. The Custom Script handler downloads your files (authenticating to private blob storage with the VM’s managed identity) and runs your command, capturing stdout, stderr and an exit code. The DSC handler compiles your configuration to a MOF, applies it resource-by-resource, and may reboot and resume. The Azure Monitor Agent handler installs the agent binary, which reaches back out — using the VM’s managed identity — to the ingestion endpoint, collecting only what an associated Data Collection Rule tells it to, landing data in a Log Analytics workspace.

Now overlay the failure map. A stuck-at-Creating symptom lives at the agent hop (agent Not Ready, or no route to 168.63.129.16) — nothing downstream started. A download 403 lives at the handler-fetch or blob-auth hop. A non-zero exit lives in your payload box. And the AMA “succeeded but no data” symptom is unique: the install box is green, but the identity, DCR-association, or egress box further along is red. Every path reports status back through the agent to ARM, which is why the agent log (waagent.log / WaAppAgent.log) is the spine: if the agent is sick, every box downstream is suspect; if it’s Ready, the failure is exactly one handler or payload, and the per-extension log tells you which. The first question on every incident — “terminal status, or hung below my payload?” — is just asking which half of this diagram you’re in.

Real-world scenario

Northwind Logistics runs a fleet of Linux web VMs (Ubuntu 22.04, Standard_D2s_v5) in Central India, provisioned by Terraform. Each VM gets two extensions on first boot: a Custom Script Extension that pulls bootstrap.sh from a private storage account and installs the app, and the Azure Monitor Linux Agent for telemetry. The platform team is three engineers; a typical terraform apply rolls five VMs.

The incident began during a routine scale-out. terraform apply ran for forty minutes then failed five VMs at once with “…processing extension ‘CustomScript’. Error message: ‘Enable failed: failed to download files…’”. The on-call engineer’s first reflex: re-run terraform apply. Identical failure. Second reflex: re-create one VM by hand — same failure. An hour in, the scale-out was blocked and a deploy window was closing.

The breakthrough came from the right first question: did the script ever run, or fail before that? The error said “failed to download files,” so the script never executed — not a script bug. The engineer SSH’d into one VM (it was Running; only the extension failed) and ran sudo grep -iE "download|403|denied" /var/log/waagent.log, which showed an HTTP 403 fetching bootstrap.sh. The script was fine; the VM couldn’t authorise the download. The cause: that morning, stbootstrap had been moved behind selected networks with public access tightened, and a Terraform module refactor had dropped the user-assigned managed identity the CSE relied on to read the blob. The managedIdentity reference resolved to no identity, the blob refused anonymous access, and the download 403’d before the script ran. Every VM failed identically — the tell that it was infrastructure, not code.

A second, quieter problem lurked. Even on VMs where the download was manually fixed, the Azure Monitor Agent showed Provisioning succeeded but dashboards were blank — the team had assumed “agent installed = telemetry flowing.” az monitor data-collection rule association list --resource <vmId> returned zero rows: the refactor had also dropped the DCR association. The agent was a radio with no station tuned in.

The fix landed in two parts. Immediately: restore the managed identity and grant it Storage Blob Data Reader on stbootstrap, clearing the 403 — terraform apply went green. Within the day: re-add the dataCollectionRuleAssociation so every VM associates the standard dcr-linux-perf-syslog rule on creation, and Heartbeat rows appeared in minutes. They also hardened the script with set -euo pipefail and idempotent installs. The next scale-out rolled five VMs in eight minutes with telemetry flowing. The lesson on the wall: “‘Provisioning failed’ and ‘provisioning succeeded’ are both lies until you read the layer underneath — the agent log for the failure, the Heartbeat query for the success.”

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

The agent-driven extension model both causes this class of problem and makes it diagnosable. Weigh it honestly:

The model is right when you want VMs that arrive configured and monitored without hand-touching them. It bites hardest on private-storage bootstraps (download/identity), DSC-heavy estates (apply/reboot complexity), and AMA migrations (the silent no-data trap). The disadvantages are all manageable — but only if you know the agent log is the spine and that “succeeded” needs a second proof for data-shipping extensions.

Hands-on lab

Reproduce a Custom Script Extension failure on purpose, read it the way you would in an incident, then fix it — all on the cheapest VM (delete at the end). Run in Cloud Shell (Bash).

Step 1 — Variables and resource group.

RG=rg-ext-lab
LOC=centralindia
VM=vm-ext-lab
az group create -n $RG -l $LOC -o table

Step 2 — Create a small Linux VM.

az vm create -n $VM -g $RG --image Ubuntu2204 --size Standard_B1s \
  --admin-username azureuser --generate-ssh-keys -o table

Expected: a JSON/table row with provisioningState: Succeeded and a public IP.

Step 3 — Run a Custom Script Extension that deliberately fails (exit 1). This mirrors the most common CSE failure — a script that returns non-zero:

az vm extension set -n CustomScript --publisher Microsoft.Azure.Extensions \
  --vm-name $VM -g $RG \
  --protected-settings '{"commandToExecute":"echo starting; ls /no/such/path; exit 1"}'

Expected: the command fails with “VM has reported a failure when processing extension ‘CustomScript’.”

Step 4 — Read it the way you would in an incident. First what ARM saw:

az vm get-instance-view -n $VM -g $RG \
  --query "instanceView.extensions[?contains(name,'CustomScript')].substatuses[].message" -o tsv

The substatus carries your script’s stderr (the ls error) and the exit code. Then read the real thing on the box via run-command (no SSH needed):

az vm run-command invoke -n $VM -g $RG --command-id RunShellScript \
  --scripts "tail -n 50 /var/log/waagent.log; echo '--- stdout ---'; cat /var/lib/waagent/custom-script/download/0/stdout 2>/dev/null; echo '--- stderr ---'; cat /var/lib/waagent/custom-script/download/0/stderr 2>/dev/null"

You’ll see the script’s starting echo, the ls error in stderr, and exit code 1 — proof the script ran and failed (this is a payload bug, not a download/agent bug).

Step 5 — Fix it by replacing the command with one that succeeds. Because the config changed, CSE re-runs:

az vm extension set -n CustomScript --publisher Microsoft.Azure.Extensions \
  --vm-name $VM -g $RG \
  --protected-settings '{"commandToExecute":"echo bootstrap-ok; uname -a; exit 0"}'

Expected: the command succeeds; provisioningState for the extension is now Succeeded.

Step 6 — Confirm the green state and prove the difference between “ran” and “succeeded.”

az vm get-instance-view -n $VM -g $RG \
  --query "instanceView.extensions[].{name:name, state:statuses[0].displayStatus}" -o table

Expected: CustomScript → Provisioning succeeded.

Validation checklist. You reproduced a CSE failure from a non-zero exit, read both the ARM substatus and the on-box stdout/stderr via run-command, and fixed it by changing the command (which is what re-triggers CSE). The steps map to what each proves:

Cleanup (avoid lingering charges).

az group delete -n $RG --yes --no-wait

Cost note. A Standard_B1s VM is a few rupees per hour; an hour of this lab is well under ₹50, and deleting the resource group stops everything.

Common mistakes & troubleshooting

This is the playbook — the part you bookmark. First as a scannable table you can read mid-incident, then the same entries with the full confirm-command detail underneath. It spans the basic failures (a script exits non-zero, a re-run that won’t trigger) and the advanced ones (a dead guest agent, a DSC reboot loop, an AMA with no DCR).

The expanded form, with the full reasoning for the entries that bite hardest:

1. “Enable failed” with an exit code and your script’s stderr. Root cause: Your script ran and returned a non-zero exit code; the handler faithfully reported it as the extension failure. Confirm: az vm get-instance-view -n <vm> -g <rg> --query "instanceView.extensions[?contains(name,'CustomScript')].substatuses[].message"; on Linux read /var/lib/waagent/custom-script/download/0/stdout and stderr. Fix: Fix the failing line. Add set -euo pipefail (bash) so a mid-script failure actually fails the script; log loudly; return an honest exit code.

2. “failed to download files” — the script never executed. Root cause: CSE couldn’t fetch the file from storage — either a 403 (private blob, missing identity, expired SAS) or no outbound route to storage. Confirm: sudo grep -iE "download|403|denied|timed out" /var/log/waagent.log. A 403 = auth; a timeout/refused = egress. Fix: For auth, give the VM a managed identity + Storage Blob Data Reader and reference it (protectedSettings.managedIdentity), or use a valid, narrowly-scoped SAS. For egress, fix the NSG/UDR/firewall — see Diagnosing Connectivity with Network Watcher: Connection Monitor, Connection Troubleshoot and Next Hop.

3. CSE 403 specifically on a private blob, identity-related. Root cause: The VM has no managed identity, the wrong one, or the identity lacks the data-plane role (Reader on the resource is not enough — blob data needs Storage Blob Data Reader). Confirm: az vm identity show -n <vm> -g <rg> (empty = none); az role assignment list --assignee <principalId> --scope <storageId>. Fix: Assign the identity, grant Storage Blob Data Reader at the account/container scope, and reference it in the extension’s protected settings. Background: Managed Identities Demystified: System vs User-Assigned and When to Use Each.

4. Extension stuck at Creating/Transitioning for 20+ minutes. Root cause: A layer below your payload never finished — the guest agent is Not Ready or can’t reach the fabric host 168.63.129.16 to fetch goal state/handler. Confirm: az vm get-instance-view ... --query "vmAgent.statuses[].displayStatus" (Not Ready), and from the box Test-NetConnection 168.63.129.16 -Port 80 / curl -m 5 http://168.63.129.16/. Fix: Restart the agent (systemctl restart walinuxagent / Restart-Service WindowsAzureGuestAgent); restore outbound to 168.63.129.16 and the agent CDN; check for clock skew (a badly wrong clock breaks the agent’s TLS to the fabric).

5. Every extension on the VM fails or stalls together. Root cause: It’s the guest agent, not any one handler — a single shared dependency failing all of them. Confirm: vmAgent.statuses shows Not Ready; systemctl status walinuxagent (or the Windows services) shows it stopped/crashed; the image may simply lack the agent. Fix: Restart or reinstall the agent; verify the image includes it; check time sync. Once the agent is Ready, re-trigger the extensions.

The remaining entries are tighter — symptom, the one-line root cause, and the move:

• 6. Fixed the script in storage but CSE won’t re-run. It re-executes only when its config changes, not the referenced file. Bump settings.timestamp (or change any setting) to force a fresh run.
• 7. A Windows-authored script fails oddly on Linux. CRLF line endings — the shell sees \r and reports command not found on valid lines (cat -A shows ^M$). Save as LF, or set settings.skipDos2Unix: false.
• 8. DSC fails early with “configuration failed.” It didn’t compile to MOF, or an imported module is missing/wrong version (the DSC handler log and Get-DscConfigurationStatus name it). Compile locally first; package + pin modules in the config zip.
• 9. DSC reboot-loops, never terminal. The config never converges — it reboots, re-applies, still finds drift (Get-DscConfigurationStatus -All shows repeated RebootRequested=True). Fix the always-drifting resource; gate RebootNodeIfNeeded or set ActionAfterReboot=StopConfiguration to debug.
• 10. AMA “succeeded” but no data. No DCR associated — the healthy agent collects nothing (...association list = 0 rows; Heartbeat empty). Associate a DCR, then re-check Heartbeat.
• 11. AMA installed + DCR set, still no data. No managed identity, or the agent can’t reach the ingestion endpoint (DCE/AMPLS in Private Link). Assign identity; open egress — No Logs Showing Up? Troubleshooting Empty Log Analytics Tables and Ingestion Gaps.
• 12. “Another operation is in progress.” A concurrent VM/extension op. Serialise — one op at a time; sequence dependent extensions with dependsOn / depends_on.
• 13. CSE exit code 51. The handler refused the operation — usually an unsupported OS/distro. Use a supported image and current handler version.
• 14. “Succeeded” but the app is broken. The script swallowed an error (... || true) and exit 0’d — green status, broken VM, nothing alerts (the most dangerous class). Remove blanket || true; fail on real errors.
• 15. Fine on create, fails on later re-deploy. The script isn’t idempotent — it trips over its own prior run (“already exists”). Make every step check-then-act.

Best practices

• Read the agent first, then the handler, then your payload. az vm get-instance-view → vmAgent.statuses is the spine. If the agent is Not Ready, stop looking at your script. If it’s Ready, the failure is one handler — read that log.
• Write idempotent bootstrap scripts. CSE can run more than once (re-deploy, model change). Every step should be safe to repeat; use check-then-act, not blind create.
• Fail loudly, exit honestly. set -euo pipefail in bash; never || true over a real failure; log a clear success/failure line. A green provisioning state must mean the work actually succeeded.
• Use managed identity for downloads, not SAS in config. Give the VM an identity + Storage Blob Data Reader and reference it in protectedSettings.managedIdentity. No secrets in the resource, auditable access, nothing to expire.
• Put secrets in protectedSettings, never settings. commandToExecute, SAS tokens, passwords belong in the encrypted block; settings is world-readable in the resource.
• Re-trigger CSE by bumping timestamp. Changing the referenced file doesn’t re-run the extension; changing the config does. Use the timestamp integer as the clean re-run lever.
• For AMA, prove data flow with Heartbeat, not the provisioning state. “Succeeded” means installed. Check identity → DCR association → Heartbeat rows. No DCR = no data, every time.
• Package and pin DSC modules with the configuration. Don’t rely on what’s on the box; ship the modules in the config zip and pin versions so applies are deterministic.
• Keep long work out of the extension. Background slow bootstrap, or bake it into a custom image. An extension that takes 40 minutes is fragile; one that returns in seconds and hands off to a service is robust.
• Serialise dependent extensions. Sequence them with dependsOn (Bicep) / explicit depends_on (Terraform) so two operations don’t collide into “another operation in progress.”
• Pin handler versions for production-critical extensions. autoUpgradeMinorVersion: false plus a tested typeHandlerVersion keeps a surprise handler update from breaking a config you didn’t change.
• Use run-command as your break-glass diagnostic. It runs through the same agent and reaches the on-box logs without opening SSH/RDP — perfect for reading waagent.log during an incident.

Security notes

• Managed identity over secrets for every download. Prefer the VM’s system- or user-assigned managed identity with Storage Blob Data Reader over SAS tokens or account keys in the extension. Grant least privilege — data-plane Reader on the specific container, not Owner on the account.
• Never put secrets in settings. The settings block is returned in plaintext by az vm extension show and stored in the resource; anything sensitive (SAS, passwords, connection strings, commandToExecute with secret args) must go in protectedSettings, which is write-only and encrypted.
• Treat SAS tokens as credentials. If you must use a SAS, scope it to read-only on the single blob, give it a short expiry, and rotate it. A broad, long-lived SAS in an extension definition is a standing leak.
• Lock down the bootstrap storage account. Restrict the storage account to selected networks / Private Endpoint, and reach it from the VM via the managed identity — don’t make the bootstrap blob public to “fix” a 403.
• Pin and scan what the extension pulls. For container or package pulls inside a script, pin digests/versions and scan images; an extension that installs “latest” from an open URL is a supply-chain risk.
• Restrict who can write extensions. An attacker with rights to add a CustomScript extension to a VM gets arbitrary code execution as root/SYSTEM through the agent. Control Microsoft.Compute/virtualMachines/extensions/write via RBAC and audit it — it is effectively remote-root.
• Keep diagnostics off the public surface. Don’t echo secrets into stdout/stderr (the handler captures them into status files and the substatus message). Log identifiers, not credentials.

The security knobs that also prevent these incidents — secure and reliable pull in the same direction:

Cost & sizing

Extensions themselves are free — you don’t pay for CustomScript, DSC, or the agent extension as a line item. The cost shows up indirectly, in time and the resources the extension touches:

• Compute time during long extensions. You pay for the VM while it runs; a 40-minute DSC apply blocks the deploy and bills the whole time — a reason to keep extensions fast or move heavy setup into a custom image built once.
• AMA’s real cost is ingestion, not the agent. AMA is free; the Log Analytics ingestion and retention is the bill. A too-broad DCR (every perf counter at 15s, full verbose syslog) can quietly cost more than the VM — right-size it to what you query: How to Create a Log Analytics Workspace: Region, Access Modes and Resource-Context RBAC.
• Storage egress is negligible per script, but pulling multi-GB artifacts on every VM at scale adds up; a same-region account + Private Endpoint keeps it cheap.
• The hidden cost is engineer time — a misdiagnosed extension failure costs hours; reading the right log turns a half-day stall into minutes, by far the biggest lever here.

A rough picture for a small fleet: extensions add ₹0 directly; AMA-driven Log Analytics ingestion is the variable cost (often ₹1,000s/month by DCR breadth and VM count); compute is briefly held during long applies. The cost drivers and what each buys:

Interview & exam questions

1. A VM extension is stuck at “Creating” for 30 minutes with no error. What’s the most likely layer, and how do you confirm? A layer below the payload — most likely the guest agent is Not Ready or can’t reach the Azure fabric at 168.63.129.16 to fetch goal state/the handler. Confirm with az vm get-instance-view → vmAgent.statuses (Not Ready) and, on the box, Test-NetConnection 168.63.129.16 -Port 80. A stuck (non-terminal) state is an agent/egress problem; a Failed state is a payload/handler problem.

2. Custom Script Extension fails with “failed to download files.” Did your script run? What do you check? No — the failure is before execution; the handler couldn’t fetch the file. Check waagent.log for the download line and HTTP code: a 403 means auth (private blob, missing managed identity, expired SAS), a timeout means no egress to storage. Fix the identity/role or the NSG/UDR accordingly.

3. What’s the difference between an extension’s provisioning state and its handler status? The provisioning state (Succeeded/Failed) says the handler ran and returned; the handler status/substatus (and your own log) says whether the work was correct. A script can exit 0 on a swallowed error → provisioning “Succeeded” while the app is broken. Always read the substatus, not just the green tick.

4. The Azure Monitor Agent shows “Provisioning succeeded” but no data appears in Log Analytics. Most common cause? No Data Collection Rule is associated with the VM. AMA is config-driven and has no implicit default — without a DCR it collects nothing despite being installed and healthy. Confirm with az monitor data-collection rule association list --resource <vmId> (0 rows) and an empty Heartbeat query; fix by associating a DCR.

5. AMA has a DCR associated and the VM has an identity, but still no data. What’s left? Network egress to the ingestion endpoint (or the Data Collection Endpoint / AMPLS in a Private Link design) is blocked by an NSG/UDR/firewall. The AMA log shows connection/timeout errors to the ingestion host. Fix the egress or wire a DCE + Private Link.

6. Why does AMA need a managed identity when the old Log Analytics agent used a workspace key? AMA authenticates to the ingestion endpoint with the VM’s managed identity (Entra ID), not a shared workspace key — more secure and centrally revocable. No identity means no authenticated channel and therefore no data, even though the agent is installed.

7. You fixed the script in blob storage, but re-running the deployment doesn’t re-run the Custom Script Extension. Why? CSE re-executes only when its configuration changes, not when the referenced file changes. Bump the settings.timestamp integer (or change any setting) to force a fresh run. This catches many teams who update the blob and expect the extension to notice.

8. A DSC extension applies, reboots, and re-applies endlessly. What’s happening and how do you break it? A reboot loop: the configuration requests a reboot but never converges (a resource always reports “not in desired state”). Confirm with Get-DscConfigurationStatus -All showing repeated RebootRequested=True. Break it by fixing the always-drifting resource, and temporarily disabling RebootNodeIfNeeded or setting ActionAfterReboot=StopConfiguration to investigate.

9. Where do you read the real output of a Custom Script Extension on a Linux VM? In the handler’s capture of your script’s stdout/stderr (e.g. /var/lib/waagent/custom-script/download/<n>/stdout and stderr) and the status file under the handler’s status/ directory; the agent log (/var/log/waagent.log) covers download/agent issues. You can reach all of these via az vm run-command without SSH.

10. What does exit code 0 vs non-zero mean to a Custom Script Extension, and why does it matter for reliability? 0 = success, non-zero = failure, and that code is what bubbles to ARM as the extension result. It matters because a script that does real work but returns 0 on error makes the extension “succeed” with a broken VM — the worst failure mode since nothing alerts. Honest exit codes (set -euo pipefail, no blanket || true) are the foundation of reliable bootstrapping.

11. Why can adding a Custom Script Extension be a security-sensitive action? Because the extension runs through the guest agent as root/SYSTEM — anyone with Microsoft.Compute/virtualMachines/extensions/write on a VM effectively has remote code execution on it. The permission should be tightly RBAC-scoped and audited; it’s as powerful as shell access.

12. An entire VM’s extensions all fail at once. Is it more likely your script or the platform? The platform/agent — when every extension fails or stalls together, suspect the shared dependency: the guest agent is Not Ready (stopped, crashed, missing from the image, or clock-skewed) or there’s no route to the fabric. A single script bug fails one extension while the agent stays Ready.

These map to AZ-104 (Administrator) — deploy and manage Azure compute resources, VM extensions, and monitor resources with the Azure Monitor Agent and DCRs — and AZ-204 (Developer) where bootstrap/config-as-code and managed identities appear. The networking-egress angle (fabric, ingestion endpoints, Private Link) touches AZ-700, and the identity/RBAC angle touches AZ-500. A compact cert-mapping for revision:

Quick check

1. An extension sits at “Creating” for 25 minutes with no error message. Is this more likely a script bug or a guest-agent/egress problem, and what’s the one command you run first?
2. Custom Script Extension fails with “failed to download files.” Did your script execute, and what’s the first log line you grep for?
3. True or false: changing the bootstrap.sh file in blob storage will cause the Custom Script Extension to re-run on the next deployment.
4. The Azure Monitor Agent reports “Provisioning succeeded” but Log Analytics shows no data. Name the single most common cause and the command that confirms it.
5. A bootstrap script does its work and returns exit 0, but the application is broken. What failure class is this, and why is it the most dangerous?

Answers

1. A guest-agent/egress problem — a stuck (non-terminal) state means a layer below your payload never finished. Run az vm get-instance-view -n <vm> -g <rg> --query "vmAgent.statuses[].displayStatus"; if it’s Not Ready, the agent (or its route to 168.63.129.16) is the issue, not your script.
2. No — the download failed before the script ran. Grep /var/log/waagent.log for download/403/denied: a 403 is auth (managed identity / SAS), a timeout is egress.
3. False. CSE re-runs only when its configuration changes, not when the referenced file changes. Bump settings.timestamp (or change any setting) to force a re-run.
4. No Data Collection Rule is associated with the VM (AMA has no implicit default, so it collects nothing). Confirm with az monitor data-collection rule association list --resource <vmId> returning 0 rows (and an empty Heartbeat query).
5. A silent success-on-failure (the script swallowed an error and exit 0’d). It’s the most dangerous because the provisioning state is green and nothing alerts, so a broken VM looks healthy. Fix with set -euo pipefail, no blanket || true, and honest exit codes.

Glossary

• VM extension — a small agent-run payload (install software, apply configuration, stream telemetry) attached as a child resource of an Azure VM and executed inside the guest after boot.
• Guest agent — the service Azure pre-installs in the VM (waagent/walinuxagent on Linux, WindowsAzureGuestAgent + RdAgent on Windows) that polls goal state, downloads handlers, runs them, and reports status to ARM.
• Handler — the per-extension program the agent downloads and runs; it executes your payload and writes the status/exit code that bubbles up to ARM.
• Goal state — the declared “what should be installed” for a VM, recorded by ARM and polled by the guest agent over the fabric.
• instanceView — the runtime status view of a VM and its extensions (az vm get-instance-view), carrying the agent status, per-extension display status, and substatus messages.
• Substatus / status message — the terse reason the handler reports to ARM; the same text the portal shows (often truncated) and the first thing you read.
• Exit code — a script/handler return value; 0 = success, non-zero = failure, and the non-zero code is the failure ARM surfaces.
• 168.63.129.16 — the special Azure “wire server” host the guest agent uses for goal state and health; if the VM can’t reach it, extensions stall.
• Custom Script Extension (CSE) — Microsoft.Azure.Extensions/CustomScript (Linux) / Microsoft.Compute/CustomScriptExtension (Windows); downloads files and runs a command on the VM.
• protectedSettings — the encrypted, write-only extension settings block for secrets (commandToExecute, SAS tokens, the managedIdentity reference); contrast with the plaintext settings.
• timestamp (CSE) — an integer in settings; changing it is the clean way to force CSE to re-run when the referenced file changed but the config didn’t.
• DSC extension — Microsoft.Powershell.DSC; compiles a PowerShell Desired State Configuration to MOF and enforces it on the VM, with reboot/resume semantics.
• Get-DscConfigurationStatus — the on-box cmdlet that reports per-resource DSC results and apply history (including reboot requests) — more detailed than the extension status.
• Azure Monitor Agent (AMA) — AzureMonitorWindowsAgent/AzureMonitorLinuxAgent; the current monitoring agent, config-driven by Data Collection Rules and authenticated by managed identity.
• Data Collection Rule (DCR) — Microsoft.Insights/dataCollectionRules; defines what AMA collects and where it sends it. Without a DCR association, AMA ships nothing.
• DCR association — the link binding a DCR to a VM (dataCollectionRuleAssociations); the step that’s most often forgotten, causing “installed but no data.”
• Managed identity — the VM’s Entra ID identity (system- or user-assigned) used to authenticate downloads (Storage Blob Data Reader) and AMA’s data shipping — replacing keys/SAS.
• run-command — az vm run-command, a way to run a command via the same guest agent without SSH/RDP; invaluable for reading on-box logs during an incident.
• Reboot loop (DSC) — a configuration that reboots, re-applies, finds drift, and reboots again without ever converging — the extension never reaches terminal success.

Next steps

You can now localise any extension failure to the agent, the handler, or your payload and fix it. Build outward:

• Next: Cannot RDP or SSH into Your Azure VM? A Beginner’s Connectivity Troubleshooting Checklist — when the extension failure is really a connectivity problem, this is the box-level checklist.
• Related: Managed Identities Demystified: System vs User-Assigned and When to Use Each — get the identity right so CSE downloads and AMA never 403 or sit silent.
• Related: No Logs Showing Up? Troubleshooting Empty Log Analytics Tables and Ingestion Gaps — the deep playbook for AMA’s “installed but no data” half.
• Related: Diagnosing Connectivity with Network Watcher: Connection Monitor, Connection Troubleshoot and Next Hop — confirm the VM’s egress to storage, the fabric, and ingestion endpoints.
• Related: Azure Monitor and Application Insights: Full-Stack Observability — how AMA, DCRs and workspaces fit into the bigger observability picture.