When Azure Backup Jobs Fail: Diagnosing VM Agent, Snapshot, and Extension Errors

The job ran at 02:00 like every night. This morning the Recovery Services vault shows a red Failed against one VM, error code UserErrorGuestAgentStatusUnavailable, and the backup admin has already restarted the VM twice “just in case.” The other forty VMs in the same vault succeeded. Nothing changed — except, as always, something did. This is the most common operational incident in Azure Backup, the managed service that takes point-in-time copies of your Azure VMs into a vault you don’t run. It is maddening because the error you see is reported by the backup orchestrator in the vault, not by the thing that failed: the vault is saying “I asked the VM to take a snapshot and didn’t get a good answer.” Why it didn’t is the whole game, and at least a dozen root causes hide behind a handful of error strings.

This is the diagnostic playbook for failed VM backup jobs. We treat the failures not as one bug but as a small set of symptom classes — the agent is unreachable, the VMSnapshot extension failed, VSS couldn’t quiesce the disks, the snapshot succeeded but the data transfer to the vault failed, or a restore won’t complete — each with a fan-out of causes you confirm with specific commands. You will learn to read the backup pipeline end to end — vault orchestrator → Azure VM agent → VMSnapshot extension → in-guest VSS / pre-post scripts → instant-recovery snapshot → background copy into the vault — and to localise a failure to exactly one hop. Every diagnosis comes with the precise path to confirm it (the Backup jobs blade, the job error detail, the in-guest agent and extension logs, az backup, az vm) and the exact fix, with az CLI and Bicep where it applies. Because this is a reference you reach for mid-incident, the playbook, the error codes and the agent/extension knobs are all laid out as scannable tables.

By the end you will stop guessing and stop blindly restarting VMs. When a job fails you will know within minutes whether you face a stopped or out-of-date VM agent, a VMSnapshot extension in a failed state, a VSS writer that timed out under disk pressure, an NSG or firewall blocking egress to Azure Storage, a vault at its instant-restore snapshot limit, or simply a VM that was deallocated at backup time. Knowing which — and confirming it rather than assuming — turns a recurring nightly red into a one-time fix.

What problem this solves

Azure Backup hides a lot of moving parts so you can tick “backup enabled” and move on — a gift until a job fails, when it becomes an opaque wall. The job-failure pane gives you an error code and a deliberately generic one-line message, while the real evidence is split across the vault’s job detail, the VM’s extension status, and log files inside the guest OS the vault cannot show you. Not knowing which place maps to which failure costs you an afternoon of reading the wrong logs.

What breaks without this knowledge: an operator retries the job (sometimes succeeding by accident as a transient clears, teaching the wrong lesson), restarts the VM (occasionally fixing a wedged extension, usually doing nothing), or — worst — assumes the backup is fine because most nights are green, and only discovers the gap during a restore, when the recovery point they need was never created. A silent backup failure is a disaster waiting for the day you need the data.

Who hits this: essentially everyone running Azure VM backup. It bites hardest on Linux VMs where the agent or Python dependencies drift, application servers where VSS struggles to quiesce a busy database, locked-down networks with no clean egress to Azure Storage, VMs deallocated on a schedule (dev boxes, spot VMs), and subscriptions where someone tightened RBAC and the operator quietly lost permission. The fix is almost never “restart the VM” — it is “find the hop that failed and make it tell you why.”

Every symptom class this article covers, the question it forces, and the first place to look:

Learning objectives

By the end of this article you can:

• Map any failed Azure VM backup job to a specific hop in the backup pipeline — agent, extension, VSS, snapshot, or data transfer — and name the most likely root cause for each.
• Diagnose an agent-unreachable failure as a deallocated VM, a stopped/outdated Azure VM agent, or a wireserver/IMDS reachability problem — and confirm which with exact commands.
• Diagnose a VMSnapshot extension failure as a failed provisioning state, a missing dependency (Python/handler on Linux), a disk-state or lock conflict, or an extension that needs a clean reinstall.
• Recognise an application-consistency (VSS) failure, decide when crash-consistent is acceptable, and fix VSS writer timeouts and pre/post-script errors.
• Diagnose a data-transfer failure as an NSG / firewall / UDR blocking egress to Azure Storage, a private-endpoint/DNS misconfiguration, or a transient condition.
• Break a restore failure by isolating it to a staging-account problem, a disk-SKU or region mismatch, target capacity/quota, or RBAC on the target.
• Drive the core diagnostic tools fluently: the Backup jobs blade and job error detail, az backup job, VM extension status, in-guest logs, and Backup Center — and read the error-code and retention-limit tables to pick the right fix fast.

Prerequisites & where this fits

You should understand the Azure Backup basics: a Recovery Services vault is the regional resource that stores recovery points and runs the backup policy (schedule + retention); a backup item is one protected resource (here, a VM); a backup job is a single run you watch succeed or fail. Know that VM backup is agent-based but agentless to install — it rides the Azure VM agent already on the VM and pushes a VMSnapshot extension the first time you protect it. Be comfortable running az in Cloud Shell, reading JSON, and connecting into a VM to read a log. New to backup? Start with Protect Your First Azure VM with Azure Backup: A Guided Walkthrough and come back when a job goes red.

This sits in the Observability & Troubleshooting track and assumes the protection fundamentals from Azure Backup and Site Recovery: Protecting Workloads from Loss and the resilience framing in BCDR Foundations on Azure: Making Sense of RTO, RPO, and the Resilience Spectrum. It pairs with Managed Identities Demystified: System vs User-Assigned and When to Use Each, since RBAC and identity sit behind several failure classes, and Azure Storage Redundancy Decoded: LRS vs ZRS vs GRS vs RA-GRS and How to Choose.

A quick map of who confirms what during a backup incident, so you read the right evidence first:

Core concepts

Five mental models make every later diagnosis obvious.

The error code names the orchestrator’s complaint, not the in-guest root cause. The job lives in the vault; the work happens inside the VM. UserErrorGuestAgentStatusUnavailable or a VMSnapshot failure tells you which step the vault couldn’t complete — not why the guest failed it; the “why” is in the VM’s extension status and the in-guest logs. Reading the vault error and stopping is the biggest time-waster: the error names the hop to inspect, the inspection names the cause.

VM backup rides the Azure VM agent and a snapshot extension. The Azure VM agent (guest agent) — WindowsAzureGuestAgent on Windows, walinuxagent on Linux — ships on Marketplace images and reports Ready/Not Ready. The first time you protect a VM, Azure Backup uses the agent to install the VMSnapshot extension (VMSnapshot / VMSnapshotLinux), and every later backup asks that extension to snapshot. So a backup needs two healthy things in the guest: an agent that is Ready, and a VMSnapshot extension whose provisioning state is Succeeded. If either is unhealthy, the job fails before it touches your disks.

Application-consistent is the goal; crash-consistent is the fallback. Backup aims for an application-consistent point — on Windows via VSS (Volume Shadow Copy Service), on Linux via pre-/post-snapshot scripts you provide. If those fail, it falls back to a crash-consistent snapshot (like pulling the power — disk intact, apps may need their own recovery). A job that succeeds with warnings usually means a crash-consistent point because app-consistency failed — a real signal, not noise.

The snapshot is fast; the transfer is where the network bites. Backup first takes an instant-recovery snapshot on the managed disks (default retention historically up to 5 days, configurable down to 1) — fast, and what powers near-instant restores — then a background copy moves the data into the vault. The phases fail differently: the snapshot phase on agent/extension/VSS problems; the transfer phase when the VM cannot reach Azure Storage (the data is read out via the guest), where NSGs, UDRs and firewalls cause grief unrelated to the snapshot.

Allocation and retention are finite. A vault item has an instant-restore snapshot retention window — keep too many on-demand snapshots and the next job fails until older ones age out. The VM must be running at backup time for an application-consistent result. And the operator needs the right RBAC. Run out of any of these — snapshot slots, allocation, permission — and you get a failure that looks like a bug but is a limit.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

The backup error-code reference

Before the per-symptom anatomy, here is the lookup table you scan first: the error codes you realistically see on a failed Azure VM backup job, what each means, how to confirm it, and the first fix. The non-obvious ones are the UserError* agent/extension codes (which all point into the guest) and the gap between a job that fails and one that completes with warnings (crash-consistent fallback).

Three reading notes that save the most time:

Anatomy of an agent-unreachable failure

Here the orchestrator never gets a usable answer from inside the VM: codes like UserErrorGuestAgentStatusUnavailable and GuestAgentSnapshotTaskStatusError. Four causes — scan the matrix, then read the matching detail:

Cause 1 — The VM was deallocated (or stopped) at backup time

The most common cause of all. The backup fires at 02:00 but the VM is Stopped (deallocated) — a dev box on a shutdown schedule, an evicted spot VM, or left off. With no running guest agent, the orchestrator gets no status and the job fails. Confirm the power state:

# Current power state of the VM
az vm get-instance-view --name vm-app01 --resource-group rg-app-prod \
  --query "instanceView.statuses[?starts_with(code,'PowerState')].displayStatus" -o tsv
# Expect: "VM running"  (anything else at backup time explains the failure)

The activity log corroborates with the deallocate/start events. Fix. Align the schedule to when the VM is reliably up, or change the shutdown schedule. If the VM is meant to be off and you only need disk integrity, a crash-consistent point is acceptable — just know you aren’t getting app-consistency on a powered-off VM.

Cause 2 — The VM agent is stopped or reporting Not Ready

The agent is installed but its service isn’t running or is wedged, so it reports Not Ready (or nothing) and the orchestrator can’t get snapshot status. Confirm:

az vm get-instance-view --name vm-app01 --resource-group rg-app-prod \
  --query "instanceView.vmAgent.statuses[].displayStatus" -o tsv
# "Ready" is healthy; "Not Ready" / empty points at the agent

VM → Properties → Agent status shows Ready / Not Ready. Fix. Restart the agent in-guest — Windows: Restart-Service WindowsAzureGuestAgent; Linux: systemctl restart walinuxagent. The logs that hold the truth differ by OS: on Windows, C:\WindowsAzure\Logs\WaAppAgent.log (extensions under ...\Logs\Plugins\...); on Linux, /var/log/waagent.log (extensions to /var/log/azure/, the VMSnapshotLinux handler in its own subdirectory).

Cause 3 — The VM agent is out of date or broken

An old or corrupted agent can fail to push or run the extension even while appearing partly functional — common where walinuxagent/Python dependencies drift, or on long-lived custom images. Confirm the version and look for handler errors in waagent.log / WaAppAgent.log. Fix. Update or reinstall the agent, then validate with an on-demand backup.

Cause 4 — The platform endpoint (wireserver / IMDS) is unreachable

The agent reaches the platform on the host channel — 168.63.129.16 (wireserver) and IMDS at 169.254.169.254. A too-aggressive NSG, a UDR forcing all traffic through an appliance, or an in-guest firewall that blocks these leaves the agent unable to function, reporting Not Ready. Confirm with the in-guest agent log (it records wireserver failures) plus the NSG/UDR; fix by restoring reachability — these addresses must not be blackholed or denied. If the VM is also unreachable for management, Cannot RDP or SSH to an Azure VM? A First-Principles Troubleshooting Path walks the same plumbing.

Anatomy of a snapshot / extension failure

Here the agent is Ready but the VMSnapshot extension can’t do its job — the orchestrator reached the guest and asked for a snapshot, but the snapshot step failed. Four causes:

Cause 1 — The VMSnapshot extension is in a failed state

The extension can land in a Failed provisioning state — corrupted state, an interrupted run, or an in-guest condition it can’t recover from — and once failed, every job fails until it’s healthy. Confirm:

# Extension provisioning state for VMSnapshot (name differs Win vs Linux)
az vm extension list --vm-name vm-app01 --resource-group rg-app-prod \
  --query "[?contains(name,'VMSnapshot')].{name:name, state:provisioningState}" -o table

Fix. Remove the extension and let the next backup reinstall it — Azure Backup re-pushes it automatically, so deleting the broken instance is safe. Then run an on-demand backup to re-provision and validate:

# Remove the failed VMSnapshot extension; the next backup re-installs it cleanly
az vm extension delete --vm-name vm-app01 --resource-group rg-app-prod \
  --name VMSnapshot   # use VMSnapshotLinux on Linux VMs

Cause 2 — A missing in-guest dependency (especially Linux)

VMSnapshotLinux relies on the agent and its Python runtime; if the distro’s Python is upgraded out from under the agent, a package is missing, or the handler can’t launch, the snapshot fails even though the agent reports Ready. Confirm in /var/log/azure/ (the VMSnapshotLinux handler directory). Fix. Repair the dependencies — update walinuxagent, ensure a supported Python, reinstall the extension. On custom or hardened images this is the usual culprit; validate the agent before mass-deploying.

Cause 3 — A disk-state conflict, and Cause 4 — a resource lock

If another operation is touching the disks — a resize, another snapshot, a disk attach/detach, or a previous backup that hasn’t released — the compute layer rejects the snapshot with UserErrorCrpReportedUserError or UserErrorVmNotInDesiredState. Confirm via the VM’s activity log (in-flight ops) and the Backup jobs blade (an overlapping run); fix by letting the other op finish, then retrying. Serialise on-demand runs so they don’t collide with the schedule — UserErrorBackupOperationInProgress is two jobs racing. Separately, a ReadOnly resource lock can block the staging/snapshot resources Backup creates (a CanNotDelete lock generally does not) — list locks with az lock list, then scope or remove the ReadOnly lock for the window. The extension-repair options compared:

Anatomy of an application-consistency (VSS) failure

This one hides in plain sight: the job often completes with warnings rather than failing outright, because Backup created a crash-consistent point after VSS (Windows) or the pre/post scripts (Linux) failed. The disk is intact, but a database may need its own crash recovery on restore — treat a warning as a real defect, not a green light. The causes:

Cause 1 — A VSS writer is in a bad state

Each application registers a VSS writer (SQL Server, AD, etc.). A writer in a non-Stable state blocks the application-consistent snapshot, and Backup falls back to crash-consistent. Confirm inside the Windows guest:

:: Elevated command prompt — every writer should read "State: [1] Stable", "Last error: No error"
vssadmin list writers

Fix. Restart the service behind the bad writer (for SQL, the SQL Server service), then re-run an on-demand backup. A persistently bad writer usually means the app is under stress at snapshot time.

Cause 2 — Disk pressure makes VSS time out

VSS briefly quiesces I/O for the shadow copy; on a VM hammering its disks at the backup hour the quiesce times out and app-consistency fails. Confirm by correlating the job time with disk IOPS / queue-depth metrics — failing only under load is the tell. Fix. Move the window to a quieter time and/or a higher-throughput disk SKU; Azure VM Disk Types Demystified: Standard HDD, Standard/Premium SSD, Premium v2 and Ultra Disks lays out the IOPS/throughput ceilings that govern how fast it flushes.

Cause 3 — Free space, and Cause 4 — Linux scripts

VSS needs room for the shadow copy’s differences; on a nearly-full volume it can’t allocate that space and the snapshot degrades — confirm free space on each protected volume and fix by freeing or growing it (a chronically full data disk is the usual offender). On Linux there is no VSS: application-consistency requires pre-/post-snapshot scripts (referenced from the agent config) that quiesce and unquiesce your app. Absent, wrong, or non-zero-exit scripts mean only crash-consistent points — provide correct, fast, idempotent scripts that exit 0, which is why “my Linux VM is never application-consistent.” The two models compared:

Anatomy of a data-transfer failure

Here the snapshot succeeded — a recovery point may even appear — but the background copy into the vault failed. Because the data is read out via the guest, this phase fails almost entirely on egress to Azure Storage. The tell: a job that snapshotted fine yet went red, or *NoNetwork*/DataTransferFailed codes. The causes:

Cause 1 — An NSG or firewall blocks egress to Azure Storage

The most common transfer failure. The snapshot data must reach Azure Storage, and a locked-down NSG or an in-path Azure Firewall/NVA without the right allow-rule blocks it: the snapshot succeeds (local), the transfer fails (UserErrorSnapshotFailedNoNetwork). Confirm the effective NSG rules on the VM’s NIC:

# Effective outbound rules on the VM's NIC — is Storage egress allowed?
az network nic list-effective-nsg \
  --name vm-app01VMNic --resource-group rg-app-prod -o table

Fix. Allow outbound to the Storage service tag, or open the firewall/NVA for it — the service tag is the clean way, with no hard-coded IPs:

resource allowStorageEgress 'Microsoft.Network/networkSecurityGroups/securityRules@2023-11-01' = {
  parent: nsg
  name: 'Allow-Backup-Storage-Egress'
  properties: {
    priority: 200
    direction: 'Outbound'
    access: 'Allow'
    protocol: 'Tcp'
    sourceAddressPrefix: 'VirtualNetwork'
    sourcePortRange: '*'
    destinationAddressPrefix: 'Storage'   // service tag; region-scoped form: Storage.<region>
    destinationPortRange: '443'
  }
}

Cause 2 — A UDR blackholes the traffic

A forced-tunnel UDR (0.0.0.0/0 → an on-prem appliance) sends Storage-bound traffic somewhere with no return path, and the copy hangs/fails. Confirm the effective routes:

az network nic show-effective-route-table \
  --name vm-app01VMNic --resource-group rg-app-prod -o table

Fix. Route Storage-bound traffic correctly — exempt the Storage service tag from the tunnel, or send it via an endpoint that reaches Storage; the route table is the source of truth. Cause 3 — private endpoint / DNS. With the vault or storage behind private endpoints, the VM must resolve those private names via the correct private DNS zone; a missing zone link means it resolves a public name it can’t reach and the transfer fails. Confirm by resolving the name from the VM (expect the private IP); fix the DNS zone link. Cause 4 — throttling / transient. Occasionally the transfer fails transiently and a plain retry succeeds — don’t over-engineer a one-off, but if it recurs across many VMs at the same minute, stagger schedules. What’s allowed to reach where, by topology:

Architecture at a glance

Hold the whole pipeline as a single left-to-right path and every error code tells you where to stand. On the far left is the Recovery Services vault — the regional brain. It owns the policy (when to run, how long to keep) and the job you see go green or red, but does no in-guest work; it orchestrates. When the schedule fires, the vault reaches into the VM through the Azure VM agent, which must be Ready before anything else happens. If the vault can’t get a status from that agent (VM deallocated, agent stopped, platform endpoint unreachable), you never leave this first hop — the agent-unreachable class.

One hop in, the agent hands off to the VMSnapshot extension inside the guest — the worker that freezes and captures the disks. It must be in a Succeeded provisioning state; if it’s wedged or missing a dependency, the request dies here (the extension-failure class). The extension then triggers application quiescing: on Windows it calls VSS so writers flush; on Linux it runs your pre/post scripts. Succeed and you get an application-consistent point; fail and Backup falls back to crash-consistent and the job warns — the consistency class, easy to miss because the job still looks “done.”

With the disks quiesced, the platform takes the instant-recovery snapshot on the managed disks — fast, local, the thing that powers near-instant restores — held for a finite retention window (pile up on-demand snapshots and you hit the cap). Finally, a background copy reads the snapshot out of the guest and ships it to the vault over the network to Azure Storage — the only phase that depends on the VM’s egress, and where NSGs, UDRs and firewalls bite. The mental shortcut: agent → extension → VSS/scripts → snapshot → transfer — read the error code, place it on that line, inspect that one hop, ignore the other four. Restores run the line in reverse (vault → recovery point → new disks/VM in a target resource group), which is why restore failures cluster around the target — capacity, SKU, region, RBAC — not the source guest.

Real-world scenario

Meghdoot Logistics runs 60 Azure VMs across two regions — Windows application servers, SQL-on-VM database hosts, and a dozen Linux API workers — backed up by two Recovery Services vaults on a nightly policy (app-consistent at 02:00, 30-day retention, 5-day instant-restore). For a year it was boring and green. Then a platform refresh changed two things at once: the network team rolled out a forced-tunnel UDR pushing all egress through a new Azure Firewall, and the database team migrated three SQL hosts onto larger data disks running much busier overnight ETL. Next morning the dashboard lit up with two different failure patterns, and the on-call instinct — “restart the VMs” — fixed nothing.

The first pattern hit all the Linux API workers and several app servers: UserErrorSnapshotFailedNoNetwork. The engineer almost opened a support ticket, then remembered the pipeline — a NoNetwork code means the snapshot succeeded but the transfer failed, so it’s egress, not the guest. az network nic show-effective-route-table on one worker showed a 0.0.0.0/0 route to the firewall with no rule permitting the Storage service tag: the new forced tunnel was blackholing the backup copy. The network team added a Storage egress allow-rule on the firewall (plus a Storage NSG rule for belt-and-braces), and the next on-demand run transferred cleanly — fleet-wide green within the hour.

The second pattern was subtler and only on the three migrated SQL hosts: the jobs succeeded — but with warnings, “completed with crash-consistency.” Because the dashboard looked green-ish, nobody had flagged it — the kind of thing that bites only at restore. Inside one host, vssadmin list writers showed the SQL Server VSS writer in a non-Stable state: the heavy ETL was pinning the disks at exactly 02:00 and VSS was timing out on the quiesce. The two-part fix moved that vault’s schedule to 23:30 (before the ETL) for the database hosts and bumped their data disks to a higher-throughput SSD tier so the quiesce flushes faster. After that, the writers read Stable at backup time and the jobs returned to application-consistent, no warning.

The lesson, written into their runbook: read the error class before you touch the VM. A NoNetwork code is a network ticket, not a VM restart; a warning is a real defect — a crash-consistent SQL backup is a landmine you only find at restore time. Two simultaneous changes produced two unrelated failure classes that “restart everything” could never have fixed. Total cost of the actual fixes: one firewall rule, one schedule change, one disk-tier bump — no support ticket, no new vault, a far more trustworthy nightly run.

Advantages and disadvantages

Knowing how Azure VM backup fails is inseparable from knowing what the agent-and-snapshot design buys you and what it costs. The trade-offs:

The agentless, policy-driven model scales beautifully across hundreds of VMs — enable a policy and walk away. The downside bites the moment the guest or network changes: the failure is reported in one place (the vault) and caused in another (the guest or network), making diagnosis harder than with a self-contained product — exactly why this playbook exists. Crash-consistent fallback is itself a feature and a trap: protective when VSS fails, but it lets a silently-degraded backup look healthy unless you watch for warnings.

Hands-on lab

This lab provisions a VM, protects it, deliberately breaks two failure classes, observes the errors, fixes them, and validates with an on-demand backup. It uses a B2s VM (~₹2–3/hour when running; deallocate or delete after) and Cloud Shell.

1. Create a resource group, vault, and VM.

az group create --name rg-bk-lab --location centralindia

az backup vault create --name rsv-bk-lab --resource-group rg-bk-lab \
  --location centralindia

az vm create --name vm-bk-lab --resource-group rg-bk-lab \
  --image Ubuntu2204 --size Standard_B2s \
  --admin-username azureuser --generate-ssh-keys

2. Enable backup with the default policy and run one on-demand backup.

az backup protection enable-for-vm \
  --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --vm vm-bk-lab --policy-name DefaultPolicy

# First on-demand run — this also pushes/validates the VMSnapshot extension
az backup protection backup-now \
  --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --container-name vm-bk-lab --item-name vm-bk-lab \
  --backup-management-type AzureIaasVM \
  --retain-until 05-07-2026

3. Watch the job and read its detail.

# List recent jobs and their status
az backup job list --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --query "[].{name:name, op:properties.operation, status:properties.status}" -o table

# Drill into the most recent job (substitute its name)
az backup job show --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --name <job-name> --query "properties.{status:status, error:extendedInfo.propertyBag}" -o json

Expected: Completed. Note the VMSnapshot extension now exists on the VM (az vm extension list --vm-name vm-bk-lab --resource-group rg-bk-lab -o table).

4. Break failure class A — agent unreachable (deallocate the VM). Stop and deallocate the VM, then trigger a backup:

az vm deallocate --name vm-bk-lab --resource-group rg-bk-lab
az backup protection backup-now \
  --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --container-name vm-bk-lab --item-name vm-bk-lab \
  --backup-management-type AzureIaasVM --retain-until 05-07-2026

Watch the job: a deallocated VM yields UserErrorGuestAgentStatusUnavailable / UserErrorVmNotInDesiredState — no running agent to talk to. Confirm:

az vm get-instance-view --name vm-bk-lab --resource-group rg-bk-lab \
  --query "instanceView.statuses[?starts_with(code,'PowerState')].displayStatus" -o tsv
# "VM deallocated" — there's your cause

5. Fix class A and revalidate. Start the VM, wait for the agent to report Ready, re-run:

az vm start --name vm-bk-lab --resource-group rg-bk-lab
az vm get-instance-view --name vm-bk-lab --resource-group rg-bk-lab \
  --query "instanceView.vmAgent.statuses[].displayStatus" -o tsv   # expect "Ready"
az backup protection backup-now --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --container-name vm-bk-lab --item-name vm-bk-lab \
  --backup-management-type AzureIaasVM --retain-until 05-07-2026

6. Break failure class B — data-transfer egress. Add an NSG rule that denies all outbound (simulating a locked-down network that blocks Storage egress), then back up:

NSG=$(az network nsg list -g rg-bk-lab --query "[0].name" -o tsv)
az network nsg rule create --resource-group rg-bk-lab --nsg-name "$NSG" \
  --name Deny-All-Out --priority 4000 --direction Outbound --access Deny \
  --protocol '*' --source-address-prefixes '*' --destination-address-prefixes '*' \
  --destination-port-ranges '*'
az backup protection backup-now --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --container-name vm-bk-lab --item-name vm-bk-lab \
  --backup-management-type AzureIaasVM --retain-until 05-07-2026

The job should fail in the transfer phase (a *NoNetwork*-class error). Confirm the block with effective rules: az network nic list-effective-nsg --name vm-bk-labVMNic --resource-group rg-bk-lab -o table.

7. Fix class B (allow Storage egress) and validate.

az network nsg rule create --resource-group rg-bk-lab --nsg-name "$NSG" \
  --name Allow-Storage-Out --priority 300 --direction Outbound --access Allow \
  --protocol Tcp --source-address-prefixes VirtualNetwork \
  --destination-address-prefixes Storage --destination-port-ranges 443
az backup protection backup-now --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --container-name vm-bk-lab --item-name vm-bk-lab \
  --backup-management-type AzureIaasVM --retain-until 05-07-2026   # expect Completed

8. Teardown. Disable protection and delete the resource group:

az backup protection disable --vault-name rsv-bk-lab --resource-group rg-bk-lab \
  --container-name vm-bk-lab --item-name vm-bk-lab \
  --backup-management-type AzureIaasVM --delete-backup-data true --yes
az group delete --name rg-bk-lab --yes --no-wait

You have now seen two distinct failure classes produce two distinct error families, confirmed each at the correct hop, and validated each fix — exactly the loop you run in production.

Common mistakes & troubleshooting

This is the centerpiece. Read the prose once; keep the table open at 02:00. It spans the everyday failures (deallocated VM, blocked egress) and the nastier ones (failed extension, VSS in a bad state, snapshot-retention cap), with the exact command or path to confirm and the real fix.

Per-symptom detail for the ones that trip people most:

Agent-unreachable (rows 1–2). Separate “is the VM even up?” from “is the agent healthy?” A deallocated VM is row 1; a running VM with a Not Ready agent is row 2 — az vm get-instance-view answers both (PowerState and vmAgent.statuses). Restarting the VM occasionally fixes row 2 by restarting the agent but tells you nothing; restart the agent service instead and you fix it and confirm the cause.

Extension failure (rows 3–5). The reliable repair for a wedged VMSnapshot extension is to delete it — Azure Backup re-pushes it on the next run, so this is safe, not destructive. On Linux, suspect the agent’s Python/dependencies first; a distro upgrade that moved Python is the classic cause and bites whole fleets built from one image.

Consistency warning (rows 6–8). Treat completed with warnings as a defect. A crash-consistent database backup will likely need crash recovery on restore — fine for a stateless web server, a problem for a transactional database. vssadmin list writers is your truth on Windows (every writer should read Stable / No error); on Linux, “always crash-consistent” almost always means missing or failing pre/post scripts.

Transfer failure (rows 9–11). A NoNetwork code is a network problem, not a VM one — the snapshot already succeeded. Go straight to effective NSG rules and the route table; don’t read in-guest application logs. The Storage service tag is the clean allow-target so you never hard-code IP ranges.

Best practices

• Monitor backup health, not just the dashboard colour. Wire Backup Center / Azure Monitor alerts on job failures and on “completed with warnings” — a crash-consistent fallback is a silent defect to catch before a restore. Treat that warning as a failure to investigate, especially on database VMs.
• Keep the VM agent current and Ready. Most agent-unreachable failures trace to a stopped, outdated, or dependency-broken agent. Validate the agent on any custom image before mass-deploying it.
• Repair a broken VMSnapshot extension by removing it. Backup re-installs it on the next run; don’t fight a wedged extension in place.
• Align the backup window with VM uptime and off peak I/O. A deallocated VM can’t be app-consistent; a hammered-disk VM fails VSS quiesce. Schedule for “up and quiet.”
• Allow egress to Storage explicitly when you lock down the network. Use the Storage service tag in NSGs and exempt it from forced tunnels — the data transfer depends on it.
• Author and test Linux pre/post scripts for any Linux VM that needs application-consistency; never assume Linux is app-consistent by default.
• Right-size instant-restore retention and on-demand frequency so you don’t hit the snapshot cap; let snapshots age out rather than piling on more.
• Validate with on-demand backups after any change (agent update, network change, extension repair) instead of waiting for the nightly run to tell you.
• Practise restores, not just backups. A backup you’ve never restored is a hypothesis. Periodically restore a recovery point to a scratch resource group to prove RTO and surface target-side issues early.
• Manage policy, RBAC, and network rules as code (Bicep) so a tightened NSG or a revoked role doesn’t silently break backups unreviewed.
• Stagger schedules across the fleet so dozens of VMs don’t all push into the vault at the same minute.

Security notes

• Least-privilege RBAC for backup. Grant operators Backup Operator (run/restore) rather than broad roles, and Backup Contributor only where policy management is needed. A revoked or missing role is itself a failure cause (config/restore denied), so manage these deliberately.
• Protect the vault from accidental and malicious deletion. Enable soft delete (retains deleted backup data for a recovery window) and consider an immutable vault / multi-user authorisation so ransomware or a rogue admin can’t wipe your recovery points. Disabling protection is a high-value action — alert on it and restrict it; turning off backups before encrypting is a known ransomware pattern these features defend against. A backup you can delete in one click is not a safe backup.
• Encrypt the data at rest. Vault data is encrypted by default; for stricter control use customer-managed keys (CMK) so you hold the key — and note the key’s availability becomes part of your restore path (a deleted/disabled key blocks restore). Get the Key Vault model right per Azure Key Vault: Secrets, Keys & Certificates, and use managed identity for integrations rather than embedding secrets.
• Keep the network lockdown intentional. The same NSG/firewall discipline that secures the VM breaks the transfer if you forget Storage egress — secure and allow the one path Backup needs.

Cost & sizing

The bill drivers and how they interact with the failure classes:

• Protected-instance fee + storage dominate. Azure Backup charges a per-protected-instance fee (bucketed by used size — ≤50 GB, ≤500 GB, then per 500 GB) plus the storage the recovery points consume. Failures don’t cost compute, but they cost coverage — a VM whose jobs keep failing pays for nothing useful.
• Storage replication and instant-restore retention drive the storage rate: LRS cheapest, GRS more for cross-region durability, ZRS between (see Azure Storage Redundancy Decoded: LRS vs ZRS vs GRS vs RA-GRS and How to Choose); and snapshot storage scales with the instant-restore window — more days and on-demand snapshots mean more storage and a higher chance of the retention cap that fails jobs.
• Transfer egress is rarely the painful line item — but a NAT/firewall path you add to fix a transfer break has its own small hourly + per-GB cost. The cost of a missed backup dwarfs it.

A rough monthly picture for one ~200 GB Windows VM, daily backup, 30-day retention on LRS: a per-instance fee plus the incremental storage of ~30 restore points typically lands in the low thousands of INR per VM per month — less with shorter retention, more with GRS. The drivers:

Interview & exam questions

1. A nightly VM backup fails with UserErrorGuestAgentStatusUnavailable. What does that mean and what do you check first? The orchestrator couldn’t get a usable status from the Azure VM agent inside the guest. First check whether the VM is even running (az vm get-instance-view → PowerState) — a deallocated VM is the most common cause — then the agent status (Ready vs Not Ready). Fix by starting the VM and/or restarting/updating the agent so it reports Ready.

2. A job shows green “completed with warnings.” Is the backup fine? Not necessarily — a warning typically means Backup created a crash-consistent recovery point because application-consistency failed (VSS in a bad state on Windows, or missing/failing pre/post scripts on Linux). The disk is recoverable, but a database may need crash recovery on restore. Open the job to read the consistency, fix the underlying VSS writer or scripts, and re-validate.

3. The snapshot phase succeeds but the job still fails with a NoNetwork error. Where’s the problem? The data-transfer phase — the background copy into the vault — failed because the VM has no egress to Azure Storage. Confirm with the NIC’s effective NSG rules and route table; the cause is almost always an NSG/firewall blocking outbound to Storage or a forced-tunnel UDR. Fix by allowing the Storage service tag (443) and/or exempting it from the tunnel.

4. How do you repair a VMSnapshot extension stuck in a failed provisioning state? Remove the extension (az vm extension delete --name VMSnapshot / VMSnapshotLinux); Azure Backup re-installs it on the next run, so deleting the broken instance is the safe, reliable repair. Then trigger an on-demand backup to re-provision and confirm.

5. Why might a Linux VM consistently get only crash-consistent backups while a Windows VM gets application-consistent ones? Windows uses VSS (built in); Linux has no VSS, so application-consistency requires pre-/post-snapshot scripts that you author and the agent runs. Absent, misconfigured, or non-zero-exit scripts mean Backup falls back to crash-consistent. Provide correct, fast scripts that exit 0.

6. What is the instant-recovery snapshot, and how can it cause job failures? Azure Backup first takes a fast instant-recovery snapshot on the managed disks (default retention historically up to 5 days, configurable) to enable near-instant restores, then copies data to the vault. Accumulate too many snapshots (frequent on-demand runs) and you hit the retention cap, and the next job fails until older snapshots age out. Right-size the retention window and on-demand frequency.

7. A VM is running and the agent is Ready, but the job fails UserErrorVmNotInDesiredState. What’s going on? The compute layer can’t take the snapshot because the VM or its disks are in a conflicting state — a concurrent resize, another snapshot, a disk attach/detach, or an overlapping backup. Check the VM’s activity log for in-flight operations and the Backup jobs blade for an overlapping run; let it finish, then retry.

8. How do you protect backups from accidental or malicious deletion? Enable soft delete so deleted backup data is retained for a recovery window, use an immutable vault / multi-user authorisation to prevent retention reduction or deletion, and restrict who can disable protection with least-privilege RBAC (Backup Operator vs Backup Contributor). These defend against the ransomware pattern of disabling backups before encrypting.

9. Two VMs back up to the same vault; one fails UserErrorRequestDisallowedByPolicy. Why one and not both? An Azure Policy denies a resource Backup must create for that VM’s scope (blocking a snapshot or staging resource by location, SKU, or naming). The activity log shows the deny event. Fix by scoping the policy to exclude backup-created resources or adding an exemption — the other VM simply isn’t in the denying scope.

10. App-consistency fails only on busy database nights. Cause and fix? Heavy disk I/O at the backup hour makes the VSS quiesce time out, so Backup falls back to crash-consistent — confirm by correlating the job time with disk IOPS/queue-depth metrics. Fix by moving the window off the peak I/O period and/or to a higher-throughput disk SKU so the quiesce flushes within the window.

11. Which RBAC roles govern who can run and configure backups, and why does it matter for failures? Backup Operator can run backups and restores; Backup Contributor can also manage policy/protection; broad roles like Owner/Contributor cover everything. It matters because a missing or revoked role is itself a failure cause — config or restore operations get permission-denied — so least-privilege must be deliberate, not accidental.

12. How do you validate a fix without waiting for the nightly schedule? Trigger an on-demand backup (az backup protection backup-now) and watch the job to Completed (and check it’s application-consistent, not just completed-with-warnings). On-demand runs also re-push the VMSnapshot extension, so they double as the validation step after an agent update or extension repair.

These map to AZ-104 (Administrator) — back up and restore Azure VMs — and AZ-305 (Solutions Architect) — design business continuity; the redundancy/identity angles touch storage replication and RBAC. A compact cert-mapping for revision:

Quick check

1. A backup fails with UserErrorGuestAgentStatusUnavailable. What is the first thing to check, and the single command that answers it?
2. A job reports “completed with warnings — crash-consistent.” Why is that not necessarily fine, and where do you look on Windows to find the cause?
3. The snapshot succeeds but the job fails with a NoNetwork error. Which phase failed, and what do you allow to fix it?
4. The VMSnapshot extension is stuck in a Failed state. What is the safe, reliable repair?
5. Your Linux VM only ever produces crash-consistent recovery points. Why, and what’s the fix?

Answers

1. Whether the VM is actually running — a deallocated VM has no agent to talk to. az vm get-instance-view shows both the PowerState and the agent’s Ready/Not Ready status in one call.
2. Because a warning means Backup created a crash-consistent point after application-consistency failed — a database may need crash recovery on restore. On Windows, run vssadmin list writers (elevated); every writer should read Stable / No error, and a non-Stable one is the cause.
3. The data-transfer phase (the background copy of the snapshot into the vault), which depends on the VM’s egress to Azure Storage. Allow outbound to the Storage service tag (port 443) in the NSG and exempt it from any forced-tunnel UDR.
4. Remove the extension (az vm extension delete --name VMSnapshot / VMSnapshotLinux); Azure Backup re-installs it automatically on the next backup run. Then trigger an on-demand backup to re-provision and validate.
5. Linux has no VSS; application-consistency requires pre-/post-snapshot scripts that you author and the agent runs. If they’re missing or failing, Backup falls back to crash-consistent. Provide correct, fast scripts that exit 0.

Glossary

• Recovery Services vault — the regional Azure resource that stores recovery points and runs the backup policy and jobs; it orchestrates but does no in-guest work itself.
• Backup job — a single backup run (scheduled or on-demand) you watch succeed or fail; a backup item is the protected resource it runs against.
• Azure VM agent (guest agent) — the in-guest agent (WindowsAzureGuestAgent / walinuxagent) that hosts extensions and reports Ready/Not Ready; backup depends on it.
• VMSnapshot extension — the in-guest worker (VMSnapshot / VMSnapshotLinux) Azure Backup pushes to take the snapshot; must be in a Succeeded provisioning state.
• VSS (Volume Shadow Copy Service) — the Windows service that quiesces applications so a snapshot is application-consistent; its writers must be Stable.
• VSS writer — a per-application component (SQL, AD, etc.) VSS coordinates; a writer in a bad state breaks application-consistency.
• Pre/post-snapshot scripts — the Linux equivalent of VSS: scripts you author to quiesce/unquiesce an app for application-consistency.
• Application-consistent — a recovery point taken with apps quiesced (VSS/scripts), restorable without manual recovery.
• Crash-consistent — a recovery point equivalent to pulling power: disk intact, apps may need their own recovery; the fallback when app-consistency fails.
• Instant-recovery snapshot — a fast snapshot kept on the managed disks for a short retention window, enabling near-instant restores before data reaches the vault.
• Data transfer — the background copy that moves snapshot data from the guest into the vault; depends on the VM’s egress to Azure Storage.
• Soft delete — a vault feature that retains deleted backup data for a recovery window, defending against accidental or malicious deletion.
• Service tag (Storage) — a named, Azure-maintained set of IP ranges you allow in NSG/firewall rules instead of hard-coding addresses.
• Wireserver / IMDS — the platform endpoints (168.63.129.16 and 169.254.169.254) the agent needs to reach; blocked access makes the agent fail.

Next steps

You can now localise any failed Azure VM backup job to a hop — agent, extension, VSS, snapshot, or transfer — and fix it. Build outward:

• Next: Protect Your First Azure VM with Azure Backup: A Guided Walkthrough — the end-to-end setup these failures sit on top of.
• Related: Azure Backup and Site Recovery: Protecting Workloads from Loss — where VM backup fits alongside replication and DR.
• Related: BCDR Foundations on Azure: Making Sense of RTO, RPO, and the Resilience Spectrum — the RTO/RPO framing that decides your retention and restore strategy.
• Related: Azure Storage Redundancy Decoded: LRS vs ZRS vs GRS vs RA-GRS and How to Choose — the vault-storage durability choice and what it costs.
• Related: Cannot RDP or SSH to an Azure VM? A First-Principles Troubleshooting Path — the same network plumbing that breaks agent reachability and the data transfer.