Diagnosing Azure VM Boot Failures: Boot Diagnostics, Screenshots and Common Stuck-at-Boot Fixes

The VM was running for eight months. You applied a routine kernel update, rebooted, and now it will not come back. SSH times out. RDP times out. The portal cheerfully says the VM is Running — power is on, the host is healthy, the platform did its job — but nothing answers on any port, and you have no idea whether the guest OS reached a login prompt, panicked halfway through init, or is sitting at an fsck prompt waiting for a keystroke nobody can send. This is the stuck-at-boot failure, and it is uniquely frustrating because every tool you normally reach for — SSH, RDP, the VM agent, run-command — needs the guest to have already booted. When the guest never finishes booting, all of them are dead too.

The way out is to stop trying to connect into the OS and look at it from the platform side. Azure gives you two platform-level windows that work even when the guest network is down, the disk is read-only, or the kernel panicked. Boot Diagnostics captures a serial console log (everything the guest writes to its first serial port, COM1 — the stream a physical server prints to a screen during POST and OS startup) plus a boot screenshot (a snapshot of the VM’s console framebuffer). The Serial Console is the live, interactive twin: a keyboard-and-text session to that same port, so you can answer the fsck prompt, pick a previous kernel in GRUB, or break into the boot loader.

This article is that translation layer. We treat “VM won’t boot” not as one bug but as a family of symptom classes — kernel panic, GRUB/boot-loader failure, fsck/filesystem errors, a full OS disk, a missing driver, a network stack that never came up, a Windows boot-manager error — each with its own tell-tale serial line and its own confirmed fix. You will enable Boot Diagnostics correctly, read the serial log and screenshot like a console operator, drive the Serial Console to recover interactively, and — when the guest is truly wedged — repair it offline by attaching its OS disk to a healthy rescue VM. By the end, the portal saying “Running” while nothing answers will be the start of a five-minute diagnosis, not a two-hour mystery.

What problem this solves

A VM that is powered on but unreachable is one of the most common and most misdiagnosed incidents in Azure. The instinct is to treat it as a connectivity problem — check the NSG, check the public IP, restart the VM, redeploy it — and sometimes that is right. But very often the network is fine and the guest operating system never finished booting. Those two failures look identical from the outside (timeout on every port) and have completely different fixes. Without a way to see inside the boot sequence you cannot tell them apart, so you guess, and guessing during an outage is expensive.

What breaks without this skill: an engineer reboots three times (useless if the OS disk is corrupted or a kernel update broke the initramfs), then redeploys to a new host (which migrates the same broken disk and changes nothing), then escalates and waits hours — while the answer, a single readable line like Kernel panic - not syncing: VFS: Unable to mount root fs or error: file '/boot/vmlinuz...' not found, sat in the boot diagnostics log the whole time. The cost is not just downtime but the wrong remediations along the way, several of which (reimage, delete-and-recreate) can destroy state if done in a panic.

Who hits this: everyone who runs IaaS VMs long enough. It bites hardest after kernel or OS patching (a bad kernel, a broken initramfs, a GRUB entry pointing at a removed kernel), after disk operations (a disk that filled up, an /etc/fstab edit referencing a device that isn’t there), after agent or extension changes that wedge a Windows boot, and after image customisation (a generalised image that boots in one region and panics in another for a missing driver). The fix is almost never “redeploy” — it is “read the serial log, find the exact boot stage that failed, and repair that.”

To frame the whole field before the deep dive, here is every stuck-at-boot symptom class this article covers, the boot stage it lives in, and the one place to look first.

Learning objectives

By the end of this article you can:

• Explain what Boot Diagnostics actually captures — the serial console log and the boot screenshot — and why both work when SSH, RDP and the VM agent are all unreachable.
• Enable Boot Diagnostics in the correct mode (managed storage vs a custom storage account) with az, Bicep and the portal, and avoid the misconfigurations that leave the screenshot blank.
• Read a Linux serial log and localise a failure to a specific boot stage: firmware, GRUB, kernel, initramfs, fsck/mount, or userspace networking.
• Recognise the signature lines for each failure — Kernel panic - not syncing, Give root password for maintenance, error: file not found, You are in emergency mode, Windows 0xc000000e/INACCESSIBLE_BOOT_DEVICE — and name the fix for each.
• Use the Serial Console interactively to answer an fsck prompt, boot a previous kernel from the GRUB menu, enter a Linux rescue/emergency shell, or run SAC/bcdedit on Windows.
• Recover a wedged VM offline by attaching its OS disk to a healthy rescue VM (manually and via az vm repair), fix the disk, and swap it back.
• Diagnose a “boots but unreachable” VM as a guest networking problem rather than a boot problem, using the boot screenshot to prove the OS reached a login prompt.

Prerequisites & where this fits

You should already be comfortable deploying and managing a basic Azure VM — creating one, attaching a data disk, and connecting over SSH or RDP when things are healthy. If that is new, start with Your First Azure Virtual Machine: A Step-by-Step Deployment in Portal, CLI and PowerShell. You should know how to run az in Cloud Shell, read JSON output, and understand that a VM separates compute (the VM resource and its size) from storage (the OS disk and any data disks as independent managed-disk resources). Basic Linux boot vocabulary (boot loader, kernel, init system, /etc/fstab) and basic Windows boot vocabulary (boot manager, BCD, drivers) help; we define the load-bearing terms as we go.

This sits in the Compute Troubleshooting track and is the boot-stage sibling to two connection-stage guides. When the OS is up but you still cannot reach it, the problem is connectivity, covered in Cannot RDP or SSH into Your Azure VM? A Beginner’s Connectivity Troubleshooting Checklist; this article is what you read before that, to confirm the OS even reached a login prompt. It pairs with VM Extension Provisioning Failed: Debugging Custom Script, DSC and Azure Monitor Agent Errors, because a wedged extension can stall a boot, and with Azure VM Disk Types Demystified: Standard HDD, Standard/Premium SSD, Premium v2 and Ultra Disks, since several boot failures are really disk failures. The offline-repair workflow leans on understanding the OS disk as a detachable managed-disk resource.

A quick map of which boot stage owns which failure, so you call the right fix fast:

Core concepts

Five mental models make every later diagnosis obvious.

“Running” describes the host, not the guest. The portal power state (PowerState/running) means the hypervisor has the VM powered on; it says nothing about whether the guest OS reached a usable state. A kernel-panicked VM, one stuck at an fsck prompt, and a perfectly healthy VM all report Running. The platform’s view of “up” stops at the VM’s virtual power button — to see past it you need a window into the guest, which is exactly what Boot Diagnostics provides.

Boot Diagnostics is a platform-side window, not an in-guest agent. It captures two things. The serial console log is the byte stream the guest writes to its first serial port (COM1 / ttyS0 on Linux) — the modern equivalent of a serial terminal cabled to a physical server. The boot screenshot is a periodic image of the VM’s console framebuffer (what a monitor on the box would show). Crucially, both are produced by the hypervisor reading the VM’s virtual hardware, not by software inside the OS, so they keep working when the guest network is down, the VM agent is dead, the disk is read-only, or the kernel has panicked — the exact moments you need them.

The boot is a fixed sequence of stages, and the failure lives in exactly one. Firmware (BIOS/UEFI) does POST and hands off to the boot loader (GRUB on Linux, Windows Boot Manager/BCD on Windows), which loads the kernel; the kernel brings up core drivers and, with help from the initramfs (a tiny in-memory root filesystem holding just enough drivers to find the real root disk), mounts the real root filesystem; early userspace checks and mounts filesystems (fsck); then the init system (systemd) starts services, the agent and networking. A failure stops at one stage and prints a stage-specific message just before it dies — identify the stage from the last meaningful line and you have identified the failure class.

The Serial Console is the interactive twin of the serial log. The log is read-only history; the Serial Console is a live, bidirectional text session to the same COM1 port. Being text-over-serial (not a graphical desktop, not SSH), it works with zero guest networking, so you can interact with the boot: type the root password at a maintenance prompt, boot an older kernel from GRUB, drop into an emergency shell to fix /etc/fstab, or on Windows talk to SAC (Special Administration Console) and run bcdedit. It needs the guest to offer a console on the serial port — Azure’s marketplace images mostly do; custom images often don’t (a gotcha we cover).

When the guest is too broken to fix from inside, fix the disk from outside. Some failures (a corrupted boot loader, an /etc/fstab typo that wedges before any shell, a missing driver) never give you a usable prompt. The escape hatch is offline repair: stop the broken VM, snapshot-and-copy its OS disk, attach the copy as a data disk to a healthy rescue VM, fix the file/config/boot-loader, then swap the repaired disk back as the original VM’s OS disk. The az vm repair extension automates this. It is the universal backstop: if you can read and write the OS disk from a working VM, almost any boot failure is fixable.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side.

Enabling and reading Boot Diagnostics

You cannot diagnose what you did not capture. Boot Diagnostics must be on for the serial log and screenshot to exist, and — critically — the Serial Console depends on it too (it reuses the same storage path). Turn it on at create time for every production VM; turning it on after a VM is already wedged still works for the live screenshot but you will have lost the historical serial log from the failing boot.

Managed vs custom storage — pick the simpler one

Boot Diagnostics writes the log and screenshot to a storage account. You have two modes.

For nearly everyone, managed is correct: nothing to create, secure, or misconfigure. Use custom only when policy requires diagnostics in a storage account you control (retention, geography, audit). A large share of “the screenshot is blank” tickets comes from custom storage in the wrong region, a Premium account (Boot Diagnostics needs a Standard general-purpose account), or access changed underneath the VM.

Enable managed Boot Diagnostics on an existing VM with one command:

# Managed storage — simplest, recommended. No storage account argument.
az vm boot-diagnostics enable \
  --name vm-app-01 --resource-group rg-app-prod

To use a specific custom storage account instead, pass its blob endpoint:

az vm boot-diagnostics enable \
  --name vm-app-01 --resource-group rg-app-prod \
  --storage "https://stbootdiagprod.blob.core.windows.net/"

In Bicep, managed Boot Diagnostics is a two-line block on the VM (omit storageUri to use managed storage):

resource vm 'Microsoft.Compute/virtualMachines@2024-07-01' = {
  name: 'vm-app-01'
  location: location
  properties: {
    diagnosticsProfile: {
      bootDiagnostics: {
        enabled: true
        // No storageUri => Azure-managed storage account (recommended)
      }
    }
    // ... hardwareProfile, storageProfile, osProfile, networkProfile ...
  }
}

Reading the boot screenshot and serial log

Once enabled, two artefacts become available on the VM under Help → Boot diagnostics in the portal: the Screenshot tab (the framebuffer image) and the Serial log tab (the text stream). Both are also reachable from the CLI.

# Fetch the latest serial console log to a file you can read/grep
az vm boot-diagnostics get-boot-log \
  --name vm-app-01 --resource-group rg-app-prod > vm-app-01-serial.log

# Get the SAS URLs for the screenshot and serial log blobs
az vm boot-diagnostics get-boot-log-uris \
  --name vm-app-01 --resource-group rg-app-prod -o json

How to read each artefact, and what it is best at telling you:

The two are complementary. The screenshot is unbeatable for Windows (the stop code on a blue screen is the whole diagnosis) and for confirming a Linux VM reached a login prompt (which proves the boot finished and the problem is networking). The serial log is unbeatable for Linux boot failures, where the last few hundred lines name the stage that died. An empty serial log on a booting Linux VM almost always means the guest isn’t printing to the serial console — the kernel command line lacks console=ttyS0 (marketplace images include it; hand-built images frequently don’t).

The first lines to scan in any Linux serial log, in priority order:

Linux boot failures: stage by stage

Linux gives you the richest serial output of any guest, so most Linux boot failures are diagnosable from the serial log alone. We walk the boot in order; the failure is whichever stage prints its signature line last.

GRUB and boot-loader failures

GRUB (the GRand Unified Boot loader), after firmware POST, presents the boot menu and loads the selected kernel and initramfs from /boot. It fails when its config points at a kernel file that no longer exists, when /boot is corrupted or out of space, or when the GRUB install is damaged. The classic post-patching case: a kernel update wrote a new entry but the package transaction was interrupted, leaving grub.cfg referencing a vmlinuz/initramfs that isn’t on disk.

Confirm. The boot screenshot shows a GRUB menu that hangs, or a bare grub> / grub rescue> prompt; the serial log shows error: file '/boot/vmlinuz-5.15.0-1064-azure' not found, error: you need to load the kernel first, or error: no such partition.

Fix (interactive, fastest). Open the Serial Console, reboot, and at the GRUB menu pick a previous, known-good kernel under “Advanced options for…”. If the menu doesn’t appear, hold Shift (BIOS) or press Esc (UEFI) early to force it. Once booted on the good kernel, repair the default: reinstall the broken kernel package, regenerate the config, and reinstall GRUB to the disk.

# After booting a working kernel via the Serial Console (Ubuntu/Debian example)
sudo apt-get install --reinstall linux-image-azure
sudo update-grub                 # regenerate /boot/grub/grub.cfg
sudo grub-install /dev/sda       # reinstall GRUB to the OS disk (BIOS)

The boot-loader signatures and their fixes:

Kernel panic

A kernel panic is the kernel hitting an unrecoverable error and halting — the Linux equivalent of a blue screen. The most common Azure trigger is being unable to mount the root filesystem because the initramfs lacks the driver for the root device (the Hyper-V storage driver hv_storvsc, or the disk/filesystem module) — typical of a regenerated-but-incomplete initramfs after patching, or a generalised custom image built without the Azure/Hyper-V drivers. The other trigger is simply a broken or incompatible kernel.

Confirm. The serial log ends with Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) or similar, often preceded by No filesystem could mount root, tried: .... The boot screenshot may freeze on the panic backtrace.

Fix. Boot a previous kernel from the GRUB menu via the Serial Console (the fastest test — if the old kernel boots, the new kernel/initramfs is the culprit). Then regenerate the initramfs ensuring the Azure drivers are included, or reinstall the kernel:

# Ubuntu/Debian: rebuild initramfs for the running kernel (after booting a good one)
sudo update-initramfs -u -k all
# RHEL/CentOS/Alma: regenerate initramfs with Hyper-V drivers
sudo dracut -f --add-drivers "hv_vmbus hv_storvsc hv_netvsc" --kver $(uname -r)

If no kernel boots at all (so you cannot get a shell), repair offline via a rescue VM: chroot into the mounted OS disk and run the same update-initramfs/dracut there. A few panic variants to recognise: No filesystem could mount root means the root-filesystem module (not just the disk driver) is absent from the initramfs; Attempted to kill init! means a corrupt init/userspace and needs an offline chroot reinstall of the base packages; and a panic that appears only after image generalisation almost always means the custom image lacks the Azure/Hyper-V drivers — rebuild the image with them (see the generalise-and-capture guide).

fsck failures and emergency mode

When a filesystem is marked dirty (unclean shutdown, forced stop, host issue) or an /etc/fstab entry can’t be satisfied, the boot pauses to run fsck (filesystem check) or drops into emergency/maintenance mode asking for the root password. On a physical server an operator answers; on an Azure VM with no console the boot simply hangs there forever — which from outside looks identical to a hard crash. A single bad /etc/fstab line (a wrong UUID after a disk change, or a data disk referenced without nofail that isn’t attached) is one of the most common self-inflicted causes.

Confirm. The serial log shows Give root password for maintenance (or press Control-D to continue), You are in emergency mode, a systemd job like Dependency failed for /mnt/data and Failed to mount, or an fsck run reporting errors and pausing. The boot screenshot shows the maintenance prompt.

Fix (interactive). Open the Serial Console, supply the root password at the maintenance prompt (or the emergency shell appears), and either run fsck to repair the filesystem or fix the offending /etc/fstab line:

# At the emergency/maintenance shell over the Serial Console:
journalctl -xb | tail -50        # see exactly which mount/unit failed
fsck -y /dev/sda1                # repair the root or named filesystem
# Fix the bad fstab entry, then add nofail so a missing data disk never blocks boot:
#   UUID=<uuid>  /mnt/data  ext4  defaults,nofail  0  2
mount -a                         # validate fstab now mounts cleanly
reboot

If you cannot get the maintenance shell (no root password set for single-user, or it hangs before the prompt), repair offline via a rescue VM and edit /etc/fstab on the mounted disk. The golden rule: every non-root mount in /etc/fstab should carry nofail so that a detached or renamed data disk degrades to “not mounted” instead of “no boot.” The fsck/mount signatures:

A full OS disk

A VM whose OS disk is 100% full can fail to boot or boot into a broken state: services that need to write (logs, lock files, temp) fail, the agent can’t start, and sometimes the boot stalls. The usual culprits are runaway logs under /var/log, an unrotated journal, a package cache, or an application writing to the OS disk instead of a data disk.

Confirm. The serial log shows No space left on device from various services, or you connect via the Serial Console and df -h shows / at 100%.

Fix (interactive). Via the Serial Console, find and clear the space, then prevent recurrence by capping the journal and moving heavy writers to a data disk:

df -h /                              # confirm / is full
journalctl --disk-usage              # the journal is a frequent offender
journalctl --vacuum-size=200M        # trim it immediately
du -xh / | sort -rh | head -20       # find the biggest directories on /
# Cap it permanently in /etc/systemd/journald.conf: SystemMaxUse=200M

If / is so full that even the shell misbehaves, expand the OS disk (resize the managed disk, then grow the partition and filesystem) or repair offline. The durable fixes are always the same: cap the journal with SystemMaxUse, move heavy writers (app logs, databases) to a data disk so they can never fill /, and add a disk-percentage alert that fires at ~85% so you hear about it before a service does.

Windows boot failures and SAC

Windows emits far less to the serial port than Linux, so for Windows the boot screenshot is your primary tool: a Windows boot failure is usually a blue screen with a stop code or a Windows Boot Manager error screen, and the code on that screen is most of the diagnosis. The interactive serial channel for Windows is SAC (Special Administration Console), an emergency text console exposed over the serial port via EMS (Emergency Management Services); from SAC you can open a CMD channel and run bcdedit and other recovery commands without RDP.

Confirm. Read the boot screenshot. Common screens and their meaning:

Fix (interactive via SAC). If EMS/SAC is enabled (Azure Windows marketplace images generally have it), open the Serial Console; you land at the SAC> prompt. From there create and enter a CMD channel:

SAC>cmd                       (creates a channel, e.g. "Channel: Cmd0001")
SAC>ch -si 1                  (switch into channel 1)
... log in, then at the CMD prompt:
bcdedit /enum                 (inspect boot entries)
bcdedit /set {default} recoveryenabled No   (stop an auto-recovery loop while you work)

Fix (offline, the reliable path for BCD/driver problems). Most Windows boot failures — BCD corruption, INACCESSIBLE_BOOT_DEVICE, a bad driver — are repaired by attaching the OS disk to a rescue Windows VM and using bcdedit/bootrec against the attached disk, or removing/rolling back the faulting driver. The az vm repair workflow (next section) handles the disk swap; the in-rescue commands target the attached disk’s BCD store and Windows\System32\drivers folder.

Windows boot failures rarely benefit from “restart and hope.” If the screenshot shows a stop code, go straight to offline repair against the named cause; rebooting an INACCESSIBLE_BOOT_DEVICE just reproduces the same blue screen.

Offline repair with a rescue VM

When you cannot get a usable prompt — the boot wedges before any shell, GRUB or BCD is too broken to interact with, or /etc/fstab stops the boot cold — the universal fix is to repair the OS disk from a healthy VM. The principle is simple: an unbootable OS disk is still a perfectly readable data disk when attached to a working VM. You mount it, fix the file/config/boot record, detach, and put it back.

The manual workflow

Azure recommends doing this against a snapshot/copy of the OS disk (so the original is never at risk), but here is the core flow conceptually:

1. Stop (deallocate) the broken VM so its OS disk can be detached safely.
2. Snapshot the OS disk and create a new managed disk from the snapshot (your working copy).
3. Attach that copy as a data disk to a healthy rescue VM in the same region.
4. Mount and repair the disk inside the rescue VM (fix /etc/fstab, rebuild initramfs/GRUB, run chkdsk/bcdedit, free disk space, remove a bad driver).
5. Detach the repaired disk from the rescue VM.
6. Swap it back as the broken VM’s OS disk (az vm update --os-disk <repaired-disk-id>), then start the VM.

# 1. Stop the broken VM (deallocate)
az vm deallocate --name vm-app-01 --resource-group rg-app-prod

# 6. After repair: swap the fixed disk back in as the OS disk, then start
az vm update --name vm-app-01 --resource-group rg-app-prod \
  --os-disk /subscriptions/<sub>/resourceGroups/rg-app-prod/providers/Microsoft.Compute/disks/vm-app-01-osdisk-fixed
az vm start --name vm-app-01 --resource-group rg-app-prod

The automated workflow: az vm repair

The vm-repair extension scripts the entire dance — snapshot the OS disk, build a rescue VM, attach the copy — with three commands. This is the path to prefer because it never touches the original disk and it cleans up after itself.

# Add the extension once
az extension add -n vm-repair

# 1) Create a rescue VM with a COPY of the broken VM's OS disk attached
az vm repair create \
  --verbose --name vm-app-01 --resource-group rg-app-prod \
  --repair-username azureuser --repair-password '<StrongP@ssw0rd!>'

# 2) Run a repair: either a built-in script or your own commands inside the rescue VM
az vm repair run \
  --name vm-app-01 --resource-group rg-app-prod \
  --run-id linux-alar2 --run-on-repair        # ALAR runs guided Linux repairs

# 3) When fixed, swap the repaired disk back to the original VM and tear down the rescue VM
az vm repair restore \
  --name vm-app-01 --resource-group rg-app-prod

When each repair path is the right tool:

“It boots but I can’t reach it” — the networking impostor

The most over-diagnosed boot failure is the one that isn’t a boot failure. The VM boots completely, reaches a login prompt, the OS is healthy — but SSH or RDP still times out. From the outside this is indistinguishable from a stuck boot, and engineers waste an hour in the serial log when the boot was never the problem.

The boot screenshot settles it in seconds. If the screenshot shows a normal Linux login: prompt or the Windows lock screen, the OS finished booting and your problem is connectivity, not boot — go to Cannot RDP or SSH into Your Azure VM? A Beginner’s Connectivity Troubleshooting Checklist. If instead the screenshot shows a kernel panic, GRUB menu, BSOD, or a maintenance prompt, it is a real boot failure and stays in this article.

The “booted but unreachable” causes split into guest networking failed during boot (which the serial log can show) versus the OS is fine and the path to it is blocked (which it can’t). Quick triage:

The discipline is fixed: always look at the boot screenshot first. Two seconds of looking tells you which half of the universe you are in — boot or network — and stops the single most common hour-long detour in VM troubleshooting.

Architecture at a glance

Hold the boot as a single left-to-right pipeline, because the whole diagnostic method is “find the stage where the line stops moving.” On the far left is the Azure host: the hypervisor powers the VM on and reports PowerState/running — and that report stops at the virtual power button. To its right, inside the guest, the boot runs through fixed stations: firmware (UEFI/BIOS POST finds the OS disk) → boot loader (GRUB or Windows Boot Manager loads the kernel) → kernel + initramfs (core drivers up, real root filesystem mounted) → early userspace (fsck checks and mounts) → systemd/services (init, the VM agent, your app) → networking (DHCP, interface, routing). A healthy boot flows all the way right and lands at a login prompt with SSH/RDP listening; a broken boot stops at exactly one station and prints a stage-specific line just before it halts.

Running underneath that pipeline, independent of it, are the two platform-side windows you use. The hypervisor continuously taps the VM’s virtual serial port (COM1) — capturing the byte stream as the serial log and offering the live Serial Console — and periodically photographs the console framebuffer as the boot screenshot, both feeding Boot Diagnostics storage. Because they read the virtual hardware, not anything inside the OS, they keep reporting no matter how early or badly the guest fails. So the model is two layers: the guest boot marching left to right and stalling at one station, and beneath it the always-on serial/framebuffer tap that lets you see which station stalled and — via the Serial Console or an offline disk swap — reach in and restart it. Diagnosis is just: read the last meaningful line, name the station, apply that station’s fix.

Real-world scenario

Northwind Retail runs its order-processing API on a fleet of Ubuntu 22.04 VMs (Standard_D4s_v5, Premium SSD OS disks) behind a load balancer in centralindia. One Tuesday night, the platform team rolled the monthly OS patch baseline across the fleet via their configuration-management tool, which installed a new kernel and rebooted each VM in waves. The first two waves came back clean. The third wave — four VMs — went to Running in the portal and never returned to the load balancer’s healthy pool. The on-call engineer, paged at 23:40, did the natural thing: restarted one of the four. It came back Running and still failed the health probe. He restarted it again. Same result. He was forty minutes in and reaching for “redeploy.”

Instead, his lead opened Boot diagnostics → Screenshot on the wedged VM. The framebuffer showed a GRUB rescue artefact and, on the Serial log tab, the smoking gun near the end: error: file '/boot/vmlinuz-5.15.0-1071-azure' not found followed by Kernel panic - not syncing: VFS: Unable to mount root fs. The new kernel’s package transaction had been interrupted by the wave reboot on exactly these four machines, leaving grub.cfg pointing at a kernel whose initramfs was never fully generated. Every restart faithfully tried the same broken default kernel and panicked the same way — which is precisely why rebooting “fixed” nothing.

The lead opened the Serial Console on the first VM, rebooted it, and at the GRUB menu chose Advanced options → the previous kernel (5.15.0-1064-azure). The VM booted in under a minute on the known-good kernel and immediately rejoined the load balancer. With a real shell now available, he repaired the default properly:

sudo apt-get install --reinstall linux-image-5.15.0-1071-azure
sudo update-initramfs -u -k 5.15.0-1071-azure
sudo update-grub

He confirmed the new kernel now had a valid initramfs, rebooted once to verify it came up on the patched kernel unaided, then repeated the exact sequence on the other three VMs via their Serial Consoles. Total time from “open the screenshot” to “all four healthy on the patched kernel”: about 25 minutes — most of it the deliberate verify-reboots. The earlier 40 minutes of blind restarts had produced nothing.

Two changes came out of the postmortem. First, the patch playbook was amended so kernel installs complete and verify the initramfs before the reboot is allowed, and waves are gated on each VM rejoining the pool before the next wave proceeds. Second — the cheap insurance — they confirmed Boot Diagnostics is enabled by policy on every VM (it was, which is the only reason the serial log from the failing boot existed to read) and wrote a one-page runbook: VM stuck after patch → open the boot screenshot first → read the serial log’s last lines → boot the prior kernel via Serial Console → repair the default. The next time a patch wedged a VM, the incident was eight minutes, not two hours, because nobody guessed.

Advantages and disadvantages

Boot Diagnostics and the Serial Console are not a silver bullet; they are a specific, powerful window with real limits. Weigh them honestly.

When each side dominates: for Linux, the serial log + Serial Console are decisive and rarely need a fallback — the empty-log disadvantage only bites on custom images you fix once with console=ttyS0. For Windows, lean on the screenshot (the stop code is the diagnosis) and accept that real BCD/driver fixes mean offline repair, the slower workflow being the price of reliability. The “must enable it beforehand” limit hurts most in practice — which is why the highest-value action in this whole topic is to turn Boot Diagnostics on for every VM at create time, so the window exists before you need it.

Hands-on lab

This lab deliberately breaks a throwaway Linux VM’s boot, then diagnoses and fixes it exactly as you would in production. Use a B-series VM (cheap) and delete everything at the end. Budget ~30 minutes and a few rupees of compute.

1. Create a small Ubuntu VM with Boot Diagnostics on.

RG=rg-bootlab
LOC=centralindia
az group create -n $RG -l $LOC

az vm create -g $RG -n vm-bootlab \
  --image Ubuntu2204 --size Standard_B1s \
  --admin-username azureuser --generate-ssh-keys

# Enable managed Boot Diagnostics (if not already on)
az vm boot-diagnostics enable -g $RG -n vm-bootlab

2. Confirm the serial log and screenshot work on a healthy VM. This is your baseline — you should see a clean boot ending at a login prompt.

az vm boot-diagnostics get-boot-log -g $RG -n vm-bootlab | tail -30

In the portal, open vm-bootlab → Boot diagnostics → Screenshot: you should see a Linux login prompt. That confirms both windows are capturing.

3. Break the boot on purpose — a bad /etc/fstab entry. SSH in and add a mount for a device that does not exist, without nofail. This reproduces the single most common self-inflicted boot hang.

ssh azureuser@<public-ip>
# Add a non-existent disk to fstab WITHOUT nofail (this is the mistake):
echo '/dev/sdz1  /mnt/ghost  ext4  defaults  0  2' | sudo tee -a /etc/fstab
sudo reboot

4. Observe the failure from the platform side. SSH will now time out. In the portal, Boot diagnostics → Screenshot shows a maintenance prompt; the Serial log (or get-boot-log) shows Dependency failed for /mnt/ghost, Failed to mount, and You are in emergency mode / Give root password for maintenance. Note that you diagnosed a totally unreachable VM without any connection into the OS.

5. Fix it interactively via the Serial Console. In the portal open vm-bootlab → Serial console. Press Enter to get the maintenance prompt, supply the root context (the emergency shell), then repair /etc/fstab and reboot.

# In the Serial Console emergency shell:
journalctl -xb | tail -20            # confirms /mnt/ghost is the failed mount
# Remove the bad line (or add nofail). Easiest: delete the line you added:
sed -i '/\/mnt\/ghost/d' /etc/fstab
mount -a                             # should now succeed with no error
reboot

6. Confirm recovery. After the reboot, the boot screenshot returns to a login prompt and SSH works again. You have completed the full loop: capture → read the failing line → fix over serial → verify.

7. Tear down.

az group delete -n $RG --yes --no-wait

What each step proved:

Common mistakes & troubleshooting

This is the centrepiece. The table below is the playbook you keep open during an incident: a symptom, its likely root cause, the exact command or portal path that confirms it, and the fix. It spans the basic failures (disk full, bad fstab, wrong storage) and the deeper ones (missing drivers, BCD corruption, generalised-image panics). Read the prose once; keep the table open at 02:00.

Three reading notes that save the most time during an incident:

Two of these rows account for the majority of real incidents, so internalise their reflex. For bad /etc/fstab (rows 5–6) — the most common self-inflicted hang — the rule is nofail on every non-root mount; confirm the failed unit with journalctl -xb over the Serial Console, fix the line, mount -a to validate, and always reboot once to prove it boots clean unattended (an emergency shell that mounts manually can mask an fstab that still fails on a fresh boot). For kernel/initramfs after patching (rows 3–4, 11) the decisive test is to boot the previous kernel from GRUB’s “Advanced options” via the Serial Console: if the old kernel boots, the new kernel/initramfs is the culprit (reinstall it, regenerate the initramfs); if no kernel boots, the problem is below the kernel and you go offline.

Best practices

• Enable Boot Diagnostics on every VM at create time — by policy, with managed storage. The single most valuable habit in this topic: the historical serial log of a failing boot only exists if diagnostics was already on.
• Prefer managed storage over a custom account unless compliance forces otherwise; if you must use custom, make it a Standard GPv2 account in the same region as the VM, and never a Premium account (blank-screenshot trap).
• Always read the boot screenshot first when a VM is unreachable. Two seconds tells you whether it is a boot failure or a networking failure and stops the most common hour-long detour.
• Put nofail on every non-root mount in /etc/fstab. A detached or renamed data disk should never be able to stop the OS booting.
• Keep at least two kernels installed and never let a patch remove the running kernel; a known-good previous kernel in the GRUB menu is your fastest recovery from a bad update.
• Make patch pipelines complete-and-verify the initramfs/boot config before rebooting, and gate fleet waves on each VM rejoining the pool before proceeding.
• Cap the journal (SystemMaxUse) and move heavy writers to a data disk so the OS disk cannot fill; alert on OS-disk usage at ~85%.
• Keep console=ttyS0 on the Linux kernel command line (and EMS/SAC enabled on Windows) so the serial log and Serial Console actually work — verify this on every custom/generalised image.
• For risky in-guest changes (kernel, GRUB, fstab, drivers), reboot once to verify the change boots cleanly before you walk away or apply it fleet-wide.
• Repair a copy of the OS disk, not the original, when the stakes are high — snapshot first (or use az vm repair, which does this for you) so a botched repair never destroys the only good state.
• Write a one-page boot-failure runbook (screenshot → serial line → stage → fix) so on-call follows the method instead of guessing during an outage.

Security notes

• The serial log and screenshot can contain secrets. Boot output and a console screenshot may reveal hostnames, usernames, paths, and occasionally tokens printed at boot. Treat the boot diagnostics storage account as sensitive: prefer managed storage (Microsoft-managed access), and if custom, lock it behind network rules and least-privilege RBAC rather than shared keys.
• The Serial Console is a privileged backdoor into the OS — whoever opens it can reach the boot loader and, depending on configuration, a root/admin context without SSH/RDP. Gate it with RBAC (it requires contributor-level VM access plus access to the diagnostics storage), and note it can be disabled at the subscription level for true lockdown. Set a Linux root password only if you rely on maintenance mode — and weigh that against the interactive-console risk.
• Offline repair widens the blast radius briefly. Attaching a broken VM’s OS disk to a rescue VM lets that VM read every secret on the disk. Use a dedicated, short-lived rescue VM in a controlled subnet and delete it after; never attach a production OS disk to a shared utility VM.
• Boot Diagnostics does not bypass disk encryption. If the OS disk uses Azure Disk Encryption (BitLocker/dm-crypt), offline repair must unlock it on the rescue VM with the key from Key Vault — keep those keys and access policies intact, or recovery becomes impossible.
• Audit who enabled/disabled diagnostics and who opened the Serial Console via the activity log; an attacker who reaches a VM’s serial console has a powerful, low-visibility foothold.

Cost & sizing

The diagnostic tooling itself is essentially free; the costs that matter are the tiny storage footprint and the rescue-VM compute you spin up briefly during a repair.

Sizing guidance: Boot Diagnostics has no meaningful sizing decision — turn it on everywhere; the storage is a rounding error against the VM’s own cost. The only real spend is during an offline repair, and it is transient: a Standard_B2s rescue VM running for the thirty minutes of a repair costs a few rupees, and az vm repair restore (or a manual delete) tears down the rescue VM and its temporary disks so nothing lingers. The expensive mistake is forgetting a rescue VM running — set a reminder, or always finish with az vm repair restore, which cleans up for you. Free-tier note: an Azure free account’s monthly B1s credit hours comfortably cover the lab above if you delete the resource group promptly.

Interview & exam questions

1. The portal shows a VM as “Running” but SSH and RDP both time out. What does “Running” actually tell you, and what is your first diagnostic step? “Running” reflects the host/hypervisor power state — the VM is powered on — and says nothing about whether the guest OS finished booting. The first step is to open Boot Diagnostics → Screenshot (and read the serial log): if it shows a login prompt the issue is connectivity, otherwise it is a real boot failure and the last serial line names the stage. (Relevant to AZ-104.)

2. What two artefacts does Boot Diagnostics capture, and why do they work when the VM agent does not? A serial console log (the guest’s COM1 byte stream) and a boot screenshot (the console framebuffer). They are captured by the hypervisor reading the VM’s virtual hardware, not by any software inside the guest, so they keep working through kernel panics, dead networking, and a read-only disk — exactly when the in-guest agent is unavailable. (AZ-104.)

3. A Linux VM’s serial log ends with Kernel panic - not syncing: VFS: Unable to mount root fs. What are the two most likely causes and the fastest test? Either the initramfs is missing the storage driver needed to mount root (often after a patch or a generalised image lacking Hyper-V drivers) or the kernel itself is broken. The fastest test is to boot the previous kernel from the GRUB menu via the Serial Console; if it boots, the new kernel/initramfs is the culprit — regenerate the initramfs (update-initramfs/dracut) and reinstall the kernel. (AZ-104.)

4. When would you use the Serial Console instead of just reading the serial log? When you need to interact with the boot, not merely observe it: answer an fsck/maintenance prompt, pick a previous kernel at the GRUB menu, drop into an emergency shell to fix /etc/fstab, or — on Windows — use SAC to run bcdedit. The log is read-only history; the Serial Console is a live session to the same COM1 port and needs no guest networking. (AZ-104.)

5. A /etc/fstab entry for a data disk that isn’t attached has wedged the boot into emergency mode. What is the fix, and the one-word prevention? Via the Serial Console, supply the maintenance shell, remove or correct the bad line, run mount -a to validate, and reboot. The prevention is nofail on every non-root mount, so a missing/renamed disk degrades to “not mounted” instead of “no boot.” (AZ-104.)

6. Describe the offline repair workflow and when you reach for it. You reach for it when you cannot get a usable prompt (boot wedges before any shell, GRUB/BCD too broken, fstab stops the boot cold). Stop the VM, snapshot/copy the OS disk, attach the copy as a data disk to a healthy rescue VM, mount and fix it, then swap the repaired disk back as the OS disk. az vm repair create/run/restore automates the whole sequence against a copy. (AZ-104.)

7. The boot screenshot of a Windows VM shows INACCESSIBLE_BOOT_DEVICE (0x7B). Why is “just restart it” the wrong move? 0x7B means Windows cannot access its boot disk — typically a storage-driver or BCD problem — which is deterministic, so a restart reproduces the same blue screen. The fix is offline repair: attach the OS disk to a rescue Windows VM and repair the BCD (bootrec/bcdedit) or re-enable the correct storage driver. (AZ-104.)

8. Your Linux serial log is completely empty even though the VM appears to be booting. What’s wrong and how do you fix it? The guest isn’t configured to emit boot messages to the serial port — the kernel command line lacks console=ttyS0 (common on hand-built/custom images; Azure marketplace images include it). Add console=ttyS0 to the GRUB command line, run update-grub, and reboot so future serial logs are populated. (AZ-104.)

9. What is the difference between the boot screenshot and the serial log, and which is primary for Windows vs Linux? The screenshot is a single image of the console framebuffer (great for Windows stop codes, GRUB menus, login prompts); the serial log is a scrolling text transcript (great for Linux kernel/systemd messages). For Windows, the screenshot is primary (the stop code is the diagnosis); for Linux, the serial log is primary (the last lines name the failing stage). (AZ-104.)

10. A VM patched fine in eastus but kernel-panics after you copied its generalised image to centralindia. What’s the likely cause? The generalised image was built without the Azure/Hyper-V drivers (hv_storvsc, hv_netvsc) baked into the initramfs, so the kernel can’t mount root (or bring up networking) on the new VM. Rebuild the image with those drivers included; this is a classic image-hardening gap covered by the generalise-and-capture process. (AZ-104.)

11. How can the boot diagnostics data be a security concern, and how do you mitigate it? The serial log, an emergency shell, and the screenshot can expose hostnames, paths, usernames and occasionally secrets printed at boot, and the Serial Console is a privileged backdoor. Mitigate with managed diagnostics storage (or a locked-down custom account), RBAC on both the VM and the storage, optional subscription-level disabling of the Serial Console, and auditing who opens it. (AZ-500 / AZ-104.)

12. A repair requires attaching an encrypted OS disk to a rescue VM. What extra step is needed? If the disk uses Azure Disk Encryption, the rescue VM must unlock the volume using the encryption key from Key Vault before you can mount and repair it. The keys and their access policies must be intact for recovery to be possible — losing them means the disk cannot be read offline. (AZ-500.)

Quick check

1. The portal says a VM is Running but nothing answers. What single fact does “Running” guarantee, and what does it not guarantee?
2. Name the two artefacts Boot Diagnostics captures and the one reason both keep working when SSH and the VM agent are dead.
3. A Linux serial log shows Give root password for maintenance. Which boot stage failed, and what is the one-word /etc/fstab setting that prevents it for data disks?
4. Your Linux serial log is empty even though the VM is booting. What’s missing from the kernel command line?
5. When do you abandon the Serial Console and switch to an offline rescue-VM repair?

Answers

1. It guarantees the host/hypervisor has the VM powered on (the virtual power is on, the host is healthy). It does not guarantee the guest OS booted — a panicked, fsck-wedged, or healthy VM all report Running.
2. The serial console log (COM1 byte stream) and the boot screenshot (console framebuffer). Both work because the hypervisor captures the VM’s virtual hardware from the platform side, requiring nothing running inside the guest.
3. Early userspace — fsck/mount failed and dropped to a maintenance shell. The preventive setting is nofail on every non-root mount, so a missing/renamed data disk never blocks the boot.
4. console=ttyS0 — without it the guest doesn’t mirror boot messages to the serial port, so the serial log stays empty (common on custom images; marketplace images include it).
5. When you cannot get a usable prompt — the boot wedges before any shell, GRUB/BCD is too broken to interact with, or a bad /etc/fstab stops the boot before a console is offered. Then stop the VM, copy the OS disk, attach it to a rescue VM, fix it, and swap it back (az vm repair).

Glossary

• Boot Diagnostics — The Azure VM feature that captures the guest’s serial console log and a boot screenshot to a storage account, from the platform side, so you can see a boot that no in-guest tool can reach.
• Serial console log — The byte stream the guest writes to its first serial port (COM1 / ttyS0), mirroring boot messages; the primary diagnostic for Linux boot failures.
• Boot screenshot — A periodic snapshot of the VM’s console framebuffer (what a monitor would show); the primary diagnostic for Windows stop codes and for confirming a Linux login prompt.
• Serial Console — A live, interactive text session to the VM’s COM1 port (using the boot diagnostics storage), letting you fix a boot — answer fsck, pick a kernel, enter a rescue shell, run SAC — without any guest networking.
• GRUB — The Linux boot loader that presents the boot menu and loads the kernel and initramfs from /boot; a wrong/missing kernel entry stops the boot before the kernel runs.
• Kernel panic — An unrecoverable kernel error that halts the system; the most common Azure trigger is being unable to mount the root filesystem due to a missing driver in the initramfs.
• initramfs / initrd — A tiny in-memory root filesystem holding just enough drivers for the kernel to find and mount the real root disk; if it lacks the storage driver, the kernel panics with “unable to mount root fs.”
• /etc/fstab — The Linux filesystem mount table read at boot; a bad entry (wrong UUID, a missing disk without nofail) drops the boot into emergency mode.
• nofail — An /etc/fstab mount option that lets the boot continue even if the device is absent; mandatory on every non-root mount to prevent self-inflicted boot hangs.
• fsck — The Linux filesystem check/repair run at boot when a filesystem is marked dirty; without a console to answer its prompt, a VM can hang at it indefinitely.
• systemd — The Linux init system and service manager; a failed required mount or unit can stall the whole boot into emergency mode.
• Rescue VM — A healthy VM, in the same region, to which you attach an unbootable VM’s OS disk (as a data disk) to repair it offline.
• az vm repair — The CLI extension that automates offline repair: snapshot the OS disk, build a rescue VM with a copy attached, run a repair, then swap the fixed disk back.
• SAC (Special Administration Console) — The Windows emergency text console exposed over the serial port via EMS, used to run recovery commands (bcdedit, CMD) without RDP.
• BCD (Boot Configuration Data) — The Windows boot configuration store the Boot Manager reads; corruption produces errors like 0xc000000e and is repaired with bootrec/bcdedit, usually offline.
• INACCESSIBLE_BOOT_DEVICE (0x7B) — A Windows stop code meaning the OS cannot access its boot disk, typically a storage-driver or BCD mismatch; fixed by offline repair, not by restarting.

Next steps

• When the boot screenshot proves the OS did reach a login prompt, the problem is connectivity — work through Cannot RDP or SSH into Your Azure VM? A Beginner’s Connectivity Troubleshooting Checklist.
• If a stuck or failed VM agent/extension is stalling provisioning or boot, dig into VM Extension Provisioning Failed: Debugging Custom Script, DSC and Azure Monitor Agent Errors.
• Several boot failures are really disk failures — ground your disk knowledge with Azure VM Disk Types Demystified: Standard HDD, Standard/Premium SSD, Premium v2 and Ultra Disks.
• To stop a single failing boot from becoming downtime, design for resilience with Availability Sets vs Availability Zones: Choosing the Right Azure VM Resilience Model.
• To prevent the “missing driver after image copy” panic, bake the Azure drivers in correctly when you build images: Generalize and Capture: Building Reusable Custom Azure VM Images with Sysprep and waagent.