Troubleshooting Arc-Enabled Servers: Disconnected Agents, Missing Heartbeats & Extension Failures

The Arc portal blade says the worst single word in hybrid operations: Disconnected. The server itself is up — you are SSH’d into it right now — but Azure has stopped hearing from it, so every Arc capability silently degrades: Azure Policy compliance goes stale, Azure Monitor Agent stops shipping logs, Defender for Servers loses coverage, and any extension you queue just sits there. Azure Arc is the control plane that projects a non-Azure machine — an on-prem VM, a bare-metal box, an EC2 or GCE instance — into Azure Resource Manager as a first-class Microsoft.HybridCompute/machines resource, kept alive by a small local agent phoning home. When the call stops, the resource is still in the portal, but it is a ghost.

This is the diagnostic playbook for when that agent misbehaves. We treat the three classic failures — a Disconnected agent, a missing heartbeat (looks connected but status fields are stale), and extension failures (AMA, Custom Script, Defender stuck in Creating or Failed) — not as random gremlins but as a short list of concrete causes you confirm with specific commands. You read the local side (himds, the azcmagent CLI, the agent logs) and the cloud side (the resource’s status, the extension state, Resource Graph) to localise any failure to one layer.

By the end you will stop restarting the agent and hoping. When the blade says Disconnected you will know within minutes whether the himds service died, egress is blocked, the identity token failed to renew, the clock skewed, or onboarding was never completed. Knowing which is the difference between a one-command fix and a day of guessing — and it is almost never “delete the resource and re-onboard,” the expensive reflex this article exists to prevent.

What problem this solves

Arc’s promise — one inventory, policy engine, identity model and monitoring agent across clouds and on-prem — rests entirely on a slim local agent maintaining a continuous, authenticated connection. When it breaks the failure is uniquely frustrating: the machine is fine and serving traffic, but Azure has gone blind to it, and every governance and security control on top quietly stops working.

What breaks without this knowledge: an operator sees Disconnected, restarts himds, watches it flip to Connected for a few minutes, then go stale again — because the real cause (an expired service principal, a proxy dropping long-lived TLS, a firewall change that closed *.his.arc.azure.com) was never addressed. Or they disconnect and re-onboard, minting a new resource ID that orphans the old one, breaks every policy and RBAC grant scoped to it, and double-counts the machine in inventory and Defender billing. The machine looks fixed for a day; the governance debt lingers for months.

Who hits this: anyone running Arc at any scale. It bites hardest on proxy-mandated networks (the endpoint requirements are unforgiving), machines onboarded with a short-lived service principal secret that later expired, fleets where TLS interception mangles the agent’s calls, and anyone who layered AMA or Defender on Arc then wondered why a connected machine ships no data. The fix is almost never re-onboarding — it is finding the one layer that’s lying and making it tell the truth.

To frame the field, here is every symptom class this article covers, the question it forces, and where to look first:

Learning objectives

By the end of this article you can:

• Explain what the Connected Machine agent actually is (the himds service, the azcmagent CLI, the GC and extension manager) and what keeps a machine Connected versus letting it drift to Disconnected.
• Diagnose a Disconnected agent as a stopped local service, blocked Arc egress, an expired onboarding credential, or clock skew — and confirm which with azcmagent show, azcmagent check and the agent logs.
• Pinpoint a missing heartbeat even when the portal says Connected, by reading lastStatusChange, the agent’s send loop, and proxy behaviour.
• Recover a failed onboarding by isolating identity (service principal / RBAC) from network (endpoints / proxy) from “already connected” without minting a duplicate resource.
• Fix extension failures — Azure Monitor Agent with no data, Custom Script Extension non-zero exit, Defender, the Dependency agent — by reading the extension provisioningState, the per-extension log, and the data-collection-rule association.
• Verify and unblock the exact network endpoints, ports and proxy an Arc machine needs, and recognise the TLS-inspection failure mode that mangles agent calls.
• Use the system-assigned managed identity correctly from a script on an Arc machine, and confirm token acquisition via the local IMDS endpoint.

Prerequisites & where this fits

You should understand Arc at a control-plane level — that it projects a non-Azure machine into Azure as a Microsoft.HybridCompute/machines resource so policy, RBAC, monitoring and Defender apply like an Azure VM. If that’s new, read Azure Arc Explained: One Control Plane for Servers, Kubernetes & Data Across Any Cloud first. You should also have onboarded at least one machine; the bulk-enrolment mechanics and onboarding credential are covered in Onboarding Servers to Azure Arc: Connected Machine Agent, Service Principals & Bulk Enrollment, and we refer back to that credential as a root cause.

You should be comfortable running az in Cloud Shell and getting a root/Administrator shell on the machine to run azcmagent and inspect services and logs, with basic familiarity with TLS, proxies and DNS. Because so many failures here are really identity failures, a mental model of managed identities helps — see Managed Identities Demystified: System vs User-Assigned and When to Use Each.

This sits in the Hybrid & Troubleshooting track, upstream of every Arc capability: a machine that isn’t reliably Connected can’t be governed by Remediate Non-Compliant Resources: Remediation Tasks and Managed Identity Setup, monitored via No Logs Showing Up? Troubleshooting Empty Log Analytics Tables and Ingestion Gaps, or protected by Defender. Six layers can each cause a distinct failure, and knowing which tells you who to call: the machine OS (service down, clock skew), the agent (himds/extension manager), network/proxy (firewall, TLS inspection), Entra identity (SPN, system identity), the ARM resource (stale status, extension Failed), and extension config (DCR, CSE script).

Core concepts

Five mental models make every later diagnosis obvious.

Arc is a connection, not an install. Onboarding performs a connect that registers the machine in ARM, provisions a system-assigned managed identity in Entra ID, and stores a local certificate the agent uses to authenticate every subsequent call. The machine’s value is the live connection, maintained by the local himds daemon. When you see Disconnected, the resource still exists — but the connection that made it useful has lapsed.

The status is a heartbeat verdict, not a health check of your server. The agent sends a status roughly every 5 minutes; Azure marks it Connected while those arrive and flips to Disconnected once the machine stops reporting for long enough. “Disconnected” never means “the server is down” — it means “the connection lapsed,” and your app can be serving traffic happily on a machine Azure considers disconnected.

The agent is three cooperating parts. The himds daemon maintains the connection, renews the identity, and exposes a local IMDS token endpoint for on-machine code; the extension manager installs and reports on extensions; the azcmagent CLI is your control surface. A failure almost always lives in exactly one of these three, and knowing which narrows the fix instantly.

Egress is the lifeline, and it is specific. The agent needs only outbound 443 — no inbound — but to a defined set of endpoints (detailed in the network section). Block one, or have a proxy intercept TLS by re-signing the agent’s certificate-validated calls, and the connection fails in ways that look like the machine’s fault. This is the single most common root-cause class in real estates.

The system-assigned identity is the machine’s passport, and it expires. The agent renews the credential automatically — but if the renewal can’t reach Entra or the clock skews beyond the token’s tolerance (a few minutes), token validation fails, and anything needing a token — the agent’s own status calls, the AMA/Defender data plane — returns 401/403. A surprising share of “Disconnected” and “no data” incidents are really clock-skew or token-renewal failures in a network costume.

The agent status and error reference

Before the per-symptom anatomy, here is the lookup table you scan first: the status and error situations you realistically see from an Arc machine, what each means, how to confirm it, and the first fix. The non-obvious ones are Disconnected (heartbeat lapsed) versus Expired (the credential itself lapsed), and the proxy-induced failures that present as generic connection errors.

One reading note saves the most time: if azcmagent show says Connected but an extension is Failed or no logs arrive, the connection is fine — it’s an extension/DCR problem, so read the extension log, not the agent log.

Anatomy of a Disconnected agent

A Disconnected status means Azure hasn’t received a heartbeat within its window. Four distinct causes — scan the matrix, then read the matching detail:

Cause 1 — The local agent service stopped or crashed

Services stop: a reboot without auto-start, an OS patch that broke a dependency, a disk-full condition, or a manual stop someone forgot to undo. With himds down, no heartbeat goes out and Azure flips the machine to Disconnected — even though the server is otherwise healthy.

Confirm. Ask the agent first (azcmagent show reports status, version and last attempt), then the service:

# Linux — the agent self-report, then the service
azcmagent show
systemctl status himds
journalctl -u himds --since "1 hour ago" --no-pager

On Windows the service is “Azure Hybrid Instance Metadata Service” (himds): Get-Service himds | Format-List Status, StartType.

Fix. Start and enable the service so it survives reboots, then re-check (Set-Service himds -StartupType Automatic; Start-Service himds on Windows):

# Linux
sudo systemctl enable --now himds
azcmagent show --query 'status' -o tsv   # expect: Connected (allow a few minutes)

If it starts then dies, read journalctl -u himds / the Windows Event Log for the crash reason (commonly disk full, a corrupt config, or a failed dependency) rather than restarting in a loop. The agent is really three services — himds (connection/identity), the extension manager (GCArcService on Windows), and guest configuration — so a Disconnected symptom is himds, a stuck-Creating symptom the extension manager.

Cause 2 — Egress to the Arc endpoints is blocked

The agent needs outbound 443 to a specific set of hosts. A firewall change, a new proxy mandate, or an over-zealous security baseline closes one, and the heartbeat can’t get out — the machine online, the service running, but the call failing. This is the most common Disconnected cause, and the one most often misdiagnosed as “the agent is broken.”

Confirm. The agent’s built-in connectivity checker tests every required endpoint and names the one that fails:

# Tests DNS + TLS reachability to every required Arc endpoint for your region
azcmagent check
# Add the cloud/region if needed:
azcmagent check --location centralindia --cloud AzureCloud

A pass prints each URL as reachable; a failure names the URL and the reason (DNS, connection, or TLS). If ambiguous, corroborate from the agent log (sudo tail -n 100 /var/opt/azcmagent/log/himds.log).

Fix. Allow the endpoints (full list below); if the machine uses a proxy, tell the agent:

# Point the agent at an outbound proxy (persists in agent config)
azcmagent config set proxy.url "http://proxy.corp.local:3128"
# Exclude hosts that must NOT go via proxy (e.g. local IMDS, internal):
azcmagent config set proxy.bypass "Arc,WireServer"   # known bypass keywords

Read the azcmagent check failure by layer: DNS = blocked/split-horizon DNS (or a missing Private DNS zone); connection = the firewall blocking 443; TLS/certificate = the most damaging, TLS interception re-signing the agent’s calls (bypass inspection for the Arc endpoints, don’t trust the CA). If every endpoint is reachable yet still Disconnected, move to Cause 3 (expired) or Cause 4 (clock).

Cause 3 — The onboarding credential or machine identity expired

Arc tolerates going offline, but not forever. If a machine is off for a long stretch (the documented window is around 45 days), its credential lapses and the status becomes Expired — the agent can no longer authenticate with the stored certificate, so even with perfect egress it can’t reconnect itself.

Confirm. azcmagent show reports whether the credential is valid; the portal surfaces an Expired indicator. From Azure, the resource status confirms it:

az connectedmachine show -n srv-onprem-01 -g rg-arc-prod \
  --query "{status:status, lastStatusChange:lastStatusChange, agentVersion:agentVersion}" -o json

Fix. Re-run connect. Reconnecting the same machine to the same resource does not mint a duplicate — it re-establishes the identity for the existing resource (no disconnect first for an expired recovery):

# Re-establish identity for the SAME resource (service-principal onboarding)
azcmagent connect --resource-group rg-arc-prod --subscription-id <sub-id> \
  --location centralindia --service-principal-id <appId> --service-principal-secret <secret>

This is the one place reconnecting is correct — bindings are preserved, unlike the wrong reflex (disconnect + re-onboard a merely-Disconnected machine) which orphans the resource. The distinction in three: a lapsed heartbeat with a valid credential is Disconnected (it reconnects itself once you fix service/egress/clock); an expired credential needs azcmagent connect (same resource ID); a decommissioned machine is azcmagent disconnect’d and deleted.

Cause 4 — Clock skew breaks token validation

Entra tokens are time-bound; if the machine’s clock drifts beyond the allowed tolerance (a few minutes), the agent’s tokens are rejected as not-yet-valid or expired and authenticated calls — including the heartbeat — fail with 401/403. The machine looks network-healthy (azcmagent check passes) but won’t connect, which is maddening until you check the clock. VMs resumed from a long pause/snapshot and machines with broken NTP are the usual victims.

Confirm and fix. Check the offset, then restore sync — once the clock is correct, validation succeeds and the agent reconnects on its next cycle (Windows: w32tm /query /status, then w32tm /resync /force):

# Linux — check the offset, then enable NTP and step the clock immediately
timedatectl status
chronyc tracking 2>/dev/null || ntpq -p 2>/dev/null
sudo systemctl enable --now chronyd && sudo chronyc makestep

A quick probe ties clock to identity: ask the local IMDS endpoint for a token. A 401 asking for the challenge token is expected (a security guard — see Security notes); a timeout means himds is down or blocked; a nbf/time/expiry error is clock skew; a 403 after the challenge is RBAC on the target, not Arc.

# A 401 asking for the challenge token here is EXPECTED and healthy.
# A timeout, or a time/nbf error in the body, points at the real problem.
curl -s -D - -H "Metadata:true" \
  "http://localhost:40342/metadata/identity/oauth2/token?api-version=2020-06-01&resource=https://management.azure.com/" \
  -o /dev/null

Anatomy of a missing heartbeat

Sometimes the portal says Connected but lastStatusChange is hours old and a Heartbeat query shows the machine fell silent and came back — the agent running but failing to send reliably. Three causes share one spine (gaps in the Heartbeat table) but diverge on what the log shows around the gap:

Confirm. The Heartbeat table records each agent check-in, so a gap is unambiguous:

// Gaps in Arc agent heartbeats over the last day (a healthy machine has steady beats)
Heartbeat
| where TimeGenerated > ago(1d)
| where Category == "Azure Monitor Agent" or ResourceType == "machines"
| summarize beats = count() by bin(TimeGenerated, 5m), Computer
| where beats == 0
| order by TimeGenerated desc

Then read the agent log around those timestamps to tell them apart:

# Connection resets/timeouts → Cause 1; 401/token → Cause 2; silence → Cause 3 (check host CPU/IO)
sudo grep -iE "reset|timeout|proxy|401|unauthorized|token" /var/opt/azcmagent/log/himds.log | tail -n 40
chronyc tracking   # 'System time' offset should be tiny and stable; jitter points at Cause 2

Fix per the matrix: align the proxy idle timeout (and make any TLS-inspection bypass consistent — partial inspection produces exactly this flicker), pick one authoritative time source, or relieve a starved host. The agent is innocent in all three; it just needs to reach Azure on schedule.

Anatomy of a failed onboarding

When azcmagent connect fails, the machine never becomes a healthy resource. Three cause families — identity, network, state — scan then read the detail:

Cause 1 — The onboarding principal lacks the right role

Bulk onboarding uses a service principal that must hold Azure Connected Machine Onboarding on the target scope; without it, connect fails with an authorization error before any resource exists.

Confirm. The connect output names the failure; verify the assignments:

# What roles does the onboarding SPN actually hold at the RG?
az role assignment list \
  --assignee <appId> \
  --resource-group rg-arc-prod \
  --query "[].{role:roleDefinitionName, scope:scope}" -o table

Fix. Grant the onboarding role at the resource group (least privilege — not subscription Owner):

az role assignment create \
  --assignee <appId> \
  --role "Azure Connected Machine Onboarding" \
  --scope /subscriptions/<sub-id>/resourceGroups/rg-arc-prod

There are two Arc roles, scoped to the RG: Azure Connected Machine Onboarding (create/read machines, install the agent) is all the onboarding SPN needs — not Owner — while Azure Connected Machine Resource Administrator (manage machines and extensions) is for ongoing fleet management.

Cause 2 — The service principal secret is wrong or expired

The most common bulk-onboarding failure isn’t permissions — it’s an expired secret. The SPN was created with a 1- or 2-year secret, the script still carries the old value, and connect fails with no resource created. The tell is that it hits en masse — every machine in a wave fails identically. Confirm the validity window, rotate the secret, update the onboarding store, and re-run:

# When does the SPN secret expire? A past end date is the cause.
az ad app credential list --id <appId> --query "[].{end:endDateTime}" -o table

A short-lived secret baked into a script is a time bomb — treat it as a vaulted, rotated credential. (For bulk-enrolment mechanics, see Onboarding Servers to Azure Arc: Connected Machine Agent, Service Principals & Bulk Enrollment.)

Cause 3 — Endpoints blocked, or a stale resource already exists

The last two are quick to rule out. Blocked egress at connect time gives a connection error (often AZCM0061) naming the unreachable host — onboarding has the same endpoint requirements as steady state, so run azcmagent check before connect, and pass --proxy-url if you onboard through a proxy:

azcmagent check --location centralindia --cloud AzureCloud   # pre-flight before connect

A Resource already exists error means a resource with that identity is already there — how teams accidentally create duplicates by re-onboarding instead of reconnecting. Check with az connectedmachine show -n <name> -g <rg>: if it’s the same machine, fix the connection in place (or reconnect for an expired credential); only if it’s a stale leftover from a decommissioned machine do you azcmagent disconnect (or az connectedmachine delete) first and onboard cleanly.

Anatomy of an extension failure

Once a machine is reliably Connected, the next pain is extensions — AMA, Custom Script, Defender, the Dependency agent. The connection is fine; the extension won’t go healthy. Cardinal rule: if azcmagent show says Connected, read the extension log, not the agent log. Five causes — scan, then detail:

Cause 1 — Azure Monitor Agent installed but shipping no data

The number-one extension confusion: AMA installs cleanly, shows Succeeded, the agent is Connected — yet Heartbeat, Syslog and Perf are empty. AMA collects nothing until a Data Collection Rule (DCR) is associated. Unlike the legacy Log Analytics agent, it’s inert without a DCR telling it what to collect and where.

Confirm. The extension shows Succeeded — so check the DCR association, almost always the missing piece:

# Are any DCRs associated with this machine? (empty = the smoking gun)
az monitor data-collection rule association list \
  --resource "$(az connectedmachine show -n srv-onprem-01 -g rg-arc-prod --query id -o tsv)" -o table

An empty list is the smoking gun; an empty Heartbeat table confirms it.

Fix. Associate a DCR (create one targeting your workspace and data sources, then associate it):

az monitor data-collection rule association create \
  --name "dcra-srv-onprem-01" \
  --resource "$(az connectedmachine show -n srv-onprem-01 -g rg-arc-prod --query id -o tsv)" \
  --rule-id "$(az monitor data-collection rule show -n dcr-linux-syslog -g rg-arc-prod --query id -o tsv)"

As a short decision: no DCR = inert, associate one; DCR but no Heartbeat = downstream (workspace/egress/data-plane); only some tables = the DCR’s sources are too narrow; Failed = AMA didn’t install, read its log. (For the full empty-tables tree, see No Logs Showing Up? Troubleshooting Empty Log Analytics Tables and Ingestion Gaps.)

Cause 2 — Custom Script Extension exits non-zero

CSE runs your script; a non-zero exit makes the extension Failed, the message being whatever the script wrote to stderr plus the exit code. People stare at the extension state when the answer is in the script’s output log.

Confirm. Read the status from Azure, then the script’s stdout/stderr on the machine:

az connectedmachine extension show \
  --machine-name srv-onprem-01 -g rg-arc-prod --name customScript \
  --query "{state:provisioningState, status:instanceView.status}" -o json

The on-machine CSE output (Linux) lives under the extension’s directory:

sudo find /var/lib/GuestConfig /var/lib/waagent -iname "*stdout*" -o -iname "*stderr*" 2>/dev/null

Fix. Make the script exit 0 and idempotent (CSE may re-apply; a mkdir without -p that fails on re-run creates phantom failures), then re-push. By fingerprint: a first-run failure is a script bug; a re-run-only failure is non-idempotency; a timeout is a hanging script; a download error is an expired blob SAS or blocked storage egress.

Cause 3 — Stuck Creating, wrong settings, or an unhealthy Defender extension

An extension stuck in Creating/Transitioning means the extension manager can’t complete — usually the agent is unhealthy, or egress to *.guestconfiguration.azure.com is blocked while the agent’s own endpoint is open. Verify agent health first (a stuck extension on a Disconnected agent is downstream), confirm the guestconfig URL with azcmagent check, then delete and re-push:

azcmagent show --query 'status' -o tsv        # must be Connected first
azcmagent check                               # confirm the guestconfiguration URL is reachable
az connectedmachine extension delete --machine-name srv-onprem-01 -g rg-arc-prod --name <ext> --yes

Wrong settings or an unsupported version fail immediately on push, the message quoting the offending field — correct to the documented schema and pin a supported version (settings are reviewed Bicep). And an unhealthy Defender extension is a security gap behind a green agent: it’s auto-provisioned when Defender for Servers is enabled, so if the plan or auto-provisioning is off, a Connected machine has no protection — enable both at the subscription. The extensions you’ll most often manage, and what each needs:

The network and endpoint requirements

Half of all Arc agent incidents are an egress problem, so keep this reference open. The agent needs only outbound TCP 443 — no inbound — to these hosts (region-specific subdomains apply; azcmagent check validates your exact set):

Three hard truths the table can’t convey. Private endpoints change the URL set: with an Azure Arc Private Link Scope the agent talks to private endpoints, and a missing Private DNS zone shows up as DNS failures in azcmagent check (see Diagnosing Connectivity with Network Watcher: Connection Monitor, Connection Troubleshoot and Next Hop). TLS inspection is the silent killer — exclude the Arc endpoints rather than trusting the inspection CA. And tell the proxy to the agent: azcmagent config set proxy.url, proxy.bypass "Arc,WireServer" so IMDS/local traffic skips it, and --proxy-url on azcmagent connect.

The diagnostic toolkit: exact commands

Knowing where to look is half the battle. The tools matrix:

Two earn extra emphasis: azcmagent show (Connected locally but Disconnected in the portal = a send problem, ruling out half the causes) and azcmagent check (resolves most incidents on its own by naming the blocked URL and distinguishing DNS from TCP from TLS). For every Disconnected machine across subscriptions at once, Azure Resource Graph is the fleet view:

// All Arc machines and their connection status, newest status-change first
// (run in the portal's Resource Graph Explorer, or via `az graph query -q "..."`)
resources
| where type =~ "microsoft.hybridcompute/machines"
| extend status = tostring(properties.status), lastChange = todatetime(properties.lastStatusChange)
| project name, resourceGroup, status, agentVersion = tostring(properties.agentVersion), lastChange
| order by lastChange desc

Architecture at a glance

Hold this mental model and every diagnosis falls into place. Inside the machine — anywhere — the agent’s core himds daemon loops three things: sends a heartbeat every few minutes, renews the system-assigned identity against Entra ID, and exposes a local IMDS token endpoint (localhost:40342) for on-box code; alongside it the extension manager runs extensions. From the machine, only outbound 443 leaves — no inbound — fanning to the Hybrid Identity Service (*.his.arc.azure.com) for heartbeat and identity, Entra for tokens, ARM for the resource status, and guest configuration (*.guestconfiguration.azure.com) for extensions. Those heartbeats keep the Microsoft.HybridCompute/machines resource Connected; the moment they stop, ARM flips it to Disconnected — regardless of your server’s health.

That picture localises every failure to one hop: a dead himds = no heartbeat leaves the box; a firewall on the 443 fan-out = it can’t reach Azure; a skewed clock or blocked Entra = tokens rejected so even reachable calls fail; a blocked guestconfiguration hop = the connection is fine but extensions hang; an inert extension (AMA without a DCR) = both green while no data flows. Read the symptom, name the hop, run the one command that interrogates it.

Real-world scenario

Northwind Logistics runs a hybrid estate: 240 servers across two on-prem data centres and a VMware cluster, all onboarded to Azure Arc so a team of three can apply Remediate Non-Compliant Resources: Remediation Tasks and Managed Identity Setup, ship logs via AMA, and get Defender coverage. Onboarding was done eighteen months earlier with a service principal whose secret expires in two years. Monthly Arc-adjacent spend (Monitor + Defender) ran about ₹140,000.

The incident surfaced on a Monday: the dashboard showed 38 machines Disconnected — all in the same data centre, all gone within the same two-hour weekend window. The on-call engineer restarted himds on one; it flipped to Connected for about four minutes, then went straight back. Second reflex: disconnect and re-onboard it — which worked, but minted a brand-new resource, orphaning the old one and breaking its three policy assignments. Across 38 machines, that path would have created 38 governance messes.

The breakthrough came from the right first question: local service, or egress? azcmagent show said Connected locally — service fine — yet Azure said Disconnected, ruling out Cause 1. azcmagent check was decisive: it failed on *.his.arc.azure.com and login.microsoftonline.com with a TLS/certificate error while DNS resolved fine — the fingerprint of TLS inspection. The network team had rolled out SSL inspection on the proxy for “all outbound HTTPS” that weekend, silently re-signing the agent’s calls. The restart “fixed” it for four minutes only because the cached token outlived the next failed renewal.

The fix had two parts. Immediately: a bypass rule excluding the Arc endpoints from inspection, after which machines flipped back to Connected within one heartbeat cycle — no re-onboarding, all bindings intact — and the one wrongly re-onboarded machine was reverted (delete the duplicate, reconnect to the original). The following week: they rotated the onboarding secret (four months from its own expiry) into Key Vault and added a Resource Graph alert firing when any machine stays Disconnected over 15 minutes, so the next change is caught in minutes, not on a Monday dashboard.

The lesson on the wall: “Disconnected is a network verdict until the local agent proves otherwise — azcmagent check before you touch the service, and never re-onboard a machine you can reconnect.” The incident as a timeline, because the order is the lesson:

Advantages and disadvantages

The agent-maintains-a-connection model both causes this class of problem and makes it diagnosable. Weigh it honestly:

The model is right for any organisation wanting Azure governance over machines it can’t move — hybrid, multicloud, edge. It bites hardest on tightly-controlled networks, fleets onboarded with short-lived secrets, and teams that bolt on AMA/Defender without wiring the DCR or plan. Every disadvantage is manageable — but only if you know it exists.

Hands-on lab

Onboard one machine, watch a healthy connection, break it the way real estates do (block egress), confirm with azcmagent check, and fix it. We use a free-tier-friendly Azure VM as the “non-Azure” machine (Arc happily manages an Azure VM for labs), so no external hardware is needed. Run az in Cloud Shell and azcmagent on the VM.

Step 1 — A resource group and a small Linux VM to act as the machine.

RG=rg-arc-lab
LOC=centralindia
VM=vm-arc-lab
az group create -n $RG -l $LOC -o table
az vm create -n $VM -g $RG --image Ubuntu2204 --size Standard_B1s \
  --admin-username azureuser --generate-ssh-keys -o table

Step 2 — Create the onboarding service principal with least privilege.

SUB=$(az account show --query id -o tsv)
az ad sp create-for-rbac --name "sp-arc-lab-onboard" \
  --role "Azure Connected Machine Onboarding" \
  --scopes /subscriptions/$SUB/resourceGroups/$RG -o json
# Note the appId and password from the output for Step 4.

Step 3 — Install the agent on the VM. SSH in, then fetch and run the install script:

# On the VM (over SSH)
wget https://aka.ms/azcmagent -O install_linux_azcmagent.sh
sudo bash install_linux_azcmagent.sh
azcmagent version

Expected: the agent installs and prints a version; himds is present but not yet connected.

Step 4 — Pre-flight, then connect. Always check before connecting:

# On the VM
azcmagent check --location centralindia --cloud AzureCloud   # expect all endpoints reachable
sudo azcmagent connect \
  --resource-group rg-arc-lab --subscription-id <SUB> --location centralindia \
  --service-principal-id <appId> --service-principal-secret <password> \
  --tenant-id "$(az account show --query tenantId -o tsv)"

Expected: Connected to Azure. Confirm both sides:

azcmagent show --query 'status' -o tsv          # on the VM → Connected
# In Cloud Shell:
az connectedmachine show -n $VM -g $RG --query status -o tsv   # → Connected

Step 5 — Break it: block egress. Add an NSG rule denying outbound 443 to simulate a firewall closing the Arc endpoints:

# Cloud Shell — deny outbound HTTPS on the VM's NIC subnet (simulate the firewall change)
NSG=$(az network nsg list -g $RG --query "[0].name" -o tsv)
az network nsg rule create -g $RG --nsg-name $NSG -n deny-https-out \
  --priority 100 --direction Outbound --access Deny --protocol Tcp \
  --destination-port-ranges 443 --destination-address-prefixes Internet

Wait several minutes, then observe: azcmagent check now fails on the Arc endpoints and the machine drifts to Disconnected — Cause 2 reproduced exactly, the service running but egress gone.

azcmagent show --query 'status' -o tsv   # still Connected locally for a bit, then the heartbeat fails
azcmagent check                          # endpoints now unreachable — the smoking gun

Step 6 — Restore egress. Remove the deny rule; the agent reconnects on its own — no re-onboarding:

az network nsg rule delete -g $RG --nsg-name $NSG -n deny-https-out
# Wait one heartbeat cycle, then:
az connectedmachine show -n $VM -g $RG --query status -o tsv   # → Connected again

Validation checklist. You onboarded with a least-privilege SPN, confirmed the connection from both sides, reproduced a Disconnected purely from blocked egress, identified it with azcmagent check, and fixed it by restoring the network — the agent self-recovering with resource and identity preserved. No re-onboarding. The steps mapped to what each proves:

Cleanup (avoid lingering charges).

az group delete -n $RG --yes --no-wait
az ad sp delete --id <appId>

Cost note. A Standard_B1s VM is a few rupees per hour; this lab is well under ₹50, and deleting the resource group stops everything. (Arc itself is free for the connected-machine resource — this lab enables no billed services.)

Common mistakes & troubleshooting

This is the playbook — the part you bookmark. Every row maps a symptom to its root cause, the way to confirm it, and the fix; the most-misdiagnosed entries are detailed in the Anatomy sections above.

When in doubt the order is the same: azcmagent show to split send from service, then azcmagent check to name the blocked hop, then the resource/extension state in Azure.

Best practices

• Run azcmagent check before you touch anything else. On any Disconnected or onboarding incident, the connectivity pre-flight tells you in seconds whether it’s a network problem — most are — and saves you from restarting a service that was never the issue.
• Never re-onboard a machine you can reconnect. Disconnect + re-onboard mints a new resource and orphans every policy/RBAC grant on the old one. For an expired credential, azcmagent connect reuses the same resource — always prefer it.
• Bypass Arc endpoints from TLS/SSL inspection. An inspecting proxy that re-signs the agent’s certificate-validated calls breaks Arc silently. Exclude the Arc URLs rather than trusting the interception CA.
• Keep one authoritative time source. Clock skew breaks tokens and masquerades as a network failure. Pick hypervisor time or NTP — not both fighting.
• Treat the onboarding secret as a vaulted, rotated credential. A short-lived SPN secret baked into a script is a time bomb that takes out an entire wave when it expires.
• Grant least privilege for onboarding. Azure Connected Machine Onboarding at the resource group is enough; don’t hand the SPN Owner or subscription scope.
• Associate a DCR the moment you deploy AMA. AMA is inert without one. Deploy the agent and its DCR association together (Bicep/Policy) so you never have a “healthy agent, no data” gap.
• Manage extensions and settings as code. A typo’d extension setting is the same boot-time landmine as a bad app setting; keep settings in Bicep, reviewed, with auto-upgrade enabled.
• Make CSE scripts idempotent and exit-0. CSE may re-apply; a script that fails on a second run creates phantom failures. Every step must be safe to re-run and signal via the exit code.
• Alert on Disconnected, don’t discover it on a dashboard. A Resource Graph / Activity Log alert beyond a threshold turns a Monday surprise into a minutes-old page.
• Keep agents current. A stale version causes odd failures and misses fixes; enable automatic agent upgrade or patch on a schedule.
• Use least-privilege RBAC for the machine identity. Grant the system-assigned identity only the role it needs on the specific target — not a broad role at the subscription.

The signals worth alerting on before the next incident — the leading indicators, not the lagging “machine down”:

Security notes

• Outbound-only, no inbound. The agent needs only outbound 443 and no inbound ports — anything suggesting otherwise is misconfiguration. One of Arc’s strongest security properties.
• Scope the system-assigned identity tightly. Code on the machine can mint tokens for the machine’s identity via the local IMDS endpoint, so grant that identity least privilege on only the resources it must reach — a broad grant turns one compromised box into broad Azure access.
• Protect the local token endpoint. The Arc IMDS endpoint (localhost:40342) requires a challenge file readable only by privileged users before issuing a token — a deliberate guard so no local process can silently grab the machine’s identity. Don’t weaken those permissions.
• TLS everywhere, but don’t inspect Arc traffic. The agent’s calls are certificate-validated; exclude Arc endpoints from inspection rather than disabling TLS or trusting an interception CA.
• Least privilege for onboarding and management. Azure Connected Machine Onboarding to onboard, Azure Connected Machine Resource Administrator to manage — scoped to the resource group, never Owner.
• Rotate and vault the onboarding credential. The SPN secret can register machines into your tenant — keep it in Key Vault, rotate it, never in plaintext in a script or repo.
• Use Defender for Servers as a posture, not a checkbox. Enable it and auto-provisioning at the subscription; a Connected machine without the Defender extension is an unmonitored host hiding behind a green agent.
• Private Link for sensitive estates. An Azure Arc Private Link Scope keeps agent traffic on private endpoints — at the cost of correct Private DNS, a new failure surface to monitor.

Cost & sizing

The bill drivers. The Arc connected-machine resource is free — resource, agent, identity and inventory cost nothing; you pay for the capabilities you layer on top, where the spend (and the troubleshooting motivation) lives. Azure Monitor ingestion is usually the largest line, billed per GB; a too-broad DCR balloons it, so scoping to what you query also cuts the noise during incidents. And Defender for Servers is per-server, per-month, with a cost trap in duplicates — a machine re-onboarded as a second resource is double-counted and pays twice, another reason never to re-onboard a reconnectable machine.

A rough monthly picture for a 100-server estate: Arc resources free; Defender (Plan 2) at a few hundred rupees per server adds up; Monitor ingestion depends entirely on DCR scope. The cost drivers and what each buys you:

The sizing lesson mirrors the troubleshooting one: the platform is cheap; the spend (and the pain) comes from the layered services — fix the cause (a duplicate resource, a sloppy DCR) rather than paying around it.

Interview & exam questions

1. The Arc portal shows Disconnected but you’re SSH’d into the machine and it’s healthy. What does Disconnected mean, and your first command? It means Azure hasn’t received the agent’s heartbeat within its window — a connection verdict, not server health. Run azcmagent show first; if it’s locally Connected you have a send problem (egress/identity), and azcmagent check is next. Don’t restart the service blindly.

2. Difference between a Disconnected and an Expired Arc machine, and why it matters for the fix? Disconnected = a recent heartbeat gap, credential still valid, usually fixable in place (the agent reconnects itself). Expired = offline ~45 days so the credential lapsed; you must run azcmagent connect again, which reuses the same resource. It matters because the wrong fix (disconnect + re-onboard) mints a duplicate and orphans policy/RBAC.

3. A whole data centre goes Disconnected after a network change and azcmagent check shows TLS/certificate errors. What happened? A proxy/NGFW started TLS/SSL inspection, re-signing outbound HTTPS and breaking the agent’s certificate-validated calls — DNS resolves and TCP opens, but the handshake is tampered. Fix by excluding the Arc endpoints from inspection, not by trusting the interception CA.

4. What network access does the agent require, and any inbound? Only outbound TCP 443 to a defined set — the Hybrid Identity Service (*.his.arc.azure.com), guest configuration (*.guestconfiguration.azure.com), Entra (login.microsoftonline.com), ARM (management.azure.com), plus a few supporting hosts. No inbound at all. azcmagent check validates the exact set for your region.

5. AMA shows Succeeded on an Arc machine but no logs arrive. Why? AMA is inert without a Data Collection Rule — unlike the legacy agent, it collects nothing until a DCR is associated. Confirm with az monitor data-collection rule association list --resource <machine-id> (empty = the cause) and the empty Heartbeat table, then associate a DCR.

6. Onboarding fails with AuthorizationFailed. Cause and least-privilege fix? The onboarding SPN lacks the Azure Connected Machine Onboarding role at the target scope. Confirm with az role assignment list --assignee <appId> and grant that role at the resource group — not Owner, not subscription-wide.

7. An entire onboarding wave fails with the same auth error. What do you check first? The service principal secret has expired — the tell is that every machine fails identically at once. Check az ad app credential list --id <appId> for a past end date, rotate it, update the onboarding store, and re-run. Treat the secret as vaulted and rotated.

8. A machine passes azcmagent check but still won’t connect, with 401s in the log. Cause? Clock skew beyond Entra’s token tolerance — tokens are rejected as not-yet-valid or expired, failing the heartbeat and any authenticated call. Confirm with timedatectl / w32tm /query /status; fix NTP, step the clock, and ensure a single time source.

9. A Custom Script Extension shows Failed. Where do you look, and the two common script-side causes? Read the extension status (az connectedmachine extension show ... instanceView.status) for the exit code, then the script’s stdout/stderr on the machine. The two causes are a non-zero exit (script bug/missing dependency) and non-idempotency (fails on re-apply). Fix it to exit 0 and be safe to re-run.

10. Why is re-onboarding a Disconnected machine usually wrong? Disconnect + re-onboard creates a new resource with a new ID, orphaning the old one — breaking every policy assignment and RBAC grant scoped to it, double-counting inventory, and double-billing Defender. Fix a Disconnected machine in place; reconnect an Expired one (reusing the resource). Re-onboarding is only for genuinely new or replaced machines.

11. An extension is stuck in Creating and never reaches Succeeded. Your sequence? Confirm the agent is healthy first (azcmagent show = Connected) — a stuck extension on a Disconnected agent is downstream. Then confirm egress to *.guestconfiguration.azure.com with azcmagent check, since the extension manager needs it. Then delete and re-push.

12. How does code on an Arc machine get an Azure token, and what guards that endpoint? It calls the local IMDS endpoint at http://localhost:40342 for the machine’s system-assigned identity. The endpoint requires a challenge file readable only by privileged users before issuing a token, so a random local process can’t silently impersonate the machine. The identity must hold the right RBAC on the target it calls.

These map to AZ-800 / AZ-801 (Windows Server Hybrid Administrator) — manage hybrid servers with Azure Arc, onboarding, agents and extensions — and AZ-104 (Administrator) — Arc-enabled servers, monitoring and Defender coverage. The identity and network angles touch AZ-500 (Security Engineer). A compact cert-mapping for revision:

Quick check

1. The portal says a machine is Disconnected but you’re logged into it and it’s healthy. What does Disconnected mean, and what’s the single first command you run on the machine?
2. azcmagent check reports DNS resolves and TCP connects, but the TLS handshake fails on *.his.arc.azure.com. What is almost certainly happening, and what’s the fix?
3. True or false: when a Disconnected machine won’t recover, the correct fix is to disconnect it and onboard it again.
4. An Azure Monitor Agent extension on an Arc machine shows Succeeded, yet Heartbeat is empty for that machine. Name the most likely cause and the one command that confirms it.
5. Your onboarding pipeline suddenly fails for every machine in a wave with an authentication error. What’s the most probable cause, and where do you check?

Answers

1. Disconnected means Azure hasn’t received the agent’s heartbeat within its window — a connection verdict, not server health. The first command is azcmagent show on the machine: if it reports Connected locally, the service is fine and you have a send (egress/identity) problem; run azcmagent check next.
2. A proxy/NGFW is doing TLS/SSL inspection, re-signing the agent’s certificate-validated calls and breaking the handshake (DNS and TCP are fine; only TLS is tampered). The fix is to exclude the Arc endpoints from inspection — not to trust the interception CA.
3. False. Disconnect + re-onboard mints a new resource and orphans the old one’s policy/RBAC bindings (and can double-bill Defender). For a Disconnected machine you fix the connection in place; only an Expired machine needs azcmagent connect, which reuses the same resource.
4. No Data Collection Rule (DCR) is associated — AMA collects nothing without one. Confirm with az monitor data-collection rule association list --resource <machine-id> (an empty result is the smoking gun). Associate a DCR to fix it.
5. The service principal’s secret has expired — the tell is that every machine fails identically at once. Check az ad app credential list --id <appId> for a past end date; rotate the secret, update the onboarding store, and re-run.

Glossary

• Azure Arc — the control plane that projects non-Azure machines (and Kubernetes, data services) into Azure Resource Manager so policy, RBAC, monitoring and Defender apply to them.
• himds (Hybrid Instance Metadata Service) — the core agent daemon: maintains the connection and heartbeat, renews the identity, and exposes the local IMDS token endpoint.
• azcmagent — the local CLI for the agent: connect, disconnect, show, check, logs, config.
• Connected / Disconnected — Azure’s verdict on whether it has recently received the machine’s heartbeat; Disconnected means the connection lapsed, not that the server is down.
• Expired — the state when a machine’s credential has lapsed (offline beyond ~45 days); recovered with azcmagent connect, reusing the same resource.
• Heartbeat / status report — the periodic call (~5 minutes) the agent makes to keep the machine marked Connected; its absence drives Disconnected.
• System-assigned managed identity — the Entra service principal minted for the machine at onboarding; lets code on the box get Azure tokens without storing credentials.
• Local IMDS endpoint — http://localhost:40342, where on-machine code requests tokens for the machine’s identity; guarded by a challenge file.
• Data Collection Rule (DCR) — the configuration that tells Azure Monitor Agent what to collect and where to send it; AMA is inert until a DCR is associated.
• Azure Connected Machine Onboarding — the least-privilege Entra role that lets a service principal create Arc machine resources during onboarding.
• Arc endpoints — the outbound-443 hosts the agent must reach (*.his.arc.azure.com, *.guestconfiguration.azure.com, login.microsoftonline.com, management.azure.com, and supporting hosts).
• TLS inspection — a proxy/NGFW decrypting and re-signing HTTPS; breaks the agent’s certificate-validated calls and is the most common fleet-wide Disconnect cause.
• Azure Arc Private Link Scope — routes agent traffic over private endpoints instead of the public internet, requiring correct Private DNS.

Next steps

You can now localise any Arc agent failure to a layer — service, egress, identity, or extension — and fix it without minting a duplicate. Build outward:

• Next: Onboarding Servers to Azure Arc: Connected Machine Agent, Service Principals & Bulk Enrollment — get the onboarding credential and bulk-enrolment mechanics right so half these failures never start.
• Related: Azure Arc Explained: One Control Plane for Servers, Kubernetes & Data Across Any Cloud — the control-plane model this whole article assumes.
• Related: Arc-Enabled SQL Server Gone Wrong: Fixing Failed Onboarding, Billing Tags & Missing Inventory — the same agent, the SQL extension, and its billing/inventory failure modes.
• Related: No Logs Showing Up? Troubleshooting Empty Log Analytics Tables and Ingestion Gaps — the workspace-side decision tree when AMA is healthy but tables are empty.
• Related: VM Extension Provisioning Failed: Debugging Custom Script, DSC and Azure Monitor Agent Errors — the sibling extension-failure playbook for Azure VMs.