Patching Failover Clusters with Cluster-Aware Updating and Stretch Clusters via Storage Replica

Patching a failover cluster by hand is how you turn a routine Tuesday into a quorum-loss incident. You drain a node, suspend it, patch it, reboot, resume, wait for storage to resync, and only then move to the next one — and somewhere around node nine at 03:00 you fat-finger the order, reboot a second node in the same site before the first finished resyncing, and the cluster drops below majority. Everything hosting on it goes dark. Cluster-Aware Updating (CAU) exists precisely so a human never has to remember that dance. Get CAU right and a 16-node cluster patches itself overnight with every clustered role staying online, respecting quorum and storage resync at every step. Get it wrong — or skip it — and you are one tired operator away from a self-inflicted outage.

The second half of this article extends that same cluster across two datacenters. A Storage Replica stretch cluster keeps a synchronous, block-level copy of your volumes in a second site, so a metro failure costs you a failover — seconds — not a restore — hours. But a stretch cluster is a different animal from a single-site cluster: quorum math that can split-brain, site-awareness that decides where roles land, a replication log that stalls your entire application if you under-provision it, and a failover that must flip replication direction cleanly under pressure. Every one of those has a correct answer and a dozen ways to get it subtly wrong.

This is the full field guide. We treat CAU and Storage Replica not as two features but as the two halves of one discipline — keeping a clustered, replicated Windows workload available through both planned maintenance and unplanned site loss. You will learn to choose updating modes, wire pre/post scripts that gate on real health, define fault domains and sites, size the SR log from a measured workload, place the witness where it survives the failure it is meant to survive, and read the cluster log and SR event channel when something is wrong at 03:00. Every mechanism comes with real PowerShell, the exact cmdlet to confirm it, and the failure mode it prevents. Because this is a reference you will return to mid-incident, the modes, parameters, limits, events and failure classes are laid out as scannable tables — read the prose once, then keep the tables open. Both features ship in Windows Server Datacenter; the commands target Windows Server 2022 and apply cleanly to 2019 and 2025 except where noted.

What problem this solves

Two production pains, one shared root cause: a cluster is only highly available if you can lose a node — planned or not — without taking down what runs on it. Manual patching breaks that promise. An operator patching a 12-node Storage Spaces Direct cluster by hand has to track, per node, which roles have drained, whether the storage pool has finished repairing after the last node came back, and whether suspending the next node would drop quorum. Miss any one and you get partial data-unavailability (a virtual disk goes offline because too many nodes are down mid-repair) or a full cluster stop (quorum lost). CAU encodes all of that logic — drain, quorum check, resync wait, serial ordering — so the machine tracks it, not a human at 03:00.

The disaster-recovery pain is different but adjacent. A single-site cluster survives node and disk failure; it does not survive the datacenter burning, the metro power event, or the flood. Backups give you recovery, but at the cost of hours of restore and whatever RPO your last backup allows — unacceptable for a settlement database or an EHR that regulators hold to zero data loss. Storage Replica closes that gap by keeping a block-level, synchronous copy of the volume in a second site beneath the filesystem, so a site failover brings the workload up in the surviving site with RPO = 0 and an RTO measured in seconds-to-minutes. Extend the cluster itself across the two sites — a stretch cluster — and the failover is automatic: the cluster’s own health monitoring detects the site loss and brings roles up on the far side.

Who hits this: anyone running Windows Failover Clustering at scale — SQL Server Failover Cluster Instances, Hyper-V clusters, Scale-Out File Servers, general-use file servers — where both planned maintenance without downtime and survival of a site loss are hard requirements. It bites hardest on regulated workloads with contractual RPO/RTO (payments, healthcare, trading), on large clusters where manual patching is genuinely unsafe, and on two-site deployments where the quorum and witness placement decisions are non-obvious and, done wrong, guarantee a split-brain outage the first time the inter-site link blips.

To frame the whole field before the deep dive, here is the map of what this article covers, the pain each piece removes, and the single command that proves it is working:

Learning objectives

By the end of this article you can:

• Choose between CAU self-updating and remote-updating modes deliberately, and explain the Update Coordinator hand-off that lets a self-updating cluster patch its own last node.
• Configure the CAU clustered role with the right MaxFailedNodes, MaxRetriesPerNode, run profile, plug-in and pre/post-update scripts, and write a pre-script that gates on genuine cluster and replication health.
• Explain and control CAU’s drain-first, quorum-respecting, resync-aware ordering — and pin a NodeOrder so a stretch cluster patches one site fully before the other.
• Distinguish Storage Replica synchronous (RPO 0, latency-bound) from asynchronous (distance-tolerant, RPO > 0), size the log volume from a Test-SRTopology measurement, and enforce the GPT / destination-size / dismounted-target rules.
• Build a two-site stretch cluster with fault domains, sites, a PreferredSite, and an SR partnership that flips replication direction automatically on failover.
• Get quorum right for a split-site cluster: dynamic quorum, dynamic witness, and the correct witness type and placement (cloud vs file-share) in a third, independent failure domain.
• Test site failover and controlled failback, and confirm SR reversed direction without a full resync.
• Troubleshoot with the cluster log, Get-ClusterLog, the Storage Replica and Failover Clustering event channels, and the SR performance counters — mapping symptoms to causes to fixes.
• Understand where cluster sets fit for scaling beyond a single cluster’s node ceiling.

Prerequisites & where this fits

You should already be comfortable building a single-site Windows Failover Cluster: creating it with New-Cluster, validating with Test-Cluster, adding storage, and understanding that a cluster is a set of nodes that vote to maintain quorum (a majority) and that clustered roles (groups) move between nodes on failover. Familiarity with Storage Spaces Direct (S2D) — software-defined storage pooling local NVMe/SSD/HDD across nodes into cluster shared volumes — helps, because S2D and stretch clusters share resync and fault-domain concepts. You should know PowerShell remoting (Invoke-Command, WinRM) and how to read cluster and event-log output.

This sits at the top of the Servers / High Availability & DR track. It assumes the foundation from Windows Failover Clustering and Storage Spaces Direct: A Production Build — that article builds the single-site cluster this one patches and stretches. It pairs with Hyper-V Live Migration and Replica for Zero-Downtime VM Mobility (the VM-mobility layer that rides on top of a healthy cluster) and Resilient File Services with DFS Namespaces and DFS Replication (an alternative, file-level replication story where SR’s block-level, single-writer model does not fit). It complements Hardening Windows Server and Building a Reliable WSUS Patch Pipeline — WSUS is the update source CAU pulls from, and this article is the orchestration on top of it.

A quick map of who owns what during a cluster-patching or site-failover event, so you engage the right team fast:

Core concepts

Six mental models make every later decision obvious.

A cluster stays up only while it holds quorum. Quorum is a majority of votes. In a classic model every node has one vote and a witness (disk, file share, or cloud) can hold one more; the cluster stays online while more than half the votes are reachable and can talk to each other. Lose majority — too many nodes down, or a network partition where no side has a majority — and the cluster stops to prevent split-brain (two halves both thinking they own the workload, corrupting shared state). Everything about patching order and witness placement is, at bottom, about never accidentally dropping below majority.

CAU is an orchestrator, not a patch source. CAU does not decide which updates to install — a plug-in does that, pulling from WSUS, Windows Update for Business, or Microsoft Update. What CAU owns is the safe sequence: for each node, put it in maintenance, drain its roles elsewhere, install updates via the plug-in, reboot, resume, wait for storage to resync, and only then move on. It is serial (one node at a time), drain-first (nothing serving on a node being rebooted), quorum-respecting (never suspend a node that would break majority), and resync-aware (on S2D/stretch it waits for storage to rebuild before touching the next node).

Storage Replica works beneath the filesystem, one writer at a time. SR does block-level, volume-to-volume replication under NTFS/ReFS — it does not know about files, only blocks. Critically it is single-writer: the destination volume is dismounted and inaccessible while it is a replication target. You cannot read the second site’s copy as a live share; it is a cold, exact block replica that becomes writable only when replication direction flips (on failover). This is the fundamental difference from DFS Replication, which is file-level, multi-master, and lets both copies be read. SR trades that flexibility for exact, crash-consistent, real-time replication with a defined RPO.

Synchronous vs asynchronous is the RPO decision, bounded by latency. In synchronous mode, a write is not acknowledged to the application until it is committed to the SR log on both source and destination — so a site loss loses nothing in flight (RPO = 0), but every write pays the round-trip to the second site (so you need low latency, ≈ ≤ 5 ms round-trip). In asynchronous mode the write acks locally and ships shortly after — surviving high-latency, long-distance links but with a non-zero RPO (you can lose in-flight writes). Metro → synchronous; cross-region → asynchronous. The distance dictates the mode, and the mode dictates the RPO.

The SR log is the beating heart, and it is in the write path. Every write lands in the log volume first, then drains to the data volume. The log absorbs bursts and bridges brief link slowdowns. If the log fills — destination fell behind, link saturated — replication stalls and (synchronous) application writes back up. So the log must be on your fastest storage (NVMe/SSD, because log latency is in the critical path of every sync write) and large enough to ride out the worst burst plus link blip you will realistically see. Under-provisioning the log is the single most common Storage Replica production failure.

A stretch cluster is one cluster in two sites, and quorum is the hard part. Nodes live in two datacenters; each site has its own storage (asymmetric — this is not shared storage); SR replicates one site’s volumes to the other; and the cluster’s site awareness (fault domains) keeps roles in their home site, fails over within a site before crossing the WAN, and enables automatic site failover. The danger is even votes: 2 nodes + 2 nodes, link dies, each side has exactly half — both lose quorum, total outage. The witness in a third, independent failure domain breaks that tie. Dynamic quorum and dynamic witness adjust vote weights automatically to keep the surviving side in the majority through sequential failures.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

CAU updating modes: self-updating versus remote-updating

CAU runs as an Update Coordinator that walks the cluster one node at a time. There are two ways to run it, and the choice shapes your whole patch pipeline.

Remote-updating mode. The coordinator runs outside the cluster — on an admin workstation or a management server — and drives an Update Run on demand with Invoke-CauRun. Nothing is installed in the cluster itself. This is the mode for ad-hoc, attended patching, for dry runs before you commit to automation, and for emergency out-of-band patching (a zero-day you must push now, not on Sunday). The coordinator host must stay up for the whole run; if it reboots or loses the network mid-run, the run is interrupted.

Self-updating mode. You add a clustered CAU role (Microsoft.ClusterAwareUpdating) to the cluster. The role schedules itself and, when the window opens, one node temporarily hosts the coordinator, patches every other node, then hands the coordinator role to an already-patched node so the last one can be updated too. This hand-off is the clever bit — it is how a cluster patches its own coordinator without an external box. Fully unattended, no management host required, survives a coordinator-node reboot because the role fails over.

The mechanism (drain → update → reboot → resume → wait) is identical in both modes; the difference is who owns the schedule and where the coordinator lives. My default is self-updating for steady-state monthly patching, with a remote-updating run kept in the toolbox for emergencies and dry runs. The two modes compared on every axis that matters:

Install the tooling on every node and on the management host:

$nodes = 'clu-n1','clu-n2','clu-n3','clu-n4'

Invoke-Command -ComputerName $nodes -ScriptBlock {
    Install-WindowsFeature -Name RSAT-Clustering-PowerShell, FS-FileServer
}

# CAU UI + automation cmdlets on the management workstation (remote-updating):
Install-WindowsFeature -Name RSAT-Clustering-AutomationServer, RSAT-Clustering-PowerShell

Confirm the cluster is actually ready to be updated by CAU. This single command is the best pre-flight check there is — it verifies WinRM, the firewall rules, the required PowerShell remoting, and (for self-updating) that the clustered role is installable:

Test-CauSetup -ClusterName clu-prod

Resolve every warning before going further. The most common failures are a missing Failover Clustering firewall group and the absence of the RemoteShutdown / RestartCluster inbound rule, both of which CAU needs to reboot nodes remotely. What Test-CauSetup checks and what each failure means:

Turn off each node’s own Automatic Updates auto-install. If Windows Update installs and reboots a node on its own schedule, it will fight CAU and can reboot a node CAU did not drain. CAU wants to be the only thing that patches and reboots these nodes.

Configuring the CAU role, run profiles, and pre/post scripts

For self-updating, register the clustered role and bake the schedule, plug-in, and options into it. The plug-in Microsoft.WindowsUpdatePlugin pulls from whatever the node points at — WSUS, Windows Update for Business, or Microsoft Update.

Add-CauClusterRole -ClusterName clu-prod `
    -DaysOfWeek Sunday `
    -WeeksOfMonth @(2,4) `
    -IntervalWeeks 1 `
    -MaxFailedNodes 1 `
    -MaxRetriesPerNode 2 `
    -RequireAllNodesOnline `
    -CauPluginName Microsoft.WindowsUpdatePlugin `
    -CauPluginArguments @{ 'IncludeRecommendedUpdates' = 'False' } `
    -PreUpdateScript  '\\fs01\cau$\pre-update.ps1' `
    -PostUpdateScript '\\fs01\cau$\post-update.ps1' `
    -StartDate '2026-06-14 02:00:00' `
    -Force

The knobs that matter, what they default to, and the trade-off in each:

The plug-ins are the pluggable “what to install” layer. The two that ship, and when to use each:

A common CauPluginArguments set for the Windows Update plug-in — filter what gets offered so CAU installs exactly what your change window approved:

A pre-update script worth its salt blocks the run if the cluster is unhealthy. A non-zero exit aborts the whole run before a single node is touched:

# pre-update.ps1 — abort the Update Run if anything is already unhealthy
$bad = Get-ClusterNode | Where-Object State -ne 'Up'
if ($bad) {
    Write-Error "Nodes not Up before patching: $($bad.Name -join ', ')"
    exit 1
}
# Storage Replica must be healthy before we start rebooting nodes
$lag = Get-SRGroup | Get-SRPartnership |
       Where-Object ReplicationStatus -ne 'ContinuouslyReplicating'
if ($lag) { Write-Error 'Storage Replica not in sync; aborting.'; exit 1 }
exit 0

Register the scripts in source control and deploy them to a share the cluster’s machine accounts can read. CAU runs them as SYSTEM on the coordinator, so permission the share to the cluster name object (CNO) and node computer accounts, not a user. Pre- versus post-script responsibilities and the traps:

A post-update script that confirms the cluster came back healthy and re-enables what the pre-script quiesced:

# post-update.ps1 — confirm health and resume normal operations
$down = Get-ClusterNode | Where-Object State -ne 'Up'
if ($down) { Write-Warning "Nodes still down after run: $($down.Name -join ', ')" }

# Confirm every clustered role is Online
Get-ClusterGroup | Where-Object State -ne 'Online' |
    ForEach-Object { Write-Warning "Group not Online: $($_.Name)" }

# Re-enable the backup schedule the pre-script paused, notify on-call, etc.
Enable-ScheduledTask -TaskName 'Nightly-Backup' -ErrorAction SilentlyContinue
exit 0

Draining roles, fault domains, and update ordering

The reason CAU is safe is the drain. Before patching a node, CAU puts it into maintenance mode and live-migrates or fails over every clustered role off it — exactly what Suspend-ClusterNode -Drain does interactively. Nothing on that node is serving traffic when the reboot hits. On a Hyper-V cluster the VMs live-migrate (no guest downtime); on a general-use file server the file-server role fails over (a brief SMB reconnect); on S2D the node’s storage contribution is safely taken offline and the pool tolerates it.

CAU’s ordering is availability-aware: by default it updates the node hosting the fewest roles first and, critically, respects the cluster’s quorum so it never suspends a node that would break the majority. On an S2D or stretch cluster, CAU also waits for storage to finish resyncing before moving on — it will not pull a second node while virtual disks are still rebuilding from the first. This resync-await is what prevents the classic manual-patching disaster: two nodes down at once mid-repair, a virtual disk losing too many copies, data going offline.

You can pin the order with -NodeOrder. On a stretch cluster you almost always want to patch one site fully, confirm it is healthy, then patch the other — never interleave sites, or a mid-patch metro link blip could leave you with one rebooting node per site and no quorum:

# Patch Site A nodes first, then Site B, never interleaved
Invoke-CauRun -ClusterName clu-prod `
    -CauPluginName Microsoft.WindowsUpdatePlugin `
    -NodeOrder clu-n1,clu-n2,clu-n3,clu-n4 `
    -MaxFailedNodes 1 -MaxRetriesPerNode 2 `
    -RequireAllNodesOnline -Force -Wait

The drain/suspend/resume lifecycle CAU runs per node, and the interactive equivalent for each step so you can reproduce or debug it by hand:

Verify a drain by hand before trusting automation — this is the single most important pre-flight for a first CAU run:

Suspend-ClusterNode -Name clu-n1 -Drain -Wait
Get-ClusterGroup | Where-Object OwnerNode -eq 'clu-n1'   # should be EMPTY
Get-ClusterNode  -Name clu-n1 | Select-Object Name, State # Paused
Resume-ClusterNode -Name clu-n1 -Failback Immediate

Fault domains give CAU (and the cluster) topology awareness beyond just “site.” Windows Failover Clustering understands a four-level hierarchy; defining it makes both placement and CAU ordering smarter:

CAU never reboots two nodes at once. The whole design is serial, drain-first, quorum-respecting, and (on S2D/stretch) resync-aware. If you ever see two nodes down during a CAU run, something is wrong — stop and investigate; do not assume CAU did it. The most likely culprit is a node’s own Windows Update auto-installing and rebooting outside CAU’s control.

Storage Replica modes: synchronous, asynchronous, and log sizing

Storage Replica (SR) does block-level, volume-to-volume replication beneath the filesystem. Two modes, and the distinction is the whole disaster-recovery conversation.

Synchronous. The write is not acknowledged to the application until it is committed to the SR log on both the source and destination. Zero data loss — RPO = 0. The price is latency: every write pays the round-trip to the second site, so you need roughly ≤ 5 ms round-trip and fat, low-jitter links. This is the mode for a metro stretch cluster.

Asynchronous. The write acks locally and ships to the destination shortly after. Survives long-distance, high-latency links, but your RPO is non-zero — you can lose the in-flight writes not yet shipped. Use it across regions, not across a metro. Async also introduces a recovery point objective window you can tune (-AsyncRPO), the maximum seconds the destination may lag before SR warns.

The two modes on every axis that drives the choice:

The log volume is the heart of SR and the part people under-provision. Every write lands in the log first, then drains to the data volume. If the log fills — because the destination fell behind or the link saturated — replication stalls. Microsoft’s guidance is at least 8 GB of log per data volume, sized up for write-heavy workloads and to bridge the longest link blip you expect. Put the log on the fastest storage you have (NVMe/SSD); log latency is in the critical path of every synchronous write. Log sizing rules of thumb:

Two iron rules SR enforces, and the surprises they cause:

1. Log and data volumes must be GPT, not MBR. An MBR volume is silently ineligible; New-SRPartnership refuses it.
2. The destination data volume must be at least as large as the source, and the destination volume is dismounted and inaccessible while it is a replication target — you cannot read it on the passive side. This surprises people: the second site’s copy is not a live, browsable share; it is a cold, exact block replica.

The hard SR requirements and limits, as a reference:

Validate a server pair before committing — Test-SRTopology runs a real workload sample and emits an HTML report with measured latency, throughput, IOPS, and a recommended log size:

Test-SRTopology `
    -SourceComputerName clu-n1 -SourceVolumeName D: -SourceLogVolumeName E: `
    -DestinationComputerName clu-n3 -DestinationVolumeName D: -DestinationLogVolumeName E: `
    -DurationInMinutes 30 -ResultPath \\fs01\sr$\report

Run it during a busy period — sizing on idle numbers is exactly how you fill a log in production. The report’s recommended log size assumes it sampled your real peak; if you ran it at 3 a.m. on an idle box, its recommendation is meaningless. What Test-SRTopology measures and how to read it:

Building a two-site stretch cluster with replicated volumes

A stretch cluster is a single failover cluster whose nodes live in two sites, with SR keeping each site’s storage in sync. The shape: two nodes in Site A, two in Site B, asymmetric storage (each site has its own disks — this is not shared storage), and SR replicating Site A’s volumes to Site B. On failover the cluster brings roles up in the surviving site and SR flips replication direction so the survivor becomes the source.

First, install the feature everywhere and tell the cluster which node lives where using fault domains and sites. Site awareness is what makes the cluster keep roles local, fail over within a site before crossing the WAN, and order placement sensibly:

$all = 'clu-n1','clu-n2','clu-n3','clu-n4'
Invoke-Command -ComputerName $all -ScriptBlock {
    Install-WindowsFeature -Name Storage-Replica, FS-FileServer -IncludeManagementTools -Restart
}

# Define sites and assign nodes (fault-domain awareness)
New-ClusterFaultDomain -Name SiteA -Type Site -Location 'DC-East'
New-ClusterFaultDomain -Name SiteB -Type Site -Location 'DC-West'
Set-ClusterFaultDomain -Name clu-n1 -Parent SiteA
Set-ClusterFaultDomain -Name clu-n2 -Parent SiteA
Set-ClusterFaultDomain -Name clu-n3 -Parent SiteB
Set-ClusterFaultDomain -Name clu-n4 -Parent SiteB

# Prefer keeping groups in their home site; cross-site failover only when needed
(Get-Cluster).PreferredSite = 'SiteA'

# Confirm the topology
Get-ClusterFaultDomain | Format-Table Name, Type, ParentName, Location

What site awareness buys you once fault domains are defined — the behaviours that turn a “cluster that happens to span two sites” into a real stretch cluster:

Cross-site heartbeat parameters matter on a stretch cluster: the default sub-second heartbeat thresholds tuned for a LAN are too twitchy for a WAN and will cause false failovers on a normal latency blip. Loosen them for the cross-site path only:

# Tolerate WAN jitter on the inter-site heartbeat (values are examples — tune to your link)
(Get-Cluster).CrossSiteDelay     = 1000   # ms between heartbeats across sites
(Get-Cluster).CrossSiteThreshold = 20     # missed heartbeats before the site is 'down'

Now establish the replication partnership. Add the disks to the cluster first, then create the SR partnership between the Site A data volume and its Site B counterpart, each with its own log:

New-SRPartnership `
    -SourceComputerName clu-n1 -SourceRGName rg-SiteA `
    -SourceVolumeName 'C:\ClusterStorage\DataA' -SourceLogVolumeName 'L:' `
    -DestinationComputerName clu-n3 -DestinationRGName rg-SiteB `
    -DestinationVolumeName 'C:\ClusterStorage\DataB' -DestinationLogVolumeName 'L:' `
    -ReplicationMode Synchronous `
    -LogSizeInBytes 32GB

ReplicationMode Synchronous is the line that buys you RPO = 0. The cluster now sees the replicated volume as a single resource that can move between sites; whichever site owns it is the source, and SR flips replication direction automatically on failover. The key New-SRPartnership parameters:

Watch the initial sync complete before declaring the stretch cluster ready — until it does, the destination is not yet a valid failover target:

# Initial sync progress — NumOfBytesRemaining trends to ~0, status becomes ContinuouslyReplicating
while ((Get-SRGroup -Name rg-SiteA).Replicas.ReplicationStatus -ne 'ContinuouslyReplicating') {
    Get-SRGroup -Name rg-SiteA |
        Select-Object -ExpandProperty Replicas |
        Format-Table ReplicationStatus, NumOfBytesRemaining
    Start-Sleep -Seconds 30
}
Write-Host 'Initial sync complete — destination is now a valid failover target.'

Quorum: dynamic quorum, witness types, and split-site placement

A two-site cluster with an even number of nodes (2+2) has a fatal quorum problem: if the inter-site link dies, each site holds exactly half the votes and both sites lose quorum — a split brain that takes the whole cluster down. The witness is what breaks the tie, and where you put it is the entire decision.

The witness must live in a third, independent failure domain — never in Site A or Site B. If it sits in Site A and Site A is the site that fails, you lose the witness and two nodes at once: game over. Witness types compared:

• Cloud witness is the right answer for stretch clusters. It is a tiny blob in an Azure Storage account, reachable from both sites over HTTPS, neutral to either datacenter failing. My default.
• File-share witness works when there is no internet egress — but only if you have a genuine third site to host it. A share inside one of the two datacenters defeats the purpose.
• Disk witness is not an option for a stretch cluster: there is no shared disk spanning both sites (storage is asymmetric).

# Cloud witness — the third, neutral vote for a stretch cluster
Set-ClusterQuorum -CloudWitness `
    -AccountName 'stcluwitness' `
    -AccessKey  '<storage-account-key>' `
    -Endpoint   'core.windows.net'

# Or a file-share witness in a genuine THIRD site:
# Set-ClusterQuorum -FileShareWitness '\\fs-thirdsite\clu-witness$'

# Confirm every node and the witness each hold one vote
Get-ClusterQuorum | Format-List *
Get-ClusterNode | Select-Object Name, DynamicWeight, NodeWeight

Dynamic quorum and dynamic witness are on by default and they are your friends here: the cluster automatically adjusts vote weights so the surviving site retains the majority during a sequential failure, and toggles the witness vote to keep the total odd. You do not manage these by hand, but you must read them correctly in logs — a node’s effective weight (DynamicWeight) changes over time and differs from its assigned weight (NodeWeight). The two quorum mechanisms:

The failure scenarios a 2+2 stretch cluster faces, and what keeps it alive in each:

Dynamic quorum protects against sequential failures. A simultaneous, instantaneous loss of two same-site nodes before the cluster recomputes weights can still momentarily drop quorum. The cloud witness is what carries you through that window — which is exactly why it must not share a failure domain with either site. Place it wrong and you have engineered a single point of failure into your DR design.

If quorum is genuinely lost (a real double failure), you can force a surviving partition online — the last-resort recovery. Use it knowingly; forcing the wrong partition online is how you lose the writes that were only on the other side:

# LAST RESORT: force quorum on a surviving node when the cluster is stopped
# Run ON the node you are CERTAIN has the most-current data (usually the SR source side)
Start-ClusterNode -Name clu-n3 -FixQuorum
# Then adjust node weights so the survivors form a majority, and bring groups online.

Automatic site failover and controlled failback

A properly built stretch cluster fails over automatically when a site is lost: the cluster’s health monitoring detects that Site A’s nodes are unreachable, quorum is preserved (Site B + witness), the clustered roles come online in Site B, and SR flips replication direction so the Site B copy becomes the writable source. No human, no restore. What you must test is that this actually happens cleanly, and that you can fail back to the primary site once it recovers.

The failover sequence, step by step, and where it can go wrong:

Move the role to force a failover for testing — this is the drill that matters. Simulate losing Site A and confirm roles and storage come up in Site B, then fail back cleanly:

# Move the replicated role to Site B and let SR reverse direction
Move-ClusterGroup -Name 'FS-Stretch' -Node clu-n3
Get-ClusterGroup 'FS-Stretch' | Select-Object Name, OwnerNode, State

# Confirm SR flipped: clu-n3 is now source, clu-n1 is destination
Get-SRPartnership | Select-Object SourceComputerName, DestinationComputerName

# Controlled failback to Site A once it is healthy again
Move-ClusterGroup -Name 'FS-Stretch' -Node clu-n1

Watch the SR event channel during the failover — Microsoft-Windows-StorageReplica/Admin. Key events mark direction changes and resync; a clean failover shows the destination becoming source within seconds and replication resuming in the new direction without a full resync. If you see a full resync kick off, the failover was not clean (the destination was behind, or the log was inconsistent). The SR events to know:

To manually reverse replication (e.g., after a planned site swap where you want Site B to be the long-term source), use Set-SRPartnership:

# Manually reverse direction: make Site B the source, Site A the destination
Set-SRPartnership -NewSourceComputerName clu-n3 -SourceRGName rg-SiteB `
                  -DestinationComputerName clu-n1 -DestinationRGName rg-SiteA

Cluster sets: scaling beyond one cluster

A single failover cluster tops out — practically and supportability-wise — around 64 nodes, and operationally you often want fault isolation between clusters (a bad patch or a quorum event on one cluster should not touch another). Cluster sets are the answer: a loose federation of multiple failover clusters into a single, larger software-defined datacenter, with a management cluster coordinating and a unified cluster-set namespace so VMs can live-migrate between member clusters. This is where you go when one stretch cluster is not enough — many clusters, unified management, fault domains that are entire clusters.

Cluster sets vs a single (stretch) cluster, so you place the boundary correctly:

A minimal cluster-set creation (each member cluster and the management cluster already exist):

# Create the cluster set on a management cluster, then add member clusters
New-ClusterSet -Name CS-Prod -NamespaceRoot 'SOFS-ClusterSetMaster' `
    -CimSession mgmt-cluster

Add-ClusterSetMember -ClusterName clu-prod-a -CimSession CS-Prod `
    -InfraSOFSName 'SOFS-A'
Add-ClusterSetMember -ClusterName clu-prod-b -CimSession CS-Prod `
    -InfraSOFSName 'SOFS-B'

# Live-migrate a set VM across member clusters
Move-ClusterSetVM -CimSession CS-Prod -VMName 'vm-app01' `
    -Node 'clu-prod-b-n1'

The rule of thumb: use a stretch cluster for HA + metro DR of a workload; reach for cluster sets when you have outgrown a single cluster’s node count or need entire-cluster fault isolation. They are not mutually exclusive — member clusters within a set can themselves be stretch clusters.

Architecture at a glance

Picture the system as two datacenters and a neutral tie-breaker, with two orchestration layers riding on top. In Site A (DC-East) sit clu-n1 and clu-n2; in Site B (DC-West) sit clu-n3 and clu-n4. All four are members of one failover cluster — this is the stretch cluster — but their storage is asymmetric: Site A owns its local NVMe/SSD pool, Site B owns its own. There is no shared disk spanning the metro. What keeps the two sides consistent is Storage Replica, running beneath the filesystem: the Site A data volume (C:\ClusterStorage\DataA) replicates synchronously — every write committed to the SR log on both sides before the application sees an ack — to its Site B counterpart (C:\ClusterStorage\DataB), with a dedicated NVMe log volume on each side absorbing the write stream. The destination volume is dismounted; it is a cold, exact block replica, not a browsable share.

Sitting above the storage is the cluster’s quorum brain. Each of the four nodes holds one vote; a fifth vote lives in a cloud witness — a tiny blob in an Azure Storage account, reachable from both sites over HTTPS, deliberately placed in a third failure domain that shares nothing with either datacenter. Dynamic quorum trims a failed node’s vote so the survivors keep a majority; dynamic witness toggles the witness vote to keep the total odd; PreferredSite = SiteA decides who wins a clean 50/50 partition. Trace the failure: Site A burns → its two votes vanish → Site B’s two nodes plus the cloud witness make three of the five remaining votes → majority preserved → roles fail over to Site B → SR flips direction, making the Site B copy the writable source, with no full resync. That is the whole point: a failover (seconds), not a restore (hours), at RPO = 0.

The two orchestration layers complete the picture. Cluster-Aware Updating patches the whole thing without downtime: an Update Coordinator (a cluster node in self-updating mode, handing off for its own last node) walks the cluster one node at a time, draining roles off each node before it reboots, respecting quorum so it never suspends a node that breaks majority, waiting for storage to resync between nodes, and — via a pinned NodeOrder — patching Site A fully, confirming health, then Site B, never interleaving the sites. A pre-update script gates the whole run on real health: if any node is down or SR is genuinely diverged, it exits non-zero and the run aborts before a single reboot. If you outgrow this one cluster, cluster sets federate many such clusters under one management plane with VM mobility between them — each member cluster keeping its own quorum blast radius. The reader’s mental walk is: storage replicates synchronously beneath the filesystem → quorum plus a third-domain witness survives a site loss → the cluster fails over automatically and SR reverses → CAU keeps it all patched, one drained node at a time, one site before the other.

Real-world scenario

Meridian Settlements, a payments processor, ran a 4-node stretch cluster across two metro datacenters 38 km apart, hosting the SQL Server FCI behind their settlement service. RPO was contractually zero — the settlement ledger could not lose a committed transaction — so Storage Replica ran synchronous over dark fibre with a measured ≈ 2 ms round-trip. Two nodes in DC-East (clu-n1, clu-n2), two in DC-West (clu-n3, clu-n4), a cloud witness in an Azure Storage account in a third region, PreferredSite = DC-East. Monthly patching had been a quarterly all-hands maintenance window — four engineers, a bridge call, hours of manual drain-patch-reboot — until they moved to CAU self-updating.

The first automated run aborted halfway through, on a Sunday at 02:30, with one node (clu-n1) already patched and the run refusing to continue. The on-call engineer’s first instinct was that CAU had broken. It had not — the abort was their own pre-update gate working exactly as designed.

The cause: their pre-update.ps1 checked Get-SRGroup for ContinuouslyReplicating before each Update Run, and the dark-fibre provider had a 6 ms latency spike that night from a re-route. At 6 ms — three times the normal round-trip — synchronous writes backed up, the SR log on the source crept toward full, and ReplicationStatus flipped out of ContinuouslyReplicating into a catching-up state. The gate did its job and aborted rather than reboot a node while replication was unhealthy — which, had it proceeded, would have taken a node down while writes to the settlement DB were already stalling. Exactly the outcome the gate exists to prevent.

But the gate was too binary. It treated any deviation from ContinuouslyReplicating — including a transient, self-healing catch-up of a few megabytes — as a hard stop. The fix was not to weaken the gate; it was to make it tolerant of transient lag while still blocking on real divergence, and to split CAU into a per-site NodeOrder so Site A fully completed and resynced before Site B was touched:

# pre-update.ps1 — tolerate transient lag, block on genuine divergence
$g = Get-SRGroup | Get-SRPartnership
$diverged = $g | Where-Object {
    $_.ReplicationStatus -ne 'ContinuouslyReplicating' -and
    $_.NumOfBytesRemaining -gt 64MB    # bounded catch-up is acceptable
}
if ($diverged) { Write-Error 'SR genuinely behind; aborting run.'; exit 1 }
Start-Sleep -Seconds 30   # ride out a brief link blip, then re-check
$g = Get-SRGroup | Get-SRPartnership
if ($g.ReplicationStatus -contains 'Error') { exit 1 }
exit 0

They also oversized the SR log from 8 GB to 32 GB on NVMe so a latency spike no longer threatened to fill it, and loosened the cross-site heartbeat threshold slightly so a normal jitter would not risk a false failover. The next month’s run completed unattended in 41 minutes, every settlement transaction served throughout, the FCI staying online as each node drained, patched, and rejoined. The quarter after, they added a second workload cluster and federated both with a cluster set for unified management.

The incident as a timeline, because the order of realization is the lesson:

The lesson on the wall: “Synchronous SR makes link quality a first-class patching dependency. Your CAU pre-script is the right place to encode that — tolerant of jitter, intolerant of real data loss. And a gate that aborts is CAU protecting you, not failing you.”

Advantages and disadvantages

Running a clustered, replicated, self-patching Windows workload buys enormous resilience — and adds real operational surface area. Weigh it honestly:

This model is right when you have a hard requirement for both zero-downtime maintenance and survival of a site loss — regulated workloads (payments, healthcare, trading), business-critical SQL/Hyper-V, anything with contractual RPO/RTO. It is over-engineered for a single dev cluster or a workload where hours of restore and non-zero RPO are acceptable (use backups or async SR there). The disadvantages are all manageable — but only if you know they exist, size the log from a real measurement, place the witness in a third domain, and write the pre-gate to tell a blip from divergence.

Hands-on lab

You will build a two-node Storage Replica pair and simulate the failover, then configure CAU on a small cluster and watch a drain. Storage Replica requires Windows Server Datacenter; this lab uses two VMs (sr-src, sr-dst) each with a data disk and a fast log disk. Run from an elevated PowerShell that can remote into both.

Step 1 — Prep both servers. Install the feature and confirm edition:

$pair = 'sr-src','sr-dst'
Invoke-Command -ComputerName $pair -ScriptBlock {
    Install-WindowsFeature -Name Storage-Replica, FS-FileServer -IncludeManagementTools
    (Get-WindowsEdition -Online).Edition   # must be *Datacenter*
}
Restart-Computer -ComputerName $pair -Wait -For PowerShell

Expected: Storage-Replica installs; edition contains Datacenter.

Step 2 — Prepare GPT volumes on each server. Bring the data and log disks online as GPT, format, and assign letters (D: data, E: log). SR refuses MBR:

Invoke-Command -ComputerName $pair -ScriptBlock {
    # Data disk → D:, Log disk → E: (adjust disk numbers to your VM)
    Get-Disk | Where-Object PartitionStyle -eq 'RAW' |
        Initialize-Disk -PartitionStyle GPT -PassThru |
        Out-Null
    # (In a real run, create partitions and format D: and E: as NTFS/ReFS here.)
    Get-Volume | Select-Object DriveLetter, FileSystem, SizeRemaining
}

Expected: D: and E: exist on both, GPT, NTFS/ReFS. The destination D: must be ≥ the source D:.

Step 3 — Test topology BEFORE committing. Measure latency and get a recommended log size:

Test-SRTopology -SourceComputerName sr-src -SourceVolumeName D: -SourceLogVolumeName E: `
    -DestinationComputerName sr-dst -DestinationVolumeName D: -DestinationLogVolumeName E: `
    -DurationInMinutes 10 -ResultPath C:\Temp\SRReport
Invoke-Item C:\Temp\SRReport\TestSrTopologyReport-*.html

Expected: an HTML report with round-trip latency, throughput, and a recommended log size. In a lab the latency is ~0 ms (same host); read the shape of the report.

Step 4 — Create the replication partnership (synchronous).

New-SRPartnership -SourceComputerName sr-src -SourceRGName rg-src `
    -SourceVolumeName D: -SourceLogVolumeName E: `
    -DestinationComputerName sr-dst -DestinationRGName rg-dst `
    -DestinationVolumeName D: -DestinationLogVolumeName E: `
    -ReplicationMode Synchronous -LogSizeInBytes 8GB

Expected: the partnership is created and initial sync begins. Note the destination D: dismounts — you can no longer browse it on sr-dst.

Step 5 — Watch it reach ContinuouslyReplicating.

Get-SRGroup -Name rg-src |
    Select-Object -ExpandProperty Replicas |
    Format-Table ReplicationStatus, NumOfBytesRemaining, NumOfBytesRecovered

Expected: ReplicationStatus becomes ContinuouslyReplicating and NumOfBytesRemaining trends to ~0. That is healthy synchronous replication.

Step 6 — Reverse direction (simulate a failover). Make the destination the new source:

Set-SRPartnership -NewSourceComputerName sr-dst -SourceRGName rg-dst `
    -DestinationComputerName sr-src -DestinationRGName rg-src
Get-SRPartnership | Select-Object SourceComputerName, DestinationComputerName

Expected: sr-dst is now the source, sr-src the destination; D: mounts on sr-dst and dismounts on sr-src. Watch Microsoft-Windows-StorageReplica/Admin for event 1215 (direction change).

Step 7 — (Cluster half) Confirm CAU readiness on your cluster and watch a drain by hand:

Test-CauSetup -ClusterName clu-lab
Suspend-ClusterNode -Name clu-lab-n1 -Drain -Wait
Get-ClusterGroup | Where-Object OwnerNode -eq 'clu-lab-n1'   # should be empty
Resume-ClusterNode -Name clu-lab-n1 -Failback Immediate

Expected: Test-CauSetup passes; the drain moves all groups off clu-lab-n1; resume brings it back.

Validation checklist. You proved the SR fundamentals hands-on: GPT-only volumes, Test-SRTopology before committing, the destination dismounting as a cold replica, ContinuouslyReplicating as the healthy synchronous state, and a clean direction reversal — the mechanics behind a stretch-cluster failover. And you watched a CAU drain empty a node without downtime. What each step proved:

Teardown.

Remove-SRPartnership -SourceComputerName sr-dst -SourceRGName rg-dst `
    -DestinationComputerName sr-src -DestinationRGName rg-src
Remove-SRGroup -ComputerName sr-src -Name rg-src
Remove-SRGroup -ComputerName sr-dst -Name rg-dst
# Optionally uninstall the feature:
# Invoke-Command $pair { Uninstall-WindowsFeature Storage-Replica }

Common mistakes & troubleshooting

The playbook — the part you bookmark. First as a scannable table you read at 03:00, then the highest-value entries expanded with the full confirm-command detail.

The highest-value entries, expanded:

1. CAU run aborts almost immediately. Root cause: The pre-update script exited non-zero — usually a health gate catching a down node or SR not in ContinuouslyReplicating. Confirm: Get-CauReport -ClusterName clu-prod -Last -Detailed shows the pre-script step and its exit; the message names the failed condition. Read the cluster’s %windir%\cluster\Reports for the run log. Fix: If the gate caught a real problem (a down node, genuine SR divergence), fix that first. If it tripped on a transient blip (brief latency spike, small NumOfBytesRemaining), make the gate tolerant — a bounded lag threshold plus a short re-check window (see the scenario). The abort is CAU protecting you; do not “fix” it by removing the gate.

3. SR replication stalled; application writes backing up. Root cause: The SR log filled — undersized for the workload, or the link saturated so the destination fell behind and the log couldn’t drain. Synchronous writes then back up because they wait for the remote commit. Confirm: Get-SRGroup | Select Name, LogVolume, LogSizeInBytes and the replica’s used/remaining log; event 1237 in Microsoft-Windows-StorageReplica/Admin is the log-full marker. SR performance counters (Storage Replica Statistics) show the log approaching capacity. Fix: Grow the log on NVMe/SSD, sized from a peak-load Test-SRTopology; fix the link bandwidth; for cross-region use async so a slow link doesn’t block app writes. Under-provisioning the log is the number-one SR failure — treat the log as production-critical fast storage, not an afterthought.

6/7. Split-brain / cluster down when a site or the link fails. Root cause: Witness placement. An even-node stretch cluster with the witness inside Site A (or B) loses the witness and two nodes when that site fails — dropping below majority. Or on a clean link partition with no reachable neutral witness, both halves lose quorum. Confirm: Get-ClusterQuorum shows the witness type and resource; check where the witness lives (a file share inside Site A is the classic mistake). Get-ClusterNode | Select Name, DynamicWeight, NodeWeight shows current votes. Fix: Put the witness in a third, independent failure domain — a cloud witness (Azure blob) is ideal because it shares nothing with either datacenter. If you must use a file-share witness, it must be in a genuine third site. Set PreferredSite so a clean 50/50 partition resolves in favour of the intended survivor.

9. Failover triggered a full resync instead of a clean flip. Root cause: At the moment of failover the destination was behind or the log was inconsistent, so SR couldn’t just reverse direction — it had to resync. Often follows a failover done while ReplicationStatus was not ContinuouslyReplicating. Confirm: Microsoft-Windows-StorageReplica/Admin shows event 5015 (resync started) rather than 1215 (direction changed) cleanly. Get-SRGroup shows NumOfBytesRemaining climbing during recovery. Fix: Only fail over (in a planned move) when replication is healthy — check ContinuouslyReplicating first. Size the log so a burst never leaves the destination behind. On an unplanned site loss a resync may be unavoidable, but a correctly sized, healthy partnership minimizes it.

Reading the cluster log

When a failover, a quorum event, or a CAU abort needs deep forensics, the cluster log is the truth. It is not on by default in a readable place — you generate it with Get-ClusterLog, which dumps a detailed, timestamped log per node:

# Generate cluster logs for all nodes into a folder, last 24h, local timezone timestamps
Get-ClusterLog -Destination C:\Temp\ClusterLogs -TimeSpan 1440 -UseLocalTime

# For a single node:
Get-ClusterLog -Node clu-n1 -Destination C:\Temp -TimeSpan 240

The diagnostic sources and when to reach for each:

Best practices

• Let CAU be the only thing that patches and reboots cluster nodes. Set each node’s Automatic Updates to notify-only; a node auto-rebooting outside CAU is how you get two nodes down at once.
• Run Test-CauSetup to zero warnings before enabling self-updating, and re-run it after any hardening baseline change that might disable a firewall rule CAU needs.
• Keep MaxFailedNodes low (1 on small clusters). A bad patch should abort the run, not march through your whole cluster taking nodes down.
• Write the pre-update gate to tell a blip from divergence. Block on genuine health failures (a down node, real SR divergence beyond a bounded threshold); tolerate transient lag with a short re-check. Keep the scripts in source control on a CNO-readable share.
• Pin NodeOrder on a stretch cluster to patch one site fully, confirm health, then the other. Never interleave sites — a mid-patch link blip must never leave one rebooting node per site.
• Size the SR log from a peak-load Test-SRTopology, on NVMe/SSD, GPT volumes. The log is production-critical fast storage; under-provisioning it is the number-one SR failure. When in doubt, oversize.
• Match replication mode to distance: synchronous for metro (≤ 5 ms round-trip, RPO 0); asynchronous across regions (RPO > 0). Never run synchronous over a high-latency link — it cripples every application write.
• Place the quorum witness in a third, independent failure domain. A cloud witness is the default for a stretch cluster; a file-share witness only counts if it’s in a genuine third site. A witness inside Site A or B engineers a single point of failure into your DR.
• Define fault domains and sites, set PreferredSite, and confirm destination ≥ source. Site awareness is what makes failover fail over within a site first and keeps roles local.
• Loosen cross-site heartbeat thresholds so normal WAN jitter never triggers a false failover — but not so far that a real site loss is detected slowly.
• Test the site failover and controlled failback regularly with Move-ClusterGroup, and confirm SR reversed direction without a full resync (event 1215, not 5015). A DR design you have never failed over is a hope, not a plan.
• Monitor Get-CauReport, the StorageReplica Admin channel, and SR performance counters after every run — log usage, bytes remaining, replication status, and per-node patch results.

Security notes

• Least-privilege for the CAU scripts. Pre/post scripts run as SYSTEM on the coordinator. Store them on a share writable only by administrators and readable by the CNO and node computer accounts — not world-readable, and never edited in place on the share (source-control them, deploy signed).
• Sign your pre/post-update scripts and set an appropriate ExecutionPolicy so an attacker who can write to the CAU share can’t inject arbitrary SYSTEM-level code that runs on every patch cycle.
• Protect the cloud-witness storage account key. The AccessKey passed to Set-ClusterQuorum -CloudWitness is a credential to an Azure Storage account; rotate it, store it in a secret manager, and prefer a dedicated, minimally-permissioned storage account for the witness. Consider Azure AD / managed-identity-based witness where supported.
• Storage Replica traffic should be on a segregated, trusted network. SR replicates raw blocks over SMB (TCP 445); run it over a dedicated replication VLAN/subnet between sites, and use SMB encryption if the inter-site link is not otherwise protected (dark fibre you own vs a shared carrier).
• Harden the SMB and cluster attack surface. The same clustering ports CAU and SR use are lateral-movement paths; scope firewall rules to the cluster’s own subnets, require SMB signing, and keep the cluster off any general-purpose management VLAN. See SMB Hardening: Credential Guard and Lateral-Movement Defense for the SMB side.
• Use gMSAs for service accounts where clustered roles need a service identity, so credentials rotate automatically and aren’t standing secrets — see Hardening Service Accounts with gMSA and Windows LAPS.
• Audit quorum and failover events. A forced-quorum (-FixQuorum) or an unexpected witness change is a high-signal event — forward the Failover Clustering and StorageReplica channels to your SIEM.

The security controls that also improve resilience — they pull in the same direction:

Cost & sizing

The bill drivers and how they interact with the design:

• Windows Server Datacenter licensing dominates. Both Storage Replica (unrestricted) and stretch clustering effectively require Datacenter edition per physical host (licensed per core, minimum 16 cores/host). Standard edition allows only a single ~2 TB volume / one partnership — not a production stretch cluster. This is the biggest line item and it scales with core count across all nodes.
• The SR log volume is fast storage you must dedicate — NVMe/SSD, sized from Test-SRTopology (8 GB floor, commonly 32–64 GB+). Cheap in absolute terms, but it must be fast, not spinning disk, because it’s in the write path.
• Inter-site bandwidth for synchronous replication must sustain your peak write rate; for async, your average (the log buffers bursts). Dark fibre or a dedicated metro link is a recurring operational cost and often the real constraint — latency, not money, decides whether sync is even viable.
• The cloud witness is nearly free — a tiny blob in an Azure Storage account, a few paise a month of storage plus negligible transactions. There is no reason a stretch cluster shouldn’t have one.
• Cluster sets add no licensing beyond the member clusters, but the management cluster and infrastructure SOFS are additional nodes to run.

A rough monthly picture for a 4-node metro stretch cluster (2+2), illustrative INR:

Sizing rules of thumb, as a table:

Interview & exam questions

1. What are CAU’s two updating modes and how does self-updating patch its own last node? Remote-updating runs the Update Coordinator on an external host driving an on-demand Invoke-CauRun; self-updating adds a clustered CAU role that schedules itself. Self-updating patches its own last node via a coordinator hand-off: one node coordinates, patches every other node, then hands the coordinator role to an already-patched node so the original coordinator can be updated too. No external host needed.

2. Why is CAU safe to run unattended on a large cluster? Because it is serial (one node at a time), drain-first (every clustered role is live-migrated/failed over off a node before it reboots), quorum-respecting (it never suspends a node that would break majority), and on S2D/stretch resync-aware (it waits for storage to finish rebuilding before touching the next node). It encodes the exact logic a careful human would follow, without the human error.

3. Synchronous vs asynchronous Storage Replica — what’s the trade-off? Synchronous acks a write only after it’s committed to the SR log on both sites → RPO = 0, but every write pays the round-trip, so it needs low latency (≈ ≤ 5 ms) and suits metro distances. Asynchronous acks locally and ships shortly after → tolerates high-latency/long-distance links but with a non-zero RPO (you can lose in-flight writes). Distance dictates the mode; the mode dictates the RPO.

4. Why is the SR log volume so important, and how do you size it? Every write lands in the log first, then drains to the data volume; the log absorbs bursts and bridges brief link slowdowns. If it fills, replication stalls (and synchronous, app writes back up). Size it from a peak-load Test-SRTopology run (8 GB floor, often 32–64 GB+), on NVMe/SSD because log latency is in the write path. Under-provisioning it is the most common SR failure.

5. Why can’t you read the DR-site copy of a Storage Replica volume? SR is single-writer: the destination data volume is dismounted and inaccessible while it’s a replication target — it’s a cold, exact block replica, not a live share. It only mounts and becomes writable when replication direction flips (on failover). This is the fundamental difference from DFS Replication, which is file-level and multi-master.

6. A 2+2 stretch cluster loses its inter-site link. What happens, and how do you prevent an outage? Each site holds exactly half the votes, so both sides can lose quorum — a split-brain that takes the whole cluster down. Prevent it with a witness in a third, independent failure domain (a cloud witness is ideal) so one side plus the witness holds the majority, and set PreferredSite so a clean 50/50 partition resolves in favour of the intended survivor.

7. Where must you NOT place the quorum witness for a stretch cluster, and why? Never inside Site A or Site B. If the witness is in Site A and Site A fails, you lose the witness and two nodes simultaneously, dropping below majority — the cluster stops. The witness must share no failure domain with either site, which is why a cloud witness (neutral, third region) is the standard choice.

8. What do dynamic quorum and dynamic witness do? Dynamic quorum removes a failed node’s vote so the survivors still hold a majority, letting a cluster survive down to the last node in a sequential cascade. Dynamic witness adds/removes the witness’s vote to keep the total number of votes odd, avoiding an even-vote deadlock. Both are on by default; you read them (a node’s DynamicWeight vs NodeWeight) rather than manage them.

9. On a stretch cluster, why pin CAU’s NodeOrder to one site before the other? So the cluster never has one rebooting node in each site at the same time. If a metro link blip hit while a node in each site was down mid-patch, you could drop quorum. Patching Site A fully — confirming health and storage resync — before touching Site B keeps at least one full, healthy site available throughout.

10. During a failover you see SR event 5015 (resync) instead of 1215 (direction change). What does that mean? The failover was not clean: at the moment of failover the destination was behind (or the log was inconsistent), so SR had to run a resync rather than simply reverse direction. It usually follows failing over while replication was not ContinuouslyReplicating. Prevent it by only doing planned failovers when replication is healthy, and by sizing the log so bursts never leave the destination behind.

11. When do you reach for cluster sets instead of a single (stretch) cluster? When you’ve outgrown a single cluster’s ~64-node ceiling, or you need entire-cluster fault isolation (a quorum event or bad patch on one cluster must not affect another), while still wanting unified management and VM mobility across clusters. A cluster set federates multiple failover clusters under a management cluster with a set namespace; member clusters can themselves be stretch clusters.

12. What’s the single command to verify a cluster is ready for CAU, and name two things it checks? Test-CauSetup -ClusterName <name>. It verifies PS remoting/WinRM, the Failover Clustering firewall group, the RemoteShutdown rule CAU needs to reboot nodes, that Automatic Updates isn’t set to auto-install (fighting CAU), and — for self-updating — that the clustered CAU role is installable. Resolve every warning before enabling self-updating.

These map to AZ-800 (Administering Windows Server Hybrid Core Infrastructure) — failover clustering, Storage Spaces Direct, storage migration/replication — and AZ-801 (Configuring Windows Server Hybrid Advanced Services) — high availability, disaster recovery, stretch clustering, and cluster patching. A compact cert-mapping:

Quick check

1. In CAU self-updating mode, how does the cluster manage to patch the node that is hosting the Update Coordinator?
2. You must replicate a volume to a datacenter 900 km away over a 40 ms link with zero tolerance for the application being slow. Which SR mode, and what’s the RPO consequence?
3. A 2+2 stretch cluster’s inter-site link dies cleanly. Both sites go offline. What single design change would have kept one site running?
4. True or false: on an S2D stretch cluster, CAU sometimes waits a long time between nodes, and you should force it to hurry up.
5. During a planned site-failover test you notice a full resync started (SR event 5015). What did you most likely do wrong?

Answers

1. Via a coordinator hand-off: one node hosts the coordinator and patches every other node, then passes the coordinator role to an already-patched node, which then patches the original coordinator node. That hand-off is exactly why self-updating needs no external host.
2. Asynchronous — a 40 ms link is far beyond the ≈ 5 ms synchronous latency budget, and forcing synchronous would make every application write wait 40 ms+ (crippling). The consequence is a non-zero RPO: on a site loss you can lose the in-flight writes not yet shipped to the destination.
3. Add a witness in a third, independent failure domain — a cloud witness — so one site plus the witness holds a majority when the link partitions. (Setting PreferredSite determines which site survives a clean 50/50 split.) The outage happened because 2+2 with no reachable neutral witness leaves neither half with a majority.
4. False. The wait is CAU (correctly) waiting for storage to resync before pulling the next node. Forcing it would risk taking a second node down while virtual disks are still rebuilding, potentially dropping too many copies and taking data offline. Let the resync finish (Get-StorageJob empties).
5. You most likely failed over while replication was not ContinuouslyReplicating — the destination was behind, so SR had to resync rather than cleanly reverse direction (which would show event 1215). Always confirm ContinuouslyReplicating before a planned failover; size the log so bursts don’t leave the destination behind.

Glossary

• Cluster-Aware Updating (CAU) — the feature that orchestrates safe, serial, drain-first, quorum-respecting patching of a failover cluster, one node at a time.
• Update Coordinator — the CAU component that drives an Update Run; runs on an external host (remote-updating) or a cluster node with hand-off (self-updating).
• Self-updating / remote-updating — CAU modes: a clustered role that schedules itself vs an external host driving Invoke-CauRun on demand.
• CAU plug-in — the pluggable “what to install” layer (Microsoft.WindowsUpdatePlugin for WSUS/WUfB/MU; Microsoft.HotfixPlugin for arbitrary hotfixes from a share).
• Pre/post-update script — cluster-wide hooks run once per Update Run as SYSTEM on the coordinator; a non-zero pre-script exit aborts the run.
• Drain / maintenance mode — pausing a node and moving all its clustered roles elsewhere before it reboots (Suspend-ClusterNode -Drain).
• Quorum — the majority of votes a cluster must maintain to stay online; lose it and the cluster stops to prevent split-brain.
• Witness — an extra tie-breaking vote (cloud, file-share, or disk); on a stretch cluster it must live in a third, independent failure domain.
• Dynamic quorum — automatically removes a failed node’s vote so survivors keep a majority through sequential failures.
• Dynamic witness — automatically adds/removes the witness’s vote to keep the total odd.
• Storage Replica (SR) — block-level, single-writer, volume-to-volume replication beneath the filesystem, with a defined RPO.
• Synchronous / asynchronous replication — ack after remote commit (RPO 0, latency-bound) vs ack locally (RPO > 0, distance-tolerant).
• SR log — the volume every write lands in first before draining to the data volume; must be fast (NVMe) and sized to absorb bursts, or replication stalls.
• Test-SRTopology — the cmdlet that samples a real workload and recommends a log size and confirms latency before you create a partnership.
• Stretch cluster — a single failover cluster spanning two sites with asymmetric storage kept in sync by SR, giving automatic site failover at RPO 0.
• Fault domain / site — cluster topology grouping (node/chassis/rack/site) that drives site-aware placement and failover.
• PreferredSite — the cluster property naming the favoured site for role placement and for winning a clean 50/50 partition.
• Cluster set — a federation of multiple failover clusters under a management cluster with a unified namespace and cross-cluster VM mobility.
• Cluster log — the detailed, per-node forensic log generated by Get-ClusterLog; the source of truth for quorum, failover, and resource events.

Next steps

You can now patch a cluster without downtime and stretch it across two sites with zero data loss. Build outward:

• Foundation: Windows Failover Clustering and Storage Spaces Direct: A Production Build — the single-site cluster and software-defined storage this article patches and stretches.
• Related: Hyper-V Live Migration and Replica for Zero-Downtime VM Mobility — the VM-mobility layer that rides on a healthy, patched cluster.
• Related: Resilient File Services with DFS Namespaces and DFS Replication — file-level, multi-master replication for the cases where SR’s single-writer block model doesn’t fit.
• Related: Hardening Windows Server and Building a Reliable WSUS Patch Pipeline — the WSUS update source CAU orchestrates on top of.
• Related: Highly Available DNS and DHCP on Windows Server, End to End — the network services your clustered workloads depend on to stay reachable through a failover.
• Related: SMB Hardening: Credential Guard and Lateral-Movement Defense — securing the SMB surface that SR replication and clustering ride on.