Windows Failover Clustering and Storage Spaces Direct: A Production Build

Hyper-converged infrastructure on Windows Server collapses compute and storage onto the same nodes using Storage Spaces Direct (S2D), with Failover Clustering providing the high-availability fabric. Done right, you get a software-defined storage pool that survives a node loss without a SAN in sight — every node contributes its local NVMe, SSD, and HDD to a shared pool, presented back to the cluster as Cluster Shared Volumes (CSV) that every node reads and writes concurrently. Done wrong, you build a cluster that loses quorum the first time a node reboots for patching, or a pool whose resync storm pins storage latency at 60 ms every time a node comes back. This is the production build I run: understand the machine first — the software storage bus, the cache binding, the resiliency math — validate before you commit, enable S2D in a controlled step, and prove it survives failure before it carries load.

The reference throughout is a four-node cluster, each node with dual 25 GbE RDMA NICs and a mix of NVMe cache and SSD/HDD capacity drives. Commands target Windows Server 2022 (S2D requires Datacenter edition); the mechanics apply to 2019 and 2025 too. What separates this from a “click Enable-ClusterS2D and hope” walkthrough is that S2D fails in specific, diagnosable ways — a cache drive that silently detaches, a mirror volume that can’t rebuild because reserve capacity was consumed, a RoCE fabric that degrades to lossy without a single disk going unhealthy — each a distinct mechanism you can see, confirm, and fix. The tables here enumerate every resiliency type, fault-domain level, drive state, and failure flow, so mid-incident you read the prose once and keep the tables open.

By the end you will stop treating S2D as a black box: why a three-way mirror survives two disk losses but a two-way mirror can’t survive a drive failure during a node reboot, how much of your raw capacity the mirror and reserve consume, what the cache actually caches (and what changes on an all-NVMe box), how the Software Storage Bus (SSB) and Storage Bus Layer (SBL) move every write across RDMA to its copies, and exactly what happens — command by command, state by state — when a drive dies at 3 a.m. or a node drops mid-patch.

What problem this solves

Traditional Windows high availability meant a SAN: a shared array on Fibre Channel or iSCSI, a disk witness on a shared LUN, and a storage team owning the array separately from the server team. It worked, but it concentrated risk and cost in one box, required specialist skills, and made a two-node branch cluster absurdly expensive. S2D removes the SAN entirely — storage is the local drives in each server, pooled in software and mirrored across nodes over the network, so a node can die and its data still lives on the survivors. That is the whole promise of hyper-converged infrastructure: commodity servers with local NVMe/SSD/HDD, SAN-grade resilience in software.

What breaks without understanding the machinery: teams enable S2D, drop VMs onto it, and hit failures that live in layers the SAN world never exposed. A two-way mirror looks fine until a drive fails while a node is rebooting for patches — now two fault domains are unavailable and the volume goes offline. A pool fills to 95% because nobody reserved rebuild capacity, and the next drive failure can’t rebuild — degraded with no path back to healthy. RoCE storage traffic degrades to lossy after a switch firmware update resets DCB, and “storage is slow” tickets chase the disk layer for weeks while every disk reports green. A node is rebooted for firmware without pausing S2D on it first, triggering a full resync of everything it held. None of these are bugs; they are the system behaving exactly as designed against an operator who didn’t know the design.

Who hits this: anyone running Hyper-V or SQL on Windows who wants HA without a SAN — branch offices (2-node clusters), mid-size datacentres (4–8 nodes replacing an aging array), and SQL/VDI shops needing low-latency storage with node fault tolerance. It bites hardest on teams migrating from SAN thinking, because the failure modes, the sizing math (usable ≠ raw), and the maintenance discipline (pause storage before every reboot) are all new. The fix is never “add more disks and hope” — it is knowing the resiliency math, reserving rebuild capacity, and driving maintenance and repair with the exact PowerShell that keeps the pool healthy.

To frame the whole field before the deep dive, here is every layer of an S2D cluster, what lives there, and the failure class it owns:

Learning objectives

By the end of this article you can:

• Explain the S2D stack end to end — the Software Storage Bus, the Storage Bus Layer (SBL) cache, cache and capacity tiers, the storage pool, resilient virtual disks, ReFS on CSV — and name what each layer contributes to performance and fault tolerance.
• Choose the correct resiliency type per volume (two-way mirror, three-way mirror, dual parity, mirror-accelerated parity) from the node count and the workload, and compute the usable capacity and reserve each one costs.
• Design fault domains correctly (drive → node → chassis → rack) so the cluster tolerates the failure unit you actually care about, and understand why S2D is node-fault-tolerant, not just drive-fault-tolerant.
• Reason about the cache precisely — what binds to cache, read vs write caching by media mix, the cache-to-capacity ratio, and what changes on an all-flash or all-NVMe deployment.
• Build the RDMA storage network (RoCEv2 vs iWARP, PFC/DCB, SET switch, SMB Multichannel) and confirm RDMA is live end to end, because most “S2D is slow” incidents are a broken fabric, not a broken disk.
• Validate, create, and enable an S2D cluster in controlled steps (Test-Cluster → New-Cluster -NoStorage → witness → Enable-ClusterStorageSpacesDirect), and size volumes with the correct reserve capacity so rebuilds always have room.
• Operate the cluster safely: storage maintenance mode before every reboot, Cluster-Aware Updating for rolling patches, and read health faults, storage jobs, and the drive/node failure and repair flows command by command.

Prerequisites & where this fits

You should already be comfortable with Failover Clustering fundamentals — that a cluster stays online only while it holds quorum (a majority of votes), that each node has one vote and a witness breaks ties, and that Cluster Shared Volumes let every node access the same volume concurrently. You should know how to run PowerShell remotely (Invoke-Command), read Get-* output, and understand basic storage concepts (RAID-like resilience, mirror vs parity, IOPS vs throughput). Familiarity with SMB, RDMA at a conceptual level, and Hyper-V helps but isn’t assumed.

This sits in the Windows Server storage and availability track. It is the hyper-converged sibling of the SAN-based cluster: where a classic cluster uses shared external storage and a disk witness, S2D pools local drives and uses a cloud or file-share witness. It pairs tightly with Patching Failover Clusters with Cluster-Aware Updating and Stretch Clusters via Storage Replica — CAU is how you patch this cluster without downtime, and Storage Replica is how you stretch it across two sites once the single-site build is bulletproof. If you run Hyper-V on it (the most common case), Hyper-V Live Migration and Replica for Zero-Downtime VM Mobility covers the compute-mobility half. For headless operation, Managing Windows Server Core Headless with Windows Admin Center and PowerShell is how you run these nodes on Server Core (recommended for HCI). Configuration drift across nodes — a killer for S2D, which demands identical firmware/drivers — is addressed in Configuration Management for Windows Server with PowerShell DSC and Ansible.

A quick map of who confirms what during an S2D incident, so you open the right tool first:

Core concepts

Six mental models make every later decision — and every incident — obvious.

S2D is a software-defined virtual SAN built from local drives. There is no shared array. Each node’s local drives are claimed into a single storage pool spanning all nodes. When you create a volume, S2D spreads its data — and its redundant copies — across drives in different nodes, so losing a node loses only one copy. The Software Storage Bus (SSB) stitches every node’s drives into that one pool: it presents remote drives to the local storage stack as if local, and moves I/O between nodes over SMB3 with RDMA. This is why the network is the storage backplane in S2D — a degraded RDMA fabric is a degraded storage array, even though every disk reads healthy.

Resiliency is copies across fault domains, and the fault domain is the node, not the drive. A two-way mirror keeps 2 copies of every slab; a three-way mirror keeps 3. Parity keeps data plus computed parity symbols (like RAID-5/6) for space efficiency at a write-performance cost. S2D places these copies to survive the loss of a whole node (the default fault domain), not merely a drive. That’s the value proposition and the source of the sizing math: a three-way mirror on a 4-node cluster survives any two nodes failing (non-concurrently) because the third copy is always on a survivor — but it costs 3× the raw capacity for the data you store.

The cache is a write-back / read cache in front of the capacity tier, bound automatically. With mixed media, S2D binds the fastest device class (NVMe, then SSD) as cache and the slower class as capacity. Every write lands in cache first (low latency), de-stages to capacity later, and hot reads come from cache. Cache drives show Usage = Journal; capacity drives show Usage = Auto-Select. The cache is not extra usable space — it’s a performance layer sacrificed from the fast drives. On an all-NVMe or single-media box there is no separate cache tier by default (nothing faster to cache with), which changes both sizing and performance.

ReFS on CSV is the mandatory file system, not optional plumbing. S2D volumes use CSVFS_ReFS — ReFS presented through the Cluster Shared Volume layer. ReFS provides integrity streams (checksums that let S2D detect and auto-repair bit rot from a good copy), block cloning (near-instant VHDX operations), and real-time tiering for mirror-accelerated-parity volumes. CSV makes the volume accessible from every node simultaneously — essential for live migration and Scale-Out File Server. NTFS on S2D throws away integrity streams and tiering; ReFS is required.

Reserve capacity is the room a rebuild needs, and it is not optional. When a drive or node fails, S2D rebuilds the missing copies onto the remaining drives to restore full resiliency — but only if there is free space to rebuild into. Fill the pool and the next failure leaves you degraded with no path to healthy until you free space or add drives. Leave one capacity drive’s worth of free space per node (Microsoft’s guidance) as reserve — deducted from usable capacity up front, never “found” later.

The quorum model is the same clustering quorum — but the witness must never be a disk. S2D still needs cluster quorum, and a witness is still the tiebreaker for even-node clusters. But there is no shared disk in an HCI S2D cluster, so a disk witness is impossible — use a cloud witness (a tiny Azure blob) or a file-share witness (a share on an independent server, never a cluster node). Getting this wrong — no witness on a 2- or 4-node cluster — is the single most common self-inflicted outage: quorum is lost the first time a node reboots for patching.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this is the mental model side by side:

The S2D I/O path: SSB, SBL cache, and how a write becomes redundant

To operate S2D you must know what a single write does, because every performance and resiliency behaviour follows from it. Here is one 64 KB write to a three-way-mirror CSV, from the VM down to persistent media on three nodes:

The consequences, which drive every later decision:

• The write latency the VM feels is cache-write + network RTT, not capacity-drive latency — which is why NVMe cache in front of HDD gives SSD-class write latency on cheap bulk storage, and why a broken RDMA path (step 4) spikes latency even though the disks (step 7) are fine.
• Redundancy is created synchronously on the write path. A three-way-mirror write isn’t acknowledged until enough copies are durable — mirror writes cost network bandwidth (3× the data crosses the fabric), so the fabric must be fast.
• Hot reads come from cache; cold reads hit capacity. On HDD-capacity pools, cold random reads are HDD-slow — size the cache to hold the working set.

What binds to cache, by media mix

Cache behaviour is not uniform — it depends on what media types are present. Get this wrong in your hardware order and you either waste NVMe or get no write cache at all:

For all-flash boxes where you still want a cache (usually to spare capacity-drive endurance by absorbing writes on higher-endurance cache drives), force the cache device model before enabling S2D:

# Force NVMe as cache in an all-NVMe box to spare capacity-drive endurance (optional)
Enable-ClusterStorageSpacesDirect -CimSession 'hci-clus01' `
    -CacheDeviceModel 'INTEL SSDPE2KX010T8' `
    -Confirm:$false

The cache-to-capacity ratio matters: the cache must hold the active working set. Microsoft’s guidance is a minimum of ~2 cache drives per node, and capacity drives should be a multiple of the cache drives so binding is even (e.g. 2 cache + 4 capacity, each cache drive fronting 2 capacity drives).

Resiliency: mirror, parity, and mirror-accelerated parity

Resiliency is set per volume — the single decision defining your fault tolerance and usable capacity. This is where SAN thinking fails people: usable capacity is a fraction of raw, and the fraction depends on the resiliency and node count. Study the full matrix:

Why two-way mirror is dangerous in production

Two-way mirror keeps two copies. On a 3+ node cluster it can survive one node — but consider the real failure sequence that bites teams: you reboot node A for patching (one fault domain unavailable), and during that window a drive fails on node B. Slabs that had one copy on node A and one on the failed drive of node B now have zero available copies — the volume goes offline and VMs crash. Patch-window drive failures are exactly when clusters are most stressed, and two-way mirror has no margin for a fault during maintenance. Use it only on genuine 2-node clusters, and even then prefer nested resiliency if you must survive a drive failure while a node is down.

Three-way mirror: the production default

Three copies across three fault domains: any two nodes (or drives) can be lost non-concurrently and every slab still has a copy. This is the safe default for active VM/SQL workloads — mirror write performance (no parity penalty), and enough margin that a drive failure during a patch reboot still leaves one copy. The cost is capacity: 3× the raw for the data you keep, i.e. 33% efficiency.

Parity and mirror-accelerated parity

Dual parity (erasure coding) is space-efficient — like RAID-6 it stores data plus two parity symbols and tolerates two failures, and efficiency improves with more nodes. But it carries a write penalty (read-modify-write of parity on every write), so use it for cold, large, sequentially-written data (backups, archives, media), not busy VMs.

Mirror-accelerated parity (MAP) is the middle path, and the reason ReFS real-time tiering matters. A MAP volume has a mirror tier and a parity tier: writes land in the fast mirror tier (mirror-speed), and ReFS rotates cold data down to parity in the background. You get near-mirror write performance for hot data and near-parity efficiency for the cold majority. The trade-off: size the mirror tier for your write working set, or if it fills faster than ReFS can rotate, writes slow to parity speed. MAP shines for a hot write front over a large cold tail (file servers, backup targets with active ingest).

Computing usable capacity and reserve — worked

The arithmetic on the reference 4-node cluster, assuming each node has 8 × 4 TB capacity SSDs (cache NVMe is separate and doesn’t count): 8 × 4 TB × 4 nodes = 128 TB raw.

Two rules the table encodes:

• Reserve is deducted, not discovered. Leave one capacity drive’s worth of free space per node unallocated so a failed drive can rebuild into it — ~16 TB you never provision here.
• Parity efficiency scales with node count. Dual parity is ~50% efficient on 4 nodes, climbing toward 80% on 8–16 (more data columns per parity stripe). Never quote a single parity efficiency number.

Set the reserve implicitly by not allocating the last drive-per-node of capacity. Confirm the pool has headroom:

# Pool free space must exceed at least one capacity drive per node (the reserve)
Get-StoragePool -IsPrimordial $false |
    Select-Object FriendlyName,
        @{n='Size(TB)';    e={[math]::Round($_.Size/1TB,1)}},
        @{n='Alloc(TB)';   e={[math]::Round($_.AllocatedSize/1TB,1)}},
        @{n='Free(TB)';    e={[math]::Round(($_.Size-$_.AllocatedSize)/1TB,1)}}

Fault domains: surviving the unit you actually care about

By default S2D uses the node (StorageScaleUnit) as the fault domain — copies are placed so any one node can fail without data loss. But you can tell S2D about larger physical boundaries — chassis and rack — so it survives the loss of a whole chassis or rack. This matters when nodes share a chassis (blade enclosures) or a rack (shared power/top-of-rack switch): if all copies of a slab sit in nodes in the same rack and that rack loses power, you lose data even though no single node failed.

The fault-domain hierarchy and what each buys you:

Inspect the fault-domain topology (before enabling S2D, so placement is right from the start):

# Show the fault-domain tree the cluster currently sees
Get-StorageFaultDomain -Type StorageScaleUnit |
    Select-Object FriendlyName, FaultDomainType, HealthStatus

# For rack/chassis awareness, assign nodes to racks/chassis BEFORE enabling S2D
Set-ClusterFaultDomain -Name 'hci-n1' -Parent 'Rack-A'
Set-ClusterFaultDomain -Name 'hci-n2' -Parent 'Rack-A'
Set-ClusterFaultDomain -Name 'hci-n3' -Parent 'Rack-B'
Set-ClusterFaultDomain -Name 'hci-n4' -Parent 'Rack-B'

# Create the rack fault domains first
New-ClusterFaultDomain -Name 'Rack-A' -Type Rack
New-ClusterFaultDomain -Name 'Rack-B' -Type Rack

Rack/chassis awareness changes copy placement, which changes the minimum node count for a given resiliency: a three-way mirror surviving a rack loss needs its three copies in three racks — with only two racks, that’s impossible. Design fault domains against the failure unit you’re insuring, and confirm the node/rack count supports the resiliency you want.

Networking: RDMA, DCB, SET, and SMB Multichannel

S2D moves all east-west storage traffic over SMB3, the performance-critical path. It must run over RDMA (Remote Direct Memory Access) for low latency and CPU offload — without it, the CPU cost of moving storage between nodes at scale is crippling. There are two RDMA flavours, and the choice has real operational consequences.

RoCE vs iWARP

The pragmatic take from production: iWARP unless you have a reason not to. RoCE’s marginal edge is real only when PFC/DCB is configured and stays configured end to end — and the most common S2D storage incident is a RoCE fabric that silently degrades to lossy after a switch firmware update or config drift, producing “slow storage” that chases the disk layer for weeks. iWARP has no switch DCB dependency, so there’s nothing to drift. If you inherit a RoCE cluster, treat the switch DCB state as part of the cluster and monitor RDMA error counters.

DCB / PFC for RoCE (only if you run RoCE)

RoCEv2 needs lossless Ethernet: Priority Flow Control (PFC) to prevent drops on the storage priority, and DCB (Data Center Bridging) to reserve bandwidth — identical on every NIC and every switch port in the storage path.

# --- RoCE only: configure DCB/PFC on each node (must match the physical switches) ---
Invoke-Command -ComputerName $nodes -ScriptBlock {
    # SMB Direct traffic on priority 3; a class for it
    New-NetQosPolicy -Name 'SMB'    -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
    New-NetQosPolicy -Name 'Cluster' -Cluster -PriorityValue8021Action 7

    # Enable PFC on priority 3 (the SMB Direct priority); disable on others
    Enable-NetQosFlowControl  -Priority 3
    Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7

    # Reserve 50% bandwidth for SMB Direct
    New-NetQosTrafficClass -Name 'SMB' -Priority 3 -BandwidthPercentage 50 -Algorithm ETS

    # Apply DCB to the storage NICs and turn on hardware flow control
    Enable-NetAdapterQos -InterfaceAlias 'SMB1','SMB2'
}

The knobs that must match host-and-switch, and what happens if they don’t:

SET switch and storage vNICs

Most builds are converged: the same dual NICs carry storage and tenant/management traffic behind a Switch Embedded Teaming (SET) virtual switch. SET is the only supported teaming for S2D (classic LBFO isn’t supported for RDMA storage). Build the SET switch, add host vNICs for storage, and pin each vNIC to a physical adapter so each RDMA path stays on its own NIC:

Invoke-Command -ComputerName $nodes -ScriptBlock {
    # Create the SET switch across both storage NICs (no host OS adapter yet)
    New-VMSwitch -Name 'ConvergedSwitch' `
        -NetAdapterName 'SMB1','SMB2' `
        -EnableEmbeddedTeaming $true `
        -AllowManagementOS $false

    # Two host vNICs for SMB storage traffic
    Add-VMNetworkAdapter -ManagementOS -Name 'SMB_1' -SwitchName 'ConvergedSwitch'
    Add-VMNetworkAdapter -ManagementOS -Name 'SMB_2' -SwitchName 'ConvergedSwitch'

    # Pin each storage vNIC to a separate physical NIC (affinity → predictable RDMA paths)
    Set-VMNetworkAdapterTeamMapping -ManagementOS `
        -VMNetworkAdapterName 'SMB_1' -PhysicalNetAdapterName 'SMB1'
    Set-VMNetworkAdapterTeamMapping -ManagementOS `
        -VMNetworkAdapterName 'SMB_2' -PhysicalNetAdapterName 'SMB2'

    # Put the storage vNICs on their VLANs and enable RDMA on them
    Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName 'SMB_1' -Access -VlanId 711
    Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName 'SMB_2' -Access -VlanId 712
    Enable-NetAdapterRdma -Name 'vEthernet (SMB_1)','vEthernet (SMB_2)'
}

Confirm RDMA is actually live — the step everyone skips

This is where most “S2D is slow” tickets are resolved. Enabling RDMA on the adapter is not the same as RDMA being used for storage traffic:

# 1. Adapters report RDMA-capable AND enabled
Get-NetAdapterRdma | Select-Object Name, Enabled, MaxQueuePairs

# 2. SMB sees RDMA-capable interfaces
Get-SmbClientNetworkInterface | Where-Object RdmaCapable |
    Select-Object FriendlyName, RdmaCapable, RssCapable

# 3. THE proof: SBL (the S2D storage instance) is actively using RDMA
Get-SmbMultichannelConnection -SmbInstance SBL |
    Select-Object ServerName, ClientRdma, ServerRdma, ClientInterfaceIndex

# 4. Under load, SMB Direct connection count should be non-zero
Get-SmbConnection -SmbInstance SBL | Measure-Object   # activity present

# 5. RoCE only: verify DCB/PFC actually applied
Get-NetAdapterQos | Where-Object Enabled | Select-Object Name, OperationalFlowControl
Get-NetQosFlowControl

The three SMB instances and what each is for:

RoCEv2 requires PFC and DCB consistent on both NICs and switches; without it end to end, RoCE falls back to lossy — retransmits and latency spikes with every disk still green. iWARP needs no switch DCB, which is why many teams prefer it. Whichever you run, Get-SmbMultichannelConnection -SmbInstance SBL showing ClientRdma/ServerRdma = True proves the storage bus is accelerated.

Finally, tell live migration to ride SMB (hence RDMA) with sane concurrency:

Invoke-Command -ComputerName $nodes -ScriptBlock {
    Set-VMHost -VirtualMachineMigrationPerformanceOption SMB `
        -MaximumVirtualMachineMigrations 2 `
        -MaximumStorageMigrations 2
}

Quorum and the witness on S2D

S2D uses the same cluster quorum model as any failover cluster, with one hard rule: the witness is never a disk. A cluster stays online only while it holds quorum — a majority of votes; each node gets one vote, a witness gets one as a tiebreaker.

Dynamic quorum recalculates the working majority as nodes leave gracefully, so a cluster can survive to a last node standing if nodes drop sequentially. Dynamic witness toggles the witness vote on/off to keep the total odd — why a 4-node cluster gives the witness a vote and an odd cluster may not. Both are on by default; understand them when reading logs, because the vote a node holds changes over time.

Dynamic quorum protects against sequential failures, not simultaneous ones. If two of four nodes die at the exact same instant before the cluster recomputes votes, you still lose quorum. Witness placement is what saves you in that window.

Witness type selection on S2D:

Validate, create, and enable S2D — the controlled sequence

Never enable S2D without a clean validation and a staged build. The sequence: install features → validate → create the cluster with no storage → configure the witness → enable S2D → verify cache → create volumes. One-shot builds hide which step failed.

Step 1 — Features on every node

$nodes = 'hci-n1','hci-n2','hci-n3','hci-n4'

Invoke-Command -ComputerName $nodes -ScriptBlock {
    Install-WindowsFeature -Name Failover-Clustering, FS-FileServer, Hyper-V, Data-Center-Bridging `
        -IncludeManagementTools -Restart:$false
}
# Reboot nodes that requested it (Hyper-V install typically does), then continue.

Step 2 — Validate with Test-Cluster (the report Microsoft support demands)

Test-Cluster with the Storage Spaces Direct suite exercises S2D readiness — identical drive firmware, unclaimed drives, consistent NIC drivers, network communication. Treat every Failed as a hard stop and review every Warning.

Test-Cluster -Node $nodes `
    -Include 'Storage Spaces Direct','Inventory','Network','System Configuration' `
    -ReportName 'C:\ClusterReports\precheck'
# Open the generated HTML report. Any 'Failed' = do not proceed.

The validation checks that matter most for S2D, and what a failure means:

If a drive shows a stale partition table from a prior build, clear it before clustering — destructive, triple-check the host list:

Invoke-Command -ComputerName $nodes -ScriptBlock {
    Get-Disk | Where-Object { $_.IsSystem -eq $false -and $_.IsBoot -eq $false } |
        Clear-Disk -RemoveData -RemoveOEM -Confirm:$false -ErrorAction SilentlyContinue
}

Step 3 — Create the cluster with -NoStorage

Create the cluster without claiming storage, so S2D can claim drives in a controlled later step. Supply a static management IP:

New-Cluster -Name 'hci-clus01' `
    -Node $nodes `
    -StaticAddress '10.20.0.50' `
    -NoStorage

Step 4 — Configure the witness (cloud or file share)

# Cloud witness (preferred): needs a storage account name + one access key
Set-ClusterQuorum -Cluster 'hci-clus01' `
    -CloudWitness `
    -AccountName 'sthciwitness01' `
    -AccessKey '<storage-account-key>' `
    -Endpoint 'core.windows.net'

# OR — air-gapped: a file-share witness on an INDEPENDENT server (never a node)
Set-ClusterQuorum -Cluster 'hci-clus01' `
    -FileShareWitness '\\fsw-srv01\hci-clus01-witness'

Confirm quorum and dynamic behaviour:

Get-ClusterQuorum -Cluster 'hci-clus01' | Format-List *
(Get-Cluster).DynamicQuorum   # expect 1
Get-ClusterNode | Select-Object Name, State, DynamicWeight, NodeWeight
# DynamicWeight = vote the node currently holds; NodeWeight = eligible to vote at all. Both 1 = healthy.

Step 5 — Enable Storage Spaces Direct

One cmdlet claims eligible drives, builds the pool, and (with mixed media) auto-binds the fastest device as cache:

Enable-ClusterStorageSpacesDirect -CimSession 'hci-clus01' -Confirm:$false

Verify the pool and the cache bindings:

Get-StoragePool -IsPrimordial $false |
    Select-Object FriendlyName, Size, AllocatedSize, HealthStatus

# Cache drives show Usage=Journal; capacity drives show Usage=Auto-Select
Get-PhysicalDisk |
    Select-Object FriendlyName, MediaType, Usage, Size, HealthStatus |
    Sort-Object Usage

Step 6 — Create volumes with the right resiliency and reserve

Create volumes as CSV ReFS so every node can access them concurrently. PhysicalDiskRedundancy 2 is what makes a mirror three-way (two copies can be lost):

# Three-way mirror volume for VMs (production default)
New-Volume -FriendlyName 'vm-prod-01' `
    -FileSystem CSVFS_ReFS `
    -StoragePoolFriendlyName 'S2D on hci-clus01' `
    -ResiliencySettingName 'Mirror' `
    -PhysicalDiskRedundancy 2 `
    -Size 4TB

# Mirror-accelerated parity volume for bulk/archive (mirror tier + parity tier)
New-Volume -FriendlyName 'archive-01' `
    -FileSystem CSVFS_ReFS `
    -StoragePoolFriendlyName 'S2D on hci-clus01' `
    -StorageTierFriendlyNames 'Performance','Capacity' `
    -StorageTierSizes 1TB, 9TB

# STOP allocating once free space ≈ one capacity drive per node remains (the reserve)

The New-Volume parameters that define resiliency, decoded:

Deploy clustered roles

With storage and networking in place, deploy workloads onto CSV paths.

Hyper-V VMs as clustered roles

New-VM -Name 'app-vm-01' `
    -MemoryStartupBytes 16GB `
    -Generation 2 `
    -Path 'C:\ClusterStorage\vm-prod-01' `
    -NewVHDPath 'C:\ClusterStorage\vm-prod-01\app-vm-01\disk0.vhdx' `
    -NewVHDSizeBytes 128GB

# Make it highly available (clustered)
Add-ClusterVirtualMachineRole -VirtualMachine 'app-vm-01' -Cluster 'hci-clus01'

Scale-Out File Server (SOFS) for app data

For SQL-over-SMB or Hyper-V-over-SMB, a Scale-Out File Server gives active-active share access from every node:

Add-ClusterScaleOutFileServerRole -Name 'sofs-app01' -Cluster 'hci-clus01'
New-SmbShare -Name 'sqldata' -Path 'C:\ClusterStorage\vm-prod-01\Shares\sqldata' `
    -FullAccess 'CONTOSO\SQLEngineAccounts' -ContinuouslyAvailable $true

Tune failover policy

The key knobs: how many failures are tolerated in a window before the cluster gives up, and whether a role fails back automatically. Leave failback disabled in most cases — it can move a VM back to a healthy-but-cold-cache node:

$grp = Get-ClusterGroup -Name 'app-vm-01'
$grp.FailoverThreshold = 2      # max failovers ...
$grp.FailoverPeriod    = 6      # ... within this many hours
$grp.AutoFailbackType  = 0      # 0 = no failback (prevents ping-pong / cold-cache moves)

The failover-policy knobs and how to reason about them:

Patching: Cluster-Aware Updating and storage maintenance mode

Patching is where the quorum design and storage discipline earn their keep. Cluster-Aware Updating (CAU) orchestrates a rolling update: drain one node, maintenance its storage, patch, reboot, return it, wait for resync, move to the next — the cluster stays online throughout. Enable self-updating mode to patch on a schedule:

Add-CauClusterRole -ClusterName 'hci-clus01' `
    -DaysOfWeek Sunday `
    -WeeksOfMonth 2 `
    -IntervalWeeks 4 `
    -MaxFailedNodes 1 `
    -MaxRetriesPerNode 2 `
    -EnableFirewallRules `
    -Force

Patch on demand:

Invoke-CauRun -ClusterName 'hci-clus01' `
    -CauPluginName 'Microsoft.WindowsUpdatePlugin' `
    -MaxFailedNodes 1 -MaxRetriesPerNode 2 -Force

For manual maintenance (firmware, hardware, driver swaps), you must pause S2D on the node first — the single most important S2D operational habit. Skipping it triggers a full resync of everything the node held the moment it goes down. The correct sequence:

# 1. Drain clustered roles off the node
Suspend-ClusterNode -Name 'hci-n2' -Drain

# 2. Put the node's STORAGE into maintenance mode (this is the step people skip)
Get-StorageFaultDomain -Type StorageScaleUnit |
    Where-Object FriendlyName -eq 'hci-n2' |
    Enable-StorageMaintenanceMode

# 3. ... do the work: firmware, driver, reboot ...

# 4. Take storage OUT of maintenance mode
Get-StorageFaultDomain -Type StorageScaleUnit |
    Where-Object FriendlyName -eq 'hci-n2' |
    Disable-StorageMaintenanceMode

# 5. Resume the node and let it re-absorb roles (no auto-failback)
Resume-ClusterNode -Name 'hci-n2' -Failback Immediate

# 6. WAIT for resync to finish before touching the next node
Get-StorageJob   # must be empty (or only 'Regeneration' completing) before proceeding

Why storage maintenance mode matters:

Always confirm storage is fully healthy (Get-StorageJob empty) before touching the next node. Patching a second node while the first is still resyncing puts two fault domains in flux at once — the exact condition three-way mirror protects you from, squandered.

Health, faults, and storage jobs — reading the machine

S2D exposes its state through health faults, physical-disk states, virtual-disk states, and storage jobs — reading each is the difference between a five-minute triage and a two-hour guess.

Health faults

# The cluster's own health report — run this first when something's wrong
Get-StorageSubSystem -FriendlyName 'Clustered*' | Debug-StorageSubSystem
Get-StorageSubSystem -FriendlyName 'Clustered*' | Get-StorageHealthReport
Get-HealthFault   # active faults with descriptions + recommended actions

Physical disk states

Virtual disk (volume) states

Storage jobs — how you watch a rebuild

# Every async pool operation (repair, regeneration, rebalance) with % complete
Get-StorageJob | Select-Object Name, JobState, PercentComplete, BytesProcessed, BytesTotal

Drive failure and node failure: the exact flows

This is what you’re really buying S2D for — and where knowing the flow command-by-command turns a 3 a.m. panic into a checklist.

A single drive fails

Locate and replace a failed drive:

# Identify the failed drive and blink its slot LED so the tech pulls the right one
$bad = Get-PhysicalDisk | Where-Object HealthStatus -ne 'Healthy'
$bad | Select-Object FriendlyName, SerialNumber, MediaType, HealthStatus, OperationalStatus
$bad | Enable-PhysicalDiskIndication

# If S2D hasn't already retired it, retire it explicitly (drains its data)
$bad | Set-PhysicalDisk -Usage Retired

# After replacement the new drive auto-joins; confirm and rebalance
Get-PhysicalDisk | Where-Object Usage -eq 'Auto-Select' | Measure-Object
Get-StoragePool -IsPrimordial $false | Optimize-StoragePool
Get-StorageJob   # watch the rebalance complete

A node fails (unplanned)

# Confirm the cluster survived
Get-ClusterNode | Select-Object Name, State, StatusInformation
Get-ClusterQuorum | Format-List *
Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus

# Node coming back: resume it (delta resync only)
Resume-ClusterNode -Name 'hci-n3' -Failback Immediate
Get-StorageJob   # small resync

# Node permanently dead: evict it, then S2D rebuilds full copies onto survivors' free space
Remove-ClusterNode -Name 'hci-n3' -Cluster 'hci-clus01' -Force
Get-StorageJob   # larger rebuild — ensure reserve/free space exists!

The reason three-way mirror is the production default is here: while a node is down (fault domain 1), a drive failure on a second node (fault domain 2) still leaves a third copy — on two-way mirror that same second failure is data loss. And the reason reserve capacity is mandatory is step 5b: a permanent node loss must rebuild full copies into free space, and without reserve the pool is stuck degraded.

Architecture at a glance

Picture the reference four-node cluster as three stacked planes that every I/O passes through. The compute plane is the four Hyper-V hosts running your VMs; each VM’s VHDX lives on a CSV path like C:\ClusterStorage\vm-prod-01, and because CSV presents the volume to every node at once, a VM can live-migrate between hosts without moving a byte of storage. Below it, the storage plane is a single pool spanning all four nodes: each node contributes local NVMe (bound as cache, Usage=Journal) and SSD/HDD (bound as capacity, Usage=Auto-Select), and from that pool you carve resilient virtual disks — a three-way mirror for busy VMs (three copies, one per node, survives two non-concurrent node losses) and a mirror-accelerated-parity volume for bulk data (mirror tier for hot writes, parity tier for the cold majority). Threading the two together is the network plane: the Software Storage Bus carrying east-west traffic over SMB3 + RDMA across dual 25 GbE NICs behind a SET switch. Off to the side sits the cloud (or file-share) witness — one vote that keeps quorum sane on the even node count, never a shared disk because there is none.

Follow one write to see the whole system move: a VM on node 1 issues a 64 KB write; ReFS stamps an integrity checksum; Spaces maps its three copies to drives on nodes 1, 2, 3; the SSB ships two copies over RDMA to the peers; each node’s SBL cache (NVMe) durably acknowledges; once a quorum of copies is safe in cache, the write is acknowledged — at NVMe-cache latency, not HDD latency — and de-stages to capacity later. Now overlay failure: pull a capacity drive and the affected virtual disk goes Degraded while a repair job rebuilds the missing copy into reserve space, VMs untouched; drop a whole node and dynamic quorum plus the witness keep the cluster online, VMs restart on survivors, and volumes stay Degraded (not Detached) because a third copy always survives. That’s the entire mental model — compute on top, one pooled storage layer beneath, an RDMA bus stitching them, a witness holding the tie — and every performance number and failure outcome in this article falls out of it.

Real-world scenario

Meridian Health Systems ran clinical VMs — an EHR database, PACS imaging file servers, and ~110 supporting VMs — on an aging Fibre Channel SAN nearing end of support, a steep renewal quote and a single blast radius. The infrastructure team of five moved to a four-node S2D cluster: each node 2 × 1.6 TB NVMe cache + 8 × 3.84 TB SSD capacity (~123 TB raw), dual 25 GbE NICs, Windows Server 2022 Datacenter. They chose three-way mirror for the EHR and general VMs and a mirror-accelerated-parity volume for PACS (a large, mostly-cold dataset with active ingest). The build went in clean — Test-Cluster passed, New-Cluster -NoStorage, a cloud witness, then Enable-ClusterStorageSpacesDirect — sized to ~26 TB usable after reserving one capacity drive per node. RDMA was iWARP, a deliberate call to avoid DCB risk on a small team.

Then the first patch cycle delivered the first self-inflicted lesson. An engineer rebooted hci-n2 for firmware without Enable-StorageMaintenanceMode. The drives went missing abruptly, S2D began treating them as failed, and when the node returned sixty seconds later it triggered a full resync of everything it held. Storage p99 write latency spiked from sub-millisecond to 35–50 ms for ~20 minutes; the EHR team logged a “slowness” ticket. Nothing was broken — every disk Healthy, quorum held, no data lost — but the resync storm was avoidable. The fix was procedural: a runbook that always drains the node, enables storage maintenance mode, does the work, disables it, resumes, and waits for Get-StorageJob to clear before the next node — plus Cluster-Aware Updating in self-updating mode so the sequence couldn’t be skipped.

The second lesson was capacity discipline. Six months in, PACS growth had quietly filled the pool to ~92%, eating the reserve. When a capacity SSD failed on hci-n4, the repair couldn’t fully rebuild — no free space to recreate the missing copies — and the pool sat Degraded with a stalled repair. They freed space by archiving cold imaging, expanded capacity, and added a hard alert at 80% utilization plus a synthetic check on Get-StorageJob duration. The cluster has since survived a genuine dead-node event (a failed mainboard on hci-n1) with zero VM data loss — VMs restarted on survivors, volumes went Degraded not Detached, and the delta resynced cleanly. The lesson on the wall: “S2D doesn’t fail like a SAN — it fails in resiliency math and maintenance discipline. Reserve the rebuild space, pause storage before every reboot, and never patch the next node until the last one is fully healthy.”

The incident timeline, because the order is the lesson:

Advantages and disadvantages

S2D’s model — pooled local drives, resilience in software over an RDMA fabric — both earns HCI’s benefits and creates its distinctive failure modes. Weigh it honestly:

The model is right when you want SAN-grade resilience without a SAN, run Hyper-V/SQL, and have the discipline (or automation) to reserve rebuild space and pause storage before reboots. It bites hardest on teams that treat it like a SAN — over-fill the pool, patch without maintenance mode, run RoCE without owning DCB, or pick two-way mirror to “save space.” Every disadvantage above is manageable — but only if you know it exists, which is the point of this article.

Hands-on lab

You can’t do a real four-node build without hardware, but you can build a working S2D cluster in nested virtualization on a single Hyper-V host (or an Azure VM that supports nested virt, e.g. Dv3/Ev3). This lab stands up a 2-node cluster with virtual data disks, enables S2D, creates a mirror volume, and injects a drive failure to watch the repair — the exact flow, at zero hardware cost. Run from an elevated PowerShell on the Hyper-V host.

Nested S2D is for learning the cmdlets and flows only — not supported for production, and the “RDMA” is emulated. The value is muscle memory on the exact commands.

Step 1 — Create two nested VMs with data disks (on the Hyper-V host).

$vms = 'lab-s2d-1','lab-s2d-2'
foreach ($v in $vms) {
    New-VM -Name $v -MemoryStartupBytes 4GB -Generation 2 -Path 'D:\LabVMs' `
        -NewVHDPath "D:\LabVMs\$v\os.vhdx" -NewVHDSizeBytes 60GB
    Set-VM -Name $v -ProcessorCount 4
    # Nested virtualization + MAC spoofing (needed inside the guest cluster)
    Set-VMProcessor -VMName $v -ExposeVirtualizationExtensions $true
    Get-VMNetworkAdapter -VMName $v | Set-VMNetworkAdapter -MacAddressSpoofing On
    # Four "capacity" data disks per node
    1..4 | ForEach-Object {
        $p = "D:\LabVMs\$v\data$_.vhdx"
        New-VHD -Path $p -SizeBytes 30GB -Dynamic | Out-Null
        Add-VMHardDiskDrive -VMName $v -Path $p
    }
}

Install Windows Server 2022 Datacenter in each VM and join them to a test domain (S2D requires domain membership). Expected: two VMs, each with four unformatted 30 GB data disks.

Step 2 — Inside the guests: install features (run against both nodes).

$nodes = 'lab-s2d-1','lab-s2d-2'
Invoke-Command -ComputerName $nodes -ScriptBlock {
    Install-WindowsFeature Failover-Clustering, FS-FileServer -IncludeManagementTools
}

Expected: Success = True, RestartNeeded = No for the clustering feature.

Step 3 — Validate and create the cluster with no storage.

Test-Cluster -Node $nodes -Include 'Storage Spaces Direct','Inventory','Network','System Configuration' `
    -ReportName 'C:\ClusterReports\lab'
New-Cluster -Name 'lab-clus' -Node $nodes -StaticAddress '10.99.0.50' -NoStorage

Expected: a validation report (review it), then a cluster object lab-clus with both nodes Up.

Step 4 — Configure a file-share witness (simplest for a lab).

# Create a share on a separate machine (your host or a third VM), then:
Set-ClusterQuorum -Cluster 'lab-clus' -FileShareWitness '\\host\lab-witness'
Get-ClusterQuorum | Format-List *

Expected: QuorumResource shows the file-share witness; (Get-Cluster).DynamicQuorum = 1.

Step 5 — Enable S2D and create a two-way mirror volume.

Enable-ClusterStorageSpacesDirect -CimSession 'lab-clus' -Confirm:$false
Get-StoragePool -IsPrimordial $false | Select-Object FriendlyName, Size, HealthStatus

# Two-node cluster → two-way mirror (PhysicalDiskRedundancy 1)
New-Volume -FriendlyName 'lab-vol' -FileSystem CSVFS_ReFS `
    -StoragePoolFriendlyName 'S2D on lab-clus' `
    -ResiliencySettingName 'Mirror' -PhysicalDiskRedundancy 1 -Size 20GB
Get-VirtualDisk | Select-Object FriendlyName, ResiliencySettingName, HealthStatus, OperationalStatus

Expected: a pool spanning both nodes (HealthStatus=Healthy), and a volume mounted at C:\ClusterStorage\lab-vol with HealthStatus=Healthy.

Step 6 — Inject a drive failure and watch the repair.

# On the Hyper-V host: yank a data disk from node 1 to simulate a drive failure
Get-VMHardDiskDrive -VMName 'lab-s2d-1' | Select-Object -First 1 | Remove-VMHardDiskDrive

# Inside the cluster: observe the effect
Get-PhysicalDisk | Where-Object HealthStatus -ne 'Healthy'      # the "failed" drive
Get-VirtualDisk  | Select-Object FriendlyName, OperationalStatus # expect Degraded (still serving)
Get-StorageJob                                                   # a repair may run if capacity allows

Expected: one physical disk unhealthy/lost, the virtual disk Degraded (not Detached — data still available), and the volume still mounted and readable. Validation: you’ve proven that a drive loss degrades but doesn’t down the volume, and you drove every core cmdlet (Enable-ClusterStorageSpacesDirect, New-Volume, Get-PhysicalDisk, Get-VirtualDisk, Get-StorageJob) on the exact flow you’d run in production.

Step 7 — Teardown.

# Inside: remove the cluster
Remove-Cluster -Cluster 'lab-clus' -Force -CleanupAD
# On the host: delete the lab VMs and their disks
$vms | ForEach-Object { Stop-VM $_ -TurnOff -Force -EA SilentlyContinue; Remove-VM $_ -Force }
Remove-Item 'D:\LabVMs' -Recurse -Force

The lab steps mapped to what each proves:

Common mistakes & troubleshooting

The failure modes that actually bite, as a scannable table first, then the detail for the ones that hurt most:

The expanded reasoning for the ones that cost the most time:

2. “S2D is slow” with every disk healthy — the RDMA/DCB trap. Root cause: The storage bus rides RDMA; if a RoCE fabric drifts to lossy (a switch firmware update reset DCB/PFC) or a storage path drops, latency spikes on the storage bus while every physical disk reads Healthy. Confirm: Get-SmbMultichannelConnection -SmbInstance SBL shows ClientRdma/ServerRdma = False or fewer connections than expected; Get-NetAdapterStatistics | Select Name, RdmaConnectionErrors climbs during spikes; Get-NetQosFlowControl / Get-NetAdapterQos show PFC not applied. Fix: Reapply DCB/PFC to the drifted switch and pin switch firmware in the change baseline; monitor RDMA error counters and PFC pause-frame asymmetry. If DCB discipline is a recurring problem on a small team, iWARP removes the switch dependency entirely.

3. A reboot causes a huge resync storm. Root cause: The node was rebooted without Enable-StorageMaintenanceMode, so S2D saw its drives vanish abruptly and, on return, resynced everything the node held instead of just the delta. Confirm: Get-StorageJob is full of Regeneration jobs immediately after a reboot; storage latency spikes cluster-wide. Fix: Always drain the node and enable storage maintenance mode before any reboot; disable it after; wait for Get-StorageJob to clear before the next node. Use CAU so this can’t be skipped.

5. A drive failed but the pool can’t rebuild — stuck Degraded. Root cause: No reserve capacity — the pool was too full for S2D to rebuild the missing copies into free space. Confirm: Get-StoragePool -IsPrimordial $false shows free space less than one capacity drive per node; the repair Get-StorageJob stalls. Fix: Free space (archive/delete) or add drives so the rebuild can complete; going forward, reserve one capacity drive’s worth of free space per node and alert at 80% utilization.

12. Events 1135 / 1177 in the FailoverClustering log. Root cause: 1135 = a node was removed from active membership (usually a network/communication problem); 1177 = the cluster lost quorum (the one you never want — a quorum design or witness failure); 1069/1146 = resource/role failures. Confirm: Get-WinEvent -LogName 'Microsoft-Windows-FailoverClustering/Operational' -MaxEvents 100 | Where LevelDisplayName -in 'Error','Warning'. Fix: For 1135, fix the storage/heartbeat network; for 1177, fix the quorum design (add/repair the witness) — 1177 almost always means the vote math was wrong for the failure that occurred.

Best practices

• Always configure a witness — cloud preferred, file-share for air-gapped, never disk. It costs nothing, turns even-node clusters from fragile to resilient, and prevents the most common S2D outage.
• Use three-way mirror for production VM/SQL workloads. Two-way can’t survive a fault during maintenance; reserve it (and nested resiliency) for genuine 2-node clusters.
• Reserve rebuild capacity — one capacity drive per node, deducted up front — and alert at 80% utilization. A full pool can’t rebuild after a failure.
• Pause S2D storage before every reboot: Suspend-ClusterNode -Drain → Enable-StorageMaintenanceMode → work → Disable-StorageMaintenanceMode → Resume-ClusterNode.
• Patch with Cluster-Aware Updating in self-updating mode, and never touch the next node until Get-StorageJob is empty.
• Validate with Test-Cluster (Storage Spaces Direct suite) before creating the cluster — and keep the report; Microsoft support will ask for it.
• Create the cluster with -NoStorage, then Enable-ClusterStorageSpacesDirect separately so drive claiming and cache binding are controlled and verifiable.
• Use ReFS on CSV (CSVFS_ReFS) for every volume, never NTFS — integrity streams, block cloning, and MAP tiering all depend on it.
• Prove RDMA is live on the SBL instance (Get-SmbMultichannelConnection -SmbInstance SBL). On RoCE, treat switch DCB state as part of the cluster; if DCB discipline is hard, use iWARP.
• Keep firmware and drivers identical across nodes — manage it as code (DSC/Ansible) and pin firmware in the change baseline.
• Disable auto-failback so a recovered node doesn’t pull warm workloads onto a cold cache.
• Run failure-injection drills (node loss, drive pull, storage-path loss, witness loss) before production load — an untested HA design is a hope, not a guarantee.
• Monitor the right signals: pool utilization, Get-StorageJob duration, virtual-disk and physical-disk health, node state, and RDMA error counters — not just “is a disk healthy.”

The alerts worth wiring before the next incident — leading indicators, not “cluster down”:

Security notes

• Domain-join and use least-privilege administration. Nodes must be domain-joined; use scoped admin accounts (or group Managed Service Accounts for service contexts) rather than Domain Admins for daily operation.
• Protect the witness credentials. A cloud witness’s Azure storage key belongs in a secret manager, rotated, never in source control. A file-share witness must live on an independent hardened server (not a node), with the share ACL scoped to the cluster name object.
• Encrypt data at rest where required. S2D supports cluster-aware BitLocker on CSV volumes; for regulated data (the Meridian case), enable it so pulled drives are useless outside the cluster.
• Isolate the storage network. Put SMB/SBL traffic on a dedicated VLAN restricted to the nodes; consider SMB encryption/signing for sensitive SOFS shares (weigh CPU cost vs RDMA offload). The storage bus should never be routable from the general network.
• Harden SMB on SOFS shares with the same discipline as Hardening SMB and Enabling Credential Guard to Block Lateral Movement — signing, encryption for sensitive shares, no SMBv1.
• Guard the management plane. Run nodes on Server Core, manage via Windows Admin Center and remote PowerShell, and keep RDP off the storage network.
• Keep firmware and OS patched — CAU makes rolling patching downtime-free, so there’s no excuse for lagging updates on an HA cluster.

Cost & sizing

The bill and sizing math differ from a SAN because you buy servers, not an array — and usable capacity is a fraction of raw that depends on the resiliency choice.

• Servers dominate; there’s no SAN line item. The economic case is replacing a SAN + maintenance with commodity servers you’d partly buy anyway. Windows Server Datacenter licensing (per physical core, all cores) is the notable software cost — S2D is Datacenter-only, so budget the premium per node.
• Size for usable-after-reserve, not raw. Three-way mirror is 33% efficient; on ~123 TB raw you get ~40 TB before reserve and ~26 TB after reserving one drive per node. Parity/MAP recover efficiency for cold data at a write cost. Compute raw × efficiency − reserve for your node count before ordering drives.
• Cache drives are a performance tax, not capacity. NVMe cache (2+ per node) buys latency, not usable space — budget it separately and size it to the working set.
• Network is a real cost. Dual 25 GbE RDMA NICs plus capable switches (DCB-capable for RoCE) are part of the build. iWARP lets you use simpler switches, trading a little peak performance for lower cost and risk.
• Grow by adding nodes/drives, not forklift upgrades. S2D scales 2→16 nodes; each added node adds capacity, compute, fault-domain headroom, and parity efficiency — a genuine advantage over SAN refresh cycles.

For the reference 4-node cluster: ~123 TB raw → ~26 TB usable on three-way mirror after reserve (mix in a MAP volume to reclaim efficiency for cold data). The dominant recurring cost is Datacenter licensing per core across all four nodes; hardware is capex amortized over the refresh cycle. The cost/sizing drivers:

Interview & exam questions

1. What is Storage Spaces Direct and how does it differ from a SAN-based cluster? S2D pools the local drives of every cluster node into a single pool and mirrors/erasure-codes data across nodes over RDMA — there is no shared SAN. A SAN-based cluster uses external shared storage and a disk witness; an S2D cluster has no shared disk, so it uses a cloud or file-share witness and provides node fault tolerance in software.

2. Explain two-way vs three-way mirror, and why two-way is risky in production. Two-way keeps 2 copies (50% efficiency) and survives one node/drive loss; three-way keeps 3 (33%) and survives two non-concurrent losses. Two-way has no margin for a fault during maintenance — a drive failing while a node reboots can drop slabs to zero copies and take the volume offline. Use three-way for production; two-way only on genuine 2-node clusters.

3. What is the cache tier, what binds to it, and what does it cache? With mixed media, S2D binds the fastest class (NVMe, then SSD) as cache (Usage=Journal) in front of the slower capacity class. On NVMe+HDD or SSD+HDD it’s a read+write cache; on NVMe+SSD it’s write-only; on all-NVMe/all-SSD there’s no cache tier by default. It absorbs writes and serves hot reads — a performance layer, not extra usable capacity.

4. Why enable storage maintenance mode before rebooting an S2D node? Without it, S2D sees the drives vanish abruptly and, on return, triggers a full resync of everything the node held — a cluster-wide latency storm. Enable-StorageMaintenanceMode marks the drives temporarily offline, so S2D holds off rebuilding and resyncs only the delta on return.

5. What is reserve capacity and what happens without it? Free pool space (guidance: one capacity drive per node) kept so S2D can rebuild missing copies after a failure. Without it, a failure leaves the pool degraded with no path to healthy — nowhere to recreate the copies. It’s deducted from usable capacity up front, not discovered later.

6. Why does an S2D cluster need a witness, and why can’t it be a disk? Even-node clusters (2 or 4) split votes 50/50 when a node is lost — not a majority — so they lose quorum without a witness tiebreaker. It can’t be a disk witness because there is no shared disk; use a cloud witness (Azure blob) or a file-share witness on an independent server.

7. RoCE vs iWARP — the operational trade-off? Both deliver RDMA. RoCEv2 needs lossless Ethernet (PFC + DCB) end to end; DCB drift silently degrades storage. iWARP runs over standard TCP with no switch DCB requirement — slightly less peak performance, far less operational risk. Pick iWARP unless you have strong network engineering and DCB tooling.

8. What is mirror-accelerated parity and when do you use it? A single ReFS volume with a mirror tier (fast writes) and a parity tier (efficient cold storage); ReFS rotates cold data down to parity. It gives near-mirror write performance for hot data and near-parity efficiency for the cold majority — ideal for an active write front over a large cold tail (file servers, backup targets). Size the mirror tier for the write working set or writes hit parity speed.

9. A four-node three-way-mirror cluster has a node down for patching, then a drive fails on a second node. What happens? The cluster survives with no data loss: three copies across three nodes means with one node down (copy 1 gone) and a drive failing elsewhere (copy 2 gone), a third copy survives. Volumes go Degraded, not Detached. On two-way mirror the same sequence is data loss — which is why three-way is the production default.

10. How do you validate and build an S2D cluster in the correct order? Install-WindowsFeature on all nodes → Test-Cluster -Include 'Storage Spaces Direct',... and fix every failure → New-Cluster -NoStorage → configure the witness → Enable-ClusterStorageSpacesDirect → verify cache → New-Volume -FileSystem CSVFS_ReFS ... with correct resiliency and reserve. Staging it (-NoStorage, then enabling S2D separately) makes each step verifiable.

11. What file system do S2D volumes use and why does it matter? ReFS on CSV (CSVFS_ReFS): integrity streams (auto bit-rot repair from a good copy), block cloning (fast VHDX operations), and real-time tiering for MAP; CSV makes the volume accessible from every node at once (live migration, SOFS). NTFS throws away integrity streams and tiering — ReFS is required.

12. Which cluster event IDs matter most on S2D? 1135 — node removed from membership (usually network/heartbeat). 1177 — cluster lost quorum (a quorum/witness design failure; the one you never want). 1069/1146 — resource/role failures. A 1177 almost always means the vote math was wrong for the failure that occurred — fix the witness/quorum design.

These map to Windows Server Hybrid Administrator (AZ-800/AZ-801) — manage Windows Server storage, high availability, and failover clustering, including S2D, CSV, quorum, and CAU — and general Windows Server storage/HA competency. A compact cert-mapping for revision:

Quick check

1. Your four-node cluster has no witness. A single node reboots for a patch. What happens to the cluster, and what’s the fix?
2. On an NVMe + HDD S2D deployment, which drives are cache and which are capacity, and does the cache serve reads, writes, or both?
3. True or false: to fix an S2D pool that’s Degraded after a drive failure and can’t rebuild, you should scale out by adding another node’s worth of volumes.
4. You need to reboot a node for firmware. List the correct sequence of S2D-specific steps before and after the reboot.
5. Every disk in the pool reads Healthy, but storage latency is spiking under load and users complain. On a RoCE-based cluster, what do you check first?

Answers

1. With no witness, a four-node cluster drops to exactly 50% of votes when one node is lost — not a majority — so the cluster loses quorum and goes offline. The fix is to configure a cloud witness or file-share witness (Set-ClusterQuorum -CloudWitness ...), which adds a tiebreaker vote so the survivors hold a majority. Never a disk witness on S2D.
2. The NVMe drives are cache (Usage=Journal) and the HDDs are capacity (Usage=Auto-Select). On NVMe+HDD the cache serves both reads and writes — it absorbs writes (write-back) and serves hot reads, hiding HDD latency in both directions.
3. False. Scaling out volumes does nothing for a stuck rebuild. The pool can’t rebuild because there’s no free space (reserve) to rebuild into — the fix is to free space or add drives/capacity so the repair can recreate the missing copies, and to reserve one capacity drive per node going forward.
4. Suspend-ClusterNode -Drain <node> (move roles off) → Enable-StorageMaintenanceMode on that node’s StorageScaleUnit (so S2D holds off rebuilding) → do the firmware/reboot → Disable-StorageMaintenanceMode → Resume-ClusterNode → wait for Get-StorageJob to clear before touching the next node.
5. The RDMA/DCB fabric, not the disks. Check Get-SmbMultichannelConnection -SmbInstance SBL for ClientRdma/ServerRdma = True, and Get-NetAdapterStatistics | Select Name, RdmaConnectionErrors for rising errors during the spikes — a RoCE switch whose DCB/PFC drifted (e.g. after a firmware update) makes the storage bus lossy while every disk stays green. Reapply DCB/PFC (or move to iWARP).

Glossary

• Storage Spaces Direct (S2D) — Windows Server (Datacenter) software-defined storage that pools the local drives of cluster nodes and mirrors/erasure-codes data across nodes over RDMA; the storage half of hyper-converged infrastructure.
• Software Storage Bus (SSB) — the virtual-SAN fabric that presents every node’s drives to the local storage stack as if local and moves I/O between nodes over SMB3 + RDMA; the storage backplane.
• Storage Bus Layer (SBL) cache — the write-back / read cache the SSB manages on the fastest media; absorbs writes and serves hot reads. Its SMB instance is SBL.
• Cache drive — the fastest media class (NVMe/SSD) bound as cache (Usage=Journal); a performance layer, not usable capacity.
• Capacity drive — the slower media class (SSD/HDD) that holds the data (Usage=Auto-Select); your usable-before-resiliency raw capacity.
• Storage pool — the single collection of all eligible drives across all nodes; one per S2D cluster.
• Virtual disk / Space — a resilient volume carved from the pool; resiliency (mirror/parity) is set here.
• Two-way / three-way mirror — resiliency that keeps 2 / 3 copies of every slab across fault domains (50% / 33% efficiency); three-way survives two non-concurrent losses.
• Dual parity — erasure-coded resiliency (data + 2 parity, RAID-6-like) that tolerates two failures with efficiency that improves with node count; has a write penalty (cold data).
• Mirror-accelerated parity (MAP) — a single ReFS volume with a mirror tier (fast writes) and a parity tier (efficient cold storage), with ReFS rotating cold data down to parity.
• Fault domain — the failure unit S2D tolerates: PhysicalDisk, StorageScaleUnit (node, the default), StorageChassis, StorageRack, StorageSite.
• Reserve capacity — free pool space kept for rebuilds (≈ one capacity drive per node); without it a failure can’t rebuild.
• CSV (Cluster Shared Volume) — a volume accessible from every node simultaneously; enables live migration and SOFS.
• ReFS (Resilient File System) — the S2D file system, providing integrity streams (auto bit-rot repair), block cloning, and real-time tiering; used as CSVFS_ReFS.
• RDMA (RoCEv2 / iWARP) — Remote Direct Memory Access for low-latency, low-CPU east-west storage traffic; RoCE needs lossless Ethernet (PFC/DCB), iWARP runs over standard TCP.
• SET (Switch Embedded Teaming) — the only supported NIC teaming for S2D RDMA storage; a virtual switch that teams the physical NICs and hosts the storage vNICs.
• DCB / PFC — Data Center Bridging / Priority Flow Control; the lossless-Ethernet configuration RoCE requires end to end on NICs and switches.
• Storage maintenance mode — pauses S2D on a node’s scale unit before a reboot so its drives aren’t treated as failed, avoiding a full resync storm.
• Cluster-Aware Updating (CAU) — orchestrated rolling patching that drains, patches, and returns nodes one at a time so the cluster stays online.
• Storage job — an async pool operation (Repair/Regeneration, Rebalance, Optimize, Retiring) tracked via Get-StorageJob.
• Witness (cloud / file share) — the quorum tiebreaker for even-node clusters; on S2D it’s a cloud (Azure blob) or file-share witness, never a disk.

Next steps

You can now design, build, and operate a hyper-converged S2D cluster and reason about its failures. Build outward:

• Next: Patching Failover Clusters with Cluster-Aware Updating and Stretch Clusters via Storage Replica — deepen CAU and add cross-site DR by stretching S2D with Storage Replica.
• Related: Hyper-V Live Migration and Replica for Zero-Downtime VM Mobility — the compute-mobility half of an HCI cluster.
• Related: Managing Windows Server Core Headless with Windows Admin Center and PowerShell — run these nodes on Server Core, the recommended HCI footprint.
• Related: Configuration Management for Windows Server with PowerShell DSC and Ansible — keep firmware, drivers, and config identical across nodes, which S2D demands.
• Related: Resilient File Services with DFS Namespaces and DFS Replication — present S2D-hosted shares resiliently to clients.
• Related: Hardening SMB and Enabling Credential Guard to Block Lateral Movement — secure the SMB/SOFS surface your cluster exposes.