Advanced LVM: Thin Provisioning, Snapshots, and Cache Pools

Classic (“thick”) LVM allocates every extent up front, so a 500 GiB logical volume consumes 500 GiB the instant you create it. That is wasteful when you run dozens of volumes that are mostly empty, and its old copy-on-write snapshots are a footgun: size the snapshot too small and it invalidates silently the moment it fills, taking your point-in-time copy with it. Thin provisioning fixes both. Blocks are allocated on first write from a shared pool, snapshots cost only metadata until something diverges, and dm-cache lets you front a slow spinning volume with a fast NVMe tier. The price is a new and exotic way to cause an outage: you can now promise more space than you physically own, and walk the pool straight into 100% full while every volume on it remounts read-only at once.

This article turns LVM from “the thing the installer set up” into a tool you operate deliberately under pressure. We treat the whole stack as one layered machine — PV → VG → LV, with a thin pool and its metadata as a fourth layer, a snapshot as a fifth, and a cache device bolted on the side — and walk every layer option by option: what it is, the choices, the default, when to pick which, the trade-off, the limit, and the specific way it bites at 3 a.m. The danger sections (a full pool, exhausted metadata, a runaway snapshot, an unflushed writeback cache yanked off a dying disk) are the centerpiece, because those are the incidents that page you. Everything is real: real lvcreate/lvconvert/pvmove flags, real lvm.conf keys, real dmsetup status output, real dmesg strings you grep for mid-incident.

Everything below assumes lvm2 2.03+ and a kernel with dm-thin-pool, dm-cache and dm-writecache (any current RHEL 8/9, Debian 12, Ubuntu 22.04+, or SUSE 15 SP4+). Confirm the tooling and the kernel modules before anything else:

lvm version | head -1                       # expect 2.03.x or newer
modprobe dm_thin_pool dm_cache dm_writecache
lsmod | grep -E 'dm_thin_pool|dm_cache|dm_writecache'

What problem this solves

Storage is the one resource where over-provisioning is both the smartest and the most dangerous thing you can do. A database team asks for 500 GiB “to be safe” and uses 60. A VM fleet provisions 40 root disks at 80 GiB each — 3.2 TiB promised — against 900 GiB of real usage. Thick allocation forces you to buy all 3.2 TiB up front or constantly play capacity Tetris. Thin provisioning lets you buy what you actually use and grow the backing store as real consumption climbs, exactly as every SAN and cloud block store does under the hood. The mechanism is not exotic; operating it safely is the skill.

What breaks without that skill is specific and brutal. A thin pool that hits 100% data does not politely slow down — every thin volume it backs starts erroring on write, filesystems remount read-only or corrupt, and because it is one shared pool, the blast radius is every tenant on it at once. Metadata exhaustion is the same outage from a direction nobody watches: the data area can read 60% while a snapshot-heavy workload silently fills the tiny metadata LV, which is far harder to grow under pressure. A legacy thick snapshot sized at 5 GiB against a volume that takes 6 GiB of writes doesn’t warn you — it drops to Invalid and your backup source is gone. A writeback cache on a single NVMe that dies takes your unflushed acknowledged writes with it. None of these are bugs; they are the documented behaviour of a powerful tool used without guardrails.

Who hits this: anyone running virtualization on local disk (Proxmox, oVirt, libvirt LVM-thin pools), container hosts with the old devicemapper thin driver, database fleets wanting cheap point-in-time snapshots for restore testing, and any server adding a fast SSD tier in front of bulk HDD without rebuilding the storage stack. The reflex that fails them is the cloud reflex — “the disk filled, make it bigger.” On a thin pool, adding capacity is the right emergency move only if the VG has free extents; if it doesn’t, autoextend silently does nothing, and the real fix might be reaping a runaway snapshot, not buying disk. This article is about knowing which.

Learning objectives

By the end of this article you can:

• Explain the full LVM object model — PV, VG, PE, LV, plus the thin pool’s data LV (_tdata) and metadata LV (_tmeta) — and read any of them from pvs/vgs/lvs -a.
• Create thin pools with a deliberately sized metadata LV and chunk size, and articulate the chunk-size trade-off (metadata footprint and IO overhead vs snapshot CoW granularity and sequential throughput).
• Reason about overcommit quantitatively: virtual vs physical capacity, safe ratios per workload, and why you alert on the pool’s Data%/Meta%, never the per-LV percentages.
• Distinguish thick (old) CoW snapshots from thin snapshots in mechanism, cost, failure mode, and merge/rollback behaviour — and avoid the “snapshot fills and invalidates” trap.
• Accelerate a slow volume with dm-cache (cachepool/cachevol) or dm-writecache, choose writethrough vs writeback as a durability decision, and detach a cache safely with --splitcache.
• Build RAID LVs (raid1, raid5, raid6, raid10) directly in LVM, and migrate live data off a failing disk with pvmove with zero downtime.
• Survive the incidents: recover a full thin pool, repair exhausted/corrupt metadata with lvconvert --repair, and configure dmeventd autoextend and external alerting so you never discover a full pool from dmesg.
• Operate the whole stack from the real CLI and lvm.conf — autoextend thresholds, issue_discards, thin_check, monitoring — and run a symptom→cause→confirm→fix playbook against the failure modes that actually happen.

Prerequisites & where this fits

You should be comfortable on a Linux shell as root, know what a block device and a partition are, and have formatted and mounted at least one filesystem (mkfs.ext4/mkfs.xfs, /etc/fstab). Helpful but not required: a working mental model of copy-on-write, and exposure to the device-mapper (dmsetup) layer that LVM is built on. You do not need prior LVM depth — Section 5 (Core concepts) builds the object model from the ground up — but if classic LVM (pvcreate/vgcreate/lvcreate -L/lvextend) is already muscle memory, you can skim to Section 7 where thin provisioning starts.

Where this sits: LVM is the volume-management layer between raw disks (or RAID arrays, or multipath/SAN LUNs) and your filesystems. It pairs naturally with software RAID — you can stack LVM on top of an md array, or use LVM’s own RAID LV types and skip mdadm entirely; if you run the former, read Building Resilient Linux Storage with mdadm Software RAID for the array layer underneath. The IO-scheduler and sysctl tuning that affects how a cached or thin volume actually performs is covered in Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers. The dmeventd and fstrim.timer automation here is driven by systemd, so Mastering systemd: Units, Timers, Resource Control, and Service Hardening is the companion for the scheduling side. On the Windows side, the equivalent thin-provisioning-plus-tiering story lives in Windows Failover Clustering and Storage Spaces Direct: A Production Build.

A quick map of who owns what when an LVM storage incident lands, so you escalate to the right layer rather than blaming the filesystem:

Core concepts

Six mental models make every later section obvious. Internalize these and the failure modes stop being surprises.

LVM stacks three abstractions, and thin adds a fourth. A Physical Volume (PV) is a raw block device — a disk (/dev/sdb), a partition, an md array, or a multipath device — stamped with an LVM label and divided into fixed-size Physical Extents (PEs), 4 MiB by default. A Volume Group (VG) pools PVs into one allocation arena of interchangeable PEs. A Logical Volume (LV) is a named sequence of extents you mkfs and mount. Thin provisioning inserts a fourth layer: a thin pool is a special LV with two hidden sub-LVs — a large data LV (<pool>_tdata) holding real blocks, and a small metadata LV (<pool>_tmeta) holding the btrees that record which pool block backs which logical address for every thin volume and snapshot. Thin volumes draw blocks from that pool on demand.

Thin means “allocate on write,” and that is the whole game. A thin LV created with -V 200G advertises 200 GiB but consumes nothing until written; each write allocates a chunk from the pool’s data area. This lets ten 100 GiB thin volumes share a 500 GiB pool — as long as their combined real usage stays under 500 GiB. The instant pool data hits 100%, writes to every thin volume block or error. That is the central risk you manage for the rest of this article; monitoring, autoextend, overcommit ratios and snapshot reaping all exist to keep real consumption below physical size.

Metadata is small, separate, and just as fatal when full. _tmeta is tiny relative to data — often hundreds of MiB for a multi-hundred-GiB pool — but if it fills, the pool can’t record new mappings and wedges as hard as data exhaustion, sometimes worse because it’s harder to grow under pressure. Metadata consumption scales with the number of chunks and snapshots, not bytes written, so a snapshot-heavy workload can exhaust metadata while the data area looks healthy. You watch two numbers, always: Data% and Meta%.

Snapshots come in two different mechanisms. A thick (classic) snapshot of a normal LV pre-allocates a fixed CoW exception store; old blocks are copied in on first write, and when that store fills, the snapshot is dropped to Invalid and lost — silently, no second chance. A thin snapshot is just another thin volume sharing the origin’s blocks; it costs only metadata at creation, allocates pool blocks only where origin or snapshot diverge, and cannot “invalidate” — it consumes pool space like any thin volume. Thin is strictly better for modern use, but makes the pool fill faster, looping back to the central risk.

Caching is a separate volume bolted onto a slow LV. dm-cache (the LVM front-end calls it a cache pool or cache volume) puts NVMe/SSD in front of a slow origin LV and promotes hot blocks transparently — the app sees one cached LV, no API change. dm-writecache is different: a pure write buffer for sync writes only, no read caching, no hotspot promotion. The writethrough vs writeback choice for dm-cache is a durability decision, not a performance dial: writeback acks from the fast device and can lose unflushed writes if that single device dies.

Almost everything is online, and growing is always safe. Extending an LV (and its filesystem), extending a thin pool’s data or metadata, adding a PV, and evacuating a disk with pvmove all happen while volumes stay mounted. Shrinking is the dangerous direction — shrink the filesystem first and the LV second or you truncate live data; XFS cannot shrink at all, and thin pools cannot shrink, period. Build runbooks around growth and migration, never shrinkage.

The vocabulary in one table

Pin down every moving part before the deep sections. The glossary at the end repeats these for lookup; this is the mental model side by side:

The LVM object model and the device-mapper truth underneath

Before thin provisioning, you must be fluent in the classic stack, because every thin object is built on it and every recovery command operates at this level. Initialize devices as PVs, pool them into a VG, carve LVs:

# Initialize two whole disks as PVs (wipes any existing signature — be sure)
pvcreate /dev/sdb /dev/sdc

# Pool them into a VG with the default 4 MiB extent size
vgcreate vg_data /dev/sdb /dev/sdc

# Inspect: PE size, total/free extents
vgdisplay vg_data | grep -E 'PE Size|Total PE|Free  PE|Alloc PE'

# A plain (thick) LV: 100 GiB allocated up front
lvcreate -L 100G -n classic01 vg_data
mkfs.xfs /dev/vg_data/classic01

Every LVM device is ultimately a device-mapper target — LVM is a friendly management layer over dm. When something is too broken for lvs to show it, dmsetup still can:

dmsetup ls --tree          # the full dm dependency tree (pool -> tdata/tmeta -> PVs)
dmsetup status vg_data-thinpool-tpool   # raw thin-pool counters the kernel exposes

The three reporting commands you live in are pvs, vgs, and lvs (with -a to reveal hidden sub-LVs). Their most useful columns, and what each tells you:

The lv_attr field is a dense status string you learn to read at a glance. Position 1 is the volume type; the most common values you will see:

Position 4 carries the open/health state and position 6 the target type; the bits that matter most in an incident are the health flags. Decode them with the dedicated columns rather than counting characters:

lvs -o name,lv_attr,lv_health_status,raid_sync_action vg_data

Two lvm.conf and tooling realities to know up front, because they affect every command above:

Building thin pools: chunk size, metadata sizing, and the spare

You can create a thin pool in one command and let LVM auto-size metadata, but on anything production-grade you size both the metadata LV and the chunk size deliberately. Metadata exhaustion is just as fatal as data exhaustion and harder to fix under pressure, and the chunk size is a one-time decision that shapes the pool’s whole performance and snapshot economics.

The quick way — fine for a lab, risky for production because you don’t control chunk or metadata:

# 400 GiB thin pool; LVM auto-calculates metadata and picks a chunk size
lvcreate --type thin-pool -L 400G -n thinpool vg_data

The deliberate way — explicit chunk, explicit metadata, then assemble:

# 1) Create the data LV with an explicit chunk size.
#    Larger chunks = less metadata + better sequential throughput,
#    but coarser snapshot CoW (more write amplification on small writes).
lvcreate -L 400G --chunksize 128K -n thinpool_tdata vg_data

# 2) Create the metadata LV explicitly (size it for growth + many snapshots).
lvcreate -L 1G -n thinpool_tmeta vg_data

# 3) Assemble the pool from the two LVs.
lvconvert --type thin-pool \
  --poolmetadata vg_data/thinpool_tmeta \
  vg_data/thinpool_tdata

# 4) Rename to the clean pool name.
lvrename vg_data/thinpool_tdata thinpool

Metadata size scales with pool size and chunk size, roughly pool_size / chunk_size * 64 bytes of mapping data, with overhead. The defaults and limits you must respect:

Choosing a chunk size is a genuine trade-off. Small chunks (64 KiB) keep snapshot CoW cheap but balloon metadata and add per-IO overhead; large chunks (256 KiB–1 MiB) suit big sequential workloads but waste space when snapshots diverge by tiny amounts. The decision table:

The metadata-size math, worked for common pool sizes at the default 64 KiB chunk (round up and add headroom for snapshots):

Always carry the metadata spare. vgcreate/pool creation reserves a [lvol*_pmspare] LV by default so lvconvert --repair has somewhere to write a rebuilt copy of metadata. Do not delete it to reclaim a gigabyte — you will regret it the one time you need --repair and there is nowhere to land the rebuild. Confirm it exists with lvs -a vg_data | grep pmspare.

Provisioning thin volumes and the discipline of overcommit

Now carve thin volumes from the pool. The -V (virtual size) flag is the size the volume advertises to the filesystem; it is not what it consumes:

# Three thin volumes, each advertising 200 GiB, from a 400 GiB pool.
# Total virtual = 600 GiB on 400 GiB physical: 1.5x overcommit.
lvcreate --type thin -V 200G --thinpool thinpool -n app01 vg_data
lvcreate --type thin -V 200G --thinpool thinpool -n app02 vg_data
lvcreate --type thin -V 200G --thinpool thinpool -n db01  vg_data

mkfs.ext4 /dev/vg_data/app01
mount /dev/vg_data/app01 /srv/app01

The numbers you watch are the pool’s Data% and Meta% — the pool row aggregates real consumption across every thin LV and snapshot it backs:

lvs -o name,lv_size,data_percent,metadata_percent,pool_lv vg_data

#   LV        LSize    Data%  Meta%  Pool
#   thinpool  400.00g  41.27  3.10
#   app01     200.00g  18.44         thinpool
#   app02     200.00g  62.10         thinpool
#   db01      200.00g  43.05         thinpool

The per-volume Data% is the fraction of each thin LV’s virtual size that has been written; the pool’s Data% (41.27) is the share of physical pool capacity in use — the only number that predicts survival. Each per-LV figure can look modest while the pool quietly approaches full. Alert on the pool row, never the volumes. The distinction, made explicit because it is the most common monitoring mistake:

Overcommit is the whole point of thin provisioning and the whole risk. Pick a ratio per workload and document it; the ratios that experienced teams actually run:

Two behaviours make reclaim work — without them, deleting files inside a thin volume does not return blocks to the pool, and the pool only ever grows:

# Option A (recommended): periodic batch TRIM via systemd timer — gentle on perf.
systemctl enable --now fstrim.timer

# Option B: inline discard at the FS layer (more overhead per delete).
#   ext4/xfs mount option 'discard', OR set issue_discards=1 in lvm.conf
#   so lvremove/lvreduce also TRIM the layers below.

How reclaim flows down the stack, and what each setting controls:

Snapshots: thick CoW vs thin, the merge dance, and the fill trap

Snapshots are where thin earns its keep and where the worst silent data loss in classic LVM hides. The two mechanisms are not variations on a theme — they are different machines.

Thin snapshots — cheap, safe, the modern default

A thin snapshot is just another thin volume that initially shares every block with its origin; it allocates new pool blocks only where origin or snapshot subsequently diverges. It costs only metadata at creation, needs no pre-sized CoW area, and cannot “invalidate” — it consumes pool space like any thin volume and that is its only cost:

# Snapshot of a live volume. Thin snapshots are created with activation
# skipped by default (so they don't auto-mount on boot); -kn + -ay activates one.
lvcreate --snapshot --name app01_snap_pre_deploy vg_data/app01
lvchange -kn -ay vg_data/app01_snap_pre_deploy

To roll the origin back to the snapshot’s state, merge it. A merge of an in-use origin is deferred until the next deactivation/reactivation, so for a mounted root or busy volume you unmount (or reboot) to let it complete:

umount /srv/app01
lvconvert --merge vg_data/app01_snap_pre_deploy   # origin becomes the snapshot's contents
mount /dev/vg_data/app01 /srv/app01               # snapshot LV is consumed by the merge

A snapshot is also a clean backup source — snapshot, mount read-only, back up the stable image, drop it:

lvcreate -s -kn -ay --name app01_bak vg_data/app01
mount -o ro /dev/vg_data/app01_bak /mnt/snap
# ... back up /mnt/snap ...
umount /mnt/snap
lvremove -y vg_data/app01_bak

Thick snapshots — the legacy mechanism and its silent-loss trap

A classic snapshot of a normal (non-thin) LV pre-allocates a fixed copy-on-write exception store. Every first write to an origin block copies the old block into that store. The store has a size you pick at creation — and when it fills, the kernel marks the snapshot Invalid and it is gone, with no warning beyond a dmesg line and the snapshot’s Snap% hitting 100:

# Thick CoW snapshot — note the explicit -L (the CoW store size).
lvcreate -s -L 10G -n classic01_snap vg_data/classic01

# Watch the CoW store fill. If Snap% reaches 100, the snapshot INVALIDATES.
lvs -o name,origin,snap_percent,lv_attr vg_data

The side-by-side that should drive every snapshot decision:

The thick-snapshot fill trap, made concrete: a thick snapshot sized at 10 GiB against an origin that subsequently takes 11 GiB of first writes invalidates at the 10 GiB mark — your point-in-time copy vanishes the moment you most rely on it (a long backup, a risky migration). The defences:

Crucially for thin: snapshots count against pool capacity. A long-lived thin snapshot of a write-heavy volume can consume as much pool space as a full second copy, because every changed origin block forces a new allocation to preserve the snapshot’s view. This is the single most common root cause of a runaway full pool. Time-box snapshots and reap them; treat every snapshot as a pool consumer with a deadline. The lvm.conf autoextend keys for thick snapshots (distinct from the thin-pool keys in Section 11):

LVM cache: dm-cache, dm-writecache, and the durability decision

Caching fronts a slow origin LV with a fast device and serves hot data from it. LVM exposes two mechanisms with different goals — read+write hotspot caching (dm-cache) and a pure sync-write buffer (dm-writecache). Pick by what you are actually trying to fix.

dm-cache via a cache pool

dm-cache promotes frequently-accessed blocks to the fast device and serves them there. Build a cache pool (data + its own metadata) on the fast PV, then attach it to the slow origin:

# Add the fast device to the VG, then build a cache pool on it.
vgextend vg_data /dev/nvme0n1
lvcreate --type cache-pool -L 64G \
  --poolmetadatasize 128M \
  --cachemode writethrough \
  -n cachepool vg_data /dev/nvme0n1

# Attach the cache pool to a slow origin LV -> one cached LV.
lvconvert --type cache \
  --cachepool vg_data/cachepool \
  vg_data/db01

The mode is a durability decision, not a performance dial:

Inspect and detach. Always --splitcache (which flushes dirty blocks to origin) rather than yanking the NVMe:

lvs -o name,cache_mode,cache_total_blocks,cache_used_blocks,cache_dirty_blocks vg_data

# Detach cleanly: flush dirty blocks to origin, then remove the cache.
lvconvert --splitcache vg_data/db01

With writeback, an unflushed detach — or a dead single cache device — is data loss of every acknowledged-but-not-destaged write. Detach the cache before a pvmove of the origin’s slow device, and never pull the cache device physically without --splitcache first.

The cache also has a replacement policy and tunables, set at attach or later with lvchange --cachepolicy/--cachesettings:

dm-writecache — when only write latency is the problem

If reads are already fine (big page cache, read-mostly) but fsync-heavy writes (a database WAL, a busy mail spool) are slow on spinning disk, dm-writecache is the sharper tool: it buffers writes only on the fast device and writes back to origin lazily, with no read caching and no hotspot logic.

# Build a writecache and attach it (note: cachevol, not cache-pool, for writecache)
lvcreate --type writecache -L 16G \
  --cachevol /dev/nvme0n1 \
  --cachesettings 'high_watermark=50 low_watermark=45' \
  -n db01 vg_data /dev/sdb   # origin on slow disk, writecache on nvme

Choosing between the two mechanisms:

A worked decision: a Postgres box on HDD with a 200 GiB hot index and bursty commits wants dm-cache (read promotion of the hot index and writeback if the cache is mirrored). A logging box that streams sequential writes and rarely re-reads wants dm-writecache to absorb fsync latency without wasting fast capacity caching reads it will never do.

RAID logical volumes and the device-mapper RAID stack

LVM can build redundant LVs itself via dm-raid (the same kernel RAID engine as md), so you can get mirroring and parity inside a VG without a separate mdadm array. Each RAID LV is composed of hidden rimage/rmeta sub-LVs you see with lvs -a.

# Mirror across two PVs (n.b. --type raid1, the modern replacement for the old "mirror")
lvcreate --type raid1 -m 1 -L 100G -n mirror01 vg_data /dev/sdb /dev/sdc

# RAID5 across >=3 PVs (single-parity)
lvcreate --type raid5 -i 3 -L 300G -n raid5lv vg_data

# RAID6 across >=4 PVs (double-parity)
lvcreate --type raid6 -i 4 -L 400G -n raid6lv vg_data

# RAID10 (stripe of mirrors) across >=4 PVs
lvcreate --type raid10 -i 2 -m 1 -L 200G -n raid10lv vg_data

The RAID LV types, their minimum PVs, and what they buy:

Operating RAID LVs — sync status, scrubbing, and replacing a failed leg:

# Sync / scrub status
lvs -o name,lv_attr,raid_sync_action,sync_percent,raid_mismatch_count vg_data

# Trigger a scrub (check) or a repair (fix mismatches)
lvchange --syncaction check  vg_data/raid6lv
lvchange --syncaction repair vg_data/raid6lv

# Replace a failed device's leg with a spare PV
lvconvert --replace /dev/sdd vg_data/raid6lv /dev/sde

mdadm vs LVM-RAID — when to use which underneath your volumes:

A thin pool’s data and metadata can each live on a RAID LV for redundancy — e.g. _tmeta on a raid1 so a single disk loss can’t destroy the mapping btrees while leaving data intact. Losing metadata while data survives is the worst kind of half-failure; mirroring metadata is cheap insurance.

Online resize, pvmove migration, and the shrink rules

LVM’s headline capability is doing all of this live. Growing is always safe; shrinking is dangerous and order-sensitive.

# Grow an LV and its ext4/xfs filesystem in one shot (--resizefs calls resize2fs/xfs_growfs)
lvextend -L +50G --resizefs vg_data/app02

Reducing is where people destroy data. You must shrink the filesystem first, then the LV — -r does both in the safe order, but XFS cannot shrink at all:

# REDUCE (ext4 only): -r shrinks the FS before the LV. ALWAYS fsck first.
umount /srv/app01
e2fsck -f /dev/vg_data/app01
lvreduce -r -L 120G vg_data/app01

The resize rules in one reference — internalize the “XFS can’t shrink” and “thin pool can’t shrink” lines:

To evacuate a failing disk with zero downtime, pvmove relocates its extents to other PVs in the VG while the volumes stay mounted. It is restartable if interrupted (it journals progress):

# Move EVERYTHING off a dying PV onto the rest of the VG, then remove it.
pvmove -i 5 /dev/sdc          # -i 5 = progress every 5 s
vgreduce vg_data /dev/sdc     # drop the now-empty PV from the VG
pvremove /dev/sdc             # wipe the LVM label

# Move only one LV's extents (targeted migration)
pvmove -n vg_data/db01 /dev/sdc /dev/sdd

pvmove gotchas worth knowing before you run it on production:

The full thin pool: the incident that pages you

This is the section to read twice. When a thin pool’s Data% or Meta% reaches 100%, every thin volume it backs starts erroring on write; filesystems remount read-only or, worse, corrupt. Because it is one shared pool, the blast radius is every volume on it. Diagnose immediately:

lvs -o name,data_percent,metadata_percent,lv_health_status vg_data
dmesg | grep -iE 'thin|out-of-data-space|metadata operation failed|needs_check'
dmsetup status vg_data-thinpool-tpool   # raw used/total data and metadata blocks

The fastest non-destructive fix is to add capacity — if the VG has free extents or you can add a PV in seconds:

# If no free extents remain, add a disk to the VG first.
vgextend vg_data /dev/sdd

# Relieve data pressure, and/or metadata pressure, independently.
lvextend -L +100G vg_data/thinpool                    # grow the data area
lvextend --poolmetadatasize +512M vg_data/thinpool    # grow metadata (up to ~16 GiB)

The fastest destructive relief is deleting expendable snapshots, which immediately returns their diverged blocks to the pool — which is exactly why a runaway snapshot is the most common root cause of a full pool:

lvs -o name,lv_attr,data_percent,origin vg_data | grep -i ' V'   # find snapshots
lvremove -y vg_data/app01_snap_pre_deploy                        # reclaim its blocks

If the pool is wedged (transactional inconsistency, needs_check flag, won’t activate), do an offline metadata repair. Deactivate, repair from the spare, reactivate:

vgchange -an vg_data
lvconvert --repair vg_data/thinpool   # rebuilds tmeta into the pmspare, swaps it in
vgchange -ay vg_data

The full-pool decision tree — match your situation to the row:

The behaviour-under-full setting that bounds your blast radius — choose it before the incident:

lvchange --errorwhenfull y vg_data/thinpool   # fail fast and cleanly instead of hanging

The metadata-repair toolchain you should know exists (the offline tools lvconvert --repair calls, and that you can run by hand on a deactivated pool):

Monitoring and autoextend: never learn it from dmesg

You should never discover a full pool from dmesg. LVM’s dmeventd can automatically grow a monitored pool when it crosses a threshold; you configure it in /etc/lvm/lvm.conf and back it with independent external alerting that fires before the autoextend point.

# /etc/lvm/lvm.conf  ->  activation { ... }
# When the pool exceeds 80% used, grow it by 20% of current size.
# Set threshold to 100 to disable autoextend (monitoring/warnings still fire).
thin_pool_autoextend_threshold = 80
thin_pool_autoextend_percent   = 20

Autoextend only works while the VG has free extents and the pool is monitored. Confirm monitoring and add a human-facing alert below the autoextend threshold:

# Ensure dmeventd is monitoring the pool (look for 'monitored' / the 'm' state bit)
lvchange --monitor y vg_data/thinpool
systemctl status lvm2-monitor.service

# Independent alert at 75% so a human reacts BEFORE autoextend at 80%.
USED=$(lvs --noheadings -o data_percent vg_data/thinpool | tr -d ' %' | cut -d. -f1)
[ "${USED:-0}" -ge 75 ] && echo "WARN: vg_data/thinpool data ${USED}%" | logger -t lvm-watch

The four lvm.conf keys that govern thin-pool growth and what each does:

Wire the check into a systemd timer or your monitoring agent. A Prometheus node_exporter textfile collector reading lvs is the standard pattern; export both percentages:

# /usr/local/bin/lvm-thin-metrics.sh  (run by a systemd timer into the textfile dir)
{
  lvs --noheadings --units b --nosuffix -o vg_name,lv_name,data_percent,metadata_percent \
    --select 'lv_attr=~"^t"' | while read VG LV DATA META; do
    echo "lvm_thinpool_data_percent{vg=\"$VG\",pool=\"$LV\"} ${DATA:-0}"
    echo "lvm_thinpool_metadata_percent{vg=\"$VG\",pool=\"$LV\"} ${META:-0}"
  done
} > /var/lib/node_exporter/textfile/lvm_thin.prom.$$
mv /var/lib/node_exporter/textfile/lvm_thin.prom.$$ /var/lib/node_exporter/textfile/lvm_thin.prom

The alert thresholds that experienced teams actually set — note that the human alert sits below autoextend so a person is engaged before automation acts, and metadata gets its own alert because it is the forgotten ceiling:

Architecture at a glance

Picture the stack as a five-storey building that data falls through, top to bottom, with one annex bolted to the side. On the top floor sit the things you mount: a handful of thin volumes (app01, app02, db01) and their thin snapshots, each advertising a generous virtual size (200 GiB) to its filesystem, none owning blocks directly. One floor down is the thin pool, the shared reservoir they all draw from; it is physically only 400 GiB even though the floor above promises 600 GiB — that gap is the overcommit, the load-bearing risk of the whole structure. The pool floor has two rooms: a large data room (_tdata) holding real chunks, and a small but critical metadata room (_tmeta) holding the btrees that remember which chunk backs which logical address for every volume and snapshot. A dmeventd smoke detector watches both; cross 80% in either and it grows the room (autoextend, if the basement has spare capacity) or screams.

Below the pool is the Volume Group (vg_data) — the foundation slab of pooled physical extents — and beneath it the Physical Volumes, the actual disks (/dev/sdb, /dev/sdc, plus /dev/sdd held in reserve as free extents the smoke detector can grab during a fire). To the side, an annex holds the fast NVMe (/dev/nvme0n1): a cache pool or writecache wired into one slow origin LV so its hot or sync-write traffic is served from flash, transparently. Trace a write: an app on db01 issues a 4 KiB write; the filesystem turns it into a block write to the thin LV; the pool checks its metadata, and if that address has no chunk yet, allocates one from the data room and records the mapping in the metadata room; if db01 is cached writeback, the block lands on NVMe and is acknowledged immediately, destaged to the slow PV later. Every step consumes something finite — a data chunk, a metadata entry, a slice of pool capacity — and the whole discipline of operating LVM is keeping both rooms below 100% while the floor above keeps promising more than the basement holds. The danger is structural, not incidental: it is what you accepted the moment you overcommitted, and the smoke detector plus a reserved disk in the basement are what keep “promising more than you own” from becoming “the building is on fire.”

Real-world scenario

Meridian Pay, a payments startup, ran a multi-tenant PostgreSQL fleet on a single LVM thin pool to claw back capacity from databases that each provisioned 500 GiB but used 80. The host: a bare-metal box in a Mumbai colo, 6× 1.92 TB NVMe behind an LVM-RAID6 array, a 6 TiB thin pool at 3x overcommit (eighteen tenants × 1 TiB virtual), 64 KiB chunk, a 2 GiB _tmeta. The killer feature for them was cheap restore testing: a nightly pg_basebackup plus per-tenant thin snapshots that QA could mount and validate, then drop. It worked beautifully for five months. Monthly storage cost, all-in, about ₹48,000 — roughly a third of what 18 TiB of thick allocation would have cost.

The incident hit at quarter-end. A batch job rewrote a large partitioned table across most tenants in the same window. Each rewrite forced copy-on-write divergence on every tenant’s open restore-test snapshot — snapshots QA had created the night before and not reaped — plus the raw write surge. The pool went from a comfortable 68% to 100% data in nineteen minutes. Because it was one shared pool with the default --errorwhenfull n, writes didn’t fail cleanly — they queued, Postgres connections hung, then filesystems remounted read-only as the kernel gave up. Eighteen tenants down simultaneously from one shared reservoir. The on-call’s first reflex was the cloud reflex: “grow the disk.” But vgs showed vg_free at zero — the RAID6 array was fully allocated to the pool, so lvextend had nothing to take and autoextend (set to 80%) had silently no-opped at the moment of truth for exactly that reason.

The breakthrough was reading lvs instead of reaching for hardware. The fat consumers weren’t the databases — they were fourteen un-reaped QA snapshots, each holding gigabytes of diverged blocks. Dropping them with lvremove returned blocks to the pool instantly and brought Data% from 100% back to 74% in under two minutes; the read-only filesystems were remounted rw, and a couple needed a quick fsck after their ENOSPC episode. Total downtime: 41 minutes, almost all of it spent reaching for the wrong fix first.

The remediation had four parts, each mapping onto this article. One — fail safe: lvchange --errorwhenfull y so a future exhaustion is a clean read-only remount, not a hang. Two — real headroom: they shrank the pool’s claim on the array to leave 15% of the VG permanently free as autoextend fuel, set autoextend to 80%/25%, and paged at 70% on both Data% and Meta%. Three — snapshot discipline: restore-test snapshots became strictly time-boxed, a systemd timer reaping any older than 60 minutes so divergent CoW blocks can’t accumulate unbounded. Four — smaller blast radius: overcommit capped at 1.5x (down from the crept-in 3x), the three largest tenants split onto their own pools, and _tmeta moved onto its own raid1. The next quarter-end batch peaked at Data% 79% — autoextend grabbed free extents once, the page fired, a human glanced and went back to bed. The pool has not filled since. The runbook line: “A full pool is a snapshot problem until proven otherwise. Read lvs before you touch hardware.”

Advantages and disadvantages

Thin provisioning, snapshots and caching are powerful precisely because they decouple what you promise from what you own — which is also exactly why they bite. Weigh it honestly:

The model is right when capacity is genuinely sparse and you will operate it: VM and container fleets, restore-test snapshot workflows, mixed HDD+SSD hosts wanting a transparent cache. It bites hardest on workloads that write in correlated bursts (everything fills the pool at once), on teams that treat snapshots as free-forever, and on anyone who deploys with defaults and never sets up the smoke detector. Every disadvantage above is manageable — but only if you know it exists and built the guardrail before the incident, which is the entire point of this article.

Hands-on lab

Build a thin pool, overcommit it, take a thin snapshot, then deliberately fill the pool to watch the failure and recover it — all on loopback files so you need no spare disks. Run as root. Everything is torn down at the end.

Step 1 — Create two loopback “disks” and PVs.

mkdir -p /root/lvmlab && cd /root/lvmlab
truncate -s 2G disk1.img
truncate -s 1G disk2.img
LOOP1=$(losetup --find --show disk1.img)
LOOP2=$(losetup --find --show disk2.img)
pvcreate "$LOOP1" "$LOOP2"
echo "PVs: $LOOP1 (2G), $LOOP2 (1G)"

Expected: Physical volume "/dev/loopN" successfully created. for each.

Step 2 — Build the VG using only the 2 GiB disk (keep the 1 GiB as future autoextend fuel).

vgcreate vglab "$LOOP1"
vgs vglab          # VFree should be ~2.00g

Step 3 — Create a small thin pool with explicit metadata, deliberately overcommitted.

lvcreate --type thin-pool -L 1G --chunksize 128K --poolmetadatasize 16M -n tpool vglab
lvs -a -o name,lv_attr,lv_size,data_percent,metadata_percent vglab

Expected: a tpool row with lv_attr starting twi-, plus hidden [tpool_tdata], [tpool_tmeta], [lvol*_pmspare] rows. Data% and Meta% near 0.

Step 4 — Carve two 800 MiB thin volumes from a 1 GiB pool (1.6x overcommit).

lvcreate --type thin -V 800M --thinpool tpool -n vol1 vglab
lvcreate --type thin -V 800M --thinpool tpool -n vol2 vglab
mkfs.ext4 -q /dev/vglab/vol1
mount /dev/vglab/vol1 /mnt 2>/dev/null || { mkdir -p /mnt/lvmlab && mount /dev/vglab/vol1 /mnt/lvmlab; }

Step 5 — Take a thin snapshot (near-free) and confirm pool usage barely moves.

lvcreate -s -kn -ay --name vol1_snap vglab/vol1
lvs -o name,lv_attr,origin,data_percent,pool_lv vglab

Expected: vol1_snap appears as a thin volume (Vwi-); pool Data% essentially unchanged — the snapshot cost only metadata.

Step 6 — Set fail-fast, then deliberately fill the pool and watch it error.

lvchange --errorwhenfull y vglab/tpool
MP=$(findmnt -no TARGET /dev/vglab/vol1)
# Write more than the 1 GiB pool can hold -> pool hits 100% -> writes fail
dd if=/dev/zero of="$MP/fill" bs=1M count=1100 2>&1 | tail -2
dmesg | grep -iE 'out-of-data-space|thin' | tail -3
lvs -o name,data_percent,metadata_percent,lv_health_status vglab/tpool

Expected: dd fails partway with a write error (or the FS goes read-only); dmesg shows an out-of-data-space thin message; pool Data% reads 100.00.

Step 7 — Recover non-destructively by adding the reserved disk and extending the pool.

vgextend vglab "$LOOP2"               # add the 1 GiB disk as free extents
lvextend -L +800M vglab/tpool         # grow the data area into the new space
lvs -o name,data_percent,metadata_percent vglab/tpool

Expected: vgextend succeeds, lvextend grows the pool, and Data% drops well below 100 — the volumes are writable again. (If a filesystem went read-only, mount -o remount,rw "$MP" brings it back; a real incident might need e2fsck.)

Validation checklist — what each step proved:

Teardown (removes everything, including the loopbacks):

umount "$MP" 2>/dev/null
vgchange -an vglab
vgremove -y vglab
pvremove "$LOOP1" "$LOOP2"
losetup -d "$LOOP1" "$LOOP2"
cd / && rm -rf /root/lvmlab

Common mistakes & troubleshooting

The playbook — bookmark this. First as a scannable symptom→cause→confirm→fix table you can read mid-incident, then the reasoning for the entries that bite hardest.

The extra reasoning for the ones that cause the worst incidents:

1. All thin volumes error / go read-only — pool data 100%. The defining LVM incident. Fix in order of preference: (a) lvremove an expendable snapshot to instantly reclaim diverged blocks — usually the actual culprit; (b) lvextend the pool if vgs vg_free > 0; © vgextend a new PV then lvextend. Set --errorwhenfull y beforehand so this is a clean remount, not a hang.

2. Pool wedged but data looks fine — metadata exhausted. The forgotten ceiling: _tmeta fills from snapshot count, not bytes written, so Meta% hits 100 while Data% is comfortable. Grow it with lvextend --poolmetadatasize +512M (up to ~16 GiB), and alert on Meta% separately — data-area alerts never catch this.

3. Pool won’t activate, needs_check. A transactional inconsistency (often after an unclean shutdown mid-allocation) sets the check-needed flag. Deactivate the VG and run lvconvert --repair vg_data/pool, which rebuilds metadata into the pmspare and swaps it in — exactly why you never delete the spare. Rehearse it on a lab pool first.

4. Thick snapshot invalidated. A classic CoW snapshot whose store filled is gone (lv_attr shows the I flag, Snap% hit 100); there is no recovery of that point-in-time. Recreate with a larger -L and enable snapshot_autoextend_threshold, but the real fix is migrating to a thin pool where snapshots can’t invalidate.

5. lvextend fails with no free extents. The trap that turns a 2-minute fix into a 20-minute one: the VG is fully allocated, so the pool has nowhere to grow and autoextend silently did nothing for the same reason. vgextend vg_data /dev/sdX first. Prevent it by keeping 15% of the VG permanently free as autoextend fuel and alerting on vg_free.

6. Deletes don’t free pool space. Filesystems mark blocks free internally but don’t tell the pool unless discards flow down. Run fstrim, enable fstrim.timer, and verify the pool’s --discards passdown and lvm.conf issue_discards for your backing.

7. Data loss after cache failure. writeback acks from the fast device; if that single device dies before destaging, those writes are gone. Restore from backup; going forward only run writeback on a mirrored cache (or use writethrough, which has no write risk), and always --splitcache (flush) before any planned cache removal.

Best practices

• Watch Data% AND Meta% on the pool row, never per-LV. The per-volume percentages reassure you while the pool dies. Alert on the pool; metadata gets its own alert because it fills from snapshot count, invisibly to data alerts.
• Keep 15% of the VG permanently free as autoextend fuel. Autoextend silently no-ops when the VG is out of extents — the exact moment you need it. Alert on vg_free too.
• Set --errorwhenfull y on multi-tenant pools. A full pool then fails as a clean read-only remount with a bounded blast radius, not an indefinite hang that corrupts.
• Size _tmeta deliberately and never delete the pmspare. Metadata exhaustion is as fatal as data exhaustion and harder to fix; lvconvert --repair needs the spare to land a rebuild.
• Treat every snapshot as a time-boxed pool consumer. Run a reaper (a systemd timer dropping snapshots past a TTL). A runaway snapshot is the most common root cause of a full pool.
• Prefer thin snapshots over thick. Thin snapshots can’t silently invalidate; thick ones drop to Invalid the instant their CoW store fills, taking your point-in-time copy with them.
• Use writethrough cache unless the cache device is redundant. writeback on a single device is a data-loss hazard. Always --splitcache (flush) before removing or migrating a cached origin.
• Cap and document overcommit per pool. Pick a ratio matched to how sparse and bursty the workload genuinely is (1.5x for mixed prod, not the 4x that creeps in), and write it down.
• Enable fstrim.timer so deletes return blocks to the pool. Without discards flowing down, the pool only grows.
• Mirror _tmeta (and consider RAID for _tdata) on important pools. Losing metadata while data survives is the worst half-failure; a raid1 _tmeta is cheap insurance.
• Remember the shrink asymmetry. Grow freely; shrink only ext4 (FS-first, lvreduce -r), never XFS, never a thin pool. Build runbooks around growth and pvmove, not shrinkage.
• Confirm monitoring is actually on after every change. lvchange --monitor y and systemctl status lvm2-monitor — autoextend and warnings depend on it, and it is easy to assume rather than verify.
• Rehearse lvconvert --repair on a deactivated lab pool before you need it. Recovery you’ve practised is recovery you can do at 3 a.m.

Security notes

LVM is a local block-storage layer, so its security story is about data-at-rest confidentiality, integrity of the metadata that maps your data, and not leaking through reclaimed blocks. The essentials:

• Encrypt at rest with LUKS, and choose the layer deliberately. dm-crypt/LUKS under LVM (encrypt the PV) gives one passphrase covering everything — the common, simpler choice; over LVM (encrypt individual LVs) gives per-volume keys and lifecycles.
• Mind the cache device’s confidentiality. A dm-cache/dm-writecache on NVMe holds copies of hot (and, in writeback, not-yet-destaged) data. If the slow origin is encrypted, the fast cache must be encrypted too, or plaintext lands on an unencrypted SSD.
• Reclaimed thin blocks can leak between tenants if not zeroed. When a freed chunk is reused by another volume, the pool must zero new blocks (lvm.conf thin_pool_zero / pool -Z y, the default) so a new tenant can’t read the previous one’s data. Verify it’s on for shared pools.
• Protect the metadata backups. /etc/lvm/backup/ and /etc/lvm/archive/ hold your VG layout (structure, not data); vgcfgrestore from them can undo a destructive change. Guard them, back them up off-host, treat write access to /etc/lvm/ as privileged.
• Snapshots are full-fidelity copies — same access controls. A read-only snapshot of a database is the database; never mount one with looser permissions than the origin, and reap them so old data doesn’t linger mounted.
• Least privilege on the tooling. LVM commands need root; don’t hand out broad sudo to lvremove/vgremove (a fat-fingered vgremove -y is catastrophic). Scope automation to the specific operations it needs.

The security-relevant settings in one place:

Cost & sizing

LVM itself is free — it ships with every distro — so “cost” here is the capacity you buy and the engineering time the failure modes consume. The drivers and how to right-size:

• Thin provisioning’s value is buying capacity as real usage grows. On a genuinely sparse fleet (the 500-provisioned/80-used pattern), 3–6x overcommit can cut backing-store spend 60–80%. The bill for getting it wrong is an outage, so spend some of the saving on headroom and monitoring.
• Headroom is the cost of safety, cheap relative to downtime. Reserving 15% of a VG as autoextend fuel “wastes” 15% — far less than revenue lost to a fleet-wide read-only remount. Size it to absorb the largest realistic write surge plus snapshot divergence.
• The cache device is a small, high-leverage spend. One NVMe fronting a large array transforms latency for the hot working set; you buy one fast device, not the whole tier in flash. Mirror it only for writeback durability — that doubles the cache cost, not the array.
• Metadata is tiny but has a hard ceiling. _tmeta costs MiB to a few GiB; the risk isn’t its cost but its ~16 GiB cap — past a certain pool-size + small-chunk combination you can’t grow it further, forcing a larger chunk at design time. Size chunk and metadata for the pool’s eventual size.
• RAID level is a capacity-efficiency decision. raid1/raid10 cost 50%; raid5 costs one disk; raid6 costs two. For a pool’s data on large disks raid6 is usually right (long rebuilds favour double parity); for _tmeta, a small raid1 cheaply protects the irreplaceable mapping.

A rough sizing picture for a mid-size host (figures are illustrative INR for on-prem disk, to anchor the trade-offs):

The right-sizing discipline: measure actual usage and growth rate, pick an overcommit ratio you can defend, reserve real headroom, and let autoextend plus alerts manage the rest. The cheapest pool is not the one with zero free space — it’s the one that never fills.

Interview & exam questions

1. Explain the LVM object model and where thin provisioning inserts itself. PVs are block devices stamped for LVM and divided into physical extents; a VG pools PVs into one allocation arena; LVs are mountable mappings of extents. Thin provisioning adds a thin pool (a special LV) with a data sub-LV (_tdata) and a metadata sub-LV (_tmeta); thin LVs draw blocks from the pool on demand rather than mapping VG extents directly.

2. What exactly happens when a thin pool reaches 100% data, and what’s your first move? Every thin volume needing a new chunk starts erroring on write; filesystems remount read-only or corrupt, and because the pool is shared, all its volumes are hit at once. First move: check lvs for a runaway snapshot and lvremove it (instant reclaim), or lvextend the pool if the VG has free extents (vgextend first if it doesn’t). dmesg shows out-of-data-space.

3. Why must you monitor metadata separately from data on a thin pool? Metadata (_tmeta) fills from the number of allocated chunks and snapshots, not bytes written, so Meta% can hit 100 while Data% looks healthy — wedging the pool from a direction data alerts never see. Metadata also has a ~16 GiB hard ceiling and is harder to grow under pressure, so you alert on metadata_percent independently.

4. Contrast thick and thin snapshots, focusing on the failure mode. A thick snapshot pre-allocates a fixed CoW store; when that store fills it is marked Invalid and lost silently — no recovery. A thin snapshot is just another thin volume sharing the origin’s blocks; it consumes pool space as it diverges but cannot invalidate. Thin is strictly safer for backups and restore testing.

5. When do you choose dm-cache vs dm-writecache? dm-cache caches reads (and optionally writes via writeback) and promotes hotspots — use it for mixed read/write with a hot working set. dm-writecache buffers sync writes only with no read caching — use it when only fsync latency on a read-mostly origin is the problem.

6. writethrough vs writeback — why is this a durability decision? writethrough acknowledges a write only after it lands on the slow origin, so a dead cache device never loses acknowledged data (read acceleration, no write risk). writeback acknowledges from the fast device and destages later — faster, but a single failed cache device loses unflushed acknowledged writes. Run writeback only on a redundant (mirrored) cache.

7. Your lvextend to relieve a full pool fails with “insufficient free extents.” What went wrong and what do you do? The VG is fully allocated, so there’s nothing to grow into — and autoextend silently no-opped for the same reason. Run vgextend vg_data /dev/sdX to add a PV first, then lvextend. Prevent it by keeping ~15% of the VG permanently free and alerting on vg_free.

8. Why can deleting files inside a thin volume fail to free pool space, and how do you fix it? Filesystems mark blocks free internally but only tell the pool if discards flow down. Run fstrim (or enable fstrim.timer), and ensure the pool’s --discards passdown and lvm.conf issue_discards are set so reclaim reaches the pool/device.

9. Walk through recovering a wedged thin pool that won’t activate. Confirm needs_check in dmesg. Deactivate the VG (vgchange -an), run lvconvert --repair vg_data/pool (which rebuilds metadata into the pmspare and swaps it in), then reactivate (vgchange -ay). This is why you never delete the metadata spare. Rehearse it on a lab pool beforehand.

10. What are the rules for shrinking, and what’s the asymmetry with growing? Growing (LV+FS, pool data, pool metadata, adding PVs, pvmove) is always online and safe. Shrinking requires shrinking the filesystem first then the LV (lvreduce -r); XFS cannot shrink at all and thin pools cannot shrink at all. Always e2fsck before reducing ext4. Build runbooks around growth and pvmove, never shrinkage.

11. How does pvmove let you evacuate a failing disk with zero downtime, and what’s the cache gotcha? pvmove relocates a PV’s extents to other PVs in the VG while volumes stay mounted; it journals progress and is resumable after an interruption (pvmove to resume, --abort to roll back). If the PV backs a cached origin’s slow device, --splitcache (flush) the cache before moving, then reattach.

12. Why mirror _tmeta, and what does losing metadata vs data mean? Losing the data area means losing data; losing the metadata means losing the map of which chunk backs which address — potentially making otherwise-intact data unrecoverable. Putting _tmeta on a small raid1 ensures a single disk loss can’t destroy the mapping while data survives — cheap insurance against the worst half-failure.

These map to vendor Linux certifications and storage modules: RHCSA/RHCE (EX200/EX294) cover LVM creation, extension, and snapshots; LFCS/LFCE cover the device-mapper stack, thin provisioning and recovery; storage-architecture interviews probe overcommit reasoning, the full-pool failure mode, and cache durability. A compact mapping:

Quick check

1. A 600 GiB-virtual set of thin volumes lives on a 400 GiB pool. The pool’s Data% reads 96% while app01 shows only 30% used. Which number do you page on, and why?
2. A thick snapshot you took for an overnight backup is gone by morning and shows Invalid. What happened, and what’s the durable fix?
3. lvextend to relieve a full pool fails: “insufficient free extents in volume group.” What’s the one command that unblocks it?
4. You’re about to pvmove a disk that backs a writeback-cached database volume. What must you do first, and why?
5. True or false: scaling out to more disks (adding PVs to the VG) fixes an exhausted _tmeta. If false, what does?

Answers

1. Page on the pool’s Data% (96%) — it’s the share of physical pool capacity in use and predicts the outage; app01’s 30% is just that one volume’s fill against its virtual size and tells you nothing about pool survival. At 96% pool data you’re minutes from every volume erroring on write.
2. The thick snapshot’s pre-sized CoW store filled and the kernel invalidated it — there’s no recovery of that point-in-time. The durable fix is to use thin snapshots (which can’t invalidate; they just consume pool space), or at minimum size the CoW store for worst-case churn and enable snapshot_autoextend_threshold.
3. vgextend vg_data /dev/sdX — the VG is out of free extents, so the pool has nothing to grow into (and autoextend silently no-opped for the same reason). Add a PV first, then lvextend.
4. lvconvert --splitcache vg_data/<vol> first — it flushes dirty (unflushed acknowledged) blocks to the origin before detaching. With writeback, moving the origin while the cache holds undestaged writes risks losing them; split (flush) first, pvmove, then reattach.
5. False. Adding PVs gives the VG more free extents but does nothing for metadata until you actually grow it. The fix is lvextend --poolmetadatasize +N vg_data/pool (up to the ~16 GiB ceiling); reaping snapshots also lowers Meta% by removing mappings.

Glossary

• PV (Physical Volume) — a block device (disk, partition, md array, multipath LUN) initialized with an LVM label and divided into physical extents.
• PE (Physical Extent) — the fixed-size unit of LVM allocation, 4 MiB by default; every LV is an integer number of extents.
• VG (Volume Group) — one or more PVs pooled into a single allocation arena; must have free extents to grow anything in it.
• LV (Logical Volume) — a named mapping of extents you format and mount; can be thick, thin, a snapshot, cached, or a RAID LV.
• Thin pool — a special LV that hands blocks to thin LVs on demand; composed of a data LV (_tdata) and a metadata LV (_tmeta).
• _tdata / _tmeta — the thin pool’s real-data sub-LV and its block-mapping-metadata sub-LV; Data%/Meta% at 100% each wedge the pool.
• Thin LV — a volume that advertises a virtual size but consumes pool blocks only as data is written (allocate-on-write).
• Chunk size — the thin pool’s allocation granularity (64 KiB–1 GiB); set at creation and immutable; trades metadata size and IO overhead against CoW granularity and sequential throughput.
• Overcommit — total virtual capacity divided by physical pool capacity; the point of thin provisioning (> 1.0) and its central risk.
• Thick (classic) snapshot — a CoW snapshot of a normal LV with a pre-sized exception store; invalidates and is lost when the store fills.
• Thin snapshot — a thin LV sharing the origin’s blocks; near-free at creation, consumes pool space as it diverges, cannot invalidate.
• CoW (copy-on-write) — preserving an old block by copying it before an overwrite, so a snapshot keeps the original view.
• dm-cache / cache pool / cachevol — device-mapper read+(optionally)write caching of hot blocks on a fast device fronting a slow origin LV.
• dm-writecache — a device-mapper write-only buffer for sync writes on a fast device; no read caching, no hotspot promotion.
• writethrough / writeback — cache durability modes: writethrough acks from the slow origin (no write-loss risk); writeback acks from the fast device (faster, loses unflushed writes if it dies).
• pvmove — live, resumable relocation of a PV’s extents to other PVs in the VG, used to evacuate a failing disk with zero downtime.
• RAID LV — a redundant LV built directly in LVM via dm-raid (raid1/raid5/raid6/raid10), composed of hidden rimage/rmeta sub-LVs.
• pmspare — the reserved metadata-spare LV ([lvol*_pmspare]) that lvconvert --repair writes a rebuilt metadata copy into; never delete it.
• dmeventd — the device-mapper event daemon that monitors thin pools (and snapshots) and triggers autoextend when a threshold is crossed.
• Autoextend — automatic growth of a monitored thin pool (or thick snapshot) past a configured threshold; silently no-ops if the VG has no free extents.
• --errorwhenfull — pool setting: y makes a full pool error immediately (clean failure, bounded blast radius); n (default) queues writes (hangs).
• fstrim / discards — the mechanism by which freed filesystem blocks are returned (TRIMmed) to the thin pool and the device below; without it the pool only grows.

Next steps

You can now build, operate, monitor and recover an LVM thin/snapshot/cache stack. Build outward into the layers above and below it:

• Next: Building Resilient Linux Storage with mdadm Software RAID — the redundancy layer you often put under an LVM PV, and the mdadm-vs-LVM-RAID decision in full.
• Related: Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers — how IO schedulers, readahead and sysctl shape what a cached or thin volume actually delivers.
• Related: Mastering systemd: Units, Timers, Resource Control, and Service Hardening — the timer and unit machinery behind fstrim.timer, lvm2-monitor, and your snapshot reaper.
• Related: Building a Linux Audit Trail with auditd and eBPF Runtime Visibility — observe and audit who runs the destructive lvremove/vgremove commands on your storage.
• Related: Windows Failover Clustering and Storage Spaces Direct: A Production Build — the Windows equivalent of thin provisioning, tiering and snapshots, for comparison and mixed estates.