Advanced Storage: Software RAID (mdadm), NFS, Samba/CIFS, autofs & iSCSI

Single-disk, single-host storage runs out of road in two directions at once. Down, a disk dies — platters seize, a controller flakes, a cloud volume returns I/O errors — and if that disk held your only copy, the outage is total and the recovery is a restore-from-backup marathon. Out, more than one machine needs the same data — ten web servers must serve the same media library, a design team on Windows wants a shared drive, a database VM wants a raw block device it can format itself — and copying files around by scp stops scaling the moment two people can edit them. This lesson solves both problems with the tools every RHCSA/LFCS candidate and every real ops engineer is expected to wield cold.

The first half is redundancy on one host: Linux software RAID via mdadm, where several disks become one /dev/md0 that keeps serving through a disk death — including the failure-and-rebuild drill you must be able to perform in your sleep. The second half is sharing across hosts: three network storage protocols that hand the same filesystem to other machines — NFS (the Unix-native file share), Samba/CIFS (the Windows-native file share), and iSCSI (raw block storage over IP) — plus autofs for mounting shares on demand, and the firewall and SELinux settings without which none of it works. We finish with the decision that trips up most people: given a sharing problem, which of the three (or object storage) is the right tool? Everything runs on a single throwaway VM using loop devices and localhost, so you can do all of it hands-on. ⚠️ Several commands here destroy data (mdadm --create, mkfs, --fail) — they are flagged, and the lab confines them to scratch files.

Why this matters

Four situations, each of which you will meet in production:

• A disk dies at 03:00 and nothing goes down. A drive in your /dev/md0 array throws errors; mdadm marks it faulty, a hot spare kicks in, the array rebuilds itself, and the only evidence is an email and a line in /proc/mdstat. That calm is not luck — it is RAID configured correctly before the failure, plus an operator who has rehearsed --fail/--remove/--add on a test rig and doesn’t panic when the real one lights up.
• Ten app servers, one media library. You cannot ship a 400 GB library to ten servers and keep them in sync. You export it once from a storage host over NFS and every server mounts nfs-server:/srv/media read-only. Add an eleventh server and it mounts the same path — no copy, no drift.
• The Windows design team wants a drive letter. They don’t speak NFS; they speak SMB. Samba turns a Linux directory into \\fileserver\design that maps to Z: on every Windows box, honours a username/password, and — done right — never has that password sitting in a script or a shell history.
• A database wants its own disk, not a shared folder. Some workloads want a block device they format and own, not a shared filesystem. iSCSI ships raw sectors over the network; the client sees a new /dev/sdX, partitions it, runs mkfs, and treats it exactly like a local disk — because to the client, it is one.

The mental model to hold for the whole lesson is a single distinction that untangles all four:

RAID gives you redundancy on ONE host. NFS/Samba/iSCSI give you sharing ACROSS hosts. And within sharing, NFS and Samba serve files (the server owns the filesystem, many clients read/write files inside it) while iSCSI serves blocks (the client owns the filesystem; the server just hands out sectors).

Get that straight and every command below lands in the right place. RAID is not a backup and not sharing. NFS is not block storage. iSCSI is not a shared folder. This lesson builds all three layers and shows exactly where each one fits. It assumes you are comfortable with disks, partitions, filesystems and /etc/fstab — if lsblk, blkid, mkfs and a UUID-based fstab line aren’t yet second nature, do Storage: disks, partitions, filesystems & fstab first, because everything here sits on top of it.

Software RAID with mdadm

RAID — Redundant Array of Independent Disks — combines several physical disks into one logical device that is bigger, faster, more resilient, or some blend of the three, depending on the level you choose. Linux does this in the kernel’s md (“multiple devices”) layer, driven entirely from user space by one command: mdadm. No special controller, no licence, no vendor lock-in — any disks the kernel can see, mdadm can weave into an array that presents as a single /dev/md0 you then partition, format and mount like any other block device.

The RAID levels, compared

Choosing a level is a trade between three things you cannot maximise at once: usable capacity, fault tolerance, and write performance. This table is the one you memorise.

Three rules of thumb turn that table into a decision. RAID 0 is not redundancy — it multiplies your failure risk (any one disk kills the whole stripe) and exists only for raw speed on disposable data. RAID 5 is out of favour for large disks: with multi-terabyte drives, rebuilding after a failure reads every remaining disk end-to-end for hours, and a second failure or an unreadable sector during that rebuild loses the array — which is why RAID 6 (tolerates two failures) is the default for anything big. RAID 10 wins when you care about write latency and IOPS (databases) more than raw capacity, because it has no parity to compute — the “write penalty” column is why: every RAID 5 write costs four physical I/Os (read old data, read old parity, write new data, write new parity) and RAID 6 costs six, while RAID 10 just writes to two mirrors.

Where software RAID sits: mdadm vs LVM vs hardware RAID

RAID is not the only “combine disks” tool on the box, and knowing which layer to use — and how they stack — prevents a lot of muddle.

The clean division of labour most systems use: mdadm (or hardware RAID) for the redundancy, LVM on top for the flexibility. You build /dev/md0 from your disks for fault tolerance, then make md0 a single LVM physical volume and carve logical volumes out of it — so you get RAID’s resilience and LVM’s online resize, snapshots and multi-volume management. LVM can also do its own RAID (lvcreate --type raid1), which is handy when you want mirroring and volume management in one tool, but the two-layer md + LVM stack remains the most common and the most transparent. (If LVM is new, see LVM: logical volume management.) Hardware RAID earns its keep mainly for its battery-backed write cache and for presenting a single bootable disk to legacy firmware; on modern Linux, software RAID matches it on reliability and beats it on portability and observability.

Creating an array

You build a RAID array from whole disks or from partitions typed fd00 (Linux RAID). Using partitions is tidier — it labels the disk’s intent and leaves room for metadata. The core verb is mdadm --create.

⚠️ mdadm --create wipes the member devices. Everything on /dev/sdb1 … /dev/sde1 is destroyed. Confirm the device list with lsblk first; a create aimed at the wrong disk is unrecoverable.

# Build a RAID6 array named /dev/md0 from four devices, keeping one hot spare:
sudo mdadm --create --verbose /dev/md0 \
     --level=6 --raid-devices=4 \
     /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 \
     --spare-devices=1 /dev/sdf1
# mdadm: array /dev/md0 started.

# It starts building immediately — check progress:
cat /proc/mdstat

The mdadm verbs (--modes) you actually use:

The --level accepts 0, 1, 5, 6, 10 (and linear); --raid-devices is the count of active members; --spare-devices adds idle standbys. Chunk size (--chunk=512K, the stripe unit) rarely needs tuning — the default is fine unless you’re chasing a specific workload.

Reading the array: /proc/mdstat and mdadm --detail

Two views tell you everything. /proc/mdstat is the live, kernel-maintained summary — the first thing to cat (or watch) when anything RAID happens:

cat /proc/mdstat
# Personalities : [raid6] [raid5] [raid4]
# md0 : active raid6 sdf1[4](S) sde1[3] sdd1[2] sdc1[1] sdb1[0]
#       2093056 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
#       [===========>.........]  resync = 58.9% (...) finish=0.4min speed=...
# unused devices: <none>

Decode that middle block — it’s dense but every token matters:

The [4/4] [UUUU] line is the one you glance at daily: all members present and up. The instant it reads [4/3] [UU_U], a disk is gone and (if a spare exists) a recovery line will be counting up beside it. mdadm --detail is the fuller, human-readable report — use it when /proc/mdstat flags something:

sudo mdadm --detail /dev/md0
#            Version : 1.2
#      Creation Time : ...
#         Raid Level : raid6
#         Array Size : 2093056 (2044.00 MiB ...)
#       Raid Devices : 4
#      Total Devices : 5
#              State : clean                 <-- clean | active | degraded | recovering
#     Active Devices : 4
#    Working Devices : 5
#     Failed Devices : 0
#      Spare Devices : 1
#               UUID : 3aa5c8f2:...:...      <-- the array UUID for mdadm.conf
#     Number Major Minor RaidDevice State
#        0     8    17        0      active sync   /dev/sdb1
#        ...
#        4     8    81        -      spare         /dev/sdf1

The State field is your headline: clean/active (healthy), degraded (a member down, running on redundancy), recovering/resyncing (rebuilding). The UUID line is what you’ll pin into the config file next.

Persisting the array and monitoring it

An array you built with --create is assembled and running now, but nothing yet tells the system to reassemble it at the next boot — that’s the job of mdadm.conf. Generate the definition and append it:

# RHEL/Fedora/Rocky — config lives at /etc/mdadm.conf:
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf
# ARRAY /dev/md0 metadata=1.2 spares=1 name=host:0 UUID=3aa5c8f2:...:...
sudo dracut -f                       # rebuild initramfs so md0 assembles early at boot

# Debian/Ubuntu — config lives at /etc/mdadm/mdadm.conf:
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u             # the Debian equivalent of dracut -f

⚠️ Regenerate the initramfs (dracut -f or update-initramfs -u) after editing mdadm.conf. If the root filesystem lives on RAID and the initramfs can’t assemble the array, the machine won’t boot — the array must be defined inside the early boot image, not just in the on-disk config the running system reads later.

Monitoring is the difference between “a disk failed and a spare took over silently” and “a disk failed weeks ago, then a second one died and the array is gone.” mdadm ships a monitor daemon that emails on events:

Set MAILADDR, enable mdmonitor, and send yourself the --test alert once so you know mail actually reaches you — an alerting pipeline you’ve never tested is not an alerting pipeline.

The array is now built, readable, persistent and monitored. That single fault-tolerant /dev/md0 — with a filesystem on it — is exactly what we’ll hand to other machines in the second half. Here is the whole picture, left to right: disks become one mdadm array, a filesystem on it is exported three ways, and clients across the network mount whichever flavour they need.

The disk-failure and rebuild drill

This is the section to do, not just read — the muscle memory that turns a failed disk from an emergency into a chore. The cycle is always the same three verbs: fail → remove → add, while you watch /proc/mdstat narrate the rebuild.

⚠️ --fail deliberately kicks a healthy disk out of the array. Only do this on a test array (like the lab’s loop devices) or on a real disk you are genuinely replacing. On a degraded array with no redundancy left, failing another member destroys data.

Walking through it, with the output you’ll actually see:

# 1. Fail a member (test rig only!). If a hot spare exists, recovery starts at once:
sudo mdadm /dev/md0 --fail /dev/sdb1
# mdadm: set /dev/sdb1 faulty in /dev/md0

cat /proc/mdstat
# md0 : active raid6 sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0](F)
#       [4/3] [_UUU]
#       [=====>...............]  recovery = 27.4% (.../...)  finish=... speed=...
#                                ^ the spare sdf1 is rebuilding into the missing slot

The array kept serving reads and writes the whole time — that’s the entire point of redundancy. Once recovery finishes it shows [4/4] [UUUU] again, now with sdf1 promoted to active and sdb1[0](F) still hanging around as a failed member. Clear it out and add a fresh disk:

# 2. Remove the failed disk from the array:
sudo mdadm /dev/md0 --remove /dev/sdb1
# mdadm: hot removed /dev/sdb1 from /dev/md0

# 3. Add a replacement. On a still-degraded array it rebuilds; on a healthy one it's a new spare:
sudo mdadm /dev/md0 --add /dev/sdb1
# mdadm: added /dev/sdb1

Two rules keep this drill safe. Never fail a second disk while the first rebuild is running on a single-parity array (RAID 5) — you have no redundancy mid-rebuild and will lose everything; RAID 6’s second parity is exactly the margin that makes a rebuild survivable. And replace, don’t just remove — an array running degraded with no spare is one disk away from disaster, so treat “degraded” as a same-day, not same-week, ticket.

To retire an array completely: unmount it, sudo mdadm --stop /dev/md0, then sudo mdadm --zero-superblock /dev/sd{b,c,d,e,f}1 to erase the RAID metadata so the disks don’t get re-assembled into a phantom array on the next boot. ⚠️ --zero-superblock is destructive to the array; only run it once the data is safely elsewhere.

NFS: file sharing for Unix clients

NFS (Network File System) is the native way Unix and Linux machines share files. The server exports a directory; clients mount it over the network and it behaves like a local filesystem — same paths, POSIX permissions, ownership, the works. It is the right tool when both ends speak Unix: shared home directories, a common media or code tree, scratch space for a compute cluster.

The server: /etc/exports and its options

You declare what to share, to whom, and with what rights in /etc/exports (or a drop-in under /etc/exports.d/). One line per exported directory:

# /etc/exports  —  <directory>  <client>(options) [<client>(options) ...]
/srv/share      192.168.1.0/24(rw,sync,root_squash,no_subtree_check)
/srv/media      *(ro,sync,root_squash)                      # world-readable, read-only
/srv/secure     10.0.0.5(rw,sync,root_squash,sec=krb5p)     # Kerberos-encrypted
/export/home    192.168.1.0/24(rw,sync,root_squash,no_subtree_check)

The client can be an IP, a CIDR range, a hostname, a wildcard (*.example.com), or * for everyone. ⚠️ There is no space between the client and its (options) — 192.168.1.0/24 (rw) (with a space) means “read-only to that network and read-write to everyone else,” a genuinely dangerous typo. The options that matter:

The single most important option to understand is root_squash. NFS’s traditional sec=sys model trusts whatever UID the client claims — so without squashing, someone with root on any client could chown and read every file on your export as root. root_squash (on by default) neuters that by mapping the client’s UID 0 to the unprivileged nobody, so remote root has fewer rights than a normal user, not more. You turn it off (no_root_squash) only for something like a backup server that must preserve root-owned files — and you scope that to one trusted IP. For genuine security across an untrusted network, sec=sys isn’t enough at all; you move to sec=krb5p, which authenticates users with Kerberos and encrypts the traffic.

Applying exports and the server service

Editing /etc/exports doesn’t change anything until you tell the NFS server to re-read it with exportfs, and the daemon itself has to be installed and running:

sudo mkdir -p /srv/share && sudo chown nobody:nobody /srv/share
sudoedit /etc/exports                 # add the export line
sudo exportfs -rav                    # -r re-export all, -a all, -v verbose
# exporting 192.168.1.0/24:/srv/share
sudo exportfs -s                      # confirm what's live
# /srv/share  192.168.1.0/24(sync,wdelay,hide,...,rw,secure,root_squash,no_subtree_check)

exportfs -rav is the command you run after every /etc/exports edit — -r re-exports everything to match the file (adding new, dropping removed), -a covers all, -v shows you what it did. Note it filled in the full option set including the defaults you didn’t type (sync, root_squash, secure).

NFSv3 vs NFSv4: pseudo-root, ports and rpcbind

There are two NFS versions in the wild, and the differences decide your firewall rules and mount syntax. Prefer NFSv4 for anything new.

The pseudo-root is the NFSv4 concept that surprises people. In v4 the server presents a single virtual namespace: you can nominate one export as the root with fsid=0, and clients then mount paths relative to it. Two ways it plays out:

# Style A — explicit v4 pseudo-root (fsid=0):
#   /etc/exports:
#   /export          192.168.1.0/24(rw,fsid=0,root_squash)     # this is v4 "/"
#   /export/share    192.168.1.0/24(rw,root_squash)            # bind-mounted under /export
# Client mounts the path RELATIVE to the pseudo-root:
sudo mount -t nfs4 server:/share /mnt/share

# Style B — modern nfs-utils auto-builds the pseudo-root; mount the real path:
sudo mount -t nfs -o vers=4.2 server:/srv/share /mnt/share

Modern nfs-utils synthesises the pseudo-root automatically, so Style B (mount the real server path, let the version negotiate) is what you’ll usually type. The firewall story follows directly from the port table — this is where a lot of “it mounts locally but not from another host” tickets die:

# RHEL/Fedora — open NFSv4 through firewalld:
sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --reload

# Debian/Ubuntu with ufw:
sudo ufw allow 2049/tcp

If you must run v3, pin mountd/statd to fixed ports (in /etc/nfs.conf on RHEL, /etc/default/nfs-common + /etc/default/nfs-kernel-server on Debian) before writing firewall rules — otherwise they roam to a different port on each restart and your rules stop matching. Every one of these protocols needs its ports opened; the mechanics of firewalld, nftables and zones are covered in Firewalls: firewalld, nftables & iptables.

Mounting NFS on the client

The client needs the NFS client package (nfs-utils on RHEL, nfs-common on Debian), then it’s a normal mount:

sudo mkdir -p /mnt/share
# Mount once (auto-negotiates the highest version):
sudo mount -t nfs -o vers=4.2 server:/srv/share /mnt/share
findmnt /mnt/share
# TARGET      SOURCE               FSTYPE OPTIONS
# /mnt/share  server:/srv/share    nfs4   rw,relatime,vers=4.2,...

Make it persistent in /etc/fstab — the key extra is _netdev, which tells systemd “this is a network mount, wait for the network before trying it”:

# /etc/fstab
server:/srv/share  /mnt/share  nfs  defaults,_netdev,vers=4.2  0  0

The hard vs soft choice matters more than it looks. hard (the default) means if the server vanishes, I/O blocks forever and processes wedge in uninterruptible D state until it returns — safe for data integrity, miserable for availability. soft makes I/O return an error after retrans retries — better for a nice-to-have mount, risky for one you’re writing important data to (a transient blip can surface as a write error). The usual advice: hard for read-write data you care about, soft,timeo=… only for read-only or non-critical mounts. Which leads directly to the better answer for anything that isn’t always needed: don’t hard-mount it in fstab at all — mount it on demand.

autofs: on-demand mounts done right

A permanent fstab NFS mount has two failure modes: if the server is down at boot the client can hang (or, with _netdev, at least delay), and a mount you rarely use sits there consuming a connection and blocking clean shutdowns. autofs fixes both by mounting a share the instant something touches its path and unmounting it again after an idle timeout. This is the correct pattern for NFS home directories and any “sometimes needed” share — a laptop that never reaches the file server simply never triggers the mount, so it never hangs.

autofs is driven by a master map (/etc/auto.master) that points at one or more mount maps, each of which describes what to mount under a base directory:

The mechanics: the base /mnt/nfs is managed by autofs, and when a process references /mnt/nfs/share, autofs reads the share key from /etc/auto.nfs, mounts server:/srv/share there on the fly, and unmounts it 60 seconds after last use. The map entry format is always key -options source.

The killer application is home directories. One wildcard line serves every user without editing anything when people join:

# /etc/auto.master
/home/net   /etc/auto.home

# /etc/auto.home  —  '&' is replaced by whatever key was requested
*   -rw,soft   server:/export/home/&

Now cd /home/net/vinod mounts server:/export/home/vinod on demand; cd /home/net/asha mounts server:/export/home/asha. No per-user config, no boot-time hang, no idle mounts. Enable it with the autofs package (same name on both families) and sudo systemctl enable --now autofs; after editing any map, sudo systemctl reload autofs. Debug with automount -f -v in a spare terminal (foreground, verbose) to watch mounts happen as you touch paths.

Samba/CIFS: file sharing for Windows

When the clients are Windows (or a mix of Windows, macOS and Linux), the native protocol is SMB/CIFS, and the Linux server that speaks it is Samba. Samba turns a directory into a share that appears as \\server\share in Windows Explorer, honours a username and password, and integrates with Active Directory when you need it. The Linux client side uses mount -t cifs (from cifs-utils) to mount an SMB share — from a Samba server or from an actual Windows box.

Configuring shares: /etc/samba/smb.conf

Samba’s config is /etc/samba/smb.conf: a [global] section for server-wide settings, then one [sharename] section per share.

# /etc/samba/smb.conf
[global]
   workgroup = WORKGROUP
   server string = KloudVin File Server
   security = user                 # authenticate against Samba user accounts
   map to guest = never
   server min protocol = SMB3      # refuse ancient, insecure SMB1

[data]
   path = /srv/samba/data
   comment = Shared data
   valid users = vinod, @smbteam   # a user and a group (@)
   read only = no                  # i.e. writable
   browseable = yes
   create mask = 0664
   directory mask = 0775

Managing Samba: users, testparm, services

Two facts trip beginners. First, Samba users are separate from Linux users: the Linux account must exist, and then you give it a Samba password with smbpasswd -a. Second, always validate the config with testparm before restarting — a syntax error can take the whole service down.

sudo useradd -M -s /sbin/nologin vinod   # a real Linux account (no home/login needed)
sudo smbpasswd -a vinod                  # give it a SEPARATE Samba password
# New SMB password: ...  Retype new SMB password: ...  Added user vinod.
testparm                                 # validate BEFORE restarting
# Loaded services file OK.  Server role: ROLE_STANDALONE
sudo systemctl restart smb nmb           # (smbd nmbd on Debian)

smbd serves the files; nmbd handles legacy NetBIOS name resolution and browsing. On a pure-SMB3 network you can often skip nmbd, but enabling both is the safe default. Samba’s ports are 445/tcp (modern SMB, the one that matters) and 139/tcp + 137-138/udp (legacy NetBIOS); open the firewalld samba service (firewall-cmd --permanent --add-service=samba).

Mounting CIFS with a credentials file

On the Linux client, mount -t cifs attaches an SMB share. The trap everyone falls into first is putting the password on the command line:

⚠️ Never mount CIFS with -o username=x,password=y. The password is then visible to every user via ps aux, sits in your shell history, and lands in logs. Put credentials in a root-owned, chmod 600 file instead.

# Create the credentials file (mode 600, readable only by root):
sudo tee /root/.smbcreds >/dev/null <<'EOF'
username=vinod
password=SuperSecret
domain=WORKGROUP
EOF
sudo chmod 600 /root/.smbcreds

# Mount using the file — no secret on the command line or in history:
sudo mkdir -p /mnt/winshare
sudo mount -t cifs //server/data /mnt/winshare \
     -o credentials=/root/.smbcreds,uid=1000,gid=1000,vers=3.1.1

In /etc/fstab, reference the same credentials file and add _netdev:

//server/data  /mnt/winshare  cifs  credentials=/root/.smbcreds,uid=1000,gid=1000,_netdev  0  0

Because SMB carries no Unix UIDs/GIDs, uid=/gid= decide which local account appears to own everything under the mount — set them to the user who needs to work there, or files show up as root and non-root writes fail.

SELinux for NFS and Samba

On the RHEL family (Fedora, Rocky, RHEL, AlmaLinux), SELinux is enforcing by default, and it is the single most common reason an NFS or Samba share with perfect file permissions still returns “permission denied.” SELinux confines the daemons: nfsd and smbd are only allowed to serve files whose type they’re permitted to, gated by booleans you flip with setsebool. (Debian/Ubuntu use AppArmor instead; there the equivalent gotcha is an AppArmor profile confining smbd — usually permissive enough out of the box, but check dmesg for apparmor="DENIED" if a share misbehaves.)

The -P flag makes the boolean persistent across reboots — forget it and your share works until the next boot, then mysteriously breaks. Two philosophies: the quick blanket boolean (samba_export_all_rw) lets the daemon serve anything, while the tidy approach labels only the specific directory with the right type (samba_share_t for Samba, or leave NFS with the default public_content_t/nfs_t handling) using semanage fcontext + restorecon. Blanket booleans are fine on a dedicated file server; targeted labels are better on a multi-role host. When a share denies access despite correct ls -l permissions, your first two commands are getsebool -a | grep -E 'nfs|samba' and sudo ausearch -m avc -ts recent (or tail /var/log/audit/audit.log) to see the exact SELinux denial.

iSCSI: block storage over IP

NFS and Samba share files — the server owns the filesystem and clients read and write files inside it. iSCSI is fundamentally different: it ships raw blocks (disk sectors) over TCP/IP, so the client receives what looks like a brand-new local disk (/dev/sdX) that it must partition, format and mount itself. The server (the target) is just serving sectors; the client (the initiator) owns the filesystem. This is the right tool when a workload wants a disk of its own — a database that manages its own storage, a VM datastore, a system that needs a block device rather than a shared folder.

The vocabulary is small but non-negotiable:

The target side: targetcli

The server uses targetcli — an interactive shell that edits the kernel’s LIO target and persists to /etc/target/saveconfig.json. Install targetcli (RHEL) / targetcli-fb (Debian), enable the service (target.service on RHEL; rtslib-fb-targetctl.service on Debian), then build four things: a backstore, a target, a LUN mapping the backstore, and an ACL for the client’s IQN.

sudo targetcli
/> /backstores/fileio create name=disk0 file_or_dev=/srv/iscsi/disk0.img size=1G
/> /iscsi create iqn.2026-07.com.kloudvin:target0
/> /iscsi/iqn.2026-07.com.kloudvin:target0/tpg1/luns create /backstores/fileio/disk0
/> /iscsi/iqn.2026-07.com.kloudvin:target0/tpg1/acls create iqn.2026-07.com.kloudvin:client0
/> saveconfig
/> exit

The ACL’s IQN must match the client’s initiator name (from the client’s /etc/iscsi/initiatorname.iscsi) or the client will discover the target but see no LUNs. Open 3260/tcp on the target’s firewall (firewall-cmd --permanent --add-service=iscsi-target). You can layer CHAP username/password auth under the TPG for real deployments; the ACL alone is IP/IQN-scoped trust.

The initiator side: iscsiadm

The client installs iscsi-initiator-utils (RHEL) / open-iscsi (Debian), sets its IQN, then discovers and logs in with iscsiadm. Login is the moment a new /dev/sdX appears.

# 1. Set the initiator IQN to match the target's ACL:
echo "InitiatorName=iqn.2026-07.com.kloudvin:client0" | sudo tee /etc/iscsi/initiatorname.iscsi
sudo systemctl restart iscsid

# 2. Discover, then log in:
sudo iscsiadm -m discovery -t st -p 192.168.1.10
# 192.168.1.10:3260,1 iqn.2026-07.com.kloudvin:target0
sudo iscsiadm -m node -T iqn.2026-07.com.kloudvin:target0 -p 192.168.1.10 --login
# Login to [iface: default, target: iqn.2026-07..., portal: 192.168.1.10,3260] successful.

# 3. A NEW block device just appeared — treat it as a local disk:
lsblk --scsi
sudo mkfs.xfs /dev/sdX            # ⚠️ formats the LUN — it's now yours to own
sudo mkdir -p /data && sudo mount /dev/sdX /data

That is the whole mental shift: after --login, the LUN is /dev/sdX and you do everything you’d do to a local disk — fdisk/parted, mkfs, mount, even pvcreate if you want LVM on it. In fstab, use the filesystem’s UUID (not /dev/sdX, which can renumber) plus _netdev so the mount waits for the network and the iSCSI session.

⚠️ The single most dangerous iSCSI mistake: two initiators mounting the same LUN with an ordinary filesystem. Because each client thinks it owns the filesystem and caches metadata independently, two writers on one ext4/xfs LUN will corrupt it almost immediately. A LUN is single-writer unless you put a cluster filesystem (GFS2, OCFS2) on it with a full cluster stack. Share a LUN to exactly one host at a time — if you need many hosts writing shared files, that’s an NFS/Samba job, not iSCSI.

Which to use: NFS vs Samba vs iSCSI vs object

Given a “we need shared storage” request, the right answer falls out of two questions: files or blocks? and who are the clients?

The decision in one breath: NFS for sharing files between Unix/Linux hosts; Samba for sharing files with Windows (or mixed) clients; iSCSI when a single host wants a raw block device to format and own; object storage when you’re storing huge numbers of blobs behind an HTTP API rather than a mounted filesystem. The classic mistakes are reaching for iSCSI to share files among many servers (it isn’t a shared filesystem — you’ll corrupt it), or reaching for NFS/Samba when an application really wants block-level control (databases often prefer their own iSCSI LUN or local disk over a network file share). Object storage is a different universe — no POSIX, no mount, an HTTP PUT/GET API — and it wins for backups, archives and cloud-native apps precisely because it drops the filesystem semantics the other three preserve.

Hands-on lab

A complete, safe, self-contained lab on one VM. It uses loop devices (scratch files exposed as block devices) for the RAID array and localhost (127.0.0.1) for every network protocol, so you exercise the real commands without a second machine or a real disk. Run it on any Linux VM/cloud instance/WSL2 with root. Each step has the command, what you’ll see, and a one-line “what just happened.”

⚠️ The mdadm --create, mkfs and --fail commands here are destructive — but only to throwaway loop files. Do not substitute a real /dev/sdX.

1. Create five loop devices to act as disks.

sudo mkdir -p /srv/lab && cd /srv/lab
for i in 1 2 3 4 5; do sudo fallocate -l 512M disk$i.img; done
# attach each file to the next free loop device and print the name it got:
for i in 1 2 3 4 5; do sudo losetup -f --show disk$i.img; done
# /dev/loop0
# /dev/loop1
# ... through /dev/loop4  (assuming loop0-4 were free on this fresh VM)
losetup -a | grep /srv/lab

What happened: five real block devices (/dev/loop0..4) backed by plain files — safe to build a RAID array on and abuse exactly like disks.

2. Build a RAID5 array with a hot spare, and watch it build.

# 3 active + 1 spare (RAID5 needs a minimum of 3 active disks):
sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 \
     /dev/loop0 /dev/loop1 /dev/loop2 --spare-devices=1 /dev/loop3
# mdadm: array /dev/md0 started.
cat /proc/mdstat
# md0 : active raid5 loop3[4](S) loop2[3] loop1[1] loop0[0]
#       1046528 blocks super 1.2 level 5, 512k chunk ... [3/3] [UUU]

What happened: three loop devices became one RAID5 /dev/md0 (with loop3 as a spare (S)); on tiny loop files the resync is instant, so you see [3/3] [UUU] — fully healthy.

3. Put a filesystem on the array and mount it.

sudo mkfs.xfs /dev/md0
sudo mkdir -p /srv/share
sudo mount /dev/md0 /srv/share
echo "shared over three protocols" | sudo tee /srv/share/hello.txt
df -h /srv/share
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/md0       1016M   40M  977M   4% /srv/share

What happened: the fault-tolerant array now carries an xfs filesystem at /srv/share — this is the single dataset we’ll export three ways.

4. The failure/rebuild drill — fail a disk, watch the spare rebuild. ⚠️ Test array only.

sudo mdadm /dev/md0 --fail /dev/loop0
# mdadm: set /dev/loop0 faulty in /dev/md0
cat /proc/mdstat
# md0 : active raid5 loop3[4] loop2[3] loop1[1] loop0[0](F)
#       ... [3/2] [_UU]  recovery = ...   <-- spare loop3 rebuilding into slot 0
sudo mdadm /dev/md0 --remove /dev/loop0     # detach the failed member
sudo mdadm /dev/md0 --add /dev/loop0        # re-add it as the new spare
sudo mdadm --detail /dev/md0 | grep -E 'State|spare|active'

What happened: you failed a disk, saw the array go degraded ([_UU]) and the hot spare auto-rebuild, then removed the dead member and added a replacement — the entire real-world recovery cycle, with the filesystem staying mounted throughout.

5. Persist the array config.

# RHEL path shown; on Debian use /etc/mdadm/mdadm.conf + update-initramfs -u
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf
# ARRAY /dev/md0 metadata=1.2 spares=1 name=... UUID=...

What happened: the array’s UUID is recorded so it reassembles by name on the next boot instead of getting a random md127.

6. Export it over NFS and mount it from localhost.

# Install + start the server (RHEL: nfs-utils / nfs-server; Debian: nfs-kernel-server)
sudo dnf install -y nfs-utils 2>/dev/null || sudo apt-get install -y nfs-kernel-server
sudo systemctl enable --now nfs-server 2>/dev/null || sudo systemctl enable --now nfs-kernel-server
echo '/srv/share  127.0.0.1(rw,sync,root_squash,no_subtree_check)' | sudo tee -a /etc/exports
sudo exportfs -rav
# exporting 127.0.0.1:/srv/share
sudo mkdir -p /mnt/nfs
sudo mount -t nfs 127.0.0.1:/srv/share /mnt/nfs
cat /mnt/nfs/hello.txt
# shared over three protocols

What happened: the same filesystem is now reachable over NFS; mounting 127.0.0.1:/srv/share proves the export works and you can read the file the RAID array holds. (On the RHEL family, if a real write is denied, sudo setsebool -P nfs_export_all_rw on.)

7. Share it over Samba and mount it with a credentials file.

sudo dnf install -y samba cifs-utils 2>/dev/null || sudo apt-get install -y samba cifs-utils
sudo useradd -M -s /sbin/nologin labuser 2>/dev/null; printf 'labpass\nlabpass\n' | sudo smbpasswd -a -s labuser
sudo tee -a /etc/samba/smb.conf >/dev/null <<'EOF'

[labshare]
   path = /srv/share
   valid users = labuser
   read only = no
EOF
testparm -s 2>/dev/null | head -n 3
sudo systemctl restart smb nmb 2>/dev/null || sudo systemctl restart smbd nmbd
# credentials in a 600 file — NEVER on the command line:
printf 'username=labuser\npassword=labpass\n' | sudo tee /root/.smbcreds >/dev/null
sudo chmod 600 /root/.smbcreds
sudo mkdir -p /mnt/smb
sudo mount -t cifs //127.0.0.1/labshare /mnt/smb -o credentials=/root/.smbcreds,uid=0
ls /mnt/smb
# hello.txt

What happened: the same directory is now a Windows-style share, mounted via CIFS using a chmod 600 credentials file — the password never touched the command line or your history. (RHEL SELinux: sudo setsebool -P samba_export_all_rw on if writes are denied.)

8. Serve a separate LUN over iSCSI and consume it as a local disk.

sudo dnf install -y targetcli iscsi-initiator-utils 2>/dev/null || sudo apt-get install -y targetcli-fb open-iscsi
sudo mkdir -p /srv/iscsi
sudo systemctl enable --now target 2>/dev/null || sudo systemctl enable --now rtslib-fb-targetctl
# Build a target with a fileio LUN and an ACL for our local initiator:
sudo targetcli <<'EOF'
/backstores/fileio create name=lun0 file_or_dev=/srv/iscsi/lun0.img size=512M
/iscsi create iqn.2026-07.com.kloudvin:target0
/iscsi/iqn.2026-07.com.kloudvin:target0/tpg1/luns create /backstores/fileio/lun0
/iscsi/iqn.2026-07.com.kloudvin:target0/tpg1/acls create iqn.2026-07.com.kloudvin:client0
saveconfig
exit
EOF
# Initiator side (same box): set our IQN to match the ACL, then discover + login:
echo "InitiatorName=iqn.2026-07.com.kloudvin:client0" | sudo tee /etc/iscsi/initiatorname.iscsi
sudo systemctl restart iscsid
sudo iscsiadm -m discovery -t st -p 127.0.0.1
sudo iscsiadm -m node -T iqn.2026-07.com.kloudvin:target0 -p 127.0.0.1 --login
lsblk --scsi        # a NEW /dev/sdX appeared — that's the LUN

What happened: iSCSI handed the client a raw block device (a fresh /dev/sdX), not a folder — the fundamental file-vs-block difference, live. Note we used a separate backstore file for iSCSI, never /dev/md0, because md0 is already a mounted filesystem and block-sharing a live filesystem would corrupt it.

9. Format and use the iSCSI LUN.

ISCSI_DEV=$(lsblk -S -o NAME,TRAN -dn | awk '$2=="iscsi"{print "/dev/"$1}')
echo "iSCSI disk is $ISCSI_DEV"
sudo mkfs.ext4 "$ISCSI_DEV"          # ⚠️ you own this filesystem — format it
sudo mkdir -p /mnt/iscsi && sudo mount "$ISCSI_DEV" /mnt/iscsi
echo "block storage, formatted by the client" | sudo tee /mnt/iscsi/blk.txt

What happened: you partitioned/formatted and mounted the LUN exactly like a local disk — proving the client, not the server, owns an iSCSI filesystem.

10. Clean up completely. ⚠️ Tears down every layer built above.

# Unmount clients
sudo umount /mnt/iscsi /mnt/smb /mnt/nfs 2>/dev/null
# iSCSI logout + wipe target
sudo iscsiadm -m node -T iqn.2026-07.com.kloudvin:target0 -p 127.0.0.1 --logout 2>/dev/null
sudo targetcli clearconfig confirm=True 2>/dev/null
# NFS + Samba config back out
sudo sed -i '\#/srv/share  127.0.0.1#d' /etc/exports && sudo exportfs -rav
sudo sed -i '/\[labshare\]/,/read only = no/d' /etc/samba/smb.conf
# RAID + loop teardown
sudo umount /srv/share
sudo mdadm --stop /dev/md0
sudo mdadm --zero-superblock /dev/loop{0,1,2,3} 2>/dev/null
# detach every loop device backed by a /srv/lab scratch file:
losetup -a | grep /srv/lab | cut -d: -f1 | xargs -r -n1 sudo losetup -d
sudo rm -rf /srv/lab /srv/iscsi /mnt/{nfs,smb,iscsi} /root/.smbcreds

What happened: every artefact — mounts, iSCSI session, target, exports, Samba share, RAID array, loop devices, scratch files — is removed, leaving the system as you found it. Always remove network-mount fstab lines and exports when you tear down the storage, or the next boot trips over them.

Common mistakes and troubleshooting

The three nastiest gotchas, in prose:

1. “Permission denied” that isn’t a permission — it’s SELinux. On RHEL/Rocky/Fedora, the number-one shared-storage support call is a share with textbook-correct chmod/chown that still refuses access. The daemon is being blocked by SELinux, not by the file mode. The tell is that ls -l looks perfect and audit.log shows an AVC denied line. The fix is a boolean (setsebool -P nfs_export_all_rw on or samba_export_all_rw on) or the proper file-context label (samba_share_t). Reach for getsebool -a | grep -E 'nfs|samba' and ausearch -m avc -ts recent before you start loosening file permissions — loosening perms won’t help and just weakens security.

2. The /etc/exports space that inverts your intent. /srv/share 192.168.1.0/24(rw) exports read-write to that network. /srv/share 192.168.1.0/24 (rw) — one space before the paren — parses as two things: read-only (default) to 192.168.1.0/24, and read-write to everyone else on Earth. A single stray space turns a locked-down export into a world-writable one. Always exportfs -s after editing and read back exactly who has rw.

3. Two hosts, one iSCSI LUN, instant corruption. iSCSI feels like shared storage — several machines can discover the same target — but a LUN is a raw disk, and an ordinary filesystem assumes it is the only writer. Mount one LUN read-write on two initiators and each caches its own idea of the metadata; within minutes the filesystem is scrambled beyond fsck. If you truly need many hosts writing the same blocks you must run a cluster filesystem (GFS2, OCFS2) with fencing — otherwise iSCSI is single-host, and “many hosts share files” is precisely the job NFS and Samba exist to do.

Cheat-sheet

Interview and exam questions

Q: A colleague says “we have RAID, so we’re backed up.” Correct them. A: RAID is redundancy on one host, not a backup. It protects against disk failure, but rm -rf, filesystem corruption, ransomware, a bad deploy, or the whole machine dying all propagate to every member instantly. You still need real backups (separate media/site). RAID improves availability; backups provide recoverability — different problems.

Q: When would you choose RAID 6 over RAID 5, and why not always RAID 5? A: RAID 6 keeps double parity and survives two simultaneous disk failures; RAID 5 survives one. On large modern disks a rebuild reads every surviving disk for hours, and a second failure (or an unreadable sector) during that window destroys a RAID 5 array — so RAID 6 is the default for big arrays. RAID 5’s cost is one fewer disk of overhead and a lighter write penalty (4 vs 6 I/Os per write).

Q: Walk through replacing a failed disk in an mdadm array. A: cat /proc/mdstat / mdadm --detail to confirm the failure and degraded state; mdadm /dev/md0 --remove /dev/sdX the faulty member; physically swap the disk; mdadm /dev/md0 --add /dev/sdX the replacement, which rebuilds (watch /proc/mdstat). If a hot spare existed, the rebuild started automatically at failure. Never fail a second disk on a single-parity array mid-rebuild.

Q: What does root_squash do, and when would you disable it? A: It maps a client’s root (UID 0) to the unprivileged nobody, so remote root can’t own or overwrite files on the export — the safe default under NFS’s trust-the-client sec=sys model. You disable it (no_root_squash) only for a specific trusted host that must preserve root ownership, e.g. a backup server, scoped to one IP. For real security over untrusted networks, use sec=krb5p instead.

Q: Why prefer NFSv4 over NFSv3 for a new deployment? A: v4 uses a single port (2049/tcp), so the firewall is trivial; v3 needs rpcbind (111) plus mountd/statd/lockd on dynamic ports you must pin. v4 is stateful with built-in locking, presents a single pseudo-filesystem namespace, and has first-class Kerberos and ACL support. v3’s only edge is legacy client compatibility.

Q: Why is autofs the right way to mount NFS home directories? A: A fstab hard-mount hangs the client if the server is down and keeps idle mounts alive; autofs mounts a path only when it’s touched and unmounts it after an idle timeout. One wildcard map line (* -rw server:/export/home/&, where & = the username) serves every user with no per-user config and no boot-time dependency on the server being reachable.

Q: You’re mounting a CIFS share. Why not put the password in the mount command? A: A password in -o username=,password= is visible to any user via ps aux, is written to your shell history, and can land in logs. Put username=/password= in a root-owned chmod 600 credentials file and pass credentials=/root/.smbcreds. It’s both more secure and cleaner in fstab.

Q: An NFS export has correct permissions but a client still gets “permission denied” on a RHEL server. Where do you look? A: SELinux. nfsd/smbd are confined by booleans; correct chmod/chown isn’t enough. Check getsebool -a | grep nfs and ausearch -m avc -ts recent for an AVC denial, then setsebool -P nfs_export_all_rw on (or apply the proper file context). Don’t loosen file permissions — that won’t fix an SELinux denial.

Q: What’s the fundamental difference between iSCSI and NFS/Samba? A: NFS/Samba serve files — the server owns the filesystem and many clients share files inside it, with locking. iSCSI serves blocks — the client receives a raw /dev/sdX, and the client partitions, formats and owns the filesystem. So iSCSI is single-writer (two hosts on one LUN with an ordinary fs = corruption), whereas NFS/Samba are built for concurrent multi-client access.

Q: (RHCSA-style) Export /srv/data read-write to 192.168.10.0/24 with root squashing, and open the firewall. A:

echo '/srv/data 192.168.10.0/24(rw,sync,root_squash,no_subtree_check)' | sudo tee -a /etc/exports
sudo exportfs -rav
sudo systemctl enable --now nfs-server
sudo setsebool -P nfs_export_all_rw on
sudo firewall-cmd --permanent --add-service=nfs && sudo firewall-cmd --reload

Q: (LFCS-style) Attach an iSCSI LUN from target 192.168.10.5 and mount its filesystem persistently. A:

sudo iscsiadm -m discovery -t st -p 192.168.10.5
sudo iscsiadm -m node -T iqn.2026-07.example:target0 -p 192.168.10.5 --login
# new /dev/sdX appears:
sudo mkfs.xfs /dev/sdX && UUID=$(sudo blkid -s UUID -o value /dev/sdX)
echo "UUID=$UUID /data xfs _netdev 0 0" | sudo tee -a /etc/fstab   # _netdev is mandatory
sudo iscsiadm -m node -T iqn.2026-07.example:target0 -p 192.168.10.5 --op update -n node.startup -v automatic

Q: An mdadm array comes up as /dev/md127 with a random name after reboot. Why, and how do you fix it? A: The array isn’t defined in mdadm.conf (and the initramfs), so the kernel auto-assembles it under a fallback name. Fix: mdadm --detail --scan >> /etc/mdadm.conf to record its UUID and name, then rebuild the initramfs (dracut -f on RHEL, update-initramfs -u on Debian) so it assembles correctly at boot.

Key takeaways

• RAID is redundancy on one host — not a backup and not sharing. mdadm builds /dev/md0 from disks; choose the level by trading capacity vs fault tolerance vs write speed (RAID 6 is the safe default for big arrays, RAID 10 for databases, never RAID 0 for real data). Pair it with LVM on top for flexibility.
• Rehearse the failure drill until it’s boring: --fail → watch /proc/mdstat → --remove → --add. A hot spare rebuilds automatically; a degraded array with no spare is an urgent, same-day ticket. Persist every array in mdadm.conf + the initramfs or it comes back as md127.
• NFS shares files with Unix clients. Get /etc/exports right (rw,sync,root_squash, and mind the fatal space before (), apply with exportfs -rav, prefer NFSv4 (one port, 2049), and mount with _netdev. root_squash is your default protection; sec=krb5p for real security.
• autofs is the correct pattern for home dirs and sometimes-needed shares — on-demand mount, idle unmount, one wildcard map (& = key), no boot-time hang if the server is away.
• Samba shares files with Windows. Samba users are separate (smbpasswd -a), validate with testparm, run smb/nmb, and on the client always use a chmod 600 credentials file — never a password on the command line.
• On the RHEL family, SELinux is the number-one “permission denied” cause for shares whose file modes are perfect. Flip the persistent boolean (nfs_export_all_rw, samba_export_all_rw) or set the file context; check ausearch -m avc before touching permissions.
• iSCSI serves blocks, not files. The client sees a raw /dev/sdX it formats and owns — brilliant for a single host wanting its own disk, catastrophic if two hosts mount one LUN with an ordinary filesystem. Build the target with targetcli (backstore → target → LUN → ACL), attach with iscsiadm discovery + login.
• Pick by files-or-blocks and who the clients are: NFS (Unix files, multi-writer), Samba (Windows files), iSCSI (single-host block device), object storage (HTTP blobs at scale, no filesystem). Matching the tool to the problem is half the job — and every one of them needs its firewall port opened.