Shell /proc, /sys & sysctl: Kernel Introspection, Runtime Tuning, Persistent Configs & Per-Process Forensics From the Command Line

In a nutshell

Most operating systems seal the kernel inside a black box and make you buy special tools — a debugger, a profiler, a proprietary agent — to peek inside. Linux does the opposite. It turns the running kernel inside out and lays it on the floor as a pile of ordinary text files you can cat. Want to know a process’s memory, its open files, which namespace it lives in? Read a file. Want to know how much RAM is free or how loaded the machine is? Read a file. Want to change how aggressively the kernel swaps, or how big a network backlog it will accept? Write a file.

Think of it as the instrument panel and control console of a running engine. /proc and /sys are the gauges — the tachometer, the fuel level, the temperature — and reading a gauge never breaks anything. sysctl (which is just the /proc/sys corner of that panel) is the set of knobs you can turn while the engine runs: turn one and the behaviour of the whole machine changes right now, for every process, no reboot, no recompile. That asymmetry is the single most important idea in this lesson: reading is free and safe; writing is a live change to a running kernel.

There are three doors, and it pays to know which is which:

/proc — processes and kernel state. /proc/$pid/ is a live X-ray of one running program; files like /proc/meminfo and /proc/loadavg are the system-wide numbers that top and free are really just reading for you.
/sys — hardware and drivers. One knob per device: a NIC’s link state, a disk’s I/O scheduler, a CPU’s frequency governor.
/proc/sys — the tunable kernel parameters, the exact same set sysctl shows. sysctl -w writes them at runtime; a file in /etc/sysctl.d/ makes the change survive a reboot.

Master those three filesystems and a huge amount of “I need a special tool for this” collapses into cat, echo, awk, and sysctl.

Level: Advanced · Time: ~40 min

Prerequisites: You should be comfortable with what a PID is and how processes fork from Shell anatomy & the process model, managing background jobs and PIDs from Process management: subshells, jobs & wait, what a file descriptor is from I/O redirection & file descriptors, and enough awk to pull a field out of a line from Globbing, regex, find, grep, sed & awk. Everything here is Linux-only — macOS and the BSDs have sysctl but no /proc or /sys (see Going deeper), so the labelled output below is representative of a Linux host.

After this lesson you will be able to:

Read any running process’s state — name, memory, open files, memory map, namespaces — straight from /proc/$pid/, no lsof or ps required.
Pull system-wide numbers (memory, load, CPU %, uptime) out of /proc/meminfo, /proc/loadavg, and /proc/stat with cat + awk.
Read and, where safe, write device knobs under /sys (link state, MTU, I/O scheduler, CPU governor).
Explain the difference between reading a gauge and turning a knob, and tell writable tunables from read-only ones before you touch them.
Tune the kernel with sysctl -w, understand why the change is lost on reboot, and persist it correctly via /etc/sysctl.d/*.conf with a rollback path.
Write safe introspection-and-tuning helpers that handle the racy, root-gated, container-namespaced reality of /proc and /sys.

The kernel exposed as files: the shell reads gauges from /proc and /sys with cat and awk, turns tunable knobs with sysctl, and persists the ones that must survive a reboot into /etc/sysctl.d

Read the diagram left → right. You touch the kernel with two verbs — cat/awk to read (always safe) and echo >/sysctl -w to write (a live change). /proc gives you per-process X-rays (status, fd/, maps, ns/) and system-wide numbers (meminfo, loadavg, stat); /sys gives you the same trick aimed at hardware; /proc/sys (a.k.a. sysctl) holds the tunable knobs — some read-only, some clamped, some root-only. A runtime write vanishes on reboot, so the ones you mean to keep go into /etc/sysctl.d/99-*.conf and sysctl --system reapplies them at boot.

Why /proc, /sys, and sysctl Matter for Shell Operators

Most monitoring you do from shell — checking memory, listing open files, reading network connections, looking at namespaces, tuning kernel limits — does not need a special tool. The kernel exposes everything you need as text files under three magic directories:

/proc — process and kernel state. Per-process data lives in /proc/$pid/. Global kernel state lives in /proc/<various> (e.g. /proc/meminfo, /proc/cpuinfo).
/proc/sys — runtime-tunable kernel parameters. The same as sysctl shows, but exposed as files: cat /proc/sys/net/ipv4/ip_forward, echo 1 > /proc/sys/net/ipv4/ip_forward.
/sys — device and driver state. Per-device knobs live under /sys/class/, /sys/devices/, /sys/block/. CPU frequency scaling, block-device queue depth, network interface flags — all live here.

When you understand these three filesystems, a huge category of tooling becomes “just use cat and echo”:

“What’s the memory layout of my Java process?” → cat /proc/$pid/status
“What network namespace is this container in?” → readlink /proc/$pid/ns/net
“Which files does this process have open?” → ls -l /proc/$pid/fd/
“How much swap is being used right now?” → awk '/^Swap/' /proc/meminfo
“Tune TCP buffers on a flaky network” → sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
“Persist across reboots” → write to /etc/sysctl.d/99-tcp-tuning.conf

This lesson covers the layout, the conventions, the persistence model, and a lib/proc.sh of helper functions you can use to query and tune from any script.

/proc — The Process Filesystem

/proc is a virtual filesystem. The files don’t exist on disk; the kernel synthesizes them every time you read. This has consequences:

You cannot trust file size. stat /proc/meminfo shows size 0; the actual content materializes only on read.
Reads are not atomic with respect to the underlying state. The contents of /proc/$pid/maps can change between consecutive reads if the process is allocating memory.
Some files are accessible only to root or to the process owner (look at the file’s owner: ls -l /proc/1/maps is root, ls -l /proc/$$/maps is you).
Some entries disappear when the process exits. Operating on /proc/$pid/... is racy with the process’s lifecycle.

Per-process layout

The directory /proc/$pid/ (where $pid is a PID, or self for the current process) contains:

/proc/$pid/
├── status              # human-readable summary: name, state, uid, mem, threads
├── stat                # space-separated, one line; same data, machine-readable
├── statm               # memory in pages (size, resident, shared, ...)
├── cmdline             # NUL-separated argv (the actual process arguments)
├── environ             # NUL-separated environment (root or owner only)
├── exe                 # symlink → the executable file
├── cwd                 # symlink → current working directory
├── root                # symlink → root directory (different in chroots)
├── fd/                 # one symlink per open file descriptor
│   ├── 0 -> /dev/pts/2
│   ├── 1 -> /dev/null
│   └── 4 -> /var/log/myapp.log
├── fdinfo/             # offset, flags per fd
├── maps                # VM memory map: ranges, perms, mapped files
├── smaps               # detailed per-mapping memory accounting
├── io                  # bytes read/written by the process
├── limits              # rlimit values: max files, stack, ...
├── ns/                 # namespaces: net, mnt, pid, user, uts, ipc, cgroup
│   ├── net -> 'net:[4026531992]'
│   └── mnt -> 'mnt:[4026531840]'
├── cgroup              # cgroup memberships
├── sched               # scheduler statistics
├── stack               # current kernel stack trace (CONFIG_STACKTRACE)
└── task/$tid/          # one subdirectory per thread (same layout as $pid/)

Recipe: read process info portably

# Read PID, name, and state.
proc_status() {
  local pid="$1"
  [[ -d "/proc/$pid" ]] || { echo "no such pid: $pid" >&2; return 1; }

  local name state ppid threads vmrss
  while IFS=$'\t' read -r key value; do
    case "$key" in
      "Name:")    name=$value ;;
      "State:")   state=$value ;;
      "PPid:")    ppid=$value ;;
      "Threads:") threads=$value ;;
      "VmRSS:")   vmrss=$value ;;
    esac
  done < "/proc/$pid/status"

  printf 'pid=%s name=%s state=%s ppid=%s threads=%s rss=%s\n' \
    "$pid" "$name" "$state" "$ppid" "$threads" "$vmrss"
}

proc_status $$
# pid=12345 name=bash state=S (sleeping) ppid=12340 threads=1 rss=4096 kB

Recipe: list a process’s open files (tiny lsof)

proc_fds() {
  local pid="$1"
  [[ -d "/proc/$pid/fd" ]] || return 1
  local fd target
  for fd in /proc/$pid/fd/*; do
    target=$(readlink "$fd" 2>/dev/null) || continue
    printf 'fd=%-3s target=%s\n' "$(basename "$fd")" "$target"
  done
}

proc_fds 1234
# fd=0   target=/dev/null
# fd=1   target=pipe:[123456]
# fd=2   target=/var/log/myapp.log
# fd=4   target=socket:[789012]

This is what lsof does, but lsof walks every PID; if you know the PID you care about, /proc/$pid/fd/ is much faster (single readdir).

Recipe: identify a socket from its inode

socket:[INODE] from /proc/$pid/fd/ is opaque. Resolve it via /proc/net/tcp (or /proc/net/udp):

# /proc/net/tcp columns:
# sl  local_address rem_address st tx_queue:rx_queue tr:tm->when retrnsmt uid timeout inode
#  0: 0100007F:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000  0      0 12345

socket_inode_to_addr() {
  local inode="$1"
  awk -v ino="$inode" '
    NR>1 && $10==ino {
      # local_address is hex IP:hex PORT, little-endian for IP.
      split($2, a, ":")
      ip=a[1]; port=a[2]
      # Convert IP from hex little-endian to dotted decimal.
      printf "%d.%d.%d.%d:%d  state=%s\n",
        strtonum("0x"substr(ip,7,2)), strtonum("0x"substr(ip,5,2)),
        strtonum("0x"substr(ip,3,2)), strtonum("0x"substr(ip,1,2)),
        strtonum("0x"port), $4
    }' /proc/net/tcp
}

This is reverse-engineerable but well-documented. Most production scripts use ss -tnp or lsof -i instead — but knowing where the data comes from helps when those tools are unavailable (minimal containers).

Recipe: detect what namespace a process is in

# Each ns symlink has the form 'net:[INODE]'. Two PIDs in the same namespace
# share the same inode.
ns_id() { readlink "/proc/$1/ns/$2"; }

# Compare two processes' namespaces:
[[ "$(ns_id 1234 net)" == "$(ns_id 5678 net)" ]] && echo "same network ns"

# Find the host's network namespace inode (PID 1 is init):
ns_id 1 net
# Output: net:[4026531992]

This is how nsenter and container tooling figure out which namespace to enter. For diagnostics: “is this process in a different network namespace from the host?” — compare ns_id $pid net with ns_id 1 net.

Recipe: scrape memory layout from /proc/$pid/maps

# /proc/$pid/maps lines:
# 7fc0a8e4f000-7fc0a9000000 r-xp 00000000 fd:00 524294  /usr/lib/x86_64-linux-gnu/libc-2.31.so

proc_libs() {
  local pid="$1"
  awk '$6 ~ /\.so/ { print $6 }' "/proc/$pid/maps" | sort -u
}

proc_libs $$
# /usr/lib/.../libc.so.6
# /usr/lib/.../libdl.so.2
# /usr/lib/.../libtinfo.so.6

maps reveals every shared library, every mapped file, every executable region. For forensics: “is this binary loading something it shouldn’t?” → grep maps for unexpected paths.

System-wide /proc: the numbers `top`, `free`, and `uptime` actually read

Everything above was per-process (/proc/$pid/...). The other half of /proc is global kernel state — flat files at the top of /proc that summarise the whole machine. These are the files the classic monitoring commands read for you: free parses /proc/meminfo, uptime parses /proc/loadavg, top’s CPU line comes from /proc/stat. Read them directly and you never have to shell out to another binary — which is exactly why this keeps working in a stripped-down container or a rescue shell where top and free aren’t installed.

/proc/loadavg — one line, five fields:

cat /proc/loadavg
# 0.45 0.30 0.25 2/512 12345    (representative)
#  |    |    |    |     └ PID of the most recently created process
#  |    |    |    └─────── running/total tasks (2 runnable of 512)
#  |    |    └──────────── 15-minute load average
#  |    └───────────────── 5-minute load average
#  └────────────────────── 1-minute load average

Load average is the number of tasks runnable or waiting on uninterruptible I/O, exponentially averaged. On an 8-core box, a 1-minute load of 8.0 means “fully busy”; 16.0 means “twice as much work as cores”. Pull one field with awk:

awk '{print $1}' /proc/loadavg          # 1-minute load, e.g. 0.45

/proc/meminfo — memory in kB, key by key:

awk '/^MemTotal:|^MemAvailable:|^SwapFree:/ {printf "%-14s %s kB\n",$1,$2}' /proc/meminfo
# MemTotal:      32825036 kB   (representative)
# MemAvailable:  16307180 kB
# SwapFree:       2097148 kB

The single most important line is MemAvailable, not MemFree. MemFree is RAM sitting completely idle; MemAvailable is the kernel’s own estimate of how much a new process could use without swapping — it accounts for reclaimable page cache. “Used memory %” the way a human means it:

mem_used_pct() {
  awk '/^MemTotal:/{t=$2} /^MemAvailable:/{a=$2}
       END{ printf "%.1f\n", 100*(t-a)/t }' /proc/meminfo
}
mem_used_pct     # 50.3   (representative)

/proc/stat — CPU time in jiffies:

The first line, cpu, is aggregate CPU time since boot, split into user/nice/system/idle/iowait/… measured in USER_HZ “jiffies” (usually 1/100 s). A single read is meaningless — CPU utilisation is the change over an interval. Take two snapshots and diff them:

# %CPU busy over an interval, from /proc/stat.
cpu_busy_pct() {
  local interval="${1:-1}"
  _cpu_idle_total() {
    awk '/^cpu /{ idle=$5+$6; total=0; for(i=2;i<=NF;i++) total+=$i;
                  print idle, total }' /proc/stat
  }
  local i1 t1 i2 t2
  read -r i1 t1 < <(_cpu_idle_total)
  sleep "$interval"
  read -r i2 t2 < <(_cpu_idle_total)
  awk -v i1="$i1" -v t1="$t1" -v i2="$i2" -v t2="$t2" \
    'BEGIN{ di=i2-i1; dt=t2-t1; printf "%.1f\n", dt ? 100*(dt-di)/dt : 0 }'
}
cpu_busy_pct 1   # 14.8   (representative: 14.8% busy over 1s)

idle here is idle + iowait ($5+$6); total is the sum of every field on the line. Busy% is 100 × (Δtotal − Δidle) / Δtotal. This is exactly what top computes — you’ve just inlined it.

Other high-value global files:

File	What it gives you	One-liner
`/proc/uptime`	uptime + summed idle seconds	`awk '{print $1"s up"}' /proc/uptime`
`/proc/cpuinfo`	per-core model, flags, MHz	`grep -c ^processor /proc/cpuinfo` (core count)
`/proc/swaps`	active swap devices + usage	`cat /proc/swaps`
`/proc/diskstats`	per-device I/O counters	`awk '$3=="sda"{print $6,$10}' /proc/diskstats`
`/proc/mounts`	the kernel’s live mount table	`grep ' /data ' /proc/mounts`
`/proc/cmdline`	kernel boot parameters	`cat /proc/cmdline`
`/proc/net/dev`	per-interface byte/packet counters	`awk -F: '/eth0/{print $2}' /proc/net/dev`

The pattern is always the same: a virtual text file, awk to pick the field, arithmetic if you need a rate. No agent, no /usr/bin dependency.

/sys — The Device Filesystem

/sys is similar in spirit to /proc but tied to the kernel’s device model. The shape:

/sys/
├── class/                 # by-functionality view (block, net, leds, thermal)
│   ├── net/eth0/          # symlink to /sys/devices/.../eth0
│   │   ├── operstate      # 'up' | 'down'
│   │   ├── mtu            # 1500
│   │   └── statistics/
│   │       └── rx_bytes
│   └── block/sda/
│       ├── size           # in 512-byte sectors
│       └── queue/scheduler  # 'mq-deadline [bfq] none'
├── devices/               # the underlying device tree (PCI, USB, ...)
├── module/                # loaded kernel modules and their parameters
└── kernel/                # kernel state knobs (rcu, debug, ...)

Useful one-liners

# Network interface link state.
cat /sys/class/net/eth0/operstate          # up

# Total bytes received on eth0 (no parsing /proc/net/dev needed).
cat /sys/class/net/eth0/statistics/rx_bytes

# Block device size in bytes.
echo $(( $(cat /sys/class/block/sda/size) * 512 ))

# Current I/O scheduler for sda.
cat /sys/class/block/sda/queue/scheduler   # mq-deadline [bfq] none
# Change it (writable):
echo deadline > /sys/class/block/sda/queue/scheduler

# CPU frequency governor.
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor   # performance / powersave / ...
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Thermal zone temperature (millidegrees C).
cat /sys/class/thermal/thermal_zone0/temp   # 47000 means 47.000 °C

`/sys` vs `sysctl`

/sys is per-device. sysctl (which is /proc/sys/...) is per-subsystem and global. Tuning a single network interface’s MTU is /sys/class/net/eth0/mtu. Tuning system-wide TCP buffer sizes is sysctl net.ipv4.tcp_rmem. They don’t overlap; learn both.

sysctl — Runtime Kernel Tuning

sysctl is the canonical interface for kernel tunables. The mapping is mechanical:

# These three are the same setting.
sysctl net.ipv4.ip_forward
cat /proc/sys/net/ipv4/ip_forward
# (also exposed via /sys depending on subsystem)

# Read all current values.
sysctl -a   # huge; pipe through grep for what you care about

# Read one.
sysctl -n net.ipv4.ip_forward   # -n: print value only, no key=value

# Write (runtime-only; lost on reboot).
sysctl -w net.ipv4.ip_forward=1
# Equivalent:
echo 1 > /proc/sys/net/ipv4/ip_forward

Reading is safe; writing is a live kernel change

The single most important habit in this whole lesson: cat never hurts, echo > might. Reading any file under /proc, /sys, or /proc/sys is non-destructive — you are asking the kernel a question. Writing to /proc/sys/... (via echo > or sysctl -w) mutates the running kernel immediately and globally: the new value applies to every process on the machine, this instant, with no confirmation and no undo button.

That asymmetry drives three rules.

1. Not every knob is writable — check before you try. Many entries under /proc/sys are informational or root-only, and inside a container most are read-only. The permission bit tells you:

# Is this tunable writable by me right now?
if [[ -w /proc/sys/net/ipv4/ip_forward ]]; then
  echo "writable"
else
  echo "read-only (need root, or namespaced/locked)"
fi

ls -l shows the same thing: a -rw-r--r-- sysctl is root-writable; -r--r--r-- is read-only to everyone (the kernel exposes it for inspection only).

2. The kernel may silently clamp what you write — always re-read. Ask for a socket buffer of a billion bytes and the kernel may quietly give you its maximum instead, with no error. The write “succeeds”; the value that took is different. So writing is a two-step operation — set, then verify:

sysctl -w net.core.rmem_max=999999999      # "succeeds"
sysctl -n net.core.rmem_max                # re-read: kernel may show a smaller, clamped value

3. Snapshot before you change, so you can roll back. A live tuning change on a production box should always be reversible in one command:

prev=$(sysctl -n vm.swappiness)            # remember the old value
sysctl -w vm.swappiness=10                 # try the new one
# ... observe; if it made things worse:
sysctl -w "vm.swappiness=$prev"            # instant rollback

None of this persists yet — every write here is runtime-only and disappears on the next reboot. Persisting a change you’ve decided to keep is the next section.

Persistence: `/etc/sysctl.conf` vs `/etc/sysctl.d/`

Settings written via sysctl -w or echo > /proc/sys/... are runtime-only. To persist across reboot, write them to a config file that’s applied at boot.

The 3-tier loading order (modern systemd-based distros):

/usr/lib/sysctl.d/*.conf — distro defaults (e.g. 99-sysctl.conf).
/run/sysctl.d/*.conf — runtime overrides (rarely used).
/etc/sysctl.d/*.conf — your overrides.
/etc/sysctl.conf — legacy single-file config (still supported; consider deprecated).

Files within each directory are loaded in lexicographic order; later wins. This is why convention is 99-myapp.conf (run last) for overrides and 10-something.conf (run early) for defaults. Use /etc/sysctl.d/, not /etc/sysctl.conf — multiple management tools cooperate via per-tool files, and per-tool files are easier to enable/disable.

Recipe: persist a sysctl change with rollback

sysctl_persist() {
  local key="$1" value="$2" reason="${3:-no reason given}"
  local file="/etc/sysctl.d/99-$(printf '%s' "$key" | tr '.' '-').conf"

  # Write the persistent file.
  cat >"$file" <<EOF
# Set by ensure_sysctl on $(date -u +%FT%TZ): $reason
$key = $value
EOF

  # Apply now (so the change takes effect without a reboot).
  sysctl -w "$key=$value" >/dev/null

  # Verify.
  local actual
  actual=$(sysctl -n "$key")
  if [[ "$actual" != "$value" ]]; then
    echo "sysctl_persist: failed to apply $key=$value (got '$actual')" >&2
    return 1
  fi
}

# Usage:
sysctl_persist net.ipv4.ip_forward 1 "enable IP forwarding for k8s networking"
sysctl_persist vm.swappiness 10 "favor cache over swap on this DB host"

Recipe: validate sysctl values before applying

Some sysctl values are constrained (e.g. net.core.rmem_max must be ≥ net.core.rmem_default). Always verify:

sysctl_safe_set() {
  local key="$1" value="$2"
  # Snapshot current value for rollback.
  local prev
  prev=$(sysctl -n "$key" 2>/dev/null) || { echo "no such sysctl: $key" >&2; return 1; }
  # Apply.
  if ! sysctl -w "$key=$value" >/dev/null 2>&1; then
    echo "sysctl rejected $key=$value; keeping $prev" >&2
    return 1
  fi
  # Validate via re-read (kernel may have clamped).
  local actual
  actual=$(sysctl -n "$key")
  if [[ "$actual" != "$value" ]]; then
    echo "warning: requested $value, kernel set $actual (clamping)" >&2
  fi
}

Recipe: dump current vs default to detect drift

# Compare the running kernel's values to /etc/sysctl.d/* declared values.
sysctl_drift() {
  local file actual declared key value
  for file in /etc/sysctl.d/*.conf; do
    while IFS='=' read -r key value; do
      [[ -z "${key// }" || "${key:0:1}" == "#" ]] && continue
      key="${key// /}"; value="${value# }"
      actual=$(sysctl -n "$key" 2>/dev/null) || continue
      if [[ "$actual" != "$value" ]]; then
        printf '%s: declared=%s actual=%s (file=%s)\n' "$key" "$value" "$actual" "$file"
      fi
    done < "$file"
  done
}

Useful in CI: “did someone change a sysctl at runtime that’s out of sync with the persistent config?” Drift before audit, not during.

/proc/sys Tunables Worth Knowing

A reference of high-leverage tunables most production hosts touch:

Tunable	Meaning	Common values
`vm.swappiness`	0=never swap unless OOM, 100=swap aggressively	1–10 for DB; 60 default
`vm.overcommit_memory`	0=heuristic, 1=always allow, 2=strict accounting	1 for Redis; 2 for paranoid
`vm.dirty_ratio`	% of RAM dirty before sync writeback blocks	10–20 (default 20)
`vm.dirty_background_ratio`	% of RAM dirty before bg writeback starts	5–10 (default 10)
`net.ipv4.ip_forward`	Enable routing	1 for routers/k8s nodes
`net.ipv4.tcp_fin_timeout`	TIME_WAIT seconds	15–30 for high-conn servers
`net.ipv4.tcp_tw_reuse`	Reuse TIME_WAIT sockets	1 for outbound-heavy clients
`net.core.somaxconn`	Listen backlog cap	4096+ for high-RPS servers
`net.core.rmem_max` / `wmem_max`	Max socket buffer	16777216 for big BDP links
`net.ipv4.tcp_rmem` / `tcp_wmem`	TCP buffer min/default/max	4096 87380 16777216
`net.ipv4.tcp_keepalive_time`	Idle before keepalives	300 (default 7200)
`fs.file-max`	System-wide max open files	2097152+ for fd-heavy hosts
`fs.inotify.max_user_watches`	inotify watches per user	524288+ for k8s hosts
`kernel.pid_max`	Max PID value	4194304 for high-fork hosts
`kernel.panic_on_oops`	Panic on kernel oops (cluster reset)	0 default; 1 in HA

Add kernel.dmesg_restrict=1 and kernel.kptr_restrict=2 for hardening.

Putting It Together: `lib/proc.sh`

# lib/proc.sh — process and kernel introspection helpers.

# ─── Process queries ───────────────────────────────────────────────────────

proc_exists() { [[ -d "/proc/$1" ]]; }

proc_name() {
  [[ -r "/proc/$1/comm" ]] && cat "/proc/$1/comm"
}

# Read /proc/$pid/status into associative-array-style output.
proc_status_kv() {
  local pid="$1" key value
  while IFS=$'\t' read -r key value; do
    key="${key%:}"
    printf '%s=%s\n' "$key" "$value"
  done < "/proc/$pid/status"
}

# RSS in kB.
proc_rss() {
  awk '/^VmRSS:/ {print $2}' "/proc/$1/status" 2>/dev/null
}

# UID owning the process.
proc_uid() {
  awk '/^Uid:/ {print $2}' "/proc/$1/status"
}

# Walk children of a PID.
proc_children() {
  local parent="$1"
  for pid in /proc/[0-9]*; do
    pid=${pid##*/}
    [[ "$(awk '/^PPid:/ {print $2}' "/proc/$pid/status" 2>/dev/null)" == "$parent" ]] \
      && echo "$pid"
  done
}

# ─── Open-file inspection (mini lsof) ──────────────────────────────────────

proc_open_files() {
  local pid="$1" fd target
  for fd in /proc/$pid/fd/*; do
    target=$(readlink "$fd" 2>/dev/null) || continue
    printf '%s\t%s\n' "$(basename "$fd")" "$target"
  done
}

# Find PIDs that have a given path open.
proc_holders_of() {
  local path="$1" pid fd
  path=$(readlink -f "$path")
  for pid in /proc/[0-9]*; do
    pid=${pid##*/}
    for fd in /proc/$pid/fd/*; do
      [[ "$(readlink "$fd" 2>/dev/null)" == "$path" ]] && {
        echo "$pid"
        break
      }
    done
  done
}

# ─── Namespace inspection ──────────────────────────────────────────────────

ns_inode() { readlink "/proc/$1/ns/$2"; }

ns_same_as_host() {
  [[ "$(ns_inode "$1" "$2")" == "$(ns_inode 1 "$2")" ]]
}

# ─── sysctl helpers ────────────────────────────────────────────────────────

sysctl_get() { sysctl -n "$1" 2>/dev/null; }

sysctl_set() {
  local key="$1" value="$2"
  sysctl -w "$key=$value" >/dev/null
}

sysctl_persist() {
  local key="$1" value="$2" reason="${3:-managed}"
  local fname
  fname=$(printf '%s' "$key" | tr '.' '-')
  local file="/etc/sysctl.d/99-${fname}.conf"

  cat >"$file" <<EOF
# Managed: $reason
# Set by lib/proc.sh on $(date -u +%FT%TZ)
$key = $value
EOF
  chmod 0644 "$file"
  sysctl_set "$key" "$value"
}

# Rollback: remove the managed file and reload.
sysctl_unmanage() {
  local key="$1"
  local fname
  fname=$(printf '%s' "$key" | tr '.' '-')
  rm -f "/etc/sysctl.d/99-${fname}.conf"
  sysctl --system >/dev/null
}

# ─── /sys helpers ──────────────────────────────────────────────────────────

block_size_bytes() {
  local dev="$1"   # e.g. sda
  local sectors
  sectors=$(cat "/sys/class/block/${dev}/size" 2>/dev/null) || return 1
  echo $((sectors * 512))
}

iface_link() { cat "/sys/class/net/$1/operstate" 2>/dev/null; }
iface_mtu()  { cat "/sys/class/net/$1/mtu" 2>/dev/null; }

iface_rx_bytes() { cat "/sys/class/net/$1/statistics/rx_bytes" 2>/dev/null; }
iface_tx_bytes() { cat "/sys/class/net/$1/statistics/tx_bytes" 2>/dev/null; }

# Compute throughput between two snapshots.
iface_throughput_bps() {
  local iface="$1" interval="${2:-1}"
  local r1 r2
  r1=$(iface_rx_bytes "$iface")
  sleep "$interval"
  r2=$(iface_rx_bytes "$iface")
  echo $(( (r2 - r1) * 8 / interval ))
}

Real-World Recipes

Recipe 1: Find what’s holding `/var/log/myapp.log` open

. lib/proc.sh
proc_holders_of /var/log/myapp.log
# 12345
# 12346
ps -fp 12345 12346
# Output: which processes still have the deleted log open

This is the “why isn’t my disk space freed after rm?” debugging tool. Restart the listed processes or close their fds and the kernel reclaims the inode.

Recipe 2: Tune for a database host

# A reasonable baseline for a Postgres host.
sysctl_persist vm.swappiness 1 "DB host: avoid swapping"
sysctl_persist vm.dirty_background_ratio 5 "smaller writeback bursts"
sysctl_persist vm.dirty_ratio 10 "smaller writeback bursts"
sysctl_persist vm.overcommit_memory 2 "strict accounting; refuse oversubscription"
sysctl_persist vm.overcommit_ratio 80 "with 20% reserved for kernel"
sysctl_persist net.core.somaxconn 4096 "DB connection pool backlog"
sysctl_persist fs.file-max 2097152 "many DB connections + WAL files"
sysctl_persist kernel.shmmax 17179869184 "for big shared_buffers"

# Now persist and verify in one pass.
sysctl --system   # reload all /etc/sysctl.d/*.conf

Recipe 3: Audit drift between expected and actual sysctl

# CI check: read a manifest of expected sysctl values and compare.
audit_sysctl_manifest() {
  local manifest="$1" key value actual fail=0
  while IFS='=' read -r key value; do
    [[ -z "${key// }" || "${key:0:1}" == "#" ]] && continue
    key="${key// /}"; value="${value# }"
    actual=$(sysctl -n "$key" 2>/dev/null)
    if [[ "$actual" != "$value" ]]; then
      printf 'DRIFT %s: expected %s got %s\n' "$key" "$value" "$actual"
      fail=1
    fi
  done < "$manifest"
  return "$fail"
}

# manifest format:
# vm.swappiness = 1
# net.ipv4.ip_forward = 1
audit_sysctl_manifest /etc/myapp/sysctl-baseline.conf || exit 1

Recipe 4: Detect container vs host

# Container detection from /proc.
detect_container() {
  if [[ -f /.dockerenv ]]; then echo docker; return; fi
  if grep -qa 'kubepods\|docker' /proc/1/cgroup 2>/dev/null; then echo container; return; fi
  if [[ "$(awk -F/ '$2=="systemd" {print $NF}' /proc/1/cgroup 2>/dev/null)" != "$(hostname)" ]]; then
    # cgroup path differs from hostname-named systemd scope: probably container
    :
  fi
  # Compare PID 1's mount namespace to the host's (won't work inside container).
  # Better: check PID 1's parent. Host has none; container's PID 1 is /sbin/init or app.
  if [[ "$(proc_name 1)" =~ ^(systemd|init)$ ]]; then echo host; else echo container; fi
}

/proc/1/cgroup contains the cgroup path; in containers it usually mentions docker, kubepods, or lxc. This is far more reliable than [[ -f /.dockerenv ]] (which Docker can hide).

Recipe 5: Read scheduler stats for a tight-loop process

# /proc/$pid/sched has cumulative scheduler stats.
sched_summary() {
  local pid="$1"
  awk '
    /se.sum_exec_runtime/   { runtime  = $3 }
    /se.statistics.wait_sum/ { wait     = $3 }
    /nr_voluntary_switches/  { vol      = $3 }
    /nr_involuntary_switches/{ invol    = $3 }
    END {
      printf "runtime_ms=%.1f wait_ms=%.1f vol_switches=%s invol_switches=%s\n",
        runtime, wait, vol, invol
    }' "/proc/$pid/sched"
}

invol_switches rising fast = the process is being preempted by other CPU-hungry processes. wait_ms rising = the process is waiting in the run queue. Useful diagnostic when “the app is slow but CPU isn’t pegged.”

Going deeper

How `/proc` is actually implemented (and why the rules are weird)

/proc is a synthetic filesystem (procfs) with no backing storage. When you read() a file, a kernel function runs right then and generates the bytes; when you close it, they’re gone. Most /proc files are built on the kernel’s seq_file interface, which produces output record-by-record. Three surprising behaviours all fall out of this:

stat reports size 0. There’s nothing on disk to measure; the content only exists during a read. wc -c < /proc/meminfo gives a real number because it actually reads to EOF; stat -c %s /proc/meminfo gives 0.
A read is a snapshot in motion, not atomic. A large file like /proc/$pid/maps is generated in chunks as you read; if the process mutates its address space mid-read, you can get a slightly inconsistent picture. For a truly consistent view you must freeze the process (e.g. SIGSTOP) first.
Entries appear and vanish with the objects they describe. /proc/$pid/ exists exactly as long as the process does. Every access is a race against the process’s lifecycle — hence the defensive [[ -d /proc/$pid ]] || guards throughout this lesson.

`/proc` and `/sys` inside containers — why `free` lies

A container is just namespaced processes on the host kernel, and /proc is not fully namespaced. /proc/$pid/ is filtered to the PID namespace (a container sees only its own processes), but the system-wide files — /proc/meminfo, /proc/loadavg, /proc/stat, /proc/cpuinfo — show the host’s numbers, because there is only one kernel. This is why free and top inside a container report the host’s 256 GB of RAM even though the cgroup limits the container to 512 MB. Tools like lxcfs exist precisely to bind-mount cgroup-aware versions of those files over the real ones so container tools see their limits.

For /proc/sys, writability is gated by namespaces: net.* tunables are per–network-namespace (a container with its own netns can set some), but vm.*, fs.*, and kernel.* are host-global and read-only from inside an unprivileged container. The reliable “am I containerised?” signal is /proc/1/cgroup — a line mentioning docker, kubepods, or lxc — or, on cgroup v2, a single 0::/… line whose path is not the host’s. /proc/1/cgroup beats [[ -f /.dockerenv ]], which is trivial to hide.

Security: `/proc` is a goldmine, so it’s increasingly locked down

/proc/$pid/ can leak secrets: environ often holds credentials passed as env vars, cmdline holds tokens passed as flags, and maps reveals ASLR layout useful to an exploit. Modern kernels ship several hardening switches you will meet:

hidepid= (a mount option on /proc): hidepid=1 hides other users’ /proc/$pid/ contents; hidepid=2 (or hidepid=invisible on newer kernels) hides the directories entirely, so you can’t even see that another user’s PIDs exist. A gid= option lets a monitoring group keep visibility.
kernel.yama.ptrace_scope: 0 = classic (any process can ptrace another with the same UID); 1 = only a parent may; 2 = admin only; 3 = ptrace disabled. At 2/3, even root may need CAP_SYS_PTRACE to read /proc/$pid/environ or /proc/$pid/mem.
kernel.kptr_restrict and kernel.dmesg_restrict: hide kernel pointers (from /proc/kallsyms etc.) and restrict dmesg to root — both worth setting to harden a host.

Scripts that read environ/maps should therefore degrade gracefully (2>/dev/null, a clear error) rather than assuming access.

Performance: some `/proc` files are expensive

Reading is cheap — except when it isn’t. /proc/$pid/smaps walks the process’s entire page table to produce per-mapping accounting; on a multi-gigabyte JVM or database it can take hundreds of milliseconds and briefly stall the target process. Prefer /proc/$pid/statm or the VmRSS line of status for cheap memory snapshots, and reach for smaps/smaps_rollup only when you truly need the PSS breakdown. Likewise, scanning /proc/[0-9]* on a box with tens of thousands of processes is a lot of readdir/open; do it once and cache, don’t poll it in a tight loop.

sysctl load order, precisely

The simplified list earlier is close; the exact rule that systemd-sysctl.service follows is worth knowing when two files fight. Files from /etc/sysctl.d/, /run/sysctl.d/, and /usr/lib/sysctl.d/ are collected and sorted by basename across all three directories. If the same basename exists in more than one, the one in the earliest directory in that list wins (/etc overrides /run overrides /usr) — the others are ignored. The surviving files are then applied in basename lexicographic order, and for a given key the last file to set it wins. /etc/sysctl.conf is applied at the end for backward compatibility. Net effect: name your override 99-myapp.conf in /etc/sysctl.d/ and it beats a distro default of the same or lower number. Apply the whole set on demand with sysctl --system; apply one file with sysctl -p FILE.

One booby-trap: a value can also be set on the kernel command line (/proc/cmdline) or by a subsystem at module-load time, and a sysctl.d file applied later will override it — or a module loaded after boot can reset a value your sysctl.d file set earlier. When a tunable mysteriously “won’t stick”, check /proc/cmdline, module load order, and whether something is re-applying it at runtime.

Portability: there is no `/proc` on macOS or the BSDs

This entire lesson is Linux. macOS and the BSDs have no /proc and no /sys — the “kernel as a filesystem” model is a Linux invention. They do have sysctl, but it is a different animal:

The namespace is a flat MIB (management information base), not a directory tree: sysctl -a, sysctl hw.memsize, sysctl -n hw.ncpu, sysctl vm.swapusage, sysctl kern.maxfiles. The names differ from Linux’s (kern.ipc.somaxconn, not net.core.somaxconn).
There is no /proc/sys, no /etc/sysctl.d, and no sysctl --system. On FreeBSD, persistence is the single file /etc/sysctl.conf (read by rc). On macOS, /etc/sysctl.conf is read only early and unreliably; the supported way to persist a sysctl -w is a launchd job, and some knobs live in nvram boot-args instead.
To introspect a process on macOS you use ps, lsof, vmmap, dtruss/dtrace — not a per-PID filesystem.

So: write and test your /proc-based tooling on Linux, and gate it behind an OS check ([[ -d /proc ]] || { echo "Linux-only" >&2; exit 1; }) so it fails loudly rather than silently misbehaving on a developer’s Mac.

Footgun List

/proc/$pid is racy. A process can exit between your [[ -d /proc/$pid ]] and your cat /proc/$pid/status. Always handle “file vanished” gracefully.
/proc/$pid/cmdline uses NUL separators, not spaces. cat shows them squished together. Use tr '\0' ' ' for human display, or xargs -0 to parse.
/proc/sys/kernel/perf_event_paranoid defaults restrict perf for non-root. If your script invokes perf, expect to need root or a tuned perf_event_paranoid.
sysctl --system reloads ALL /etc/sysctl.d files. If a stale file declares something destructive, --system will apply it. Audit periodically.
/etc/sysctl.conf is loaded by some distros and ignored by others. Use /etc/sysctl.d/*.conf only for portability.
Some sysctl values are clamped silently. sysctl -w net.core.rmem_max=999999999 may set a smaller value than requested. Always re-read after writing.
/sys writes can require specific timing. Writing to /sys/.../scheduler while the device is busy may fail with EBUSY. Stop I/O first if possible.
/proc/$pid/smaps is expensive to read. It walks the process’s entire VM. On large processes (multi-GB heaps), a single cat smaps can take seconds and cause scheduling glitches. Prefer statm for cheap memory snapshots.
readlink /proc/$pid/exe may say (deleted) if the process’s binary was upgraded after the process started. The pattern /usr/bin/myapp (deleted) means “restart this process to pick up the new binary.”
Per-PID files are subject to ptrace_scope hardening. With kernel.yama.ptrace_scope=2 or 3, even root may need CAP_SYS_PTRACE to read /proc/$pid/environ or /proc/$pid/maps. Surface a clear error in scripts that depend on these.
Inside containers, /proc/sys/... is largely read-only or namespaced. Don’t assume sysctl writes from inside a container will persist; many net.* and vm.* are host-only.
/sys/class/net/eth0/statistics/rx_bytes is a 64-bit counter that may wrap. On 1 Gbps interfaces it’s effectively unwrapping for years, but on 100 Gbps interfaces it can wrap in <1 day. Use deltas and handle wrap-around if your tooling runs long.

Common beginner mistakes

These are conceptual traps — wrong mental models — distinct from the code-level pitfalls in the Footgun List above.

“/proc files are real files on disk, so I can trust their size and treat reads as instant.” They’re synthesised by the kernel on every read; stat shows size 0, a read runs kernel code, and a big file like maps can change under you mid-read. The right model: a /proc file is a live query, not a stored document. Guard for it vanishing, and freeze the process if you need a consistent snapshot.

“sysctl -w saved my setting.” No — -w is runtime-only. It changes the running kernel and is gone on the next reboot. The right model: writing and persisting are two separate acts. sysctl -w (or echo >) changes now; a file in /etc/sysctl.d/ changes from the next boot onward. Production changes almost always need both.

“I’ll trust MemFree to see how much memory is free.” MemFree is only the RAM sitting totally idle; Linux deliberately fills spare RAM with reclaimable page cache, so a healthy busy server shows tiny MemFree and is perfectly fine. The right model: read MemAvailable — the kernel’s own estimate of what a new workload can use without swapping.

“A single read of /proc/stat tells me CPU usage.” It tells you cumulative CPU time since boot, not utilisation. Utilisation is a rate — you must diff two snapshots over an interval. The same applies to every counter in /sys/class/net/*/statistics/ and /proc/diskstats: they’re monotonic totals, and the useful number is the delta.

“The numbers in /proc/meminfo are my container’s.” Inside a container the system-wide /proc files show the host’s totals, because there’s one shared kernel. The right model: for a container’s real limits, read its cgroup (/sys/fs/cgroup/...) or rely on lxcfs — not /proc/meminfo, which is why free “sees” 256 GB in a 512 MB container.

“Reading and writing under /proc/sys are equally harmless — it’s just cat and echo.” Reading is harmless; a write is a live, system-wide kernel change affecting every process instantly. The right model: treat sysctl -w with the same care as any production config change — snapshot the old value, change one thing, re-read to confirm (the kernel may clamp it), and know your rollback.

“/etc/sysctl.conf is the place to put kernel settings.” It’s the legacy single file; it still works but doesn’t compose. The right model: drop a purpose-named file in /etc/sysctl.d/ (e.g. 99-database-tuning.conf) so multiple tools and roles each own a file you can enable, disable, or diff independently.

“If sysctl -w returned success, my value is set.” The kernel can silently clamp an out-of-range value to a legal one and still return 0. The right model: a write is only half the operation — always re-read with sysctl -n and compare to what you asked for.

Practice challenges

Work these in order on a Linux host (a VM or container is perfect) — they climb from “just cat the right file” to “tune the kernel safely and detect drift”. Try each before opening the solution. Output shown is representative of a Linux host.

Challenge 1 — X-ray your own shell (beginner)

Print your current shell’s name, run state, and resident memory (RSS) by reading only /proc/self/status. Don’t use ps.