Shell Idempotency Patterns: State Files, Reconciliation Loops, Dry-Run Flags & Idempotent Primitives

In a nutshell

Think of a thermostat. You never tell a thermostat “turn the heater on for twelve minutes.” You declare a target — “keep this room at 21°C” — and it does the rest: it reads the actual temperature, compares it to the target, and heats only if there’s a gap. When the room is already warm it does nothing. That is why you can leave it running forever and check it every minute: acting on the room a thousand times a day is completely safe, because each action is “close the gap, or do nothing.”

Idempotency is that same idea for scripts. A non-idempotent script is a to-do list you re-execute literally — “add milk to the cart” run three times puts three cartons of milk in the cart. An idempotent script instead declares the destination — “the cart should contain one carton of milk” — reads what’s actually there, and adds milk only if it’s missing. Run it once or run it every five minutes from cron; the end state is identical and nothing breaks. Formally: running the script N times produces the same result as running it once.

That single property is the line between a script you run once, nervously, by hand and a script you trust as a control plane — something cron, CI, a deploy pipeline, or a panicking on-call engineer can re-run without fear. The whole lesson is about how to build that trust into shell: read the real state on every run, act only on the difference, preview changes with --dry-run before they happen, and reach for the primitives (mkdir -p, ln -sfn, install -m, sentinel-guarded appends, atomic writes) that are safe to repeat by construction.

The diagram is the spine of everything below: declare desired → read actual → compute the delta → apply only the delta (or nothing), with --dry-run gating every write and a marker written only after success. Master that loop and you can make almost any ops script safe to re-run.

Level: Intermediate → Advanced · Time: ~40 min

Prerequisites & what you’ll be able to do

You’ll get the most from this lesson if you’re already comfortable with a few earlier topics:

• Functions, locals, and return codes — every reconciler here is a function that reads state and returns. See Functions: Local Scope, Return & Arguments.
• Strict mode — set -Eeuo pipefail interacts with idempotency in subtle ways (a guarded read must not trip errexit). See Defensive Scripting: set -Eeuo pipefail & ShellCheck.
• Atomic writes & filesystem semantics — the temp-file-plus-rename pattern rests on how rename(2) works. See Filesystem Semantics: Hard Links, Symlinks & fsync.
• Idempotent cleanup on signals — traps that run twice must not double-clean. See Signal Handling: trap, Cleanup & Idempotency.

After this lesson you’ll be able to:

1. Tell strict from convergent idempotency apart and pick which one a given script actually needs.
2. Refactor a fragile check-then-act function into a read → compare → apply-the-delta reconciler that no-ops when nothing changed.
3. Recognise a TOCTOU race (the gap between checking and acting) and shrink it with an atomic primitive or a lock.
4. Add a real --dry-run mode that previews the exact diff without touching the system — greppable, auditable, CI-ready.
5. Reach for the idempotent primitive (mkdir -p, ln -sfn, install, sentinel append, atomic write) instead of its footgun twin, on reflex.
6. Store state markers so a crash mid-run retries cleanly instead of lying that the work is done.

Why Idempotency Matters: The Five Failure Modes of Non-Idempotent Scripts

You wrote a setup script. It worked. You run it again to apply a small config tweak — and it fails because the user already exists, the cron entry got duplicated, the symlink threw File exists, or the database migration ran twice and corrupted a counter. You add 2>/dev/null || true to suppress the errors, ship it, and now the script “succeeds” whether or not it actually did anything.

This is the most common operational tax in shell automation. Idempotency is the property that running a script N times produces the same end state as running it once. It is not a coding style; it is a contract with the runtime: “I describe what should be true. The script makes it true if it isn’t, and does nothing if it already is.”

The five failure modes you eliminate by building idempotency in from the start:

The fix is structural, not cosmetic. Idempotency is built from four primitives that this lesson covers in depth:

1. State-file markers — write a sentinel after every “did the work” branch so reruns see “already done.”
2. Desired-vs-actual reconciliation — express what should be true, query what is true, apply only the delta.
3. --dry-run discipline — every change-making function must support a no-op mode that prints what it would do.
4. Idempotent primitives — use ln -sfn, install -m, mkdir -p, sed -i with sentinels, and atomic file writes instead of their non-idempotent equivalents.

By the end, you’ll have a lib/state.sh you can source into any script to make it convergent.

Ground Rules: What “Idempotent” Actually Means in Shell

Before we write code, let’s pin down the definitions, because “idempotent” gets used loosely.

Strict idempotency

f(state) = f(f(state)) = f(f(f(state)))

The script is a function from system state to system state. Applying it twice equals applying it once. The script reads the world, computes the delta, and applies the minimum set of changes.

Convergent idempotency (what we usually build)

f(state) → desired_state, regardless of starting state

Slightly weaker but more practical: the script doesn’t care what state it started from; it reaches desired_state whether the system was empty, partially configured, or fully configured. Configuration management tools (Ansible, Puppet, Chef) are convergent.

What idempotency is NOT

• It is not “the script doesn’t crash on rerun.” Suppressing errors with || true is the opposite of idempotent — it hides the fact that you no longer know what happened.
• It is not “the script is safe.” A rm -rf /var/cache/myapp is technically idempotent (the directory stays gone) but destroying running state on every rerun is not safe.
• It is not “Ansible’s job.” If your shell script wraps terraform apply, calls a REST API, or runs a database migration, it must be idempotent at its layer, regardless of what the upstream tool does.

The mental model: every change-making function is a state transition

Every function that changes the system answers four questions:

1. What is the desired state? (The argument: ensure_user "deploy" "deploy" "/home/deploy".)
2. What is the current state? (Read it: id deploy 2>/dev/null.)
3. Is current == desired? (Compare; if yes, return without changing anything.)
4. If not, apply the minimal change to transition. (Run useradd or usermod.)

This four-step pattern is the heart of every idempotent function in this lesson.

The Four Pillars: State Markers, Reconciliation, Dry-Run, Primitives

Pillar 1 — State-file markers (the “I already did this” sentinel)

The simplest idempotency primitive: write a marker file when you complete a one-shot, expensive, or non-queryable action. On rerun, check for the marker and skip if present.

#!/usr/bin/env bash
set -Eeuo pipefail

STATE_DIR="/var/lib/myapp/state"
mkdir -p "$STATE_DIR"

# One-shot: download a 2 GB model file. We can check the file's existence,
# but we also want to record WHEN and WHICH version was installed, so a
# marker with metadata is more useful than just `[[ -f /path/to/file ]]`.

ensure_model_v2() {
  local marker="$STATE_DIR/model-v2.installed"
  if [[ -f "$marker" ]]; then
    log "model-v2 already installed at $(cat "$marker")"
    return 0
  fi
  log "downloading model-v2..."
  curl -fsSL https://example.com/model-v2.bin -o /opt/myapp/model.bin
  echo "$(date -u +%FT%TZ) sha256=$(sha256sum /opt/myapp/model.bin | awk '{print $1}')" >"$marker"
  log "model-v2 installed"
}

Why this works:

• The marker is idempotent across crashes: if curl fails, no marker is written, so the next run retries from scratch.
• The marker is auditable: it records when and what version, so later you can ask “when did we install v2?” without git-blame archaeology.
• The marker is cheap: file existence check is a single stat syscall; it’s faster than re-validating a 2 GB file’s checksum on every run.

State-file rules:

1. Write the marker last. Never write the marker before the work succeeds; otherwise a crash mid-work leaves a marker that lies about state.
2. Include version/identity in the marker name. model-v2.installed, not model.installed — the v2 marker won’t satisfy a future “ensure v3” check.
3. Keep markers under one well-known directory. /var/lib/myapp/state for system services, ~/.local/state/myapp for user scripts, $STATE_DIR for tests. Easy to clear, easy to inspect, easy to back up.
4. Don’t put markers in /tmp. /tmp gets cleared on reboot on many distros (systemd-tmpfiles); your “already installed” claim evaporates.

Pillar 2 — Desired-vs-actual reconciliation (drift detection)

State markers work for one-shot operations. For ongoing config (a user, a file, a sysctl value, a cron entry), markers are wrong because the config could drift after the marker was written. Someone could delete the user, edit the file, override the sysctl. The script needs to read the current state on every run and reconcile.

# The reconciliation pattern, applied to a user account.

ensure_user() {
  local user="$1" home="$2" shell="${3:-/bin/bash}"
  local current_home current_shell

  if id -u "$user" &>/dev/null; then
    # Read actual state.
    current_home=$(getent passwd "$user" | cut -d: -f6)
    current_shell=$(getent passwd "$user" | cut -d: -f7)

    # Compare. If everything matches, no-op.
    if [[ "$current_home" == "$home" && "$current_shell" == "$shell" ]]; then
      log "user $user already correct"
      return 0
    fi

    # Apply the delta only.
    log "user $user exists but has drifted; applying $home, $shell"
    [[ "$current_home" != "$home" ]] && usermod --home "$home" --move-home "$user"
    [[ "$current_shell" != "$shell" ]] && usermod --shell "$shell" "$user"
  else
    log "creating user $user"
    useradd --home-dir "$home" --shell "$shell" --create-home "$user"
  fi
}

The four-step reconciliation pattern, made explicit:

┌─────────────────────────────────────────────┐
│ 1. Desired state from arguments             │  ensure_user "deploy" "/home/deploy" "/bin/bash"
└─────────────┬───────────────────────────────┘
              ▼
┌─────────────────────────────────────────────┐
│ 2. Read actual state from the system        │  getent passwd deploy
└─────────────┬───────────────────────────────┘
              ▼
┌─────────────────────────────────────────────┐
│ 3. Compute delta (desired - actual)         │  home differs? shell differs? exists?
└─────────────┬───────────────────────────────┘
              ▼
┌──────────────┴──────────────────────────────┐
▼ delta empty                  ▼ delta non-empty
"already correct"             apply minimal change
return 0                      (useradd or usermod)

This pattern scales. The same skeleton works for files (compare_file — checksum the desired vs actual, atomic-write if different), services (ensure_service_running — systemctl is-active, then systemctl start if needed), packages (ensure_pkg_installed — dpkg-query then apt-get install), and remote API state (ensure_dns_record — query the API, then PATCH only the changed fields).

Check-then-act, and the one race every reconciler has (TOCTOU). Look closely at the shape of ensure_user: it checks (id -u "$user") and then acts (useradd). Between those two steps there is a window in which the world can change out from under you — another admin, another cron tick of the same script, or a parallel Ansible run could create that user first. When your action then runs, it fails with “user already exists.” This gap is called TOCTOU — time-of-check to time-of-use — and every check-then-act reconciler has one. For most single-writer ops scripts the window is harmless (the worst case is a benign “already exists” error you can absorb). But when two copies of the reconciler can run at once, you need to close it. The cheapest portable fix is an atomic guard — an operation the kernel guarantees only one caller can win:

# `mkdir` is atomic: exactly one concurrent caller creates the dir; the rest fail.
# That makes it a portable mutex — no race between "check" and "act".
LOCKDIR=/var/lib/myapp/state/reconcile.lock
if mkdir "$LOCKDIR" 2>/dev/null; then
  trap 'rmdir "$LOCKDIR"' EXIT          # release on any exit
  reconcile_everything                  # only one runner is inside here
else
  log "another reconcile is running; skipping this tick"
  exit 0
fi

We go deep on locking, flock, and noclobber in Going deeper — for now, just internalise the shape: check and act are not one step, and the gap between them is where concurrency bugs live.

Pillar 3 — Dry-run discipline (the no-op mode)

Every change-making function must support --dry-run. Two reasons:

1. Operator safety. Before applying a script in production, the operator wants to see the diff: “what would this change?” A dry-run that prints “would create user deploy, would write /etc/cron.d/myjob, would not change /etc/hosts” is the audit log before the fact.
2. CI integration. PR checks should run the script against a snapshot of prod state with --dry-run and fail if the diff is unexpected.

The pattern: a global DRY_RUN=0 flag, and a wrapper do_or_say that gates every mutating call.

DRY_RUN=0

# Parse args; $1 is "--dry-run" if present.
[[ "${1:-}" == "--dry-run" ]] && { DRY_RUN=1; shift; }

# Wrapper: in dry-run, print the command instead of running it.
do_or_say() {
  if (( DRY_RUN )); then
    printf 'WOULD: %s\n' "$*" >&2
  else
    "$@"
  fi
}

ensure_user_dry() {
  local user="$1" home="$2"
  if id -u "$user" &>/dev/null; then
    log "user $user exists"
  else
    do_or_say useradd --home-dir "$home" --create-home "$user"
  fi
}

Dry-run rules:

• Reads stay live. id -u, getent, systemctl is-active, curl GET — all run normally in dry-run. The point is to see what changes would happen given the current real state.
• Writes are gated. Every useradd, usermod, cp, chmod, systemctl start, curl POST — all wrapped in do_or_say.
• Output is unambiguous. Print WOULD: useradd ... not # useradd ... (looks like a comment); not useradd ... # skipped (parsers will run it). The WOULD: prefix is greppable: myscript --dry-run | grep ^WOULD.
• Dry-run must reach the same code paths as live. If --dry-run skips entire sections of the script, you’re testing nothing. The do_or_say wrapper means the control flow is identical; only the side-effecting commands are gated.

Pillar 4 — Idempotent primitives (use the right tool)

Many shell tools have idempotent and non-idempotent variants. Use the idempotent one by default, even when you “know” the script will only run once.

The append-only-if-missing pattern (sentinel comments):

# Add a line to a file only if it's not already there. The sentinel comment
# uniquely identifies our line so we can find it on rerun.

ensure_line_in_file() {
  local file="$1" line="$2" sentinel="$3"
  # sentinel example: "# managed-by:myapp:cron-job"
  if grep -qF "$sentinel" "$file" 2>/dev/null; then
    # Line is already there. Update it in place if content differs.
    local current
    current=$(grep -F "$sentinel" "$file" | head -n1)
    if [[ "$current" != "$line" ]]; then
      # Use a delimiter unlikely to appear in $line; '|' is common.
      sed -i.bak "/$(printf '%s' "$sentinel" | sed 's:[][\\/.^$*]:\\&:g')/c\\
$line" "$file"
    fi
  else
    printf '%s\n' "$line" >>"$file"
  fi
}

# Usage:
ensure_line_in_file /etc/cron.d/myapp \
  "*/5 * * * * deploy /opt/myapp/bin/heartbeat # managed-by:myapp:heartbeat" \
  "managed-by:myapp:heartbeat"

Atomic file writes (the mv-into-place pattern):

# Idempotent file generation. Compare desired content to current, write only on diff.

ensure_file_content() {
  local path="$1" mode="$2" owner="$3" group="$4"
  local desired_content
  desired_content=$(cat)  # read from stdin

  # Compare.
  if [[ -f "$path" ]] && diff -q <(printf '%s' "$desired_content") "$path" >/dev/null; then
    log "$path: content unchanged"
    return 0
  fi

  # Write atomically: temp file in same directory, then rename.
  local tmp
  tmp=$(mktemp "${path}.XXXXXX")
  printf '%s' "$desired_content" >"$tmp"
  install -m "$mode" -o "$owner" -g "$group" "$tmp" "$path"
  rm -f "$tmp"  # install copies; remove the temp
  log "$path: updated"
}

# Usage:
ensure_file_content /etc/myapp/server.conf 0644 root root <<'EOF'
listen_port = 8080
log_level = info
EOF

Why atomic writes: if the script crashes mid-write with >"$path", the file is truncated and the service that reads it sees garbage. Writing to ${path}.XXXXXX and renaming is atomic on POSIX filesystems — readers see either the old file or the new one, never a partial.

A Drop-In Library: lib/state.sh

Compose the patterns above into a reusable library. Source it from any idempotent script.

# lib/state.sh — drop-in idempotency helpers.
# Source from any script: . /opt/myapp/lib/state.sh

# ─── Configuration (override before sourcing if needed) ────────────────────
: "${STATE_DIR:=/var/lib/$(basename "${0##*/}" .sh)/state}"
: "${DRY_RUN:=0}"
: "${LOG_PREFIX:=$(basename "${0##*/}" .sh)}"

# ─── Logging (timestamped, structured-ish) ─────────────────────────────────
log()  { printf '[%s] [%s] [INFO]  %s\n'  "$(date -u +%FT%TZ)" "$LOG_PREFIX" "$*" >&2; }
warn() { printf '[%s] [%s] [WARN]  %s\n'  "$(date -u +%FT%TZ)" "$LOG_PREFIX" "$*" >&2; }
err()  { printf '[%s] [%s] [ERROR] %s\n'  "$(date -u +%FT%TZ)" "$LOG_PREFIX" "$*" >&2; }
chg()  { printf '[%s] [%s] [CHANGE] %s\n' "$(date -u +%FT%TZ)" "$LOG_PREFIX" "$*" >&2; }

# ─── Dry-run wrapper ───────────────────────────────────────────────────────
# Run the command, or print "WOULD: ..." in dry-run mode.
do_or_say() {
  if (( DRY_RUN )); then
    printf 'WOULD: %s\n' "$*" >&2
    return 0
  fi
  "$@"
}

# ─── Init ──────────────────────────────────────────────────────────────────
state_init() {
  if (( ! DRY_RUN )); then
    mkdir -p "$STATE_DIR"
    chmod 0700 "$STATE_DIR"
  fi
}

# ─── State markers ─────────────────────────────────────────────────────────
state_marker_path() { printf '%s/%s.marker' "$STATE_DIR" "$1"; }

state_marker_exists() { [[ -f "$(state_marker_path "$1")" ]]; }

state_marker_set() {
  local name="$1" content="${2:-$(date -u +%FT%TZ)}"
  do_or_say bash -c "umask 077; printf '%s\n' '$content' >'$(state_marker_path "$name")'"
}

state_marker_clear() {
  do_or_say rm -f "$(state_marker_path "$1")"
}

# Run $@ once-and-only-once across reruns; mark with $name.
state_once() {
  local name="$1"; shift
  if state_marker_exists "$name"; then
    log "skip ($name): already done at $(cat "$(state_marker_path "$name")" 2>/dev/null || echo unknown)"
    return 0
  fi
  log "running ($name): $*"
  if "$@"; then
    state_marker_set "$name"
    chg "completed: $name"
  else
    err "failed: $name (no marker written; will retry next run)"
    return 1
  fi
}

# ─── Idempotent primitives ─────────────────────────────────────────────────

# Ensure a directory exists with mode/owner/group.
ensure_dir() {
  local path="$1" mode="${2:-0755}" owner="${3:-root}" group="${4:-root}"
  if [[ ! -d "$path" ]]; then
    do_or_say install -d -m "$mode" -o "$owner" -g "$group" "$path"
    chg "created dir $path ($mode $owner:$group)"
  else
    # Reconcile mode/owner/group.
    local cur_mode cur_owner cur_group
    cur_mode=$(stat -c %a "$path" 2>/dev/null || stat -f %A "$path")
    cur_owner=$(stat -c %U "$path" 2>/dev/null || stat -f %Su "$path")
    cur_group=$(stat -c %G "$path" 2>/dev/null || stat -f %Sg "$path")
    [[ "$cur_mode"  != "${mode#0}" ]] && do_or_say chmod  "$mode"          "$path" && chg "chmod $path $mode"
    [[ "$cur_owner" != "$owner"   ]] && do_or_say chown  "$owner"         "$path" && chg "chown $path $owner"
    [[ "$cur_group" != "$group"   ]] && do_or_say chgrp  "$group"         "$path" && chg "chgrp $path $group"
  fi
}

# Ensure a symlink target -> link, idempotent and no-prompt.
ensure_symlink() {
  local target="$1" link="$2"
  if [[ -L "$link" ]]; then
    local current
    current=$(readlink "$link")
    [[ "$current" == "$target" ]] && return 0
    chg "symlink $link drifted ($current -> $target)"
  elif [[ -e "$link" ]]; then
    err "$link exists and is not a symlink; refusing to clobber"
    return 1
  fi
  do_or_say ln -sfn "$target" "$link"
  chg "symlink $link -> $target"
}

# Ensure file content matches stdin; preserve mode/owner/group.
ensure_file() {
  local path="$1" mode="${2:-0644}" owner="${3:-root}" group="${4:-root}"
  local desired tmp
  desired=$(cat)
  if [[ -f "$path" ]] && diff -q <(printf '%s' "$desired") "$path" >/dev/null 2>&1; then
    # Content matches; reconcile mode/owner only.
    local cur_mode cur_owner cur_group
    cur_mode=$(stat -c %a "$path" 2>/dev/null || stat -f %A "$path")
    cur_owner=$(stat -c %U "$path" 2>/dev/null || stat -f %Su "$path")
    cur_group=$(stat -c %G "$path" 2>/dev/null || stat -f %Sg "$path")
    [[ "$cur_mode"  != "${mode#0}" ]] && do_or_say chmod "$mode"  "$path" && chg "chmod $path $mode"
    [[ "$cur_owner" != "$owner"   ]] && do_or_say chown "$owner" "$path" && chg "chown $path $owner"
    [[ "$cur_group" != "$group"   ]] && do_or_say chgrp "$group" "$path" && chg "chgrp $path $group"
    return 0
  fi
  if (( DRY_RUN )); then
    printf 'WOULD: write %s (mode %s owner %s:%s, %d bytes)\n' \
      "$path" "$mode" "$owner" "$group" "${#desired}" >&2
    return 0
  fi
  # Atomic write.
  tmp=$(mktemp "${path}.XXXXXX")
  trap 'rm -f "$tmp"' EXIT
  printf '%s' "$desired" >"$tmp"
  install -m "$mode" -o "$owner" -g "$group" "$tmp" "$path"
  rm -f "$tmp"
  trap - EXIT
  chg "wrote $path ($mode $owner:$group, ${#desired} bytes)"
}

# Ensure a line is in a file, identified by a sentinel comment.
ensure_line() {
  local file="$1" line="$2" sentinel="$3"
  if [[ ! -f "$file" ]]; then
    do_or_say bash -c "printf '%s\n' '$line' >'$file'"
    chg "created $file with $sentinel"
    return 0
  fi
  if grep -qF "$sentinel" "$file"; then
    local current
    current=$(grep -F "$sentinel" "$file" | head -n1)
    if [[ "$current" == "$line" ]]; then
      return 0  # already correct
    fi
    # Replace the line containing the sentinel.
    if (( DRY_RUN )); then
      printf 'WOULD: replace line in %s containing "%s" with "%s"\n' "$file" "$sentinel" "$line" >&2
      return 0
    fi
    local escaped_sentinel
    escaped_sentinel=$(printf '%s' "$sentinel" | sed 's:[][\\/.^$*]:\\&:g')
    sed -i.bak "/$escaped_sentinel/c\\
$line" "$file"
    rm -f "${file}.bak"
    chg "updated line in $file ($sentinel)"
  else
    do_or_say bash -c "printf '%s\n' '$line' >>'$file'"
    chg "appended line to $file ($sentinel)"
  fi
}

# Ensure a user exists with home and shell.
ensure_user() {
  local user="$1" home="$2" shell="${3:-/bin/bash}"
  if id -u "$user" &>/dev/null; then
    local cur_home cur_shell
    cur_home=$(getent passwd "$user" | cut -d: -f6)
    cur_shell=$(getent passwd "$user" | cut -d: -f7)
    [[ "$cur_home"  != "$home"  ]] && do_or_say usermod --home "$home" --move-home "$user" && chg "user $user home -> $home"
    [[ "$cur_shell" != "$shell" ]] && do_or_say usermod --shell "$shell" "$user" && chg "user $user shell -> $shell"
  else
    do_or_say useradd --home-dir "$home" --shell "$shell" --create-home "$user"
    chg "created user $user"
  fi
}

# Ensure a systemd service is enabled and running.
ensure_service() {
  local svc="$1" state="${2:-running}"  # running | stopped
  local enabled="${3:-enabled}"          # enabled | disabled

  case "$enabled" in
    enabled)
      systemctl is-enabled "$svc" &>/dev/null || \
        { do_or_say systemctl enable "$svc"; chg "enabled $svc"; }
      ;;
    disabled)
      systemctl is-enabled "$svc" &>/dev/null && \
        { do_or_say systemctl disable "$svc"; chg "disabled $svc"; } || true
      ;;
  esac

  case "$state" in
    running)
      systemctl is-active "$svc" &>/dev/null || \
        { do_or_say systemctl start "$svc"; chg "started $svc"; }
      ;;
    stopped)
      systemctl is-active "$svc" &>/dev/null && \
        { do_or_say systemctl stop "$svc"; chg "stopped $svc"; } || true
      ;;
  esac
}

Real-World Recipes

Recipe 1: Idempotent package installation across distros

# Source lib/state.sh first.

ensure_pkg() {
  local pkg="$1"
  if command -v dpkg-query &>/dev/null; then
    if dpkg-query -W -f='${Status}\n' "$pkg" 2>/dev/null | grep -q "ok installed"; then
      return 0
    fi
    do_or_say apt-get install -y "$pkg"
    chg "installed $pkg (apt)"
  elif command -v rpm &>/dev/null; then
    if rpm -q "$pkg" &>/dev/null; then
      return 0
    fi
    do_or_say dnf install -y "$pkg" 2>/dev/null || do_or_say yum install -y "$pkg"
    chg "installed $pkg (rpm)"
  else
    err "no supported package manager"; return 1
  fi
}

ensure_pkg curl
ensure_pkg jq
ensure_pkg "$([[ "$(uname -s)" == Linux ]] && echo iproute2 || echo iproute)"

Recipe 2: Idempotent cron entry with sentinel

ensure_dir /etc/cron.d 0755 root root

ensure_line /etc/cron.d/myapp \
  '*/5 * * * * deploy /opt/myapp/bin/heartbeat >/dev/null 2>&1 # managed-by:myapp:heartbeat' \
  'managed-by:myapp:heartbeat'

# Removing a managed cron entry is also idempotent:
remove_line() {
  local file="$1" sentinel="$2"
  [[ -f "$file" ]] || return 0
  grep -qF "$sentinel" "$file" || return 0
  do_or_say sed -i.bak "/$sentinel/d" "$file"
  rm -f "${file}.bak"
  chg "removed line from $file ($sentinel)"
}

Recipe 3: Idempotent remote API state (DNS record)

# Reconcile a DNS A record via Cloudflare API.
ensure_dns_a_record() {
  local zone_id="$1" name="$2" desired_ip="$3" cf_token="${CLOUDFLARE_API_TOKEN:?}"

  # 1. Read current state.
  local current_json current_id current_ip
  current_json=$(curl -fsS \
    -H "Authorization: Bearer $cf_token" \
    "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records?name=$name&type=A")

  current_id=$(jq -r '.result[0].id // empty' <<<"$current_json")
  current_ip=$(jq -r '.result[0].content // empty' <<<"$current_json")

  # 2. Compare.
  if [[ -n "$current_id" && "$current_ip" == "$desired_ip" ]]; then
    log "$name -> $desired_ip already correct"
    return 0
  fi

  # 3. Apply minimal delta.
  if [[ -z "$current_id" ]]; then
    # Create.
    if (( DRY_RUN )); then
      printf 'WOULD: POST DNS record %s A %s\n' "$name" "$desired_ip" >&2
    else
      curl -fsS -X POST \
        -H "Authorization: Bearer $cf_token" -H "Content-Type: application/json" \
        "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records" \
        --data "$(jq -nc --arg n "$name" --arg ip "$desired_ip" \
          '{type:"A",name:$n,content:$ip,ttl:300}')" >/dev/null
      chg "created DNS $name A $desired_ip"
    fi
  else
    # Update.
    if (( DRY_RUN )); then
      printf 'WOULD: PATCH DNS %s A from %s to %s\n' "$name" "$current_ip" "$desired_ip" >&2
    else
      curl -fsS -X PATCH \
        -H "Authorization: Bearer $cf_token" -H "Content-Type: application/json" \
        "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records/$current_id" \
        --data "$(jq -nc --arg ip "$desired_ip" '{content:$ip}')" >/dev/null
      chg "updated DNS $name A: $current_ip -> $desired_ip"
    fi
  fi
}

ensure_dns_a_record "$ZONE_ID" "api.example.com" "203.0.113.42"

The four-step pattern is identical to local file/user reconciliation: read, compare, compute delta, apply minimum. The only difference is the API instead of a syscall.

Recipe 4: Database migration with idempotent runner

# Run SQL migration files only once each, tracked in a migrations table.
# Uses Postgres; the same shape works for any DB.

PGCONN="${PGCONN:-postgresql://app:secret@localhost/app_prod}"

ensure_migrations_table() {
  state_once db.migrations_table psql "$PGCONN" -v ON_ERROR_STOP=1 -c "
    CREATE TABLE IF NOT EXISTS schema_migrations (
      id          TEXT PRIMARY KEY,
      applied_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
      checksum    TEXT NOT NULL
    );"
}

apply_migration() {
  local file="$1"
  local id checksum
  id=$(basename "$file" .sql)
  checksum=$(sha256sum "$file" | awk '{print $1}')

  # Already applied?
  local applied
  applied=$(psql "$PGCONN" -tAc \
    "SELECT checksum FROM schema_migrations WHERE id='$id';" 2>/dev/null || true)

  if [[ -n "$applied" ]]; then
    if [[ "$applied" != "$checksum" ]]; then
      err "migration $id already applied with different checksum ($applied vs $checksum)"
      return 1
    fi
    log "migration $id already applied"
    return 0
  fi

  # Apply in a transaction; record on success.
  if (( DRY_RUN )); then
    printf 'WOULD: apply migration %s (%s)\n' "$id" "$checksum" >&2
    return 0
  fi

  psql "$PGCONN" -v ON_ERROR_STOP=1 --single-transaction <<EOF
\i $file
INSERT INTO schema_migrations (id, checksum) VALUES ('$id', '$checksum');
EOF
  chg "applied migration $id"
}

ensure_migrations_table
for f in /opt/myapp/migrations/*.sql; do
  apply_migration "$f"
done

Why this is idempotent:

• The schema_migrations table is the durable state (the database itself, not a file).
• Re-running iterates all files; already-applied ones are no-ops.
• Checksum comparison catches a class of bugs: someone edited a migration file after it was applied. The script refuses to re-run silently and surfaces the drift.
• --single-transaction ensures the migration and the marker insert succeed atomically; a crash mid-migration leaves no marker, so the next run retries cleanly.

Visual: The Reconciliation Lifecycle

        ┌────────────────────────────────────────────────────┐
        │              SCRIPT INVOCATION                     │
        │   ./apply.sh    or    ./apply.sh --dry-run         │
        └────────────────────┬───────────────────────────────┘
                             │
                             ▼
              ┌──────────────────────────────┐
              │ Source lib/state.sh          │
              │ Init STATE_DIR, parse flags  │
              └─────────────┬────────────────┘
                            │
                            ▼
        ┌───────────────────┴───────────────────┐
        │  For each ensure_*  (the desired      │
        │  state declarations):                 │
        │                                       │
        │  ┌─────────────────────────────────┐  │
        │  │ 1. Define desired (arguments)   │  │
        │  └─────────────┬───────────────────┘  │
        │                ▼                      │
        │  ┌─────────────────────────────────┐  │
        │  │ 2. Read actual (system query)   │  │
        │  └─────────────┬───────────────────┘  │
        │                ▼                      │
        │  ┌─────────────────────────────────┐  │
        │  │ 3. Compute delta                │  │
        │  └─────────────┬───────────────────┘  │
        │                ▼                      │
        │     ┌──────────┴──────────┐          │
        │     ▼ delta empty         ▼ non-empty│
        │   no-op                 do_or_say    │
        │   (silent)              CHANGE log    │
        │                                       │
        └───────────────────────────────────────┘
                            │
                            ▼
              ┌──────────────────────────────┐
              │ Summary:                     │
              │  N actions taken             │
              │  M no-ops (already correct)  │
              │  K dry-run announcements     │
              └──────────────────────────────┘

Going deeper

You now have the loop. This section is for the reader who has to run that loop concurrently, at scale, against hostile inputs, and across five different userlands — the things that turn a tidy ensure_* function into a 2 a.m. incident.

1. TOCTOU, races, and real mutual exclusion

The check-then-act gap from Pillar 2 becomes a genuine bug the moment two runners overlap: a 5-minute cron reconciler that occasionally takes 6 minutes, a human running the script by hand while cron fires, or a fleader-election flap that starts two copies. Three portable ways to close the window, weakest to strongest:

flock is the nicest when you have it, precisely because the lock releases automatically when the process dies — no stale lock after a kill -9:

# Linux: whole-script mutual exclusion, self-healing on crash.
exec 9>/var/lib/myapp/state/reconcile.lock   # open the lock fd
if ! flock -n 9; then
  log "another reconcile holds the lock; exiting"
  exit 0
fi
# ... critical section; lock auto-released when fd 9 closes at exit ...

The mkdir lock (shown in Pillar 2) is the portable fallback, but it has the opposite problem: a kill -9 leaves the directory behind and every future run thinks it’s locked. If you use it, either put the lockdir on a tmpfs that clears on reboot and write the holder’s PID inside so a new run can check kill -0 "$pid" and break a stale lock — or just prefer flock on Linux. Rule of thumb: single-writer scripts can ignore all of this; anything that can overlap needs one of these three.

2. “Changed” is a first-class signal — report it like the tools do

Ansible prints changed=3 and Terraform prints Plan: 2 to add, 1 to change. That count is not cosmetic — it’s what makes reconcilers composable and alertable. A steady-state box should report zero changes every run; a non-zero count on an otherwise-stable host means drift, and drift is worth an alert. Bake the signal into your library:

CHANGES=0
chg() { CHANGES=$((CHANGES+1)); printf '[CHANGE] %s\n' "$*" >&2; }

# ... run all ensure_* ...

# Exit code carries the signal, Terraform-style:
#   0 = converged, nothing changed   2 = changes applied (or, in dry-run, WOULD change)
if (( CHANGES > 0 )); then
  log "$CHANGES change(s) applied"
  (( DRY_RUN )) && exit 2      # like `terraform plan -detailed-exitcode`
fi

A CI gate can now assert “prod should be converged” with ./apply.sh --dry-run; test $? -eq 0 — a non-zero exit means someone changed the box out of band. That is drift detection for free, built from the same reconcile loop.

3. Make the read cheap — reconcile cost at scale

Every reconcile reads before it acts, and reads cost syscalls, forks, and sometimes network round-trips. A loop that runs dpkg-query once per package for 300 packages forks 300 processes every five minutes. At scale, batch the read:

# SLOW: 300 forks per run
for pkg in "${packages[@]}"; do
  dpkg-query -W "$pkg" &>/dev/null || to_install+=("$pkg")
done

# FAST: one read, then set arithmetic in-shell
declare -A installed=()
while read -r name status; do
  [[ "$status" == install ]] && installed["$name"]=1
done < <(dpkg-query -W -f='${Package} ${Status}\n' 2>/dev/null | awk '{print $1, $2}')
for pkg in "${packages[@]}"; do
  [[ -v installed[$pkg] ]] || to_install+=("$pkg")
done
apt-get install -y "${to_install[@]}"    # one call for all missing

declare -A associative arrays and [[ -v arr[key] ]] are bash 4+; the < <(...) process substitution avoids a subshell that would lose the array. On bash 3.2 (stock macOS) neither works — you’d fall back to a sorted-comm diff of two temp files. The principle is universal: read once, diff in memory, act in one batch. It’s the difference between a reconciler that’s invisible on a 5-minute timer and one that shows up in top.

4. Operations you genuinely cannot make idempotent — use an idempotency key

Some actions have no natural “read current state” — you can’t ask “did my POST already create the order?” if the API returns a fresh random ID each time. The industry answer is an idempotency key: the caller generates a stable unique token and the server dedupes on it. Stripe, PayPal, and most payment APIs support an Idempotency-Key header exactly for this:

# Derive a STABLE key from the request's identity, not from $RANDOM.
# Same logical request => same key => server returns the original result, no dupe.
idem_key=$(printf '%s' "order:${customer}:${cart_hash}" | sha256sum | cut -c1-32)

curl -fsS -X POST https://api.example.com/v1/charges \
  -H "Idempotency-Key: $idem_key" \
  -H "Authorization: Bearer $TOKEN" \
  --data "amount=1999&currency=usd"

Two rules make this work: the key must be deterministic for the same logical operation (hash the inputs — never $RANDOM, date, or a fresh UUID per attempt), and the server must store the key and its result. When you control both ends, the same trick works locally: record a claim row keyed by the operation’s hash before you act, and skip if the claim already exists. This is how “exactly once” is faked on top of an “at least once” world — you never truly get exactly-once delivery, you get at-least-once delivery plus idempotent processing.

5. Partial failure and the myth of the transactional script

A shell script is not a transaction. If ensure_user succeeds, ensure_dir succeeds, and ensure_service then fails, you’re left in a partial state — and set -e will abort you there. Idempotency is exactly what saves you: because each step is independently convergent, re-running the whole script finishes the job — the succeeded steps no-op, the failed one retries. So the design rule is make every step independently convergent and order them so partial progress is always safe:

• Order for safety. Create the directory and write the config before starting the service, never after — a half-run should never leave a service pointing at a missing file.
• Don’t couple steps that must be atomic together into two shell commands. If two changes must both apply or neither, push them into one atomic operation — a single rename(2), a DB --single-transaction, an API call that accepts both fields at once.
• Never write a marker spanning multiple actions. One marker = one atomic unit of work. A setup.complete marker covering ten steps means a crash on step 7 either leaves no marker (redo all ten, hopefully idempotent) or, if you wrote it early, a marker that lies.

6. Security of the reconcile loop

Reconcilers run as root and touch config, so they’re an attack surface:

• Marker and state-dir permissions. state_init does chmod 0700 "$STATE_DIR" and markers use umask 077 for a reason: a world-writable state dir lets an unprivileged user pre-create a marker and suppress a security-relevant one-shot (e.g. skip a “harden sshd” step). Own the state dir as root, mode 0700.
• Sentinel/line injection. ensure_line interpolates $line into a printf/sed program. If $line is attacker-controlled and contains sed metacharacters or a newline, you can get command or line injection. Treat any externally-sourced content as data: prefer grep -qF (fixed strings), escape before sed, and never eval reconstructed content.
• Symlink / TOCTOU attacks on the state dir. If $STATE_DIR lives somewhere an attacker can create symlinks (a shared /tmp — another reason not to use it), your >"$marker" can be redirected to clobber a file root can write but you didn’t intend. Keep state under root-owned /var/lib.
• install over cp for perms. install -m 0600 -o root -g root sets ownership and mode in the same call as the copy, closing the window where a freshly-cp’d secret file briefly has the umask’s default (often world-readable) mode before a follow-up chmod.

7. Portability: the GNU-vs-BSD reconciler minefield

This course targets Linux + bash 4+/5 + GNU coreutils, but reconcilers get copied onto macOS laptops, Alpine containers (busybox), and BSD hosts, where the same command behaves differently. The ones that bite idempotency code specifically:

The defensive move you already saw in ensure_dir/ensure_file — stat -c … 2>/dev/null || stat -f … — is the template: try the GNU form, fall back to the BSD form. When in doubt, branch on uname -s once at the top and set a function.

8. The reconcile loop is a controller (why “level-triggered” matters)

The deepest framing: an idempotent reconciler is a control loop, and the same design underpins Kubernetes controllers, systemd, and every config-management engine. The key property is that it’s level-triggered, not edge-triggered.

• Edge-triggered = “react to the event that something changed.” Miss the event (script wasn’t running, message was dropped) and you’re permanently wrong.
• Level-triggered = “observe the current level and drive toward desired, regardless of how you got here.” Miss a hundred events; the next reconcile still converges, because it reads reality fresh every time.

That’s why a reconciler should always read actual state rather than trusting a remembered “I already did this” for anything that can drift. Markers are a deliberate, narrow exception for expensive one-shots you’ve decided can’t drift. Everything else: observe → diff → act, every single run. Build that reflex and your shell scripts inherit the same robustness that makes Kubernetes self-heal — a pod dies, the next reconcile notices the level is wrong and recreates it, no event required.

The 12-Item Idempotency Footgun List

After reviewing dozens of “almost-idempotent” scripts in production, these are the patterns that fail in practice:

1. The “marker before work” bug. Writing the state file before the operation succeeds, so a crash leaves a marker that lies. Always write the marker as the last step.

2. mkdir without -p. First run succeeds, second run fails with “File exists.” Always mkdir -p, even when you “know” the dir doesn’t exist.

3. echo "line" >> file without sentinel. Each rerun appends another copy. Always pair appends with a sentinel-comment guard.

4. useradd without an id -u guard. Fails on rerun with “user already exists.” Always read first.

5. ln -s without -fn. First run creates the link; second run fails. Worse, if the link target is a directory and -n is missing, ln -sf creates a link inside the directory. Always ln -sfn.

6. set -e masking idempotency holes. useradd user fails because user exists; set -e aborts the script. The author “fixes” it with useradd user || true, hiding the failure. The right fix is to read first.

7. sed -i without backup file disposal. sed -i.bak leaves .bak files everywhere on every run. Either sed -i (no backup, GNU only) or sed -i.bak followed by rm -f file.bak. Note: sed -i syntax differs between GNU sed and BSD sed; on macOS use sed -i '' or sed -i.bak with cleanup.

8. Dry-run that doesn’t reach all code paths. If if (( DRY_RUN )); then return; fi is at the top of every function, you’re testing nothing. Use a do_or_say wrapper around mutating commands; reads always run.

9. State stored in /tmp. systemd-tmpfiles may clean /tmp on reboot or schedule. Your “already done” marker disappears, the script re-runs the work, and you get duplicate side effects. Use /var/lib/myapp/state for system state.

10. Markers in different namespaces collide. Two scripts both write /var/lib/state/done; one clobbers the other’s state. Always namespace with the script name: $STATE_DIR/myapp/v2-installed.

11. Reconciliation that compares wrong fields. “Does the user exist?” → id -u says yes. “Is the home dir right?” → never checked. The user drifted. Reconcile every field you declare desired state for.

12. Idempotency that depends on shared mutable state. Two scripts both manage /etc/sysctl.d/99-tuning.conf. They each write their own line, but on conflict, the last one wins. The reconciliation pattern only protects what one script owns. For multi-owner files, use /etc/sysctl.d/<owner>-<name>.conf per owner.

Quick-Reference Card

┌─ THE FOUR-STEP RECONCILIATION ────────────────────────────────────────┐
│  1. desired = arguments              "I want X to be true"            │
│  2. actual  = read system            "What is true now?"              │
│  3. delta   = desired - actual       "What's different?"              │
│  4. apply minimal change             "Move only the delta"            │
└────────────────────────────────────────────────────────────────────────┘

┌─ IDEMPOTENT PRIMITIVES (use these by default) ────────────────────────┐
│  mkdir -p DIR                        idempotent dir create            │
│  ln -sfn TARGET LINK                 idempotent symlink               │
│  install -m MODE -o U -g G S D       idempotent file install w/ perms │
│  rm -f FILE                          idempotent removal               │
│  id -u USER || useradd USER          guarded user create             │
│  getent group GRP || groupadd GRP    guarded group create             │
│  systemctl is-active SVC || start    guarded service start           │
└────────────────────────────────────────────────────────────────────────┘

┌─ STATE MARKERS ───────────────────────────────────────────────────────┐
│  Where:    /var/lib/$APP/state/   (NOT /tmp)                         │
│  Name:     <action>-<version>.marker  (versioned)                    │
│  Content:  ISO 8601 timestamp + identifying metadata                 │
│  When to write:  AFTER the work succeeds, never before               │
│  When to use:  one-shot/expensive ops; NOT for ongoing config        │
└────────────────────────────────────────────────────────────────────────┘

┌─ DRY-RUN DISCIPLINE ──────────────────────────────────────────────────┐
│  Reads run live (id, getent, stat, curl GET).                        │
│  Writes gated by do_or_say: in DRY_RUN=1, prints "WOULD: cmd".       │
│  Same code paths as live mode; only side effects differ.             │
│  Output is greppable: `script --dry-run | grep ^WOULD`.              │
└────────────────────────────────────────────────────────────────────────┘

┌─ APPEND-ONLY-IF-MISSING ──────────────────────────────────────────────┐
│  grep -qF "$sentinel" "$file" && update_in_place || append          │
│  Sentinel = unique comment: "# managed-by:$APP:$RULE"                │
│  Survives reformatting; greppable in audit                           │
└────────────────────────────────────────────────────────────────────────┘

┌─ ATOMIC FILE WRITE ───────────────────────────────────────────────────┐
│  tmp=$(mktemp "$path.XXXXXX")                                         │
│  printf '%s' "$content" >"$tmp"                                       │
│  install -m MODE -o U -g G "$tmp" "$path"                            │
│  rm -f "$tmp"                                                         │
│  Reader sees old or new file, never partial.                         │
└────────────────────────────────────────────────────────────────────────┘

┌─ MUTUAL EXCLUSION (when runs can overlap) ────────────────────────────┐
│  Portable:  mkdir "$lock" 2>/dev/null || exit 0                      │
│  Linux:     exec 9>"$lock"; flock -n 9 || exit 0                     │
│  flock auto-releases on crash; mkdir-lock can go stale (add PID).    │
└────────────────────────────────────────────────────────────────────────┘

Practice challenges

Work these in order — they escalate from a one-line guard to a genuinely concurrent, network-facing reconciler. Each has a runnable solution; try it before you open the details.

Challenge 1 — Make an append idempotent (beginner)

A bootstrap script contains echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc. Running the bootstrap five times adds the line five times. Rewrite it so any number of runs leaves exactly one copy of the line.

<details> <summary>Solution</summary>

line='export PATH="$HOME/.local/bin:$PATH"'
grep -qxF -- "$line" ~/.bashrc || printf '%s\n' "$line" >> ~/.bashrc

Why: grep -qxF matches the whole line (-x) as a fixed string (-F), quietly (-q); the || only appends when it’s absent. Guarding the append with a read is the smallest possible reconciler. </details>

Challenge 2 — Convert three setup commands to idempotent primitives (beginner)

This snippet fails on its second run. Fix each line to be safe to re-run, without suppressing errors with || true.

mkdir /opt/app/releases
ln -s /opt/app/releases/v1 /opt/app/current
cp app.conf /etc/app/app.conf     # must end up mode 0640, root:root

<details> <summary>Solution</summary>

mkdir -p /opt/app/releases
ln -sfn /opt/app/releases/v1 /opt/app/current
install -m 0640 -o root -g root app.conf /etc/app/app.conf

Why: -p makes mkdir a no-op when the dir exists; ln -sfn overwrites and never links inside an existing dir; install copies and sets mode/owner atomically, so perms converge even if the file already existed with the wrong mode. </details>

Challenge 3 — Reconcile more than existence (intermediate)

This function is idempotent about whether the user exists but ignores drift in the login shell. Extend it so that if the user exists with the wrong shell, it’s corrected — and if everything already matches, the function prints nothing and changes nothing.

ensure_deploy_user() {
  id -u deploy &>/dev/null || useradd --shell /bin/bash deploy
}

<details> <summary>Solution</summary>

ensure_deploy_user() {
  local want_shell=/bin/bash
  if id -u deploy &>/dev/null; then
    local have_shell
    have_shell=$(getent passwd deploy | cut -d: -f7)
    if [[ "$have_shell" != "$want_shell" ]]; then
      usermod --shell "$want_shell" deploy
      echo "[CHANGE] deploy shell $have_shell -> $want_shell"
    fi
  else
    useradd --shell "$want_shell" deploy
    echo "[CHANGE] created deploy"
  fi
}

Why: you declared the shell as desired state, so you must read and compare it, not just existence. Reconcile every field you declare — the classic drift bug is checking one field and assuming the rest. </details>

Challenge 4 — Add a real --dry-run mode (intermediate)

Take the small installer below and add a --dry-run flag so that, in dry-run, it prints exactly what it would do (prefixed WOULD:) and touches nothing — while the existence checks still run live so the preview reflects reality.

setup() {
  mkdir -p /opt/app
  systemctl start app
}

<details> <summary>Solution</summary>

DRY_RUN=0
[[ "${1:-}" == "--dry-run" ]] && { DRY_RUN=1; shift; }
do_or_say() { if (( DRY_RUN )); then printf 'WOULD: %s\n' "$*" >&2; else "$@"; fi; }

setup() {
  [[ -d /opt/app ]]                    || do_or_say mkdir -p /opt/app
  systemctl is-active app &>/dev/null  || do_or_say systemctl start app
}

Run ./setup.sh --dry-run | grep ^WOULD to see the plan. Why: the reads ([[ -d ]], is-active) stay live in both modes; only the mutating half is wrapped, so the previewed diff is exactly what a live run would apply — same control flow, gated side effects. </details>

Challenge 5 — TOCTOU-safe “run exactly once” (advanced)

Write run_once NAME CMD... that runs CMD only if a marker NAME doesn’t exist, writes the marker after success, and is safe when two copies start simultaneously — only one should run the work; the other must not run it and must not error. (Assume a shared STATE_DIR.)

<details> <summary>Solution</summary>

: "${STATE_DIR:=/var/lib/myapp/state}"; mkdir -p "$STATE_DIR"

run_once() {
  local name="$1"; shift
  local marker="$STATE_DIR/$name.done"
  local lock="$STATE_DIR/$name.lock"

  [[ -f "$marker" ]] && return 0                 # fast path: already done

  # Atomic claim: exactly one concurrent caller wins the mkdir.
  if ! mkdir "$lock" 2>/dev/null; then
    return 0                                      # someone else is doing it
  fi
  trap 'rmdir "$lock" 2>/dev/null' RETURN         # release on return

  [[ -f "$marker" ]] && return 0                  # re-check inside the lock
  if "$@"; then
    date -u +%FT%TZ > "$marker"                   # marker AFTER success
  else
    return 1                                       # no marker => retries next run
  fi
}

Why: mkdir is the atomic gate that closes the TOCTOU window — only one of two racing callers creates the lockdir. The re-check inside the lock handles the case where the other runner finished between your fast-path check and acquiring the lock, and writing the marker only on success preserves clean retry-on-failure. </details>

Challenge 6 — Idempotent resource creation over HTTP (advanced)

You must POST to https://api.example.com/v1/widgets to create a widget for a given name, but the API has no “get by name” endpoint and returns a fresh random ID each call — so naive retries create duplicates. Make widget creation idempotent so re-running the script never creates a second widget for the same name.

<details> <summary>Solution</summary>

ensure_widget() {
  local name="$1"
  # Deterministic key from the request's identity — NOT $RANDOM/uuidgen.
  local key
  key=$(printf 'widget:%s' "$name" | sha256sum | cut -c1-32)

  curl -fsS -X POST https://api.example.com/v1/widgets \
    -H "Idempotency-Key: $key" \
    -H "Authorization: Bearer ${API_TOKEN:?}" \
    -H 'Content-Type: application/json' \
    --data "$(jq -nc --arg n "$name" '{name:$n}')"
}

Why: when there’s no way to read current state, you shift the dedupe to the server with an idempotency key derived deterministically from the request’s identity. Same name → same key → the server returns the original widget instead of creating a new one. The key must be stable across retries (hash the inputs), never random per attempt. </details>

Common beginner mistakes

These are misconceptions about what idempotency is, distinct from the tactical footguns above — get the mental model right and the tactics follow.

• “I added || true, so now it’s idempotent.” No — you made it not crash, which is the opposite of idempotent. || true throws away the exit code and hides whether the work happened. Idempotency comes from reading state and acting on the delta, not from swallowing errors. Right model: guard with a read (id -u user || useradd …), don’t suppress the failure.

• “It ran twice without an error, so it’s idempotent.” Not crashing ≠ same end state. echo line >> file never errors and still appends a duplicate every run. The test is “is the end state identical after N runs?”, not “did it exit 0?”. Right model: check the resulting state, not just the exit code.

• “The file/user/dir exists, so I’m done.” Existence is one field; you usually declared more (mode, owner, content, shell, membership). A file that exists with the wrong permissions has drifted. Right model: reconcile every field you declared as desired state, not just presence.

• “Dry-run means: if DRY_RUN, return early.” That skips the reads too, so the preview reflects nothing and you’ve tested no code path. Right model: keep reads live, wrap only the writes in do_or_say; control flow must be identical to a real run.

• “I’ll write the marker first so I know the step started.” A marker written before the work lies after a crash — the next run sees “done” and skips unfinished work. Right model: the marker is a receipt, written only after success. One marker = one atomic unit of work.

• “Markers can live in /tmp.” /tmp is cleared on reboot (and by systemd-tmpfiles), so your “already installed” claim silently evaporates and the one-shot re-runs — sometimes with duplicate side effects. Right model: durable state goes under root-owned /var/lib/$APP/state, namespaced by app.

• “Idempotent means safe.” rm -rf /var/cache/app is perfectly idempotent (the dir stays gone) and still destroys live cache on every run. Idempotency is about repeatability of end state, not harmlessness. Right model: reason about safety separately — idempotent-and-safe is the goal.

• “Two scripts can both manage the same file safely because each is idempotent.” Idempotency protects what one owner controls; two owners writing the same file just means last-writer-wins churn. Right model: give each owner its own file (/etc/sysctl.d/<owner>-name.conf), or make one script the sole owner.

Glossary

• Idempotency — the property that running an operation N times yields the same end state as running it once. The core contract of a re-runnable script.
• Convergent idempotency — the practical form most ops scripts target: from any starting state the script reaches the desired state. Ansible/Puppet/Chef are convergent.
• Strict idempotency — the mathematical form f(x) = f(f(x)): applying twice equals applying once, exactly.
• Reconciliation — reading the actual state, comparing it to the desired state, and applying only the difference. The engine of every idempotent function here.
• Desired state — what you declare should be true (usually the function’s arguments): “user deploy exists with this home and shell.”
• Actual state — what is currently true on the system, discovered by reading it (id, getent, stat, an API GET).
• Drift — when actual state diverges from desired state after the fact (someone edited the file, deleted the user, changed a sysctl). Reconcilers detect and correct drift on the next run.
• Delta — the difference between desired and actual; the minimal set of changes to apply. An empty delta means a no-op.
• No-op — “no operation”: the reconciler found actual already equals desired and changed nothing. The most important, most common outcome of a healthy reconcile.
• State marker / sentinel file — a small file written after a one-shot action to record that it completed, so reruns skip it. Lives under /var/lib/$APP/state, written after success.
• Sentinel comment — a unique tag (e.g. # managed-by:myapp:heartbeat) embedded in a config line so a reconciler can find and update its line without duplicating it.
• Dry-run / --check — a mode that reads live but gates every write, printing what it would do (WOULD: …) instead of doing it. Used for operator preview and CI drift checks.
• do_or_say — the wrapper that implements dry-run: it runs the command normally, or prints it in dry-run mode, so control flow is identical either way.
• Idempotent primitive — a command that’s safe to repeat by construction: mkdir -p, ln -sfn, install -m, rm -f, sentinel-guarded append, atomic write. Contrast with their footgun twins.
• Guard condition — a read that gates an action, e.g. id -u user || before useradd, or grep -qF sentinel || before an append.
• TOCTOU — time-of-check to time-of-use: the window between checking state and acting on it, in which another process can change the world, causing a race. Closed with atomic operations or locks.
• Atomic write — writing to a temp file on the same filesystem and rename(2)-ing it into place, so readers see either the whole old file or the whole new one, never a partial.
• Idempotency key — a caller-generated, deterministic token (e.g. a hash of the request’s identity) sent so a server can dedupe retries of an operation that has no readable current state, such as a POST.
• Level-triggered — driving toward desired state by observing the current level every run (vs edge-triggered, reacting to change events). Reconcilers are level-triggered, which is why they self-heal after missed events.
• Convergence — the process of repeatedly reconciling until actual equals desired; a converged system reports zero changes per run.

What’s Next

Idempotency is the discipline; filesystem semantics is what your idempotent operations rest on. The next lesson, Filesystem Semantics: Hard Links, Symlinks, Mount Namespaces & fsync Discipline, drills into the mechanics that determine whether your atomic writes are actually atomic, why ln -sfn behaves differently across filesystems, when mv is atomic and when it isn’t (cross-device renames!), and how fsync discipline turns “I wrote it to disk” into a guarantee instead of a hope. Together with this lesson, those primitives let you build scripts that converge correctly even on flaky filesystems and crash-prone hosts.