Every command you’ve ever run on a Unix system started life with three file descriptors already open: standard input (fd 0), standard output (fd 1), and standard error (fd 2). Almost every interesting thing the shell does — silencing noisy commands, capturing output, tee-ing logs to two places, feeding a heredoc to cat, swapping stdout and stderr — is a manipulation of those three numbers, plus optionally a few you open yourself.
The good news: once you know the file-descriptor model, every redirection idiom in shell collapses into one mental picture. The bad news: shell’s redirection syntax is terse, order-sensitive, and full of subtle gotchas that catch every beginner. The classic one is cmd 2>&1 > file — which looks like “redirect stderr to stdout and stdout to file” but actually does the opposite of what most people expect.
This lesson walks the model end-to-end. Read it slowly. Type the examples. The payoff is that every redirection you ever encounter — in someone else’s script, in a man page, in a log-rotation cron job — will read like English instead of like hieroglyphics.
1. The file-descriptor model in 90 seconds
A file descriptor is a small non-negative integer that the kernel uses to track an open file (or pipe, socket, terminal, etc.) for a particular process. When a process starts, the kernel typically gives it three pre-opened FDs:
- fd 0 — stdin — where the process reads input from. Connected to your terminal keyboard by default.
- fd 1 — stdout — where the process writes its primary output. Connected to your terminal screen.
- fd 2 — stderr — where the process writes errors and diagnostic messages. Also connected to your terminal screen — but separately from stdout.
You can open more FDs by open()-ing files (or pipes, etc.); these get the next free non-negative integer (typically starting at 3). The shell exposes exec for opening custom FDs in your script (section 6).
When you run cmd > file, the shell does this:
open("file", O_WRONLY | O_CREAT | O_TRUNC, 0644)— gets back a new FD, say 3.dup2(3, 1)— make fd 1 point to whatever fd 3 points to (the file).close(3)— close the temporary fd 3.fork()andexec()the command.
Now cmd’s fd 1 (stdout) is wired to the file, not the terminal. When cmd calls printf or write(1, ...), the bytes go to the file.
This is the entire mental model. Every redirection operator in shell is one of these open + dup2 + close sequences. Once you see them this way, the order-sensitivity of 2>&1 becomes obvious.
2. The basic redirection operators
cmd > file # redirect stdout to file (truncate file first)
cmd >> file # redirect stdout to file (append)
cmd < file # redirect stdin to read from file
cmd 2> file # redirect stderr to file (truncate)
cmd 2>> file # redirect stderr to file (append)
cmd > file 2>&1 # redirect both stdout and stderr to file (more on this in section 4)
cmd &> file # bash shorthand for "both stdout and stderr to file"
cmd >& file # same as &> (older syntax)
cmd >> file 2>&1 # both, appending
cmd &>> file # bash 4+ shorthand for "both, appending"
The number before > is the file descriptor being redirected. > alone means 1>. The default redirection target is fd 1 (stdout); to redirect stderr you must say 2>.
The truncate-vs-append distinction
> truncates the destination file before writing. >> appends. This is the most catastrophic mistake beginners make. If you have a precious file and you accidentally write > instead of >>, your file is wiped before the command even runs:
cmd > /var/log/important.log # WIPES the log first, then writes cmd's output
cmd >> /var/log/important.log # Appends — preserves existing content
Real production disasters happen here. Memorise the difference and prefer >> for any file you might care about. To prevent accidental clobbering globally, set:
set -o noclobber # equivalent to set -C
After this, > will refuse to overwrite an existing file. To force overwrite when you really mean it, use >|:
cmd >| file # force-overwrite even with noclobber
noclobber is a great safety net for interactive shells; less common in scripts.
Discarding output: /dev/null
/dev/null is a “sink” device. Anything written to it is silently discarded. To silence a command:
cmd > /dev/null # discard stdout, keep stderr visible
cmd 2> /dev/null # discard stderr, keep stdout visible
cmd > /dev/null 2>&1 # discard both
cmd &> /dev/null # bash shorthand for "discard both"
To use /dev/null as input (i.e. give the command an empty stdin):
cmd < /dev/null # cmd's stdin is empty; useful for non-interactive runs
This last form is critical for scripts that get run by cron or systemd — without it, some commands hang waiting for stdin. We’ll come back to this in lesson 9 (process management) and lesson 25 (cron).
3. Here-documents (<<EOF) and here-strings (<<<)
Sometimes you want to feed a multi-line block of text to a command’s stdin without creating a temporary file. That’s what here-docs do.
cat <<EOF
Hello, $USER.
The current date is $(date).
This is line 3.
EOF
The <<EOF says: “stdin for this command comes from a here-document. Read everything until a line containing exactly EOF, and feed it to the command’s stdin.”
EOF is just a delimiter — you can use any string, but EOF and END are conventional. Choose a delimiter that won’t appear in your content.
Variable expansion in here-docs
By default, here-docs expand variables and command substitution, just like double-quoted strings:
NAME="Alice"
cat <<EOF
Hello, $NAME!
Today is $(date +%A).
EOF
# Output:
# Hello, Alice!
# Today is Monday.
To disable expansion (treat the heredoc as literal), quote the delimiter:
cat <<'EOF'
This $VAR is literal.
$(date) is also literal.
EOF
# Output:
# This $VAR is literal.
# $(date) is also literal.
The single quotes around 'EOF' mean “treat the body as if it were single-quoted.” This is essential when you’re emitting code (shell scripts, SQL, JSON) and don’t want the shell parsing it. Use <<'EOF' for any embedded code that contains $ or `.
Indented here-docs (<<-)
The <<- form (with a dash) strips leading tabs from each line of the body. This lets you indent the body to match surrounding code:
process() {
cat <<-EOF
Line 1
Line 2
Line 3
EOF
}
Critical: only literal tabs are stripped, not spaces. If your editor is set to use spaces for indentation, <<- won’t strip them. Either configure your editor to use tabs for here-docs or don’t indent the body. (This is a common cause of “the heredoc has weird leading whitespace” bugs.)
Here-strings (<<<)
A here-string is a single-line variant: pass a string as stdin without quoting:
grep "alice" <<< "name: alice
name: bob
name: carol"
# More commonly:
read -r line <<< "this is the input line"
The here-string <<< "$LINE" is the canonical way to feed a single string to read (or any command that wants stdin). It’s equivalent to printf '%s\n' "$LINE" | cmd, but doesn’t fork a subshell — cmd runs in the current shell.
Use cases:
# Parse CSV in current shell (no subshell trap from L4)
IFS=',' read -ra FIELDS <<< "$LINE"
# Feed JSON to jq
jq '.user.name' <<< "$JSON_RESPONSE"
# Feed a single value to a tool
md5sum <<< "Hello, world"
Note: bash’s <<< adds a trailing newline to the string before piping it. printf '%s' "$LINE" does not. For read and most tools this is exactly what you want.
4. The order-of-evaluation gotcha: 2>&1 > file vs > file 2>&1
This is the single most-misunderstood piece of redirection syntax. Read carefully.
cmd > file 2>&1 # CORRECT — both stdout and stderr go to file
cmd 2>&1 > file # WRONG — stderr still goes to terminal, stdout goes to file
What’s going on? The shell processes redirections left to right. Each redirection takes effect as it’s encountered, in order.
Let’s walk through cmd 2>&1 > file:
- Initial state: stdout (fd 1) → terminal. stderr (fd 2) → terminal.
2>&1— make fd 2 point to wherever fd 1 currently points. Currently fd 1 points to the terminal. So fd 2 is now also pointing to the terminal. (No change in effect.)> file— make fd 1 point tofile. Now fd 1 → file. But fd 2 is still pointing to the terminal (we copied it earlier when fd 1 was the terminal).- Result: stdout in file, stderr on terminal.
Now cmd > file 2>&1:
- Initial state: stdout → terminal. stderr → terminal.
> file— make fd 1 point tofile. Now fd 1 → file.2>&1— make fd 2 point to wherever fd 1 currently points. fd 1 currently points tofile. So fd 2 is now also pointing tofile.- Result: both stdout and stderr in file.
The mental model: 2>&1 is “copy the current target of fd 1 to fd 2,” not “redirect stderr to stdout.” It’s a snapshot, not a link. So you must do the redirections in the right order.
The bash shorthand
Bash provides &> and &>> as a less-error-prone shorthand:
cmd &> file # equivalent to cmd > file 2>&1
cmd &>> file # equivalent to cmd >> file 2>&1
These are unambiguous — they always redirect both streams. Use them when you want both, unless you need POSIX portability (in which case write > file 2>&1 explicitly).
Swapping stdout and stderr
To send stdout to where stderr was going and vice versa (e.g., to filter on stderr in a pipe), use a temporary FD:
cmd 3>&2 2>&1 1>&3 3>&-
Decompose:
3>&2— fd 3 = stderr’s destination2>&1— fd 2 = stdout’s destination1>&3— fd 1 = (saved) stderr’s destination3>&-— close fd 3
After this, stdout and stderr are swapped. You’ll see this rarely, but it shows up in scripts that need to pipe stderr through grep or awk while leaving stdout alone.
5. tee — write to both a file and stdout
Sometimes you want output to go to both a log file and the terminal:
cmd | tee log.txt
tee reads stdin and writes it to both stdout (the next stage of the pipeline) and to the named file(s). Useful for live-tailing a long-running command:
make build 2>&1 | tee build.log
tee -a appends instead of truncating:
cmd | tee -a log.txt
You can tee to multiple files:
cmd | tee log1.txt log2.txt log3.txt
To tee with stderr also captured:
cmd 2>&1 | tee log.txt # both streams interleaved into the log
To tee stdout to one file and stderr to another (using process substitution from section 7):
cmd > >(tee out.log) 2> >(tee err.log >&2)
This is the canonical “log everything, but keep stderr visible” pattern. Unpacked: > >(tee out.log) redirects stdout to a process running tee out.log. 2> >(tee err.log >&2) redirects stderr to a process running tee err.log >&2 (which writes back to stderr after teeing).
6. exec for FD manipulation in a script
The exec builtin (already covered briefly in lesson 1 for replacing the shell with another binary) has a second form: with no command, it modifies the current shell’s file descriptors permanently.
# Redirect all of THIS SCRIPT's stdout to a log file from now on
exec > /var/log/myscript.log
# Redirect all stderr to a log file
exec 2> /var/log/myscript.err.log
# Both
exec > /var/log/myscript.log 2>&1
# Open fd 3 for reading from a config file
exec 3< /etc/myapp.conf
read -r LINE <&3 # read one line from fd 3
exec 3<&- # close fd 3
# Open fd 4 for writing to a custom log
exec 4> /var/log/audit.log
echo "Important event" >&4
exec 4>&-
The pattern of opening custom FDs is essential when you want to:
- Process two files in parallel (read fd 3, fd 4 alternately).
- Keep a long-running file open across many writes (avoid open/close overhead).
- Build complex pipelines without temporary files.
Lesson 14 (concurrency, FIFOs, flock) uses these patterns extensively.
Common idiom: redirect script’s logs once, at startup
#!/usr/bin/env bash
set -euo pipefail
# Redirect everything to a log file, with timestamps via tee+ts
LOG_FILE="/var/log/$(basename "$0").log"
exec > >(ts '%Y-%m-%dT%H:%M:%S' >> "$LOG_FILE") 2>&1
echo "Starting..." # goes to LOG_FILE
do_work # all of its output also goes to LOG_FILE
ts (from moreutils) prefixes each line with a timestamp. The combination is a tiny one-line “structured logger” for shell scripts.
7. Process substitution: the elegant alternative to temp files
Process substitution gives you a filename (or fd path) that, when read or written, runs a command. Bash creates a FIFO or /dev/fd/N device behind the scenes.
diff <(ls /var/log) <(ls /backup/var/log)
Each <(cmd) expands to a path like /dev/fd/63. diff opens each path as a file. Bash arranges for the corresponding command to write to that path. The result: diff thinks it’s diffing two files, but it’s diffing the live output of two commands.
Reading from process substitution
mapfile -t LINES < <(grep ERROR /var/log/app.log)
while IFS= read -r line; do
process "$line"
done < <(some-stream-generator)
The < <(cmd) form is one of the most useful idioms in modern bash: it’s like cmd | while read but the loop runs in the current shell (no subshell trap from L4).
Writing to process substitution
some-cmd > >(gzip > out.log.gz)
>(gzip > out.log.gz) expands to a path that, when written to, feeds bytes to gzip. gzip writes its compressed output to out.log.gz. Result: some-cmd’s stdout is compressed in real time, no temporary file.
# Tee into two simultaneous compressors
some-cmd > >(gzip > out.log.gz) 2> >(gzip > err.log.gz)
The portability caveat
Process substitution is a bash extension. Not POSIX. Doesn’t work in dash or ash or busybox sh. If you need pure POSIX, use a temporary file or a named pipe. Lesson 31 (POSIX portability) discusses workarounds.
8. Named pipes (FIFOs)
A named pipe (FIFO) is a special file on the filesystem that behaves like a pipe. One process writes to it, another reads from it, and the kernel buffers the data. Created with mkfifo:
mkfifo /tmp/myfifo
# In one terminal:
echo "Hello" > /tmp/myfifo
# In another terminal:
cat < /tmp/myfifo # prints "Hello"
rm /tmp/myfifo # clean up
Useful for coordinating between separate processes that you don’t want to chain directly with a pipe. We’ll use FIFOs for concurrency control in lesson 14.
The right pattern is to wrap the FIFO in a tempdir and clean it up with a trap:
TMPDIR=$(mktemp -d)
trap 'rm -rf -- "$TMPDIR"' EXIT
mkfifo "${TMPDIR}/myfifo"
# ...
We’ll cover trap thoroughly in lesson 10.
9. Reading from and writing to network sockets (/dev/tcp)
Bash has a built-in network feature that’s wildly useful and almost nobody knows about. The pseudo-files /dev/tcp/HOST/PORT and /dev/udp/HOST/PORT open a socket when redirected:
exec 3<>/dev/tcp/example.com/80 # bidirectional TCP socket on fd 3
echo -e "GET / HTTP/1.0\r\nHost: example.com\r\n\r\n" >&3
cat <&3 # read response
exec 3<&- # close
Or as a one-liner:
echo > /dev/tcp/database.example.com/5432 && echo "DB port reachable"
This is the cleanest way to do TCP port-checking from a shell script — no nc/netcat/telnet dependency. We cover /dev/tcp in depth in lesson 21 (network operations).
10. Common redirection patterns
Quiet mode
cmd > /dev/null 2>&1 # silence everything
cmd &> /dev/null # bash shorthand
cmd 2>/dev/null # silence errors only (keep stdout)
cmd >/dev/null # silence stdout only (keep stderr — diagnostic-friendly)
Logging
# Append to log file, both streams
cmd >> /var/log/app.log 2>&1
# Tee to log AND show on terminal
cmd 2>&1 | tee -a /var/log/app.log
# Timestamped log
cmd 2>&1 | ts '%Y-%m-%dT%H:%M:%S' >> /var/log/app.log
# Separate stdout and stderr logs, but capture both
cmd > >(tee -a /var/log/app.out >&1) 2> >(tee -a /var/log/app.err >&2)
Capture into a variable
RESULT=$(cmd) # stdout only
RESULT=$(cmd 2>&1) # stdout + stderr (interleaved)
RESULT=$(cmd 2>/dev/null) # stdout, suppress errors
Heredoc to a command
ssh user@host <<EOF
sudo systemctl restart myservice
sudo systemctl status myservice
EOF
Or with disabled expansion:
ssh user@host <<'EOF'
echo "Hostname: \$(hostname)" # \$ stays literal — runs hostname on the REMOTE
EOF
Hereformat for SQL / JSON / config files
psql "$DB_URL" <<'SQL'
CREATE TABLE users (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL
);
INSERT INTO users (name) VALUES ('alice'), ('bob');
SQL
Append a single line to a file
echo "127.0.0.1 myhost" | sudo tee -a /etc/hosts > /dev/null
Why sudo tee -a instead of sudo echo ... >> /etc/hosts? Because the >> redirection is performed by the current shell (running as you), not by sudo. To run the redirection as root, you need to invoke a privileged process (tee) that writes to the file itself.
Capture both stdout and stderr separately
RESULT_OUT=$(cmd 2>err.log)
RESULT_ERR=$(< err.log)
Or, all in one without a temp file (advanced):
{ RESULT_OUT=$(cmd 2>&1 >&3); } 3>&1
# (... and capture stderr separately — see bash-faq for details)
This is one of the genuinely awkward things about shell. For complex stdout/stderr capture, prefer dropping into a real temp file or a Python script.
Run a script’s all output through a transformer
exec > >(grep -v 'DEBUG') 2>&1 # filter out DEBUG lines from this script onward
Discard but keep the exit code
cmd >/dev/null 2>&1
echo "Exit code: $?" # cmd's exit code, output discarded
Or inline:
if cmd >/dev/null 2>&1; then
echo "succeeded"
fi
11. The noclobber / >| safety net
For interactive use, set -o noclobber (or set -C) is a classic guard against accidental > overwrites:
set -o noclobber
echo hi > /etc/passwd # bash: /etc/passwd: cannot overwrite existing file
echo hi >| /etc/passwd # force; OK
Add set -o noclobber to your ~/.bashrc if you frequently work with precious files. It’s noisy in scripts though — you’d have to use >| everywhere — so most production scripts don’t enable it.
12. The redirection cheat-sheet
Memorise this. Print it. Tape it to your monitor.
> file stdout to file (truncate)
>> file stdout to file (append)
< file stdin from file
2> file stderr to file (truncate)
2>> file stderr to file (append)
2>&1 duplicate fd 1 to fd 2 (snapshot, order-sensitive)
1>&2 duplicate fd 2 to fd 1 (e.g. echo "err" >&2 to write to stderr)
&> file both stdout and stderr to file (bash shorthand)
&>> file both, append (bash 4+)
> /dev/null discard stdout
2> /dev/null discard stderr
< /dev/null empty stdin (essential for non-interactive runs)
<<EOF here-doc, expansions ON
<<'EOF' here-doc, expansions OFF (literal)
<<-EOF here-doc, strip leading TABS only
<<< here-string (single line)
>(cmd) process substitution: a path that writes to cmd's stdin
<(cmd) process substitution: a path that reads from cmd's stdout
n> file redirect fd n to file
n< file open file for reading on fd n
exec n> file open fd n permanently in current shell (for writes)
exec n< file open fd n permanently in current shell (for reads)
exec n>&- close fd n
exec n<&- close fd n (alternate close form)
| cmd pipe stdout to cmd's stdin
|& cmd pipe both stdout and stderr to cmd's stdin (bash 4+)
13. Real example: a script that logs everything with structure
#!/usr/bin/env bash
# robust-runner.sh — runs a command, captures all output, with correct error handling
set -euo pipefail
IFS=$'\n\t'
CMD=("$@")
[[ ${#CMD[@]} -gt 0 ]] || { echo "Usage: $0 COMMAND [ARGS...]" >&2; exit 2; }
LOG_DIR="${LOG_DIR:-/tmp}"
TIMESTAMP=$(date -u +%Y%m%dT%H%M%SZ)
NAME=$(basename "${CMD[0]}")
LOG_OUT="${LOG_DIR}/${NAME}.${TIMESTAMP}.out"
LOG_ERR="${LOG_DIR}/${NAME}.${TIMESTAMP}.err"
# Run the command, capturing stdout and stderr to separate files
# while ALSO showing them live on the terminal
"${CMD[@]}" \
> >(tee "$LOG_OUT") \
2> >(tee "$LOG_ERR" >&2)
EXIT_CODE=$?
echo
echo "Command finished with exit code $EXIT_CODE"
echo "stdout: $LOG_OUT ($(wc -l < "$LOG_OUT") lines)"
echo "stderr: $LOG_ERR ($(wc -l < "$LOG_ERR") lines)"
exit "$EXIT_CODE"
Things to notice:
- Strict-mode preamble.
- Arguments collected as an array (
CMD=("$@")) so they’re properly quoted on re-invocation. - Process substitution to tee both streams independently.
- The line
2> >(tee "$LOG_ERR" >&2)is worth dissecting:2>redirects stderr;>(tee ... >&2)runsteewith the file as a destination and pipes its stdout back to the original stderr. Result: stderr is shown live AND captured. - Exit code preserved — the script exits with the wrapped command’s status.
This is the production-grade pattern for “run anything, log everything, don’t lose the exit code.” Save it as a snippet.
14. What you must internalise before lesson 8
- What are fd 0, 1, 2? (stdin, stdout, stderr.)
- What does
>vs>>do? (Truncate vs append.) - Why is
cmd 2>&1 > filewrong? (Order matters;2>&1is a snapshot of fd 1’s current target.) - What’s the shorthand for “both stdout and stderr to a file”? (
&> filein bash.) - What’s a here-doc, and how do you disable variable expansion in one? (
<<EOFfor body; quote the delimiter —<<'EOF'— to disable expansion.) - What’s a here-string and when do you use it? (
<<<for single-line input to a command, e.g.read <<< "$VAR".) - What does
< <(cmd)do? (Process substitution feeding cmd’s output as stdin to the current command.) - Why use
teewithsudoinstead ofsudo cmd >> file? (Redirection is performed by the calling shell as the calling user;sudo teeruns as root and writes the file as root.) - What’s the difference between
$(cmd)and<(cmd)? ($(cmd)is command substitution — captures stdout as a string.<(cmd)is process substitution — gives you a filename that streams stdout.) - What does
exec > log 2>&1do at the top of a script? (Redirects all of the current script’s subsequent stdout and stderr tolog.)
If any felt fuzzy, re-read. Lesson 8 (pipes and pipefail) is where redirection meets multi-stage pipelines and set -o pipefail becomes critical.
What’s next
Lesson 8 covers pipes in depth: the | operator, the PIPESTATUS array, why set -o pipefail is essential for any pipeline you actually care about, the SIGPIPE signal and why head | grep sometimes “works” with weird exit codes, multi-stage pipelines, and |& for piping both streams. Bring everything from lessons 1–7.