Secrets — API keys, passwords, tokens, certificates — are uniquely hostile to shell. Unlike a high-level language where you can keep a secret string in memory and explicitly pass it to a function, every shell command is a process invocation, and secrets pass through:
- Command-line arguments: visible in
ps,/proc/$pid/cmdline, system audit logs. - Environment variables: visible in
/proc/$pid/environto anyone with read permission. - Shell history:
~/.bash_historykeeps every command for years. - Tracing:
set -xprints every command, including secret arguments. - Logging: anything you
echofor “debug” goes to journal, syslog, or stderr files. - Core dumps: if a process crashes, secrets in memory are written to disk.
This is the common cause of secret leaks in DevOps. A senior engineer adds aws s3 cp --secret-key $KEY ... to a script, the script runs in CI, the CI uploads logs to S3, and the secret is now searchable forever. CIs regularly publish this pattern to public artifact buckets.
This lesson covers:
- The six leak channels in detail and how to close each.
- When to use environment variables, when to use files, when to use stdin.
- Integration patterns with Vault, AWS Secrets Manager, GCP, Azure Key Vault.
- Ephemeral credentials via OIDC and short-lived role assumption.
- A reusable
lib/secrets.shthat wraps the common patterns. - How to scan existing codebases for leaked secrets.
By the end, your scripts will handle secrets without leaking them, even when something goes wrong.
1. The six leak channels
1.1 Command-line arguments — visible in ps
$ aws --secret-access-key AKIAIOSFODNN7EXAMPLE s3 ls
While that command runs, anyone on the system can see it:
$ ps -ef | grep aws
ubuntu 1234 ... aws --secret-access-key AKIAIOSFODNN7EXAMPLE s3 ls
ps reads /proc/$pid/cmdline, which is world-readable on Linux (mode 444 by default). Even unprivileged users on the same host can see it. On shared CI runners, multi-tenant containers, jump hosts — this is a leak.
Fix: never pass secrets as command-line arguments. Use environment variables, files, or stdin (next sections).
1.2 Environment variables — visible in /proc/$pid/environ
Environment variables are less leaky than argv but still readable:
$ AWS_SECRET_ACCESS_KEY=foo aws s3 ls
# In another terminal:
$ cat /proc/$pid/environ | tr '\0' '\n' | grep AWS_SECRET
AWS_SECRET_ACCESS_KEY=foo
By default, /proc/$pid/environ is mode 400 — only the process owner can read it. So same-user processes can read each other’s env, but cross-user reads require root.
This is “good enough” for most cases, but be aware:
- If your script runs as root and forks to other users, those children inherit env.
- If you’re in a multi-user container, separate users still see other users’ env via
/proconly if they have CAP_DAC_READ_SEARCH or are root. - Setuid binaries clear most env vars at exec time anyway.
For most production: env vars are the standard secret transport for CLI tools (AWS, GCP, Azure all use them).
1.3 Shell history — ~/.bash_history
$ MY_TOKEN=secret ./deploy.sh
$ history | grep TOKEN
1234 MY_TOKEN=secret ./deploy.sh
Bash records every command. Years later, a colleague greps your history, finds the token. Or a backup of ~/.bash_history ends up somewhere it shouldn’t.
Fixes:
- Prefix command with a space: bash with
HISTCONTROL=ignorespaceskips space-prefixed lines from history.$ ` MY_TOKEN=secret ./deploy.sh` # leading space! - Set
HISTCONTROL=ignorebothandHISTIGNORE=*KEY*:*TOKEN*:*PASSWORD*:*SECRET*in your~/.bashrcto ignore lines matching those substrings. - Use a secret-injection wrapper so the secret never appears at the command line literally.
1.4 Tracing — set -x
#!/usr/bin/env bash
set -Eeuxo pipefail # ← `x` is the killer
KEY="$AWS_SECRET_ACCESS_KEY"
aws --secret-access-key "$KEY" s3 ls
set -x prints every command before execution, with all variables expanded:
+ KEY=AKIAIOSFODNN7EXAMPLE
+ aws --secret-access-key AKIAIOSFODNN7EXAMPLE s3 ls
If your CI captures stderr (which it always does), the secret is now in your build log. Permanent.
Fixes:
- Don’t use
-xin production scripts. It’s a debug feature, not a logging feature. - Mask secret variables explicitly in tracing: bash 5.1+ supports
BASH_XTRACEFDfor redirecting trace output, plus${variables_to_mask[*]}inPS4for selective output. - Disable -x temporarily for sensitive operations:
set +x # Disable trace aws --secret-access-key "$KEY" ... # Sensitive call set -x # Re-enable
1.5 Logging and echo
echo "DEBUG: using token=$MY_TOKEN" >&2
That goes to stderr. In production, stderr goes to journald, syslog, or a log file. Now your secret is in /var/log/syslog and shipped to your central logging system. Indexed. Searchable.
Rule: never log secrets. Even in debug mode. Even temporarily. The “I’ll remove the debug line later” pattern fails 100% of the time.
If you must log that an operation happened, log it without the secret value:
echo "Authenticating with token (length=${#MY_TOKEN})" >&2
${#var} is the length — useful for debugging “is the token even set?” without revealing its value.
1.6 Core dumps
If your script forks a binary (e.g. python, node) that crashes, the kernel writes a core dump containing memory contents — including secrets that were in the address space. Core dumps land in /var/lib/systemd/coredump/ or wherever core_pattern points.
Fix:
- Disable core dumps for security-sensitive processes:
ulimit -c 0 - System-wide: set
RLIMIT_CORE = 0viaLimitCore=0in systemd units.
For most scripts this is overkill. For privileged or secret-handling daemons, set it as a defensive baseline.
2. The four ways to pass secrets — pick one
2.1 Environment variable (most common)
# Caller:
export AWS_SECRET_ACCESS_KEY=$(get_secret aws/s3-key)
# Script:
[[ -n "${AWS_SECRET_ACCESS_KEY:-}" ]] || die "AWS_SECRET_ACCESS_KEY required"
aws s3 ls # aws CLI reads from env automatically
Pros:
- Standard for all major cloud CLIs.
- Not visible in
ps. - Can be set by parent process (CI, systemd) without ever appearing in script source.
Cons:
- Visible in
/proc/$pid/environto same-user processes. - Inherited by all child processes unless explicitly cleared.
- Lifetime is “until process exits.”
2.2 File (best for keys, certs, multi-line secrets)
# Caller writes the secret to a file with mode 600:
get_secret aws/s3-key > /tmp/aws-key.tmp.$$
chmod 600 /tmp/aws-key.tmp.$$
# Script reads:
KEY=$(< /tmp/aws-key.tmp.$$)
# Always clean up:
trap 'rm -f /tmp/aws-key.tmp.$$' EXIT
Pros:
- Not visible in
psor environ. - Permissions can restrict to owner.
- Suitable for multi-line secrets (PEM keys, JSON service account credentials).
- Persistent enough that multiple processes can read the same file (e.g. a daemon).
Cons:
- Requires careful permissions management.
- File could be backed up by accident.
- Race conditions on creation if not done with
mktemp.
For SSH keys, TLS certs, GCP service account JSON, always use file mode.
2.3 Stdin (best for one-shot operations)
# Pass secret via stdin to a tool that supports reading it:
echo "$PASSWORD" | sudo -S cmd
get_secret db/admin | psql --no-password -h "$DB_HOST" -U admin -d mydb -f schema.sql
Pros:
- Not visible in
ps, environ, or files (transient). - No cleanup needed.
- Tool-specific (must support stdin reading).
Cons:
- Only works for tools that read secrets from stdin (
sudo -S,gpg,mysql --password=, etc. — many don’t). - Can’t be used if the tool also reads other input from stdin.
2.4 Argument (the worst — avoid)
# DON'T:
mysql -u admin -psupersecret mydb
The password is in ps for the duration of the connection. Never do this. Most CLIs that accept -p PASSWORD also accept -p (no value, prompts) or have a --password-file=FILE form. Use those.
2.5 Decision matrix
| Use case | Best transport |
|---|---|
| Cloud CLI (aws, gcloud, az) | Env var (their default) |
| TLS cert / private key | File with 0600 |
| Multi-line JSON service account | File with 0600 |
| Database connection string with password | .pgpass file or env var |
| One-off admin command via sudo | Stdin |
| Container orchestration (Docker/K8s) | Mounted secret file (volume) |
| systemd-managed service | EnvironmentFile= (dropped after read) or LoadCredential= |
3. Vault, Secrets Manager, Key Vault — fetching secrets at runtime
The principle: secrets are not in code, not in env at deploy time. They’re fetched at runtime from a vault, used briefly, and discarded.
3.1 HashiCorp Vault
# Authenticate (assuming approle):
VAULT_TOKEN=$(vault write -field=token auth/approle/login \
role_id="$ROLE_ID" secret_id="$SECRET_ID")
export VAULT_TOKEN
# Fetch a secret:
SECRET=$(vault kv get -field=password secret/myapp/db)
# Use it briefly:
PGPASSWORD="$SECRET" psql -h db.example.com -U myapp -c "SELECT 1"
# Clear it:
unset SECRET PGPASSWORD VAULT_TOKEN
vault reads VAULT_ADDR and VAULT_TOKEN from env. The -field=password flag makes it print just the value, not formatted output — easy to capture into a variable.
For service identity, you bootstrap with AppRole (role_id + secret_id, similar to OIDC client credentials) or with Kubernetes auth method (the pod’s service account token authenticates to Vault). The secret_id can be short-lived and machine-specific, dramatically limiting blast radius.
3.2 AWS Secrets Manager
# Fetch with awscli:
SECRET=$(aws secretsmanager get-secret-value \
--secret-id myapp/db/password \
--query SecretString \
--output text)
# Or with jq if it's structured JSON:
RAW=$(aws secretsmanager get-secret-value --secret-id myapp/db --query SecretString --output text)
USER=$(echo "$RAW" | jq -r .username)
PASS=$(echo "$RAW" | jq -r .password)
For credentials to call AWS itself, use IAM roles attached to the EC2 instance / Lambda / ECS task — never embed AWS credentials in scripts. Secrets Manager is for secrets your app needs (database passwords, third-party API keys), not AWS credentials.
3.3 GCP Secret Manager
SECRET=$(gcloud secrets versions access latest \
--secret=myapp-db-password \
--project=my-project)
gcloud authenticates from the metadata server (when running on GCE/GKE/Cloud Run) or from ~/.config/gcloud (when on a developer machine). No secrets in scripts.
3.4 Azure Key Vault
SECRET=$(az keyvault secret show \
--vault-name my-vault \
--name myapp-db-password \
--query value \
--output tsv)
Same pattern: managed identity authenticates az, no secrets in scripts.
3.5 The reusable lib/secrets.sh
# lib/secrets.sh — drop-in secret helpers
# Source from any script. Usage: secret=$(get_secret aws|gcp|az|vault PATH)
get_secret_aws() {
aws secretsmanager get-secret-value \
--secret-id "$1" \
--query SecretString --output text
}
get_secret_gcp() {
gcloud secrets versions access latest --secret="$1"
}
get_secret_az() {
local vault=${VAULT_NAME:?VAULT_NAME required}
az keyvault secret show --vault-name "$vault" --name "$1" --query value --output tsv
}
get_secret_vault() {
local field=${SECRET_FIELD:-value}
vault kv get -field="$field" "$1"
}
# Generic dispatch — picks backend from SECRET_BACKEND env var:
get_secret() {
local path=$1
case ${SECRET_BACKEND:-aws} in
aws) get_secret_aws "$path" ;;
gcp) get_secret_gcp "$path" ;;
az) get_secret_az "$path" ;;
vault) get_secret_vault "$path" ;;
*) echo "Unknown SECRET_BACKEND: ${SECRET_BACKEND}" >&2; return 1 ;;
esac
}
Usage:
source /usr/local/lib/myapp/secrets.sh
DB_PASSWORD=$(get_secret myapp/db/password)
PGPASSWORD="$DB_PASSWORD" psql ...
unset DB_PASSWORD PGPASSWORD
Same script works on AWS, GCP, Azure, or Vault — pick by setting SECRET_BACKEND in the environment.
4. Ephemeral credentials — short-lived is safer than long-lived
The pattern: credentials live for minutes, not weeks. If they leak, blast radius is naturally limited by their TTL.
4.1 AWS STS — assume-role for short-lived credentials
# Get 15-minute credentials by assuming a role:
CREDS=$(aws sts assume-role \
--role-arn arn:aws:iam::123456789012:role/MyAppRole \
--role-session-name "deploy-$(date +%s)" \
--duration-seconds 900)
export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | jq -r .Credentials.AccessKeyId)
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | jq -r .Credentials.SecretAccessKey)
export AWS_SESSION_TOKEN=$(echo "$CREDS" | jq -r .Credentials.SessionToken)
# Use them. After 15 minutes they expire automatically.
aws s3 ls
If those credentials leak, they’re useless after 15 minutes. The TTL is the safety net.
4.2 OIDC federation — no static credentials anywhere
GitHub Actions and most modern CI systems support OIDC: the CI runner gets a short-lived JWT token from the IDP, exchanges it for cloud credentials with no static secrets stored anywhere.
# .github/workflows/deploy.yml
permissions:
id-token: write # Required for OIDC
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/GitHubActions
aws-region: us-east-1
# Now `aws` is configured with short-lived creds, no AWS_SECRET_ACCESS_KEY needed.
- run: aws s3 ls
The OIDC trust policy on the IAM role specifies which GitHub repo and branch can assume it. No static credentials anywhere in the repo, in GitHub, or in CI. Compromise of the GitHub repo gives an attacker code-edit access but no cloud credentials.
This is the modern best practice. If you’re still using long-lived AWS access keys in GitHub secrets, migrate to OIDC.
4.3 Kubernetes service accounts
In Kubernetes, pods carry their service account’s token via a mounted file (/var/run/secrets/kubernetes.io/serviceaccount/token). With the IRSA pattern (AWS) or workload identity (GCP/Azure), these tokens federate to cloud credentials — same OIDC mechanism, automatic.
# In a pod with IRSA configured:
aws s3 ls # Just works. AWS SDK reads the K8s token, exchanges for IAM creds.
No secrets in the script, no secrets in the pod spec. The CSI driver provides the token; AWS SDK does the federation.
5. The “no_log” discipline
Borrowed from Ansible: mark sensitive operations as no-log, suppress all output, and audit the script for accidental exposure.
5.1 The pattern
# A wrapper that ensures the inner command's output and trace are silenced:
no_log() {
local saved_xtrace=""
case $- in *x*) saved_xtrace=on; set +x ;; esac
"$@"
local rc=$?
[[ $saved_xtrace == on ]] && set -x
return $rc
}
# Use it for sensitive operations:
no_log mysql -u admin -p"$PASSWORD" -e "SELECT 1"
It saves whether set -x is currently on, disables it during the command, restores afterwards. The command’s stderr/stdout is unchanged; only the trace is suppressed.
5.2 Suppressing output entirely
For commands whose output might leak secrets:
# Run this command and discard all output:
no_log_quiet() {
no_log "$@" >/dev/null 2>&1
}
no_log_quiet mysql -u admin -p"$PASSWORD" -e "DROP DATABASE temp_data"
Drops both the trace and the output. Only the exit code is observable.
5.3 Audit the script for leaks
# In CI:
grep -nE '(echo|printf).*\$.*(PASSWORD|TOKEN|SECRET|KEY)' bin/* lib/*.sh
Catches lines like echo "Using token: $TOKEN" that would leak. Add to your pre-commit hook or CI lint.
5.4 The set +x zone discipline
For long sections that touch secrets:
set +x # Disable trace
{
PASSWORD=$(get_secret db/admin)
PGPASSWORD="$PASSWORD" psql ...
unset PASSWORD PGPASSWORD
} >/dev/null 2>&1 # And suppress stdout/stderr
set -x # Re-enable
# Continue with normal operations.
Wrapping in braces creates a logical zone; the > /dev/null 2>&1 is for the output of the commands, not just the trace.
6. Container secrets — the right way
6.1 Docker build-time vs run-time
The biggest mistake: baking secrets into images.
# DON'T:
FROM ubuntu:22.04
ENV DB_PASSWORD=supersecret
RUN apt-get install -y mypackage
That DB_PASSWORD is now in the image forever, in a layer, recoverable by anyone who has the image. Every push to a public registry is a leak.
The fix: secrets are runtime-only, never image-time.
# OK — image is generic, accepts secrets at run time:
FROM ubuntu:22.04
RUN apt-get install -y mypackage
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
# Run with secret via env (one-time):
docker run --env-file <(get_secret myapp/env) myimage
6.2 Docker BuildKit secrets
If you genuinely need a secret during build (to download a private artifact), use BuildKit secret mounts:
# syntax=docker/dockerfile:1.4
FROM ubuntu:22.04
RUN --mount=type=secret,id=npmtoken \
NPM_TOKEN=$(cat /run/secrets/npmtoken) && \
npm install --registry=https://my-private-npm
DOCKER_BUILDKIT=1 docker build --secret id=npmtoken,src=$HOME/.npmrc -t myimage .
The secret is mounted as a tmpfs file during the RUN, never written to a layer. After the build, it’s gone.
6.3 Kubernetes secrets
Mount as files (preferred over env):
volumes:
- name: db-credentials
secret:
secretName: db-credentials
containers:
- name: app
volumeMounts:
- name: db-credentials
mountPath: /var/run/secrets/db
readOnly: true
Then in your container script:
DB_PASSWORD=$(< /var/run/secrets/db/password)
Files are visible only inside the pod, only to the container’s user, with proper mode. Env-var secrets in K8s are visible in pod spec (often readable), so prefer file mounts.
6.4 Sealed secrets / SOPS for git-stored secrets
If you must put encrypted secrets in git (gitops pattern), use SOPS:
# Encrypt:
sops -e --aws-kms arn:aws:kms:... secrets.yaml > secrets.enc.yaml
# Decrypt at runtime (in a script that already has KMS access):
sops -d secrets.enc.yaml > secrets.yaml
The encryption key (KMS, age, gpg) is the real secret; SOPS handles the encrypted content. With KMS, only your IAM principal can decrypt — even with the encrypted file, an attacker can’t read the secret without your IAM credentials.
7. Auditing existing scripts for leaked secrets
7.1 The grep-able patterns
For any codebase, these regexes catch the common leaks:
# AWS access key:
grep -rnE 'AKIA[0-9A-Z]{16}' .
# AWS secret key (40 chars base64-ish):
grep -rnE '[A-Za-z0-9/+=]{40}' . | head # Lots of false positives; review.
# GitHub PAT:
grep -rnE '(ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{36}' .
# Generic API tokens:
grep -rinE '(api[_-]?key|secret|password|token)\s*=\s*["'\'']?[A-Za-z0-9_-]{16,}' .
# Private keys:
grep -rln 'BEGIN .* PRIVATE KEY' .
# .env files:
find . -name '.env' -o -name '.env.*' | grep -v .gitignore
7.2 Tools
- gitleaks — scans repo history for secrets.
gitleaks detect --source .runs in seconds. - trufflehog — scans git history, files, and S3. Extensive ruleset.
- detect-secrets — Yelp’s tool, lower false-positive rate.
In CI:
- uses: gitleaks/gitleaks-action@v2
with:
config-path: .gitleaks.toml
This blocks any PR that introduces a secret.
7.3 If a secret leaked — what to do
- Rotate immediately. Generate a new credential, update consumers, revoke the old one. Speed matters.
- Audit usage logs for the leaked credential. AWS CloudTrail, GitHub audit log, etc. — when was it used, by whom, from where?
- Purge from history, but accept that anyone with prior
git clonestill has it. Rotation is the only real fix. - Postmortem: how did it get committed? Was the pre-commit hook missing? Was a
.envfile not in.gitignore?
The git filter-branch / git filter-repo route is for “I want this gone from history” — but if the repo has been pushed and seen, it’s already cached, scraped by bots, in CI artifacts. Rotate, don’t try to redact.
8. The reusable patterns
8.1 The strict-mode preamble for secret-handling scripts
#!/usr/bin/env bash
set -Eeuo pipefail -f
IFS=$'\n\t'
# Pin environment:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH
TZ=UTC LC_ALL=C
export TZ LC_ALL
# Disable core dumps for safety:
ulimit -c 0
# Disable shell history:
export HISTFILE=/dev/null
unset HISTSIZE
unset HISTFILESIZE
# Source the secret helpers:
source /usr/local/lib/myapp/secrets.sh
# At the END of the script (or via trap), make sure secrets are unset:
cleanup() {
unset DB_PASSWORD AWS_SECRET_ACCESS_KEY API_TOKEN
}
trap cleanup EXIT
8.2 The “credential lifetime” pattern
Always scope secrets to the smallest block possible:
do_db_thing() (
# Subshell: secret only exists in this subshell, no leak to caller.
PGPASSWORD=$(get_secret db/admin)
export PGPASSWORD
psql -h db.example.com -U admin "$@"
)
# Caller doesn't see PGPASSWORD:
do_db_thing -c "SELECT 1"
do_db_thing -f /path/to/migration.sql
The subshell ( … ) makes the variable local. After the function returns, the secret is garbage-collected.
8.3 The “no static credentials” pattern
# Top of script — assert that we're using ephemeral creds:
require_ephemeral_credentials() {
# AWS-specific:
if [[ -n "${AWS_SESSION_TOKEN:-}" ]]; then
return 0 # Has session token = ephemeral.
fi
if [[ -f ~/.aws/credentials ]]; then
if grep -q 'aws_session_token' ~/.aws/credentials; then
return 0
fi
fi
if [[ -n "${AWS_WEB_IDENTITY_TOKEN_FILE:-}" ]]; then
return 0 # OIDC.
fi
echo "ERROR: this script requires ephemeral AWS credentials." >&2
echo "Use OIDC (gh-actions), assume-role (sts), or instance profile." >&2
exit 1
}
require_ephemeral_credentials
This refuses to run with long-lived static credentials. Forces the operator to use the safer mode.
8.4 The “secret was used, prove it works” pattern
After fetching, verify before using:
DB_PASSWORD=$(get_secret myapp/db)
# Verify the password actually authenticates before doing anything destructive:
if ! PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -U myapp -c '\q' >/dev/null 2>&1; then
echo "Authentication failed. Aborting." >&2
exit 1
fi
# Now safe to do the real work:
PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -U myapp -f migration.sql
This catches “secret was rotated, my cache is stale” before you’ve already deleted half the database.
9. The full secrets-aware script template
#!/usr/bin/env bash
# myscript - description
# Handles secrets safely. See SECRETS.md for the discipline.
set -Eeuo pipefail -f
IFS=$'\n\t'
# Hardening:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH
TZ=UTC LC_ALL=C
export TZ LC_ALL
ulimit -c 0
export HISTFILE=/dev/null
unset BASH_ENV ENV CDPATH GLOBIGNORE LD_PRELOAD LD_LIBRARY_PATH
# Helpers:
source /usr/local/lib/myapp/secrets.sh
# Cleanup on exit:
cleanup() {
# Unset all secret variables:
unset -v DB_PASSWORD API_TOKEN PGPASSWORD AWS_SECRET_ACCESS_KEY
# Remove temp files:
[[ -n "${TMPDIR:-}" ]] && rm -rf -- "$TMPDIR"
}
trap cleanup EXIT INT TERM
TMPDIR=$(mktemp -d -t myscript.XXXXXX)
chmod 700 "$TMPDIR"
# Main logic:
DB_PASSWORD=$(get_secret myapp/db/password)
[[ -n "$DB_PASSWORD" ]] || { echo "Failed to fetch DB password" >&2; exit 1; }
# Verify before destructive use:
if ! PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -U myapp -c '\q' >/dev/null 2>&1; then
echo "Authentication failed. Aborting." >&2
exit 1
fi
# Use briefly, in a subshell so the secret doesn't escape:
(
export PGPASSWORD="$DB_PASSWORD"
psql -h "$DB_HOST" -U myapp -f /usr/share/myapp/migration.sql
)
# Cleanup runs automatically via trap.
This is the pattern for any script that touches secrets. Copy it as a starting point.
10. Quick reference card
The six leak channels
1. argv — visible in ps. NEVER pass secrets as args.
2. environ — visible in /proc/$pid/environ to same user.
3. history — ~/.bash_history. Use HISTCONTROL=ignorespace.
4. xtrace — set -x prints expanded vars. Wrap secrets in set +x ... set +x.
5. logging — echo/printf to stderr. NEVER log a secret value.
6. core dumps — ulimit -c 0. Or LimitCore=0 in systemd.
Pick the transport
Cloud CLI → env var (their default)
TLS / SSH key → file with 0600
Multi-line JSON → file with 0600
SQL connection → .pgpass / env var
sudo password → stdin (sudo -S)
Container → mounted secret file
systemd service → LoadCredential= or EnvironmentFile=
Vault dispatch
SECRET=$(get_secret myapp/db/password) # generic
# Set SECRET_BACKEND=aws|gcp|az|vault to choose.
Ephemeral creds
# AWS STS:
aws sts assume-role --role-arn ... --duration-seconds 900
# OIDC in CI: configure-aws-credentials@v4 with role-to-assume
# K8s: IRSA / workload identity — automatic
no_log wrapper
no_log() {
case $- in *x*) set +x; "$@"; rc=$?; set -x; return $rc ;; esac
"$@"
}
no_log psql -p "$PASSWORD" -e "..."
The 7 commandments of secrets
- Never in argv. Use env, file, or stdin.
- Never logged. Even in debug. Even temporarily.
- Never in source/git. Fetch at runtime from a vault.
- Ephemeral lifetime. STS assume-role, OIDC, ≤ 1 hour TTL.
unseton exit. Trap-based cleanup.- Permissions 0600 on secret files.
chmodimmediately after creation. - Audit with gitleaks/trufflehog in CI on every push.
11. Wrap-up
Secrets are the highest-stakes data your script ever handles. A leaked password, key, or token can cost millions, take down a service, or end careers. The good news: the discipline is mechanical:
- Don’t pass secrets as args (
psleak). - Don’t echo secrets (log leak).
- Don’t
set -xover secret-touching code (trace leak). - Don’t leave secrets in env after use (
unset). - Don’t bake secrets into images (build-time leak).
- Don’t store secrets in git (history leak).
Layer on:
- Vault-fetched, runtime-only, ephemeral credentials.
- File transports for multi-line secrets, with 0600 perms.
- Subshells / functions to scope variable lifetime.
- CI scanners (gitleaks, trufflehog) to catch regressions.
Get those right and the most common shell-secret leak class — accidental exposure — disappears. The rare hard cases (memory dumps, side-channel attacks) are real, but most leaks are these mundane ones, and most are avoidable with mechanical discipline.
Next: L27 — idempotency. We’ll cover state files, reconciliation loops, and dry-run flags — the patterns that turn “run once or break” scripts into “run any time, end up in the right state.”