Running Containers on Linux: Podman, Docker, Buildah, Rootless Containers & Quadlet/systemd

In the previous lesson you learned that a container is not a thing the kernel knows about — it is a normal process that the kernel has been told to lie to, using namespaces (its own view of PIDs, mounts, network, users) and cgroups (its slice of CPU, memory, I/O). If that mental model is fuzzy, go back to Namespaces, cgroups & containers under the hood first — everything below assumes it.

This lesson is the layer you actually type at. Nobody assembles containers by hand with unshare and clone() in production; they run podman run or docker run, and a stack of tooling turns an image and a one-line command into a correctly-namespaced, cgroup-limited, network-attached, restart-on-failure process. Our job here is to make that stack stop being magic. We will use Podman as the primary tool — it is daemonless, rootless-first, ships as the default on RHEL/Fedora, and its CLI is a drop-in for Docker’s — while showing the Docker equivalent at every step because you will meet Docker constantly.

Why this matters

Here is the problem containers solve, stated plainly: “it works on my machine” is a lie about the environment, not the code. Your app depends on a specific Python, a specific glibc, three shared libraries, two environment variables, and a directory that exists on your laptop and not on the server. A container image freezes that entire userland — every file below your app — into one immutable, content-addressed artifact that runs bit-identically on your laptop, in CI, and in production. The kernel is shared; everything above it is shipped with the app.

The second thing that matters is who runs the container. For a decade the default was Docker, whose architecture is a single root daemon (dockerd) that every user talks to through a socket. That daemon runs as root, owns every container as its own child, and — because its socket is effectively a root shell — makes membership in the docker group equivalent to sudo with no password and no audit trail. That was a security compromise everyone quietly accepted. Podman’s bet is that you don’t have to: it runs daemonless (no background service) and rootless (your unprivileged user, mapped through a user namespace, is “root” only inside the container). If a rootless container is compromised, the attacker lands as your uid 1000 on the host, owning nothing.

The third thing — and the reason this lesson ties into the rest of the course — is lifecycle. A container you started by hand dies when you close your laptop. A production service must start on boot, restart on failure, log to the journal, and be managed like every other service on the box. On Linux that means systemd, and Podman integrates with systemd through Quadlet so cleanly that a container becomes just another unit file. If you already understand systemd units, services & targets, Quadlet will feel like coming home.

Get these three ideas — immutable userland, rootless isolation, systemd lifecycle — and you can run a real fleet. Let’s build up to it.

The container ecosystem and the OCI standards

The reason podman run docker.io/library/nginx works, and the reason an image Docker built runs under Podman and then under Kubernetes without modification, is a set of open specifications governed by the Open Container Initiative (OCI). The OCI is not software; it is three text specs that everyone implements. Understanding them dissolves most “why are there so many tools?” confusion.

Below the CLI you type sits a two-layer stack. The tool you invoke (Podman, Docker, or Kubernetes’ kubelet) is a high-level manager: it resolves image names, pulls layers, prepares storage and networking, and writes an OCI config.json. It then calls a low-level OCI runtime — a small program whose entire job is the runtime-spec: create the namespaces, apply the cgroup, drop capabilities, set up the mounts, and exec your process. That is where runc and crun live.

You can see and switch the runtime directly:

# Which runtimes does this Podman know about, and which is default?
podman info --format '{{.Host.OCIRuntime.Name}}'
# crun

# Force a specific low-level runtime for one container
podman run --runtime crun --rm docker.io/library/alpine echo hi
# hi

The practical payoff: because everyone targets the same runtime-spec, an image is a portable, tool-agnostic artifact. You are never locked into the tool that built or ran it. That is the whole point of the OCI, and it is why the rest of this lesson can teach Podman without stranding you if your shop uses Docker or ships to Kubernetes.

Podman vs Docker: daemonless and rootless by design

Podman (pod + manager) and Docker do the same job and share almost the same CLI, but their architecture differs in two ways that matter enormously in production: daemon vs daemonless, and root vs rootless by default.

Docker is a client/server system. The docker command is a thin client that sends every request over a Unix socket (/var/run/docker.sock) to dockerd, a long-running daemon that runs as root and is the parent of every container. Podman has no daemon: when you run podman run, podman itself forks and execs the OCI runtime as you, then largely gets out of the way — a tiny C monitor called conmon stays behind per container to hold the TTY and report the exit code. Because your containers were never children of a daemon, there is no daemon to restart, crash, or attack, and containers keep running across a Podman upgrade.

The diagram below traces the full rootless Podman flow — from your unprivileged shell, through the user-namespace id mapping and the image pull, into the crun/runc runtime, and finally under systemd via Quadlet — with the Docker daemon model shown as the red contrast node.

Read it left to right: daemonless entry (badge 1), the rootless user-namespace id map from /etc/subuid (badge 2), the image pulled from a registry (badge 3), the OCI runtime crun/runc doing the low-level work (badge 4), the container promoted to a Quadlet systemd service (badge 5), and the Docker daemon contrast (badge 6). Here is the same contrast as a reference table.

Because the CLIs match, the single most useful fact for a Docker user moving to Podman is:

# Everything you know still works — just swap the binary.
alias docker=podman
docker run --rm docker.io/library/hello-world   # actually runs podman

A frequent point of confusion is Docker Engine vs Docker Desktop — they are not the same product.

The takeaway: on a Linux server, “install Docker” means Docker Engine (a root daemon). Podman gives you the same commands without the daemon and without root. For the rest of this lesson every podman command has a docker twin — where they differ, we will call it out.

Images: registries, layers, tags and digests

An image is the frozen userland; a container is a running (or stopped) instance of an image with a thin writable layer on top. You get images by pulling them from a registry — an HTTPS server that speaks the OCI distribution spec.

Pulling, and the short-name problem

# Fully-qualified: registry / namespace / repository : tag
podman pull docker.io/library/nginx:1.27-alpine
# Trying to pull docker.io/library/nginx:1.27-alpine...
# Getting image source signatures
# Copying blob sha256:...  done
# Writing manifest to image destination
# 9c... (image ID)

Docker silently assumes docker.io/library/ in front of a bare name like nginx. Podman does not guess by default — it consults unqualified-search-registries and, in enforcing mode, either resolves a known short-name alias or asks you which registry you meant. This is a security feature: it stops a typo or a malicious short name from silently pulling evil.io/nginx.

Registry behaviour is configured in registries.conf (system-wide /etc/containers/registries.conf, per-user ~/.config/containers/registries.conf), a TOML file:

# See exactly which registries a bare name would search, and the aliases in effect
podman info --format '{{.Registries}}'
cat /etc/containers/registries.conf.d/000-shortnames.conf | head
# [aliases]
#   "nginx" = "docker.io/library/nginx"
#   "fedora" = "registry.fedoraproject.org/fedora"

Tags, digests, and why :latest will burn you

An image reference has the shape registry/namespace/repo:tag or registry/namespace/repo@sha256:digest.

# Find the immutable digest of an image you trust, then pin to it
podman image inspect nginx:1.27-alpine --format '{{index .RepoDigests 0}}'
# docker.io/library/nginx@sha256:e4b0b2...

# Deploy by digest — this can never silently change
podman pull docker.io/library/nginx@sha256:e4b0b2...

⚠️ Pinning to :latest in a Quadlet unit or Compose file is the classic self-inflicted outage: someone repushes latest, your host restarts, it pulls a new incompatible image, and the service that worked yesterday won’t start today. Pin by digest for anything you can’t afford to have change.

Layers and the overlay filesystem

Each image is an ordered stack of read-only layers, each a tarball of filesystem changes, content-addressed by digest and shared between images. At run time the container gets a thin writable upper layer; the kernel’s overlayfs presents the union as one filesystem.

The operational consequences: layers are shared, so ten containers from one image cost roughly one image’s disk plus ten small writable layers; and anything written inside a running container lives only in its ephemeral upper layer and vanishes when the container is removed — which is exactly why volumes exist (later).

# List images and their real sizes; inspect config and layer history
podman images
# REPOSITORY                 TAG         IMAGE ID      CREATED       SIZE
# docker.io/library/nginx    1.27-alpine 9c7a54a9a... 2 weeks ago   52.1 MB

podman history docker.io/library/nginx:1.27-alpine   # layer-by-layer provenance
podman image inspect nginx:1.27-alpine               # full JSON: env, entrypoint, layers, digest

Pruning reclaims space — and is where people accidentally nuke things:

podman image prune                 # remove dangling (untagged) images — safe
podman image prune -a              # ⚠️ remove ALL images not used by a container
podman system df                   # show reclaimable image/container/volume space
podman system prune -a --volumes   # ⚠️ removes stopped containers, unused images, networks AND volumes — data loss

Running containers: podman run and the lifecycle

podman run (and identical docker run) is the command you’ll type most. It creates a container from an image and starts its process. Learn its flags well; almost every production behaviour is one flag.

# A representative production-style run
podman run -d --name web \
  -p 8080:80 \
  -v web-content:/usr/share/nginx/html:ro \
  -e NGINX_ENTRYPOINT_QUIET_LOGS=1 \
  --restart on-failure:5 \
  --memory 256m --cpus 1.5 \
  --read-only --tmpfs /var/cache/nginx --tmpfs /run \
  --cap-drop ALL --cap-add NET_BIND_SERVICE \
  docker.io/library/nginx:1.27-alpine

Once containers exist, you manage their lifecycle:

Exit codes tell you what happened

A container’s exit code is the exit code of its main process — with a few reserved values the engine adds. Reading them saves hours.

# The exit code of the last-run container
podman inspect web --format '{{.State.ExitCode}}'
# 137   ← was OOM-killed; raise --memory or fix the leak

The --restart policies deserve a table because their reboot behaviour surprises everyone:

⚠️ A crucial subtlety: --restart always does not by itself make a container come back after a reboot when rootless — the policy is enforced by the (absent) daemon-equivalent, not by systemd. For real boot persistence you use Quadlet (covered later), or systemctl --user enable podman-restart.service plus loginctl enable-linger. This is the number-one “my container didn’t come back after reboot” gotcha.

Rootless containers: the user-namespace mapping

This is Podman’s headline feature and the thing worth understanding deeply. Rootless means an ordinary user — no sudo, not in any privileged group — runs full containers, and processes that believe they are root inside are really that unprivileged user on the host.

The mechanism is the user namespace (from the previous lesson). When you, uid 1000, run a rootless container, Podman creates a user namespace in which:

• Your host uid 1000 is mapped to uid 0 (root) inside the container.
• A pre-allocated range of “subordinate” UIDs is mapped to the other in-container UIDs (1, 2, … up to 65535).

That range comes from two files that must contain an entry for your user:

So inside the container, root (uid 0) == host uid 1000 (you), and container uid 33 (www-data) == host uid 100032. Verify it:

# Inside a rootless container: it thinks it's root
podman run --rm docker.io/library/alpine id
# uid=0(root) gid=0(root) groups=0(root),...

# From the host: enter YOUR user namespace and see the real mapping
podman unshare cat /proc/self/uid_map
#          0       1000          1     ← container uid 0 = host uid 1000
#          1     100000      65536     ← container uid 1..65535 = host 100000..165535

podman unshare runs a command inside your Podman user namespace — indispensable when you need to chown files in a rootless volume to a UID that only exists inside the container:

# A rootless volume owned by container-uid 33 must be chowned in the namespace
podman unshare chown 33:33 ~/.local/share/containers/storage/volumes/data/_data

Why rootless is more secure — and its trade-offs

The security argument is simple and strong: if a rootless container is compromised and the process breaks out, it lands as your unprivileged uid 1000 on the host. It can’t read /etc/shadow, can’t load kernel modules, can’t touch other users’ files. A rootful container breakout, by contrast, lands as host root. Rootless shrinks the blast radius from “own the box” to “own your own files.”

The port-below-1024 caveat is the rootless gotcha you will hit. Unprivileged users can’t bind ports below 1024, so podman run -p 80:80 fails rootless. Three fixes:

# Option A: publish to a high host port and let a reverse proxy / NAT handle 80
podman run -d -p 8080:80 nginx

# Option B: lower the unprivileged port floor system-wide (a deliberate policy choice)
sudo sysctl net.ipv4.ip_unprivileged_port_start=80
# persist in /etc/sysctl.d/99-rootless.conf

# Option C: grant CAP_NET_BIND_SERVICE just to rootless podman (advanced)

Other rootless caveats worth internalising:

Building images: Containerfile, multi-stage and Buildah

You consume images by pulling; you produce them by building. The build recipe is a Containerfile — identical in syntax to a Dockerfile (Podman/Buildah accept either filename; Containerfile is just the vendor-neutral name). Each instruction that changes the filesystem produces a layer.

Here is a real, production-shaped, multi-stage, non-root Containerfile for a small Go service:

# ---- Stage 1: build (has the toolchain; thrown away) ----
FROM docker.io/library/golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download                        # cached unless go.mod/go.sum change
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app ./cmd/server

# ---- Stage 2: runtime (tiny, no toolchain, non-root) ----
FROM docker.io/library/alpine:3.20
RUN adduser -D -u 10001 appuser            # create an unprivileged user
COPY --from=build /out/app /usr/local/bin/app   # copy ONLY the binary from stage 1
ENV APP_ENV=production
EXPOSE 8080
USER appuser                               # ⚠️ never run as root by default
ENTRYPOINT ["/usr/local/bin/app"]
CMD ["--listen=:8080"]

The ENTRYPOINT vs CMD relationship confuses everyone, so pin it down:

Two files/practices govern build quality:

• Layer caching: instructions are cached; a layer is rebuilt only if it or an earlier one changed. Order from least- to most-frequently-changing — copy go.mod/package.json and install deps before copying source, so a code edit doesn’t bust the dependency layer.
• .containerignore (or .dockerignore): excludes files from the build context (secrets, .git, node_modules, build artifacts). Smaller context = faster builds and no accidental secret leakage.

Multi-stage builds are the single biggest image-size and security win:

Building, with Podman or Docker:

# Podman (uses Buildah internally)
podman build -t quay.io/vinod/app:1.0 -f Containerfile .

# Docker (uses BuildKit)
docker build -t quay.io/vinod/app:1.0 .

# Push to a registry (log in first)
podman login quay.io
podman push quay.io/vinod/app:1.0

Buildah: builds without a Dockerfile

Buildah is the tool Podman calls under the hood, but you can use it directly for two things: (1) buildah bud (“build using dockerfile”) is a drop-in for podman build; and (2) its native verbs let you script an image step-by-step with no Dockerfile at all — powerful when you want loops, conditionals, or to inject files from the host at build time.

# 1) Dockerfile-based, same as podman build
buildah bud -t quay.io/vinod/app:1.0 .

# 2) Scriptable, Dockerfile-free build
ctr=$(buildah from docker.io/library/alpine:3.20)   # start from a base, get a working container
buildah run  $ctr -- apk add --no-cache curl        # run commands in it
buildah copy $ctr ./app /usr/local/bin/app          # copy files in
buildah config --entrypoint '["/usr/local/bin/app"]' --user 10001 $ctr
buildah commit $ctr quay.io/vinod/app:1.0           # freeze into an image
buildah rm $ctr                                     # clean up the working container

Storage and networking

Two container facts drive everything here: the writable layer is ephemeral (deleted with the container), and each container gets its own network namespace. So persistence needs volumes, and connectivity needs a network.

Volumes vs bind mounts

There are two ways to give a container persistent or shared storage, and choosing correctly matters.

podman volume create app-data
podman volume ls
podman volume inspect app-data --format '{{.Mountpoint}}'   # real host path
podman run -d -v app-data:/var/lib/postgresql/data docker.io/library/postgres:16

# Bind mount a host config, read-only, with SELinux relabel
podman run -d -v /etc/myapp/config.yml:/app/config.yml:ro,Z docker.io/vinod/app:1.0

⚠️ The :Z / :z SELinux trap. On SELinux systems (RHEL, Fedora, Rocky) a bind-mounted host directory carries a label the container is not allowed to access, so the app inside gets Permission denied even though ls -l shows the right Unix perms. The fix is a mount-option suffix that relabels the content:

• :z (lowercase) — relabel as shared (multiple containers may use it).
• :Z (uppercase) — relabel as private to this one container.

⚠️ Never put :Z on a system directory like /usr or /home — Podman will recursively relabel it and can break the host. Only relabel directories you created for the container. This ties directly into SELinux & mandatory access control.

Container networking

By default each container joins a bridge network with a private IP, and its outbound traffic is NAT’d to the host. Podman 4+ uses netavark (replacing the older CNI plugins) as the network stack, with aardvark-dns giving containers name-based DNS resolution to each other on the same user-defined network.

The DNS point is a frequent stumbling block:

# Create a network → containers on it resolve each other BY NAME via aardvark-dns
podman network create appnet
podman run -d --name db     --network appnet docker.io/library/postgres:16
podman run -d --name web    --network appnet docker.io/vinod/app:1.0
# 'web' can now reach the database at host 'db' — no IPs, no /etc/hosts hacks
podman exec web getent hosts db
# 10.89.0.2   db

For deeper bridge, VLAN and namespace networking on the host itself, see Advanced Linux networking: bridges, VLANs & namespaces — container networking is that machinery applied automatically.

Multi-container: compose and pods

Real apps are several containers — an app, a database, a cache. Two ways to run them together:

Compose — a compose.yaml/docker-compose.yml file declaring services, networks and volumes, driven by docker compose (v2 plugin), podman compose, or the Python podman-compose. Podman can also expose a Docker-compatible socket (podman system service) so the real docker compose drives Podman.

# compose.yaml — two services on an auto-created network with DNS
services:
  web:
    image: quay.io/vinod/app:1.0
    ports: ["8080:8080"]
    depends_on: [db]
    environment: { DB_HOST: db }
  db:
    image: docker.io/library/postgres:16
    volumes: ["db-data:/var/lib/postgresql/data"]
    environment: { POSTGRES_PASSWORD_FILE: /run/secrets/pg }
volumes:
  db-data:

podman compose up -d      # start the whole stack
podman compose ps         # status
podman compose down       # stop & remove (add -v to remove volumes ⚠️)

Pods — Podman’s native, Kubernetes-style grouping. A pod is a set of containers that share a network namespace (and can share more): they see each other on localhost, share published ports, and are managed as a unit. A hidden infra container (the “pause” container) holds the shared namespaces. This is deliberately the same concept as a Kubernetes pod.

podman pod create --name app-pod -p 8080:8080
podman run -d --pod app-pod --name web docker.io/vinod/app:1.0
podman run -d --pod app-pod --name cache docker.io/library/redis:7   # web reaches redis on localhost:6379
podman pod ps

# The bridge to Kubernetes: turn a pod into k8s YAML, or run k8s YAML locally
podman kube generate app-pod > app-pod.yaml
podman kube play app-pod.yaml

That kube generate/play pair is the on-ramp from “containers on one Linux box” to orchestration — when a pod outgrows a single host you move the YAML to a cluster. If that’s where you’re headed, the Kubernetes Zero-to-Hero course picks up exactly here.

Running containers as services: Quadlet and systemd

A container you started by hand is not a service — it won’t come back after a reboot, isn’t supervised, and doesn’t log to the journal. On Linux, “a thing that starts on boot and restarts on failure” is a systemd unit. Podman’s modern answer is Quadlet: you write a declarative .container (or .pod, .network, .volume, .kube, .image, .build) file, and a systemd generator turns it into a real .service at boot.

Place units in one of:

• /etc/containers/systemd/ — rootful, system-wide.
• ~/.config/containers/systemd/ — rootless, per-user.
• /usr/share/containers/systemd/ — shipped by packages.

A minimal, hardened Quadlet unit — ~/.config/containers/systemd/web.container:

[Unit]
Description=Web app (rootless, Quadlet-managed)
After=network-online.target

[Container]
Image=quay.io/vinod/app:1.0
PublishPort=8080:8080
Volume=app-data:/data:Z
Environment=APP_ENV=production
# hardening
User=10001
ReadOnly=true
NoNewPrivileges=true
DropCapability=ALL
AddCapability=NET_BIND_SERVICE

[Service]
Restart=on-failure

[Install]
WantedBy=default.target

Then it behaves like any other unit:

systemctl --user daemon-reload           # the generator turns web.container → web.service
systemctl --user start web.service       # note: the .service name, not .container
systemctl --user status web.service
journalctl --user -u web.service -f      # logs, in the journal, like every service

# ⚠️ For a rootless service to start at BOOT (before you log in), enable linger:
loginctl enable-linger $USER

Common [Container] keys map one-to-one to podman run flags:

Before Quadlet (Podman < 4.4) the tool was podman generate systemd, which emitted a .service file from a running container. It still exists but is deprecated — prefer Quadlet for anything new.

This is the payoff of the whole lesson: a rootless, capability-dropped, read-only container that starts on boot and is managed exactly like sshd or nginx — reviewed in depth in the systemd units & journald lesson.

Container security hardening

Rootless is the foundation, but a container should also run with the least privilege it needs. The defaults are already decent (a restrictive seccomp profile, a dropped capability set, a fresh network namespace), but the hardening baseline goes further.

Capabilities are the granular pieces of “root” you add back selectively:

# See a running container's effective capabilities
podman exec web grep Cap /proc/1/status
# Then compare against a decoded set:
podman run --rm --cap-drop ALL --cap-add NET_BIND_SERVICE alpine \
  sh -c 'apk add -q libcap; capsh --print' 2>/dev/null | grep Current

⚠️ --privileged is the anti-pattern to internalise: it gives the container almost all capabilities, disables SELinux/seccomp confinement, and exposes host devices. If a tutorial tells you to use it, treat that as “I couldn’t be bothered to find the one capability I actually need.” Scanning (Trivy/Grype), signed images, and pinning by digest complete the picture — a hardened container is rootless, non-root inside, cap-dropped, read-only, from a trusted+scanned+digest-pinned image.

Hands-on lab

This lab runs on any Linux with Podman (Fedora/RHEL/Rocky ship it; on Debian/Ubuntu sudo apt install -y podman, on Fedora/RHEL sudo dnf install -y podman). Everything is rootless — no sudo. Each step says what you should see and what just happened.

Step 0 — verify rootless is wired up.

podman info --format '{{.Host.Security.Rootless}}'   # true
grep "^$(id -un):" /etc/subuid /etc/subgid           # you should have a range in each
# /etc/subuid:vinod:100000:65536
# /etc/subgid:vinod:100000:65536

What just happened: confirmed you have subordinate UID/GID ranges — the prerequisite for rootless. If either line is missing, sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER then podman system migrate.

Step 1 — run your first rootless container and prove the mapping.

podman run --rm docker.io/library/alpine id            # uid=0(root) — inside the namespace
podman unshare cat /proc/self/uid_map                  #  0 1000 1  → you ARE container-root

What just happened: the process thinks it’s root; the host maps that to your uid 1000. Isolation with no real privilege.

Step 2 — run a web server, publish a port, hit it.

podman run -d --name web -p 8080:80 docker.io/library/nginx:1.27-alpine
podman ps
curl -s localhost:8080 | head -n1     # <!DOCTYPE html> ... the nginx welcome page

What just happened: a detached, port-published, rootless nginx. Note we used 8080, not 80 — rootless can’t bind low ports.

Step 3 — inspect, log, exec, and read the exit story.

podman logs --tail 3 web
podman exec -it web sh -c 'nginx -v; ls /etc/nginx'   # poke around inside
podman inspect web --format 'IP={{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} State={{.State.Status}}'

What just happened: you observed a running container from the outside (logs, state, IP) and the inside (exec).

Step 4 — persistent storage with a named volume.

podman volume create site
podman run --rm -v site:/data alpine sh -c 'echo "hello from a volume" > /data/index.txt'
podman run --rm -v site:/data alpine cat /data/index.txt   # data survived a container that no longer exists

What just happened: the first container wrote to the volume and was removed; the second still sees the data. Volumes outlive containers.

Step 5 — build your own image (multi-stage, non-root).

mkdir -p ~/lab && cd ~/lab
cat > index.html <<'EOF'
<h1>Built rootless with Podman</h1>
EOF
cat > Containerfile <<'EOF'
FROM docker.io/library/nginx:1.27-alpine
COPY index.html /usr/share/nginx/html/index.html
EOF
echo ".git"  > .containerignore
podman build -t localhost/mysite:1.0 .
podman run -d --name mysite -p 8081:80 localhost/mysite:1.0
curl -s localhost:8081        # <h1>Built rootless with Podman</h1>

What just happened: you authored a Containerfile, built an image with Buildah (via podman build), and ran it.

Step 6 — two containers talking by name over a custom network.

podman network create labnet
podman run -d --name api   --network labnet docker.io/library/nginx:1.27-alpine
podman run -d --name probe --network labnet docker.io/library/alpine sleep 600
podman exec probe getent hosts api        # 10.89.x.x  api  ← DNS by container name (aardvark-dns)

What just happened: on a user-created network, containers resolve each other by name automatically — the default network wouldn’t.

Step 7 — promote a container to a boot-persistent systemd service with Quadlet.

mkdir -p ~/.config/containers/systemd
cat > ~/.config/containers/systemd/mysite.container <<'EOF'
[Unit]
Description=Lab site via Quadlet
[Container]
Image=localhost/mysite:1.0
PublishPort=8082:80
[Service]
Restart=on-failure
[Install]
WantedBy=default.target
EOF
systemctl --user daemon-reload
systemctl --user start mysite.service
systemctl --user status mysite.service --no-pager | head -n4
curl -s localhost:8082       # your page, now served by a systemd-managed container
loginctl enable-linger "$USER"   # so it starts on boot even before you log in

What just happened: a declarative .container file became a real systemd service — supervised, journald-logged, boot-persistent. This is production shape.

Step 8 — clean up.

systemctl --user stop mysite.service
rm ~/.config/containers/systemd/mysite.container && systemctl --user daemon-reload
podman rm -f web mysite api probe
podman network rm labnet
podman volume rm site
# ⚠️ optional deep clean — removes ALL unused images/containers/networks/volumes:
# podman system prune -a --volumes

What just happened: you tore down every artifact. The commented deep-clean is destructive — read it before you run it.

Common mistakes and troubleshooting

Three gotchas deserve extra prose because they eat the most hours:

1. The SELinux :Z denial is invisible in ls -l. On RHEL-family hosts you bind-mount /opt/appdata into a container, the Unix permissions look perfect, and the app still gets Permission denied. Nothing about the file mode is wrong — it’s the SELinux type label on the host directory, which the container’s confined process may not access. ls -Z /opt/appdata shows the label; ausearch -m avc -ts recent shows the denial. The fix is the :Z/:z mount suffix, which relabels the content to a container-accessible type. ⚠️ Only ever apply it to directories you made for the container — :Z on /home or /var will recursively relabel and break the host.

2. --restart always is a lie about reboots (rootless). People assume the restart policy means “starts on boot.” It doesn’t — there is no daemon to enforce it while you’re logged out. Rootless containers only survive a reboot if (a) they’re managed by a Quadlet/systemd unit and (b) you’ve run loginctl enable-linger $USER so your user manager runs without an active login session. Miss either and the service is silently gone after a reboot, usually discovered during an outage.

3. :latest and un-pinned images cause “worked yesterday, broken today.” A tag is a mutable label. If your Quadlet unit or Compose file references :latest (or any tag), a publisher repushing that tag plus a host restart pulls a new image that may not be compatible. The whole point of content-addressing is defeated. Pin production images by @sha256: digest so the artifact is frozen, and update the digest deliberately as a reviewed change.

Cheat-sheet

Interview and exam questions

Q: What does “daemonless” mean for Podman, and name one concrete operational advantage over Docker. A: Podman has no central background service; podman run forks/execs the OCI runtime directly as the invoking user, with a small conmon monitor per container. Advantages include: no root daemon to attack; membership in a group isn’t root-equivalent; containers survive a Podman upgrade because they were never the daemon’s children; and it integrates cleanly with systemd instead of fighting it.

Q: A process shows uid=0(root) inside a rootless container. Is that a security problem? Explain the mapping. A: No. It’s root only inside a user namespace. Your host uid (e.g. 1000) is mapped to container uid 0, and your /etc/subuid range (e.g. 100000–165535) maps to the other in-container UIDs. A breakout lands as your unprivileged host user, not host root.

Q: What are the three OCI specifications and why do they matter? A: The image spec (image format — layers, config, manifest by digest), the runtime spec (how to turn a rootfs + config.json into a running namespaced/cgrouped process — implemented by runc/crun), and the distribution spec (the registry push/pull HTTP API). They make images portable and tool-agnostic across Podman, Docker, containerd and Kubernetes.

Q: podman run -p 80:80 fails for a normal user with a bind/permission error. Why, and give two fixes. A: Unprivileged users cannot bind ports below 1024. Fixes: publish a high host port (e.g. -p 8080:80) and reverse-proxy/NAT 80 to it; or lower the floor with sysctl net.ipv4.ip_unprivileged_port_start=80; or run that specific workload rootful.

Q: Difference between a named volume and a bind mount — and when do you need :Z? A: A named volume is Podman-managed storage under its storage dir, portable and lifecycle-independent; a bind mount maps an arbitrary host path you own. On SELinux hosts, bind-mounting a host directory needs a :z (shared) or :Z (private) suffix so Podman relabels the content to a type the confined container can access — otherwise the app gets Permission denied despite correct Unix modes.

Q: Explain ENTRYPOINT vs CMD, and exec vs shell form. A: ENTRYPOINT is the fixed executable; CMD supplies default arguments (or the default command if there’s no ENTRYPOINT) and is overridden by args after the image name. Exec form (["app","-x"]) makes your app PID 1 and lets it receive signals for clean shutdown; shell form (app -x) runs it under /bin/sh, which becomes PID 1 and swallows signals — so prefer exec form.

Q: Why is a multi-stage build better than a single-stage one? A: The build toolchain (compilers, headers, git, build secrets) lives only in an early stage that’s discarded; the final image copies just the built artifact. Result: dramatically smaller images, a smaller attack surface, and no build-time secrets leaking into shipped layers.

Q: You set --restart always on a rootless container but it’s gone after a reboot. Why, and what’s the correct fix? A: The restart policy isn’t enforced across a reboot when rootless — there’s no daemon running while you’re logged out. The correct approach is a Quadlet .container unit (or podman-restart.service) plus loginctl enable-linger $USER so your user’s systemd instance runs without an active login.

Q: What is Quadlet and why is it preferred over podman generate systemd? A: Quadlet lets you write a declarative .container (or .pod/.network/.volume/.kube) file in /etc/containers/systemd/ (or the rootless ~/.config/...) that a systemd generator turns into a real .service. It’s the single source of truth, versionable and reproducible, whereas the deprecated podman generate systemd emits a static .service that drifts from the container and must be regenerated on every change.

Q: runc vs crun? A: Both implement the OCI runtime spec. runc is the original Go implementation (Docker’s default). crun is a smaller, faster C implementation with better cgroups v2 support and lower per-container memory — the Fedora/RHEL default and preferable for rootless and high-density hosts. Images run identically under either.

Q (LFCS/RHCSA-style task): Configure a rootless Podman container running quay.io/library/redis that starts at boot as a systemd service and restarts on failure. A: Create ~/.config/containers/systemd/redis.container with [Container] Image=…redis, PublishPort=6379:6379, [Service] Restart=on-failure, [Install] WantedBy=default.target; run systemctl --user daemon-reload && systemctl --user start redis.service; then loginctl enable-linger $USER so it survives reboot without a login session.

Q (task): Prove a rootless container’s in-container root maps to your unprivileged host user. A: podman unshare cat /proc/self/uid_map shows 0 <your-uid> 1 (container uid 0 = your host uid), and podman run --rm alpine id shows uid=0(root) inside — the two together demonstrate the user-namespace mapping.

Key takeaways

• A container is an immutable userland + isolation. Images (layered, content-addressed, OCI-standard) ship the whole environment above the shared kernel; runc/crun turn one into a namespaced, cgroup-limited process. The OCI specs make images portable across every tool.
• Podman is daemonless and rootless by default; the CLI mirrors Docker. No root daemon to attack, no root-equivalent group, clean systemd integration — and alias docker=podman covers almost everything you already know.
• Rootless is real security, not a toggle. A user namespace maps your unprivileged host uid to in-container root via /etc/subuid//etc/subgid, so a breakout owns nothing. The main price is the sub-1024 port caveat and needing linger for boot persistence.
• Pin images by digest, build multi-stage, run non-root. :latest is a mutable label that will change under you; multi-stage keeps toolchains and secrets out of the shipped image; a non-root USER plus --cap-drop ALL, --read-only and no-new-privileges is the hardening baseline. Never --privileged.
• Volumes persist; bind mounts share host paths; SELinux needs :Z. The writable layer is ephemeral — use named volumes for state, and remember the :Z/:z relabel on SELinux hosts.
• User-created networks give DNS; the default one doesn’t. Put related containers on a podman network create’d bridge so they resolve each other by name via aardvark-dns.
• Quadlet makes a container a first-class systemd service. A declarative .container file in /etc/containers/systemd/ becomes a supervised, journald-logged, boot-persistent unit — the modern replacement for the deprecated podman generate systemd, and the right way to run containers in production on a single Linux host. When one host isn’t enough, podman kube generate is your on-ramp to Kubernetes.