The container conversation defaulted to “just run a daemon as root” for a decade, and most production fleets are still paying interest on that decision. Podman flips the model: no long-lived daemon, no root-owned socket, and containers that run entirely inside an unprivileged user’s namespace. A podman run is a plain fork-exec — your shell forks conmon, conmon execs an OCI runtime (crun or runc), and the container is a child of your process tree, not a client request to a privileged service. Pair that with Quadlet — the systemd generator that turns declarative .container files into real units — and containers become first-class systemd services, supervised by PID 1, with restart policies, resource caps, ordering dependencies, and journald logging for free. This is a walk through standing that up the way a platform team should, not the way a five-minute quickstart does.
The pain this removes is concrete. A root Docker daemon is a single point of total host compromise: whoever can reach /var/run/docker.sock can mount the host root filesystem into a container and walk away as root, which is why every serious security assessment flags it. Rootless Podman collapses that blast radius — a remote code execution inside a container lands an attacker as an unprivileged UID with no capabilities and no path to the host. And because the container is a systemd unit rather than a child of a daemon, “restart policy,” “resource limit,” and “starts at boot” stop being Docker-specific concepts and become the same Restart=, MemoryMax=, and WantedBy= you already use for every other service on the box. You stop running a second init system (the daemon) next to systemd.
By the end of this article you will understand the daemonless architecture precisely — where conmon sits, why there is no PID 1 inside the container by default, how a crash propagates to systemd — and you will be able to stand up a rootless service that survives reboots, updates itself from a registry with automatic rollback on a failed healthcheck, keeps the real client source IP through pasta, enforces memory and CPU limits through delegated cgroup v2 controllers, and lives entirely as version-controlled .container files. Everything below assumes a recent Podman (v4.4+ for Quadlet, v5.x for pasta-by-default networking) on a cgroup v2 host. Confirm with podman version and stat -fc %T /sys/fs/cgroup/ — you want cgroup2fs. Quadlet shipped in Podman 4.4 and is the supported successor to the now-removed podman generate systemd.
What problem this solves
Three problems, really, and they compound. The first is privilege. Docker’s default is a root daemon that owns every container; your CLI is a thin client to a root-owned Unix socket. Anyone in the docker group is effectively root on the host, because docker run -v /:/host hands them the host filesystem. Security teams issue hard controls against this — “no root-owned container daemon,” “no privileged ports bound by application processes” — and a root Docker daemon simply cannot satisfy them. Rootless Podman does, by construction: your login UID becomes UID 0 inside a user namespace whose other IDs map to a block of unprivileged host UIDs that belong to nobody.
The second is supervision drift. A Docker daemon is a second supervisor running beside systemd, with its own restart semantics (--restart=always), its own logging (json-file driver, log rotation you configure separately), and its own lifecycle that systemd does not understand. When the box reboots, the daemon starts your containers — but ordering against non-container services (a mounted filesystem, a network target, a database on the same host) is awkward, and the daemon’s health is now a dependency of your app’s health. Podman + Quadlet erases the second supervisor: every container is a normal systemd unit, so After=, Requires=, Restart=on-failure, journalctl -u, and boot ordering all work exactly as they do for nginx.service or postgresql.service.
The third is drift between “works on my machine” and “runs in the fleet.” Hand-written docker run lines rot in shell scripts and READMEs; nobody can reconstruct the exact flags a running container was started with. Quadlet makes the .container file the single declarative source of truth — checked into Git alongside your Ansible or Terraform — and the systemd unit is generated from it on daemon-reload. It is GitOps on a single host: edit the file, reload, done. Who hits all this hardest: teams on RHEL/Fedora/CentOS Stream where Podman is the default and Docker is not shipped; anyone with a PCI/CIS mandate against the Docker socket; edge and bare-metal fleets where “systemd-native” beats “run a Kubernetes cluster”; and anyone tired of the daemon being the thing that goes down at 3 a.m.
To frame the whole field before the deep dive, here is the shape of what changes when you move from a root Docker daemon to rootless Podman + Quadlet:
| Concern | Root Docker daemon | Rootless Podman + Quadlet | Why it matters |
|---|---|---|---|
| Privilege at runtime | Container runs as root (unless userns remapped) | Runs as your mapped UID; container root == unprivileged host UID | RCE in a container ≠ root on host |
| Supervisor | dockerd (a second init) |
systemd (the one you already run) | One lifecycle, one restart policy, one log stream |
| Boot persistence | Daemon starts containers | systemd unit + linger | Ordering against other services works |
| Source of truth | Shell scripts / docker run lines |
.container files in Git |
Reproducible, reviewable, declarative |
| Socket exposure | root-owned /var/run/docker.sock |
none by default; optional per-user socket | Removes the classic host-compromise path |
| Logs | json-file driver + your rotation |
journald automatically | journalctl -u, retention via journald config |
| Networking (rootless) | n/a (root bridge) | pasta / slirp4netns userspace stack | No CAP_NET_ADMIN needed on host |
| Updates | docker pull + restart, no rollback |
podman auto-update with healthcheck-gated rollback |
Safe unattended updates |
Learning objectives
By the end of this article you can:
- Explain the daemonless fork-exec architecture — where
conmon,crun/runc, and your shell or systemd unit sit in the process tree — and contrast it with the Docker client/daemon split, including what each buys and costs. - Configure rootless mode end to end: allocate non-overlapping
subuid/subgidranges of at least 65536 IDs, enable lingering so services survive logout and start at boot, and reason about the user-namespace UID map. - Choose and verify the right rootless network stack — slirp4netns vs pasta — handle privileged ports without running as root, and get container-to-container DNS working via
aardvark-dns. - Enforce CPU, memory, and PID limits on rootless containers by confirming cgroup v2 controllers are delegated to your user slice, and set them both via
PodmanArgsand idiomatic[Service]directives. - Author Quadlet
.container,.pod,.network, and.volumeunit files, understand how the generator synthesises thepodman runline and where the generated units live, and drive the declarative reload-and-start loop. - Handle volumes and SELinux correctly with
:U,:z, and:Z, and know when a named volume is the right answer instead of a bind mount. - Wire
podman auto-updatewith a per-user timer and understand why a realHealthCmdis what makes its automatic rollback meaningful. - Use
podman kube play/podman kube generate, the optional Docker-compat socket, and a concrete plan to migrate a Docker fleet to Podman + Quadlet.
Prerequisites & where this fits
You should be comfortable with Linux service administration and systemd fundamentals — units, systemctl, journalctl, targets, and the difference between the system manager and a --user manager. Container basics help: images, layers, an entrypoint, port publishing, bind mounts, and what an OCI runtime does. You do not need prior Podman experience; if your muscle memory is Docker, most of it transfers directly (alias docker=podman covers the majority of the CLI). You should be able to edit /etc/subuid, run loginctl, and read podman info output. A cgroup v2 host is assumed — every current RHEL 9/10, Fedora, Debian 12+, and Ubuntu 22.04+ boots cgroup v2 by default.
This sits in the Linux server administration track and leans hard on systemd. If the [Service]/[Unit]/[Install] sections, timers, or resource control feel shaky, read Mastering systemd: Units, Timers, Resource Control, and Service Hardening first — Quadlet is a systemd generator, so everything it produces is a systemd unit. On RHEL-family hosts the volume-labelling story is pure SELinux; Running SELinux in Enforcing Mode: Troubleshooting and Writing Custom Policy explains the :Z relabel you will use constantly, and on Ubuntu/Debian the equivalent confinement lives in Authoring AppArmor Profiles: Confining Services on Ubuntu and Debian. Resource limits ride on cgroup v2; the kernel-side tuning is in Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers. And if your workloads are cloud-bound rather than single-host, the compute-model decisions live in Azure App Service vs Container Apps vs AKS: Choose the Right Compute and AWS ECS vs EKS vs Fargate: Choose Your Container Path.
A quick map of who owns what during a rootless-Podman incident, so you escalate to the right layer:
| Layer | What lives here | Typical owner | Failure it causes |
|---|---|---|---|
| Identity ranges | /etc/subuid, /etc/subgid |
Platform / IAM | lchown invalid argument, no rootless at all |
| systemd user manager | linger, --user units, delegation |
Platform | Container vanishes on logout; no boot start |
| Quadlet generator | .container/.pod/.network/.volume |
App + platform (in Git) | Unit not generated; typo’d key ignored |
| Networking | pasta/slirp4netns, netavark, aardvark-dns | Platform / network | No egress, wrong source IP, port bind denied |
| Storage / SELinux | overlay driver, :Z labels, subuid chown |
Platform | Permission denied, avc: denied in audit log |
| Image supply | registry, digests, auto-update |
App + pipeline | Bad push crash-loops; auto-update rollback |
Core concepts
Six mental models make every later section obvious.
There is no daemon — the container is a child of your process, or of a systemd unit. Docker’s dockerd is a privileged, always-on service that owns every container; docker run sends it a request and returns. Podman has nothing like it. podman run forks a small supervisor called conmon (the container monitor), which execs the OCI runtime (crun by default on modern systems, or runc), which becomes the container’s first process. conmon holds the container’s stdio, writes logs, and reaps the child. When you launch under systemd, conmon becomes a child of the unit, so the kernel and systemd — not a daemon — are the source of truth for “is this container alive.” Kill the machine’s Podman binary and running containers keep running; there is no daemon whose death takes them all down.
Rootless works by mapping a range of host UIDs into a user namespace. Your single login UID (say 1000) becomes UID 0 inside the container’s user namespace. The container’s other UIDs (1–65535) map to a block of host UIDs that the system reserved for you in /etc/subuid//etc/subgid — typically 100000–165535. So container root is host UID 100000, an account that owns nothing else. A setuid-helper called newuidmap/newgidmap (from the shadow-utils/uidmap package) performs the mapping at container start. This is why an escaped process is harmless on the host: outside the namespace it is UID 100000 with no capabilities over anything real.
cgroups v2 is what lets rootless enforce limits — but only if controllers are delegated. On cgroup v1, unprivileged users could not set CPU/memory limits at all. cgroup v2 plus systemd’s delegation changes that: systemd hands your user slice control of the cpu, memory, pids, and io controllers via user@.service, so a rootless container can actually cap itself. If MemoryMax= silently does nothing, the memory controller was not delegated — that is the first thing to check, not a Podman bug.
Networking has no host bridge — a userspace stack fakes it. Creating a veth pair against a host bridge needs CAP_NET_ADMIN on the host, which a rootless user does not have. So rootless outbound traffic and published ports flow through a userspace network stack running in your namespace: historically slirp4netns, and on Podman 5 the faster, source-IP-preserving pasta (from the passt project). The container-facing side is managed by netavark (the network backend) with aardvark-dns providing name resolution between containers on a user-defined network.
A pod is a shared-namespace group borrowed from Kubernetes. A pod holds an infra container that keeps a set of namespaces (network, and optionally others) open, so member containers share one IP, one localhost, and one port space. You publish ports on the pod, not the members; members reach each other over 127.0.0.1. This is the sidecar pattern — proxy, app, metrics exporter — co-scheduled and co-supervised, and it is also the bridge to real Kubernetes via podman kube generate/play.
Quadlet is a systemd generator, not a runtime. You do not hand-write [Service] ExecStart=podman run ... and you do not use the removed podman generate systemd. You write a declarative .container file; systemd’s generator infrastructure runs Quadlet on every daemon-reload, and Quadlet synthesises a real .service unit (with the full podman run command) into a read-only generated directory. Your source of truth is the .container file in Git; the deploy step is systemctl daemon-reload.
The vocabulary in one table
Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side.
| Concept | One-line definition | Where it lives | Why it matters |
|---|---|---|---|
| conmon | Per-container monitor holding stdio + reaping the child | Child of your shell/unit | The “supervisor” in a daemonless world |
| OCI runtime | crun/runc — actually creates the container |
Exec’d by conmon | The kernel-facing layer |
| User namespace | Remaps UIDs/GIDs so you are “root” inside only | Per rootless container | The whole security model |
| subuid / subgid | Host ID ranges delegated to your user | /etc/subuid, /etc/subgid |
Without them, rootless can’t map IDs |
| Linger | Keeps your --user systemd manager alive with no login |
loginctl enable-linger |
Boot-persistent rootless services |
| slirp4netns | Original userspace rootless network | Per container | Reliable; slower; hides source IP |
| pasta | Modern userspace rootless network (Podman 5 default) | Per container | Fast; preserves source IP |
| netavark / aardvark-dns | Network backend + container DNS | Host-side | Name-based container-to-container resolution |
| cgroup v2 delegation | Controllers handed to your user slice | user@.service |
Rootless CPU/memory/pids limits |
| Pod | Shared-namespace group with an infra container | Podman/host | Sidecars; one IP; K8s bridge |
| Quadlet | systemd generator for .container/.pod/… files |
~/.config/containers/systemd/ |
Declarative, GitOps-able units |
| AutoUpdate=registry | Marks an image for podman auto-update |
Quadlet [Container] key |
Unattended updates + rollback |
:Z / :z |
SELinux relabel of a bind mount (private / shared) | Volume mount option | Access on RHEL-family hosts |
Podman versus Docker: daemonless and rootless by design
The architectural difference is not cosmetic, and it changes how you operate the system. Docker runs a privileged daemon (dockerd, which itself talks to containerd and containerd-shim) that owns every container; your docker CLI is a thin client speaking HTTP over a root-owned Unix socket. Compromise the socket, compromise the host — a container started with -v /:/host --privileged is game over. Podman has no daemon. podman run forks conmon, conmon execs crun/runc, and the container is a child of your process tree. When you wire that into systemd, conmon becomes a child of the unit, and the kernel — not a daemon — is the source of truth for liveness.
| Property | Docker (default) | Podman (rootless) |
|---|---|---|
| Central daemon | dockerd runs as root |
None |
| Container ownership | Child of the daemon | Child of your shell or systemd unit |
| Privilege at runtime | Root unless userns-remapped | Your UID, mapped via subuid/subgid |
| Supervision | Daemon --restart policy |
systemd unit (Restart=) |
| Socket exposure | root-owned /var/run/docker.sock |
none by default; optional per-user |
| Build tool | docker build (BuildKit) |
podman build / Buildah |
| Compose | docker compose (native) |
podman compose (wraps external), or Quadlet/kube play |
| Boot start | daemon restores containers | systemd + linger |
| Logs | json-file/journald driver |
journald (native, via conmon) |
| Kubernetes YAML | third-party tools | podman kube generate / play built in |
The practical wins compound. A crashed container is restarted by systemd with the same policy you use for every other service; logs land in the journal so journalctl --user -u myapp just works; boot ordering against a mounted filesystem or a local database uses the same After=/Requires= you already know; and an RCE inside a container lands an attacker as an unprivileged UID with no capabilities and no route to the host. The costs are honest too: rootless networking is a userspace stack (a throughput and feature ceiling versus a kernel bridge), some images that assume they can bind port 80 or write to root-owned paths need adjustment, and docker compose files map to Podman through a wrapper or a rewrite to Quadlet/Kube YAML rather than natively.
Where Podman deliberately stays Docker-compatible: the CLI surface (alias docker=podman covers the common verbs), the image format and registries (it pulls the same OCI images from Docker Hub, Quay, GHCR, and private registries), and an optional Docker-compatible API socket so tools hard-wired to docker.sock — Testcontainers, some CI runners — keep working. Buildah is the dedicated image-build tool (Podman shells out to it for podman build); Skopeo is the companion for copying and inspecting images between registries without pulling them.
| Docker command | Podman equivalent | Notes |
|---|---|---|
docker run |
podman run |
Near-identical flags |
docker build |
podman build / buildah bud |
Buildah under the hood |
docker compose up |
podman compose up |
Wraps external compose; or use Quadlet |
docker ps / logs / exec |
podman ps / logs / exec |
Same |
docker system prune |
podman system prune |
Same |
docker save / load |
podman save / load |
Same |
--restart=always |
systemd Restart=always in the .container |
The idiomatic replacement |
docker sock for tools |
podman system service (Docker-compat socket) |
Opt-in, per-user |
Setting up rootless mode: subuid, subgid, and lingering
Rootless containers map a range of host UIDs/GIDs into the container’s user namespace. Your login UID becomes root (UID 0) inside the container; the container’s other UIDs map to a block of unprivileged host UIDs reserved for you. Those ranges live in /etc/subuid and /etc/subgid, one line per user in the form name:start:count.
On a fresh service user, allocate a non-overlapping range of at least 65536 IDs. On most distributions creating a user with useradd already assigns a subuid/subgid block; verify it, and only add one if missing:
# Verify the user already has a range (many distros do this at useradd time)
grep '^svc-api:' /etc/subuid /etc/subgid
# /etc/subuid:svc-api:100000:65536
# /etc/subgid:svc-api:100000:65536
# If missing, grant a 65536-wide subordinate range (auto-picks the next free block)
sudo usermod --add-subuids 200000-265535 --add-subgids 200000-265535 svc-api
# After editing the maps, force Podman to re-read them
sudo -u svc-api podman system migrate
The 65536 width matters: it lets a container reference the full standard UID space (0–65535). Too narrow and images that chown to high UIDs fail with lchown ... invalid argument — the single most common “why won’t this image run rootless” error. Ranges must not overlap between users; an overlap means two users share host UIDs, defeating the isolation.
| Symptom / need | Cause | Fix |
|---|---|---|
lchown <uid>: invalid argument on podman pull/run |
subuid/subgid range narrower than the image’s max UID | Widen range to ≥ 65536; podman system migrate |
there might not be enough IDs available |
No subuid/subgid line for the user at all | Add a range with usermod --add-subuids/--add-subgids |
| Changed the maps, still broken | Podman cached the old mapping | podman system migrate (recreates the userns) |
| Two users’ containers clobber each other’s files | Overlapping ranges in /etc/subuid |
Assign disjoint 65536-wide blocks per user |
newuidmap: write to uid_map failed |
uidmap/shadow-utils missing or not setuid |
Install the package; ensure newuidmap/newgidmap are setuid root |
For services that must survive logout and start at boot, enable lingering. Without it, the user’s systemd --user instance is torn down when their last session ends, taking every container with it. This is the single most-forgotten step: the container works in your SSH session, you log out, and it vanishes.
# Keep the user's `systemd --user` manager running with no active login session
sudo loginctl enable-linger svc-api
# Confirm
loginctl show-user svc-api --property=Linger
# Linger=yes
Root/system-wide Podman is a real alternative and is sometimes correct — a system service that genuinely needs a privileged port or host devices, running under the system systemd manager. The trade-off is that “rootful Podman” gives back the privilege you were trying to shed; you keep the daemonless model but lose the unprivileged-UID blast-radius win. Choose per workload:
| Mode | Manager | Files live in | subuid/subgid | Linger needed? | Use when |
|---|---|---|---|---|---|
| Rootless (user) | systemd --user |
~/.config/containers/systemd/ |
Required per user | Yes, for boot | The default; least privilege |
| Rootful (system) | system systemd | /etc/containers/systemd/ |
Not used (real root) | No | Needs host devices / true privileged ports |
| Rootless as a system unit | system systemd running a --user-style unit for an account |
/etc/containers/systemd/ with User= |
Required for that user | No (system-managed) | Fleet automation wanting boot start without login |
User namespaces, storage drivers, and volumes
Inside the namespace the container believes it is root; on the host it is not. Prove it:
podman run --rm docker.io/library/alpine id
# uid=0(root) gid=0(root) ... <- inside the container
podman unshare cat /proc/self/uid_map
# 0 100000 65536 <- container UID 0 == host UID 100000
That second line is the entire security model in three numbers: container UID 0 is host UID 100000, an account that owns nothing else on the system. podman unshare drops you into the same user namespace your containers use — invaluable for fixing ownership on rootless storage from the host side.
Storage driver
Rootless Podman defaults to overlay via the kernel’s unprivileged overlayfs (kernel 5.11+) or, failing that, fuse-overlayfs in userspace. You do not need vfs on any modern kernel and you should avoid it — vfs deep-copies every layer on every container and wastes enormous disk. Confirm and know what you are looking at:
podman info --format '{{.Store.GraphDriverName}}'
# overlay
| Driver | How it works | Rootless support | Disk efficiency | Use when |
|---|---|---|---|---|
| overlay (native) | Kernel unprivileged overlayfs (5.11+) | Yes (best) | Copy-on-write layers | Default on modern kernels |
| fuse-overlayfs | Userspace overlay via FUSE | Yes | CoW, slight FUSE overhead | Older kernels lacking unprivileged overlay |
| vfs | Full copy of every layer | Yes | Terrible (deep copy) | Only as a last-resort fallback |
| btrfs / zfs | Filesystem-native snapshots | Limited rootless | Excellent | Root-mode on btrfs/zfs hosts |
Rootless storage lives under $HOME/.local/share/containers/storage, not /var/lib/containers. Put $HOME on a filesystem that supports the driver (xfs or ext4 are fine; some network filesystems are not, and NFS in particular fights overlay).
Volumes, the :U trick, and SELinux :Z
Because of the UID shift, a host bind mount owned by your real UID appears unowned inside the container (the files map to a UID the container doesn’t have). Two mount-option families fix the two distinct problems — ownership and SELinux labelling:
:Urecursivelychowns the source to the container user’s mapped UID/GID so the container can read/write it.:z/:Zrelabels the source for SELinux on RHEL-family hosts. Lowercase:zapplies a shared label (multiple containers may use it); uppercase:Zapplies a private label (exactly one container). Getting this wrong is the classic “Permission denied but the file is world-readable” on RHEL.
# :U -> chown source to the container user's mapped UID/GID
# :Z -> apply a PRIVATE SELinux label so this one container can access it
podman run --rm \
-v /srv/appdata:/data:U,Z \
docker.io/library/alpine touch /data/written-by-container
| Mount option | What it does | When to use | Gotcha |
|---|---|---|---|
| (none) | Mounts as-is | Named volumes; already-correct ownership | Bind mount often shows as unowned inside |
:U |
Recursively chowns source to mapped UID | Bind mount the container must write to | Recursive chown is O(files); slow on huge trees |
:z |
Shared SELinux relabel | A dir shared by several containers | Over-broad label if you only need one container |
:Z |
Private SELinux relabel | A dir for exactly one container | If two containers use it, the second is denied |
:ro |
Read-only mount | Config, certs, secrets | Combine as :ro,Z; app can’t write |
:z on /etc or $HOME |
Relabels a system path | Never | Recursively relabels the host path — dangerous |
For state you do not need to inspect from the host, prefer named volumes (podman volume create). They live inside Podman storage, already carry the correct mapped ownership, and sidestep the relabel dance entirely. Quadlet’s .volume file is the declarative way to create them.
| Named volume | Bind mount | |
|---|---|---|
| Ownership under rootless | Correct automatically | Often needs :U |
| SELinux label | Handled | Needs :z/:Z |
| Inspect from host | Awkward (inside Podman storage) | Trivial (real path) |
| Portability across hosts | Higher (Podman manages it) | Path must exist identically |
| Best for | Opaque app state (DB data dirs) | Config you edit, logs you tail |
Grouping containers into pods and shared namespaces
A pod is Podman’s borrowing from Kubernetes: a set of containers that share a network namespace (one IP, one localhost, one port space) and optionally other namespaces (--share ipc,uts etc.). The pod owns an infra container — a tiny placeholder (pause-like) that holds the namespaces open so individual member containers can start and stop without tearing the shared network down.
# Create a pod, publishing 8080 on the host ONCE -- all members share it
podman pod create --name web --publish 8080:80
# A reverse proxy and the app it fronts, both in the pod
podman run -d --pod web --name proxy docker.io/library/nginx:alpine
podman run -d --pod web --name cache docker.io/library/redis:7-alpine
# Inside the pod, 'proxy' reaches 'cache' on 127.0.0.1:6379 -- same netns
podman pod inspect web --format '{{.InfraContainerID}}'
The key networking insight: you publish ports on the pod, not the member containers, because they share one network namespace. Containers in a pod talk to each other over localhost. This maps cleanly onto the sidecar pattern — proxy, app, and a metrics exporter co-scheduled and co-supervised — and it is the natural unit that podman kube generate turns into a Kubernetes Pod manifest.
| Grouping | Network model | Container-to-container | Publish ports on | Best for |
|---|---|---|---|---|
| Standalone containers | Each its own netns (rootless) | Via a shared user-defined network + DNS name | Each container | Independent services |
| Pod | Shared netns (one IP) | localhost |
The pod | Sidecars; tightly-coupled group |
| User-defined network | Separate netns, common bridge (rootless: netavark) | By container name (aardvark-dns) | Each container | Microservices that resolve each other by name |
Quadlet: generating systemd units from .container files
This is the heart of a production setup. You write a declarative .container file and let the Quadlet generator translate it into a real unit at boot and on daemon-reload. Rootless Quadlet files go in ~/.config/containers/systemd/; root/system-wide units go in /etc/containers/systemd/ (or /usr/share/containers/systemd/ for vendor-shipped ones). Create ~/.config/containers/systemd/myapp.container:
[Unit]
Description=My application container
After=network-online.target
Wants=network-online.target
[Container]
Image=ghcr.io/acme/myapp:1.4.2
ContainerName=myapp
PublishPort=8080:8080
Environment=LOG_LEVEL=info
Volume=myapp-data.volume:/var/lib/myapp:Z
# Drop all capabilities, add back only what the app needs
DropCapability=ALL
AddCapability=NET_BIND_SERVICE
# Mark this image as a candidate for `podman auto-update`
AutoUpdate=registry
HealthCmd=curl -fsS http://localhost:8080/healthz || exit 1
HealthInterval=30s
HealthRetries=3
HealthStartPeriod=20s
[Service]
Restart=on-failure
RestartSec=5
[Install]
WantedBy=default.target
Note there is no [Service] ExecStart — Quadlet synthesises the podman run line from the [Container] section. The myapp-data.volume:/... reference points at a .volume file you create alongside it:
# ~/.config/containers/systemd/myapp-data.volume
[Volume]
# Quadlet names the real Podman volume "systemd-myapp-data"
# (add Driver=, Options=, etc. here if needed)
Now reload and start. The unit name is the filename stem plus .service:
# Re-run the generator (Quadlet is invoked by daemon-reload)
systemctl --user daemon-reload
# Start it -- the unit is named after the .container file
systemctl --user start myapp.service
systemctl --user status myapp.service
Quadlet units are generated read-only into /run/user/<uid>/systemd/generated/ (rootless) or /run/systemd/generated/ (system); you never edit those. You edit the .container file and daemon-reload. This is the declarative loop that makes Podman feel like GitOps on a single host — the .container files are your source of truth and live in version control alongside your Ansible/Terraform.
The four Quadlet unit types
Quadlet is not just .container. Four file types compose into a full application, and the generator wires the ordering dependencies between them for you (a .container referencing a .volume gets an automatic After=/Requires= on the generated volume unit).
| File type | Generates | Key section | Purpose |
|---|---|---|---|
.container |
<name>.service |
[Container] |
One container as a systemd service |
.pod |
<name>-pod.service |
[Pod] |
A pod; point .containers at it with Pod= |
.network |
<name>-network.service |
[Network] |
A user-defined network (netavark + DNS) |
.volume |
<name>-volume.service |
[Volume] |
A named volume, created on first use |
.image |
<name>-image.service |
[Image] |
Pre-pull an image before dependents start |
.kube |
<name>.service |
[Kube] |
Run a Kubernetes YAML via kube play |
For a whole pod, write a .pod file and point each .container at it with Pod=web.pod:
# ~/.config/containers/systemd/web.pod
[Pod]
PodName=web
PublishPort=8080:80
[Install]
WantedBy=default.target
# ~/.config/containers/systemd/web-app.container
[Container]
Image=ghcr.io/acme/web:latest
Pod=web.pod
Common [Container] keys and their podman run equivalents
The [Container] section maps almost one-to-one onto podman run flags. Knowing the mapping lets you convert any docker run/podman run line into a Quadlet file mechanically. A reference of the keys you will actually use:
| Quadlet key | podman run equivalent |
Notes |
|---|---|---|
Image= |
(positional image) | Registry ref or a .image unit |
ContainerName= |
--name |
Defaults to systemd-<unit> |
PublishPort= |
-p / --publish |
Repeatable; HOST:CONTAINER |
Environment= |
-e / --env |
Repeatable; or EnvironmentFile= |
Volume= |
-v |
Repeatable; supports :Z, :U, :ro |
Network= |
--network |
Name a .network, or pasta/slirp4netns |
Exec= |
(command after image) | Override the entrypoint’s CMD |
User= / Group= |
--user |
UID/GID inside the container |
DropCapability= / AddCapability= |
--cap-drop / --cap-add |
Least-privilege the caps |
NoNewPrivileges=true |
--security-opt no-new-privileges |
Blocks setuid escalation |
ReadOnly=true |
--read-only |
Read-only rootfs |
AutoUpdate=registry |
label io.containers.autoupdate=registry |
Enables podman auto-update |
HealthCmd= |
--health-cmd |
Plus HealthInterval, HealthRetries, HealthStartPeriod |
Secret= |
--secret |
Mounts a Podman secret |
PodmanArgs= |
(raw passthrough) | Escape hatch for flags with no dedicated key |
The [Container] section covers most needs; PodmanArgs= is the escape hatch for anything Quadlet has no dedicated key for. Prefer dedicated keys where they exist — they are validated and readable; PodmanArgs= is a raw passthrough with no validation.
Rootless networking with pasta, slirp4netns, and port mapping
Rootless containers cannot create veth pairs against a host bridge — that needs CAP_NET_ADMIN on the host. So outbound traffic and published ports flow through a userspace network stack. There are two, and the default flipped in Podman 5:
- slirp4netns — the original. Reliable and broadly compatible, but TCP throughput is limited and, by default, it does not preserve the real client source IP (traffic appears to come from an internal gateway like
10.0.2.x). - pasta (from the
passtproject) — the modern default in Podman 5.x for the rootless default network. Faster, lower latency, and it preserves source addresses far better. It is the default for the rootless default network on Podman 5.
# What is actually backing rootless networking?
podman info --format '{{.Host.NetworkBackend}}' # netavark (the backend)
podman info --format '{{.Host.Slirp4netns.Executable}}' # path if slirp is present
podman info --format '{{.Host.Pasta.Executable}}' # path if pasta is present
# Run a container on the default rootless network (pasta on Podman 5)
podman run -d --name web -p 8080:80 docker.io/library/nginx:alpine
# Force a specific stack explicitly
podman run -d --network pasta -p 8080:80 docker.io/library/nginx:alpine
podman run -d --network slirp4netns -p 8080:80 docker.io/library/nginx:alpine
# Publish on a specific host interface only (loopback = local-only)
podman run -d -p 127.0.0.1:8080:80 docker.io/library/nginx:alpine
| Attribute | slirp4netns | pasta |
|---|---|---|
| Podman default | ≤ 4.x | 5.x (rootless default net) |
| Throughput | Lower | Higher |
| Latency | Higher | Lower |
| Preserves client source IP | No (by default) | Yes |
| IPv6 | Supported | Better/native handling |
| Maturity | Very mature | Newer, now default |
| Pick it when | Max compatibility / old Podman | Source-IP fidelity or throughput matters |
Two production gotchas dominate:
1. Ports below 1024. A rootless user cannot bind privileged host ports by default. Three honest options, in order of preference: publish a high port and front it with a reverse proxy or a firewall DNAT; lower the unprivileged-port threshold deliberately; or (last resort) run that one workload rootful. Lowering the threshold once:
# Allow rootless processes to bind from port 80 upward
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
# Persist it
echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-rootless-ports.conf
| Approach | Privilege cost | Effort | Best for |
|---|---|---|---|
| High port + reverse proxy / DNAT | None (app stays unprivileged) | Medium | The clean default; production |
Lower ip_unprivileged_port_start |
Host-wide sysctl change | Low | You control the host and accept the trade |
AmbientCapabilities=CAP_NET_BIND_SERVICE on the user unit |
Grants one capability | Low | A single service that must bind low |
| Run the workload rootful | Full root for that container | Low | Truly requires host-level networking |
2. Source IP for logging/allowlists. With pasta you can keep the real client IP; with default slirp4netns you cannot, and every request appears to come from the internal gateway. If an app’s access logs or IP allowlists matter (fraud models, geo rules, rate limits keyed on IP), prefer pasta and verify the observed source address rather than assuming it works.
For container-to-container DNS, create a user-defined network — members resolve each other by container name through aardvark-dns, which ships with netavark:
podman network create appnet
podman run -d --network appnet --name api ghcr.io/acme/api:latest
podman run -d --network appnet --name web ghcr.io/acme/web:latest
# 'web' can now reach the API at http://api:PORT
The Quadlet-native way is a .network file that each .container references with Network=appnet.network, so the network is created and torn down as part of the unit lifecycle.
Automatic image updates with podman auto-update and rollback
Any container labelled io.containers.autoupdate=registry (the AutoUpdate=registry line in the Quadlet file sets exactly this label) participates in podman auto-update. It checks the registry for a newer digest on the same tag, pulls it, and restarts the systemd unit — only for containers managed by systemd, which Quadlet ones are by definition.
# Dry run: show what would update, change nothing
podman auto-update --dry-run
# Apply: pull newer digests and restart affected units
podman auto-update
The feature that makes this safe in production is automatic rollback. After restarting an updated unit, Podman waits for the container to become healthy. If the unit fails to come up, Podman rolls the image reference back to the previous digest and restarts again. Rollback is keyed on the unit reaching a running/healthy state — which is exactly why a real HealthCmd matters. Without one, “healthy” means merely “the process did not immediately exit,” and a subtly broken image (starts, serves errors) sails through and rollback never fires.
Wire it to a timer so it runs unattended. Podman ships podman-auto-update.service and podman-auto-update.timer; enable them per user:
systemctl --user enable --now podman-auto-update.timer
systemctl --user list-timers podman-auto-update.timer
AutoUpdate policy |
Behaviour | Rollback | Use when |
|---|---|---|---|
registry |
Poll registry for a newer digest on the same tag; pull + restart | Yes (on failed/unhealthy start) | A tag you control (:prod, :stable) |
local |
Restart if a locally-pulled image changed (e.g. you podman pulled it) |
Yes | Air-gapped / pipeline pushes the image locally |
| (unset) | Never auto-updated | n/a | Pinned digests; anything where surprise is unacceptable |
Pin to immutable digests for anything where surprise is unacceptable, and reserve AutoUpdate=registry for tags you genuinely want to track — a stable channel, an internal :prod tag your pipeline re-points only after a canary clears. Auto-update on a :latest you do not control is how you get paged at 3 a.m. The rollback safety net is real, but it only fires if your healthcheck is honest.
Logging, healthchecks, and resource limits via cgroups v2
Because the container is a systemd unit, its stdout/stderr go to the journal automatically — no log driver to configure, no rotation to wire up (journald’s retention applies).
# Follow a Quadlet unit's logs
journalctl --user -u myapp.service -f
# Or Podman's own view, which also reads the journal
podman logs -f myapp
Healthchecks declared in the Quadlet file (HealthCmd, HealthInterval, HealthRetries, HealthStartPeriod) are enforced by Podman via a transient systemd timer per container. Inspect and act:
podman healthcheck run myapp # run it once, now
podman inspect myapp --format '{{.State.Health.Status}}' # healthy | unhealthy | starting
HealthStartPeriod gives a slow-booting app grace before failures count — the same lesson as any orchestrator: separate “starting” from “unhealthy,” or you evict a container that just needed 15 seconds to warm. You can have Podman act on failure with HealthOnFailure=kill (or restart / stop), turning a failed probe into a restart that systemd then re-supervises.
| Healthcheck key | Meaning | Sensible default | Failure mode if wrong |
|---|---|---|---|
HealthCmd |
Command that returns 0 when healthy | A cheap, honest self-check | Deep dependency check → flaps on downstream blips |
HealthInterval |
Time between checks | 30s | Too tight → CPU noise; too loose → slow detection |
HealthRetries |
Consecutive failures before “unhealthy” | 3 | 1 → flaps on a transient; 10 → slow to react |
HealthStartPeriod |
Grace window at start | 10–30s | Too short → kills a slow-booting app repeatedly |
HealthOnFailure |
Action on unhealthy (none/kill/restart/stop) |
none (let systemd decide) |
restart without a start period → restart loop |
Resource limits ride on cgroup v2. Rootless CPU/memory/pids limits require the controllers be delegated to your user slice — modern systemd does this for cpu, memory, pids, and io out of the box via user@.service. Set caps declaratively. The idiomatic way is [Service] directives, because they show up in systemctl status and integrate with systemd accounting:
[Service]
MemoryMax=512M
CPUQuota=150%
TasksMax=256
Restart=on-failure
RestartSec=5
Or, if you prefer Podman’s own accounting, PodmanArgs= on the [Container]:
[Container]
Image=ghcr.io/acme/myapp:1.4.2
PodmanArgs=--memory=512m --cpus=1.5 --pids-limit=256
Verify delegation and live accounting — the check that saves you an hour when a limit “does nothing”:
# Which controllers are delegated to your user slice?
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers
# Expect: cpu memory pids io (delegation present)
# Live resource usage for running containers
podman stats --no-stream
If memory limits silently do nothing, the memory controller is not delegated — check that line before blaming Podman.
| Limit | [Service] directive |
PodmanArgs flag |
Notes |
|---|---|---|---|
| Memory hard cap | MemoryMax=512M |
--memory=512m |
OOM-kills the container at the ceiling |
| Memory high (soft) | MemoryHigh=384M |
(none direct) | Throttles reclaim before the hard cap |
| CPU quota | CPUQuota=150% |
--cpus=1.5 |
150% = 1.5 cores worth of time |
| CPU weight (share) | CPUWeight=200 |
--cpu-shares |
Relative weighting under contention |
| Max processes/threads | TasksMax=256 |
--pids-limit=256 |
Fork-bomb protection |
| Block IO weight | IOWeight=100 |
--blkio-weight |
Needs io delegated |
Architecture at a glance
Picture the request and the process tree side by side, because the whole model is that the two are the same. Left edge: an inbound TCP connection arrives at a published host port — say :8443. Because this is rootless, there is no host bridge; the connection is handled by the pasta userspace network process running in the container’s network namespace, which forwards it inward while preserving the client’s real source IP. Center: the packet reaches your application process, which is running as PID 1 inside a user namespace where it believes it is UID 0 — but on the host it is UID 100000, an unprivileged account. That application process was not started by any daemon. Trace its parentage upward and you find conmon (the per-container monitor holding its stdio and ready to reap it), and above conmon a systemd --user service unit named myapp.service, and above that the user’s systemd --user manager kept alive by linger, and at the very top the host’s PID 1. There is no dockerd anywhere in that chain.
Right side and underneath: three cross-cutting systems attach to that process tree. Storage flows down through the overlay graph driver rooted at $HOME/.local/share/containers/storage, with any bind mounts relabelled for SELinux (:Z) and chowned into the namespace (:U). Resource control flows up into the kernel’s cgroup v2 tree at user.slice/user-1000.slice/..., where the cpu, memory, and pids controllers — delegated to this user slice by systemd — enforce CPUQuota=150%, MemoryMax=512M, and TasksMax=256. And the whole thing is generated, not hand-built: a Quadlet .container file in ~/.config/containers/systemd/ is read by systemd’s generator on every daemon-reload, which synthesises the read-only .service unit under /run/user/1000/systemd/generated/ that actually carries the podman run command.
The flow the reader should be able to narrate: a connection enters via pasta → hits the app running as an unprivileged mapped UID → whose supervisor is conmon → whose supervisor is a systemd user unit → generated by Quadlet from a Git-tracked .container file → resource-capped by delegated cgroup v2 controllers → logging to journald → and, on a timer, checked against the registry by podman auto-update, which will roll the image back if the healthcheck fails. Every arrow in that chain is a standard Linux mechanism (fork-exec, namespaces, cgroups, systemd, journald) rather than a proprietary daemon protocol — which is precisely why the failure modes are debuggable with tools you already own: systemctl, journalctl, ps, cat /sys/fs/cgroup/..., and podman inspect.
Real-world scenario
Meridian Payments ran ~40 stateless API containers per edge node across a fleet of bare-metal RHEL 9 boxes, originally on a root Docker daemon. Their security organisation issued a hard control coming out of a PCI assessment that flagged the Docker socket as a single point of total host compromise: no root-owned container daemon, and no privileged ports bound by application processes. Two facts made this awkward. The app was hard-coded to bind port 443, and its access logs fed an IP-based fraud model — so they could neither run the process as root to bind 443 nor lose the real client source address. The platform team was five engineers; the fleet was 60 nodes; downtime during the migration had to be near-zero.
They moved each service to rootless Podman under a dedicated svc-api user with linger enabled, and defined every workload as a Quadlet .container file checked into the same Git repo as their Ansible. The generator produced the systemd units on each host at daemon-reload, driven by an Ansible handler. Two specifics made it stick. First, instead of granting the service the privilege to bind 443, they kept the container on a high port (8443) and lowered the unprivileged-port threshold only as far as needed, fronting it with the host firewall’s DNAT so nothing application-owned ran as root:
# /etc/containers/systemd/payments-api.container (system-managed, runs as svc-api)
[Container]
Image=registry.internal/payments/api:prod
User=svc-api
PublishPort=8443:8443
AutoUpdate=registry
HealthCmd=curl -fsS https://localhost:8443/healthz -k || exit 1
HealthInterval=15s
HealthStartPeriod=20s
DropCapability=ALL
NoNewPrivileges=true
# pasta preserves the real client source IP for the fraud model
Network=pasta
[Service]
Restart=on-failure
RestartSec=5
MemoryMax=1G
CPUQuota=200%
[Install]
WantedBy=default.target
Second, they switched the rootless network backend to pasta, which preserved the genuine client source IP end to end — slirp4netns had been rewriting every request to its internal gateway address, which would have blinded the fraud model. Auto-update was pointed at the internal :prod tag, which their pipeline re-pinned to an immutable digest only after a canary node cleared, and the built-in rollback (gated on the 15-second healthcheck) caught two bad pushes in the first quarter without a human touching a host — each time, the unit failed its healthcheck, Podman reverted to the prior digest, and the only evidence was a journal line and a Slack alert on the rollback.
The numbers: the migration ran node-by-node behind a load balancer with zero customer-visible downtime over three weeks. Per-node memory overhead dropped — removing the always-on dockerd freed roughly 150–250 MB per node that had been the daemon’s resident footprint. The Docker socket finding was closed outright (the socket no longer existed). And “restart policy” became a one-line Restart=on-failure that every other systemd service on the box already used — the platform team deleted a home-grown watchdog script that had existed only to restart crashed Docker containers. The line the team put on the wall: “We didn’t add a container platform. We deleted one — and let systemd do the job it was already doing for everything else.”
The migration as a phased plan, because the order is the lesson:
| Phase | Action | Guardrail | Rollback if it fails |
|---|---|---|---|
| 0 | Inventory every docker run/compose line and its flags |
podman ps/scripts audit |
(nothing changed yet) |
| 1 | Translate each to a .container file in Git |
Peer review the Quadlet | Delete the file; keep Docker |
| 2 | Stand up on one canary node, drain it first | Healthcheck must pass | Re-add node to LB on Docker |
| 3 | Roll node-by-node behind the LB | Two instances of each service fleet-wide | Halt the roll; nodes stay mixed |
| 4 | Enable auto-update on :prod with rollback |
Canary node updates first | Rollback is automatic |
| 5 | Decommission dockerd on migrated nodes |
Confirm no socket consumers left | Reinstall docker (kept in repo) |
Advantages and disadvantages
The daemonless, rootless, systemd-native model both removes whole classes of problem and introduces a few new edges. Weigh it honestly:
| Advantages | Disadvantages |
|---|---|
| No root daemon → RCE in a container lands as an unprivileged UID, not root | Rootless networking is a userspace stack (pasta/slirp) with a throughput/feature ceiling vs a kernel bridge |
Containers are ordinary systemd units — Restart=, After=, journalctl, boot ordering all just work |
You must learn systemd unit + Quadlet syntax; docker compose doesn’t map 1:1 |
| One supervisor (systemd), not two (systemd + dockerd) — less RAM, one lifecycle | Some images assume they can bind :80 or write root-owned paths; they need adjustment |
.container files are declarative, reviewable, and GitOps-able |
The generated-unit indirection (edit file → daemon-reload) trips people expecting to edit the .service |
| Auto-update with healthcheck-gated rollback is built in, no extra tooling | Rollback is only as good as your HealthCmd; a silent-error image can still slip through |
| Removing the Docker socket closes the classic total-host-compromise finding | subuid/subgid, linger, and cgroup delegation are new operational concepts to get right |
| journald logging is automatic; retention is your existing journald config | High-throughput or exotic-networking workloads may still want rootful or a kernel-bridge setup |
The model is right for standard stateless services, edge and bare-metal fleets, and anyone under a mandate against a root container daemon — which is a large and growing set, since it is the default on all of RHEL/Fedora/CentOS. It bites hardest on workloads that genuinely need kernel-level networking performance, images that were built assuming root, and teams that never learn the three foundational concepts (subuid, linger, cgroup delegation) and then conclude “rootless is flaky” when it is actually mis-set-up. Every disadvantage above is manageable — but only if you know it exists, which is the point of this article.
Hands-on lab
Stand up a rootless container as a Quadlet-managed systemd service, confirm it runs unprivileged, cap its resources, and (optionally) prove it survives a reboot. Free-tier-friendly — it needs only a Linux VM with Podman 4.4+ (any recent Fedora/RHEL/Debian/Ubuntu). Run as a non-root user (create a dedicated one for realism).
Step 1 — Confirm the platform.
podman version | head -3 # want 4.4+ (5.x ideal)
stat -fc %T /sys/fs/cgroup/ # want: cgroup2fs
grep "^$USER:" /etc/subuid /etc/subgid # want a range, e.g. USER:100000:65536
Expected: a Podman version, cgroup2fs, and one subuid/subgid line each. If the subuid/subgid lines are missing, add a range with sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER and run podman system migrate.
Step 2 — Enable lingering so the user manager runs at boot.
sudo loginctl enable-linger "$USER"
loginctl show-user "$USER" --property=Linger # Linger=yes
Step 3 — Prove the user-namespace mapping (the security model).
podman run --rm docker.io/library/alpine id # uid=0(root) INSIDE
podman unshare cat /proc/self/uid_map # 0 100000 65536 -> host UID
Expected: root inside the container, mapped to host UID 100000 outside. That is the whole isolation guarantee.
Step 4 — Write a Quadlet .container unit.
mkdir -p ~/.config/containers/systemd
cat > ~/.config/containers/systemd/weblab.container <<'EOF'
[Unit]
Description=Rootless nginx lab
After=network-online.target
Wants=network-online.target
[Container]
Image=docker.io/library/nginx:alpine
ContainerName=weblab
PublishPort=8080:80
DropCapability=ALL
AddCapability=NET_BIND_SERVICE
HealthCmd=curl -fsS http://localhost:80/ || exit 1
HealthInterval=15s
HealthRetries=3
HealthStartPeriod=10s
[Service]
Restart=on-failure
RestartSec=5
MemoryMax=256M
CPUQuota=100%
TasksMax=128
[Install]
WantedBy=default.target
EOF
Step 5 — Generate the unit and start it.
systemctl --user daemon-reload # Quadlet runs here, synthesising weblab.service
systemctl --user start weblab.service
systemctl --user status weblab.service --no-pager
Expected: Active: active (running). Note there is a weblab.service even though you never wrote a [Service] ExecStart — Quadlet generated it. You can see the generated unit (read-only): cat /run/user/$(id -u)/systemd/generated/weblab.service.
Step 6 — Verify it works, is healthy, and runs unprivileged.
curl -fsS http://localhost:8080/ | head -1 # nginx welcome HTML
podman inspect weblab --format '{{.State.Health.Status}}' # healthy (after ~15s)
podman top weblab huser hpid user # huser = your mapped UID, NOT 0
Expected: HTML returned, healthy, and the host user (huser) is a high mapped UID like 100000 — not root.
Step 7 — Confirm resource limits landed (cgroup v2 delegation).
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers # cpu memory pids io
podman stats --no-stream weblab # MEM LIMIT ~256MiB, PIDS bounded
Expected: the delegated controllers include memory and cpu, and podman stats shows the 256 MiB memory ceiling.
Step 8 (optional) — The real test: reboot survival.
sudo systemctl reboot
# ... wait for the box, log back in ...
systemctl --user is-active weblab.service # active (thanks to linger)
If this fails but everything else passed, you skipped loginctl enable-linger in Step 2.
Validation checklist. You ran a container with no daemon, as an unprivileged UID, supervised by systemd, generated declaratively from a Git-able .container file, capped by delegated cgroup v2 controllers, with a healthcheck — and it came back after a reboot. That is the entire production pattern in miniature.
| Step | What you did | What it proves |
|---|---|---|
| 3 | podman unshare UID map |
Container root == unprivileged host UID |
| 5 | daemon-reload generates weblab.service |
Quadlet synthesises the unit; no hand-written ExecStart |
| 6 | podman top ... huser |
The process runs unprivileged on the host |
| 7 | cgroup.controllers + podman stats |
Limits are enforced via delegated cgroup v2 |
| 8 | Reboot → still active | Linger makes it boot-persistent |
Teardown.
systemctl --user stop weblab.service
rm ~/.config/containers/systemd/weblab.container
systemctl --user daemon-reload
podman rm -f weblab 2>/dev/null; podman image rm docker.io/library/nginx:alpine 2>/dev/null
# Optional: sudo loginctl disable-linger "$USER"
Common mistakes & troubleshooting
The failure modes that actually bite, as a symptom → root cause → confirm → fix playbook. Read it once, keep it open when a rootless service misbehaves.
| # | Symptom | Root cause | Confirm (exact command) | Fix |
|---|---|---|---|---|
| 1 | Container works in SSH session, vanishes after logout | Linger not enabled; user manager torn down at session end | loginctl show-user $USER --property=Linger = no |
sudo loginctl enable-linger $USER |
| 2 | lchown <uid>: invalid argument on pull/run |
subuid/subgid range too narrow (< image’s max UID) | grep "^$USER:" /etc/subuid /etc/subgid (count < 65536) |
Widen to ≥ 65536; podman system migrate |
| 3 | Permission denied reading a bind mount that is world-readable |
SELinux label wrong on the host path | ls -Z /srv/appdata; ausearch -m avc -ts recent |
Mount with :Z (or :z if shared) |
| 4 | Bind mount appears empty/unowned inside the container | Host files owned by real UID, not mapped UID | podman run --rm -v /path:/m alpine ls -ln /m |
Add :U to chown into the namespace, or use a named volume |
| 5 | MemoryMax= / --memory has no effect |
memory controller not delegated to the user slice |
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers |
Ensure cgroup v2 + systemd delegation; reboot after enabling |
| 6 | bind: permission denied on port 80/443 |
Rootless can’t bind privileged ports | podman run -p 80:80 ... fails |
High port + proxy/DNAT, or lower ip_unprivileged_port_start, or AmbientCapabilities |
| 7 | Every request logs the same internal gateway IP | slirp4netns hiding the source IP | podman info --format '{{.Host.NetworkBackend}}'; check container access log |
Switch to pasta (Network=pasta) |
| 8 | .container edits ignored; unit unchanged |
Forgot daemon-reload, or file in the wrong dir, or a typo’d key |
systemctl --user cat weblab.service; check ~/.config/containers/systemd/ |
systemctl --user daemon-reload; fix path/key; check journalctl for generator warnings |
| 9 | podman auto-update never updates a live image |
Missing AutoUpdate=registry, or the container isn’t systemd-managed |
podman inspect --format '{{index .Config.Labels "io.containers.autoupdate"}}' |
Add AutoUpdate=registry; ensure it’s a Quadlet/systemd unit |
| 10 | A bad image update crash-loops instead of rolling back | Healthcheck missing or too shallow, so “healthy” == “process didn’t exit” | podman inspect --format '{{.State.Health.Status}}' = (none) |
Add an honest HealthCmd + HealthStartPeriod |
| 11 | Rootless storage using vfs, disk ballooning |
Kernel/overlay unavailable → fell back to vfs | podman info --format '{{.Store.GraphDriverName}}' = vfs |
Install fuse-overlayfs; on 5.11+ enable native overlay; reset storage |
| 12 | Container-to-container name resolution fails | Containers on the default net, not a shared user-defined network | podman network ls; podman inspect --format '{{.NetworkSettings.Networks}}' |
Put both on a .network/podman network create; aardvark-dns resolves names |
| 13 | newuidmap: write to uid_map failed |
uidmap/shadow-utils missing or helpers not setuid |
ls -l $(command -v newuidmap) (want setuid root) |
Install uidmap/shadow-utils; restore setuid bit |
| 14 | Boot start racy: app up before network/mount ready | Missing ordering in the .container [Unit] |
systemctl --user list-dependencies weblab.service |
Add After=/Requires= (e.g. network-online.target, a mount unit) |
| 15 | podman auto-update.timer present but nothing happens |
Timer not enabled for this user | systemctl --user is-enabled podman-auto-update.timer |
systemctl --user enable --now podman-auto-update.timer |
The expanded reasoning for the three that consume the most hours:
Linger (row 1). This is the number-one first-encounter mistake. The user’s systemd --user manager exists only while they have a login session unless linger is set; log out and the manager — and every --user unit under it, including your containers — is stopped. Everything works perfectly in your SSH session, so it looks fine, and then the box reboots (or you just log out) and the service is gone. loginctl enable-linger <user> is mandatory for any boot-persistent rootless workload, full stop.
subuid width (row 2). Rootless mapping needs enough subordinate IDs to cover whatever the image chowns to. A range of, say, 1000 IDs breaks the moment an image sets a file to UID 1001. The fix is always a ≥ 65536-wide range and a podman system migrate so Podman rebuilds the namespace with the new map. When in doubt, 100000:65536 is the safe default.
cgroup delegation (row 5). On cgroup v2, unprivileged users can only set limits for controllers that systemd has delegated to their slice. If MemoryMax= seems ignored, do not debug Podman — cat the cgroup.controllers of your user slice. If memory is absent, the controller isn’t delegated (a mis-configured host, cgroup v1, or a systemd too old), and no Podman flag will make the limit stick until you fix delegation at the host level.
Best practices
- Enable linger for every boot-persistent user.
loginctl enable-linger <user>is not optional for services meant to survive logout and reboot — it is the first thing to set and the first thing to check when a container “disappears.” - Give service users a ≥ 65536-wide, non-overlapping subuid/subgid range.
100000:65536per user is the safe default; overlapping ranges break isolation, narrow ranges break imagechowns. - Keep the storage driver on
overlay(orfuse-overlayfs), nevervfs. Confirm withpodman info;vfsdeep-copies every layer and eats disk. - Declare every workload as a Quadlet file in Git.
.container/.pod/.network/.volumefiles are your source of truth;systemctl daemon-reloadis the deploy step. Never hand-write[Service] ExecStart=podman run. - Never edit the generated
.service. It lives under/run/.../generated/and is regenerated on every reload; edit the.containerfile and reload. - Prefer named volumes for opaque state; use
:Ufor ownership and:Z/:zfor SELinux only on bind mounts. Reach for a bind mount when you need to inspect or tail the data from the host. - Use pasta where source-IP fidelity or throughput matters. slirp4netns hides the client IP by default; verify the observed source address rather than assuming it.
- Handle privileged ports without running as root — high port plus reverse proxy/DNAT, or a deliberate
ip_unprivileged_port_start, or a scopedAmbientCapabilities=CAP_NET_BIND_SERVICE. - Set
AutoUpdate=registryonly on tags you control, always with an honestHealthCmd. The rollback safety net is exactly as good as your healthcheck; a shallow one lets a broken image through. - Enable
podman-auto-update.timerper user so updates and rollbacks run unattended. - Set resource caps and confirm delegation.
MemoryMax/CPUQuota/TasksMaxin[Service], andcat cgroup.controllersto provememory/cpuare delegated to your user slice. - Drop capabilities to the minimum (
DropCapability=ALL+ a specificAddCapability=), setNoNewPrivileges=true, and use a read-only rootfs where the app allows it. - Run a full reboot test. Everything can pass in your session and still not come back at boot; only a real reboot proves linger + ordering are correct.
Security notes
- Rootless is the isolation. Container UID 0 maps to an unprivileged host UID (100000+), so a container breakout lands an attacker as an account that owns nothing — verify with
podman unshare cat /proc/self/uid_map. This is the single largest security gain over a root daemon. - Do not expose the Docker-compat socket carelessly. If you enable
podman system servicefor tools that needdocker.sock, run it rootless (a per-user socket under$XDG_RUNTIME_DIR), never a root-owned system socket — re-exposing a root socket re-creates the exact finding you moved to Podman to close. - Least-privilege capabilities. Start from
DropCapability=ALLand add back only what the app needs (NET_BIND_SERVICEfor a low port and nothing else). Combine withNoNewPrivileges=trueto block setuid escalation andReadOnly=truefor an immutable rootfs. - SELinux stays enforcing. Use
:Z(private) or:z(shared) on bind mounts so the container gets a correct type label; never disable SELinux to “make the mount work,” and never relabel a system path like/etcor$HOMEwith:z. On Ubuntu/Debian, the AppArmor profile is the equivalent confinement. - Pin images by digest for anything sensitive. Reserve
AutoUpdate=registryfor tags you re-point deliberately, and pin:prodto an immutable digest only after a canary clears — so an upstream tag move can’t silently ship into your fleet. - Use Podman secrets, not environment variables, for credentials.
Secret=mounts a secret as a file rather than leaking it into the process environment (visible in/proc/<pid>/environandpodman inspect). - Keep the healthcheck free of secrets and internal topology. It returns a status, not a system map; a
HealthCmdshould not print connection strings or dependency hostnames into the journal.
| Control | Quadlet / mechanism | Secures against |
|---|---|---|
| Rootless user namespace | (default rootless) | Container breakout → host root |
| Drop capabilities | DropCapability=ALL + AddCapability= |
Ambient kernel privileges |
| No new privileges | NoNewPrivileges=true |
setuid/setgid escalation |
| Read-only rootfs | ReadOnly=true (+ tmpfs for scratch) |
Persistence / tampering in the container |
| SELinux relabel | :Z / :z on mounts |
Cross-container / host file access |
| Secrets as files | Secret= |
Credential leak via env/inspect |
| Digest pinning | Image=...@sha256:... |
Supply-chain tag-move surprises |
| Rootless-only socket | podman system service (user) |
Re-creating the root-socket compromise path |
Cost & sizing
Podman itself is free and open-source, and rootless Podman has negative incremental cost versus a Docker daemon — you remove the always-on dockerd (and its containerd), reclaiming roughly 150–250 MB of resident memory per host that the daemon and its shims held whether or not containers were running. On a fleet of 60 edge nodes that is measurable RAM back, and one fewer service to patch, monitor, and reason about. There is no per-node licence, no control-plane to run (unlike Kubernetes), and no managed-service bill — the entire model is “systemd you already run + a container binary.”
The real sizing questions are host-level, not licence-level. Memory: size each host for the sum of your containers’ MemoryMax plus headroom; because there is no daemon reserving memory, your budget is almost entirely the workloads. CPU: CPUQuota in aggregate should leave slack for pasta’s userspace networking, which costs some CPU that a kernel bridge would not — budget a few percent per busy container for the network stack. Disk: overlay storage under $HOME/.local/share/containers grows with images and container layers; a podman system prune on a timer keeps it bounded, and staying off vfs (which deep-copies layers) is the single biggest disk saver. Networking throughput: if a workload needs line-rate networking, pasta’s userspace ceiling may push you to rootful Podman with a kernel bridge or to a different host design — that is an architecture cost, not a dollar cost.
| Cost / sizing driver | What drives it | How to right-size | Rough figure |
|---|---|---|---|
| Per-host memory | Sum of container MemoryMax + headroom |
Cap each container; no daemon to budget | ~150–250 MB reclaimed vs Docker daemon |
| Network CPU overhead | pasta/slirp userspace stack | Leave a few % CPU slack per busy container | Small; workload-dependent |
| Overlay disk | Images + layers under $HOME |
podman system prune on a timer; avoid vfs |
Bounded by prune cadence |
| Operational headcount | Fleet size / node count | Quadlet-in-Git + Ansible reload handler | Fewer moving parts than a K8s cluster |
| Licence | (none) | — | ₹0 / $0 — OSS |
For most single-host and edge fleets, the “sizing” exercise is simply: pick MemoryMax/CPUQuota/TasksMax per container from observed usage (via podman stats), sum them for the node, add headroom, and enable a prune timer. Compared with running a Kubernetes cluster for the same workloads, the cost delta is enormous — no etcd, no control plane, no worker-node overhead — which is exactly why Podman + Quadlet is the right tool when you have a handful of nodes rather than a data centre.
Interview & exam questions
1. What does “daemonless” actually mean for Podman, and what’s the process tree of a running container? There is no long-lived privileged service; podman run forks conmon, which execs an OCI runtime (crun/runc) that becomes the container’s first process. Under systemd, conmon is a child of the .service unit, so systemd and the kernel — not a daemon — supervise the container. Kill the Podman binary and running containers keep running.
2. How does rootless Podman let an unprivileged user run a container “as root”? Via a user namespace: the user’s login UID becomes UID 0 inside the namespace, while the container’s other UIDs map to a block of host UIDs reserved in /etc/subuid//etc/subgid (e.g. 100000–165535). Container root is therefore an unprivileged host UID that owns nothing — confirmable with podman unshare cat /proc/self/uid_map.
3. Why is loginctl enable-linger required, and what breaks without it? A user’s systemd --user manager normally exists only during an active login session; without linger it’s torn down at logout, stopping every --user unit including your containers. Linger keeps the manager (and its services) running with no session, which is what makes rootless containers start at boot and survive logout.
4. slirp4netns vs pasta — what’s the difference and when do you pick each? Both are userspace rootless network stacks (rootless can’t create a host bridge without CAP_NET_ADMIN). slirp4netns is the original — reliable but slower and it hides the client source IP by default. pasta is the Podman 5 default — faster, lower latency, and it preserves the real source IP. Pick pasta when source-IP fidelity (logs, allowlists, fraud models) or throughput matters.
5. What is Quadlet, and why not hand-write a [Service] ExecStart=podman run unit? Quadlet is a systemd generator that reads declarative .container/.pod/.network/.volume files (in ~/.config/containers/systemd/) and synthesises real .service units on daemon-reload. It replaces the removed podman generate systemd. You get a declarative, Git-able source of truth and never hand-maintain the podman run line; the generated unit is read-only under /run/.../generated/.
6. Why must cgroup v2 controllers be “delegated” for rootless resource limits, and how do you confirm it? On cgroup v2, an unprivileged user can only set limits for controllers systemd has delegated to their slice (modern systemd delegates cpu, memory, pids, io via user@.service). If MemoryMax= does nothing, cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers — if memory is absent, delegation is the problem, not Podman.
7. What does AutoUpdate=registry do, and why does a real HealthCmd matter for it? It labels the container io.containers.autoupdate=registry so podman auto-update polls the registry for a newer digest on the same tag, pulls it, and restarts the (systemd-managed) unit — with automatic rollback to the previous digest if the container doesn’t come up healthy. Rollback is keyed on the unit reaching a healthy state, so without an honest HealthCmd, “healthy” just means “the process didn’t exit” and a broken image can slip through.
8. When mounting a host directory into a rootless container on RHEL, what do :U, :z, and :Z each do? :U recursively chowns the source to the container’s mapped UID/GID so it’s writable; :z applies a shared SELinux label (several containers may use the path); :Z applies a private label (one container only). Wrong or missing labels are the classic “Permission denied on a world-readable file” on RHEL-family hosts.
9. How does a pod differ from independent containers, and where do you publish ports? A pod shares one network namespace (one IP, one localhost) via an infra container; members reach each other over 127.0.0.1, and you publish ports on the pod, not the members. Independent containers each have their own netns and resolve each other by name only if placed on a shared user-defined network (via aardvark-dns).
10. How do you migrate a Docker fleet to rootless Podman with minimal risk? Inventory every docker run/compose flag; translate each to a peer-reviewed .container file in Git; stand it up on a drained canary node and confirm the healthcheck; roll node-by-node behind the load balancer keeping ≥2 instances of each service; enable auto-update on a controlled tag with rollback (canary first); then decommission dockerd on migrated nodes. alias docker=podman and the Docker-compat socket ease the CLI/tooling transition.
11. Your container works over SSH but is gone after you log out and back in. What happened? Linger isn’t enabled, so the systemd --user manager (and its containers) was stopped at session end. Confirm with loginctl show-user $USER --property=Linger and fix with sudo loginctl enable-linger $USER.
12. A custom image fails to pull rootless with lchown ... invalid argument. Why, and how do you fix it? The user’s subuid/subgid range is narrower than a UID the image chowns to, so the mapping can’t satisfy it. Widen the range to ≥ 65536 (usermod --add-subuids/--add-subgids, e.g. 100000-165535) and run podman system migrate to rebuild the namespace.
These map to Linux/DevOps container competencies rather than a single vendor exam, but they overlap the Red Hat RHCSA/RHCE container objectives (running containers as systemd services, rootless containers, podman/skopeo/buildah), the LFCS/LFCE container units, and any container-security or CIS-benchmark work that scrutinises the Docker socket.
| Question theme | Where it maps |
|---|---|
| Rootless user namespaces, subuid/subgid | RHCSA/RHCE containers; container security |
| Containers as systemd services (Quadlet) | RHCSA/RHCE; LFCS |
| cgroup v2 delegation & limits | LFCE; performance/isolation |
| pasta vs slirp4netns networking | Container networking; RHCE |
SELinux :Z on volumes |
RHCSA/RHCE security; SELinux |
| Docker socket risk / migration | CIS benchmarks; container security review |
Quick check
- In the process tree of a rootless container run under systemd, what sits directly above the container’s first process, and what sits above that?
- Your rootless service runs fine in your SSH session but is gone after the host reboots. What one command fixes it, and what does it do?
MemoryMax=512Min your.containerfile appears to be ignored. What’s the single most likely cause, and how do you confirm it?- You mount
/srv/appdatainto a rootless container on RHEL and get “Permission denied” even though the files are world-readable. What mount option is missing, and what’s the difference between:zand:Z? - Why does
podman auto-update’s rollback need a realHealthCmd, and what happens without one?
Answers
- Directly above the container’s first process is conmon (the per-container monitor holding its stdio and reaping it); above conmon is the systemd
--userservice unit that Quadlet generated (kept alive by the user’ssystemd --usermanager under linger). There is nodockerdanywhere in the chain. sudo loginctl enable-linger <user>. Without linger, the user’ssystemd --usermanager (and every--userunit, including your containers) is torn down when the last login session ends; linger keeps it running with no active session, so the service starts at boot and survives logout.- The
memorycgroup v2 controller isn’t delegated to your user slice. Confirm withcat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers; ifmemoryisn’t listed, delegation (host/systemd config) is the problem, not Podman. - The SELinux relabel option (
:Zor:z) is missing.:zapplies a shared label so multiple containers can use the path;:Zapplies a private label for exactly one container. On RHEL-family hosts an unlabelled bind mount is denied regardless of Unix permissions. - Rollback triggers when an updated unit fails to reach a healthy state. Without an honest
HealthCmd, “healthy” degrades to “the process didn’t immediately exit,” so a subtly broken image (starts, serves errors) is considered fine and rollback never fires — you ship the bad image.
Glossary
- Podman — a daemonless container engine that runs containers via fork-exec (conmon → OCI runtime) rather than through a privileged always-on daemon; CLI-compatible with Docker.
- conmon — the small per-container monitor process that holds a container’s stdio, writes its logs, and reaps it; under systemd it’s a child of the container’s
.serviceunit. - OCI runtime — the low-level tool (
crunby default, orrunc) that actually creates and starts the container from an OCI spec. - Rootless mode — running containers as an unprivileged user via a user namespace, so container UID 0 maps to an unprivileged host UID.
- User namespace — a Linux namespace that remaps UIDs/GIDs; your login UID becomes root inside it while mapping to unprivileged host UIDs outside.
- subuid / subgid — host UID/GID ranges delegated to a user in
/etc/subuid//etc/subgid(e.g.user:100000:65536); the pool a rootless container maps its non-root IDs into. - Linger — a per-user setting (
loginctl enable-linger) that keeps the user’ssystemd --usermanager running with no active login session, so--userservices start at boot and survive logout. - slirp4netns — the original userspace rootless network stack; reliable but lower-throughput and hides the client source IP by default.
- pasta — the modern userspace rootless network stack (from
passt), the Podman 5 default; faster and preserves the real client source IP. - netavark / aardvark-dns — Podman’s network backend and its companion DNS server that resolves container names on user-defined networks.
- cgroup v2 delegation — systemd handing control of the
cpu/memory/pids/iocontrollers to a user slice (viauser@.service), which is what lets rootless containers enforce resource limits. - Pod — a group of containers sharing a network namespace (one IP, one
localhost) via an infra container; ports are published on the pod, and it maps to a KubernetesPod. - Quadlet — the systemd generator that turns declarative
.container/.pod/.network/.volumefiles into real.serviceunits ondaemon-reload; successor to the removedpodman generate systemd. .containerfile — a declarative Quadlet unit (in~/.config/containers/systemd/or/etc/containers/systemd/) with a[Container]section from which Quadlet synthesises thepodman runcommand.- AutoUpdate=registry — a Quadlet key (setting the
io.containers.autoupdate=registrylabel) that makespodman auto-updatepoll the registry, pull a newer same-tag digest, restart the unit, and roll back if it doesn’t come up healthy. :U/:z/:Z— bind-mount options::Uchowns the source into the container’s mapped UID;:zapplies a shared SELinux label;:Za private one.- Named volume — a Podman-managed volume (created via
podman volume createor a.volumeunit) that carries correct ownership and SELinux labels automatically, unlike a raw bind mount. - Docker-compat socket — an optional Podman-provided API socket (
podman system service) that emulates/var/run/docker.sockfor tools hard-wired to Docker; run it rootless, never as a root system socket.
Next steps
You can now run rootless containers as reboot-surviving, resource-capped, auto-updating systemd services defined entirely in Git. Build outward:
- Next: Mastering systemd: Units, Timers, Resource Control, and Service Hardening — everything Quadlet generates is a systemd unit; deepen
[Service]hardening, timers, and cgroup accounting. - Related: Running SELinux in Enforcing Mode: Troubleshooting and Writing Custom Policy — master the
:Z/:zrelabelling andavc: deniedtriage that rootless volumes depend on. - Related: Authoring AppArmor Profiles: Confining Services on Ubuntu and Debian — the Ubuntu/Debian equivalent of the SELinux confinement above.
- Related: Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers — tune the cgroup v2 and kernel knobs that back your container resource limits.
- Related: Azure App Service vs Container Apps vs AKS: Choose the Right Compute — when a single-host Podman fleet should graduate to a managed platform.
- Related: Deploy Harbor Registry on Kubernetes with Trivy Scanning, Replication, and Cosign Signing — a private registry and signed-image supply chain to feed your
AutoUpdate=registrypipeline.