Servers Administration

Running Rootless Containers in Production with Podman and Quadlet

The container conversation defaulted to “just run a daemon as root” for a decade, and most production fleets are still paying interest on that decision. Podman flips the model: no long-lived daemon, no root-owned socket, and containers that run entirely inside an unprivileged user’s namespace. A podman run is a plain fork-exec — your shell forks conmon, conmon execs an OCI runtime (crun or runc), and the container is a child of your process tree, not a client request to a privileged service. Pair that with Quadlet — the systemd generator that turns declarative .container files into real units — and containers become first-class systemd services, supervised by PID 1, with restart policies, resource caps, ordering dependencies, and journald logging for free. This is a walk through standing that up the way a platform team should, not the way a five-minute quickstart does.

The pain this removes is concrete. A root Docker daemon is a single point of total host compromise: whoever can reach /var/run/docker.sock can mount the host root filesystem into a container and walk away as root, which is why every serious security assessment flags it. Rootless Podman collapses that blast radius — a remote code execution inside a container lands an attacker as an unprivileged UID with no capabilities and no path to the host. And because the container is a systemd unit rather than a child of a daemon, “restart policy,” “resource limit,” and “starts at boot” stop being Docker-specific concepts and become the same Restart=, MemoryMax=, and WantedBy= you already use for every other service on the box. You stop running a second init system (the daemon) next to systemd.

By the end of this article you will understand the daemonless architecture precisely — where conmon sits, why there is no PID 1 inside the container by default, how a crash propagates to systemd — and you will be able to stand up a rootless service that survives reboots, updates itself from a registry with automatic rollback on a failed healthcheck, keeps the real client source IP through pasta, enforces memory and CPU limits through delegated cgroup v2 controllers, and lives entirely as version-controlled .container files. Everything below assumes a recent Podman (v4.4+ for Quadlet, v5.x for pasta-by-default networking) on a cgroup v2 host. Confirm with podman version and stat -fc %T /sys/fs/cgroup/ — you want cgroup2fs. Quadlet shipped in Podman 4.4 and is the supported successor to the now-removed podman generate systemd.

What problem this solves

Three problems, really, and they compound. The first is privilege. Docker’s default is a root daemon that owns every container; your CLI is a thin client to a root-owned Unix socket. Anyone in the docker group is effectively root on the host, because docker run -v /:/host hands them the host filesystem. Security teams issue hard controls against this — “no root-owned container daemon,” “no privileged ports bound by application processes” — and a root Docker daemon simply cannot satisfy them. Rootless Podman does, by construction: your login UID becomes UID 0 inside a user namespace whose other IDs map to a block of unprivileged host UIDs that belong to nobody.

The second is supervision drift. A Docker daemon is a second supervisor running beside systemd, with its own restart semantics (--restart=always), its own logging (json-file driver, log rotation you configure separately), and its own lifecycle that systemd does not understand. When the box reboots, the daemon starts your containers — but ordering against non-container services (a mounted filesystem, a network target, a database on the same host) is awkward, and the daemon’s health is now a dependency of your app’s health. Podman + Quadlet erases the second supervisor: every container is a normal systemd unit, so After=, Requires=, Restart=on-failure, journalctl -u, and boot ordering all work exactly as they do for nginx.service or postgresql.service.

The third is drift between “works on my machine” and “runs in the fleet.” Hand-written docker run lines rot in shell scripts and READMEs; nobody can reconstruct the exact flags a running container was started with. Quadlet makes the .container file the single declarative source of truth — checked into Git alongside your Ansible or Terraform — and the systemd unit is generated from it on daemon-reload. It is GitOps on a single host: edit the file, reload, done. Who hits all this hardest: teams on RHEL/Fedora/CentOS Stream where Podman is the default and Docker is not shipped; anyone with a PCI/CIS mandate against the Docker socket; edge and bare-metal fleets where “systemd-native” beats “run a Kubernetes cluster”; and anyone tired of the daemon being the thing that goes down at 3 a.m.

To frame the whole field before the deep dive, here is the shape of what changes when you move from a root Docker daemon to rootless Podman + Quadlet:

Concern Root Docker daemon Rootless Podman + Quadlet Why it matters
Privilege at runtime Container runs as root (unless userns remapped) Runs as your mapped UID; container root == unprivileged host UID RCE in a container ≠ root on host
Supervisor dockerd (a second init) systemd (the one you already run) One lifecycle, one restart policy, one log stream
Boot persistence Daemon starts containers systemd unit + linger Ordering against other services works
Source of truth Shell scripts / docker run lines .container files in Git Reproducible, reviewable, declarative
Socket exposure root-owned /var/run/docker.sock none by default; optional per-user socket Removes the classic host-compromise path
Logs json-file driver + your rotation journald automatically journalctl -u, retention via journald config
Networking (rootless) n/a (root bridge) pasta / slirp4netns userspace stack No CAP_NET_ADMIN needed on host
Updates docker pull + restart, no rollback podman auto-update with healthcheck-gated rollback Safe unattended updates

Learning objectives

By the end of this article you can:

Prerequisites & where this fits

You should be comfortable with Linux service administration and systemd fundamentals — units, systemctl, journalctl, targets, and the difference between the system manager and a --user manager. Container basics help: images, layers, an entrypoint, port publishing, bind mounts, and what an OCI runtime does. You do not need prior Podman experience; if your muscle memory is Docker, most of it transfers directly (alias docker=podman covers the majority of the CLI). You should be able to edit /etc/subuid, run loginctl, and read podman info output. A cgroup v2 host is assumed — every current RHEL 9/10, Fedora, Debian 12+, and Ubuntu 22.04+ boots cgroup v2 by default.

This sits in the Linux server administration track and leans hard on systemd. If the [Service]/[Unit]/[Install] sections, timers, or resource control feel shaky, read Mastering systemd: Units, Timers, Resource Control, and Service Hardening first — Quadlet is a systemd generator, so everything it produces is a systemd unit. On RHEL-family hosts the volume-labelling story is pure SELinux; Running SELinux in Enforcing Mode: Troubleshooting and Writing Custom Policy explains the :Z relabel you will use constantly, and on Ubuntu/Debian the equivalent confinement lives in Authoring AppArmor Profiles: Confining Services on Ubuntu and Debian. Resource limits ride on cgroup v2; the kernel-side tuning is in Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers. And if your workloads are cloud-bound rather than single-host, the compute-model decisions live in Azure App Service vs Container Apps vs AKS: Choose the Right Compute and AWS ECS vs EKS vs Fargate: Choose Your Container Path.

A quick map of who owns what during a rootless-Podman incident, so you escalate to the right layer:

Layer What lives here Typical owner Failure it causes
Identity ranges /etc/subuid, /etc/subgid Platform / IAM lchown invalid argument, no rootless at all
systemd user manager linger, --user units, delegation Platform Container vanishes on logout; no boot start
Quadlet generator .container/.pod/.network/.volume App + platform (in Git) Unit not generated; typo’d key ignored
Networking pasta/slirp4netns, netavark, aardvark-dns Platform / network No egress, wrong source IP, port bind denied
Storage / SELinux overlay driver, :Z labels, subuid chown Platform Permission denied, avc: denied in audit log
Image supply registry, digests, auto-update App + pipeline Bad push crash-loops; auto-update rollback

Core concepts

Six mental models make every later section obvious.

There is no daemon — the container is a child of your process, or of a systemd unit. Docker’s dockerd is a privileged, always-on service that owns every container; docker run sends it a request and returns. Podman has nothing like it. podman run forks a small supervisor called conmon (the container monitor), which execs the OCI runtime (crun by default on modern systems, or runc), which becomes the container’s first process. conmon holds the container’s stdio, writes logs, and reaps the child. When you launch under systemd, conmon becomes a child of the unit, so the kernel and systemd — not a daemon — are the source of truth for “is this container alive.” Kill the machine’s Podman binary and running containers keep running; there is no daemon whose death takes them all down.

Rootless works by mapping a range of host UIDs into a user namespace. Your single login UID (say 1000) becomes UID 0 inside the container’s user namespace. The container’s other UIDs (1–65535) map to a block of host UIDs that the system reserved for you in /etc/subuid//etc/subgid — typically 100000–165535. So container root is host UID 100000, an account that owns nothing else. A setuid-helper called newuidmap/newgidmap (from the shadow-utils/uidmap package) performs the mapping at container start. This is why an escaped process is harmless on the host: outside the namespace it is UID 100000 with no capabilities over anything real.

cgroups v2 is what lets rootless enforce limits — but only if controllers are delegated. On cgroup v1, unprivileged users could not set CPU/memory limits at all. cgroup v2 plus systemd’s delegation changes that: systemd hands your user slice control of the cpu, memory, pids, and io controllers via user@.service, so a rootless container can actually cap itself. If MemoryMax= silently does nothing, the memory controller was not delegated — that is the first thing to check, not a Podman bug.

Networking has no host bridge — a userspace stack fakes it. Creating a veth pair against a host bridge needs CAP_NET_ADMIN on the host, which a rootless user does not have. So rootless outbound traffic and published ports flow through a userspace network stack running in your namespace: historically slirp4netns, and on Podman 5 the faster, source-IP-preserving pasta (from the passt project). The container-facing side is managed by netavark (the network backend) with aardvark-dns providing name resolution between containers on a user-defined network.

A pod is a shared-namespace group borrowed from Kubernetes. A pod holds an infra container that keeps a set of namespaces (network, and optionally others) open, so member containers share one IP, one localhost, and one port space. You publish ports on the pod, not the members; members reach each other over 127.0.0.1. This is the sidecar pattern — proxy, app, metrics exporter — co-scheduled and co-supervised, and it is also the bridge to real Kubernetes via podman kube generate/play.

Quadlet is a systemd generator, not a runtime. You do not hand-write [Service] ExecStart=podman run ... and you do not use the removed podman generate systemd. You write a declarative .container file; systemd’s generator infrastructure runs Quadlet on every daemon-reload, and Quadlet synthesises a real .service unit (with the full podman run command) into a read-only generated directory. Your source of truth is the .container file in Git; the deploy step is systemctl daemon-reload.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side.

Concept One-line definition Where it lives Why it matters
conmon Per-container monitor holding stdio + reaping the child Child of your shell/unit The “supervisor” in a daemonless world
OCI runtime crun/runc — actually creates the container Exec’d by conmon The kernel-facing layer
User namespace Remaps UIDs/GIDs so you are “root” inside only Per rootless container The whole security model
subuid / subgid Host ID ranges delegated to your user /etc/subuid, /etc/subgid Without them, rootless can’t map IDs
Linger Keeps your --user systemd manager alive with no login loginctl enable-linger Boot-persistent rootless services
slirp4netns Original userspace rootless network Per container Reliable; slower; hides source IP
pasta Modern userspace rootless network (Podman 5 default) Per container Fast; preserves source IP
netavark / aardvark-dns Network backend + container DNS Host-side Name-based container-to-container resolution
cgroup v2 delegation Controllers handed to your user slice user@.service Rootless CPU/memory/pids limits
Pod Shared-namespace group with an infra container Podman/host Sidecars; one IP; K8s bridge
Quadlet systemd generator for .container/.pod/… files ~/.config/containers/systemd/ Declarative, GitOps-able units
AutoUpdate=registry Marks an image for podman auto-update Quadlet [Container] key Unattended updates + rollback
:Z / :z SELinux relabel of a bind mount (private / shared) Volume mount option Access on RHEL-family hosts

Podman versus Docker: daemonless and rootless by design

The architectural difference is not cosmetic, and it changes how you operate the system. Docker runs a privileged daemon (dockerd, which itself talks to containerd and containerd-shim) that owns every container; your docker CLI is a thin client speaking HTTP over a root-owned Unix socket. Compromise the socket, compromise the host — a container started with -v /:/host --privileged is game over. Podman has no daemon. podman run forks conmon, conmon execs crun/runc, and the container is a child of your process tree. When you wire that into systemd, conmon becomes a child of the unit, and the kernel — not a daemon — is the source of truth for liveness.

Property Docker (default) Podman (rootless)
Central daemon dockerd runs as root None
Container ownership Child of the daemon Child of your shell or systemd unit
Privilege at runtime Root unless userns-remapped Your UID, mapped via subuid/subgid
Supervision Daemon --restart policy systemd unit (Restart=)
Socket exposure root-owned /var/run/docker.sock none by default; optional per-user
Build tool docker build (BuildKit) podman build / Buildah
Compose docker compose (native) podman compose (wraps external), or Quadlet/kube play
Boot start daemon restores containers systemd + linger
Logs json-file/journald driver journald (native, via conmon)
Kubernetes YAML third-party tools podman kube generate / play built in

The practical wins compound. A crashed container is restarted by systemd with the same policy you use for every other service; logs land in the journal so journalctl --user -u myapp just works; boot ordering against a mounted filesystem or a local database uses the same After=/Requires= you already know; and an RCE inside a container lands an attacker as an unprivileged UID with no capabilities and no route to the host. The costs are honest too: rootless networking is a userspace stack (a throughput and feature ceiling versus a kernel bridge), some images that assume they can bind port 80 or write to root-owned paths need adjustment, and docker compose files map to Podman through a wrapper or a rewrite to Quadlet/Kube YAML rather than natively.

Where Podman deliberately stays Docker-compatible: the CLI surface (alias docker=podman covers the common verbs), the image format and registries (it pulls the same OCI images from Docker Hub, Quay, GHCR, and private registries), and an optional Docker-compatible API socket so tools hard-wired to docker.sock — Testcontainers, some CI runners — keep working. Buildah is the dedicated image-build tool (Podman shells out to it for podman build); Skopeo is the companion for copying and inspecting images between registries without pulling them.

Docker command Podman equivalent Notes
docker run podman run Near-identical flags
docker build podman build / buildah bud Buildah under the hood
docker compose up podman compose up Wraps external compose; or use Quadlet
docker ps / logs / exec podman ps / logs / exec Same
docker system prune podman system prune Same
docker save / load podman save / load Same
--restart=always systemd Restart=always in the .container The idiomatic replacement
docker sock for tools podman system service (Docker-compat socket) Opt-in, per-user

Setting up rootless mode: subuid, subgid, and lingering

Rootless containers map a range of host UIDs/GIDs into the container’s user namespace. Your login UID becomes root (UID 0) inside the container; the container’s other UIDs map to a block of unprivileged host UIDs reserved for you. Those ranges live in /etc/subuid and /etc/subgid, one line per user in the form name:start:count.

On a fresh service user, allocate a non-overlapping range of at least 65536 IDs. On most distributions creating a user with useradd already assigns a subuid/subgid block; verify it, and only add one if missing:

# Verify the user already has a range (many distros do this at useradd time)
grep '^svc-api:' /etc/subuid /etc/subgid
# /etc/subuid:svc-api:100000:65536
# /etc/subgid:svc-api:100000:65536

# If missing, grant a 65536-wide subordinate range (auto-picks the next free block)
sudo usermod --add-subuids 200000-265535 --add-subgids 200000-265535 svc-api

# After editing the maps, force Podman to re-read them
sudo -u svc-api podman system migrate

The 65536 width matters: it lets a container reference the full standard UID space (0–65535). Too narrow and images that chown to high UIDs fail with lchown ... invalid argument — the single most common “why won’t this image run rootless” error. Ranges must not overlap between users; an overlap means two users share host UIDs, defeating the isolation.

Symptom / need Cause Fix
lchown <uid>: invalid argument on podman pull/run subuid/subgid range narrower than the image’s max UID Widen range to ≥ 65536; podman system migrate
there might not be enough IDs available No subuid/subgid line for the user at all Add a range with usermod --add-subuids/--add-subgids
Changed the maps, still broken Podman cached the old mapping podman system migrate (recreates the userns)
Two users’ containers clobber each other’s files Overlapping ranges in /etc/subuid Assign disjoint 65536-wide blocks per user
newuidmap: write to uid_map failed uidmap/shadow-utils missing or not setuid Install the package; ensure newuidmap/newgidmap are setuid root

For services that must survive logout and start at boot, enable lingering. Without it, the user’s systemd --user instance is torn down when their last session ends, taking every container with it. This is the single most-forgotten step: the container works in your SSH session, you log out, and it vanishes.

# Keep the user's `systemd --user` manager running with no active login session
sudo loginctl enable-linger svc-api

# Confirm
loginctl show-user svc-api --property=Linger
# Linger=yes

Root/system-wide Podman is a real alternative and is sometimes correct — a system service that genuinely needs a privileged port or host devices, running under the system systemd manager. The trade-off is that “rootful Podman” gives back the privilege you were trying to shed; you keep the daemonless model but lose the unprivileged-UID blast-radius win. Choose per workload:

Mode Manager Files live in subuid/subgid Linger needed? Use when
Rootless (user) systemd --user ~/.config/containers/systemd/ Required per user Yes, for boot The default; least privilege
Rootful (system) system systemd /etc/containers/systemd/ Not used (real root) No Needs host devices / true privileged ports
Rootless as a system unit system systemd running a --user-style unit for an account /etc/containers/systemd/ with User= Required for that user No (system-managed) Fleet automation wanting boot start without login

User namespaces, storage drivers, and volumes

Inside the namespace the container believes it is root; on the host it is not. Prove it:

podman run --rm docker.io/library/alpine id
# uid=0(root) gid=0(root) ...   <- inside the container

podman unshare cat /proc/self/uid_map
#          0     100000      65536   <- container UID 0 == host UID 100000

That second line is the entire security model in three numbers: container UID 0 is host UID 100000, an account that owns nothing else on the system. podman unshare drops you into the same user namespace your containers use — invaluable for fixing ownership on rootless storage from the host side.

Storage driver

Rootless Podman defaults to overlay via the kernel’s unprivileged overlayfs (kernel 5.11+) or, failing that, fuse-overlayfs in userspace. You do not need vfs on any modern kernel and you should avoid it — vfs deep-copies every layer on every container and wastes enormous disk. Confirm and know what you are looking at:

podman info --format '{{.Store.GraphDriverName}}'
# overlay
Driver How it works Rootless support Disk efficiency Use when
overlay (native) Kernel unprivileged overlayfs (5.11+) Yes (best) Copy-on-write layers Default on modern kernels
fuse-overlayfs Userspace overlay via FUSE Yes CoW, slight FUSE overhead Older kernels lacking unprivileged overlay
vfs Full copy of every layer Yes Terrible (deep copy) Only as a last-resort fallback
btrfs / zfs Filesystem-native snapshots Limited rootless Excellent Root-mode on btrfs/zfs hosts

Rootless storage lives under $HOME/.local/share/containers/storage, not /var/lib/containers. Put $HOME on a filesystem that supports the driver (xfs or ext4 are fine; some network filesystems are not, and NFS in particular fights overlay).

Volumes, the :U trick, and SELinux :Z

Because of the UID shift, a host bind mount owned by your real UID appears unowned inside the container (the files map to a UID the container doesn’t have). Two mount-option families fix the two distinct problems — ownership and SELinux labelling:

# :U  -> chown source to the container user's mapped UID/GID
# :Z  -> apply a PRIVATE SELinux label so this one container can access it
podman run --rm \
  -v /srv/appdata:/data:U,Z \
  docker.io/library/alpine touch /data/written-by-container
Mount option What it does When to use Gotcha
(none) Mounts as-is Named volumes; already-correct ownership Bind mount often shows as unowned inside
:U Recursively chowns source to mapped UID Bind mount the container must write to Recursive chown is O(files); slow on huge trees
:z Shared SELinux relabel A dir shared by several containers Over-broad label if you only need one container
:Z Private SELinux relabel A dir for exactly one container If two containers use it, the second is denied
:ro Read-only mount Config, certs, secrets Combine as :ro,Z; app can’t write
:z on /etc or $HOME Relabels a system path Never Recursively relabels the host path — dangerous

For state you do not need to inspect from the host, prefer named volumes (podman volume create). They live inside Podman storage, already carry the correct mapped ownership, and sidestep the relabel dance entirely. Quadlet’s .volume file is the declarative way to create them.

Named volume Bind mount
Ownership under rootless Correct automatically Often needs :U
SELinux label Handled Needs :z/:Z
Inspect from host Awkward (inside Podman storage) Trivial (real path)
Portability across hosts Higher (Podman manages it) Path must exist identically
Best for Opaque app state (DB data dirs) Config you edit, logs you tail

Grouping containers into pods and shared namespaces

A pod is Podman’s borrowing from Kubernetes: a set of containers that share a network namespace (one IP, one localhost, one port space) and optionally other namespaces (--share ipc,uts etc.). The pod owns an infra container — a tiny placeholder (pause-like) that holds the namespaces open so individual member containers can start and stop without tearing the shared network down.

# Create a pod, publishing 8080 on the host ONCE -- all members share it
podman pod create --name web --publish 8080:80

# A reverse proxy and the app it fronts, both in the pod
podman run -d --pod web --name proxy docker.io/library/nginx:alpine
podman run -d --pod web --name cache docker.io/library/redis:7-alpine

# Inside the pod, 'proxy' reaches 'cache' on 127.0.0.1:6379 -- same netns
podman pod inspect web --format '{{.InfraContainerID}}'

The key networking insight: you publish ports on the pod, not the member containers, because they share one network namespace. Containers in a pod talk to each other over localhost. This maps cleanly onto the sidecar pattern — proxy, app, and a metrics exporter co-scheduled and co-supervised — and it is the natural unit that podman kube generate turns into a Kubernetes Pod manifest.

Grouping Network model Container-to-container Publish ports on Best for
Standalone containers Each its own netns (rootless) Via a shared user-defined network + DNS name Each container Independent services
Pod Shared netns (one IP) localhost The pod Sidecars; tightly-coupled group
User-defined network Separate netns, common bridge (rootless: netavark) By container name (aardvark-dns) Each container Microservices that resolve each other by name

Quadlet: generating systemd units from .container files

This is the heart of a production setup. You write a declarative .container file and let the Quadlet generator translate it into a real unit at boot and on daemon-reload. Rootless Quadlet files go in ~/.config/containers/systemd/; root/system-wide units go in /etc/containers/systemd/ (or /usr/share/containers/systemd/ for vendor-shipped ones). Create ~/.config/containers/systemd/myapp.container:

[Unit]
Description=My application container
After=network-online.target
Wants=network-online.target

[Container]
Image=ghcr.io/acme/myapp:1.4.2
ContainerName=myapp
PublishPort=8080:8080
Environment=LOG_LEVEL=info
Volume=myapp-data.volume:/var/lib/myapp:Z
# Drop all capabilities, add back only what the app needs
DropCapability=ALL
AddCapability=NET_BIND_SERVICE
# Mark this image as a candidate for `podman auto-update`
AutoUpdate=registry
HealthCmd=curl -fsS http://localhost:8080/healthz || exit 1
HealthInterval=30s
HealthRetries=3
HealthStartPeriod=20s

[Service]
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Note there is no [Service] ExecStart — Quadlet synthesises the podman run line from the [Container] section. The myapp-data.volume:/... reference points at a .volume file you create alongside it:

# ~/.config/containers/systemd/myapp-data.volume
[Volume]
# Quadlet names the real Podman volume "systemd-myapp-data"
# (add Driver=, Options=, etc. here if needed)

Now reload and start. The unit name is the filename stem plus .service:

# Re-run the generator (Quadlet is invoked by daemon-reload)
systemctl --user daemon-reload

# Start it -- the unit is named after the .container file
systemctl --user start myapp.service
systemctl --user status myapp.service

Quadlet units are generated read-only into /run/user/<uid>/systemd/generated/ (rootless) or /run/systemd/generated/ (system); you never edit those. You edit the .container file and daemon-reload. This is the declarative loop that makes Podman feel like GitOps on a single host — the .container files are your source of truth and live in version control alongside your Ansible/Terraform.

The four Quadlet unit types

Quadlet is not just .container. Four file types compose into a full application, and the generator wires the ordering dependencies between them for you (a .container referencing a .volume gets an automatic After=/Requires= on the generated volume unit).

File type Generates Key section Purpose
.container <name>.service [Container] One container as a systemd service
.pod <name>-pod.service [Pod] A pod; point .containers at it with Pod=
.network <name>-network.service [Network] A user-defined network (netavark + DNS)
.volume <name>-volume.service [Volume] A named volume, created on first use
.image <name>-image.service [Image] Pre-pull an image before dependents start
.kube <name>.service [Kube] Run a Kubernetes YAML via kube play

For a whole pod, write a .pod file and point each .container at it with Pod=web.pod:

# ~/.config/containers/systemd/web.pod
[Pod]
PodName=web
PublishPort=8080:80

[Install]
WantedBy=default.target
# ~/.config/containers/systemd/web-app.container
[Container]
Image=ghcr.io/acme/web:latest
Pod=web.pod

Common [Container] keys and their podman run equivalents

The [Container] section maps almost one-to-one onto podman run flags. Knowing the mapping lets you convert any docker run/podman run line into a Quadlet file mechanically. A reference of the keys you will actually use:

Quadlet key podman run equivalent Notes
Image= (positional image) Registry ref or a .image unit
ContainerName= --name Defaults to systemd-<unit>
PublishPort= -p / --publish Repeatable; HOST:CONTAINER
Environment= -e / --env Repeatable; or EnvironmentFile=
Volume= -v Repeatable; supports :Z, :U, :ro
Network= --network Name a .network, or pasta/slirp4netns
Exec= (command after image) Override the entrypoint’s CMD
User= / Group= --user UID/GID inside the container
DropCapability= / AddCapability= --cap-drop / --cap-add Least-privilege the caps
NoNewPrivileges=true --security-opt no-new-privileges Blocks setuid escalation
ReadOnly=true --read-only Read-only rootfs
AutoUpdate=registry label io.containers.autoupdate=registry Enables podman auto-update
HealthCmd= --health-cmd Plus HealthInterval, HealthRetries, HealthStartPeriod
Secret= --secret Mounts a Podman secret
PodmanArgs= (raw passthrough) Escape hatch for flags with no dedicated key

The [Container] section covers most needs; PodmanArgs= is the escape hatch for anything Quadlet has no dedicated key for. Prefer dedicated keys where they exist — they are validated and readable; PodmanArgs= is a raw passthrough with no validation.

Rootless networking with pasta, slirp4netns, and port mapping

Rootless containers cannot create veth pairs against a host bridge — that needs CAP_NET_ADMIN on the host. So outbound traffic and published ports flow through a userspace network stack. There are two, and the default flipped in Podman 5:

# What is actually backing rootless networking?
podman info --format '{{.Host.NetworkBackend}}'          # netavark (the backend)
podman info --format '{{.Host.Slirp4netns.Executable}}'  # path if slirp is present
podman info --format '{{.Host.Pasta.Executable}}'        # path if pasta is present

# Run a container on the default rootless network (pasta on Podman 5)
podman run -d --name web -p 8080:80 docker.io/library/nginx:alpine

# Force a specific stack explicitly
podman run -d --network pasta        -p 8080:80 docker.io/library/nginx:alpine
podman run -d --network slirp4netns  -p 8080:80 docker.io/library/nginx:alpine

# Publish on a specific host interface only (loopback = local-only)
podman run -d -p 127.0.0.1:8080:80 docker.io/library/nginx:alpine
Attribute slirp4netns pasta
Podman default ≤ 4.x 5.x (rootless default net)
Throughput Lower Higher
Latency Higher Lower
Preserves client source IP No (by default) Yes
IPv6 Supported Better/native handling
Maturity Very mature Newer, now default
Pick it when Max compatibility / old Podman Source-IP fidelity or throughput matters

Two production gotchas dominate:

1. Ports below 1024. A rootless user cannot bind privileged host ports by default. Three honest options, in order of preference: publish a high port and front it with a reverse proxy or a firewall DNAT; lower the unprivileged-port threshold deliberately; or (last resort) run that one workload rootful. Lowering the threshold once:

# Allow rootless processes to bind from port 80 upward
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
# Persist it
echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-rootless-ports.conf
Approach Privilege cost Effort Best for
High port + reverse proxy / DNAT None (app stays unprivileged) Medium The clean default; production
Lower ip_unprivileged_port_start Host-wide sysctl change Low You control the host and accept the trade
AmbientCapabilities=CAP_NET_BIND_SERVICE on the user unit Grants one capability Low A single service that must bind low
Run the workload rootful Full root for that container Low Truly requires host-level networking

2. Source IP for logging/allowlists. With pasta you can keep the real client IP; with default slirp4netns you cannot, and every request appears to come from the internal gateway. If an app’s access logs or IP allowlists matter (fraud models, geo rules, rate limits keyed on IP), prefer pasta and verify the observed source address rather than assuming it works.

For container-to-container DNS, create a user-defined network — members resolve each other by container name through aardvark-dns, which ships with netavark:

podman network create appnet
podman run -d --network appnet --name api ghcr.io/acme/api:latest
podman run -d --network appnet --name web ghcr.io/acme/web:latest
# 'web' can now reach the API at http://api:PORT

The Quadlet-native way is a .network file that each .container references with Network=appnet.network, so the network is created and torn down as part of the unit lifecycle.

Automatic image updates with podman auto-update and rollback

Any container labelled io.containers.autoupdate=registry (the AutoUpdate=registry line in the Quadlet file sets exactly this label) participates in podman auto-update. It checks the registry for a newer digest on the same tag, pulls it, and restarts the systemd unit — only for containers managed by systemd, which Quadlet ones are by definition.

# Dry run: show what would update, change nothing
podman auto-update --dry-run

# Apply: pull newer digests and restart affected units
podman auto-update

The feature that makes this safe in production is automatic rollback. After restarting an updated unit, Podman waits for the container to become healthy. If the unit fails to come up, Podman rolls the image reference back to the previous digest and restarts again. Rollback is keyed on the unit reaching a running/healthy state — which is exactly why a real HealthCmd matters. Without one, “healthy” means merely “the process did not immediately exit,” and a subtly broken image (starts, serves errors) sails through and rollback never fires.

Wire it to a timer so it runs unattended. Podman ships podman-auto-update.service and podman-auto-update.timer; enable them per user:

systemctl --user enable --now podman-auto-update.timer
systemctl --user list-timers podman-auto-update.timer
AutoUpdate policy Behaviour Rollback Use when
registry Poll registry for a newer digest on the same tag; pull + restart Yes (on failed/unhealthy start) A tag you control (:prod, :stable)
local Restart if a locally-pulled image changed (e.g. you podman pulled it) Yes Air-gapped / pipeline pushes the image locally
(unset) Never auto-updated n/a Pinned digests; anything where surprise is unacceptable

Pin to immutable digests for anything where surprise is unacceptable, and reserve AutoUpdate=registry for tags you genuinely want to track — a stable channel, an internal :prod tag your pipeline re-points only after a canary clears. Auto-update on a :latest you do not control is how you get paged at 3 a.m. The rollback safety net is real, but it only fires if your healthcheck is honest.

Logging, healthchecks, and resource limits via cgroups v2

Because the container is a systemd unit, its stdout/stderr go to the journal automatically — no log driver to configure, no rotation to wire up (journald’s retention applies).

# Follow a Quadlet unit's logs
journalctl --user -u myapp.service -f

# Or Podman's own view, which also reads the journal
podman logs -f myapp

Healthchecks declared in the Quadlet file (HealthCmd, HealthInterval, HealthRetries, HealthStartPeriod) are enforced by Podman via a transient systemd timer per container. Inspect and act:

podman healthcheck run myapp                                   # run it once, now
podman inspect myapp --format '{{.State.Health.Status}}'       # healthy | unhealthy | starting

HealthStartPeriod gives a slow-booting app grace before failures count — the same lesson as any orchestrator: separate “starting” from “unhealthy,” or you evict a container that just needed 15 seconds to warm. You can have Podman act on failure with HealthOnFailure=kill (or restart / stop), turning a failed probe into a restart that systemd then re-supervises.

Healthcheck key Meaning Sensible default Failure mode if wrong
HealthCmd Command that returns 0 when healthy A cheap, honest self-check Deep dependency check → flaps on downstream blips
HealthInterval Time between checks 30s Too tight → CPU noise; too loose → slow detection
HealthRetries Consecutive failures before “unhealthy” 3 1 → flaps on a transient; 10 → slow to react
HealthStartPeriod Grace window at start 10–30s Too short → kills a slow-booting app repeatedly
HealthOnFailure Action on unhealthy (none/kill/restart/stop) none (let systemd decide) restart without a start period → restart loop

Resource limits ride on cgroup v2. Rootless CPU/memory/pids limits require the controllers be delegated to your user slice — modern systemd does this for cpu, memory, pids, and io out of the box via user@.service. Set caps declaratively. The idiomatic way is [Service] directives, because they show up in systemctl status and integrate with systemd accounting:

[Service]
MemoryMax=512M
CPUQuota=150%
TasksMax=256
Restart=on-failure
RestartSec=5

Or, if you prefer Podman’s own accounting, PodmanArgs= on the [Container]:

[Container]
Image=ghcr.io/acme/myapp:1.4.2
PodmanArgs=--memory=512m --cpus=1.5 --pids-limit=256

Verify delegation and live accounting — the check that saves you an hour when a limit “does nothing”:

# Which controllers are delegated to your user slice?
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers
# Expect: cpu memory pids io   (delegation present)

# Live resource usage for running containers
podman stats --no-stream

If memory limits silently do nothing, the memory controller is not delegated — check that line before blaming Podman.

Limit [Service] directive PodmanArgs flag Notes
Memory hard cap MemoryMax=512M --memory=512m OOM-kills the container at the ceiling
Memory high (soft) MemoryHigh=384M (none direct) Throttles reclaim before the hard cap
CPU quota CPUQuota=150% --cpus=1.5 150% = 1.5 cores worth of time
CPU weight (share) CPUWeight=200 --cpu-shares Relative weighting under contention
Max processes/threads TasksMax=256 --pids-limit=256 Fork-bomb protection
Block IO weight IOWeight=100 --blkio-weight Needs io delegated

Architecture at a glance

Picture the request and the process tree side by side, because the whole model is that the two are the same. Left edge: an inbound TCP connection arrives at a published host port — say :8443. Because this is rootless, there is no host bridge; the connection is handled by the pasta userspace network process running in the container’s network namespace, which forwards it inward while preserving the client’s real source IP. Center: the packet reaches your application process, which is running as PID 1 inside a user namespace where it believes it is UID 0 — but on the host it is UID 100000, an unprivileged account. That application process was not started by any daemon. Trace its parentage upward and you find conmon (the per-container monitor holding its stdio and ready to reap it), and above conmon a systemd --user service unit named myapp.service, and above that the user’s systemd --user manager kept alive by linger, and at the very top the host’s PID 1. There is no dockerd anywhere in that chain.

Right side and underneath: three cross-cutting systems attach to that process tree. Storage flows down through the overlay graph driver rooted at $HOME/.local/share/containers/storage, with any bind mounts relabelled for SELinux (:Z) and chowned into the namespace (:U). Resource control flows up into the kernel’s cgroup v2 tree at user.slice/user-1000.slice/..., where the cpu, memory, and pids controllers — delegated to this user slice by systemd — enforce CPUQuota=150%, MemoryMax=512M, and TasksMax=256. And the whole thing is generated, not hand-built: a Quadlet .container file in ~/.config/containers/systemd/ is read by systemd’s generator on every daemon-reload, which synthesises the read-only .service unit under /run/user/1000/systemd/generated/ that actually carries the podman run command.

The flow the reader should be able to narrate: a connection enters via pasta → hits the app running as an unprivileged mapped UID → whose supervisor is conmon → whose supervisor is a systemd user unit → generated by Quadlet from a Git-tracked .container file → resource-capped by delegated cgroup v2 controllers → logging to journald → and, on a timer, checked against the registry by podman auto-update, which will roll the image back if the healthcheck fails. Every arrow in that chain is a standard Linux mechanism (fork-exec, namespaces, cgroups, systemd, journald) rather than a proprietary daemon protocol — which is precisely why the failure modes are debuggable with tools you already own: systemctl, journalctl, ps, cat /sys/fs/cgroup/..., and podman inspect.

Real-world scenario

Meridian Payments ran ~40 stateless API containers per edge node across a fleet of bare-metal RHEL 9 boxes, originally on a root Docker daemon. Their security organisation issued a hard control coming out of a PCI assessment that flagged the Docker socket as a single point of total host compromise: no root-owned container daemon, and no privileged ports bound by application processes. Two facts made this awkward. The app was hard-coded to bind port 443, and its access logs fed an IP-based fraud model — so they could neither run the process as root to bind 443 nor lose the real client source address. The platform team was five engineers; the fleet was 60 nodes; downtime during the migration had to be near-zero.

They moved each service to rootless Podman under a dedicated svc-api user with linger enabled, and defined every workload as a Quadlet .container file checked into the same Git repo as their Ansible. The generator produced the systemd units on each host at daemon-reload, driven by an Ansible handler. Two specifics made it stick. First, instead of granting the service the privilege to bind 443, they kept the container on a high port (8443) and lowered the unprivileged-port threshold only as far as needed, fronting it with the host firewall’s DNAT so nothing application-owned ran as root:

# /etc/containers/systemd/payments-api.container  (system-managed, runs as svc-api)
[Container]
Image=registry.internal/payments/api:prod
User=svc-api
PublishPort=8443:8443
AutoUpdate=registry
HealthCmd=curl -fsS https://localhost:8443/healthz -k || exit 1
HealthInterval=15s
HealthStartPeriod=20s
DropCapability=ALL
NoNewPrivileges=true
# pasta preserves the real client source IP for the fraud model
Network=pasta

[Service]
Restart=on-failure
RestartSec=5
MemoryMax=1G
CPUQuota=200%

[Install]
WantedBy=default.target

Second, they switched the rootless network backend to pasta, which preserved the genuine client source IP end to end — slirp4netns had been rewriting every request to its internal gateway address, which would have blinded the fraud model. Auto-update was pointed at the internal :prod tag, which their pipeline re-pinned to an immutable digest only after a canary node cleared, and the built-in rollback (gated on the 15-second healthcheck) caught two bad pushes in the first quarter without a human touching a host — each time, the unit failed its healthcheck, Podman reverted to the prior digest, and the only evidence was a journal line and a Slack alert on the rollback.

The numbers: the migration ran node-by-node behind a load balancer with zero customer-visible downtime over three weeks. Per-node memory overhead dropped — removing the always-on dockerd freed roughly 150–250 MB per node that had been the daemon’s resident footprint. The Docker socket finding was closed outright (the socket no longer existed). And “restart policy” became a one-line Restart=on-failure that every other systemd service on the box already used — the platform team deleted a home-grown watchdog script that had existed only to restart crashed Docker containers. The line the team put on the wall: “We didn’t add a container platform. We deleted one — and let systemd do the job it was already doing for everything else.”

The migration as a phased plan, because the order is the lesson:

Phase Action Guardrail Rollback if it fails
0 Inventory every docker run/compose line and its flags podman ps/scripts audit (nothing changed yet)
1 Translate each to a .container file in Git Peer review the Quadlet Delete the file; keep Docker
2 Stand up on one canary node, drain it first Healthcheck must pass Re-add node to LB on Docker
3 Roll node-by-node behind the LB Two instances of each service fleet-wide Halt the roll; nodes stay mixed
4 Enable auto-update on :prod with rollback Canary node updates first Rollback is automatic
5 Decommission dockerd on migrated nodes Confirm no socket consumers left Reinstall docker (kept in repo)

Advantages and disadvantages

The daemonless, rootless, systemd-native model both removes whole classes of problem and introduces a few new edges. Weigh it honestly:

Advantages Disadvantages
No root daemon → RCE in a container lands as an unprivileged UID, not root Rootless networking is a userspace stack (pasta/slirp) with a throughput/feature ceiling vs a kernel bridge
Containers are ordinary systemd units — Restart=, After=, journalctl, boot ordering all just work You must learn systemd unit + Quadlet syntax; docker compose doesn’t map 1:1
One supervisor (systemd), not two (systemd + dockerd) — less RAM, one lifecycle Some images assume they can bind :80 or write root-owned paths; they need adjustment
.container files are declarative, reviewable, and GitOps-able The generated-unit indirection (edit file → daemon-reload) trips people expecting to edit the .service
Auto-update with healthcheck-gated rollback is built in, no extra tooling Rollback is only as good as your HealthCmd; a silent-error image can still slip through
Removing the Docker socket closes the classic total-host-compromise finding subuid/subgid, linger, and cgroup delegation are new operational concepts to get right
journald logging is automatic; retention is your existing journald config High-throughput or exotic-networking workloads may still want rootful or a kernel-bridge setup

The model is right for standard stateless services, edge and bare-metal fleets, and anyone under a mandate against a root container daemon — which is a large and growing set, since it is the default on all of RHEL/Fedora/CentOS. It bites hardest on workloads that genuinely need kernel-level networking performance, images that were built assuming root, and teams that never learn the three foundational concepts (subuid, linger, cgroup delegation) and then conclude “rootless is flaky” when it is actually mis-set-up. Every disadvantage above is manageable — but only if you know it exists, which is the point of this article.

Hands-on lab

Stand up a rootless container as a Quadlet-managed systemd service, confirm it runs unprivileged, cap its resources, and (optionally) prove it survives a reboot. Free-tier-friendly — it needs only a Linux VM with Podman 4.4+ (any recent Fedora/RHEL/Debian/Ubuntu). Run as a non-root user (create a dedicated one for realism).

Step 1 — Confirm the platform.

podman version | head -3                    # want 4.4+ (5.x ideal)
stat -fc %T /sys/fs/cgroup/                  # want: cgroup2fs
grep "^$USER:" /etc/subuid /etc/subgid       # want a range, e.g. USER:100000:65536

Expected: a Podman version, cgroup2fs, and one subuid/subgid line each. If the subuid/subgid lines are missing, add a range with sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER and run podman system migrate.

Step 2 — Enable lingering so the user manager runs at boot.

sudo loginctl enable-linger "$USER"
loginctl show-user "$USER" --property=Linger     # Linger=yes

Step 3 — Prove the user-namespace mapping (the security model).

podman run --rm docker.io/library/alpine id            # uid=0(root) INSIDE
podman unshare cat /proc/self/uid_map                  # 0  100000  65536  -> host UID

Expected: root inside the container, mapped to host UID 100000 outside. That is the whole isolation guarantee.

Step 4 — Write a Quadlet .container unit.

mkdir -p ~/.config/containers/systemd
cat > ~/.config/containers/systemd/weblab.container <<'EOF'
[Unit]
Description=Rootless nginx lab
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/library/nginx:alpine
ContainerName=weblab
PublishPort=8080:80
DropCapability=ALL
AddCapability=NET_BIND_SERVICE
HealthCmd=curl -fsS http://localhost:80/ || exit 1
HealthInterval=15s
HealthRetries=3
HealthStartPeriod=10s

[Service]
Restart=on-failure
RestartSec=5
MemoryMax=256M
CPUQuota=100%
TasksMax=128

[Install]
WantedBy=default.target
EOF

Step 5 — Generate the unit and start it.

systemctl --user daemon-reload         # Quadlet runs here, synthesising weblab.service
systemctl --user start weblab.service
systemctl --user status weblab.service --no-pager

Expected: Active: active (running). Note there is a weblab.service even though you never wrote a [Service] ExecStart — Quadlet generated it. You can see the generated unit (read-only): cat /run/user/$(id -u)/systemd/generated/weblab.service.

Step 6 — Verify it works, is healthy, and runs unprivileged.

curl -fsS http://localhost:8080/ | head -1        # nginx welcome HTML
podman inspect weblab --format '{{.State.Health.Status}}'   # healthy (after ~15s)
podman top weblab huser hpid user                 # huser = your mapped UID, NOT 0

Expected: HTML returned, healthy, and the host user (huser) is a high mapped UID like 100000 — not root.

Step 7 — Confirm resource limits landed (cgroup v2 delegation).

cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers   # cpu memory pids io
podman stats --no-stream weblab                    # MEM LIMIT ~256MiB, PIDS bounded

Expected: the delegated controllers include memory and cpu, and podman stats shows the 256 MiB memory ceiling.

Step 8 (optional) — The real test: reboot survival.

sudo systemctl reboot
# ... wait for the box, log back in ...
systemctl --user is-active weblab.service          # active  (thanks to linger)

If this fails but everything else passed, you skipped loginctl enable-linger in Step 2.

Validation checklist. You ran a container with no daemon, as an unprivileged UID, supervised by systemd, generated declaratively from a Git-able .container file, capped by delegated cgroup v2 controllers, with a healthcheck — and it came back after a reboot. That is the entire production pattern in miniature.

Step What you did What it proves
3 podman unshare UID map Container root == unprivileged host UID
5 daemon-reload generates weblab.service Quadlet synthesises the unit; no hand-written ExecStart
6 podman top ... huser The process runs unprivileged on the host
7 cgroup.controllers + podman stats Limits are enforced via delegated cgroup v2
8 Reboot → still active Linger makes it boot-persistent

Teardown.

systemctl --user stop weblab.service
rm ~/.config/containers/systemd/weblab.container
systemctl --user daemon-reload
podman rm -f weblab 2>/dev/null; podman image rm docker.io/library/nginx:alpine 2>/dev/null
# Optional: sudo loginctl disable-linger "$USER"

Common mistakes & troubleshooting

The failure modes that actually bite, as a symptom → root cause → confirm → fix playbook. Read it once, keep it open when a rootless service misbehaves.

# Symptom Root cause Confirm (exact command) Fix
1 Container works in SSH session, vanishes after logout Linger not enabled; user manager torn down at session end loginctl show-user $USER --property=Linger = no sudo loginctl enable-linger $USER
2 lchown <uid>: invalid argument on pull/run subuid/subgid range too narrow (< image’s max UID) grep "^$USER:" /etc/subuid /etc/subgid (count < 65536) Widen to ≥ 65536; podman system migrate
3 Permission denied reading a bind mount that is world-readable SELinux label wrong on the host path ls -Z /srv/appdata; ausearch -m avc -ts recent Mount with :Z (or :z if shared)
4 Bind mount appears empty/unowned inside the container Host files owned by real UID, not mapped UID podman run --rm -v /path:/m alpine ls -ln /m Add :U to chown into the namespace, or use a named volume
5 MemoryMax= / --memory has no effect memory controller not delegated to the user slice cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers Ensure cgroup v2 + systemd delegation; reboot after enabling
6 bind: permission denied on port 80/443 Rootless can’t bind privileged ports podman run -p 80:80 ... fails High port + proxy/DNAT, or lower ip_unprivileged_port_start, or AmbientCapabilities
7 Every request logs the same internal gateway IP slirp4netns hiding the source IP podman info --format '{{.Host.NetworkBackend}}'; check container access log Switch to pasta (Network=pasta)
8 .container edits ignored; unit unchanged Forgot daemon-reload, or file in the wrong dir, or a typo’d key systemctl --user cat weblab.service; check ~/.config/containers/systemd/ systemctl --user daemon-reload; fix path/key; check journalctl for generator warnings
9 podman auto-update never updates a live image Missing AutoUpdate=registry, or the container isn’t systemd-managed podman inspect --format '{{index .Config.Labels "io.containers.autoupdate"}}' Add AutoUpdate=registry; ensure it’s a Quadlet/systemd unit
10 A bad image update crash-loops instead of rolling back Healthcheck missing or too shallow, so “healthy” == “process didn’t exit” podman inspect --format '{{.State.Health.Status}}' = (none) Add an honest HealthCmd + HealthStartPeriod
11 Rootless storage using vfs, disk ballooning Kernel/overlay unavailable → fell back to vfs podman info --format '{{.Store.GraphDriverName}}' = vfs Install fuse-overlayfs; on 5.11+ enable native overlay; reset storage
12 Container-to-container name resolution fails Containers on the default net, not a shared user-defined network podman network ls; podman inspect --format '{{.NetworkSettings.Networks}}' Put both on a .network/podman network create; aardvark-dns resolves names
13 newuidmap: write to uid_map failed uidmap/shadow-utils missing or helpers not setuid ls -l $(command -v newuidmap) (want setuid root) Install uidmap/shadow-utils; restore setuid bit
14 Boot start racy: app up before network/mount ready Missing ordering in the .container [Unit] systemctl --user list-dependencies weblab.service Add After=/Requires= (e.g. network-online.target, a mount unit)
15 podman auto-update.timer present but nothing happens Timer not enabled for this user systemctl --user is-enabled podman-auto-update.timer systemctl --user enable --now podman-auto-update.timer

The expanded reasoning for the three that consume the most hours:

Linger (row 1). This is the number-one first-encounter mistake. The user’s systemd --user manager exists only while they have a login session unless linger is set; log out and the manager — and every --user unit under it, including your containers — is stopped. Everything works perfectly in your SSH session, so it looks fine, and then the box reboots (or you just log out) and the service is gone. loginctl enable-linger <user> is mandatory for any boot-persistent rootless workload, full stop.

subuid width (row 2). Rootless mapping needs enough subordinate IDs to cover whatever the image chowns to. A range of, say, 1000 IDs breaks the moment an image sets a file to UID 1001. The fix is always a ≥ 65536-wide range and a podman system migrate so Podman rebuilds the namespace with the new map. When in doubt, 100000:65536 is the safe default.

cgroup delegation (row 5). On cgroup v2, unprivileged users can only set limits for controllers that systemd has delegated to their slice. If MemoryMax= seems ignored, do not debug Podman — cat the cgroup.controllers of your user slice. If memory is absent, the controller isn’t delegated (a mis-configured host, cgroup v1, or a systemd too old), and no Podman flag will make the limit stick until you fix delegation at the host level.

Best practices

Security notes

Control Quadlet / mechanism Secures against
Rootless user namespace (default rootless) Container breakout → host root
Drop capabilities DropCapability=ALL + AddCapability= Ambient kernel privileges
No new privileges NoNewPrivileges=true setuid/setgid escalation
Read-only rootfs ReadOnly=true (+ tmpfs for scratch) Persistence / tampering in the container
SELinux relabel :Z / :z on mounts Cross-container / host file access
Secrets as files Secret= Credential leak via env/inspect
Digest pinning Image=...@sha256:... Supply-chain tag-move surprises
Rootless-only socket podman system service (user) Re-creating the root-socket compromise path

Cost & sizing

Podman itself is free and open-source, and rootless Podman has negative incremental cost versus a Docker daemon — you remove the always-on dockerd (and its containerd), reclaiming roughly 150–250 MB of resident memory per host that the daemon and its shims held whether or not containers were running. On a fleet of 60 edge nodes that is measurable RAM back, and one fewer service to patch, monitor, and reason about. There is no per-node licence, no control-plane to run (unlike Kubernetes), and no managed-service bill — the entire model is “systemd you already run + a container binary.”

The real sizing questions are host-level, not licence-level. Memory: size each host for the sum of your containers’ MemoryMax plus headroom; because there is no daemon reserving memory, your budget is almost entirely the workloads. CPU: CPUQuota in aggregate should leave slack for pasta’s userspace networking, which costs some CPU that a kernel bridge would not — budget a few percent per busy container for the network stack. Disk: overlay storage under $HOME/.local/share/containers grows with images and container layers; a podman system prune on a timer keeps it bounded, and staying off vfs (which deep-copies layers) is the single biggest disk saver. Networking throughput: if a workload needs line-rate networking, pasta’s userspace ceiling may push you to rootful Podman with a kernel bridge or to a different host design — that is an architecture cost, not a dollar cost.

Cost / sizing driver What drives it How to right-size Rough figure
Per-host memory Sum of container MemoryMax + headroom Cap each container; no daemon to budget ~150–250 MB reclaimed vs Docker daemon
Network CPU overhead pasta/slirp userspace stack Leave a few % CPU slack per busy container Small; workload-dependent
Overlay disk Images + layers under $HOME podman system prune on a timer; avoid vfs Bounded by prune cadence
Operational headcount Fleet size / node count Quadlet-in-Git + Ansible reload handler Fewer moving parts than a K8s cluster
Licence (none) ₹0 / $0 — OSS

For most single-host and edge fleets, the “sizing” exercise is simply: pick MemoryMax/CPUQuota/TasksMax per container from observed usage (via podman stats), sum them for the node, add headroom, and enable a prune timer. Compared with running a Kubernetes cluster for the same workloads, the cost delta is enormous — no etcd, no control plane, no worker-node overhead — which is exactly why Podman + Quadlet is the right tool when you have a handful of nodes rather than a data centre.

Interview & exam questions

1. What does “daemonless” actually mean for Podman, and what’s the process tree of a running container? There is no long-lived privileged service; podman run forks conmon, which execs an OCI runtime (crun/runc) that becomes the container’s first process. Under systemd, conmon is a child of the .service unit, so systemd and the kernel — not a daemon — supervise the container. Kill the Podman binary and running containers keep running.

2. How does rootless Podman let an unprivileged user run a container “as root”? Via a user namespace: the user’s login UID becomes UID 0 inside the namespace, while the container’s other UIDs map to a block of host UIDs reserved in /etc/subuid//etc/subgid (e.g. 100000–165535). Container root is therefore an unprivileged host UID that owns nothing — confirmable with podman unshare cat /proc/self/uid_map.

3. Why is loginctl enable-linger required, and what breaks without it? A user’s systemd --user manager normally exists only during an active login session; without linger it’s torn down at logout, stopping every --user unit including your containers. Linger keeps the manager (and its services) running with no session, which is what makes rootless containers start at boot and survive logout.

4. slirp4netns vs pasta — what’s the difference and when do you pick each? Both are userspace rootless network stacks (rootless can’t create a host bridge without CAP_NET_ADMIN). slirp4netns is the original — reliable but slower and it hides the client source IP by default. pasta is the Podman 5 default — faster, lower latency, and it preserves the real source IP. Pick pasta when source-IP fidelity (logs, allowlists, fraud models) or throughput matters.

5. What is Quadlet, and why not hand-write a [Service] ExecStart=podman run unit? Quadlet is a systemd generator that reads declarative .container/.pod/.network/.volume files (in ~/.config/containers/systemd/) and synthesises real .service units on daemon-reload. It replaces the removed podman generate systemd. You get a declarative, Git-able source of truth and never hand-maintain the podman run line; the generated unit is read-only under /run/.../generated/.

6. Why must cgroup v2 controllers be “delegated” for rootless resource limits, and how do you confirm it? On cgroup v2, an unprivileged user can only set limits for controllers systemd has delegated to their slice (modern systemd delegates cpu, memory, pids, io via user@.service). If MemoryMax= does nothing, cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers — if memory is absent, delegation is the problem, not Podman.

7. What does AutoUpdate=registry do, and why does a real HealthCmd matter for it? It labels the container io.containers.autoupdate=registry so podman auto-update polls the registry for a newer digest on the same tag, pulls it, and restarts the (systemd-managed) unit — with automatic rollback to the previous digest if the container doesn’t come up healthy. Rollback is keyed on the unit reaching a healthy state, so without an honest HealthCmd, “healthy” just means “the process didn’t exit” and a broken image can slip through.

8. When mounting a host directory into a rootless container on RHEL, what do :U, :z, and :Z each do? :U recursively chowns the source to the container’s mapped UID/GID so it’s writable; :z applies a shared SELinux label (several containers may use the path); :Z applies a private label (one container only). Wrong or missing labels are the classic “Permission denied on a world-readable file” on RHEL-family hosts.

9. How does a pod differ from independent containers, and where do you publish ports? A pod shares one network namespace (one IP, one localhost) via an infra container; members reach each other over 127.0.0.1, and you publish ports on the pod, not the members. Independent containers each have their own netns and resolve each other by name only if placed on a shared user-defined network (via aardvark-dns).

10. How do you migrate a Docker fleet to rootless Podman with minimal risk? Inventory every docker run/compose flag; translate each to a peer-reviewed .container file in Git; stand it up on a drained canary node and confirm the healthcheck; roll node-by-node behind the load balancer keeping ≥2 instances of each service; enable auto-update on a controlled tag with rollback (canary first); then decommission dockerd on migrated nodes. alias docker=podman and the Docker-compat socket ease the CLI/tooling transition.

11. Your container works over SSH but is gone after you log out and back in. What happened? Linger isn’t enabled, so the systemd --user manager (and its containers) was stopped at session end. Confirm with loginctl show-user $USER --property=Linger and fix with sudo loginctl enable-linger $USER.

12. A custom image fails to pull rootless with lchown ... invalid argument. Why, and how do you fix it? The user’s subuid/subgid range is narrower than a UID the image chowns to, so the mapping can’t satisfy it. Widen the range to ≥ 65536 (usermod --add-subuids/--add-subgids, e.g. 100000-165535) and run podman system migrate to rebuild the namespace.

These map to Linux/DevOps container competencies rather than a single vendor exam, but they overlap the Red Hat RHCSA/RHCE container objectives (running containers as systemd services, rootless containers, podman/skopeo/buildah), the LFCS/LFCE container units, and any container-security or CIS-benchmark work that scrutinises the Docker socket.

Question theme Where it maps
Rootless user namespaces, subuid/subgid RHCSA/RHCE containers; container security
Containers as systemd services (Quadlet) RHCSA/RHCE; LFCS
cgroup v2 delegation & limits LFCE; performance/isolation
pasta vs slirp4netns networking Container networking; RHCE
SELinux :Z on volumes RHCSA/RHCE security; SELinux
Docker socket risk / migration CIS benchmarks; container security review

Quick check

  1. In the process tree of a rootless container run under systemd, what sits directly above the container’s first process, and what sits above that?
  2. Your rootless service runs fine in your SSH session but is gone after the host reboots. What one command fixes it, and what does it do?
  3. MemoryMax=512M in your .container file appears to be ignored. What’s the single most likely cause, and how do you confirm it?
  4. You mount /srv/appdata into a rootless container on RHEL and get “Permission denied” even though the files are world-readable. What mount option is missing, and what’s the difference between :z and :Z?
  5. Why does podman auto-update’s rollback need a real HealthCmd, and what happens without one?

Answers

  1. Directly above the container’s first process is conmon (the per-container monitor holding its stdio and reaping it); above conmon is the systemd --user service unit that Quadlet generated (kept alive by the user’s systemd --user manager under linger). There is no dockerd anywhere in the chain.
  2. sudo loginctl enable-linger <user>. Without linger, the user’s systemd --user manager (and every --user unit, including your containers) is torn down when the last login session ends; linger keeps it running with no active session, so the service starts at boot and survives logout.
  3. The memory cgroup v2 controller isn’t delegated to your user slice. Confirm with cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.controllers; if memory isn’t listed, delegation (host/systemd config) is the problem, not Podman.
  4. The SELinux relabel option (:Z or :z) is missing. :z applies a shared label so multiple containers can use the path; :Z applies a private label for exactly one container. On RHEL-family hosts an unlabelled bind mount is denied regardless of Unix permissions.
  5. Rollback triggers when an updated unit fails to reach a healthy state. Without an honest HealthCmd, “healthy” degrades to “the process didn’t immediately exit,” so a subtly broken image (starts, serves errors) is considered fine and rollback never fires — you ship the bad image.

Glossary

Next steps

You can now run rootless containers as reboot-surviving, resource-capped, auto-updating systemd services defined entirely in Git. Build outward:

linuxpodmancontainerssystemdrootlessquadletcgroups-v2selinux
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments

Keep Reading