Mastering systemd: Units, Timers, Resource Control, and Service Hardening

Most engineers touch systemd through three commands — systemctl start, stop, and status — and never open the unit file underneath. That is the most expensive habit in Linux operations, because a unit file is not boilerplate: it is a declarative contract that hands you a process supervisor, a dependency resolver, a cron replacement, a cgroup-based resource controller, and a syscall-filtering sandbox, all from one INI file with no wrapper scripts and no external daemon. The difference between a service that crash-loops silently at 3am and one that trips a watchdog, sheds memory under pressure, and pages you with a clean journal is usually fifteen lines of [Service] config that the person who wrote it never learned existed.

This is the deep dive. We walk every unit type (service, socket, timer, target, mount, path, slice, scope), the full anatomy of the [Unit]/[Service]/[Install] sections, all five service Type= values and exactly when each is correct, the orthogonal-and-constantly-confused pair of dependency (Wants=/Requires=) versus ordering (After=/Before=), timers as a strictly better cron, cgroup v2 resource control (CPU/Memory/IO/Tasks accounting and hard/soft limits, organised into slices), the sandboxing directives that replace a pile of AppArmor/SELinux work for common cases, drop-in overrides that survive package upgrades, and reading journald like a database instead of grepping flat files. Everything assumes a modern distribution — systemd v245 or newer with the unified cgroup v2 hierarchy as default. Confirm yours with systemctl --version and stat -fc %T /sys/fs/cgroup/ (you want the answer cgroup2fs).

By the end you will write a production service unit from a blank file — one with a real readiness gate, a crash-loop circuit breaker, a memory ceiling that reclaims before it kills, a dropped-capability set, and a seccomp allowlist — validate it before it ever runs, migrate a cron job to a timer that catches up on missed runs, and diagnose a slow boot or an ordering cycle from the journal. The tables are the reference you keep open; the prose explains the why.

What problem this solves

Before systemd, keeping a daemon alive in production meant a supervisor (runit, supervisord, monit), a PID file the daemon might or might not write correctly, an init script full of start-stop-daemon incantations, ulimit lines in a wrapper, nice/ionice prefixes, a cron entry for the periodic job with its own >> logfile 2>&1 redirection, logrotate config for that logfile, and — if you cared about security — an AppArmor or SELinux profile authored and debugged separately. Six tools, six config formats, six ways to get it subtly wrong. The classic failures were legion: a daemon that “started” (forked) but wasn’t ready when the next service tried to connect; a cron job that silently didn’t run because the machine was asleep at 2am; a memory leak that took down the whole box because nothing capped the process; a log that filled /var because nobody rotated it.

systemd collapses all of that into one declarative unit file that the init system — which is PID 1, present on every boot, aware of every dependency — enforces directly. What breaks without this knowledge is subtle and expensive: engineers put a crash-loop limiter in [Service] where it is silently ignored, order a service on network.target when it needed network-online.target and it fails intermittently on reboot, use Requires= where Wants= belonged and cascade a non-critical dependency failure into a full outage, or edit a vendor-shipped unit in place so the next package upgrade quietly reverts their fix. Each of these is a one-line mistake that costs hours of incident time because the symptom is far from the cause.

Who hits this: everyone running Linux servers — which is to say, everyone running anything on AWS, GCP, Azure VMs, or on-prem. It bites hardest on teams migrating off SysV init scripts (the mental model doesn’t transfer), teams running memory-hungry JVM or Node services on shared boxes (no resource limits, noisy-neighbour outages), teams that “replaced cron with a systemd timer” but never set Persistent= (missed runs on machines that sleep), and anyone who thinks “the service started” means “the service is ready.” The fix is almost never a bigger box or another supervisor — it is understanding the contract the unit file already offers you for free.

Learning objectives

By the end of this article you can:

• Name every systemd unit type — service, socket, timer, target, mount, path, slice, scope, device, swap — and say what each one manages and when you author one directly.
• Read and write the [Unit], [Service], and [Install] sections from scratch, and explain what [Install] does and why a missing one means enable silently does nothing.
• Choose the correct service Type= (simple, exec, forking, oneshot, notify, dbus, idle) for any daemon, and explain why “started” is not “ready” under Type=simple.
• Distinguish dependency (Wants=/Requires=/BindsTo=) from ordering (After=/Before=), use both correctly, and know why network-online.target differs from network.target.
• Replace cron with timers — build the .service/.timer pair, write and validate OnCalendar= expressions, and use Persistent=, RandomizedDelaySec=, OnBootSec=, and OnUnitActiveSec=.
• Fence a service with cgroup v2 resource control — CPUQuota/CPUWeight, MemoryHigh/MemoryMax, IOWeight/bandwidth caps, TasksMax — and organise services into slices with tiered budgets.
• Harden a service with the sandboxing directives (NoNewPrivileges, ProtectSystem=strict, CapabilityBoundingSet, SystemCallFilter, PrivateTmp, DynamicUser) and drive the systemd-analyze security score down.
• Override vendor units safely with drop-ins, template one unit into many instances, and read journald with structured, indexed filters instead of grep.

Prerequisites & where this fits

You should be comfortable on a Linux command line as root or with sudo: editing files under /etc, reading process output from ps, and understanding that a “daemon” is a long-running background process. You should know what a TCP port and a Unix socket are, what an environment variable is, and roughly what “the kernel schedules CPU time” means. No prior systemd depth is assumed beyond having typed systemctl status once. Familiarity with INI-style config (sections in [brackets], Key=Value lines) helps because that is exactly the unit-file format.

This sits at the foundation of the Linux server operations track — almost everything else builds on it. Resource control here is the mechanism behind Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers, which layers system-wide tuning on top of the per-service cgroup limits you set here. The sandboxing section is the systemd-native complement to full mandatory-access-control in Authoring AppArmor Profiles: Confining Services on Ubuntu and Debian and Running SELinux in Enforcing Mode: Troubleshooting and Writing Custom Policy — systemd directives handle the common cases; MAC handles the rest. Timers and units are the substrate for Running Rootless Containers in Production with Podman and Quadlet, where Quadlet generates systemd units for containers, and journald is one input to Building a Linux Audit Trail with auditd and eBPF Runtime Visibility.

A quick map of who owns what during an incident, so you look in the right place first:

Core concepts

Five mental models make every later section obvious.

A unit is a typed, declarative object that PID 1 manages. systemd does not run shell scripts on your behalf and forget about them; it loads units — objects of a specific type identified by the filename suffix (.service, .timer, .socket, .target, .slice, .mount, .path, .scope, .device, .swap) — into an in-memory dependency graph, resolves that graph, and drives each unit through a state machine (inactive → activating → active → deactivating → failed). Because everything is a unit, the same verbs (start, stop, enable, status) and the same dependency mechanics apply uniformly whether the thing is a daemon, a mount point, a periodic timer, or an abstract grouping point.

“Started” is not “ready.” This is the single most consequential idea. With the default Type=simple, systemd considers your service started the instant it successfully forks the process — before the process has parsed its config, bound its socket, or connected to its database. Any service ordered After= yours may launch into a service that isn’t actually serving yet. Genuine readiness requires either Type=notify (the service explicitly signals READY=1 via sd_notify(3) when it can serve) or socket activation (systemd holds the socket, so the dependent can connect immediately and its first bytes queue until your service is up). Every “works on reboot, breaks under load” race traces back to conflating started with ready.

Dependency and ordering are orthogonal. Requires=B says “if I start, pull B in, and if B fails, fail me too” — it is about whether B runs. After=B says “do not start me until B has finished starting” — it is about when, and says nothing about pulling B in. You almost always need both, and each without the other is a bug: Requires= without After= starts them in parallel (a race); After= without Wants=/Requires= orders a dependency that might not even be present. Internalise the two-axis model and half of systemd’s surprises evaporate.

Every service lives in a cgroup, and cgroups are a tree. On cgroup v2 (unified hierarchy) systemd places each service in its own control group, nested under a slice. The default tree is -.slice (root) → system.slice (system daemons), user.slice (per-user sessions), and machine.slice (VMs/containers). Because a cgroup is the unit of accounting and control, you cap CPU, memory, and IO with [Service] directives that map directly onto kernel controllers — no ulimit guesswork, no external cgcreate. Slices let you set a shared budget across a group of services: put your batch jobs in system-batch.slice with a hard CPU and memory ceiling and they can never starve the foreground app.

Hardening is subtractive and additive at once. systemd’s sandbox works by removing capability from the service (drop Linux capabilities, filter syscalls, make the filesystem read-only, hide /home, mask kernel tunables) — every directive shrinks the blast radius of a compromise. The catch is that removing too much breaks a legitimate service on an unrelated syscall or path, so hardening is empirical: add a directive, restart the running service (not just check that it starts), watch for EPERM/SIGSYS, and repeat. The systemd-analyze security tool scores the result so you have a target to drive down.

The vocabulary in one table

Pin down every moving part before the deep sections. The glossary at the end repeats these for lookup; this table is the mental model side by side.

The unit types, end to end

The filename suffix declares the type, and each type manages a different kind of thing. You author .service, .timer, .socket, .slice, and .path units by hand; .mount, .device, .swap, and most .target units are usually generated or shipped, but you should recognise them. Here is the complete set:

Three of these deserve a mental model beyond the one-liner.

Targets are the closest systemd has to SysV runlevels, but richer: a target pulls in a set of units and provides a synchronisation point. multi-user.target is “multi-user text-mode system up” (the old runlevel 3); graphical.target adds the display manager (runlevel 5); network-online.target is “the network is actually configured and routable.” You rarely write a target, but you constantly reference them — WantedBy=multi-user.target in [Install] is how a service wires itself into boot, and After=network-online.target is how it waits for the network. The default target the system boots into is set by a symlink:

systemctl get-default            # e.g. multi-user.target or graphical.target
sudo systemctl set-default multi-user.target   # boot to text mode (no GUI)

Sockets enable socket activation: systemd creates and listens on the socket itself, then starts your service only when the first connection arrives (or eagerly, if you prefer), passing it the already-bound file descriptor. Two payoffs: on-demand start (the service consumes nothing until used) and zero-lost-connection restarts (systemd holds the socket across a service restart, so client connections queue instead of getting connection-refused). A .socket unit and its .service share a name stem; systemd links them automatically.

Slices are cgroup nodes. The name encodes the tree with - as the separator: system-batch.slice nests under system.slice, which nests under -.slice (root). You set resource limits on a slice and every service in it shares that budget — the cleanest way to guarantee background work can never starve foreground work.

Where unit files live, and who wins

Service units are searched in several directories in strict priority order — the first match wins, and a higher-priority file masks a lower one entirely (a zero-length or /dev/null-symlinked file disables the unit). Know these paths cold, because “I edited the unit and nothing changed” is almost always editing the wrong copy:

To customise a package-shipped unit, you never copy it into /etc — you add a drop-in (covered below), which merges on top and survives upgrades because the base file stays untouched. To see the effective, merged result of a unit plus all its drop-ins, run systemctl cat <unit>.

Anatomy of a unit file: the three sections

A .service unit has up to three sections. [Unit] and [Install] are generic (they appear in other unit types too); [Service] is service-specific. Here is what each section owns and the highest-value keys in each.

The [Unit] section — identity, dependency, ordering

[Unit] carries human metadata and the entire dependency/ordering graph. The keys you use constantly:

Two of these are landmines. StartLimitIntervalSec and StartLimitBurst live in [Unit], not [Service] — put them in [Service] and they are silently ignored, so your crash-loop breaker does nothing and a broken service spins forever. And Conditions differ from Asserts: a failed Condition* marks the unit as skipped (success, it just didn’t run), while a failed Assert* marks it failed. Use ConditionPathExists=/etc/widgetd/enabled to make a unit opt-in without erroring when the file is absent.

The [Service] section — how the process runs

[Service] is the largest section: process type, the commands, restart policy, timeouts, the watchdog, resource control, and the entire sandbox. The core (non-resource, non-sandbox) keys:

The [Install] section — enable-time wiring

[Install] is inert while the service runs; it only takes effect when you run systemctl enable. It describes how the unit should be wired into the boot graph. The keys:

The gotcha is stark: no [Install] section means systemctl enable has nothing to do, so the unit never starts at boot — even though systemctl start works fine right now. Engineers write a perfect unit, start it, confirm it runs, walk away, and discover after the next reboot that it never came up. The fix is one block:

[Install]
WantedBy=multi-user.target

enable reads WantedBy=multi-user.target and creates the symlink /etc/systemd/system/multi-user.target.wants/<unit> — that symlink is what makes the unit start at boot. Confirm with systemctl is-enabled <unit> (expect enabled).

Service Type=: when is it “started”?

The Type= field tells systemd how to decide the service has finished starting — which gates when After=-ordered units may launch, and whether ExecStartPost= runs against a ready service. Getting it wrong is the most common systemd bug in the field. All the values:

The hierarchy in one sentence: prefer notify when your framework supports it; use exec over bare simple otherwise; use forking only for legacy double-forking daemons (and always set PIDFile=); use oneshot for tasks that run and exit. The reason notify matters is concrete — a database ordered After= a Type=simple app races the app’s own socket bind, so on a busy boot the database connection is refused, the app crashes, and it “works after a restart” because the second time the race falls the other way. Switch the app to Type=notify, have it call sd_notify(0, "READY=1") only after its socket is bound and its pool is primed, and the race is gone.

Dependencies vs ordering: the two-axis model

This is the concept that generates the most confusion and the most subtle outages, so it earns its own section. There are two independent questions about any two units A and B:

1. Should starting A cause B to start too? — the dependency axis (Wants=, Requires=, Requisite=, BindsTo=).
2. If both are starting, which goes first? — the ordering axis (After=, Before=).

They do not imply each other. Requires=B with no After=B pulls B in but starts A and B in parallel — a race. After=B with no Wants=/Requires=B orders A after B if B happens to be present, but does nothing to ensure it is. You almost always want both together. The full matrix of dependency strengths:

Default to Wants= and reserve Requires=/BindsTo= for genuinely hard dependencies. The classic over-coupling bug: a web app declares Requires=redis.service because it uses Redis for caching; Redis has a blip and restarts; because of Requires=, systemd stops the web app too, turning a degraded-cache event (which the app could ride out) into a full outage. Wants=redis.service would have kept the app up. “Start my app after the database, and pull the database in if it isn’t running” is the canonical correct pair:

[Unit]
Description=Widget API daemon
Wants=postgresql.service
After=postgresql.service network-online.target

Note network-online.target, not network.target. This distinction bites constantly:

If your service needs a routable IP the moment it starts (it connects to a remote database or API at boot), you need both After=network-online.target and Wants=network-online.target — and the relevant wait-online service (systemd-networkd-wait-online or NetworkManager-wait-online) must be enabled, or the target completes instantly and gates nothing. Ordering on plain network.target is the number-one “fails intermittently on reboot, fine on restart” cause.

Writing a production service unit

Enough theory — here is a complete, opinionated unit for a long-running daemon, with every deliberate choice explained. Save it as /etc/systemd/system/widgetd.service:

[Unit]
Description=Widget API daemon
Documentation=https://internal.example.com/runbooks/widgetd
Wants=network-online.target
After=network-online.target postgresql.service
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=notify
ExecStart=/usr/local/bin/widgetd --config /etc/widgetd/config.toml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
TimeoutStartSec=30
TimeoutStopSec=20
WatchdogSec=30

# Run as a transient, system-managed identity with private state dirs
DynamicUser=yes
StateDirectory=widgetd
RuntimeDirectory=widgetd

[Install]
WantedBy=multi-user.target

The reasoning behind each non-obvious choice:

• Type=notify + WatchdogSec=30 gives you a real readiness gate and a real liveness gate. The daemon calls sd_notify(0, "READY=1") when it can serve; ordered units then wait correctly. It must also ping sd_notify(0, "WATCHDOG=1") at least every 15 seconds (half WatchdogSec); miss it and systemd treats the service as hung and restarts it — catching deadlocks a “process still alive” check never would.
• Restart=on-failure restarts on non-zero exit, killing signals, timeouts, and watchdog failures — but not on a clean exit 0. Use Restart=always only if a clean exit should also trigger a restart, and Restart=on-watchdog if only watchdog timeouts should.
• StartLimitIntervalSec=60 + StartLimitBurst=5 is the crash-loop circuit breaker: more than 5 restarts in 60 seconds and systemd stops trying and marks the unit failed, instead of hammering a downstream forever. These live in [Unit].
• DynamicUser=yes is the single highest-leverage hardening directive: systemd allocates a transient UID/GID for the service’s lifetime, gives it a private /tmp, and locks down filesystem access by default. With StateDirectory=widgetd, systemd creates and chowns /var/lib/widgetd; RuntimeDirectory=widgetd gives /run/widgetd, cleaned on stop.

The values in this template you will tune per service:

Load and start it — daemon-reload is mandatory after any unit change because systemd caches parsed units in memory:

sudo systemctl daemon-reload
sudo systemctl enable --now widgetd.service
systemctl status widgetd.service

Replacing cron with systemd timers

Timers are a strictly better cron for anything you want to observe: full journald logging (no >> /var/log/x 2>&1 and no logrotate to configure), dependency ordering (run only After= the database is up), resource control (the timer’s service is just a service, so it inherits cgroup limits and sandboxing), and Persistent= to catch up on runs missed while the machine was off. A timer is always two units: a .service that does the work and a .timer that triggers it.

/etc/systemd/system/backup.service:

[Unit]
Description=Nightly database backup
Wants=postgresql.service
After=postgresql.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/db-backup.sh
DynamicUser=yes
StateDirectory=db-backup

/etc/systemd/system/backup.timer:

[Unit]
Description=Run database backup nightly at 02:30

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=300
AccuracySec=1m

[Install]
WantedBy=timers.target

You enable the timer, not the service (the timer starts the service on schedule):

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers --all

The [Timer] directives

Timers come in two flavours: monotonic (relative to an event like boot or the last run) and realtime/calendar (wall-clock). The full directive set:

Three of these do the heavy lifting. Persistent=true records the last run on disk; if the machine was off at 02:30, the job fires shortly after the next boot — the cron-with-anacron behaviour you almost always want and the single best reason to switch. RandomizedDelaySec= jitters the start so a hundred machines don’t all hit the backup target at exactly 02:30:00. AccuracySec= lets systemd batch nearby timers to reduce wakeups (great on laptops and dense hosts); the default one minute is fine unless you genuinely need second precision.

For “every 15 minutes” style intervals, use the monotonic pair — OnBootSec= for the first run after boot and OnUnitActiveSec= for the repeat:

[Timer]
OnBootSec=15min
OnUnitActiveSec=15min

Writing and validating OnCalendar expressions

OnCalendar= uses the grammar DayOfWeek Year-Month-Day Hour:Minute:Second, where * is a wildcard, .. is a range, / is a step, and , is a list. Common patterns:

Always validate an expression before trusting it — systemd-analyze calendar normalises it and prints the next fire times, which is the fastest way to prove it means what you think:

systemd-analyze calendar "Mon..Fri 09:00" --iterations 5
# Prints Normalized form and the next 5 elapse timestamps

Timers versus cron, feature by feature, so the migration case is explicit:

Resource control with cgroup v2

Every service gets its own cgroup, so you control CPU, memory, IO, and task count with [Service] directives (or a drop-in) that map straight onto kernel controllers — no ulimit guesswork and no manual cgcreate. A representative block:

[Service]
# CPU: hard-cap at 1.5 cores; weight only matters under contention
CPUQuota=150%
CPUWeight=200

# Memory: soft-throttle at 512M, hard OOM-kill the cgroup at 768M
MemoryHigh=512M
MemoryMax=768M

# Block IO, per device
IOWeight=100
IOWriteBandwidthMax=/dev/sda 20M

# Fork-bomb protection
TasksMax=256

CPU control

The distinction: CPUQuota=150% allows 1.5 cores of CPU time even if the box is otherwise idle — an absolute ceiling. CPUWeight=200 only does anything when cgroups compete for the CPU; it divides contended time proportionally (a weight-200 service gets twice the contended share of a weight-100 one) and is invisible on an idle machine.

Memory control

The pattern that matters: set MemoryHigh below MemoryMax so the workload gets a chance to reclaim and stall (MemoryHigh) before the hammer of the OOM killer (MemoryMax) drops. A JVM with MemoryHigh=3500M MemoryMax=4G can shed heap under GC pressure at 3.5G instead of being killed mid-request at 4G.

IO and task control

IOWeight= is proportional like CPUWeight=; the *BandwidthMax=/*IOPSMax= directives are absolute per-device caps and require the device path (/dev/sda). TasksMax=256 is a cheap, effective fork-bomb guard — cap the PIDs a compromised or buggy service can spawn.

Applying and inspecting limits live

Apply resource limits without editing the unit using set-property, which writes a drop-in and applies it immediately, then inspect the live accounting:

# Write a drop-in and apply it live (survives reboot)
sudo systemctl set-property widgetd.service MemoryMax=1G CPUQuota=200%

# Confirm the effective values
systemctl show widgetd.service -p MemoryMax -p CPUQuota -p TasksMax

# top for cgroups — per-service CPU / memory / IO, live
systemd-cgtop

systemd-cgtop is the fastest way to find the noisy neighbour on a box — it is top for cgroups, sorted by resource. If systemctl show reports zeros, accounting is off; set MemoryAccounting=yes/CPUAccounting=yes explicitly (most distros enable them by default, but not all).

Slices: shared budgets for a group of services

A slice is a cgroup node that groups services under a shared limit — the clean way to guarantee background work can never starve foreground work. Create /etc/systemd/system/batch.slice:

[Unit]
Description=Batch/background workload slice (capped)

[Slice]
CPUQuota=200%
MemoryMax=4G
IOWeight=50

Then assign services to it with Slice=batch.slice in their [Service] section. Every service in batch.slice now shares that 2-core, 4 GB, low-IO-priority budget, collectively — no single batch job, and no sum of them, can exceed it. The default slice layout you build on:

Sandboxing and service hardening

This is where systemd quietly replaces a pile of AppArmor/SELinux work for common cases. Each directive shrinks the blast radius of a compromised process. A hardened [Service] block:

[Service]
# Filesystem
ProtectSystem=strict              # whole filesystem read-only...
ReadWritePaths=/var/lib/widgetd   # ...except these
ProtectHome=true
PrivateTmp=true

# Privilege
NoNewPrivileges=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Kernel / device / syscall exposure
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
PrivateDevices=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
SystemCallArchitectures=native
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictSUIDSGID=true

The directives, grouped by what they defend, with the high-value ones flagged:

The four highest-value moves, in order: DynamicUser=yes (or NoNewPrivileges=true if you need a fixed user) is the keystone — turn it on everywhere unless the service genuinely needs to escalate; ProtectSystem=strict + ReadWritePaths= makes the filesystem read-only except where the service must write; CapabilityBoundingSet= drops every capability the service doesn’t need (the canonical case is a web server that needs only CAP_NET_BIND_SERVICE to bind port 80/443, granted to the unprivileged process via AmbientCapabilities=); and SystemCallFilter=@system-service is a seccomp allowlist of syscalls appropriate for typical services, implicitly blocking dangerous groups like @swap, @reboot, @module, and raw @debug.

The critical operational rule: hardening is empirical and must be tested against the running service, not just “it started.” A directive like SystemCallFilter or MemoryDenyWriteExecute can let a service start and then kill it on the first request that hits a filtered syscall or a JIT compile. Add one directive, restart, exercise the service, watch the journal for EPERM/Operation not permitted/SIGSYS, and only then add the next.

systemd ships a scoring tool — run it after hardening and treat the score as a checklist to drive down:

systemd-analyze security widgetd.service
# Rates each unit and directive from "UNSAFE" through "EXPOSED" to "OK",
# with a numeric exposure score and a per-directive breakdown.

The exposure score bands:

Drop-in overrides and templated units

Drop-ins: never edit a shipped unit

To tweak a vendor unit, you do not copy it into /etc/systemd/system/. You add a drop-in — a fragment merged on top of the base unit, which survives package upgrades because the shipped file is untouched. The clean path:

sudo systemctl edit nginx.service

This opens an editor for /etc/systemd/system/nginx.service.d/override.conf. Add only the deltas:

[Service]
LimitNOFILE=65536
Restart=on-failure
RestartSec=5

Verify the merge and the effective value:

systemctl cat nginx.service              # base unit + every drop-in, in order
systemctl show nginx.service -p LimitNOFILE

The list-valued gotcha: most directives are additive or last-one-wins, but list-valued keys like ExecStart= are not simply overridden — a second ExecStart= appends, giving you two start commands. To replace one, clear it first with an empty assignment, then set the new value:

[Service]
ExecStart=
ExecStart=/usr/local/bin/nginx-wrapper

The same empty-then-set pattern applies to Environment=, ExecStartPre=, and any other list-valued directive. How the merge behaves by key type:

Templated (instanced) units

A unit file with @ in its name is a template. The string after @ is the instance, available in the file as %i (escaped) or %I (unescaped). One template file backs any number of running instances, each with independent state, logs, and lifecycle. /etc/systemd/system/tunnel@.service:

[Unit]
Description=SSH tunnel to %i
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/autossh -M 0 -N %i
Restart=always
RestartSec=10
DynamicUser=yes

[Install]
WantedBy=multi-user.target

Start as many instances as you like from the one file:

sudo systemctl enable --now tunnel@db-replica.service
sudo systemctl enable --now tunnel@cache-01.service

The commonly used specifiers:

Use systemd-escape to safely build instance names containing slashes or special characters, which would otherwise break the unit name:

systemd-escape --template tunnel@.service "user@10.0.0.5"
# tunnel@user\x4010.0.0.5.service

Architecture at a glance

Picture PID 1 — the systemd manager — as the hub every arrow points to. At boot it reads unit files from the priority-ordered directories (/etc/systemd/system overriding /run overriding /usr/lib), parses each into a typed object, and assembles them into a single directed graph. Two kinds of edge run through that graph: dependency edges (Wants=/Requires=), which decide which units get pulled in, and ordering edges (After=/Before=), which decide the sequence. systemd computes a transaction that satisfies both sets simultaneously and drives units through their state machines toward the default target (usually multi-user.target) — the milestone that means “the system is up.” Targets like network-online.target sit in this graph as synchronisation points that gate everything ordered after them.

Below the graph sits the cgroup v2 tree, and this is the part most mental models miss: it is not a separate system bolted on, it is where every service lives. -.slice at the root branches into system.slice (your daemons), user.slice (login sessions), and machine.slice (guests); each service is a leaf cgroup nested under its slice, and the resource directives you write in [Service] (CPUQuota, MemoryMax, IOWeight, TasksMax) are values written into that leaf’s controller files. A custom system-batch.slice with its own ceiling is just another branch, and every service assigned to it shares that branch’s budget. Wrapping each service is its sandbox — a private mount namespace (from ProtectSystem/PrivateTmp), a reduced capability set, and a seccomp filter — so the process sees a stripped-down view of the machine.

To the side runs journald, receiving structured log records from every unit’s stdout/stderr and from sd_journal calls, tagged with the unit name, PID, priority, and boot ID, and indexed so journalctl -u <unit> -p err -b is a database query, not a grep. Timers hang off the graph too: a .timer unit is an edge that says “at this wall-clock time (or interval), start this .service.” Follow any incident backward through this picture — a failed unit → its journal → its place in the ordering graph → its cgroup accounting — and the diagnosis is mechanical rather than mysterious. There is no separate image for this article; the model above is the map to keep in your head.

Real-world scenario

Meridian Payments ran a JVM settlement service — call it settlementd — on a fleet of 12 bare-metal Linux hosts (systemd v249, cgroup v2), one instance per host, under a unit that a departing engineer had written years earlier. The unit was minimal: Type=simple, Restart=always, MemoryMax=4G, no watchdog, no start-limit, ordered on network.target. It processed batch settlements every few minutes, fanning out to a payment provider and a Postgres cluster. The platform team was five engineers; the fleet cost roughly ₹9,00,000/month in colocation and support.

The incident began during a payment-provider outage. The provider’s API started hanging — accepting connections but never responding — and settlementd’s worker threads blocked one by one on those calls. Within minutes every thread was parked, the heap filled with queued work, GC went into a thrash, and the JVM stopped making progress. But it did not die: the process was alive, the port was open, systemctl status was green. Because the unit had no watchdog, systemd saw a perfectly healthy service. Settlements silently backed up for 40 minutes until on-call noticed the queue depth on a business dashboard — not from any systemd alert.

It got worse. As the heap grew, the cgroup hit MemoryMax=4G and the cgroup OOM killer reaped the JVM mid-batch, corrupting an in-flight settlement file. Restart=always immediately restarted it — but because Type=simple reports “started” at fork, the fresh JVM took traffic before its DB pool had reconnected, threw a wall of connection errors, and the whole cycle began again. With no StartLimitBurst, systemd restarted it in a tight loop, each restart hammering the recovering Postgres cluster with a reconnect storm. A single upstream hang had turned into a self-inflicted outage.

The fix was a redesign of the unit around true liveness, not process-alive. They switched to Type=notify and added a dedicated watchdog thread that pinged sd_notify(WATCHDOG=1) only when the work loop was making progress and the DB pool was reachable — so a wedge trips it in seconds. They split the memory limits so the JVM could shed heap under pressure before being killed, and added a crash-loop breaker:

[Unit]
Description=Settlement processor
Wants=network-online.target postgresql.service
After=network-online.target postgresql.service
StartLimitIntervalSec=120
StartLimitBurst=4

[Service]
Type=notify
WatchdogSec=20
Restart=on-watchdog
RestartSec=5
MemoryHigh=3500M
MemoryMax=4G
TimeoutStopSec=45

// Ping the watchdog only when genuinely able to serve — every WatchdogSec/2 = 10s
if (workLoop.isMakingProgress() && dbPool.isReachable()) {
    sdNotify.notifyWatchdog();
}

The next provider outage was a non-event. The wedge tripped the watchdog in 20 seconds; Restart=on-watchdog cycled the service cleanly; MemoryHigh let the heap shed at 3.5G so the OOM killer never fired and no settlement file was corrupted; and StartLimitBurst=4 in a 120-second window meant that if the service itself were genuinely broken it would give up after four tries instead of storming Postgres. TimeoutStopSec=45 gave in-flight batches time to drain on a clean stop. The incident timeline, because the order of what changed is the lesson:

The line that went on the runbook: “Restart= fires on death; a deadlocked service never dies. If you care about liveness, you want a watchdog, not a restart policy.”

Advantages and disadvantages

The unified-init-plus-cgroups-plus-sandbox model is powerful precisely because it is centralised — and that same centralisation is what people trip over. Weigh it honestly:

The model is right for essentially any Linux server workload — it is the default init on every mainstream distribution and the mechanism behind container runtimes (Quadlet, systemd-nspawn) too. It bites hardest on teams carrying SysV mental models forward, opaque third-party binaries that can’t call sd_notify, JIT runtimes that break under MemoryDenyWriteExecute, and anyone who copies a vendor unit into /etc instead of using a drop-in. Every disadvantage above is manageable — but only once you know it exists, which is the entire point of learning the contract rather than poking it with start/stop.

Hands-on lab

Build a real service unit from scratch, add resource limits and hardening, migrate a task to a timer, watch it in the journal, and tear it all down. Everything runs on a single modern Linux VM (Ubuntu 22.04+/Fedora 38+/Debian 12+) with sudo; no network services are required. Confirm your baseline first.

Step 1 — Confirm systemd version and cgroup v2.

systemctl --version | head -1
stat -fc %T /sys/fs/cgroup/

Expected: version >= 245, and cgroup2fs (the unified hierarchy). If you see tmpfs/cgroup, you are on the legacy hybrid v1 hierarchy and the resource directives behave differently.

Step 2 — Write a minimal service that runs a script. Create the payload:

sudo tee /usr/local/bin/labworker.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
echo "labworker starting (pid $$)"
# systemd-notify tells Type=notify we are ready; works in shell via the CLI
systemd-notify --ready 2>/dev/null || true
while true; do
  echo "labworker heartbeat $(date -Is)"
  sleep 5
done
EOF
sudo chmod +x /usr/local/bin/labworker.sh

Create the unit /etc/systemd/system/labworker.service:

sudo tee /etc/systemd/system/labworker.service >/dev/null <<'EOF'
[Unit]
Description=Lab worker (systemd deep-dive)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=notify
ExecStart=/usr/local/bin/labworker.sh
Restart=on-failure
RestartSec=2
DynamicUser=yes
StateDirectory=labworker

[Install]
WantedBy=multi-user.target
EOF

Step 3 — Validate before running, then start. Never skip the verify:

sudo systemd-analyze verify /etc/systemd/system/labworker.service   # silent = clean
sudo systemctl daemon-reload
sudo systemctl enable --now labworker.service
systemctl status labworker.service --no-pager

Expected: status shows Active: active (running) and — because Type=notify — a Status: line only after the script called systemd-notify --ready. systemctl is-enabled labworker.service returns enabled.

Step 4 — Watch its logs in the journal.

journalctl -u labworker.service -n 20 --no-pager
journalctl -u labworker.service -f          # follow; Ctrl-C to stop

Expected: labworker starting followed by labworker heartbeat … lines every 5 seconds, each tagged with the unit.

Step 5 — Apply resource limits live and inspect them.

sudo systemctl set-property labworker.service CPUQuota=50% MemoryMax=128M TasksMax=32
systemctl show labworker.service -p CPUQuota -p MemoryMax -p TasksMax
systemctl status labworker.service --no-pager | grep -A3 'Memory\|CPU'

Expected: CPUQuotaPerSecUSec reflects 50%, MemoryMax=134217728 (128 MiB), TasksMax=32. set-property wrote a drop-in under /etc/systemd/system/labworker.service.d/ — confirm with systemctl cat labworker.service (you will see the base unit plus a 50-CPUQuota.conf-style fragment).

Step 6 — Harden it and score the result. Add a drop-in via edit (use --full off; append only deltas). For a scripted lab, write the drop-in directly:

sudo mkdir -p /etc/systemd/system/labworker.service.d
sudo tee /etc/systemd/system/labworker.service.d/hardening.conf >/dev/null <<'EOF'
[Service]
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/labworker
ProtectHome=true
PrivateTmp=true
ProtectKernelTunables=true
ProtectKernelModules=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
SystemCallFilter=@system-service
LockPersonality=true
EOF
sudo systemctl daemon-reload
sudo systemctl restart labworker.service
systemctl status labworker.service --no-pager | head -5
systemd-analyze security labworker.service | tail -5

Expected: the service still runs (heartbeats continue — confirm with journalctl -u labworker.service -n 5), and the systemd-analyze security exposure score is markedly lower than before hardening. If the service had failed after a directive, that directive was too strict — the empirical loop in action.

Step 7 — Migrate a task to a timer. Create a oneshot service and a timer that fires every minute for the lab (you’d use OnCalendar= in production):

sudo tee /etc/systemd/system/labreport.service >/dev/null <<'EOF'
[Unit]
Description=Lab periodic report
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "report run at $(date -Is), uptime: $(uptime -p)"'
DynamicUser=yes
EOF

sudo tee /etc/systemd/system/labreport.timer >/dev/null <<'EOF'
[Unit]
Description=Run the lab report every minute
[Timer]
OnBootSec=30s
OnUnitActiveSec=1min
Persistent=true
RandomizedDelaySec=10
[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now labreport.timer
systemd-analyze calendar "*-*-* 02:30:00" --iterations 3     # validate a real cron-style expr
systemctl list-timers labreport.timer --no-pager

Expected: list-timers shows labreport.timer with a NEXT fire time within a minute and its UNIT/ACTIVATES columns. Wait ~90 seconds, then read the results:

journalctl -u labreport.service -n 10 --no-pager

Expected: one or more report run at … lines, each a separate oneshot invocation triggered by the timer.

Validation checklist. You wrote a Type=notify service that gates on real readiness, gave it a crash-loop breaker, applied live cgroup limits, hardened it and verified it still runs under the sandbox, and replaced a periodic task with a logged, jittered, catch-up-capable timer. What each step proved:

Teardown (leave the box clean):

sudo systemctl disable --now labworker.service labreport.timer
sudo rm -f /etc/systemd/system/labworker.service \
           /etc/systemd/system/labreport.service \
           /etc/systemd/system/labreport.timer \
           /usr/local/bin/labworker.sh
sudo rm -rf /etc/systemd/system/labworker.service.d
sudo systemctl daemon-reload
sudo systemctl reset-failed

Common mistakes & troubleshooting

The failure modes that eat the most incident time, as a symptom → cause → confirm → fix playbook. Read the table at 02:14; read the prose underneath once, now.

The entries that bite hardest, expanded:

2. Crash-loop breaker never trips. StartLimitIntervalSec and StartLimitBurst moved to the [Unit] section in modern systemd; put them in [Service] and they are silently dropped. A genuinely broken service then restarts forever, hammering downstreams. Confirm with systemctl cat <u> and move them to [Unit]. This is the single most common “why is it still restarting?” cause.

4. Started ≠ ready races. With Type=simple, a database ordered After= your app connects before the app has bound its listener, gets connection-refused, and crashes; the second boot the race falls the other way and it “works.” The fix is Type=notify with the app calling sd_notify(READY=1) after it can actually serve — or socket activation, where systemd holds the socket so the first connection queues instead of failing.

10. Over-hardening kills the running service. A service that starts fine under SystemCallFilter=@system-service may die on the first request that hits a filtered syscall, or a JIT runtime may SIGSYS under MemoryDenyWriteExecute=true. The journal shows Operation not permitted or a SIGSYS. Remove the last directive you added, confirm the service is healthy under load, then add syscall groups back one at a time (SystemCallFilter=@system-service @pkey etc.). Always test the running service, not just that it starts.

16. Ordering cycles. If your dependency graph has a loop, systemd breaks it by deleting a unit from the transaction — and it tells you which one in the journal (“Found ordering cycle on X, deleting Y to break it”). Do not let systemd choose: journalctl -b | grep -i 'ordering cycle' names the units, and you fix it by relaxing one After=/Requires= in the loop yourself.

The debugging commands worth memorising:

Reach for systemd-analyze critical-chain rather than blame when boot is slow: blame lists slow units in isolation, but critical-chain shows the serialised path that actually gated reaching the target, accounting for parallelism — the unit at the top of the chain is the one to fix.

Best practices

• Run systemd-analyze verify on every custom unit before deploy. It catches typos, deprecated keys, and bad references without starting anything. Wire it into CI for units under version control.
• Prefer Type=notify (or socket activation) wherever readiness matters; use Type=exec over bare simple. “Started” must mean “ready,” or ordered dependents race a half-started service.
• Always include an [Install] section with WantedBy=multi-user.target (or the right target) so enable actually wires the unit into boot. Confirm with systemctl is-enabled.
• Put the crash-loop breaker in [Unit]: StartLimitIntervalSec + StartLimitBurst, paired with Restart=on-failure. Verify placement with systemctl cat.
• Default to Wants=; reserve Requires=/BindsTo= for genuinely hard dependencies. And always pair a dependency with the matching After= ordering, or you get a race.
• Order network-needing services on network-online.target with Wants=network-online.target, and ensure the relevant wait-online service is enabled — never plain network.target when you need a routable address.
• Customise vendor units with systemctl edit drop-ins, never by editing /usr/lib/systemd/system. Drop-ins survive package upgrades; edited shipped files get reverted.
• Migrate cron to timers with Persistent=true and RandomizedDelaySec=. You gain journald logging, catch-up on missed runs, fleet jitter, ordering, and resource control for free.
• Set resource limits with MemoryHigh < MemoryMax, a sane CPUQuota, and TasksMax, and use slices (system-batch.slice) to give background work a shared ceiling that can’t starve the foreground.
• Harden empirically: turn on NoNewPrivileges/DynamicUser, ProtectSystem=strict (+ReadWritePaths), a trimmed CapabilityBoundingSet, and SystemCallFilter=@system-service — then exercise the running service after each directive and drive systemd-analyze security down.
• Make journald persistent (Storage=persistent) with sane retention (SystemMaxUse=, MaxRetentionSec=) so post-mortem evidence survives a reboot.
• Manage units, drop-ins, and timers as version-controlled files deployed by config management, reviewed in PRs — a unit is code, and a bad ExecStart= or a wrong Requires= is a production landmine.

Security notes

• Least privilege by default. DynamicUser=yes (transient UID, private state, locked-down FS) is the strongest single move; where a fixed identity is required, run as a dedicated unprivileged User= and never as root. Add NoNewPrivileges=true universally so setuid binaries and file capabilities can’t escalate.
• Drop capabilities to the minimum. Set CapabilityBoundingSet= to exactly what the service needs (often nothing, or just CAP_NET_BIND_SERVICE for a low-port bind granted via AmbientCapabilities=). Every capability you leave in is a privilege an attacker inherits.
• Filter syscalls. SystemCallFilter=@system-service plus SystemCallArchitectures=native shrinks the kernel attack surface dramatically and blocks whole classes of exploit primitives (@module, @swap, @reboot, raw @debug).
• Make the filesystem read-only and hide what the service doesn’t need. ProtectSystem=strict + narrow ReadWritePaths=, ProtectHome=true, PrivateTmp=true, PrivateDevices=true, and ProtectKernelTunables/ProtectKernelModules/ProtectControlGroups=true.
• Prefer socket activation for privileged ports. systemd can bind the low port and pass the descriptor, so the service itself never needs CAP_NET_BIND_SERVICE at all.
• Protect the journal and its retention. Persistent logs under /var/log/journal are an audit source — restrict read access (the systemd-journal group), forward to a central store, and set retention so evidence isn’t rotated away before an investigation. This complements a full Linux audit trail with auditd and eBPF.
• Layer MAC on top for the cases systemd can’t cover. systemd sandboxing is namespace/seccomp/capability based; for label-based confinement of complex file access, pair it with AppArmor or SELinux in enforcing mode. The two are complementary, not redundant.
• Score and track. Run systemd-analyze security across the fleet, treat every UNSAFE/EXPOSED unit as a backlog item, and re-score after every change so hardening doesn’t regress.

Cost & sizing

systemd itself is free and already running — the “cost” is entirely in the compute you right-size with it and the incidents you prevent. The levers:

• Resource limits are consolidation economics. Per-service cgroup caps let you safely pack more workloads onto one box without a runaway service taking the machine down — the difference between one service per VM (wasteful) and a tiered set of services under slices on a shared host. On a ₹6,000/month cloud VM, moving three lightly-loaded services from three separate boxes onto one with system-batch.slice and per-service MemoryMax saves roughly ₹12,000/month with no availability loss, because the slice ceiling guarantees isolation.
• The watchdog and crash-loop breaker prevent the expensive failure. The Meridian scenario — a 40-minute silent settlement backlog — is the real cost of a missing WatchdogSec. One prevented incident of that class dwarfs any infrastructure line item.
• journald retention is a storage tradeoff. Persistent logs consume disk; bound it with SystemMaxUse= (cap total size, e.g. 2G) and MaxRetentionSec= (e.g. 14day). Under-provisioning fills /var and can wedge the box; over-provisioning wastes disk. A typical server does fine with a 1–4 GB journal cap.
• Timers cost nothing extra over cron — they run the same work on the same schedule — but the catch-up (Persistent=) and jitter (RandomizedDelaySec=) prevent the fleet-stampede pattern where a hundred machines hit a backup target simultaneously and you over-provision the target to survive the spike. Jitter is free headroom.

There is no free-tier or SKU dimension here — the sizing question is CPU/RAM/disk of the underlying host, and systemd’s contribution is letting you use less of it safely. Rough guidance: set MemoryMax at ~1.3× a service’s steady-state working set and MemoryHigh at ~0.9× MemoryMax; set CPUQuota only where a service has a genuine ceiling (leave latency-sensitive services uncapped but give them a high CPUWeight); and cap the journal at 1–4 GB on a general server, more on a log-heavy one.

Interview & exam questions

1. What is the difference between a dependency and an ordering directive, and why do you usually need both? Dependency (Wants=/Requires=) controls whether another unit is pulled in when this one starts; ordering (After=/Before=) controls when it runs relative to others. They are orthogonal: Requires= without After= starts both in parallel (a race), and After= without Wants=/Requires= orders a unit that may not even be present. You typically pair them — e.g. Wants=postgresql.service + After=postgresql.service.

2. Why is Type=simple dangerous, and what fixes it? Type=simple marks the service “started” the instant it forks — before it has bound sockets or connected to dependencies — so units ordered After= it can launch into a not-yet-ready service and race. The fix is Type=notify (the service calls sd_notify(READY=1) when it can genuinely serve) or socket activation (systemd holds the socket so connections queue). Use Type=exec at minimum, which at least confirms the binary execve()'d.

3. You put StartLimitBurst in [Service] and the service still crash-loops forever. Why? StartLimitIntervalSec and StartLimitBurst must be in the [Unit] section in modern systemd; in [Service] they are silently ignored, so the crash-loop breaker does nothing. Move them to [Unit], pair with Restart=on-failure, and confirm placement with systemctl cat.

4. What does the [Install] section do, and what happens without it? [Install] is inert while the service runs; it only takes effect at systemctl enable, which reads WantedBy=/RequiredBy= and creates the symlink that wires the unit into a boot target. Without [Install], enable has nothing to do and the unit never starts at boot — even though start works right now. systemctl is-enabled returns static instead of enabled.

5. How is a systemd timer better than cron, concretely? Timers give per-unit journald logging (no manual redirection), Persistent=true catch-up for runs missed while the machine was off, RandomizedDelaySec= jitter to avoid fleet stampedes, dependency ordering (After=/Wants=), full cgroup resource control and sandboxing on the triggered service, and visible history via systemctl status/list-timers. The service the timer triggers is just a normal .service, so it inherits every [Service] capability.

6. Explain MemoryHigh vs MemoryMax. MemoryMax is the hard ceiling — exceeding it invokes the cgroup OOM killer, which reaps a process. MemoryHigh is a soft throttle — past it the kernel aggressively reclaims and stalls the cgroup but does not kill. Set MemoryHigh below MemoryMax so the workload can shed memory under pressure before the OOM killer fires — critical for GC’d runtimes like the JVM.

7. What is the difference between CPUQuota and CPUWeight? CPUQuota is an absolute ceiling on CPU time (150% = 1.5 cores) enforced even on an idle box. CPUWeight (1–10000, default 100) is relative and only matters under contention — it divides contended CPU proportionally among competing cgroups and is invisible when the CPU isn’t saturated. Use CPUQuota for a hard cap, CPUWeight for prioritisation.

8. How do you customise a package-shipped unit so your change survives an upgrade? Use a drop-in: systemctl edit <unit> creates /etc/systemd/system/<unit>.d/override.conf, merged on top of the base unit, which stays untouched by the package. Never copy the unit into /etc/systemd/system (that shadows it and can drift), and never edit /usr/lib/systemd/system (upgrades revert it). For list-valued keys like ExecStart=, empty-reset first (ExecStart=) then set the new value, or you get two entries.

9. Name four high-value hardening directives and what each does. NoNewPrivileges=true (process/children can never gain privileges via setuid/fscaps); ProtectSystem=strict (whole filesystem read-only in the service’s namespace, with ReadWritePaths= exceptions); CapabilityBoundingSet= (drops every Linux capability not listed); SystemCallFilter=@system-service (seccomp allowlist blocking exotic/dangerous syscall groups). DynamicUser=yes is arguably the strongest single directive. Always test the running service and check systemd-analyze security.

10. A Type=forking daemon shows as failed or with the wrong PID. What’s missing? PIDFile=. A forking daemon double-forks and the parent exits, so systemd needs the PID file to know which surviving process is the real daemon. Without it, systemd tracks the wrong process (or none). Set PIDFile= to where the daemon writes it — or, better, switch the daemon to Type=notify/Type=exec if it supports foreground operation.

11. Your service starts fine but dies on the first real request after you added SystemCallFilter. How do you diagnose and fix it? The seccomp filter is blocking a syscall the service needs at runtime (not at startup). The journal (journalctl -u <u>) shows SIGSYS or Operation not permitted. Remove the last directive to confirm it’s the cause, then add back the specific syscall group the service needs (e.g. SystemCallFilter=@system-service @pkey). The lesson: hardening must be tested against the exercised service, not just “it started.”

12. What does network-online.target guarantee that network.target does not, and how do you use it? network.target is only an ordering point for the networking stack coming up — it does not mean the network is configured. network-online.target blocks until an address is actually configured and routable, but only if the relevant wait-online service (systemd-networkd-wait-online/NetworkManager-wait-online) is enabled. A service needing a routable IP at start uses both After=network-online.target and Wants=network-online.target.

These map to the LFCS and RHCSA/RHCE objectives (manage services with systemd, control boot targets, schedule with timers, and — RHCE — configure resource limits and analyse boot), and to the LPIC-1/LPIC-2 system-startup and service-management topics. The cgroup and hardening material is directly relevant to the CKA/CKS worldview even though Kubernetes abstracts it, because the kubelet, containerd, and node-level DaemonSets all run as systemd units you must debug. A compact mapping for revision:

Quick check

1. You edit a unit file and systemctl restart it, but the change doesn’t take effect. What one command did you forget, and why is it needed?
2. A service ordered After= yours connects to yours and gets “connection refused” on some boots but not others. What Type= is your service almost certainly using, and what do you switch it to?
3. True or false: putting StartLimitBurst=5 in the [Service] section gives your service a crash-loop breaker.
4. You migrated a nightly backup from cron to a systemd timer, but on a laptop that sleeps overnight the backup silently never runs. Which one [Timer] directive did you omit?
5. What is the difference between MemoryHigh and MemoryMax, and why set MemoryHigh lower?

Answers

1. sudo systemctl daemon-reload. systemd parses unit files into memory and runs from that cache; without a reload it keeps using the old parsed unit, so your edit is ignored until you reload (then restart).
2. Your service is almost certainly Type=simple, which reports “started” the instant it forks — before it has bound its listener — so the dependent races it. Switch to Type=notify (with the service calling sd_notify(READY=1) when it can serve) or use socket activation so connections queue instead of failing.
3. False. StartLimitIntervalSec/StartLimitBurst must be in the [Unit] section; in [Service] they are silently ignored and the service can crash-loop forever. Pair them with Restart=on-failure.
4. Persistent=true. It records the last run on disk so a run missed while the machine was off fires shortly after the next boot (the cron-with-anacron behaviour). Without it, a missed scheduled time is simply skipped.
5. MemoryMax is the hard limit — breaching it triggers the cgroup OOM killer, which reaps a process. MemoryHigh is a soft throttle — past it the kernel aggressively reclaims and stalls the cgroup but does not kill. Set MemoryHigh below MemoryMax so the workload (especially a GC’d runtime) can shed memory before the OOM killer ever fires.

Glossary

• Unit — a typed object systemd manages, identified by filename suffix (.service, .timer, .socket, .target, .slice, .mount, .path, .scope, .device, .swap).
• [Unit] / [Service] / [Install] — the three sections of a service file: metadata + dependency/ordering; process type/exec/restart/resources/sandbox; enable-time wiring, respectively.
• Type= — how systemd decides a service is “started”: simple (fork), exec (execve), forking (parent exits, needs PIDFile=), oneshot (runs and exits), notify (calls sd_notify(READY=1)), dbus, idle.
• Dependency (Wants=/Requires=) — controls whether another unit is pulled in; Wants= is soft (its failure doesn’t fail you), Requires= is hard (its failure/stop takes you down).
• Ordering (After=/Before=) — controls when a unit runs relative to another; independent of dependency.
• Target — an abstract grouping/synchronisation point (multi-user.target, graphical.target, network-online.target); the systemd analogue of runlevels.
• Timer — a .timer unit that starts a companion .service on a schedule; realtime (OnCalendar=) or monotonic (OnBootSec=/OnUnitActiveSec=).
• OnCalendar= — wall-clock schedule grammar DoW Y-M-D H:M:S with *, .., /, ,; validated with systemd-analyze calendar.
• Persistent= — timer directive that records the last run on disk and catches up runs missed while the machine was off.
• cgroup v2 — the unified kernel control-group hierarchy under /sys/fs/cgroup/; the mechanism behind all systemd resource limits.
• Slice — a .slice cgroup node grouping services under a shared resource budget (system.slice, system-batch.slice).
• CPUQuota / CPUWeight — absolute CPU ceiling (percent, can exceed 100%) versus relative share under contention (1–10000).
• MemoryHigh / MemoryMax — soft reclaim throttle versus hard limit enforced by the cgroup OOM killer.
• TasksMax — cap on PIDs (threads + processes) in a service’s cgroup; a fork-bomb guard.
• Sandboxing directives — [Service] keys that shrink attack surface: NoNewPrivileges, ProtectSystem=strict, CapabilityBoundingSet, SystemCallFilter, PrivateTmp, DynamicUser, and more.
• DynamicUser=yes — allocates a transient UID/GID for the service lifetime with private /tmp and locked-down filesystem access; the highest-leverage single hardening directive.
• Drop-in — an override fragment at /etc/systemd/system/<unit>.d/*.conf merged on top of a base unit; the upgrade-safe way to customise vendor units. Created with systemctl edit.
• Templated (instanced) unit — a name@.service file where %i/%I is the instance, backing many independent instances from one file.
• journald — the systemd journal: structured, indexed, per-unit logging queried with journalctl; volatile (/run) unless Storage=persistent.
• sd_notify — the protocol a service uses to signal readiness (READY=1) and liveness (WATCHDOG=1) to systemd; underpins Type=notify and WatchdogSec=.
• systemd-analyze — the diagnostic tool: verify (validate a unit), security (score hardening), calendar (validate a timer), blame/critical-chain (boot timing).

Next steps

You can now write, harden, resource-limit, schedule, and debug systemd units from a blank file. Build outward:

• Next: Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers — layer system-wide tuning on top of the per-service cgroup limits you set here.
• Related: Running Rootless Containers in Production with Podman and Quadlet — Quadlet generates systemd units for containers, so everything above applies to containerised workloads.
• Related: Authoring AppArmor Profiles: Confining Services on Ubuntu and Debian — label-based MAC that complements systemd’s namespace/seccomp sandboxing.
• Related: Running SELinux in Enforcing Mode: Troubleshooting and Writing Custom Policy — the other major MAC system, for the confinement cases systemd directives can’t cover.
• Related: Building a Linux Audit Trail with auditd and eBPF Runtime Visibility — journald is one input; auditd and eBPF give you syscall-level forensics.
• Related: Automating Linux Patching: dnf-automatic, Live Patching, and Reboot Orchestration — orchestrate reboots and package updates that interact with the units and drop-ins you manage.