Writing systemd Units That Wrap Shell Scripts Properly: Type Selection, Restart Policy, Hardening, Watchdogs, Timers & sd_notify

In a nutshell

When you run a script by hand over SSH, you are its supervisor. You watch it, you notice when it dies, you read what it prints, you restart it, and you kill it when you log off. The trouble is that you are not there at 3 a.m., you don’t survive a reboot, and you can’t watch fifty machines at once. A systemd unit is how you write that supervision down once, on paper, and hand it to a manager that never sleeps.

Think of the unit file — a little myapp.service text file — as an employment contract for your script. The script is a new hire. The contract spells out the terms:

What is the job? ExecStart= — the one command to run.
Is this a one-off task or a permanent post? Type= — does the script run once and clock out (oneshot), or stay on shift forever (simple/notify)? Getting this wrong is like filing a full-time hire as a day-labourer: the manager keeps reporting the role “vacant” the moment they step away.
Who signs off that they’ve actually started work? sd_notify — a permanent hire says “I’m at my desk and ready” instead of the manager just assuming so the instant they badge in.
What happens if they collapse? Restart= and RestartSec= — bring them back, but not so frantically that the whole office grinds to a halt in a hiring loop.
What are they allowed to touch? The sandboxing directives — a locked-down desk, no master keys, read-only access to everything they don’t strictly need.
Where do their memos go? journald — one filing cabinet the whole company can search, not a sticky note on their monitor.

Here is the mental model to carry through the whole lesson: the two lines that “work” ([Service] + ExecStart=) are the easy 10%. The other 90% — the part that decides whether you can leave the service running unattended for a year — is everything the contract says around that one command. What counts as “started”? What happens on crash? On a hang? On reboot? What can a bug in the script reach? Where do the logs go? This lesson teaches both halves, and by the end every script you ship will come with a hardened contract instead of a hopeful nohup.

Level: Advanced · Time: ~45 min

Prerequisites: You should be comfortable with strict-mode Bash and exit codes, and you’ll get the most from this lesson if you’ve already met a few neighbours in this course: the cron vs systemd-timer decision and the idempotency it demands, structured logging to journald, handling secrets without leaking them, and trapping SIGTERM/SIGHUP for clean shutdown and reload. We lean on all four here.

After this lesson you will be able to:

Choose the right Type= (oneshot, simple, exec, notify) for a given script — and explain what “the service is active” actually proves in each case.
Write a restart policy that recovers from crashes without death-looping the CPU, and a watchdog that catches a service that’s running but wedged.
Sandbox a script from outside itself with ProtectSystem, PrivateTmp, NoNewPrivileges, capability and syscall filters — then score it with systemd-analyze security.
Signal real readiness with sd_notify from a shell script, and dodge the NotifyAccess PID trap that silently swallows the ready ping.
Feed config and secrets in safely with EnvironmentFile= and LoadCredential=, and pair a .service with a .timer to replace cron with catch-up, jitter and per-run sandboxing.
Operate any unit fluently: daemon-reload, enable --now, status, journalctl -u, drop-in overrides — and know what to reach for (launchd) when the host isn’t Linux.

Concept map for wrapping a shell script as a systemd service: the script, the [Service] contract (Type, ExecStart, User), readiness and restart/watchdog, the sandbox, and observability with journald plus a paired timer

Read the diagram left → right: your script is turned into a service by the contract in the unit file — Type= declares its shape, ExecStartPre→ExecStart orders the work, User= gives it a non-root identity. It then signals readiness with sd_notify and stays alive under a restart-plus-watchdog policy, runs inside a sandbox (read-only filesystem, dropped privileges, secrets on tmpfs), and finally becomes observable and scheduled through journald and a paired .timer. systemd is Linux-only; macOS’s native equivalent is launchd, covered near the end.

Why a systemd Unit Around Your Script Matters

Your script runs fine when you SSH in and run it. You want it to run on reboot, restart on failure, log to journald, time out after an hour, run with restricted privileges, and pull secrets from /etc/myapp/env only readable by root. You write a unit file:

[Service]
ExecStart=/opt/myapp/bin/run.sh

It works. Six months later you discover:

The script ran with Type=simple (default), which means systemd considered it “started” the moment fork returned. Health checks based on “is the service active?” return success even when the script silently exits in the first second.
It restarts on crash, but with Restart=always and no RestartSec, a buggy script that crashes immediately consumes 100% CPU in a tight respawn loop until you manually systemctl stop it.
It writes to /var/log/myapp.log — but the systemd-journald is also collecting stdout/stderr, so logs are duplicated.
It runs as root because no User= was set. Anything the script can do, root can do — a shell-injection bug means root compromise.
The EnvironmentFile=/etc/myapp/env you added contains a secret in plain text, and you didn’t restrict file mode, so any local user can read it.

This lesson covers the unit file as a contract between your script and systemd: how to declare what the script promises (will exit normally, will signal readiness, won’t fork into the background), how to harden the runtime, and how to test that the unit does what you think it does. It’s the capstone of Tier 4 — every script you write at this level should ship with a hardened unit.

The Anatomy of a Service Unit File

A systemd unit lives at /etc/systemd/system/myapp.service (system-wide) or ~/.config/systemd/user/myapp.service (per-user). Reload after editing:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
sudo systemctl status myapp.service

Unit file structure:

[Unit]                          # metadata: when to start, what it depends on
Description=My App Service
After=network-online.target     # ordering: start after this
Wants=network-online.target     # weak dep: pulls it in if available
Requires=postgresql.service     # strong dep: fail if this can't start
ConditionPathExists=/etc/myapp/config.json  # don't start if this is missing

[Service]                       # how to run
Type=simple                     # how systemd knows when "started"
User=myapp                      # don't run as root
Group=myapp
EnvironmentFile=/etc/myapp/env  # source env vars from a file
ExecStartPre=/opt/myapp/bin/preflight.sh  # run BEFORE main; failure aborts service
ExecStart=/opt/myapp/bin/run.sh # the main process
ExecReload=/bin/kill -HUP $MAINPID  # systemctl reload sends this
ExecStopPost=/opt/myapp/bin/cleanup.sh  # run AFTER stop, success or fail
Restart=on-failure
RestartSec=5
TimeoutStartSec=60
TimeoutStopSec=30

[Install]                       # who triggers `systemctl enable`
WantedBy=multi-user.target      # start at multi-user (normal boot)

Three sections. [Unit] is the metadata; [Service] is the contract; [Install] is what systemctl enable activates.

One directive worth calling out early is WorkingDirectory=. Just like a login shell, a service has a current directory; if you don’t set it, it’s / (or the user’s home for user units), which is almost never what a script that does ./config.yaml or > output/data.json expects. Set WorkingDirectory=/opt/myapp so relative paths in the script resolve where you think they do — and prefer absolute paths inside the script regardless, because “it worked in my shell” is exactly the class of bug a service surfaces.

The Everyday Workflow: Edit → Reload → Enable → Inspect

Before we go deep on individual directives, here is the loop you’ll run hundreds of times. Every systemd change follows the same four beats: edit the unit, reload the daemon, enable/start it, then inspect.

# 1. EDIT — drop a unit at /etc/systemd/system/myapp.service (or use an override, see "Going deeper")
sudoedit /etc/systemd/system/myapp.service

# 2. RELOAD — systemd caches unit files in memory; it will NOT see your edit until you say so
sudo systemctl daemon-reload

# 3. ENABLE + START — "enable" wires up boot-time autostart; "--now" also starts it immediately
sudo systemctl enable --now myapp.service     # enable + start
sudo systemctl start myapp.service            # start this boot only (no autostart)
sudo systemctl restart myapp.service          # stop then start (full bounce)
sudo systemctl reload myapp.service           # send ExecReload= (no downtime, config-only)
sudo systemctl stop myapp.service             # stop, leave enabled
sudo systemctl disable --now myapp.service    # stop + remove boot autostart

# 4. INSPECT — is it up? did it fail? what did it print?
systemctl status myapp.service                # human summary: state, PID, cgroup, last log lines
systemctl is-active myapp.service             # -> active / inactive / failed  (scriptable)
systemctl is-enabled myapp.service            # -> enabled / disabled / static
systemctl is-failed myapp.service             # -> failed / active  (use in health checks)
journalctl -u myapp.service -f                # follow this unit's logs live
systemctl cat myapp.service                   # show the effective, merged unit file

systemctl status gives you the at-a-glance picture (representative output):

● myapp.service - My App Service
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; preset: disabled)
     Active: active (running) since Sat 2025-01-11 03:00:04 UTC; 2h 14min ago
   Main PID: 4821 (run.sh)
      Tasks: 3 (limit: 4915)
     Memory: 18.4M (max: 1.0G)
     CGroup: /system.slice/myapp.service
             ├─4821 /bin/bash /opt/myapp/bin/run.sh
             └─4839 sleep 10

The two scriptable commands — is-active and is-failed — are what you wire into monitoring and other scripts, because they exit 0/non-zero instead of printing prose. is-active myapp && echo up is a clean health probe; parsing status output is not.

Type=: How systemd Knows When You’re “Ready”

Type= is the most-confused field in unit files. Pick wrong and your dependencies start before your service is actually ready, or systemd thinks your service crashed when it didn’t.

Type	Meaning	Use for
`simple` (default)	Service is “active” the moment ExecStart’s first process is forked	Daemons that don’t fork; wrong for scripts that exit
`exec`	Like `simple`, but “active” only once ExecStart’s binary is successfully `execve()`'d	Same as `simple`, but you want a missing/unexecutable `ExecStart` to count as a start failure
`forking`	Service is “active” once the parent process exits (the child becomes the daemon)	Old-style daemons that double-fork (Apache 2.2, dhcpd)
`oneshot`	Run ExecStart, wait for it to exit, mark service as completed (or failed)	Setup scripts, one-shot jobs, anything that’s not long-running
`notify`	Service must call `sd_notify(READY=1)` to signal it’s ready	Long-running daemons that have a non-trivial init phase
`notify-reload`	Same as notify, but supports reload via sd_notify	Services with reload handling
`dbus`	Service is ready when it claims a D-Bus name	D-Bus services
`idle`	Like simple, but delay execution until other jobs finish	Avoid log-spam at boot

Type=simple is wrong for shell scripts that exit

# WRONG: a script that runs to completion, with Type=simple.
[Service]
Type=simple
ExecStart=/opt/myapp/bin/sync-data.sh

This unit “succeeds” the moment fork returns, even if the script exits 200ms later. systemctl is-active returns “active” briefly, then “inactive” — confusing for monitoring. Worse, dependents that say After=myapp.service will start concurrently with sync-data.sh, not after.

Type=exec: catch a broken ExecStart at start time

Type=exec (systemd 240+) is the strict cousin of simple. Both treat a long-running process as “started,” but they differ on when:

simple reports success the instant systemd forks — before it has even tried to run your binary. If ExecStart=/opt/myapp/bin/run.sh is misspelled, non-executable, or has a broken shebang (#!/usr/bin/env pythn), simple still reports the start as successful, and you only discover the failure later.
exec reports success only after the child has successfully execve()'d the target. A missing binary, a bad interpreter, a User= that doesn’t exist, or a sandbox directive that denies the exec now surfaces immediately as a failed start — systemctl start returns non-zero and status shows the real error.

For a script wrapped as a long-running service, prefer Type=exec over Type=simple when you’re not using notify: same runtime behaviour, but honest failures at start. (It does not wait for your init to finish — that’s still notify’s job.)

Use Type=oneshot for “run once and exit”

[Service]
Type=oneshot
ExecStart=/opt/myapp/bin/sync-data.sh
RemainAfterExit=yes   # report active even after exit (so dependents see "completed")

oneshot waits for the script to exit. RemainAfterExit=yes keeps is-active returning “active” so dependents like backup-completion services can chain off of it.

Use Type=notify for daemons with a real init phase

[Service]
Type=notify
NotifyAccess=main           # only main process may signal
ExecStart=/opt/myapp/bin/run-daemon.sh
WatchdogSec=30

Inside the script:

#!/usr/bin/env bash
set -Eeuo pipefail

# ... initialization that takes a while ...
load_config
warm_caches
open_database

# Tell systemd we're ready.
systemd-notify --ready --status="Listening on :8080"

# Main loop.
while :; do
  # ... do work ...
  systemd-notify WATCHDOG=1   # reset the watchdog timer
  sleep 10
done

systemd-notify is the shell-friendly wrapper around sd_notify(3). --ready tells systemd “I’m done initializing.” After that, dependents start.

WatchdogSec=30 says: if the service doesn’t send WATCHDOG=1 within 30 seconds, systemd assumes it’s hung and restarts it. This is the canonical way to detect a daemon that’s running but stuck.

Shell gotcha, previewed: with NotifyAccess=main (the default) and a shell ExecStart, that systemd-notify --ready line is often silently ignored, because systemd-notify runs as a child of your script, not as the main PID. The service then times out and gets killed even though you “told it” you were ready. The fix is NotifyAccess=all — spelled out in Going deeper. Keep it in mind; it bites nearly everyone once.

Restart Policy: Don’t Death-Loop

Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

Restart=:

Value	Restart on
`no`	Never (default for `oneshot`)
`on-success`	Clean exit (rare; useful for retry-loops)
`on-failure`	Non-zero exit, signal kill, watchdog timeout
`on-abnormal`	Signal kill or watchdog only (not non-zero)
`on-watchdog`	Watchdog only
`on-abort`	SIGABRT only
`always`	Every exit, success or fail

RestartSec=5: wait 5 seconds before restarting. Without this, a script that crashes immediately respawns at 100% CPU.

StartLimitIntervalSec=60 + StartLimitBurst=5: if there are 5 starts within 60 seconds, refuse further restarts. This is the kill-switch that prevents infinite respawn loops.

Combined: a buggy script gets 5 retries, then systemd gives up, marks the unit failed, and stops trying. You see systemctl status say “start request repeated too quickly” — exactly the diagnosis you want, not a CPU-burning host.

Version note: StartLimitIntervalSec= and StartLimitBurst= belong in the [Unit] section on modern systemd (they were briefly [Service] keys years ago and are still accepted there with a warning). The Restart=/RestartSec= pair stays in [Service]. If your limits appear to have no effect, this misplacement is the usual cause — systemctl cat myapp shows where they actually landed.

Sandboxing: Defense in Depth From the Service File

Hardening directives let you restrict the script’s privileges from outside the script. Even if the script has bugs (or is compromised), the impact is contained.

[Service]
# ─── Identity ────────────────────────────────────────────────────────
User=myapp
Group=myapp
DynamicUser=no            # if yes: systemd creates a transient user; great for one-shots

# ─── Filesystem isolation ───────────────────────────────────────────
ProtectSystem=strict       # /usr, /boot, /efi read-only; everything else inaccessible
ReadWritePaths=/var/lib/myapp /var/log/myapp   # opt-in writable paths
ProtectHome=true           # /home, /root, /run/user invisible
PrivateTmp=true            # private /tmp, /var/tmp; cleared on stop
PrivateDevices=true        # only /dev/null, /dev/zero, /dev/random, etc.
ProtectKernelTunables=true # /proc/sys, /sys read-only
ProtectKernelModules=true  # cannot load modules
ProtectControlGroups=true  # /sys/fs/cgroup read-only
ProtectClock=true          # cannot change system time

# ─── Capabilities & privilege ───────────────────────────────────────
NoNewPrivileges=true       # PR_SET_NO_NEW_PRIVS; cannot gain privileges via setuid
CapabilityBoundingSet=     # drop ALL capabilities (empty = none)
AmbientCapabilities=       # no ambient caps either
RestrictSUIDSGID=true      # cannot create SUID/SGID files

# ─── Network ─────────────────────────────────────────────────────────
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6   # block AF_PACKET, AF_NETLINK
PrivateNetwork=false        # set true for no network at all (offline-only scripts)
IPAddressDeny=any           # deny all (then allow specific):
IPAddressAllow=10.0.0.0/8 127.0.0.0/8

# ─── System call filtering ──────────────────────────────────────────
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources @debug @mount @raw-io
SystemCallArchitectures=native    # block 32-bit ABI on 64-bit kernels

# ─── Resource limits ────────────────────────────────────────────────
MemoryMax=512M
CPUQuota=50%
TasksMax=128
LimitNOFILE=4096

Hardening rationale

ProtectSystem=strict + ReadWritePaths= is the cleanest way to enforce “the script can write here and nowhere else.” Anything else triggers EROFS.
PrivateTmp=true gives the service its own /tmp, eliminating an entire class of /tmp race conditions and information leaks.
NoNewPrivileges=true is the hard-block: even if the script execs a setuid binary, the new process inherits the no-priv flag.
CapabilityBoundingSet= (empty) drops every Linux capability. If the script doesn’t need to bind to a port < 1024 or open raw sockets, it has zero special capabilities.
SystemCallFilter uses systemd’s groups: @system-service is a well-known set of “things a service typically does.” ~ removes from the set.
MemoryMax=512M triggers OOM-kill for that one service when it exceeds — preventing a runaway script from consuming all host memory.

Test what hardening actually applies

# Show effective security settings on a running service.
systemd-analyze security myapp.service
# Outputs a score 0-10 (lower = more hardened) and a per-directive breakdown.

# Show what each setting expanded to.
systemctl show myapp.service | grep -E '^(Protect|Restrict|Cap|System|Memory)'

systemd-analyze security is the audit tool. Score under 3 is excellent; under 5 is acceptable; over 7 means you’re running with way too much privilege.

Logging: Just Use Journald

[Service]
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

journal is the default. SyslogIdentifier sets the tag in journalctl output (otherwise journald uses the executable name).

In your script: write to stdout/stderr. Don’t open /var/log/myapp.log yourself. journald captures everything, indexes by unit, retains structured fields.

journalctl -u myapp.service                    # all logs for this unit
journalctl -u myapp.service -f                 # follow
journalctl -u myapp.service --since '1h ago'   # last hour
journalctl -u myapp.service -p err             # errors and worse
journalctl -u myapp.service -o json            # structured output

For structured logs from shell, use the journald protocol:

# In your script:
log() {
  printf '<%s>%s: %s\n' "$1" "${SYSLOG_IDENTIFIER:-myapp}" "$2"
}
log 6 "service starting"     # priority 6 = info (RFC 5424 numeric)
log 3 "database unreachable" # priority 3 = error

systemd-journald reads the leading <N> and assigns the priority field. Combined with journalctl -p err, you can filter precisely.

EnvironmentFile: Secrets and Config Without Hard-Coding

[Service]
EnvironmentFile=-/etc/myapp/env
EnvironmentFile=/etc/myapp/local-env

The leading - means “okay if missing.” Files are loaded in order; later files override earlier.

The file:

# /etc/myapp/env
DATABASE_URL=postgres://app:hidden@db.internal/app
API_KEY=secret-value
LOG_LEVEL=info

Restrict access:

chown root:myapp /etc/myapp/env
chmod 0640 /etc/myapp/env

Better: LoadCredential (systemd 250+)

For modern systemd, LoadCredential= reads a secret into the service’s ${CREDENTIALS_DIRECTORY} and exposes it without leaking via env or /proc/$pid/environ:

[Service]
LoadCredential=db-password:/etc/myapp/db-password
ExecStart=/opt/myapp/bin/run.sh

In the script:

DB_PASSWORD=$(< "${CREDENTIALS_DIRECTORY}/db-password")

CREDENTIALS_DIRECTORY is a tmpfs mount only the service can see, never visible to other processes. Strictly preferable to env-var secrets if your systemd is new enough.

Timer Units: Replacing cron

A timer unit triggers a service unit on a schedule. Two files: the timer and its corresponding service.

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
ConditionACPower=true       # don't run on battery (laptops)

[Service]
Type=oneshot
User=backup
ExecStart=/opt/backup/bin/run.sh
Nice=19                     # lowest CPU priority
IOSchedulingClass=idle      # only run when no other I/O

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup nightly

[Timer]
OnCalendar=*-*-* 03:00:00     # every day at 3 AM
RandomizedDelaySec=900        # spread load: actual run is 03:00–03:15
Persistent=true               # if missed (host was off), run on next boot
Unit=backup.service           # what to start

[Install]
WantedBy=timers.target

Enable:

sudo systemctl enable --now backup.timer
systemctl list-timers --all
# NEXT                         LEFT       LAST  PASSED  UNIT          ACTIVATES
# Tue 2025-01-14 03:11:23 UTC  14h left   -     -       backup.timer  backup.service

OnCalendar syntax

OnCalendar=daily               # 00:00:00 every day
OnCalendar=hourly              # 00:00 every hour
OnCalendar=Mon..Fri 09:00      # weekdays at 9 AM
OnCalendar=*-*-01 04:00:00     # 1st of every month at 4 AM
OnCalendar=2025-12-31 23:59:00 # one specific time
OnCalendar=*:0/15              # every 15 minutes
OnCalendar=*-*-* 03:00:00      # every day at 3 AM

systemd-analyze calendar 'Mon..Fri 09:00' validates and shows the next firing.

Cron equivalents

cron line	systemd OnCalendar
`0 3 * * *`	`--* 03:00:00`
`/15 * * *`	`*:0/15`
`0 9 * * 1-5`	`Mon..Fri 09:00`
`0 0 1 * *`	`--01 00:00:00`
`@reboot`	`OnBootSec=2min` (Timer) or `After=multi-user.target`

Why timers beat cron

Logs go to journald, not /var/log/cron.
Failed runs surface via systemctl status backup.service with exit code and recent logs.
Persistent=true catches up on missed runs after downtime; cron silently skips.
Resource limits (MemoryMax, CPUQuota) apply per-run.
Hardening directives apply (cron jobs run with full user privileges).
Dependencies work: timer can require network-online.target, etc.

Timers and cron get a full head-to-head — including anacron for hosts that sleep — in Scheduling: cron, systemd timers & anacron. This lesson focuses on the .service half of the pair; that one focuses on the choice of scheduler.

A Hardened Production Template

# /etc/systemd/system/myapp.service

[Unit]
Description=My App
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=notify
NotifyAccess=main
WatchdogSec=30

User=myapp
Group=myapp
SupplementaryGroups=

EnvironmentFile=-/etc/myapp/env
LoadCredential=db-password:/etc/myapp/db-password
WorkingDirectory=/opt/myapp

ExecStartPre=/opt/myapp/bin/preflight.sh
ExecStart=/opt/myapp/bin/run.sh
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=5
TimeoutStartSec=120
TimeoutStopSec=30
KillSignal=SIGTERM
KillMode=mixed

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp
LogRateLimitIntervalSec=10
LogRateLimitBurst=200

# Filesystem
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp /var/log/myapp
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true
ProtectProc=invisible
ProcSubset=pid

# Privilege
NoNewPrivileges=true
CapabilityBoundingSet=
AmbientCapabilities=
RestrictSUIDSGID=true
RestrictRealtime=true
LockPersonality=true
MemoryDenyWriteExecute=true

# Network
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=true

# Syscalls
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources @debug @mount @raw-io @reboot @swap
SystemCallArchitectures=native
SystemCallErrorNumber=EPERM

# Resources
MemoryMax=1G
MemoryHigh=768M
CPUQuota=200%
TasksMax=256
LimitNOFILE=8192
LimitNPROC=128
LimitCORE=0

# Misc
UMask=0027

[Install]
WantedBy=multi-user.target

This template scores well on systemd-analyze security (typically 1–2 out of 10, “OK” range). Adjust by removing things your script actually needs (e.g., remove MemoryDenyWriteExecute=true for JIT languages).

Real-World Recipes

Recipe 1: One-shot setup script with idempotent guard

[Unit]
Description=One-time database initialization
ConditionPathExists=!/var/lib/myapp/initialized
After=postgresql.service
Requires=postgresql.service

[Service]
Type=oneshot
User=myapp
ExecStart=/opt/myapp/bin/init-db.sh
ExecStartPost=/usr/bin/touch /var/lib/myapp/initialized
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The ConditionPathExists=!/var/lib/myapp/initialized means: don’t run if the marker file exists. After the script succeeds, ExecStartPost creates the marker. On reboot, the unit is “skipped (precondition not met)” and journalctl logs that. Idempotent across reboots.

Recipe 2: Long-running daemon with watchdog

/opt/myapp/bin/run-daemon.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

# Initialization phase.
load_config
warm_caches
open_database

# Tell systemd we're ready.
systemd-notify --ready --status="Listening on :8080"

# Main loop with periodic watchdog ping.
while :; do
  if ! main_iteration; then
    systemd-notify --status="Iteration failed; exiting"
    exit 1
  fi
  systemd-notify WATCHDOG=1 --status="Iteration completed at $(date -u +%FT%TZ)"
  sleep 5
done

Unit sets WatchdogSec=30, Type=notify. If main_iteration hangs > 30 seconds, watchdog fires and systemd restarts the service.

Recipe 3: Timer-driven backup with offline persistence

/etc/systemd/system/myapp-backup.service:

[Unit]
Description=MyApp backup

[Service]
Type=oneshot
User=backup
ExecStart=/opt/myapp/bin/backup.sh
Nice=19
IOSchedulingClass=idle
TimeoutStartSec=2h
StandardOutput=journal
SyslogIdentifier=myapp-backup
ProtectSystem=strict
ReadWritePaths=/var/lib/backup /var/lib/myapp
PrivateTmp=true
NoNewPrivileges=true

/etc/systemd/system/myapp-backup.timer:

[Unit]
Description=Run myapp backup daily

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=15min
Persistent=true
Unit=myapp-backup.service

[Install]
WantedBy=timers.target

Persistent=true means: if the host was off at 03:00, run the backup as soon as the host is up. This is what cron’s @reboot should be — guaranteed catch-up.

Recipe 4: Service with reload that re-reads config

[Service]
Type=notify
ExecStart=/opt/myapp/bin/run.sh
ExecReload=/bin/kill -HUP $MAINPID

In the script:

reload_config() {
  echo "received SIGHUP; reloading config"
  load_config
  systemd-notify --reloading
  warm_caches
  systemd-notify --ready --status="Reloaded at $(date -u +%FT%TZ)"
}
trap reload_config HUP

# main loop

systemctl reload myapp sends SIGHUP, the script re-reads config without exit. systemctl reload is preferred over systemctl restart when the change is config-only; no downtime.

Recipe 5: Per-instance template units

You have 5 worker queues, identical config except for the queue name. Use a template unit:

/etc/systemd/system/myapp-worker@.service:

[Unit]
Description=MyApp worker for queue %i

[Service]
Type=notify
User=myapp
Environment=QUEUE_NAME=%i
ExecStart=/opt/myapp/bin/worker.sh
Restart=on-failure

Enable with the instance name after @:

sudo systemctl enable --now myapp-worker@orders.service
sudo systemctl enable --now myapp-worker@billing.service
sudo systemctl enable --now myapp-worker@notifications.service

%i in the unit file is replaced with the part after @. Five instances, one unit file, individual control: systemctl restart myapp-worker@orders.

Going deeper

The sections above get a service running correctly. This section is the layer underneath — the process model systemd actually manages, the shell-specific sd_notify trap, how to override a unit without forking it, and what to do when the host isn’t Linux at all. This is the difference between a unit that works and one you understand.

The process tree systemd tracks: cgroups, MAINPID and KillMode

systemd does not track your service by a single PID and a PID file the way SysV init did. It puts every process the unit spawns into a cgroup (/system.slice/myapp.service) and tracks the whole tree. That has two big consequences for shell scripts, which love to fork children (sleep, subshells, pipelines, backgrounded jobs).

$MAINPID is the ExecStart process — and for a shell wrapper, that’s the shell. If run.sh launches your real program without exec, the process tree is bash → yourprogram, and $MAINPID is bash. So ExecReload=/bin/kill -HUP $MAINPID sends SIGHUP to bash, not to your program, and KillSignal=SIGTERM on stop hits bash first. The fix is to hand off with exec when the wrapper’s only job is setup:

#!/usr/bin/env bash
set -Eeuo pipefail
export APP_CONFIG=/etc/myapp/config.yaml
ulimit -n 8192
# Replace the shell with the real program: now IT is $MAINPID and receives signals directly.
exec /usr/bin/myprogram --serve

KillMode= decides what happens to the children on stop:

`KillMode`	On stop, systemd…	Use when
`control-group` (default)	Sends the signal to every process in the cgroup	Almost always — nothing leaks
`mixed`	SIGTERM to `$MAINPID`, then SIGKILL to the whole group	Main process must handle SIGTERM itself, but you still want stragglers reaped
`process`	Signals only `$MAINPID`	Rare; children are expected to outlive or self-manage — leaks if not
`none`	Signals nothing (just runs `ExecStop=`)	Almost never — deprecated footgun

For a shell service that traps SIGTERM to clean up, KillMode=mixed with KillSignal=SIGTERM is the usual choice: your trap fires on the main process, then systemd guarantees no orphaned sleep or child is left behind. TimeoutStopSec= bounds how long systemd waits after SIGTERM before escalating to SIGKILL.

sd_notify from a shell without the NotifyAccess trap

Type=notify works by handing the service an AF_UNIX datagram socket in the environment variable $NOTIFY_SOCKET. To signal readiness, something writes the line READY=1\n to that socket. systemd-notify is just a tiny program that does exactly that.

Here’s the trap. When your ExecStart=/opt/myapp/bin/run.sh runs systemd-notify --ready, that spawns a separate, short-lived process — a child of your script. With the default NotifyAccess=main, systemd only accepts notifications whose sender PID equals $MAINPID. The child’s PID isn’t the main PID, so systemd silently drops the message, never marks the service ready, and eventually kills it for TimeoutStartSec — the maddening “I clearly called --ready and it still times out” bug.

Three ways out, cleanest first:

# Option A — the reliable shell fix: accept notifications from any process in the unit's cgroup.
[Service]
Type=notify
NotifyAccess=all          # not "main"; systemd-notify is a child, so "main" rejects it
ExecStart=/opt/myapp/bin/run.sh

# Option B — no systemd-notify at all: write to the socket yourself with socat.
# $NOTIFY_SOCKET is a path like /run/systemd/notify (or an abstract "@..." socket).
notify() { [ -n "${NOTIFY_SOCKET:-}" ] && printf '%s' "$1" | socat - UNIX-SENDTO:"$NOTIFY_SOCKET"; }
notify "READY=1"
notify "WATCHDOG=1"

# Option C — systemd 253+: allow the Exec* processes (and their children) to notify.
[Service]
Type=notify
NotifyAccess=exec         # broader than main, narrower than all

NotifyAccess=all is the pragmatic answer for shell services (socat isn’t always installed, and abstract sockets — names starting with @ — need abstract-sendto: rather than unix-sendto:). The watchdog uses the same channel: WATCHDOG=1 on the same socket resets the timer, so whatever you chose for --ready you reuse for the periodic ping.

Drop-in overrides: don’t fork the vendor’s unit

When a package ships /lib/systemd/system/nginx.service (or /usr/lib/...), do not edit that file — the next package upgrade overwrites it and your change vanishes. Instead, layer a drop-in:

sudo systemctl edit nginx.service
# opens an editor; systemd creates /etc/systemd/system/nginx.service.d/override.conf

You write only the deltas:

[Service]
MemoryMax=2G
Restart=on-failure

systemd merges the drop-in over the vendor unit and runs daemon-reload for you. Two nuances that trip people up:

List-valued directives accumulate. ExecStart=, Environment=, ReadWritePaths= etc. append by default. To replace ExecStart=, you must first clear it with an empty assignment, then set the new value:
```
[Service]
ExecStart=
ExecStart=/opt/myapp/bin/run.sh --new-flag
```
See the merged result with systemctl cat nginx.service (shows base + every drop-in in order). Use systemctl edit --full nginx.service to copy the whole vendor unit into /etc/ and edit it wholesale, or systemctl revert nginx.service to throw away all your overrides.

Ad-hoc units and per-user services

You don’t need a file on disk to get systemd’s sandboxing. systemd-run creates a transient unit on the fly — handy for wrapping a one-off command in the same limits and isolation:

# Run a risky import once, memory-capped, private /tmp, as a throwaway dynamic user:
systemd-run --unit=oneoff-import --property=MemoryMax=512M \
  --property=PrivateTmp=yes --property=DynamicUser=yes \
  /opt/myapp/bin/import.sh
journalctl -u oneoff-import -f

And you don’t need root. User services live in ~/.config/systemd/user/ and are driven with systemctl --user:

systemctl --user daemon-reload
systemctl --user enable --now myjob.timer

The catch: a user manager normally exists only while you’re logged in, so your service dies at logout. To keep a user service running across logout and reboots, enable lingering:

sudo loginctl enable-linger "$USER"   # user manager starts at boot, survives logout

(Inside a user unit, $XDG_RUNTIME_DIR — usually /run/user/UID — is your writable scratch space, and there is no journald priority filtering by unit across users the way there is for system units.)

Not on Linux? launchd (macOS), SMF and rc

systemd is Linux-only. None of the above exists on macOS, the BSDs, or older Unix — so a “wrap my script as a service” task is genuinely a different tool per platform.

On macOS, the native service manager is launchd, configured with property-list (.plist) files:

User agents go in ~/Library/LaunchAgents/; system daemons in /Library/LaunchDaemons/ (root-owned).
You load and control them with launchctl (launchctl load|unload, or the modern launchctl bootstrap|bootout gui/$UID ~/Library/LaunchAgents/com.me.job.plist).
Logs don’t go to journald — you set StandardOutPath / StandardErrorPath to real files (or read the unified log with log stream --predicate '...').

A rough directive map:

systemd (Linux)	launchd (macOS) `.plist` key
`ExecStart=`	`ProgramArguments` (array)
`Restart=on-failure` / `always`	`KeepAlive` (bool or condition dict)
start at boot/login	`RunAtLoad`
`OnCalendar=` (timer)	`StartCalendarInterval` (dict of minute/hour/…)
`OnUnitActiveSec=`	`StartInterval` (seconds)
`StandardOutput=journal`	`StandardOutPath` (a file path)
`EnvironmentFile=`	`EnvironmentVariables` (dict)
`WatchdogSec=`	(no direct equivalent; `KeepAlive` restarts on exit only)

Other Unixes: illumos/Solaris uses SMF (svcadm, svcs, XML service manifests) with a comparable dependency and restart model; the BSDs and pre-systemd Linux use rc.d/init scripts — plain shell in /etc/rc.d/ or /etc/init.d/, with no supervision, sandboxing or readiness protocol at all (which is precisely the gap systemd was built to close). The portable lesson: the concepts here — “what counts as started,” restart policy, least privilege, readiness, scheduled runs — transfer everywhere; only the file format and CLI change.

Common beginner mistakes

These are misconceptions, not typos — the wrong mental model that produces subtly broken services. Each is paired with the model to replace it.

“The service is active, so my script is healthy.” active only means systemd’s Type= condition was met — for simple, merely that a fork happened. It says nothing about whether your loop is doing useful work or wedged on a dead socket. Right model: active is a lifecycle state, not a health state. Use Type=notify + a real --ready for “started,” and a WatchdogSec= heartbeat for “still alive.”
“I’ll log to /var/log/myapp.log like I always have.” Under a service, stdout/stderr already flow to journald; a second log file means double storage, two places to grep, and log rotation you now own. Right model: print to stdout, let journald be the log, and query with journalctl -u. Files are for exports, not primary logs.
“Type=simple vs oneshot is a detail.” It changes the meaning of started, failed, and whether After= dependents wait for you. A run-and-exit script under simple flaps active→inactive and lets dependents race it. Right model: if the script exits, it’s oneshot; if it stays up, it’s simple/exec/notify. Match Type= to the script’s actual lifecycle.
“Hardening is a production nicety I’ll add later.” The default is User=root with the entire system reachable — the least safe possible posture — and “later” rarely arrives. Right model: start from the hardened template and remove what your script proves it needs, driven by systemd-analyze security. Subtractive hardening is safe; additive hardening never gets done.
“Restart=always makes my service bulletproof.” Without RestartSec= and StartLimitBurst=, a script that dies on startup respawns thousands of times a second and pins a core; worse, endless restarts hide the crash instead of surfacing it. Right model: on-failure + back-off + a burst limit, then let the unit reach failed loudly so a human (or an alert) sees it.
“I edited the unit, so it’ll use the new settings.” systemd reads unit files into memory at load time; your on-disk edit is invisible until daemon-reload. Right model: every hand-edit is followed by systemctl daemon-reload, then restart. (systemctl edit does the reload for you — one reason to prefer drop-ins.)
“Type=notify is broken — I call --ready and it still times out.” Almost always the NotifyAccess=main PID trap: systemd-notify runs as a child, so its notification is rejected. Right model: from a shell service, use NotifyAccess=all (or write to $NOTIFY_SOCKET from the main process). The protocol is fine; the sender identity was wrong.

Footgun List

Type=simple for a script that exits. Use oneshot, not simple. Service will appear to “succeed” then immediately become inactive.
Restart=always without RestartSec. A crash-loop pegs the CPU. Always set RestartSec= and StartLimitBurst=.
User=root because you didn’t think about it. Default is root. Always set User= to a service account; for one-shots, consider DynamicUser=true.
After=network.target instead of network-online.target. network.target only guarantees the network stack is initialized, not that the network is up. For network-dependent services, use network-online.target and Wants=network-online.target.
EnvironmentFile= with permissive mode. chmod 0640 with group read for the service account; never world-readable.
Logging to a file you also tee to journald. Pick one. Logs in two places means a 2x storage bill and grep confusion.
ExecReload=systemctl reload-or-try-restart (recursive) — don’t do this. ExecReload is the implementation of reload, usually kill -HUP $MAINPID.
ProtectSystem=strict with no ReadWritePaths=. Service has nowhere to write. Add ReadWritePaths=/var/lib/myapp etc. for the legitimate writable paths.
WatchdogSec= without Type=notify. WatchdogSec only fires for Type=notify services that send WATCHDOG=1. Other types ignore it.
ConditionPathExists= confused with RequiresMountsFor=. ConditionPathExists is checked once at start; if false, the unit is skipped. RequiresMountsFor= ensures a path’s mount is up. Different semantics.
Editing the unit and forgetting daemon-reload. systemd caches unit files; without daemon-reload, your changes don’t apply.
Forgetting [Install] means systemctl enable does nothing. The WantedBy= is what creates the symlink that triggers auto-start.

Practice challenges

Work these in order — each builds on the last. Try before opening the solution; the one-line why is the part worth remembering.

1. (Beginner) Wrap a run-once script as a chainable one-shot. Write the minimal /etc/systemd/system/seed.service for /opt/tools/seed.sh (a script that seeds a database and exits). It must run once, and stay reported as “active” afterwards so a later unit can After=seed.service. Then activate it.