Configure BorgBackup with Append-Only Repositories for Tamper-Resistant Server Backups

A mid-size SaaS company runs forty Linux application servers and a fleet of virtual appliances, all backing up nightly to a single dedicated backup host. During a tabletop exercise the security team asks the uncomfortable question: “If an attacker gets root on one of those forty servers, what stops them from running borg delete against the backup host and wiping every restore point — the exact move modern ransomware crews make before they encrypt?” With the default setup, the honest answer is “nothing.” A backup that a compromised client can erase is not a backup; it is a convenience. This guide fixes that. We will configure BorgBackup append-only repositories so that a client key can write new archives but can never prune, delete, or compact anything — and we will pin that restriction at the SSH layer with a forced command, so the guarantee holds even if the client is fully owned. The result is a deduplicated, encrypted, ransomware-resistant backup tier that survives the compromise of any server it protects.

The threat model is specific. We are not defending against disk failure (RAID and the 3-2-1 rule cover that) — we are defending against an authenticated, malicious, or compromised backup client that wants to destroy history. BorgBackup’s append-only mode plus SSH command= restrictions plus an out-of-band pruning identity is the standard, well-trodden answer.

Prerequisites

• A dedicated backup server running a recent Linux (this guide uses Ubuntu 24.04 LTS; RHEL 9 paths noted where they differ), reachable over SSH from the clients, ideally on a management VLAN.
• One or more Linux client servers / virtual appliances to back up. BorgBackup 1.2.x or newer on both ends (borg --version); versions must be compatible across the SSH link.
• Root or sudo on both ends for the initial setup. A separate, firewalled admin host that will hold the only key allowed to prune.
• HashiCorp Vault reachable for issuing and storing the repository passphrase (we will not paste secrets into shell history).
• Outbound SSH (tcp/22, or a custom port) from clients to the backup server only — never the reverse.

Target topology

The design has three identities touching the repository, with deliberately asymmetric power:

• Client servers (the many). Each application server and virtual appliance holds an SSH key whose only capability on the backup host is to run borg serve --append-only against its own repository. It can create archives forever. It cannot delete, prune, or compact. This is the identity an attacker would capture by rooting a server — and it is harmless to history.
• The append-only repository (the vault). Lives on the dedicated backup server. New data is only ever added; the segment files that hold old archives are never rewritten under an append-only session, so even a borg delete issued by a client is recorded but not honored at the storage layer until an authorized operator decides otherwise.
• The prune admin (the one). A separate identity on a firewalled admin host holds a different SSH key that runs full (non-append-only) borg serve. Retention pruning and borg compact happen here, out of band, on a schedule the attacker has no access to. This is the only place history can actually shrink.

Around that core sit the operational tools: HashiCorp Vault issues the per-repository encryption passphrase so it never lives in a script; CrowdStrike Falcon runs on the backup server and clients for runtime threat detection; Wiz scans the backup host’s cloud posture for public-exposure drift; Dynatrace (or Datadog) ingests the backup job’s exit status and timing so a silent failure pages someone; ServiceNow receives an auto-raised incident when a backup verification fails; Ansible templates the client configuration and GitHub Actions lints and ships those playbooks.

1. Install BorgBackup and create the dedicated backup user

On both the client and the backup server, install Borg from the distro or the official one-file binary (the binary avoids version skew, which is the most common cause of “Remote: Borg server is too old” errors).

# Ubuntu / Debian
sudo apt-get update && sudo apt-get install -y borgbackup

# RHEL / Rocky / Alma (EPEL)
sudo dnf install -y epel-release && sudo dnf install -y borgbackup

# Or the pinned standalone binary (recommended for matched versions)
sudo curl -L -o /usr/local/bin/borg \
  https://github.com/borgbackup/borg/releases/download/1.4.0/borg-linux-glibc236
sudo chown root:root /usr/local/bin/borg && sudo chmod 755 /usr/local/bin/borg
borg --version

On the backup server, create an unprivileged, dedicated account that owns the repositories. Backups must never land in a user’s home that other things share.

sudo useradd --create-home --home-dir /srv/borg --shell /bin/bash borgsrv
sudo install -d -o borgsrv -g borgsrv -m 0700 /srv/borg/.ssh
sudo install -d -o borgsrv -g borgsrv -m 0700 /srv/borg/repos

2. Generate per-client SSH keys

Generate a dedicated Ed25519 keypair on each client — do not reuse an admin or login key. The private key stays on the client; only the public key goes to the backup server.

# Run on each CLIENT (as root, since the backup cron runs as root)
sudo ssh-keygen -t ed25519 -N '' \
  -f /root/.ssh/borg_appendonly \
  -C "borg-appendonly-$(hostname -f)"

sudo cat /root/.ssh/borg_appendonly.pub

Capture that public key string for each client — we restrict it in the next step. In a real fleet, Ansible distributes these keys and renders the authorized_keys entries, keeping the forced command identical and auditable across all forty hosts rather than hand-edited.

3. Lock the client key with an SSH forced command (the core control)

This is the linchpin. On the backup server, every client public key is added to /srv/borg/.ssh/authorized_keys wrapped in restrictions that (a) force borg serve no matter what command the client requests, (b) pin append-only mode, © confine the key to a single repository path, and (d) strip every SSH feature an attacker could pivot through.

# On the BACKUP SERVER, edit /srv/borg/.ssh/authorized_keys
# One line per client. Replace the path and the key with real values.

command="borg serve --append-only --restrict-to-repository /srv/borg/repos/app01",restrict ssh-ed25519 AAAAC3Nz...app01-key borg-appendonly-app01.example.com
command="borg serve --append-only --restrict-to-repository /srv/borg/repos/app02",restrict ssh-ed25519 AAAAC3Nz...app02-key borg-appendonly-app02.example.com

What each piece buys you:

• command="borg serve ..." — a forced command. Whatever the client passes over SSH is ignored; the server always runs exactly this. The client cannot ask for an interactive shell or a different borg invocation.
• --append-only — the server-side enforcement of append-only mode. Critically, this is set on the server, not trusted from the client. A client running borg delete or borg prune will have the operation recorded in the transaction log but not committed to the segments, so old data remains recoverable.
• --restrict-to-repository — confines this key to one repo, so a compromised app01 cannot read or write app02’s backups.
• restrict — the modern OpenSSH shorthand that disables port forwarding, agent forwarding, X11, PTY allocation, and ~/.ssh/rc. (On older OpenSSH, spell it out: no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty.)

Fix permissions or sshd will silently ignore the file:

sudo chown borgsrv:borgsrv /srv/borg/.ssh/authorized_keys
sudo chmod 600 /srv/borg/.ssh/authorized_keys

4. Source the repository passphrase from Vault, then initialize the repo

Borg repositories are encrypted; the passphrase must never sit in a script or shell history. We pull it from HashiCorp Vault at runtime. First, store a generated passphrase per client (done once by an operator):

# Operator, once per client — generate and store, never echo to a terminal log
vault kv put secret/borg/app01 \
  passphrase="$(openssl rand -base64 48)"

On the client, export BORG_PASSPHRASE by reading Vault on the fly so it lives only in the process environment:

# On the CLIENT
export BORG_RSH="ssh -i /root/.ssh/borg_appendonly -o StrictHostKeyChecking=accept-new"
export BORG_REPO="ssh://borgsrv@backup.example.com/srv/borg/repos/app01"
export BORG_PASSPHRASE="$(vault kv get -field=passphrase secret/borg/app01)"

Now initialize the repository. This is a write/create operation, which append-only permits. Use authenticated, modern encryption:

borg init --encryption=repokey-blake2 "$BORG_REPO"

repokey-blake2 stores the (passphrase-wrapped) key inside the repo and uses BLAKE2b for fast authenticated integrity — appropriate when the passphrase is strong and Vault-managed. Immediately export and safely store the key material out of band, because losing it means losing every backup:

borg key export "$BORG_REPO" /root/borg-app01.keyfile
vault kv put secret/borg/app01-keyfile keyfile=@/root/borg-app01.keyfile
sudo shred -u /root/borg-app01.keyfile

5. Run the first backup and wire up the nightly job

Create your first archive. Borg deduplicates globally across all archives in the repo, so the first run is large and subsequent runs are tiny deltas.

borg create --stats --compression zstd,9 \
  "::app01-{hostname}-{now:%Y-%m-%dT%H:%M:%S}" \
  /etc /home /var/www /srv \
  --exclude '/var/www/*/cache' \
  --exclude-caches

Wrap the whole flow — Vault fetch, borg create, exit-code reporting — in a script and schedule it. Here it is as a systemd timer, which gives you clean logs in the journal and a real exit status for monitoring:

# /usr/local/sbin/borg-backup.sh  (mode 0700, root)
#!/usr/bin/env bash
set -euo pipefail
export BORG_RSH="ssh -i /root/.ssh/borg_appendonly -o StrictHostKeyChecking=yes"
export BORG_REPO="ssh://borgsrv@backup.example.com/srv/borg/repos/app01"
export BORG_PASSPHRASE="$(vault kv get -field=passphrase secret/borg/app01)"

borg create --stats --compression zstd,9 \
  "::app01-{hostname}-{now:%Y-%m-%dT%H:%M:%S}" \
  /etc /home /var/www /srv --exclude-caches
rc=$?

# Emit the result to Dynatrace/Datadog so a silent failure pages someone
curl -fsS -m 10 -X POST "https://metrics.example.com/api/v1/events" \
  -H "Content-Type: application/json" \
  -d "{\"host\":\"$(hostname -f)\",\"metric\":\"borg.backup.exit\",\"value\":$rc}" || true
exit $rc

# /etc/systemd/system/borg-backup.timer
[Unit]
Description=Nightly tamper-resistant Borg backup
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=1800
Persistent=true
[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now borg-backup.timer

Note what is not in this client script: any borg prune, borg delete, or borg compact. The client physically cannot run them (the forced command blocks it), and we do not even pretend to. Retention lives elsewhere — step 6.

6. Configure out-of-band pruning from a separate admin identity

Append-only repositories grow forever unless an authorized identity prunes them. That identity must not be a backup client, or you have reintroduced the very risk you removed. Set up a second key on a firewalled admin host with a non-append-only forced command:

# On the BACKUP SERVER, a SECOND authorized_keys line for the admin host.
# Note: NO --append-only here. This identity is allowed to prune & compact.
command="borg serve --restrict-to-repository /srv/borg/repos/app01",restrict ssh-ed25519 AAAAC3Nz...PRUNE-ADMIN-key borg-prune-admin

From the admin host, run retention and reclaim space. Pruning marks old archives for deletion; borg compact is what actually frees the segment space on the server:

# On the ADMIN HOST, scheduled separately (e.g. weekly)
export BORG_RSH="ssh -i /root/.ssh/borg_prune_admin"
export BORG_REPO="ssh://borgsrv@backup.example.com/srv/borg/repos/app01"
export BORG_PASSPHRASE="$(vault kv get -field=passphrase secret/borg/app01)"

borg prune --list --stats \
  --keep-daily=7 --keep-weekly=4 --keep-monthly=6 \
  --glob-archives 'app01-*'

borg compact "$BORG_REPO"

Because this admin key is the only identity that can shrink history, an attacker who owns app01 — or even the whole client fleet — still cannot reach back through time. They would have to separately compromise the firewalled admin host, breaking the single-point-of-failure that append-only mode is designed to eliminate.

Validation

Prove the controls actually hold; do not assume them. Run all four checks.

1. Confirm the forced command genuinely restricts the client. From a client, try to get a shell or run an arbitrary command over the backup key — it must be refused:

ssh -i /root/.ssh/borg_appendonly borgsrv@backup.example.com "id"
# Expected: the connection runs `borg serve` and exits; `id` never executes.
ssh -i /root/.ssh/borg_appendonly borgsrv@backup.example.com
# Expected: no interactive shell (restrict strips PTY).

2. Verify append-only blocks destruction. This is the headline test. From the client, attempt to delete an archive, then confirm from the admin side that it still exists:

# On the CLIENT (the "attacker" with the append-only key)
borg delete "::app01-$(hostname)-OLDEST-ARCHIVE"   # appears to succeed locally

# On the ADMIN HOST — the archive is STILL THERE until an operator compacts
borg list "$BORG_REPO" | grep OLDEST-ARCHIVE

In append-only mode the delete is written to the transaction log but the data is not committed; the archive remains listable and restorable. If you ever need to honor a legitimate client-side delete, an operator inspects the append-only log and explicitly rolls the repo forward — a deliberate, audited action, never an automatic one.

3. Test a real restore. A backup you have not restored is a hypothesis. Extract a known file into a scratch directory:

mkdir -p /tmp/borg-restore && cd /tmp/borg-restore
borg extract "::app01-$(hostname)-LATEST" etc/hostname
diff /etc/hostname etc/hostname && echo "RESTORE OK"

4. Check repository integrity. Run a server-side consistency check from the admin identity (clients cannot, by design):

borg check --verify-data "$BORG_REPO"

Pipe the exit codes of checks 3 and 4 into Dynatrace/Datadog; on failure, auto-raise a ServiceNow incident so a missed verification becomes a ticket, not a log line nobody reads.

Rollback / teardown

To cleanly retire a client or the whole setup without leaving orphaned access:

# 1. Disable the client schedule
sudo systemctl disable --now borg-backup.timer

# 2. On the BACKUP SERVER, revoke the client's access by removing its
#    authorized_keys line (this instantly cuts the forced-command path)
sudo sed -i '/borg-appendonly-app01.example.com/d' /srv/borg/.ssh/authorized_keys

# 3. Destroy the client's private key
sudo shred -u /root/.ssh/borg_appendonly /root/.ssh/borg_appendonly.pub

# 4. (Full teardown only) From the ADMIN identity, delete the repository.
#    Requires the non-append-only admin key — a client could never do this.
borg delete --force "$BORG_REPO"

# 5. Remove the dedicated server account and its data
sudo userdel -r borgsrv

Revoke the matching Vault secrets last, once you are certain no restore is pending: vault kv metadata delete secret/borg/app01.

Common pitfalls

• Setting --append-only on the client instead of the server. Append-only is only a real control when it is in the server’s forced command. A client-side flag is a suggestion an attacker simply omits.
• Forgetting borg compact. borg prune alone marks archives but does not free disk; the repo keeps growing and you eventually fill the volume. Compaction must run from the admin identity.
• authorized_keys permissions too open. If /srv/borg/.ssh/authorized_keys is group- or world-writable, sshd ignores it entirely and your restrictions silently vanish. Keep it 0600 and the .ssh dir 0700.
• Version skew across the SSH link. A newer client against an older server throws cryptic remote errors. Pin the same standalone binary on both ends.
• Losing the repository key. repokey stores the key in the repo, but if the repo is destroyed and you never exported the key, the data is unrecoverable. Always borg key export into Vault.
• Letting the prune key live on a backup client. This silently collapses the whole model — the destructive identity becomes capturable from a server you are trying to protect. Keep it on a separate, firewalled host.
• Concurrent locks. Borg takes an exclusive repo lock; overlapping create and prune runs fail. Schedule client backups and admin pruning in non-overlapping windows.

Security notes

Append-only mode is one layer; defense in depth completes it. Treat the backup server as a crown-jewel asset: minimal packages, SSH key-only auth, and a host firewall allowing inbound SSH from the client management VLAN only. Run CrowdStrike Falcon on both the backup server and clients so an attacker probing the backup host — or staging mass deletions on a client — trips a runtime detection that reaches the SOC. Point Wiz at the backup host’s cloud account to alert the instant its storage or a snapshot drifts to public exposure or an over-broad IAM policy widens access. Authenticate human operators to the admin host through your IdP — Okta or Microsoft Entra ID — with MFA and conditional access, so even the prune identity sits behind strong SSO rather than a lone key. For an air-gapped tier, periodically replicate the repository to offline or object-locked (WORM) storage with borg over a one-way link, giving you a copy no online identity can touch at all. Keep all of this — playbooks and authorized_keys templates — in version control behind GitHub Actions so changes are reviewed and the forced-command line is never quietly weakened.

Cost notes

BorgBackup is open source, so the spend is storage and a little compute. The two levers that matter:

• Deduplication + compression. Borg deduplicates globally and zstd compresses well, so forty similar Linux servers store a fraction of their raw footprint — typically a large first archive and very small nightly deltas. This is what makes a long retention window (--keep-monthly=6) affordable.
• Right-size the backup volume and tier old data. Provision the repository on cost-effective block storage, and for the air-gapped copy use cheap object storage with lifecycle rules. Because pruning + compact reclaim space predictably, you can forecast growth and avoid over-provisioning.

The honest tradeoff: append-only means you knowingly hold more history than the bare minimum (you cannot let clients trim it), and you run a second admin host for pruning. That extra storage and that one small host are the price of a backup tier that survives the compromise of everything it protects — cheap insurance against the ransomware scenario that started this guide.