Ansible for Edge and IoT Fleet Management, In Depth — Pull-Mode, Signed Manifests, Constrained Devices and Intermittent Networks

In a nutshell

Imagine you run a thousand vending machines scattered across a whole country. You cannot walk to each one, and most of the time a given machine is switched off, on a patchy mobile signal, or behind a router you have no way to dial into. So you flip the direction of control. Instead of you reaching out to each machine, every machine wakes up on its own schedule, checks a noticeboard you control, applies whatever signed instructions it finds there, and goes back to sleep. If the instructions look tampered with, it ignores them. If a new instruction breaks the machine, it quietly reverts to the last version that worked.

That flip — from push (a control server opens a connection out to each device) to pull (each device connects in to fetch and apply its own configuration) — is the entire idea of edge and IoT fleet management. Ordinary datacentre Ansible pushes: it SSHes out to hosts that are always reachable. Edge Ansible pulls: ansible-pull runs on the device, from a cron job or systemd timer, clones a Git repository, and applies the playbook locally. Nobody ever SSHes in.

This lesson shows the three pull-mode patterns (a Git-based ansible-pull agent, image-based OS updates with bootc/ostree, and Kubernetes-at-the-edge with a fleet operator), how to sign everything so a device only ever runs code you trust, how to make updates roll back automatically when they fail, and how to run all of it on hardware with 1 GB of RAM that is online four minutes a day. It is the same mechanism that quietly updates your phone, your car, and your smart meter.

Prerequisites: you should be comfortable with playbooks, roles, inventories, variables and Ansible Vault (Tiers 1–4), plus the air-gapped automation lesson — many edge fleets are also air-gapped — and the signing model from the compliance lesson.

After this lesson you will be able to:

Explain when to push and when to pull, and justify the choice for a given fleet size and connectivity profile.
Stand up an ansible-pull agent with a signed-commit gate, a jittered timer, and catch-up on missed runs.
Engineer automatic, atomic rollback (btrfs snapshot or bootc rollback) behind a deadman-switch watchdog.
Ship configuration as a GitOps repository and OS changes as signed bootc/ostree images, rolled out canary → wave → full.
Give every device a hardware-backed identity plus short-lived credentials, so a stolen device is locked out in minutes.
Close the loop with drift detection and device-initiated report-back, without ever scraping a device.

Level: Advanced · Time: ~45 min

Datacentre Ansible assumes warm prerequisites: every host reachable on a stable network, fast SSH, predictable hardware, plentiful CPU and disk. Edge Ansible assumes the opposite. Devices are scattered across thousands of locations, behind NAT and shared 4G modems, with 1GB of RAM and 8GB of flash, online for minutes a day. The “fleet” is 5,000 retail kiosks, 25,000 wind-turbine controllers, 100,000 vehicle telematics units, or 1.5 million water-meter gateways. Push-mode Ansible — open SSH from a control plane to every host on a schedule — is the wrong shape.

This lesson is the specialist guide to inverting the model: pull-based agents, signed manifests, fleet operators built on Kubernetes-at-the-edge (k3s, MicroK8s, KubeEdge), and the realistic operational patterns for constrained, intermittently-connected hardware. We will use ansible-pull, the Red Hat Device Edge / Image Mode for RHEL stack, and the ostree/bootc image-based update model where appropriate.

We will be opinionated about scale: at a hundred edge hosts, AAP push-mode still works. At a thousand, mesh execution nodes start to creak. At ten thousand and beyond, you must move to pull-mode and image-based updates; if you do not, you will brick devices in the field. The patterns here scale from one to one million.

Position in the curriculum. Tier 1–4 fluency required, plus the Tier 5 air-gapped lesson — many edge fleets are also air-gapped (industrial OT, aviation, defence). The compliance lesson informs the signing model.

What “edge” means and why it changes the rules

“Edge” covers a wide range; the relevant attributes for Ansible are:

Numbers: 1k–1M devices, sometimes more.
Location: thousands of physical sites — retail stores, wind farms, hospitals, vehicles.
Connectivity: intermittent 4G/5G/satellite/WiFi, often behind carrier NAT. No inbound reachability.
Hardware: small ARM64/x86 boards (Raspberry Pi, NXP iMX, NVIDIA Jetson). 1–8GB RAM. 8–256GB flash. No swap.
Power: often constrained. Reboots cause field outages.
Update window: minutes per day, sometimes per week. Updates must be atomic and rollback-able.
Identity: device certificates, not user/password.
Lifecycle: 5–10 years in field with occasional swap-outs.

Together these break every assumption datacentre Ansible makes:

Datacentre assumption	Edge reality
Control plane connects out to host (push)	Host must connect to control plane (pull)
SSH always reachable	Device may be online for 4 minutes per day
Failed task = retry	Failed task on a 4G link = wait until tomorrow
`dnf install foo` or `apt install foo`	No network during the install window
Rollback = revert config	Rollback = revert the entire OS image (atomic)
Privilege escalation via sudo	Device root-of-trust signed firmware; no sudo at all
Inventory in CMDB updated by operators	Inventory is a serial number on a sticker, scaling weekly

The mental shift is from “playbooks executed on hosts” to “hosts that pull and apply signed manifests.”

Push versus pull — the inversion, and when each still wins

Everything else in this lesson follows from one decision, so make it consciously. In push mode (default Ansible, ansible-playbook) a control node opens an SSH (or WinRM) connection outward to each managed host and streams tasks to it. In pull mode (ansible-pull) the logic runs on the device: the device reaches inward to a Git server or registry, fetches its desired state, and applies it to itself with connection: local.

At the edge the direction of that arrow is not a style choice — it is dictated by the network. A device behind carrier-grade NAT has no reachable inbound address; the control node cannot open a connection to it even if it wanted to. The device can only ever be the initiator. That single fact eliminates push for most edge fleets before any other consideration enters the picture.

Dimension	Push (`ansible-playbook`)	Pull (`ansible-pull` / bootc / fleet agent)
Who initiates the connection	Control node → host	Host → control plane
Needs inbound reachability to the device	Yes (SSH/WinRM open)	No — the device dials out
Works behind carrier NAT / intermittent link	No	Yes
Control-plane load	Scales with concurrent forks	Offloaded onto each device
Latency to apply a change	Seconds (you run it now)	One poll interval (minutes–hours)
Natural blast radius	Whatever you target in one run	One device per run, self-limited
Central real-time inventory	Yes — the controller knows all	Eventual — devices report back
Rollback	Re-run a fixed playbook	Must be engineered on-device (snapshot/image)
Sweet spot	Datacentre, ≤ few hundred always-on hosts	Thousands–millions of intermittent devices

Push has genuine advantages you surrender when you invert: you can apply a fix right now, and the controller always knows the true state of every host. Pull trades that immediacy for reach and scale. The crossover is roughly: push is fine up to a few hundred always-on hosts on a network you control; past a thousand devices, or the moment devices sit behind NAT or go offline for long stretches, pull wins decisively. Automation Platform (AAP) push with mesh execution nodes stretches this to perhaps a thousand well-connected sites, but it does not cross the NAT/offline barrier — mesh nodes still require the controller to reach them.

A compact way to hold it in your head: push asks “are you there? do this now”; pull says “whenever you’re ready, here are your standing orders — signed, so you know they’re mine.”

Three operational patterns at the edge

There are three patterns at scale; pick deliberately.

1. ansible-pull — the simplest pull-mode. Each device runs ansible-pull from cron / systemd timer, fetches a Git repo, runs the playbook locally. Works up to about 5k devices with discipline.

2. Image-based (ostree/bootc) — every change ships as a new bootable OS image. The device reboots into the new image (atomic) or rolls back (atomic) on failure. This is the Red Hat Device Edge and Fedora bootc / RHEL Image Mode model. Scales to millions; what big telcos and automakers actually use.

3. Kubernetes-at-the-edge — k3s/MicroK8s on each device, with a fleet operator (Rancher Fleet, Argo CD edge mode, EdgeX Foundry) that pulls and reconciles workloads. Most appropriate when the workload is itself container-based (ML models, data pipelines, video analytics).

In real fleets, you usually combine: image-based for the OS and platform layer (Pattern 2), Kubernetes for the application layer (Pattern 3), and ansible-pull for one-shot operational tasks (Pattern 1) when needed.

Pattern 1: `ansible-pull` for small to medium fleets

ansible-pull is the bare-minimum pull-mode runner. It clones a Git repo, runs a playbook locally with --connection=local, and exits. Configure on each device:

# /etc/systemd/system/ansible-pull.service
[Unit]
Description=ansible-pull
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/ansible-pull \
  -U https://gitea.kv.local/edge/edge-fleet.git \
  -i localhost, \
  -C main \
  -d /var/lib/ansible-pull/repo \
  --vault-password-file /etc/ansible/vault.pwd \
  --verify-commit \
  edge-pull.yml

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/ansible-pull.timer
[Unit]
Description=ansible-pull every 30 minutes (with jitter)

[Timer]
OnBootSec=2min
OnUnitActiveSec=30min
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target

Three details that matter:

--verify-commit: requires the Git tip to be GPG-signed. Without it, anyone with push access to the repo can run code on every device. Do not skip this.
RandomizedDelaySec=10min: jitter prevents 5,000 devices from all hitting the Git server at the same minute. Without it your repo server crashes.
Persistent=true: missed timers run at next boot. Without this, a device offline for 6 days never catches up.

The repo is a normal Ansible project with one entry-point playbook (edge-pull.yml) using --connection=local. Tasks gated by host facts (when: ansible_facts.hostname.startswith('kiosk-')) let you carve out groups without separate inventories.

For small fleets up to ~5,000 devices, this is enough. The Git server (Gitea, GitLab, or the AAP-bundled Hub for collections) is the only “infrastructure.” Add a webhook from the Git server to a metrics endpoint to track which devices have pulled which commit; that is your fleet status dashboard.

Signed manifests and rollback for `ansible-pull`

ansible-pull itself doesn’t roll back; if the playbook breaks, the device is broken. Engineer rollback explicitly:

# edge-pull.yml — checkpointed apply
---
- hosts: localhost
  connection: local
  tasks:

    - name: Read currently applied commit hash
      ansible.builtin.slurp:
        src: /var/lib/ansible-pull/applied_commit
      register: applied
      ignore_errors: true

    - name: Compute pending commit hash
      ansible.builtin.command:
        cmd: git -C /var/lib/ansible-pull/repo rev-parse HEAD
      register: pending
      changed_when: false

    - name: Snapshot before applying (btrfs)
      ansible.builtin.command:
        cmd: btrfs subvolume snapshot / /.snapshots/pre-{{ pending.stdout[:7] }}
      when: applied.content | default('') | b64decode | trim != pending.stdout
      ignore_errors: true   # not all devices have btrfs

    - name: Run the actual configuration role
      ansible.builtin.import_role:
        name: kiosk_configure

    - name: Smoke test
      ansible.builtin.import_role:
        name: kiosk_smoke

    - name: Persist applied commit (only if smoke test passed)
      ansible.builtin.copy:
        dest: /var/lib/ansible-pull/applied_commit
        content: "{{ pending.stdout }}\n"

    - name: Trim old snapshots to keep only last 3
      ansible.builtin.shell: |
        ls -1t /.snapshots | tail -n +4 | xargs -r -I {} btrfs subvolume delete /.snapshots/{}
      ignore_errors: true

Rollback is then a separate playbook triggered by a watchdog: if the device cannot reach the Git server for 24 hours and the smoke test is failing, the watchdog rolls back to the previous snapshot. This is the “deadman switch” pattern; it has saved more fleets than any other single mechanism.

The GitOps config repository the device pulls

ansible-pull is only as good as the repository behind it. Treat that repo as the single source of truth for the fleet’s configuration layer (Pattern 1), the same way the image registry is the source of truth for the OS layer (Pattern 2). A workable layout:

edge-fleet/                      # the repo every device clones
├── edge-pull.yml                # entry playbook, connection: local
├── ansible.cfg                  # gather_subset minimal, fact cache on
├── group_vars/
│   ├── all.yml                  # fleet-wide defaults
│   ├── class_kiosk.yml          # per-device-class vars
│   └── class_meter.yml
├── host_vars/                   # rarely used at the edge; prefer classes
├── roles/
│   ├── kiosk_configure/         # idempotent config, NEVER installs packages
│   ├── kiosk_smoke/             # post-apply health checks
│   └── report_back/             # emit applied commit + drift summary
└── files/
    └── cosign.pub               # baked into the image too; pin trust

The device selects its own work by facts, not by a central inventory:

# edge-pull.yml — the device chooses its class locally
---
- name: Edge pull apply
  hosts: localhost
  connection: local
  gather_facts: true
  vars:
    device_class: "{{ ansible_facts.hostname | regex_replace('^([a-z]+)-.*$', '\\1') }}"
  tasks:
    - name: Load class variables
      ansible.builtin.include_vars: "group_vars/class_{{ device_class }}.yml"

    - name: Apply the class configuration role
      ansible.builtin.include_role:
        name: "{{ device_class }}_configure"

Two rules keep this sane at scale:

Branches are waves, not environments. main is what canary devices track; a fast-forward-only stable branch is what the bulk of the fleet tracks. Promotion is a signed fast-forward merge from main to stable, not a cherry-pick. Devices pin their branch via the systemd unit’s -C flag.
Tags are releases. Every promotion is an annotated, GPG-signed tag (edge-2026.07.1). --verify-commit checks the signature on whatever the branch points at; signing the tag as well gives you an auditable release history to map incidents to.

Because the repo is the desired state, a device that has been offline for a week simply converges to the current tip on its next successful pull — there is no “replay each missed change in order.” That property (converge-to-desired-state, not replay-a-log) is exactly what makes GitOps the right model for flaky links.

Pattern 2: image-based updates with ostree / bootc

For fleets above ~5k devices, the right primitive is the bootable OS image. Instead of mutating the running OS (running dnf install on a Pi in the field), you publish a new immutable image, the device boots into it, and rolls back on failure. This is what Tesla, Volkswagen, Boeing, and every modern automotive/aerospace stack does at scale.

The Red Hat way is RHEL Image Mode (bootc), which uses OSTree as the on-device store and OCI container images as the build/distribution format. bootc is a small native runtime that knows how to switch between two deployments (current and pending) atomically.

# Containerfile — your edge OS image
FROM registry.redhat.io/rhel9/rhel-bootc:9.4

RUN dnf install -y \
      podman \
      systemd-container \
      ansible-core \
      python3-cryptography \
      kiosk-app && \
    dnf clean all

# Bake the kiosk app config into the image
COPY etc/kiosk/ /etc/kiosk/

# Enable services
RUN systemctl enable kiosk-app.service

# Bake the ansible pull config too
COPY etc/systemd/system/ansible-pull.service /etc/systemd/system/
COPY etc/systemd/system/ansible-pull.timer /etc/systemd/system/
RUN systemctl enable ansible-pull.timer

Build, sign, push:

podman build -t registry.kv.local/kiosk-os:1.4.2 -f Containerfile .
cosign sign --key /etc/cosign-edge.key registry.kv.local/kiosk-os:1.4.2
podman push registry.kv.local/kiosk-os:1.4.2

On each device:

bootc switch registry.kv.local/kiosk-os:1.4.2
bootc upgrade --check     # show what would change
bootc upgrade --apply     # stages new deployment
systemctl reboot          # boots into new deployment

# if anything is broken:
bootc rollback            # boots into previous deployment, atomic

The state machine: each device has two deployments on disk (current and pending). On reboot, GRUB boots the pending. If it fails to boot 3 times (or fails health checks within N minutes), GRUB falls back to the previous one automatically. You cannot brick a device with bootc, because the rollback is hardware-enforced via the firmware/bootloader. This is the property that makes the model viable at million-device scale.

Ansible’s role here is driving the image build, not running on each device:

# build-edge-image.yml — runs on the build host
- hosts: build_host
  tasks:
    - name: Render Containerfile from template
      ansible.builtin.template:
        src: Containerfile.j2
        dest: /tmp/build/Containerfile

    - name: Build image
      containers.podman.podman_image:
        name: "registry.kv.local/kiosk-os"
        tag: "{{ release }}"
        path: /tmp/build
        push: true
        push_args:
          dest: "registry.kv.local/kiosk-os:{{ release }}"

    - name: Sign with cosign
      ansible.builtin.command:
        cmd: cosign sign --key /etc/cosign-edge.key registry.kv.local/kiosk-os:{{ release }}

    - name: Update fleet rollout config
      ansible.builtin.uri:
        url: https://fleet.kv.local/api/rollouts
        method: POST
        body_format: json
        body:
          target: kiosks
          image: "registry.kv.local/kiosk-os:{{ release }}"
          wave: canary
          percentage: 1

Then a fleet operator (Rancher Fleet, Argo CD edge, custom controller) watches the rollout config and pushes the device-side bootc switch commands.

Pattern 3: Kubernetes-at-the-edge with fleet operators

For workloads that are themselves container-based — ML inference, video analytics, data pipelines — running k3s on each device and using a fleet operator is often the cleanest pattern.

k3s is a 60MB Kubernetes distribution that runs comfortably on 1GB-RAM ARM64 boxes. Each device runs a single-node k3s cluster (or a 3-node mini-cluster across nearby devices); workloads are pods deployed via a fleet operator.

The Ansible role is to:

Install and configure k3s on each device (one-time, via the OS image).
Manage the fleet operator’s manifests (in Git, applied across the fleet).
Manage the device certificates / mTLS that authenticates the device to the operator.

# install k3s during image build (Pattern 2 + Pattern 3)
- name: Install k3s
  ansible.builtin.shell: |
    curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.5+k3s1 sh -s - \
      --token={{ vault_k3s_token }} \
      --server=https://fleet-control.kv.local:6443

The fleet manifest is GitOps:

# fleet/kiosk-app/fleet.yaml
namespace: kiosk
helm:
  chart: oci://registry.kv.local/charts/kiosk-app
  version: 1.4.2

targets:
  - name: canary
    clusterSelector:
      matchLabels:
        wave: canary
  - name: prod
    clusterSelector:
      matchLabels:
        wave: prod

Devices labelled wave: canary get the new chart first; once metrics confirm health, you bump wave: prod to the same version. The label change is tracked in Git; the rollback is git revert.

Ansible orchestrates the rollout (e.g., flipping labels in waves) but does not need to reach each device directly. The control plane connects out; the device pulls in.

Connectivity reality: NAT, MQTT, and one-way trust

Most edge devices live behind carrier NAT (4G/5G/cable), which means no inbound connectivity from the control plane. Three viable bidirectional channels:

HTTPS pull (every N minutes) — devices initiate. Used by ansible-pull, bootc, fleet-agent, Argo CD edge.
MQTT/AMQP — devices maintain a long-lived connection to a broker; the control plane publishes commands. Used by industrial IoT, EdgeX, AWS IoT Core, Azure IoT Hub.
WebSocket / gRPC stream — devices maintain a long-lived stream; bidirectional RPC. Used by Tailscale-style VPN agents, RancherD, KubeEdge.

Each has trade-offs. HTTPS pull is the simplest but high latency (your minimum response time is one poll interval). MQTT is real-time but adds a broker dependency. WebSocket gives RPC semantics but is heavyweight for very small devices.

Pick one and standardise. Mixing transports per device class makes the control plane brittle; uniform transport with class-specific topics/labels makes it manageable.

For the AAP-aware reader: AAP automation mesh nodes are not viable as edge agents. They are too heavy, too tightly coupled to the controller, and require always-on network. Edge devices need the patterns above; AAP can still be the operator-facing UI that triggers fleet rollouts via webhooks.

Identity and trust at the edge

Each device must be uniquely identifiable and authenticate cryptographically. The chain:

Hardware root of trust — TPM 2.0 module on the board, with a manufacturer-issued Endorsement Key (EK).
Device certificate — issued at provisioning by an internal CA, bound to the TPM’s EK or AIK. Stored in TPM-protected NVRAM.
Workload secrets — short-lived tokens issued by the control plane after device cert validation. Rotated automatically.

The Ansible workflow:

# device-provision.yml — runs on the build host or first-boot enrolment
- name: Generate device certificate via internal CA
  community.crypto.x509_certificate:
    path: "/var/lib/devices/{{ device_serial }}.crt"
    privatekey_path: "/var/lib/devices/{{ device_serial }}.key"
    csr_path: "/var/lib/devices/{{ device_serial }}.csr"
    provider: ownca
    ownca_path: /etc/pki/edge-ca/ca.crt
    ownca_privatekey_path: /etc/pki/edge-ca/ca.key
    ownca_privatekey_passphrase: "{{ vault_ca_passphrase }}"
    ownca_not_after: "+1095d"   # 3 years
  no_log: true

- name: Bake cert into device image at first boot
  ansible.builtin.copy:
    src: "/var/lib/devices/{{ device_serial }}.crt"
    dest: "/etc/pki/device/cert.pem"
  delegate_to: "{{ device_address }}"
  no_log: true

Workload secrets (S3 keys, MQTT credentials) are not stored on the device; the device authenticates with its certificate and the control plane returns short-lived credentials each session. This pattern is what makes a stolen-device scenario manageable: revoke the cert, rotate workload creds, the lost device is locked out within minutes.

Zero-touch provisioning and bootstrap

The hardest part of a fleet is not the thousandth update — it is the first boot of a device a technician has just bolted to a wall, with no keyboard, that must join the fleet with a unique identity and zero manual steps. This is zero-touch provisioning (ZTP). Two mechanisms, often combined.

1. First-boot enrolment unit (baked into the image). A one-shot systemd service ships inside the image, runs once, enrols the device, then disables itself:

# /etc/systemd/system/edge-enrol.service  (baked into the OS image)
[Unit]
Description=First-boot edge enrolment
After=network-online.target
Wants=network-online.target
ConditionPathExists=!/var/lib/edge/enrolled

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/edge-enrol
ExecStartPost=/usr/bin/touch /var/lib/edge/enrolled
ExecStartPost=/usr/bin/systemctl disable edge-enrol.service

[Install]
WantedBy=multi-user.target

The edge-enrol script reads the board serial, builds a CSR from a TPM-resident key, and exchanges a one-time bootstrap token (injected at manufacture) for a signed device certificate:

#!/bin/bash
# /usr/local/sbin/edge-enrol — first-boot only, then self-disables
set -euo pipefail
SERIAL=$(cat /sys/class/dmi/id/product_serial)

# Key is TPM-resident via the tpm2-openssl provider (persistent handle)
openssl req -new -provider tpm2 -key handle:0x81010001 \
  -subj "/CN=${SERIAL}" -out /var/lib/edge/dev.csr

# Exchange the single-use bootstrap token for a signed cert over mTLS
curl --fail --cert /etc/pki/bootstrap.pem \
     -H "X-Bootstrap-Token: $(cat /etc/pki/bootstrap.token)" \
     --data-binary @/var/lib/edge/dev.csr \
     https://enrol.kv.local/v1/enrol > /etc/pki/device/cert.pem

shred -u /etc/pki/bootstrap.token   # single-use; burn it after enrolment

Ansible’s job is to bake and manage this on the build host, never to run it in the field:

# provision-image.yml — build host renders enrolment into the image
- name: Stage first-boot enrolment
  hosts: build_host
  tasks:
    - name: Install enrolment unit and script
      ansible.builtin.copy:
        src: "{{ item.src }}"
        dest: "{{ item.dest }}"
        mode: "{{ item.mode }}"
      loop:
        - { src: edge-enrol.service, dest: /tmp/build/etc/systemd/system/edge-enrol.service, mode: "0644" }
        - { src: edge-enrol,         dest: /tmp/build/usr/local/sbin/edge-enrol,             mode: "0755" }

    - name: Seed a single-use bootstrap token per device
      ansible.builtin.copy:
        dest: "/tmp/build/etc/pki/bootstrap.token"
        content: "{{ lookup('community.general.random_string', length=40, special=false) }}"
        mode: "0400"
      no_log: true

2. FIDO Device Onboard (FDO). For hardware that supports it (Red Hat Device Edge ships fido-device-onboard), the standardised flow removes even the baked bootstrap token. At manufacture the device receives an ownership voucher; at first power-on it contacts a rendezvous server, which redirects it to your owner-onboarding server; that server runs ServiceInfo modules that inject the SSH key, the device certificate, and the ansible-pull URL. Ansible manages the owner-onboarding server’s ServiceInfo config, so onboarding a new device model becomes a config change rather than a site visit.

Either way the invariant is the same: a technician plugs in power and network, and nothing else. No golden password, no per-device manual step, no shared credential that leaks. If your provisioning requires a human to type anything unique per device, it will not survive ten thousand devices.

Config drift, reporting back, and observability

In push mode the controller knows the truth because it just ran. In pull mode the device knows the truth and the control plane knows only what devices report. Closing that loop has two halves: detecting drift on the device, and reporting a compact summary back.

Detecting drift with check mode. Ansible’s own idempotency is a drift sensor. Run the configuration role in check mode with diff and count what would change; a non-zero changed count between real applies means something (a person, a cosmic ray, a failing disk) altered the device out of band:

- name: Detect drift (report-only, never converge here)
  ansible.builtin.include_role:
    name: kiosk_configure
  check_mode: true
  diff: true
  register: drift

- name: Summarise drift
  ansible.builtin.set_fact:
    drift_count: "{{ drift.results | default([]) | selectattr('changed') | list | length }}"

Every task in kiosk_configure must have honest changed_when / check_mode behaviour for this to mean anything — a task that reports changed on every run (a bare command without changed_when: false, a shell that always “does something”) makes every device look permanently drifted and drowns the signal. Getting idempotency right is not cosmetic at the edge; it is what makes drift detection possible at all.

Reporting back. The device posts a small, signed payload on every check-in — not logs, not full facts, just enough to run a dashboard:

# roles/report_back/tasks/main.yml
- name: Build the report
  ansible.builtin.set_fact:
    report:
      serial: "{{ ansible_facts.product_serial | default('unknown') }}"
      class: "{{ device_class }}"
      applied_commit: "{{ pending.stdout | default('unknown') }}"
      os_release: "{{ ansible_facts.distribution_version | default('n/a') }}"
      drift: "{{ drift_count | default(0) | int }}"
      healthy: "{{ smoke_passed | default(false) | bool }}"
      ts: "{{ ansible_date_time.iso8601 }}"

- name: Push to the fleet endpoint (best-effort; queue if offline)
  ansible.builtin.uri:
    url: https://fleet.kv.local/api/v1/checkin
    method: POST
    body_format: json
    body: "{{ report }}"
    client_cert: /etc/pki/device/cert.pem
    client_key: /etc/pki/device/key.pem
    timeout: 10
  register: pushed
  failed_when: false        # telemetry must never fail the apply

- name: Queue the report locally if the push failed
  ansible.builtin.copy:
    dest: "/var/lib/edge/queue/{{ ansible_date_time.epoch }}.json"
    content: "{{ report | to_nice_json }}"
  when: pushed.status is not defined or pushed.status != 200

Note failed_when: false on the push: telemetry must never be able to fail the apply. A device whose reporting endpoint is down should still configure itself correctly and simply queue its check-ins for later. The control plane aggregates these by version, class, wave and region — answering “how many devices on 1.4.2?”, “who hasn’t checked in for 24 h?”, “what’s the canary success rate?” — without ever scraping a device, which would defeat the pull model.

Constrained-device pragmatics

A 1GB-RAM device cannot run an Ansible execution environment full of Python deps. Strategies:

Strip the EE: build a minimal EE that contains only the collections actually used at the edge. Aim for under 100MB.
Cache aggressively: each ansible-pull reuses cached facts and the Git checkout from last run.
Avoid package at runtime: the OS comes via image; ansible-pull configures, never installs. This frees RAM and removes network dependency.
Use gather_subset: !all,!facter,!ohai: limit fact gathering to essentials.
Use the local connection: never SSH to localhost; always connection: local in the entry playbook.
Watch memory: a single careless with_items over a 100k-line file blows the RAM budget. Use loop_control: { label }, prefer lineinfile over template-then-replace for small edits.

For genuinely tiny devices (microcontrollers under 256KB), Ansible is the wrong tool — you ship pre-built firmware over OTA frameworks like Mender, RAUC, or Zephyr’s MCUboot. Ansible lives one layer up: provisioning the gateway that talks to those firmware updaters.

Beyond those strategies, keep a running mental budget. A typical 1GB board that must also run the workload has perhaps 200–300MB of headroom for a management pass — spent by the Python interpreter and Ansible core (~80–120MB resident), fact gathering (which grows with gather_subset), and any module that slurps a file into memory. Three habits keep you inside it:

Cache facts between runs. Set a jsonfile fact cache in ansible.cfg so ansible-pull reuses last run’s facts instead of re-gathering the full set every 30 minutes. Combine with gather_subset: min and gather the expensive subsets only in the rare task that needs them.
Stream, do not slurp. Prefer ansible.builtin.lineinfile / ansible.builtin.blockinfile / ansible.builtin.replace for small edits over slurp-then-template. A slurp of a 50MB log to check one line will OOM-kill the run on a constrained board.
Bound every loop. loop_control: { label: "{{ item.name }}" } keeps output small, and iterating a summarised list (not a 100k-line raw file) keeps the interpreter’s memory flat.

Flash is the other scarce resource. An 8GB eMMC holding two bootc deployments (Pattern 2) plus /var has little room for accumulated logs, old snapshots, or a fat Git checkout. Cap journald (SystemMaxUse=200M), trim snapshots to the last three (shown earlier), and use a shallow Git clone (--depth=1 on the ansible-pull repo) so history does not accumulate on the device.

OTA security considerations

Any OTA update path is also an attack path. Defenses:

All artefacts signed: Containerfile-built images signed with cosign; ansible-pull commits signed with GPG; manifests signed by the fleet operator’s CA.
No public images allowed: device-side bootc config refuses unsigned or wrong-issuer images.
CRL/OCSP enforced: revoked certs get blocked even before the next reboot.
Two-person review on every fleet rollout (canary stage gated by survey).
Rate limiting: a device cannot pull the same image more than N times per hour (defends against forced-rollback attack).
Signed metrics: device-emitted metrics signed with device key, so you cannot fake “100k devices reporting healthy” from one compromised host.

Auditors love the signed-everything story. Auditors hate the “we trust whatever Git pushes” story. Make the right choice early.

Fleet operator: state machine and dashboards

The fleet operator (whether a Rancher Fleet, custom controller, or AAP workflow with EDA) tracks each device’s state machine:

[unenrolled] -- enrol --> [pending] -- ack --> [active]
                                                  |
                          rollout (canary) -------|
                                                  v
                          [pending-update] -- apply ok --> [active@new]
                                            -- apply fail --> [active@old]   (rollback)
                                                  |
                          retire ----------------|--> [retired]

Each transition emits a metric (Prometheus counter) and an event (Kafka, MQTT, or simple HTTP webhook). The dashboard answers:

How many devices on each version?
Which devices haven’t checked in in N hours?
What is the rollout success rate per wave?
Which devices are stuck in pending-update?

Wave management is critical at scale: 1% canary for 24 hours, 5% wave for 48 hours, 25% wave for 72 hours, then full rollout. Bake this into the fleet operator config; do not let release managers override it manually.

Going deeper

Everything above is enough to run a fleet. This section is for the engineer who has to defend the design in an architecture review, debug it at 3 a.m., or scale it past the point where hand-waving stops working.

bootc / ostree internals: why a rollback cannot brick

OSTree is a content-addressed, immutable object store — think “Git for the root filesystem.” A deployment is a hardlinked checkout of an ostree commit into /ostree/deploy/..., mounted read-only as /. Two deployments coexist on disk at once; /etc is a 3-way merge (your local edits preserved across upgrades) and /var is persistent and shared between them. bootc upgrade fetches an OCI image, derives an ostree commit from its layers, stages it as a new deployment, and rewrites the Boot Loader Specification entries so the new one is default with the old one as fallback.

The reason rollback is instant and cannot fail: the previous deployment’s files are still on disk, still hardlinked, never deleted until a later upgrade garbage-collects them. Rolling back is just repointing the bootloader — no download, no reinstall, no package resolution that could half-complete. That is the structural difference from dnf downgrade, which mutates a live system and can leave it in a state that is neither cleanly old nor cleanly new.

greenboot: health-gated boot, not just “did the kernel start”

The claim “you cannot brick a bootc device” rests on greenboot. After boot, greenboot runs every script in /etc/greenboot/check/required.d/. If any exits non-zero it runs /etc/greenboot/red.d/ remediation and increments a boot counter; after GREENBOOT_MAX_BOOT_ATTEMPTS (default 3) the bootloader automatically rolls back to the last green deployment.

#!/bin/bash
# /etc/greenboot/check/required.d/20-kiosk-app.sh
systemctl is-active --quiet kiosk-app.service || exit 1
curl -fsS --max-time 5 http://127.0.0.1:8080/healthz >/dev/null || exit 1

The subtlety: health, not mere liveness, gates success. A kernel that boots into a broken app is still a failure, and greenboot catches it. Write these checks to assert the thing the device exists to do (serve the kiosk, publish the meter reading), not merely that systemd reached multi-user.target.

OCI deltas and the bandwidth budget

An ostree/bootc update transfers only changed objects. ostree static deltas plus OCI layer dedup mean a 200MB application change on a 4GB image is roughly 50MB on the wire, not 4GB. The fleet math still bites, so plan it:

25,000 devices × 50MB = 1.25TB per full rollout.
At a 1% canary that is only ~12.5GB before you commit real money and risk.
Stagger with waves and RandomizedDelaySec; cap concurrent fetches per gateway; prefer off-peak cellular windows for non-urgent updates.

bootc’s fetcher is resumable: a dropped 4G link resumes from cached partial layers on the next attempt, and the atomic apply only happens after a full fetch and signature verification. This is what makes updates over a four-minute daily window feasible — progress accumulates across many partial connections.

Idempotency, check-mode and serial as edge safety rails

Because a failed task on a 4G link means “wait until tomorrow,” every task must be safely re-runnable: the next pull re-applies from the top and must converge, not compound. Concretely — changed_when: false on read-only commands, honest check_mode support in any custom module, and block/rescue/always so a partial failure cleans up rather than leaving a half-state. In the build-host orchestration layer (which is push), serial: / throttle: and max_fail_percentage bound the blast radius even before the fleet operator gets involved:

- name: Promote an image across build targets in bounded waves
  hosts: build_targets
  serial: "5%"
  max_fail_percentage: 0
  tasks:
    - name: Push wave and wait for canary health gate
      ansible.builtin.include_role:
        name: promote_wave

Sigstore: keyed versus keyless at the edge

cosign supports two signing models. Keyed (a private key you hold) works fully offline: the device trusts a pinned public key baked into the image and verifies against it with no network. Keyless (Fulcio-issued short-lived certs + a Rekor transparency log via OIDC) needs internet and a reachable log — which many edge and OT networks simply do not have. Prefer keyed for the device-trust path; keep keyless (if any) for the CI side where connectivity exists. Enforce it with a containers-policy / bootc policy that requires a matching signedBy identity and rejects everything else — an unsigned or wrong-issuer image must not even stage.

TPM measured boot and remote attestation

Storing the device cert in the TPM is table stakes. The next level is measured boot: the TPM measures the firmware, kernel and initramfs into PCRs, and the device presents a signed quote the control plane can verify before issuing workload credentials. That turns “the device says it is healthy” into “the device cryptographically proved what it booted.” Overkill for a coffee-shop kiosk; essential for payment terminals, medical devices and OT, where a cloned or tampered image is a safety and compliance event, not just an outage.

Event-Driven Ansible as the operator brain

Event-Driven Ansible (EDA) can subscribe to the fleet’s check-in stream (Kafka / MQTT / webhook) and react: pause a wave automatically when the canary error rate crosses a threshold, open a ticket when N devices miss check-in, or advance stable once the canary has been green for 24 hours. This is where AAP genuinely earns its place at the edge — not as a runtime on the device, but as the operator brain reacting to device-reported events, with EX374-style workflow, RBAC and approval gates around every promotion.

Anti-patterns that destroy edge fleets

Using ansible-playbook from a centralised control plane in push mode. SSH-to-50k-NAT’d-devices is a non-starter. Use pull-mode or image-based.
Mutating the running OS in the field. A failed dnf upgrade on a kiosk in a remote store ends with a truck roll. Use atomic image-based updates.
No randomised jitter on pull timers. All 5k devices hit Git at minute 0; Git falls over.
Unsigned manifests / unsigned images. Anyone with push access owns the fleet.
Persistent device credentials with broad scope. A stolen device == fleet compromise. Use short-lived tokens.
One-shot rollback via “ssh in and fix it”. When the device is offline 23 hours a day, you cannot SSH in. Engineer rollback to fire automatically without human intervention.
No TPM / no hardware root of trust. Your fleet identity scheme can be cloned by extracting the certificate from a flash dump.
Skipping the “deadman switch” rollback. The device must roll back when it cannot reach the control plane for too long, not require remote action.
Treating canary as optional. A bad image to 100% of fleet is a 24-hour incident; a bad image to 1% is a 1-hour incident.

Frequently asked questions

1. When is ansible-pull enough vs when do I need image-based updates? Up to ~5k stable devices with reliable network and a config-only change set: ansible-pull is enough. Beyond that, or any case where you need to update kernel / glibc / firmware atomically: move to bootc/ostree image-based.

2. Can I run AAP at the edge? No. AAP is a datacentre product — controller, hub, automation mesh assume always-on network and ample resources. Use AAP as the back-of-house operator UI for the fleet operator (trigger rollouts, run reports), not as a runtime on each device.

3. How do I pin Ansible content for edge? Build the EE once on the build host with all collection versions pinned and signed. Bake the EE into the device image (Pattern 2). Devices never resolve Galaxy, never pip install, never reach out for content at runtime.

4. What about bandwidth costs? A bootc upgrade typically transfers only the layers that changed (OCI delta). For a 200MB-app-change on a 4GB-image OS, the transfer is ~50MB. With waves and randomised jitter, fleet bandwidth at peak is manageable. Always cap concurrent rollouts per gateway, and prefer cellular off-peak windows for non-urgent updates.

5. How do I handle 4G outages during update? The bootc upgrade process is resumable: the OCI fetcher caches partial layers. If connectivity drops mid-fetch, the next attempt resumes. Atomic apply happens only after full fetch + verification.

6. What about OT / industrial devices that cannot run Ansible at all? You don’t run Ansible on them. You run Ansible on the gateway that aggregates their data (the “edge gateway”, which is a Linux box). The gateway does protocol translation (Modbus/OPC-UA → MQTT) and applies updates to the OT devices via vendor-specific tooling. Ansible owns the gateway; vendor tooling owns the deepest leaf.

7. Are there security implications to using cosign / sigstore at the edge? Yes — the device must trust the cosign public keys, and the OCSP/CRL infrastructure must be reachable for revocation checks. Bake the public key bundle into the image; for revocation, use short-lived certs (renew every 24h) and reject expired without OCSP lookup.

8. How do I handle a fleet of mixed device types and OSes? Tag the inventory by capability: arch=arm64, ram=2g, network=4g, os=rhel9, class=kiosk. Roles use when guards on tags. Image builds produce per-class images. Fleet operator targets devices by tag, not by serial number — a serial-number-by-serial-number rollout does not scale.

9. What’s the right way to monitor an edge fleet? Each device emits a small metrics blob to Prometheus pushgateway / OTel collector / MQTT topic on every check-in. The control plane aggregates by version, wave, region. Anomaly detection (a device that hasn’t checked in for 7 days) triggers the maintenance ticket flow. Avoid scraping every device; that defeats the pull-mode pattern.

10. What’s the single most underrated edge practice? The deadman watchdog rollback. Every edge device should have a small program that monitors “have I successfully completed health checks recently?” and triggers a bootc rollback (or git checkout previous) if the answer is no. Without this, a regression deployed during your 4-minute connectivity window can leave a device unrecoverable until physical service. With it, the worst case is a slightly-out-of-date device.

Hands-on lab — `ansible-pull` with signed commits

This lab simulates a small edge fleet with a single device pulling a signed playbook from a local Git server.

Prerequisites: Linux box (or Pi), git, gpg, ansible-core ≥ 2.16.

mkdir -p edge-lab/{repo,device}
cd edge-lab

# 1. Local git repo with a tiny playbook
git init repo
cat > repo/edge-pull.yml << 'EOF'
- hosts: localhost
  connection: local
  tasks:
    - debug:
        msg: "Edge pull at {{ ansible_date_time.iso8601 }}, commit {{ lookup('env','COMMIT') | default('unknown') }}"
    - copy:
        dest: /tmp/edge-status
        content: |
          last-pull: {{ ansible_date_time.iso8601 }}
          host: {{ ansible_facts.hostname }}
EOF
( cd repo && git add edge-pull.yml && git commit -S -m "v1: tiny edge play" )

# 2. Set up signing key
gpg --quick-gen-key 'edge-signer@kv.local' rsa4096 sign 1y
git -C repo config user.signingkey 'edge-signer@kv.local'

# 3. The device side: configure ansible-pull
cat > device/pull.sh << 'EOF'
#!/bin/bash
set -e
COMMIT=$(git -C /tmp/edge-repo rev-parse HEAD || echo unset)
exec ansible-pull \
  -U ${PWD}/repo \
  -i localhost, \
  -d /tmp/edge-repo \
  --verify-commit \
  edge-pull.yml
EOF
chmod +x device/pull.sh

# 4. Run it
device/pull.sh
cat /tmp/edge-status

# 5. Try to push an unsigned commit and see the verify-commit fail
( cd repo && git commit --allow-empty -m "unsigned" )
device/pull.sh   # should fail because tip is unsigned

The lab proves the signing chain end-to-end: the device pulls only commits signed by a trusted key. Extend it: add a systemd timer in device/, add a snapshot/rollback step in the playbook, add a watchdog that rolls back if /tmp/edge-status is older than N minutes.

Practice challenges

Work these in order — each builds on the lab above and escalates from beginner to advanced. Try before opening the solution.

1. (Beginner) Jitter and catch-up on a pull timer. Write a systemd .timer that runs ansible-pull every 30 minutes with up to 10 minutes of random delay, and catches up a missed run after the device has been offline.