Ansible in Air-Gapped Environments, In Depth — Sealed Networks, Internal Mirrors, Signed EEs and Cross-Boundary Workflows

In a nutshell

Picture a research station on a remote island with no bridge and no undersea cable to the mainland. Nobody there can “just order online”. Everything the island runs on — food, fuel, spare parts, medicine — has to be loaded onto a ferry, inspected at the dock on arrival, logged in a ledger, and stored in the island warehouse. If it is not in the warehouse, the islanders cannot use it. There is no same-day delivery, and there is definitely no unannounced boat.

An air-gapped Ansible environment is that island. The enclave is the island. The missing bridge is the deliberate absence of internet access. The ferry is your transport process (a jump host, a USB drive carried by hand, or a one-way hardware link). The dock inspection is signature-and-hash verification. The warehouse is your internal mirror — a private Automation Hub, a container registry, a package repo — and it is the only place your playbooks are allowed to fetch anything from. You never type ansible-galaxy collection install amazon.aws and reach the internet, because the internet is not there. Instead, someone on the mainland downloads that collection, signs it, ships it on the ferry, the dock verifies the signature, and only then does it enter the warehouse where playbooks can use it.

Why should a beginner care? Because this is where the most security-sensitive automation on earth runs: defence networks, power-grid controllers, hospital systems, bank cores. The skills are the same Ansible you already know — inventories, playbooks, roles, collections, Execution Environments — but the supply chain around them changes completely. Once you understand “nothing runs until it has been imported, signed, and pinned”, the rest of this lesson is just the mechanics of a well-run dock.

Level: Advanced · Time: ~45 min

Before you start, you should be comfortable with Ansible collections and requirements.yml, be able to describe what an Execution Environment is and roughly how ansible-builder produces one, and have met Ansible Vault at least once. If any of those feel shaky, revisit Building Collections & Execution Environments, the Automation Platform Architecture lesson, and Vault & Secrets first — air-gap work assumes all three.

After this lesson you can:

Explain the three boundary archetypes (soft air-gap, true air-gap, data diode) and pick the right runbook for each.
Redirect ansible-galaxy at an internal mirror and pin collections to exact, signed versions.
Describe the full lifecycle sync → sign → bundle → transport → verify → import → publish, and build a tiny working version of it.
Build and sign an Execution Environment offline and pin job templates to an immutable image digest.
Reason about what cannot cross the gap, and design a one-way “publishing” playbook that never opens an inbound hole.

Air-gapped automation is not Ansible-with-a-firewall. It is a different operating model: every artefact you depend on must be explicitly imported, scanned, signed, versioned and pinned before it can run. There is no ansible-galaxy collection install amazon.aws from a sealed defence network. There is no pip install from a SCADA segment. There is no ee-supported pulled from quay.io from a regulated banking core. Everything must be brought across a controlled boundary, and the boundary itself becomes the most security-sensitive thing you operate.

This lesson is the specialist guide to running Ansible properly in air-gapped, classified, and high-assurance environments: defence, intelligence, OT/SCADA, banking core, regulated medical, nuclear, and other contexts where the network has no internet egress by deliberate policy.

We will cover three boundary archetypes — firewalled (one-way TLS allowed inbound from a trusted relay), traditional air-gap (sneakernet only), and data-diode (one-way physical link, hardware-enforced) — and the runbook patterns each requires. We will also cover the AAP architecture for sealed networks and how to keep auditable provenance for every collection, EE and module that ever runs in your enclave.

Position in the curriculum. Tier 1–4 fluency is required, plus the Tier 5 compliance lesson (because air-gapped environments are usually also the most regulated). The DR lesson is also useful — air-gapped DR has its own constraints we will cover briefly.

What “air-gapped” really means

The phrase is overused. There are three sharply different network models, and each has different Ansible implications:

Soft air-gap (allow-listed egress). No general internet, but a small allow-list of vendor endpoints (Red Hat CDN, Microsoft Update, vendor APT mirrors) is permitted via an outbound proxy. Ansible mostly works as normal; you point ansible-galaxy at an internal mirror and proxy your collection installs.
Traditional air-gap (no inbound, no outbound). The enclave has zero network connectivity to anything outside. Content arrives via physical media (USB sticks, CDs, removable hard drives) following a chain-of-custody process. Ansible content, EEs, OS packages, and Python wheels must all be pre-staged inside.
One-way (data-diode) air-gap. A hardware diode allows packets to flow in one direction only — typically out of the high-side enclave for monitoring, in to the low-side from a publishing relay. Ansible workflows must be split: control plane and execution plane on the high side; only data products flow out. Updates flow in only via signed bundles.

Most enterprises that say “air-gapped” actually mean #1 with policy that aspires to #2. Real defence and OT environments are #2 or #3. The patterns scale: if you can operate cleanly in a #2/#3 environment, you can trivially operate in #1.

A core principle: everything that enters the enclave must be signed, versioned, scanned, and immutable. A collection version is pinned. An EE image is signed by your build pipeline and verified by AAP before execution. An RHEL repository is mirrored on a known-good schedule with a known-good signature. There is no “latest” anywhere; “latest” is a vector for tampering.

The reference air-gap architecture

This lesson assumes a representative enclave:

Enclave network with no internet connectivity, AAP installed inside, AD or FreeIPA inside, OS hosts inside.
Sync host in a DMZ-like network that does have curated outbound access. It pulls collections, container images, RPMs and Python wheels from upstream, validates signatures, and stages them.
Transfer process: signed tar bundles move from sync host to enclave via either a one-time manual import (sneakernet), a scheduled jump-host pull, or a data diode write.
Inside the enclave: a private Galaxy-compatible mirror (Automation Hub or Pulp), a private container registry (Harbor or Quay), a private RPM mirror (Pulp/Foreman/Satellite), and a private PyPI mirror (pypi-server or bandersnatch snapshot). All of these are the only sources Ansible inside the enclave will trust.
AAP control plane inside, with all execution environments built and stored in the internal registry.

This architecture has four key invariants:

No host inside the enclave ever phones out. Even DNS lookups for galaxy.ansible.com should NXDOMAIN to fail closed.
Every artefact has a known provenance: signed by a key the enclave trusts, hash recorded on import.
The boundary is monotonic: things only enter, never leave (except via the diode publishing path).
The boundary is auditable: every import is logged, signed, and reviewable for at least the audit horizon.

Mirroring Galaxy and Automation Hub

Inside the enclave, ansible-galaxy must be redirected to the internal mirror. There are three viable backends:

Red Hat Automation Hub (AAP-bundled): official, supported, integrates with Controller’s content signing and EE image pipeline.
Pulp 3 with the pulp_ansible plugin: free, open-source, the upstream of Automation Hub.
A simple HTTP server hosting .tar.gz collection artefacts: the bare minimum, works for very small estates but lacks signing/scanning.

For anything beyond a lab, use Hub or Pulp. The sync host pulls upstream content and signs it before publishing into the internal mirror.

# sync-host playbook: mirror collections from Galaxy
---
- hosts: sync_host
  tasks:
    - name: Download collections needed inside the enclave
      ansible.builtin.command:
        cmd: >
          ansible-galaxy collection download
          {{ item }}
          -p /var/sync/collections
      loop:
        - amazon.aws:8.0.1
        - community.general:9.4.0
        - ansible.posix:1.5.4
        - community.windows:2.3.0
        - community.postgresql:3.7.0

    - name: GPG sign each tarball
      ansible.builtin.command:
        cmd: gpg --detach-sign --armor --local-user {{ signer_uid }}
              /var/sync/collections/{{ item }}
      loop: "{{ collection_tarballs.stdout_lines }}"

    - name: Compute SHA256 manifest for the bundle
      ansible.builtin.shell: |
        cd /var/sync/collections && sha256sum *.tar.gz *.asc > MANIFEST.sha256
      register: manifest

    - name: Bundle into a single tar for transport
      community.general.archive:
        path: /var/sync/collections
        dest: /var/sync/bundles/collections-{{ ansible_date_time.epoch }}.tar
        format: tar

The bundle then travels to the enclave (sneakernet, jump-host pull, or diode write). On the enclave side:

# inside-enclave playbook: import to Hub
---
- hosts: hub_host
  tasks:
    - name: Verify bundle signature before import
      ansible.builtin.command:
        cmd: gpg --verify {{ bundle_path }}.asc {{ bundle_path }}

    - name: Verify SHA256 manifest
      ansible.builtin.shell: |
        cd /tmp/extracted && sha256sum -c MANIFEST.sha256

    - name: Upload each collection to Automation Hub
      ansible.builtin.command:
        cmd: >
          ansible-galaxy collection publish
          /tmp/extracted/{{ item }}
          -s https://hub.enclave.local/api/galaxy/content/published/
          --token {{ hub_token }}
      loop: "{{ verified_collections }}"
      no_log: true

The chain is: upstream → sync host (signed) → boundary transfer → enclave (verified) → Hub (signed again with internal key) → end-user playbooks. Two signatures with different keys: the upstream signer (Red Hat for certified content, your sync engineer for community content) and the internal Hub signer (your enclave’s CA). End-user playbooks pin to specific versions; AAP enforces signature verification before execution.

# requirements.yml inside enclave
---
collections:
  - name: amazon.aws
    version: "8.0.1"   # exact, not >=
    source: https://hub.enclave.local/api/galaxy/content/published/
    signatures:
      - https://hub.enclave.local/api/galaxy/v3/artifacts/collections/amazon.aws-8.0.1.tar.gz.asc

Building signed Execution Environments offline

Execution Environments (EEs) are container images. Inside an air-gap, they cannot pull base layers from quay.io/ansible at build time, and they cannot fetch Python wheels from pypi.org. You must build them on the sync host (which has curated egress), sign the resulting image, transport it, and import into the enclave registry.

# execution-environment.yml on sync host
version: 3
images:
  base_image:
    name: registry.redhat.io/ansible-automation-platform-2.5/ee-supported-rhel9:latest
dependencies:
  galaxy: requirements.yml          # internal mirror requirements after enclave entry
  python: requirements.txt           # pinned wheels with hashes
  system: bindep.txt
options:
  package_manager_path: /usr/bin/microdnf
  skip_pip_install: false
additional_build_files:
  - src: ./offline-pip-cache         # pre-downloaded wheels with --download
    dest: configs
additional_build_steps:
  prepend_galaxy:
    - ADD _build/configs/offline-pip-cache /opt/wheels
  append_galaxy:
    - RUN pip install --no-index --find-links=/opt/wheels -r requirements.txt

Build steps on the sync host:

# pull and stash all upstream wheels with hashes
pip download -r requirements.txt --dest offline-pip-cache --require-hashes

# build the EE
ansible-builder build -t ee-airgap-base:1.0.0 --container-runtime podman

# sign with cosign (sigstore) using the enclave-trusted key
cosign sign --key cosign-airgap.key registry.local/ee-airgap-base:1.0.0

# export and bundle for transport
podman save -o /var/sync/bundles/ee-airgap-base-1.0.0.tar registry.local/ee-airgap-base:1.0.0

Inside the enclave, the import is the inverse:

# verify cosign signature
cosign verify --key cosign-airgap.pub registry.local/ee-airgap-base:1.0.0

# load image into local registry
podman load -i ee-airgap-base-1.0.0.tar
podman tag registry.local/ee-airgap-base:1.0.0 harbor.enclave.local/ee/airgap-base:1.0.0
podman push harbor.enclave.local/ee/airgap-base:1.0.0

In AAP Controller, configure the registry credential with the enclave Harbor URL, and pin every job template to a specific image tag. Never allow :latest; the entire point of the air-gap is to make every change explicit and reviewed.

RPM, APT, and Python wheel mirroring

A fleet inside an air-gap does not just need Ansible — it needs the OS packages, kernel updates, and Python interpreters that those plays install. A complete sealed environment mirrors:

RHEL repos via Red Hat Satellite or Pulp 3 with the pulp_rpm plugin. Sync the BaseOS, AppStream, and any extras you use; sign the Pulp publication with your enclave key.
Ubuntu/Debian APT mirrors via apt-mirror or debmirror, with Release.gpg re-signed by your enclave key.
PyPI via bandersnatch (full mirror, ~300GB) or devpi (selective mirror) — pinned to specific package versions.
Container registries via Harbor with replication from the sync host’s Quay/Pulp.
Helm charts via Pulp pulp_helm plugin or ChartMuseum.
Generic artefacts (binaries, ISOs) via Pulp pulp_file.

The sync schedule is its own runbook: weekly full sync of OS repos, daily delta sync of selected pinned content, monthly review of what is being mirrored versus what is needed. Auditors love the “monthly review” cadence; it provides paper trail evidence that the enclave is not slowly accumulating untracked content.

# sync-rpm.yml on sync host (typical pattern)
---
- name: Sync RHEL 9 BaseOS to internal mirror
  ansible.builtin.command:
    cmd: >
      pulp rpm repository sync --name rhel9-baseos
  delegate_to: pulp.dmz.local

- name: Publish and distribute the new version
  ansible.builtin.command:
    cmd: >
      pulp rpm publication create --repository rhel9-baseos --version latest
  delegate_to: pulp.dmz.local

- name: Sign the Pulp metadata with enclave key
  ansible.builtin.command:
    cmd: >
      pulp rpm distribution update
      --name rhel9-baseos
      --metadata-signing-service enclave-rpm-signer
  delegate_to: pulp.dmz.local

Inside the enclave, every host’s dnf.repos.d/ (or apt sources.list.d/) points at the internal Pulp distribution and trusts only the enclave key. There is no upstream URL anywhere on any host.

AAP topology inside the enclave

The Ansible Automation Platform fits cleanly into the air-gapped pattern:

Controller cluster (3 nodes): lives entirely inside the enclave; PostgreSQL also inside.
Automation Hub (1–2 nodes): serves the mirrored collections and EEs to controller and to operators.
Execution nodes: in-enclave only; no mesh links cross the boundary by default.
Hop nodes: only used if the enclave has multiple zones internally.
Event-Driven Ansible (if enabled): consumes events from in-enclave sources only (Splunk/QRadar instances inside the boundary, not external SIEM).

The interesting question is the transfer mechanism for new content. Three patterns:

A. Inbound jump host (most enterprises)

A single jump host has read-only access out to a curated set of vendor mirrors via a tightly controlled forward proxy. It also has write access in to the enclave’s Hub/Pulp/Harbor. Ansible runs on the jump host to pull-and-publish.

[upstream Galaxy]    [upstream registry.redhat.io]
        \                    /
         \                  /
        [Curated outbound proxy]
                  |
              [Jump host] — runs sync playbooks, signs artefacts
                  |
        [Enclave Hub / Harbor / Pulp]
                  |
        [Controller + execution nodes]
                  |
            [target hosts]

B. Sneakernet (true air-gap)

The sync host has no in-enclave connectivity at all. Bundles are written to media (USB, removable HDD), carried physically through a SCIF or controlled-area door, scanned by an Egress Inspection station, and imported on the enclave-side import host. This is slower (often weekly cadence) and requires written procedures and chain-of-custody logs. It is what classified networks actually do.

C. Data diode (one-way physical)

A hardware diode allows traffic in one direction only. The classic OT pattern: low-side (general corporate network) writes to the diode, high-side (the OT/SCADA enclave) reads. Ansible bundles are pushed by the low-side sync host and consumed by the high-side import host, with a UDP-based file-transfer protocol that the diode supports.

The reverse path — high-side outbound for telemetry — uses a separate diode in the opposite direction, often with a syslog/Splunk forwarder or a Prometheus remote-write target on the low side.

Each pattern has different Ansible implications, but the playbook structure is identical: sync → sign → bundle → transport → verify → import → publish. Only the transport mechanism changes.

Cross-boundary playbooks: the “publishing” pattern

Some enclaves need to publish automation results out — for example, an OT enclave that must report telemetry to a corporate IT SIEM, or a defence enclave that publishes summary reports to a less-sensitive system.

The Ansible pattern is the publishing playbook: a job that runs entirely inside the enclave, generates a sanitised artefact (a JSON report, a metric file, a scrubbed log bundle), signs it, and writes it to the diode-out side. The other side picks it up, verifies the signature, and consumes it.

# publishing-playbook.yml — runs inside enclave, writes to diode
---
- hosts: localhost
  tasks:
    - name: Generate sanitised metric bundle
      ansible.builtin.template:
        src: metrics.j2
        dest: /var/spool/diode-out/metrics-{{ ansible_date_time.epoch }}.json

    - name: Strip any classified fields
      ansible.builtin.command:
        cmd: jq 'del(.classified, .pii, .secrets)' /var/spool/diode-out/metrics-*.json

    - name: Sign the bundle
      ansible.builtin.command:
        cmd: >
          gpg --detach-sign --armor --local-user enclave-publisher
          /var/spool/diode-out/metrics-{{ ansible_date_time.epoch }}.json

    - name: Hand off to diode transmitter
      ansible.builtin.command:
        cmd: diode-tx --queue /var/spool/diode-out

Key principles:

The classification of the output is always at most equal to the lowest classification of the receiving system. Plays that generate outbound bundles must include a stripping/sanitisation step, and the strip rules must be reviewed.
No host on the other side of the diode runs Ansible against the enclave. Inbound automation comes only via signed bundles.
Output bundles are always signed and timestamped. The receiving system verifies before consuming.

Air-gapped collections and module portability

Not every collection works inside an air-gap. The ones that need internet access (e.g., a community module that fetches a script from GitHub at runtime) will fail in surprising ways. Audit your collection list before importing:

Cloud collections (amazon.aws, azure.azcollection, google.cloud) are useful inside an air-gap only if you have internal cloud equivalents (AWS GovCloud peered to your enclave, Azure Stack Hub, on-prem object storage with S3 API). The collection itself is portable; the endpoints it talks to may or may not be.
community.general modules vary widely. Some are pure-Python and work anywhere; others shell out to curl against external URLs. Read each module before relying on it.
ansible.netcommon for network device automation works fine inside an air-gap, since the devices are on the same isolated network.
community.windows / chocolatey.chocolatey: Chocolatey by default fetches from community.chocolatey.org. You must configure an internal Chocolatey repo (Sonatype Nexus, ProGet, or self-hosted) and disable the default source.
kubernetes.core: works fine; just ensure your private container registry is the only source for images and that pod manifests reference internal registry FQDNs.

A useful sanity check before publishing a new collection version into the enclave Hub:

ansible-galaxy collection install ./community-general-9.4.0.tar.gz \
  --server https://hub.enclave.local/api/galaxy/content/published/ \
  --offline    # FAILS if any role/action calls out

The --offline flag is a poor man’s network sandbox. For real assurance, run the test playbooks for the collection inside a netns with no egress.

Air-gapped leapp upgrades and OS lifecycle

Tier 5 already covered leapp; in an air-gap, the leapp content (target package set, repository pointers, leapp-data) must be present locally. The pattern:

On the sync host, download leapp packages and leapp-data for the source/target combination (e.g., RHEL 8.10 → 9.4) using leapp answer --section satellite_to_repository_mapping --add and the official Red Hat tarballs.
Sign the bundle, transport, verify, import to the enclave Pulp repo.
Configure /etc/leapp/transaction/to_install and /etc/leapp/files/repomap.json to reference enclave-local repository IDs.
Run leapp on enclave hosts as normal.

Without the offline leapp data, leapp upgrade will fail with cryptic errors about missing repositories. Pre-stage it; never improvise.

Compliance overlap: the air-gap as evidence multiplier

A side-effect of running properly in an air-gap is that you are doing most of the security-controls work that your auditors expect anyway:

CM-7 (NIST 800-53) — Least Functionality: nothing on a host that you did not explicitly provision; an air-gap forces this.
SI-7 — Software, Firmware, Information Integrity: signed artefacts everywhere.
SR-3, SR-4 — Supply Chain Risk Management: a curated, signed sync chain is exactly what these controls demand.
AC-4 — Information Flow Enforcement: the diode and inspection station are physical implementations of the control.

Ansible plays a key role here: roles/dr_evidence from the DR lesson and roles/compliance_evidence from the compliance lesson both land their bundles in immutable buckets, and an air-gapped equivalent (immutable WORM share, NetApp SnapLock volume, on-prem object store with retention) makes them auditable for the lifetime of the system.

Anti-patterns that destroy air-gap discipline

Allowing one curated outbound URL to creep into many. Each new exception is a vector; review monthly.
Letting Hub or Harbor sync directly from upstream. That is not an air-gap — that is a poorly-firewalled enclave.
Running pip install from a playbook against a public PyPI URL. Inevitable failure, sometimes silent.
Not pinning collection or EE versions. “Latest” makes the import process meaningless.
Skipping signature verification on bundle import. If you ever do this, you do not have an air-gap; you have a friendly fence.
Ignoring NXDOMAIN as a security control. Internal DNS should fail-close on *.galaxy.ansible.com; this catches the playbook that someone forgot to fix.
Operators with personal SSH keys on the bastion. Keys must be issued and rotated per the enclave’s identity model, not whatever the engineer brought from outside.
Uncontrolled become: true. In a sealed environment, privilege escalation is doubly important to log and gate.

Going deeper

Everything above is the what. This section is the how it actually behaves — the internals, edge cases, and gotchas that separate a demo from a network that passes an accreditation review.

How `ansible-galaxy` verifies signatures (the config that enforces the gap)

Redirecting ansible-galaxy at your Hub is only half the job; you also have to make signature verification mandatory and fail-closed. Two config blocks do this. First, the server list in ansible.cfg so nothing ever resolves to public Galaxy:

[galaxy]
server_list = enclave_hub

[galaxy_server.enclave_hub]
url = https://hub.enclave.local/api/galaxy/content/published/
token = <redacted-or-vaulted>

Second, the signature-verification keys (ansible-core 2.13+). Point at a GPG keyring containing only your enclave signing key, and require at least one valid signature:

[galaxy]
gpg_keyring = /etc/ansible/enclave-galaxy.gpg
required_valid_signature_count = 1
# comma list of GPG status codes you tolerate; keep this SHORT
ignore_signature_status_codes = NO_PUBKEY

The equivalent on the CLI is ansible-galaxy collection install community.general:9.4.0 --keyring /etc/ansible/enclave-galaxy.gpg --required-valid-signature-count 1. Set required_valid_signature_count to +1 (with the plus) to mean “at least one valid signature and no invalid ones” — the stricter reading you almost always want. If an attacker swaps a tarball, the GPG check fails and the install aborts before a single module lands on disk. That is the enforcement point; the mirror URL is just plumbing.

`ansible-galaxy collection download` resolves the whole dependency tree

A subtle but load-bearing detail: ansible-galaxy collection download does not just grab the one collection you named — it resolves and downloads every dependency, then writes a fresh requirements.yml into the download directory listing exact versions. That generated file is your import manifest. Beginners hand-carry one tarball, then discover inside the enclave that it needs three collections they never fetched. Always download to an empty directory and ship the entire directory, requirements.yml included:

---
- name: Sync host — resolve full dependency tree offline
  hosts: sync_host
  tasks:
    - name: Download a collection and all its dependencies
      ansible.builtin.command:
        cmd: >
          ansible-galaxy collection download
          community.general:9.4.0
          -p /var/sync/collections
      changed_when: true

The transparency-log trap in offline `cosign`

This is the single most common EE-signing failure in air-gaps. By default cosign sign uploads a record to the public Rekor transparency log (rekor.sigstore.dev), and cosign verify tries to check that log. Both operations reach the internet — which does not exist inside your enclave. You have two correct options:

# Option 1: sign without a transparency log; verify while ignoring it
cosign sign  --key cosign-airgap.key --tlog-upload=false registry.local/ee-airgap-base:1.0.0
cosign verify --key cosign-airgap.pub --insecure-ignore-tlog=true registry.local/ee-airgap-base:1.0.0

# Option 2 (higher assurance): run an INTERNAL Rekor + Fulcio and point cosign at them
export COSIGN_REKOR_URL=https://rekor.enclave.local

Option 1 is fine for most enclaves; Option 2 gives you a tamper-evident, in-enclave audit log of every signature. The mistake to avoid is copying a tutorial that omits --tlog-upload=false and then wondering why signing hangs for 30 seconds and fails on a DNS timeout.

Pin by digest, not by tag

:latest is the obvious villain, but any mutable tag is a soft spot: someone can push a new image to ee/airgap-base:1.0.0 and every job silently changes. The immutable identifier is the content digest. Capture it at import and pin job templates to it:

# resolve the digest once, at import time
podman inspect --format '{{.Digest}}' harbor.enclave.local/ee/airgap-base:1.0.0
# → sha256:9f2c...  → pin the job template to
#   harbor.enclave.local/ee/airgap-base@sha256:9f2c...

A digest is a cryptographic hash of the image contents; it cannot be re-pointed. Tags are for humans, digests are for machines and auditors.

Offline Python: hash-checking mode is your friend

pip download -r requirements.txt --require-hashes refuses to fetch anything whose hash is not pinned in the requirements file, and pip install --no-index --find-links=/opt/wheels --require-hashes refuses to install anything unpinned. Enabling hash-checking on any line enables it for all lines — so one missing hash fails the whole build, which is exactly the fail-closed behaviour you want. Remember wheels are platform-specific: a manylinux x86-64 wheel will not satisfy an aarch64 execution node, so mirror the wheels for every architecture your EEs actually run on.

Pulp’s content model, and why “publication” ≠ “distribution”

Whether you use Automation Hub or bare Pulp, the mental model is three layers: a repository holds immutable versions of content; a publication is a rendered, signable snapshot of one version; a distribution serves a publication at a stable URL (base_path). The air-gap-relevant point is that syncing new content does not change what hosts see until you create a new publication and repoint the distribution. That indirection is a feature: you stage and sign in the background, then flip the distribution atomically during a change window. Beginners sync a repo, see no change on their hosts, and assume it failed — it did not; they skipped the publish+distribute step.

One-way transfer has no retransmission — plan for it

On a data diode there is no ACK, no TCP, no retransmit — the return path physically does not exist. A dropped packet is gone forever. Diode file-transfer tools therefore layer forward error correction (FEC) and send each bundle multiple times, so the receiver can reconstruct from redundancy alone. The practical consequences for your runbook: bundles must be idempotent to re-import (verify-by-hash, skip if already present), transfers are slower than a two-way link of the same bandwidth, and you size the FEC overhead into your transfer-window planning. Treat “the bundle didn’t arrive intact” as a normal, expected event with a documented re-send procedure — not an incident.

DNS as a fail-closed tripwire

NXDOMAIN-as-a-control deserves more than a bullet. Configure the enclave resolver so that *.ansible.com, *.quay.io, pypi.org, github.com and friends return NXDOMAIN immediately, not a slow timeout. The difference matters: a timeout hides a misconfigured playbook for 30 seconds and then throws a vague error; an instant NXDOMAIN makes the mistake loud and unmistakable at the exact line that tried to escape. It is the cheapest intrusion-and-misconfiguration detector you will ever deploy, and it turns “someone forgot to fix the source URL” from a silent latent bug into an immediate, obvious failure.

Frequently asked questions

1. Can I run ansible-core inside an air-gap without any modifications? Yes. ansible-core has no network dependency at runtime beyond what your modules call out to (SSH for Linux, WinRM for Windows). The work is in the content (collections, modules, EEs) and the environment (Python, system packages). Once those are mirrored, ansible-core is happy.

2. How do I update collections on a quarterly cadence inside the enclave? Run the sync host’s pull-and-sign playbook on schedule, transport the bundle, run the enclave’s verify-and-import playbook, then publish a new requirements.yml that pins the new versions. Existing job templates continue to use the old versions until you explicitly bump them; nothing is auto-updated.

3. What about ansible-lint, molecule, and CI tooling — do those work air-gapped? Yes, with the same offline pattern. ansible-lint pulls rule packs from PyPI; mirror them. Molecule pulls container images; mirror them. CI runners (Jenkins, GitLab Runner) are inside the enclave with cached source code and inherited from the EE registry.

4. How do I handle credentials for cloud collections in an air-gapped environment? The cloud endpoint must be reachable from the enclave. For sovereign clouds (AWS GovCloud, Azure Government), this is direct; the enclave is peered. For enterprise patterns where the enclave is truly sealed and cloud is “outside,” you do not run cloud collections at all — you run on-prem/private-cloud collections (community.vmware, kubernetes.core against in-enclave clusters).

5. What’s the right way to handle EE rebuilds for security patches? The sync host rebuilds EEs nightly with the latest patched base layers and pinned content versions. The new image is signed and tagged with both :1.0.<n> (incremental) and :1.0 (latest patched within minor). The enclave imports new tags weekly. Job templates pin to specific full tags during stable periods, and bump in change windows.

6. Can Event-Driven Ansible work across the air-gap? Only one-directionally with discipline. Events from inside the enclave can be processed in-enclave (the typical pattern). Events from outside cannot trigger work inside, because that is a hole in your boundary. If you need cross-boundary events, the diode pattern applies: low-side writes a signed event bundle, high-side picks it up via diode-rx, EDA inside the enclave consumes the bundle.

7. What about secrets management — does Vault work air-gapped? HashiCorp Vault, CyberArk, and the AAP-bundled credential store all work fully air-gapped. Bring up the secrets backend inside the enclave, integrate with internal AD/IdP, and treat it as the only source of credentials. Never sync secrets from a corporate password manager into the enclave; provision separately.

8. How do I provide patching SLAs inside an air-gap? Tighten the sync cadence: critical CVEs trigger an out-of-cycle bundle build, transport, and import within hours of disclosure. Practice the cycle quarterly so the operators know the steps. The “patching window” inside the enclave is 24–72 hours from CVE publication for critical, 7 days for high, 30 days for medium — measured from when the bundle becomes available, not from CVE disclosure.

9. What’s the audit story for a sealed enclave? Every artefact import is a logged event with: source URL, upstream signature hash, internal signature hash, importer identity, timestamp, ticket reference. AAP execution events are logged with EE digest and collection version pins. Together, they let you reconstruct “what code ran on what host, sourced from what upstream version, on what date” — which is what auditors want.

10. What’s the single most underrated air-gap practice? The monthly content review. List every collection, EE, repo, and binary in your internal mirror; ask “do we still use this?”. Aggressively prune. The number of unused, unmaintained, and unwatched dependencies in a typical enterprise is staggering, and each one is a future supply-chain bug. An hour a month of review is the cheapest security improvement available.

Hands-on lab — build a tiny offline collection mirror

This lab simulates an air-gap on a single workstation: one container plays the role of the sync host (with internet), another plays the role of the enclave (network-isolated). You will set up an internal collection mirror, sign it, transfer it, verify it, and run a play from the enclave that uses the mirrored content.

Prerequisites: Podman or Docker, ansible-core ≥ 2.16, gpg.

mkdir -p airgap-lab/{sync,enclave,bundles}
cd airgap-lab
gpg --quick-gen-key 'enclave-sync@kv.local' rsa4096 sign 1y

# 1. Sync host: download a collection
podman run --rm -v $PWD/sync:/work:Z quay.io/ansible/ansible-runner \
  ansible-galaxy collection download community.general:9.4.0 -p /work

# 2. Sign it
gpg --detach-sign --armor --local-user enclave-sync@kv.local \
  sync/community-general-9.4.0.tar.gz

# 3. Build SHA256 manifest
( cd sync && sha256sum *.tar.gz *.asc > MANIFEST.sha256 )

# 4. Bundle for transport
tar cf bundles/airgap-bundle-$(date +%s).tar -C sync .

# 5. Enclave: simulate transport (just copy)
mkdir -p enclave/inbox
cp bundles/airgap-bundle-*.tar enclave/inbox/

# 6. Enclave: extract and verify
cd enclave/inbox && tar xf airgap-bundle-*.tar
gpg --verify community-general-9.4.0.tar.gz.asc community-general-9.4.0.tar.gz
sha256sum -c MANIFEST.sha256

# 7. Enclave: install offline
podman run --rm --network=none \
  -v $PWD:/work:Z quay.io/ansible/ansible-runner \
  ansible-galaxy collection install /work/community-general-9.4.0.tar.gz \
    -p /work/collections

The --network=none flag on the enclave podman run is the lab’s “air-gap”: no network at all. The collection installs cleanly because everything is local. Now run a play from inside that no-network container:

# enclave/test.yml
- hosts: localhost
  collections: [community.general]
  tasks:
    - community.general.archive:
        path: /tmp/foo.txt
        dest: /tmp/foo.tar.gz
        format: gz

Run with --network=none and observe success. You have just replicated, in miniature, the entire air-gap content lifecycle: pull, sign, bundle, transport, verify, install, run.

Practice challenges

Work these in order — the first two are approachable the moment you finish the lab, the last two are accreditation-grade. Each has a worked solution, but try it before you open it.

1. (Beginner) Redirect ansible-galaxy at your enclave Hub. Write the ansible.cfg stanza so that ansible-galaxy collection install only ever talks to https://hub.enclave.local/api/galaxy/content/published/ and never resolves public Galaxy.