Ansible for Hybrid & Multi-Cloud Orchestration: Coordinating On-Prem, AWS, Azure, GCP, and Kubernetes from a Single Workflow

In a nutshell

Imagine you run a global airline from a single control tower. Some aircraft are yours (on-prem servers in your own datacenters), some are leased from three different partners (AWS, Azure, GCP), and some are new drones (Kubernetes). You never fly a plane yourself. You issue one coordinated plan — “board, taxi, take off, in this order; if a flight aborts, hold its connections and bring it back to the gate” — and every aircraft, whoever built it, obeys the same plan. Ansible is that control tower for infrastructure change.

Hybrid and multi-cloud orchestration means driving change across environments that were never designed to cooperate — a bank’s RHEL fleet on vSphere, an AWS account, an Azure tenant, a GCP project, a Kubernetes cluster, and the network gear stitching them together — from one workflow, in the right order, with a dry-run you can approve and a rollback when something breaks. Ansible reaches each environment through its own dynamic inventory plugin (amazon.aws.aws_ec2, azure.azcollection.azure_rm, google.cloud.gcp_compute, kubernetes.core.k8s), pulls the correct credential per environment, and runs the same idempotent tasks everywhere.

Hold on to one mental split throughout this lesson: Terraform (or another IaC tool) provisions the raw infrastructure; Ansible configures and orchestrates the change across it. That division shows up everywhere here. A beginner starts with a single playbook that pings one VM in each cloud; an expert composes those plays into an Ansible Automation Platform (AAP) workflow that safely upgrades a payment processor across 12 datacenters and 3 clouds. Same ideas, different blast radius.

Level: Advanced (Tier 4 capstone) · Time: ~55 min · You’ll need: comfort with playbooks and inventory, and at least one cloud collection (AWS, Azure, or GCP).

A bank with 80 years of history runs a portfolio you can’t restart for fun: COBOL on z/OS handles ledger updates, RHEL 9 on vSphere serves the trading floor, AWS handles the mobile app, Azure handles Office 365 integration, GCP handles the data-science team’s BigQuery, and Kubernetes handles the new microservices the platform team is building. Most of these can’t move; all of them need to change weekly, and every change has to be auditable. The problem is not picking a tool — it’s picking a coordination plane that can drive all of them.

That coordination plane, for a wide swath of enterprises, is Ansible Automation Platform plus a careful approach to multi-tier inventory, automation mesh, and workflow chaining. This lesson — the capstone of the Ansible expert tier — covers how to build orchestrated, multi-environment changes that stretch across vSphere clusters, public clouds, Kubernetes, network gear, and Windows fleets, all in a single workflow with proper dependency ordering, dry-run gating, partial-failure handling, and rollback. If you’ve made it through the previous nine expert lessons, this is where they fit together.

Learning Objectives

By the end you will be able to:

Design cross-platform inventories that combine static (on-prem network, vSphere) and dynamic (AWS, Azure, GCP) sources into one logical fabric.
Use Ansible Automation Platform (AAP) Workflow Templates to chain Job Templates with on_success / on_failure / always branches.
Configure automation mesh: hop nodes, execution nodes, and control nodes across network boundaries (DMZ, restricted VLANs, partner VPCs).
Map identity across environments: AD-joined Linux + Windows, AWS IAM Roles via OIDC, Azure Workload Identity, GCP WIF, K8s ServiceAccounts.
Manage credentials across three tiers — environment variables, Ansible Vault, and short-lived Workload Identity Federation — and know when each is appropriate.
Split responsibilities correctly between Terraform (provision) and Ansible (configure/orchestrate), and hand off state between them.
Detect and remediate configuration drift across clouds using idempotent tasks and check-mode gates.
Implement dry-run gating: a --check workflow that posts a diff to Slack/PR before the actual change runs.
Build “blast-radius-aware” rollouts: serial: percentages, canary regions, ringed deployments.
Handle partial failures: workflow convergence_node, any_errors_fatal, max_fail_percentage, plus rollback Job Templates.
Log every cross-environment change to a single audit stream (Splunk, Elastic, ServiceNow CMDB).
Choose between AAP, AWX, plain CLI Ansible + GitOps, or a hybrid for your environment.

Prerequisites

All nine previous Tier 4 expert lessons (network, AWS, Azure, GCP, Windows, K8s, containers, databases, VMware).
Tier 1–3 fluency, especially AAP basics and dynamic inventory.
Read-only access to your real production inventory across all clouds you’ll orchestrate.
Comfort with the per-cloud collections — the AWS amazon.aws lesson, the Azure azure.azcollection lesson, and the GCP google.cloud lesson — since this capstone assumes you already know each cloud’s modules and inventory plugin and focuses on weaving them together.

Mental Model: Hybrid Orchestration Done Right

1. One inventory, many sources

The right approach is not “one playbook per platform” with separate inventories. It is one logical inventory that aggregates multiple sources: a static YAML for on-prem network gear and vSphere, dynamic plugins for AWS/Azure/GCP, and a kubernetes.core inventory for K8s clusters. The aggregator file in inventory/ lists each source; AAP or ansible-inventory merges them into one host graph at runtime.

2. Automation mesh routes execution, not data

In a real enterprise, the control node sits in a management VPC. AWS prod is in a different account behind a private VPC. Azure prod is on-prem-routed via ExpressRoute. The on-prem network gear is on a management VLAN that the cloud control node can’t reach. Automation mesh is AAP’s solution: deploy execution nodes close to the targets (a small EC2 in the AWS account, a small VM in Azure, a small VM in the management VLAN), wire them to the control plane via hop nodes across firewalls, and let AAP route each Job to the right execution node based on inventory location.

3. Workflows chain Job Templates with branching logic

A Workflow Template is a DAG (directed acyclic graph) of Job Templates. Edges are conditional: on_success continues the success path, on_failure triggers cleanup, always runs regardless. A real change workflow looks like: pre-check → backup → drain → upgrade → validate → traffic-shift → confirm. Each of those is a Job Template, and the workflow stitches them with the right branching logic.

4. Identity is the hard problem in hybrid

The control node is RHEL 9. It needs to talk to: vSphere (SSO user), Windows (Kerberos, AD-joined), AWS (IRSA / OIDC), Azure (Workload Identity), GCP (Workload Identity Federation), Kubernetes (kubeconfig with short-lived tokens), network gear (TACACS+ / RADIUS / SSH key). Don’t try to unify these — instead, give AAP credentials per environment and let each Job Template pull only the credentials it needs.

5. Audit trail beats clever automation

The hardest part of multi-cloud Ansible isn’t getting it to work; it’s proving what changed, when, and by whose authority. AAP’s job-event stream is the spine of this audit story — every task on every host gets a structured record. Ship those events to Splunk/Elastic/ServiceNow, tag them with the change ticket ID (extra_vars: { change_id: CHG0001234 }), and the audit becomes searchable.

Designing a Cross-Platform Inventory

The aggregator pattern: one directory of inventory sources, each loaded by its own plugin, all merged into one inventory graph.

inventory/
├── 01-static.yml         # on-prem network gear, on-prem VMs
├── 02-vsphere.yml        # community.vmware.vmware_vm_inventory
├── 03-aws.aws_ec2.yml    # amazon.aws.aws_ec2
├── 04-azure_rm.yml       # azure.azcollection.azure_rm
├── 05-gcp_compute.yml    # google.cloud.gcp_compute
├── 06-k8s.yml             # kubernetes.core.k8s
└── group_vars/
    ├── all.yml
    ├── tag_environment_prod.yml
    └── tag_role_database.yml

01-static.yml (the on-prem skeleton):

all:
  children:
    network_devices:
      hosts:
        leaf-01.dc1.corp.example.com:
        leaf-02.dc1.corp.example.com:
        spine-01.dc1.corp.example.com:
      vars:
        ansible_connection: ansible.netcommon.network_cli
        ansible_network_os: cisco.nxos.nxos
        environment: prod
        location: dc1

03-aws.aws_ec2.yml:

plugin: amazon.aws.aws_ec2
regions:
  - us-east-1
  - us-west-2
filters:
  tag:Environment: prod
  instance-state-name: running
keyed_groups:
  - key: tags.Role
    prefix: role
  - key: tags.Environment
    prefix: env
  - key: placement.region
    prefix: aws_region
hostnames:
  - tag:Name
  - private-ip-address
compose:
  ansible_host: private_ip_address

04-azure_rm.yml:

plugin: azure.azcollection.azure_rm
auth_source: auto
include_vm_resource_groups:
  - prod-rg-east
  - prod-rg-west
keyed_groups:
  - key: tags.role
    prefix: role
  - key: location
    prefix: az_region
  - prefix: env
    key: tags.environment
hostnames:
  - private_ipv4_addresses[0]

05-gcp_compute.yml:

plugin: google.cloud.gcp_compute
projects:
  - corp-prod-1234
auth_kind: serviceaccount
service_account_file: /etc/ansible/sa-prod.json
filters:
  - status = RUNNING
  - labels.environment = prod
keyed_groups:
  - key: labels.role
    prefix: role
  - key: zone
    prefix: gcp_zone
hostnames:
  - networkInterfaces[0].networkIP

Run ansible-inventory -i inventory/ --graph and you get one tree:

@all:
  |--@aws_region_us_east_1:
  |  |--10.10.1.5
  |  |--10.10.1.6
  |--@az_region_eastus:
  |  |--10.20.1.5
  |--@gcp_zone_us_central1_a:
  |  |--10.30.1.5
  |--@network_devices:
  |  |--leaf-01.dc1.corp.example.com
  |--@role_app:
  |  |--10.10.1.5
  |  |--10.20.1.5
  |--@role_db:
  |  |--10.10.1.6
  |  |--10.30.1.5
  |--@env_prod:
  |  |--10.10.1.5
  |  |--10.10.1.6
  |  |--10.20.1.5
  |  |--10.30.1.5

role_app now contains app servers from AWS and Azure together; role_db spans AWS and GCP. A play that targets role_app runs across all of them — but each task is dispatched to the right execution node by automation mesh.

The trick that makes clouds merge: the same prefix: in keyed_groups across two different plugins. AWS tags its instances Role=app and Azure tags its VMs role=app; because both plugin files declare prefix: role, the resulting groups (role_app) are the same group, and hosts from both clouds pile into it. Choose a consistent tagging convention across clouds up front — it is the single decision that makes “target all prod app servers everywhere” a one-liner instead of a scripting project. (Cloud tag keys are case-sensitive and differ by provider, so normalize with the key: expression, not by hoping the tags already match.)

Automation Mesh: Routing Execution

Automation mesh has three node types:

Control node: hosts the AAP UI, scheduler, and database.
Execution node: actually runs the Job. Each Job’s payload (collections, EE image) is shipped here.
Hop node: a relay that forwards traffic across network boundaries (e.g. across a DMZ).

A real layout for a bank’s prod workflow:

[Control] --(public)--> [Hop: DMZ-1] --(private)--> [Exec: AWS-prod]
                              |
                              +--(private)-----> [Exec: Azure-prod]
                              |
                              +--(MPLS)--> [Hop: DC1-mgmt] --(VLAN-100)--> [Exec: vSphere-prod]
                                                          \
                                                           +--(VLAN-200)--> [Exec: Network-mgmt]

Each execution node has the credentials and network reachability for its segment. The control node never directly touches the prod targets — it only schedules Jobs to the right execution nodes, which do the actual SSH/WinRM/HTTPS calls.

In AAP, configure an Instance Group per region/segment, then bind inventories to instance groups:

Inventory "AWS-prod-us-east-1" → instance_group "exec-aws-prod-us-east-1"
Inventory "Azure-prod-eastus"  → instance_group "exec-azure-prod-eastus"
Inventory "vSphere-prod-dc1"   → instance_group "exec-vsphere-prod-dc1"

When you launch a Job Template against Inventory "AWS-prod-us-east-1", AAP routes it through the mesh to the AWS execution node — no separate “did you remember to switch context?” step.

Workflow Templates: Chaining Job Templates

A Workflow Template is a DAG. Each node is a Job Template (or another workflow). Edges are conditional.

Example: production app deployment that touches AWS, Azure, and on-prem K8s:

[Pre-check]
  |
  +--on_success--> [Backup vSphere VMs]
                         |
                         +--on_success--> [Snapshot AWS RDS]
                                                |
                                                +--on_success--> [Drain LB traffic]
                                                                       |
                                                                       +--on_success--> [Rolling upgrade k8s]
                                                                                              |
                                                                                              +--on_success--> [Validate]
                                                                                                                    |
                                                                                                                    +--on_success--> [Restore traffic]
                                                                                                                    +--on_failure--> [Rollback k8s] --> [Restore traffic]
                                                                                              |
                                                                                              +--on_failure--> [Restore RDS] --> [Restore vSphere]

In AAP, this is a 9-node Workflow Template. Each node is a Job Template that already exists. Edges are configured in the Workflow visualizer.

Each Job Template can pass data to the next via set_stats:

- name: Record current image tag
  ansible.builtin.set_stats:
    data:
      previous_image_tag: "{{ current_image_tag }}"
    per_host: false

Subsequent Job Templates in the workflow can read previous_image_tag from extra_vars.

Dry-Run Gating: Diff → Approval → Apply

The single highest-leverage practice in multi-cloud Ansible: never apply without a diff. The pattern:

Dry-run Job Template runs the same playbook with --check --diff.
Output is rendered to a diff document and posted to a PR or Slack thread.
A human approves.
Apply Job Template runs the same playbook without --check.

In AAP this is two Job Templates wired in a Workflow with an Approval Node between them:

[Dry-run] --on_success--> [Approval Node] --on_approved--> [Apply] --on_success--> [Validate]
                                          --on_denied--> [Notify-denied]

The Approval Node pauses the workflow until a human clicks “Approve” in the AAP UI (or replies to a notification). This is your last-line audit gate before production change.

Identity Across Environments

The cleanest pattern: per-environment credential, no shared secrets.

Target	AAP Credential Type	Backing identity
On-prem Linux	Machine	SSH key, AD-joined service account
Windows	Machine (winrm/kerberos)	gMSA in AD
Network gear	Network	TACACS+ user
AWS	Amazon Web Services	IAM Role assumed via OIDC from AAP’s K8s SA
Azure	Microsoft Azure Resource Manager	Workload Identity from AAP’s K8s SA
GCP	Google Compute Engine	WIF mapped from AAP’s K8s SA
vSphere	VMware vCenter	SSO user with custom RBAC
Kubernetes	OpenShift / Kubernetes Bearer Token	Short-lived OIDC token

Notice the pattern: AAP runs in a K8s cluster (or OpenShift) — its workload identity (the K8s ServiceAccount of the Execution Environment pod) is the root of trust, and each cloud is configured to trust that identity via OIDC. No long-lived keys anywhere.

Credential Management: Environment, Vault, and Workload Identity

The identity table above says which credential each target uses. This section is about where the secret lives and how it reaches the module at runtime — the part beginners get wrong most often. There are three tiers, and mature shops climb them left-to-right:

Tier	Where the secret lives	Rotation	Use when
1 — Environment / config	Env vars, `~/.aws/credentials`, a SA JSON file on disk	Manual	Your laptop, a demo, a throwaway lab
2 — Ansible Vault	Encrypted vars file in git, decrypted at runtime with a vault password	Manual, but auditable in git	Static secrets that rarely change (API tokens, a bootstrap password)
3 — Workload Identity Federation (WIF/IRSA)	Nowhere persistent — a short-lived token minted per job from a trusted workload identity	Automatic (minutes-long TTL)	Production. The target state you want to reach

Tier 1 — environment. The cloud collections lean on each provider SDK’s native credential chain, so you often set nothing in the playbook:

amazon.aws.* uses boto3’s chain: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN env vars, AWS_PROFILE, or the instance/pod role.
azure.azcollection.* with auth_source: auto reads AZURE_CLIENT_ID / AZURE_SECRET / AZURE_TENANT, a managed identity, or a federated token file.
google.cloud.* reads a service-account JSON (auth_kind: serviceaccount) or Application Default Credentials (auth_kind: application).

This is fine for a laptop and dangerous for prod — plaintext keys on disk are exactly the thing you don’t want on a Tier-0 automation host.

Tier 2 — Ansible Vault. For static secrets, encrypt them at rest in git and decrypt at runtime. The Ansible Vault lesson covers this in depth; the hybrid-relevant part is that a play never contains the plaintext — it references a variable whose value came from an encrypted file:

- name: Create the application bucket
  amazon.aws.s3_bucket:
    name: "{{ app_bucket_name }}"
    state: present
    tags:
      environment: prod
      managed_by: ansible

Here app_bucket_name (and any real secret) lives in group_vars/all/vault.yml, encrypted with ansible-vault. AAP stores the vault password itself as a Vault credential, so no human types it during a job. For rotating secrets — database passwords that change nightly — reach past static Vault to an external secrets store (HashiCorp Vault, AWS Secrets Manager) via a lookup at runtime, so the secret is never committed at all.

Tier 3 — Workload Identity Federation. The production target: no static key exists to leak. AAP runs its Execution Environment pods with a Kubernetes ServiceAccount; each cloud is configured to trust that ServiceAccount’s OIDC token and mint a short-lived credential in exchange:

AWS (IRSA): the pod’s projected SA token is exchanged via sts:AssumeRoleWithWebIdentity for temporary AWS credentials — boto3 does this automatically, so the playbook sets nothing.
Azure (Workload Identity): a federated token file is exchanged for an Azure AD token; auth_source: auto picks it up.
GCP (WIF): Application Default Credentials resolve the federated token to a GCP access token — so the inventory plugin needs no key file:

plugin: google.cloud.gcp_compute
projects:
  - corp-prod-1234
auth_kind: application
filters:
  - status = RUNNING
keyed_groups:
  - key: labels.role
    prefix: role

The whole point: the credential that can change your AWS account never sits on disk, never enters git, and expires in minutes. A leaked backup or a compromised git history yields nothing usable. This is why “give AAP a per-environment credential and let each job pull only what it needs” is not bureaucracy — it is the mechanism that keeps a single compromised playbook from becoming a cross-cloud breach.

Provision with Terraform, Configure with Ansible

Beginners often try to build entire cloud estates with Ansible cloud modules — VPCs, subnets, route tables, the lot. It works, but you’re fighting the tool: Ansible has no first-class state file and no dependency graph, so it can’t cheaply answer “what exists, what must change, in what order, and what must be destroyed.” That is exactly what Terraform (or OpenTofu/Pulumi) is built for. The industry-standard division:

Concern	Terraform (provision)	Ansible (configure & orchestrate)
Create VPCs, subnets, load balancers, managed DBs	Yes — declarative, with state	Rarely (only day-0 glue)
Install/patch packages, render config, manage services	No	Yes
Multi-step change with human approval and rollback	No	Yes (workflows)
Cross-host ordering (drain → upgrade → validate)	No	Yes (`serial`, handlers, workflows)
Source of truth for “what infra exists”	Terraform state	— (reads it)
Source of truth for “how hosts are configured”	—	Ansible playbooks/roles

Read it as Day-0 vs Day-1/Day-2: Terraform stands the infrastructure up (Day-0); Ansible configures it into service (Day-1) and operates it for the rest of its life — patching, drift correction, coordinated change (Day-2).

The handoff is the interesting part, and Ansible has a native path for it. The cloud.terraform collection lets Ansible both run Terraform and read its state as inventory:

- name: Provision base network with Terraform, then configure it
  hosts: localhost
  tasks:
    - name: Apply the network stack
      cloud.terraform.terraform:
        project_path: ./terraform/network
        state: present
        force_init: true
      register: tf

And build inventory directly from Terraform state, so the machines Terraform just created are immediately targetable — no tag-scraping race:

# inventory/07-tfstate.yml
plugin: cloud.terraform.terraform_state
backend_type: s3
backend_config:
  bucket: corp-tfstate-prod
  key: network/terraform.tfstate
  region: us-east-1

In AAP the same idea becomes a Workflow: a Terraform Job Template (or a Terraform Cloud run) provisions, its outputs feed the inventory, and an Ansible Job Template configures. You get Terraform’s declarative provisioning and Ansible’s imperative, ordered, approvable orchestration — each doing what it’s best at.

Idempotency and Drift Across Clouds

Idempotency is the property that running the same playbook twice leaves the system in the same state, and the second run reports changed=0. It does not mean “the playbook never changes anything” — the first run may change plenty. It means the playbook describes a destination, and Ansible only acts when reality differs from that destination. This is what makes it safe to re-run the same play on a nightly schedule across thousands of hosts in three clouds.

Drift is the flip side: reality has diverged from the declared state — someone SSH’d in and hand-edited /etc/nginx/nginx.conf, or a cloud-console change bypassed automation. Because idempotent tasks report when they would change something, you can turn a normal playbook into a drift detector by running it in check mode and failing if anything would change:

- name: Detect configuration drift (read-only)
  hosts: role_app
  check_mode: true
  tasks:
    - name: Render nginx config against source of truth
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      register: cfg

    - name: Fail if the live config has drifted
      ansible.builtin.assert:
        that: not cfg.changed
        fail_msg: "Drift on {{ inventory_hostname }}: /etc/nginx/nginx.conf differs from source of truth"
        success_msg: "No drift on {{ inventory_hostname }}"

Schedule that in AAP nightly against role_app (which, remember, spans AWS and Azure) and you get a single cross-cloud drift alarm. When it fires, the same playbook run without check_mode remediates the drift — detection and correction share one source of truth.

The honest caveat: not every task is naturally idempotent, and not every module supports check mode. ansible.builtin.command and shell run whatever you give them every time and always report changed. Make them honest:

- name: Generate the nightly report (be honest about change)
  ansible.builtin.command: /usr/local/bin/gen-report.sh
  register: report
  changed_when: "'WROTE' in report.stdout"
  check_mode: false

changed_when teaches Ansible what “changed” actually means for this command; check_mode: false says “this task can’t be dry-run, so run it for real even during --check” (use it only for genuinely read-only or safe commands). Cloud modules vary too — most amazon.aws/azure.azcollection/google.cloud resource modules support check mode, but a few report changes imperfectly because reading current cloud state is expensive. The rule: check the module’s documentation for check-mode support before you trust a dry-run diff from it. Drift detection is only as truthful as the least-idempotent task in the play.

Blast-Radius-Aware Rollouts

For changes that touch many hosts, use these patterns to limit the damage radius:

Pattern 1: `serial:` with percentages

- hosts: webservers
  serial: "10%"   # 10% of the fleet at a time
  tasks:
    - name: Apply config
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      notify: reload nginx

serial: "10%" rolls through the fleet 10% at a time. Combined with max_fail_percentage: 5 — if more than 5% of a batch fails, abort.

Pattern 2: Canary regions

- hosts: aws_region_us_east_1   # canary first
  tasks: [ ... ]

- hosts: aws_region_us_west_2   # then west
  tasks: [ ... ]

- hosts: az_region_eastus       # then Azure east
  tasks: [ ... ]

In AAP this becomes a Workflow with one Job Template per region, gated by validation checks between each.

Pattern 3: Ringed deployment

Tag hosts with rings: ring_0 (canary, 1%), ring_1 (early adopters, 10%), ring_2 (general, 89%). Deploy ring-by-ring with bake time between rings.

Rollback Strategy

Ansible isn’t transactional. You can’t BEGIN; ... ROLLBACK; an apt install + a route change. Rollback discipline:

Snapshot stateful resources before change (vSphere snapshot, RDS snapshot, EBS snapshot, GCP disk snapshot).
Capture pre-change state in set_stats (image tag, replica count, config version).
Have a dedicated rollback Job Template for each change type. The rollback playbook reads the captured state and reverses the change.
Test rollbacks regularly. A rollback path you’ve never run is theater.

The rollback Job Template is a peer to the apply Job Template, not an afterthought:

[Apply] --on_failure--> [Rollback] --on_success--> [Restore traffic] --> [Notify-failure]

Audit Trail

AAP emits Job Events as a structured stream. Each event has timestamp, host, task, status, and diff. Ship them to your audit pipeline:

# /etc/tower/conf.d/logging.py (AAP) — log streaming
LOGGING['handlers']['external_logger'] = {
  'class': 'logging.handlers.SysLogHandler',
  'address': ('splunk-hec.corp.example.com', 5140),
  'formatter': 'json',
}

Tag every job with the change ticket ID:

# Workflow extra_vars
change_id: "{{ change_id }}"
approver: "{{ approver }}"

Now your audit pipeline has: who launched the workflow, when each task ran, what changed on which host, with which inputs — all queryable.

Hands-on Free Lab: Cross-Platform Inventory + Workflow

Free, runs against LocalStack (AWS), Azurite (Azure), kind (K8s), and a local Multipass VM (on-prem). The full lab is documented at github.com/example/ansible-hybrid-lab; here’s the skeleton:

mkdir -p ~/ansible-hybrid-lab && cd ~/ansible-hybrid-lab

# Build aggregator inventory
mkdir -p inventory
cat > inventory/01-static.yml <<'EOF'
all:
  children:
    onprem:
      hosts:
        local-vm:
          ansible_host: 192.168.64.10
          ansible_user: ubuntu
EOF

cat > inventory/02-aws.aws_ec2.yml <<'EOF'
plugin: amazon.aws.aws_ec2
endpoint: http://localhost:4566   # LocalStack
regions:
  - us-east-1
filters:
  instance-state-name: running
keyed_groups:
  - key: placement.region
    prefix: aws_region
EOF

cat > inventory/03-k8s.yml <<'EOF'
plugin: kubernetes.core.k8s
connections:
  - kubeconfig: ~/.kube/config
    context: kind-ansible-lab
EOF

# Build hybrid playbook
cat > site.yml <<'EOF'
---
- name: Phase 1 — On-prem hosts
  hosts: onprem
  tasks:
    - name: Ensure nginx
      ansible.builtin.apt:
        name: nginx
        state: present
      become: true

- name: Phase 2 — AWS hosts (LocalStack)
  hosts: aws_region_us_east_1
  gather_facts: false
  tasks:
    - name: Ping
      ansible.builtin.ping:

- name: Phase 3 — K8s namespace
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Apply baseline
      kubernetes.core.k8s:
        context: kind-ansible-lab
        state: present
        definition:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: hybrid-lab
EOF

# Run
ansible-inventory -i inventory --graph
ansible-playbook -i inventory site.yml

Three different platforms, one inventory, one playbook, three plays — that’s the hybrid pattern in miniature.

Going deeper

This is the section for the reader who already runs multi-cloud Ansible and wants the internals — the parts that decide whether your orchestration survives contact with a 10,000-host change.

Automation mesh under the hood: Receptor

Automation mesh isn’t magic — it’s the Receptor overlay network. Each control, hop, and execution node runs a receptor process; they connect (default TCP 27199) into a mesh and route work units to each other over mutually-authenticated TLS. When AAP launches a job bound to an instance group, it hands a signed work unit to the mesh; Receptor routes it hop-by-hop to the chosen execution node, where ansible-runner executes the play inside an Execution Environment container. Two consequences worth internalizing: (1) the port must be open bidirectionally between adjacent mesh nodes — mesh links are not one-way; (2) the control node never needs a route to your targets, only a route to the next mesh node, which is precisely why mesh solves the “control plane can’t reach the restricted VLAN” problem without VPN sprawl.

Execution Environments are your reproducibility boundary

An EE is an OCI image built by ansible-builder from an execution-environment.yml: a base image plus a requirements.yml (Galaxy collections), requirements.txt (pip), and bindep.txt (system packages). Because every job runs in a fresh EE container, nothing leaks between jobs — which is why set_stats exists as the sanctioned hand-off channel. The EE is also where you pin versions:

# requirements.yml — baked into the EE, this is your provenance
collections:
  - name: amazon.aws
    version: ">=7.0.0,<8.0.0"
  - name: azure.azcollection
    version: ">=2.3.0"
  - name: google.cloud
    version: ">=1.3.0"
  - name: kubernetes.core
    version: ">=3.0.0"
  - name: cloud.terraform
    version: ">=2.0.0"

A floating collection version is how “it worked yesterday, broke today, nobody touched anything” happens — a collection released a new minor and a module argument changed. Pin in the EE, and ansible-galaxy collection list in the build log becomes your audit of exactly what code ran.

How set_stats actually crosses the workflow boundary

set_stats writes into the job’s artifact data. AAP forwards those artifacts to the next Workflow node as extra_vars — but only when stats forwarding is enabled and, for a single scalar shared across the whole run, you set per_host: false (otherwise you get per-host values you didn’t expect). Because the value lands as extra_vars, it sits near the top of variable precedence — above group/host vars, below only the explicit command line and workflow-level extra_vars. That is exactly what you want for a rollback token: the captured previous_image_tag should override any default the rollback play might otherwise pick up.

Check mode is not uniform — treat it per module

--check is honest only to the degree each module implements it. File/template/package/service modules support check mode and --diff well. Many network resource modules (via ansible.netcommon) support both and show the config diff. But some cloud modules report “would change” imperfectly because reading current state is expensive, and command/shell can’t dry-run at all. Never present a --check diff as gospel across a mixed play — validate check-mode support for the specific modules on your critical path, and mark the un-checkable ones explicitly with check_mode: false + a correct changed_when.

Performance at hybrid scale

A 10,000-host cross-cloud job lives or dies on tuning:

Forks: the default is 5. Raise forks (50–100+) so Ansible addresses many hosts in parallel; the control/execution node’s CPU and memory are the ceiling.
SSH pipelining: [ssh_connection] pipelining=true (or ANSIBLE_PIPELINING=true) collapses multiple SSH round-trips per task into one — often a 2–4× wall-clock win on Linux fleets.
Fact caching: cache facts (redis/jsonfile) and use gather_subset: min so you don’t re-gather everything on every run of a huge inventory.
Strategy: strategy: free lets fast hosts race ahead instead of waiting at each task barrier — ideal when hosts are independent; keep linear when ordering across hosts matters.
Async: wrap long-running tasks in async/poll: 0 so they don’t hold a fork.
Spread the load: put execution nodes near the targets via instance groups; latency to a us-east target from a us-east execution node beats round-tripping from a central control plane.

The dedicated performance-tuning lesson drills each of these; at hybrid scale they compound, because you’re multiplying per-task latency by hosts and by cross-region network distance.

The distributed-change reality: no atomic cross-cloud transaction

There is no two-phase commit across AWS + Azure + GCP. You cannot make “upgrade the app in all three clouds” atomic — a mid-flight failure will leave one cloud changed and another not. Accept it and design for partial failure instead of pretending it away: snapshot stateful resources before the change; make each step individually reversible; give each environment a compensating rollback (saga-style) rather than one global undo; and put a convergence node at the end that validates the whole system end-to-end before declaring success. Clever orchestration doesn’t beat this constraint — disciplined, reversible, snapshot-first design does.

Practice challenges

Work these in order — they climb from “merge two sources” to “design a convergent multi-cloud workflow.” Each has a solution with the command/manifest and a one-line why.

1. (Beginner) Aggregate two inventory sources into one graph. Create an inventory/ directory with a static on-prem host and an amazon.aws.aws_ec2 source, and prove they merge into a single tree.