Ansible Lesson 40 of 42

Ansible × Backup & Storage Automation, In Depth: Veeam, Rubrik, Commvault, NetApp ONTAP, Pure & the 3-2-1-1-0 Rule as Code

40 min read Vinod

In a nutshell

Most teams set up their backups once, by hand, clicking through a vendor console — and then never look at them again until the day they’re needed. That’s the day they discover the backup is two weeks stale, or silently broken, or that nobody remembers the encryption password. This lesson is about never being in that position: making Ansible the thing that creates, checks, and proves your backups automatically, on a schedule, whether or not any human remembers.

Think of a backup you’ve never restored as a spare tyre you’ve never taken out of the boot. You assume it’s inflated and the right size — but you only find out it’s flat on the night you’re stranded on the motorway in the rain. The whole point of this lesson is to pump up the spare and check that it actually fits, on a regular cadence, without depending on anyone remembering to do it.

The industry sums up “a backup you can actually trust” in five digits — 3-2-1-1-0: 3 copies of your data, on 2 kinds of storage, with 1 copy off-site, 1 copy that is immutable (locked so nobody — not even an admin or a ransomware attacker — can delete it), and 0 errors on your most recent test restore. Everything in this lesson is about expressing those five digits as code that Ansible enforces for you, across whatever backup product and storage array you happen to own.

Level: Advanced · Time: ~45 min

Before you start — and what you’ll be able to do

You’ll get the most from this lesson if you’re already comfortable writing playbooks, roles and variables, and if you’ve met a couple of ideas from earlier lessons:

After working through it you will be able to:

There is a special kind of weariness that comes over an engineer when they are asked, mid-incident, “do we have a backup?” The answer is almost never a clean yes or no. It’s “yes, but it’s two weeks old, and we haven’t tested a restore in three months, and the storage admin just left the company.” That fragility is what this lesson exists to fix.

In modern infrastructure, backups are simultaneously:

The thesis of this lesson is that backups become trustworthy only when they are treated as automated, tested, and policy-driven — not when they are configured by hand once and reviewed annually. We’ll cover four pillars:

  1. Backup-policy-as-code with the major vendor collections (Veeam, Rubrik, Cohesity, Commvault)
  2. Storage-array automation with NetApp ONTAP, Pure, and Dell PowerStore for the snapshot tier of the 3-2-1-1-0 rule
  3. Immutability and ransomware resilience — object-lock repositories, tape-equivalents, and the air-gapped 4th copy
  4. Automated restore drills — the discipline that separates real DR readiness from theatrical compliance checkbox

This lesson assumes you’ve internalised the DR lesson (D2) — backups are the content of DR; this lesson is the plumbing that creates and maintains that content correctly.

1. The 3-2-1-1-0 rule as the unifying policy

The classic backup mantra was 3-2-1: 3 copies, 2 different media, 1 off-site. Modern ransomware threats and compliance frameworks (FFIEC, NIST 800-209, HKMA, MAS TRM) extended this to 3-2-1-1-0:

The “1 immutable” and “0 errors” additions are what most organisations get wrong. The first because immutability is operationally inconvenient (you cannot delete bad backups quickly even when you want to). The second because automated restore testing is hard to set up and easy to defer.

The whole policy can — and should — be encoded as a single Ansible group_vars file applied to every protected workload:

# group_vars/all/backup_policy.yml
---
backup_policy:
  rpo_hours: 4              # max acceptable data loss
  rto_hours: 8              # max acceptable downtime
  retention:
    daily: 14               # 14 daily backups
    weekly: 6               # 6 weekly backups
    monthly: 12             # 12 monthly backups
    yearly: 7               # 7 yearly backups (regulatory)
  copies:
    primary:
      type: snapshot        # array-based snapshot
      target: ontap-cluster-01
      retention_days: 7
    secondary:
      type: backup
      target: veeam-repo-primary
      retention_days: 30
    offsite:
      type: backup_copy
      target: veeam-repo-aws-s3-frankfurt
      retention_days: 365
    immutable:
      type: backup_copy
      target: veeam-repo-aws-s3-objectlock
      object_lock_days: 90
      retention_days: 2555  # 7 years
  restore_test:
    frequency_days: 30
    sample_size_pct: 5
    success_threshold_pct: 100

Every backup automation playbook in your repo reads from this single source of truth. When the policy changes (auditor says “we need 14 monthlies, not 12”), you change one line and the next sync run reconciles every Veeam job, Rubrik SLA, snapshot schedule, and restore-test frequency. This is the entire game. Everything else in this lesson is implementation detail in service of this principle.

2. Veeam Backup & Replication automation

Veeam is the most common backup product in enterprise Windows/VMware environments and increasingly in physical Linux estates. The official collection is veeam.backup (Veeam Backup Enterprise Manager API wrapper).

2.1 Connection and authentication

# vars
veeam_em_host: "veeam-em.corp.example.com"
veeam_em_port: 9398
veeam_em_user: "ANSIBLE\\svc-veeam"
veeam_em_password: "{{ vault_veeam_em_password }}"

---
- name: Connect to Veeam EM
  veeam.backup.session:
    host: "{{ veeam_em_host }}"
    port: "{{ veeam_em_port }}"
    username: "{{ veeam_em_user }}"
    password: "{{ veeam_em_password }}"
    validate_certs: true
  register: veeam_session
  no_log: true

The collection caches the session token for the duration of the play, so you authenticate once.

2.2 Job-as-code

A Veeam backup job in code looks like this:

- name: Reconcile backup job for payments_api
  veeam.backup.backup_job:
    session_token: "{{ veeam_session.session_token }}"
    name: "BJ-payments_api-PROD"
    description: "Managed by Ansible — do not edit manually"
    job_type: vmware
    objects:
      - vm_name: "{{ item }}"
        host: "vcenter-dc1.corp.example.com"
    schedule:
      type: daily
      time: "22:00"
      timezone: "Europe/Berlin"
    repository: "veeam-repo-primary"
    retention:
      restore_points: "{{ backup_policy.retention.daily }}"
      gfs:
        weekly:
          enabled: true
          count: "{{ backup_policy.retention.weekly }}"
        monthly:
          enabled: true
          count: "{{ backup_policy.retention.monthly }}"
        yearly:
          enabled: true
          count: "{{ backup_policy.retention.yearly }}"
    options:
      backup_mode: incremental
      synthetic_full_enabled: true
      synthetic_full_days: ["saturday"]
      compression_level: 5
      storage_block_size: 1024
      encryption:
        enabled: true
        password_id: "{{ veeam_encryption_key_id }}"
    notifications:
      enabled: true
      email_to: "platform-backup-alerts@example.com"
      on_success: false
      on_warning: true
      on_failure: true
    state: present
  loop: "{{ groups['app_payments_api'] }}"
  register: veeam_job
  no_log: false

Two key disciplines visible here:

2.3 Backup copy job (the off-site & immutable copies)

The “1 off-site, 1 immutable” rules are implemented as Veeam Backup Copy Jobs (BCJ) that pull from the primary repo to a secondary and tertiary target:

- name: Backup copy job — primary → AWS S3 (off-site copy)
  veeam.backup.backup_copy_job:
    session_token: "{{ veeam_session.session_token }}"
    name: "BCJ-payments_api-OFFSITE"
    source_job: "BJ-payments_api-PROD"
    target_repository: "veeam-repo-aws-s3-frankfurt"
    schedule:
      type: continuous
    retention:
      restore_points: "{{ backup_policy.copies.offsite.retention_days }}"
    state: present

- name: Backup copy job — primary → S3 Object Lock (immutable copy)
  veeam.backup.backup_copy_job:
    session_token: "{{ veeam_session.session_token }}"
    name: "BCJ-payments_api-IMMUTABLE"
    source_job: "BJ-payments_api-PROD"
    target_repository: "veeam-repo-aws-s3-objectlock"
    schedule:
      type: daily
      time: "04:00"
    retention:
      restore_points: "{{ backup_policy.copies.immutable.retention_days }}"
      immutability_days: "{{ backup_policy.copies.immutable.object_lock_days }}"
    state: present

The object-lock repository must be configured with Compliance Mode (not Governance Mode) on the underlying S3 bucket. Compliance Mode is irrevocable: even an AWS root account holder cannot delete objects within their lock period. This is what makes the backup ransomware-resistant — even if an attacker compromises your AWS root credentials, they cannot delete the immutable backups.

2.4 Drift detection

Veeam admins, even well-meaning ones, will edit jobs in the GUI when they’re under pressure. The reconciliation playbook should run daily and detect drift:

- name: Check for unmanaged Veeam jobs (drift detection)
  veeam.backup.backup_job_info:
    session_token: "{{ veeam_session.session_token }}"
  register: all_jobs

- name: Find drifted or unmanaged jobs
  ansible.builtin.set_fact:
    drifted_jobs: >-
      {{ all_jobs.jobs
         | rejectattr('description', 'search', 'Managed by Ansible')
         | list }}

- name: Open INC if drifted jobs exist
  servicenow.itsm.incident:
    short_description: "Unmanaged Veeam backup jobs detected"
    description: |
      The following Veeam jobs are not managed by Ansible:
      {% for j in drifted_jobs %}
      - {{ j.name }} (last modified by {{ j.last_modified_by }})
      {% endfor %}
    impact: 2
    urgency: 2
  when: drifted_jobs | length > 0

Now any out-of-band Veeam change becomes a ServiceNow incident automatically. The first time you turn this on, you will discover dozens of orphan jobs, custom retention policies, and “temporary” exceptions that were never cleaned up. Cleaning these up is its own multi-week project, but you will be in a measurably better state at the end.

3. Rubrik automation

Rubrik takes a different philosophical approach: rather than per-VM job configuration, it uses SLA Domains — declarative policies attached to objects (VMs, databases, filesets) that the Rubrik cluster autonomously satisfies. This maps cleanly to Ansible’s declarative style.

The collection is rubrikinc.cdm.

3.1 SLA-as-code

- name: Define SLA domain for production tier-1
  rubrikinc.cdm.rubrik_sla_domain:
    name: "Tier-1-PROD"
    archive_settings:
      archival_threshold_unit: hours
      archival_threshold: 4
      archival_location_id: "{{ rubrik_archive_aws_id }}"
    replication_settings:
      target_cluster_id: "{{ rubrik_dr_cluster_id }}"
      retention_days: 30
    frequencies:
      hourly:
        frequency: 4
        retention: 72
      daily:
        frequency: 1
        retention: 14
      weekly:
        frequency: 1
        day_of_week: SATURDAY
        retention: 6
      monthly:
        frequency: 1
        day_of_month: LAST_DAY
        retention: 12
      yearly:
        frequency: 1
        day_of_year: LAST_DAY
        retention: 7
    state: present
  register: sla

3.2 Object → SLA assignment

- name: Assign SLA to all production VMs
  rubrikinc.cdm.rubrik_assign_sla:
    object_name: "{{ item.name }}"
    object_type: vmware
    sla_name: "Tier-1-PROD"
  loop: "{{ vmware_vm_inventory }}"
  when: item.tags | intersect(['env:prod', 'tier:1']) | length == 2

The pattern is read-the-source-of-truth (CMDB or vCenter tags) → derive SLA → apply. When a VM gets retagged in vCenter, the next reconciliation run reassigns it to the correct SLA. There is no “VM with no SLA” state possible by construction (you can have a default “uncategorised” SLA that catches anything missed and alerts loudly).

3.3 Rubrik recovery automation

The killer feature of Rubrik is its API-driven instant recovery — mounting a backup as a live VMDK in seconds. This is what makes automated restore drills (section 7) actually feasible:

- name: Live-mount latest snapshot for restore test
  rubrikinc.cdm.rubrik_vsphere_live_mount:
    vm_name: "{{ source_vm }}"
    mounted_vm_name: "{{ source_vm }}-restore-test-{{ ansible_date_time.epoch }}"
    snapshot_id: latest
    host: "{{ restore_test_esxi_host }}"
    datastore: "{{ restore_test_datastore }}"
    power_on: true
  register: live_mount

- name: Wait for VM to boot
  ansible.builtin.wait_for:
    host: "{{ live_mount.ip_address }}"
    port: 22
    timeout: 600
    delay: 30

A live-mount completes in ~30 seconds and consumes near-zero storage (it’s a CoW overlay on the backup). Restore tests that would have taken 2-4 hours of disk copy now take 5 minutes total. This unlocks frequent, automated restore validation.

4. Cohesity and Commvault — the “everything else” tier

If you are running Cohesity, the collection is cohesity.dataprotect. Concepts are similar to Rubrik (Protection Policies are the SLA equivalent):

- name: Define Cohesity protection policy
  cohesity.dataprotect.cohesity_protection_policy:
    name: "Tier-1-PROD"
    backup_schedule:
      unit: hours
      frequency: 4
    retention:
      unit: days
      duration: 14
    extended_retention:
      - schedule:
          unit: weeks
          frequency: 1
        retention:
          unit: weeks
          duration: 6
    archival_targets:
      - target_id: "{{ cohesity_aws_archive_id }}"
        unit: days
        duration: 30
    state: present

Commvault has the commvault.cv collection. The semantics are different — Commvault uses Storage Policies and Subclients — but the pattern is the same: define policies declaratively, attach to clients, reconcile daily.

The general lesson: regardless of vendor, everything maps to the same five concepts — schedule, retention, copies, encryption, target. The Ansible policy file looks identical; only the collection’s module names change. Teams who run multi-vendor backup estates can build a thin Ansible role that takes the unified backup_policy dictionary and dispatches to the right vendor module.

5. Storage-array snapshot automation (NetApp, Pure, Dell)

The snapshot tier of the 3-2-1-1-0 rule is satisfied by storage-array native snapshots, not by backup software. These are the fastest recovery option (seconds to mount, vs minutes for backup-software restore) and the cheapest (no separate compute or media). Every modern enterprise array has Ansible support.

5.1 NetApp ONTAP

netapp.ontap is one of the most mature collections. A typical pattern:

- name: Configure SnapMirror policy (replication to DR cluster)
  netapp.ontap.na_ontap_snapmirror_policy:
    state: present
    name: "MirrorAndVault-Daily"
    cluster: "ontap-cluster-prod"
    policy_type: mirror_vault
    snapmirror_label: ["daily", "weekly", "monthly"]
    keep: [14, 6, 12]
    schedule: ["daily", "weekly", "monthly"]
    hostname: "{{ ontap_mgmt_ip }}"
    username: "{{ ontap_user }}"
    password: "{{ vault_ontap_password }}"
    https: true
    validate_certs: true
  no_log: true

- name: Configure snapshot policy on volume
  netapp.ontap.na_ontap_volume:
    state: present
    name: "vol_payments_data"
    vserver: "svm-payments"
    snapshot_policy: "MirrorAndVault-Daily"
    snapshot_auto_delete:
      state: "on"
      delete_order: "oldest_first"
      defer_delete: "user_created"
      trigger: "volume"
      target_free_space: 20
    # SnapLock for immutable snapshots (compliance mode)
    snaplock:
      type: compliance
      autocommit_period: 4hours
      retention:
        default: 7years
        minimum: 30days
        maximum: 10years
    hostname: "{{ ontap_mgmt_ip }}"
    username: "{{ ontap_user }}"
    password: "{{ vault_ontap_password }}"
    https: true
    validate_certs: true
  no_log: true

The key feature here is SnapLock Compliance — once a snapshot is committed to SnapLock, it cannot be deleted by anyone, including ONTAP admins. This is the array-level equivalent of S3 Object Lock and serves the same role: ransomware can encrypt your live volume, but it cannot touch your locked snapshots.

5.2 Pure Storage

purestorage.flasharray for FlashArray and purestorage.flashblade for FlashBlade. The patterns are similar:

- name: Set protection group for volume
  purestorage.flasharray.purefa_pg:
    name: "pg_payments_prod"
    volume:
      - "vol_payments_data"
      - "vol_payments_logs"
    target:
      - "{{ purefa_dr_array_name }}"
    state: present
    fa_url: "{{ purefa_mgmt_ip }}"
    api_token: "{{ vault_purefa_api_token }}"
  no_log: true

- name: Set snapshot schedule
  purestorage.flasharray.purefa_pgsnap:
    name: "pg_payments_prod"
    schedule:
      enabled: true
      snap_at: "22:00"
      snap_frequency: 14400  # 4 hours in seconds
      snap_per_day: 6
      replicate_at: "23:00"
      replicate_frequency: 14400
      replicate_per_day: 6
      days: 14
      weeks: 6
      months: 12
    fa_url: "{{ purefa_mgmt_ip }}"
    api_token: "{{ vault_purefa_api_token }}"
  no_log: true

- name: Enable SafeMode (ransomware protection)
  purestorage.flasharray.purefa_pg:
    name: "pg_payments_prod"
    safe_mode: true
    fa_url: "{{ purefa_mgmt_ip }}"
    api_token: "{{ vault_purefa_api_token }}"
  no_log: true

Pure’s SafeMode is similar in spirit to ONTAP SnapLock: snapshots and protection groups become undeletable except via a Pure-side multi-party approval workflow. It is the mechanism by which Pure customers survived several high-profile 2023-2024 ransomware attacks where every other storage tier was encrypted.

5.3 Dell PowerStore / IBM Storage / others

dellemc.powerstore and ibm.storage_virtualize collections follow the same shape. The pattern is identical: define protection policies, attach to volumes, enable immutability, replicate to DR.

The discipline that matters across all of these: never allow a production volume to exist without a protection policy. A daily compliance check should query each array’s API for unprotected volumes and open an incident for each one. This is the fastest way to surface “someone provisioned a database last quarter and forgot to set up backups” — exactly the kind of gap that becomes catastrophic during ransomware.

6. Immutability strategies and the “1” in 3-2-1-1-0

There are four mechanisms for true immutability, in increasing order of strength:

Mechanism Strength Cost Use case
Object-lock S3 (Compliance Mode) High Low (cloud storage cost) Off-site copy, easy automation
Storage array SnapLock / SafeMode High Medium (array licensing) On-prem snapshot tier
Hardened Linux repository (Veeam) Medium Low If you don’t have S3
Tape (LTO with WORM cartridges) Highest High (operational complexity) Highly regulated industries (banks, defence, nuclear)

The recommendation for most organisations is two of the above: S3 Object Lock for the cloud copy, and either SnapLock or SafeMode for the on-prem snapshot copy. This gives you two independent immutability mechanisms with different threat models, dramatically reducing the chance that a single attacker can defeat both.

The classic mistake is “one immutable copy” — say, only S3 Object Lock. If an attacker gets your S3 access keys and learns to wait out the lock period, they can still cause damage. Two independent immutable copies, with different access mechanisms (one needs AWS IAM, the other needs ONTAP admin credentials), are exponentially harder to defeat.

A note on tape: tape is having a renaissance specifically because of ransomware. An LTO tape that has been ejected from the library and stored in a fireproof safe is, by definition, air-gapped — no network attacker can touch it. For compliance-driven industries the cost is justified. The Ansible automation here is minimal (you can’t reach an offline tape) but the workflow automation is rich: ServiceNow CHGs for tape-vaulting events, signed manifests of what’s on each tape, and quarterly retrieval drills that prove the chain of custody works.

7. The discipline that actually matters: automated restore drills

Backups that have never been restored are not backups. They are claims about backups. The 0-errors part of 3-2-1-1-0 is about converting claims into evidence.

The minimum viable restore-drill loop:

---
- name: Monthly automated restore drill
  hosts: localhost
  gather_facts: true
  vars:
    drill_id: "drill-{{ ansible_date_time.iso8601_basic_short }}"
    target_workloads: "{{ groups['env_prod'] | random(seed=ansible_date_time.epoch | int, count=(groups['env_prod'] | length * 0.05) | int) }}"
  tasks:

    - name: Open CHG for restore drill
      servicenow.itsm.change_request:
        type: standard
        template: "Standard - DR Restore Drill"
        short_description: "{{ drill_id }} restore drill ({{ target_workloads | length }} workloads)"
        state: implement
        start_date: "{{ ansible_date_time.iso8601 }}"
      register: drill_chg

    - name: Live-mount each target VM
      rubrikinc.cdm.rubrik_vsphere_live_mount:
        vm_name: "{{ item }}"
        mounted_vm_name: "{{ item }}-{{ drill_id }}"
        snapshot_id: latest
        host: "{{ restore_test_esxi_host }}"
        datastore: "{{ restore_test_datastore }}"
        power_on: true
      loop: "{{ target_workloads }}"
      register: mounted_vms

    - name: Wait for each VM to boot and respond to SSH/WinRM
      ansible.builtin.wait_for:
        host: "{{ item.ip_address }}"
        port: 22
        timeout: 900
      loop: "{{ mounted_vms.results }}"
      loop_control:
        label: "{{ item.item }}"

    - name: Run application-level smoke tests
      ansible.builtin.include_role:
        name: kv.app_smoke_tests
      vars:
        target_host: "{{ item.ip_address }}"
        original_vm_name: "{{ item.item }}"
      loop: "{{ mounted_vms.results }}"
      register: smoke_test_results

    - name: Run filesystem integrity checks
      ansible.builtin.shell: |
        find / -type f \( -name "*.db" -o -name "*.dat" -o -name "*.idx" \) \
          -exec md5sum {} \; > /tmp/restore-checksums.txt
      delegate_to: "{{ item.ip_address }}"
      loop: "{{ mounted_vms.results }}"
      changed_when: false

    - name: Compare checksums against pre-backup baseline
      ansible.builtin.include_role:
        name: kv.checksum_compare
      vars:
        baseline_path: "/var/lib/backup-baselines/{{ item.item }}.md5"
        actual_path: "/tmp/restore-checksums.txt"
      loop: "{{ mounted_vms.results }}"
      register: checksum_results

    - name: Generate drill report
      ansible.builtin.template:
        src: drill_report.html.j2
        dest: "/var/lib/backup-drills/{{ drill_id }}/report.html"
      vars:
        smoke_results: "{{ smoke_test_results.results }}"
        checksum_results: "{{ checksum_results.results }}"
        chg_number: "{{ drill_chg.record.number }}"

    - name: Sign and archive drill evidence
      ansible.builtin.shell: |
        tar -czf {{ drill_id }}.tar.gz {{ drill_id }}/
        gpg --batch --yes --detach-sign --armor --local-user backup-drill-signer \
          {{ drill_id }}.tar.gz
        aws s3 cp {{ drill_id }}.tar.gz s3://kv-evidence-immutable/restore-drills/
        aws s3 cp {{ drill_id }}.tar.gz.asc s3://kv-evidence-immutable/restore-drills/
      args:
        chdir: /var/lib/backup-drills/

    - name: Tear down restore-test VMs
      rubrikinc.cdm.rubrik_vsphere_live_unmount:
        mounted_vm_name: "{{ item.item }}-{{ drill_id }}"
      loop: "{{ mounted_vms.results }}"

    - name: Close CHG with results
      servicenow.itsm.change_request:
        number: "{{ drill_chg.record.number }}"
        state: closed
        close_code: "{{ 'successful' if all_passed else 'unsuccessful' }}"
        close_notes: |
          Restore drill {{ drill_id }} complete.
          Workloads tested: {{ target_workloads | length }}
          Passed: {{ smoke_test_results.results | selectattr('failed', 'equalto', false) | list | length }}
          Failed: {{ smoke_test_results.results | selectattr('failed', 'equalto', true) | list | length }}
          Evidence: s3://kv-evidence-immutable/restore-drills/{{ drill_id }}.tar.gz
      vars:
        all_passed: "{{ smoke_test_results.results | rejectattr('failed', 'equalto', false) | list | length == 0 }}"

The pattern is:

The bar for success is 100% of the sampled workloads must pass. Anything less, and you have a real backup integrity problem to investigate.

7.1 The metrics that matter

Three numbers that should be on a dashboard reviewed in every operations standup:

The first metric is the only one that proves backups work. The second proves they’re operationally useful (a backup that takes 8 hours to mount is useless for most outage scenarios). The third proves they’re complete.

8. Backup chain integrity and synthetic full validation

A specific failure mode worth calling out: silent corruption of the backup chain. Backup software performs incrementals against a previous full or synthetic full. If any link in the chain is corrupt, every subsequent restore from that chain is corrupt. You will not notice until you try to restore.

The mitigations:

Encoded as Ansible:

- name: Trigger health check on backup repository
  veeam.backup.repository_health_check:
    session_token: "{{ veeam_session.session_token }}"
    repository: "veeam-repo-primary"
    full_scan: false
  register: health_check

- name: Wait for health check to complete
  veeam.backup.repository_health_check_info:
    session_token: "{{ veeam_session.session_token }}"
    job_id: "{{ health_check.job_id }}"
  register: health_status
  until: health_status.state in ['Stopped', 'Failed']
  retries: 360
  delay: 60

- name: Open INC if health check found issues
  servicenow.itsm.incident:
    short_description: "Veeam repository health check found issues"
    description: |
      Repository: veeam-repo-primary
      Issues: {{ health_status.issues_found }}
      Details: {{ health_status.details }}
    impact: 1
    urgency: 1
  when: health_status.issues_found > 0

Going deeper: internals, idempotency, and the things that bite at scale

The sections above give you the shape of backup automation. This one is for the engineer who has to make it survive contact with a real, multi-thousand-workload estate — where vendor APIs are slow, modules lie about idempotency, and one careless drill can fill a production array.

The honest-idempotency problem

Most vendor backup modules are thin wrappers over a REST API that does a full PUT of the job or policy on every run. The naive result is a reconcile play that reports changed on every execution, even when nothing actually changed. That is worse than useless: it trains your team to ignore the one signal — drift — that the whole daily reconcile exists to produce. When everything is always yellow, nobody notices the day it’s yellow for a real reason.

The fix is to make changed_when tell the truth. Read the live state first, compare it to the desired state, and only claim a change when they genuinely differ:

- name: Read the live job definition (never reports changed)
  veeam.backup.backup_job_info:
    session_token: "{{ veeam_session.session_token }}"
    name: "BJ-payments_api-PROD"
  register: current
  changed_when: false

- name: Reconcile the job only when desired state actually differs
  veeam.backup.backup_job:
    session_token: "{{ veeam_session.session_token }}"
    name: "BJ-payments_api-PROD"
    description: "Managed by Ansible — do not edit manually"
    # ... full desired spec, as in section 2.2 ...
    retention:
      restore_points: "{{ backup_policy.retention.daily }}"
    state: present
  check_mode: "{{ reconcile_dry_run | default(false) | bool }}"
  register: reconcile
  changed_when: reconcile.diff.before | default({}) != reconcile.diff.after | default({})

Run this with --check --diff in CI on every pull request to your policy repo. The diff is your change-review artefact: a reviewer sees exactly which restore points, schedules or targets a merge will alter, before it touches production. This is the single biggest lever for trust — a reconcile that is quiet on a no-op day and loud, with a readable diff, on a real change.

When there is no collection: the ansible.builtin.uri pattern

Not every product has a maintained collection. Veritas NetBackup, older Commvault releases, niche storage arrays and in-house tools often expose only a REST API. The “Ansible as the backup orchestrator” pattern still holds — you just drive the API directly. The canonical shape is authenticate → start job → poll to a terminal state → assert clean:

# Generic "drive any backup REST API" pattern — for products with no vendor collection
- name: Authenticate to the backup API
  ansible.builtin.uri:
    url: "https://{{ backup_api_host }}/api/v1/login"
    method: POST
    body_format: json
    body:
      username: "{{ backup_api_user }}"
      password: "{{ vault_backup_api_password }}"
    status_code: 200
    validate_certs: true
  register: login
  no_log: true

- name: Start the backup job
  ansible.builtin.uri:
    url: "https://{{ backup_api_host }}/api/v1/jobs"
    method: POST
    headers:
      Authorization: "Bearer {{ login.json.token }}"
    body_format: json
    body:
      policy: "Tier-1-PROD"
      client: "{{ inventory_hostname }}"
    status_code: 201
    validate_certs: true
  register: job_start
  changed_when: job_start.status == 201
  no_log: true

- name: Poll until the job reaches a terminal state
  ansible.builtin.uri:
    url: "https://{{ backup_api_host }}/api/v1/jobs/{{ job_start.json.job_id }}"
    method: GET
    headers:
      Authorization: "Bearer {{ login.json.token }}"
    status_code: 200
    validate_certs: true
  register: job_status
  until: job_status.json.state in ['completed', 'failed']
  retries: 120
  delay: 30
  changed_when: false
  no_log: true

- name: Fail the play if the backup did not complete cleanly
  ansible.builtin.assert:
    that:
      - job_status.json.state == 'completed'
      - job_status.json.errors | default(0) | int == 0
    fail_msg: >-
      Backup {{ job_start.json.job_id }} ended in {{ job_status.json.state }}
      with {{ job_status.json.errors | default('?') }} errors

Three details make this production-grade rather than a demo: until/retries/delay turns a long-running async job into a blocking task without a fragile sleep; changed_when: false on the poll stops the status check from polluting your change count; and the final assert converts “the API returned 200” into “the backup genuinely completed with zero errors” — the exact distinction this whole lesson is about.

Long-running jobs: async, not timeouts

A full backup or a restore drill can run for hours. Two mechanisms keep Ansible from blocking a fork the whole time or dying on a dropped connection:

Either way, never model a long backup as a plain synchronous task on a connection that might drop — the control node’s SSH connection or an API gateway’s idle-timeout will kill it mid-run and hand you a false failure.

Credentials at scale

The examples use {{ vault_* }} variables from Ansible Vault, which is correct for a small estate. At scale you usually want secrets to live in a dedicated manager — CyberArk, HashiCorp Vault, cloud KMS — and be fetched at run time with a lookup (e.g. community.hashi_vault.vault_read) so nothing sensitive is ever written to the repo at all. Two disciplines regardless of source:

Immutability internals you must get right

“Immutable” is not one thing. For S3 Object Lock there are two retention modes, and the difference is the whole ballgame:

Mode Who can shorten/delete within the lock window Use for
Governance Anyone holding the s3:BypassGovernanceRetention IAM permission Dev/test, “soft” immutability
Compliance Nobody — not the account root, not AWS support Real ransomware defence

Only Compliance mode is genuine immutability. The operational tax is real and you must budget for it: within the lock window you cannot delete anything, including a backup that is corrupt, oversized, or written by mistake. Set object_lock_days too high and you pay to store garbage you can’t remove; too low and you hand a patient attacker a window in which the copy becomes deletable again. The array-side equivalents — ONTAP SnapLock Compliance (governed by a tamper-proof ComplianceClock) and Pure SafeMode (deletion requires a Pure-side multi-party approval) — carry the same trade-off. This is exactly why the lesson insists on two independent mechanisms: they have different failure modes and different credentials, so no single mistake or compromise defeats both.

Restore-drill internals: why live-mount is cheap, and how it bites

A Rubrik/Veeam live-mount is fast and near-free because it doesn’t copy anything — it presents the backup as a virtual disk and captures all writes in a redirect-on-write overlay. That’s what turns a 3-hour disk restore into a 30-second mount and makes monthly drills feasible. But the overlay lives somewhere, and the reads hit some array:

Scaling the reconcile across thousands of workloads

Vendor APIs have concurrency limits; hammer them with forks: 200 and you’ll get throttled or rate-limited errors that look like failures. The levers:

Version and collection caveats

Collections lag the products they wrap, and some are community-maintained. Pin versions in requirements.yml and read the changelog before upgrading. Be aware of moving targets: rubrikinc.cdm targets the older CDM appliance API, while much of Rubrik’s newer surface lives in Rubrik Security Cloud (Polaris); the veeam.backup collection wraps Enterprise Manager, whereas Veeam’s strategic API since v12 is the native REST API you may prefer to drive with ansible.builtin.uri. The module names in this lesson are representative of the pattern, not a promise about a specific collection version — always confirm arguments against the current docs (ansible-doc <fqcn>) before you ship.

9. The economic dimension — and why it matters for adoption

A common political problem with backup automation: backup admins feel threatened by it, because it makes their job legible to engineering leadership. “We’ve always been a 4-FTE team because backups are complicated” is harder to defend when an Ansible role can spin up correct backup configuration for a new application in 30 seconds.

The way to land this politically is to be very clear that automation makes backup admins more valuable, not less. The work shifts from:

to:

In every successful rollout I’ve seen, the backup admin team becomes the trusted advisor function — they review proposed policy changes, approve PRs to the policy repo, and own the relationship with vendors. The grunt work of clicking through GUIs disappears, but their organisational standing increases. Frame it that way from day one and adoption is much easier.

10. Common failure modes (and how to spot them)

Failure mode What it looks like How to detect
Backup ran but contains no data Job succeeds, restored VM is empty Restore drill checksums mismatch
Encryption key lost Backup exists but cannot be decrypted Quarterly key-recovery drill
Object-lock retention misconfigured Auditor finds 30-day retention where 90 was required Daily policy reconciliation; alert on drift
Replication broken silently Off-site copy is N days behind last_successful_run query on every BCJ; alert if > 24h
Test snapshots fill the array Live-mount drills accumulate, eat capacity Always tear down mounted VMs in a block:/always: pattern
Backup admin GUI changes Policies drift from declared state Daily reconciliation play with diff alerts
Tape unreadable Drive degraded or tape damaged Quarterly tape retrieval drill
Ransomware deletes online backups, immutables save you But you didn’t test recovery from immutables Quarterly drill specifically restores from immutable copy
New application onboarded without backup “We didn’t know we needed it” Daily compliance scan: any production CI without protection_policy → P1 incident
Service account credentials rotated All backup jobs fail at 03:00 Backup-system credential rotation must be a coordinated change with explicit verification

The single most valuable habit is the quarterly chaos drill — pick one production application at random, simulate “your backup is the only thing left, recover the application from scratch in a clean environment, time it.” That exercise will surface every weakness in your backup automation faster than any audit.

11. Where this fits in the broader course

The compliance lesson (D1) gave you the paper-trail discipline. The DR lesson (D2) gave you the worst-case-day response. The migrations lesson (D3) gave you the bulk fleet change skills. The air-gap lesson (D4) gave you the isolated environment patterns. The SAP lesson (D5) gave you the complex-stack integration. The edge lesson (D6) gave you the device fleet primitives. The ITSM lesson (D7) gave you the governance fabric.

This lesson gives you the last-line-of-defence — the system that has to keep working when everything else has failed. It is also the system most often neglected, most often used, and most often inadequate when actually called upon.

The remaining lessons cover the database-migration patterns (D9 — online migrations, blue-green database swaps, zero-downtime cutovers) and the observability capstone (D10) that ties metrics, logs, traces, and AAP events into a coherent operational dashboard. Those final two lessons close the loop on the entire Tier 5 curriculum: from what to automate to how to be sure it’s running well.

What you should walk away with is the conviction that backups are not an IT operations side-quest. They are the foundational control that everything else depends on, and they earn that role only when they are tested, declarative, immutable, and reconciled. Anything less is a backup that exists in the same way that a fire extinguisher with an empty CO₂ canister exists — present, mounted on the wall, completely useless when needed.

Practice challenges

Work these in order; they escalate from “read the policy” to “enforce it in CI”. Each has a worked solution — try it before you open it.

Challenge 1 — One policy, one edit (beginner)

Your auditor now requires 14 monthly restore points instead of 12. Using the backup_policy dictionary from section 1, how many files and lines do you change, and which one?

<details><summary>Solution</summary>

Change one line in group_vars/all/backup_policy.yml:

retention:
  monthly: 14   # was 12

Every Veeam job, Rubrik SLA and snapshot schedule reads backup_policy.retention.monthly, so the next reconcile propagates it everywhere.

Why: a single source of truth means policy changes are one-line diffs a reviewer can approve in seconds — the entire point of policy-as-code. </details>

Challenge 2 — Stop a read task crying “changed” (beginner)

A colleague’s drift-check play reports changed on every run just from reading the current jobs. The task is a veeam.backup.backup_job_info call. Fix it.

<details><summary>Solution</summary>

- name: List all backup jobs
  veeam.backup.backup_job_info:
    session_token: "{{ veeam_session.session_token }}"
  register: all_jobs
  changed_when: false

Why: *_info / query tasks never alter state, so they must declare changed_when: false. A green “changed” on a read is noise that erodes trust in the real change signal. </details>

Challenge 3 — Drive a job through a raw REST API (intermediate)

Your NetBackup estate has no Ansible collection. Write the two tasks that (a) authenticate and (b) start a backup job for {{ inventory_hostname }} under policy Tier-1-PROD, keeping the password out of the logs.

<details><summary>Solution</summary>

- name: Authenticate
  ansible.builtin.uri:
    url: "https://{{ nbu_host }}/netbackup/login"
    method: POST
    body_format: json
    body:
      userName: "{{ nbu_user }}"
      password: "{{ vault_nbu_password }}"
    status_code: 201
    validate_certs: true
  register: login
  no_log: true

- name: Start the backup
  ansible.builtin.uri:
    url: "https://{{ nbu_host }}/netbackup/admin/manual-backup"
    method: POST
    headers:
      Authorization: "{{ login.json.token }}"
    body_format: json
    body:
      policyName: "Tier-1-PROD"
      clientName: "{{ inventory_hostname }}"
    status_code: 202
    validate_certs: true
  register: job
  changed_when: job.status == 202
  no_log: true

Why: ansible.builtin.uri + no_log: true + an explicit status_code lets you orchestrate any backup product’s REST API with the same discipline as a first-party collection. (Exact paths/codes are representative — check your NetBackup API version.) </details>

Challenge 4 — Find the jobs nobody manages (intermediate)

From all_jobs.jobs (each item has a .description), build the list of jobs whose description does not contain Managed by Ansible.

<details><summary>Solution</summary>

- name: Identify unmanaged / drifted jobs
  ansible.builtin.set_fact:
    drifted_jobs: "{{ all_jobs.jobs | rejectattr('description', 'search', 'Managed by Ansible') | list }}"

Why: rejectattr(..., 'search', ...) filters out the items whose description matches the regex, leaving exactly the out-of-band jobs — the ones that should become a ServiceNow incident. </details>

Challenge 5 — Make drill teardown bulletproof (advanced)

Your restore drill live-mounts VMs, smoke-tests them, then unmounts. Right now a failed smoke test aborts the play and leaves the test VMs mounted, slowly filling the array. Restructure so teardown always runs.

<details><summary>Solution</summary>

- name: Restore drill for {{ item }}
  block:
    - name: Live-mount
      rubrikinc.cdm.rubrik_vsphere_live_mount:
        vm_name: "{{ item }}"
        mounted_vm_name: "{{ item }}-{{ drill_id }}"
        snapshot_id: latest
        host: "{{ restore_test_esxi_host }}"
        datastore: "{{ restore_test_datastore }}"
        power_on: true
      register: mount

    - name: Smoke test (may fail)
      ansible.builtin.include_role:
        name: kv.app_smoke_tests
      vars:
        target_host: "{{ mount.ip_address }}"
  rescue:
    - name: Record the failure for the drill report
      ansible.builtin.set_fact:
        drill_failures: "{{ drill_failures | default([]) + [item] }}"
  always:
    - name: Always unmount the test VM
      rubrikinc.cdm.rubrik_vsphere_live_unmount:
        mounted_vm_name: "{{ item }}-{{ drill_id }}"

Why: tasks in always: run whether the block succeeded or the rescue caught a failure, so a leaked live-mount can never survive a failed test. Recording the failure in rescue keeps the drill honest instead of hiding it. </details>

Challenge 6 — Enforce “no prod volume without protection” in CI (advanced)

Every production volume must belong to a protection policy. Given array_volumes (a list of volume names) and protected_volumes (names that already have a policy), fail a CI gate — and open a P1 — for any gap.

<details><summary>Solution</summary>

- name: Compute unprotected production volumes
  ansible.builtin.set_fact:
    unprotected: "{{ array_volumes | difference(protected_volumes) }}"

- name: Open a P1 for each unprotected volume
  servicenow.itsm.incident:
    short_description: "Unprotected production volume: {{ item }}"
    impact: 1
    urgency: 1
  loop: "{{ unprotected }}"
  when: unprotected | length > 0

- name: Fail the pipeline so the gap can't merge
  ansible.builtin.assert:
    that: unprotected | length == 0
    fail_msg: "Volumes with no protection policy: {{ unprotected | join(', ') }}"

Why: the difference filter is the whole check; the assert turns a silent gap into a red pipeline, so “someone provisioned a database and forgot backups” is caught at merge time, not during a ransomware incident. </details>

Common beginner mistakes

These are conceptual traps — the wrong mental model — as opposed to the operational symptom→detect table in section 10.

Glossary

ansiblebackupveeamrubrikcohesitycommvaultnetappontappure-storagedell-powerstoreobject-lockimmutabilityrestore-testingransomware
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me
VH

Written by Vinod

Senior Cloud Architect with 22+ years of experience. Sharing practical knowledge from production environments.

Comments