Ansible × Observability Capstone, In Depth: Prometheus, Grafana, Loki, OpenTelemetry, AAP Metrics & the Closed Automation Feedback Loop

In a nutshell

Think of your automation estate as a professional kitchen during a full dinner service. There are two completely different jobs going on, and this lesson is about both.

The first job is making the kitchen fast. The head chef adds more line cooks (that’s Ansible forks — running more hosts in parallel), preps ingredients once and reuses them instead of re-chopping for every plate (fact caching), keeps the pass window propped open instead of walking to the walk-in fridge on every trip (SSH ControlPersist, one connection reused all night), lets stations that don’t depend on each other race ahead (the free strategy), and puts the slow-braising dish on a timer so the whole line isn’t held hostage waiting for it (async + poll). Before any of that, the chef stands at the pass with a stopwatch and times every station to find the slow one — that stopwatch is a profiling callback like profile_tasks.

The second job is proving the kitchen stays fast across a thousand covers a night. That’s the expediter’s ticket rail and the cameras: every ticket timed, every station’s throughput on a board, an alarm the instant the fryer backs up — so the moment service degrades you already know which station and why. In our world that rail is Prometheus, Loki, Tempo and Grafana, fed by an OpenTelemetry callback, plus the AAP control-plane metrics, all closing into a feedback loop where an alert can trigger the remediation playbook that fixes it.

So: first you make automation fast (performance), then you make it observable (the boards that prove it). A beginner should care because a playbook that is slow or silent doesn’t scale — at ten hosts you can eyeball it, at ten thousand you are blind without both halves. This lesson gives you the stopwatch, the tuning levers, and the full observability fabric, and shows how they connect.

Level: Advanced (with a beginner on-ramp) · Time: ~55 min · You’ll be able to: profile a slow playbook and read the summary; pick the right tuning lever (forks, pipelining, fact caching, ControlPersist, strategy) for a given bottleneck; run long tasks with async/poll without blocking the fleet; instrument runs into Prometheus/OpenTelemetry; and design the closed alert→remediation→resolve loop.

Prerequisites: you should be comfortable writing plays and roles, running playbooks against an inventory, and editing ansible.cfg. If forks, pipelining and fact caching are new, skim the dedicated Performance tuning deep-dive first; if strategies and serial are new, see Strategies, serial & rolling updates. This lesson pulls those levers together and wires the whole run into an observability stack.

---

Performance first: you can’t observe what you never profiled

Observability tells you what is slow across a fleet of thousands. Performance tuning is how you fix it. They are two ends of the same discipline, and the connective tissue is the humble callback plugin: the same event stream that a profiling callback prints as a local summary is what an OpenTelemetry callback ships to Tempo as distributed traces. Learn to profile a single run on your laptop, and instrumenting ten thousand runs in production becomes the same idea at a different scale.

The workflow, always in this order:

1. Measure — turn on a profiling callback and find the slow task or role. Never tune by guessing.
2. Tune the biggest lever first — usually fact gathering or connection overhead, not the module itself.
3. Re-measure — confirm the number moved. Tuning without a before/after is superstition.
4. Instrument for production — once it’s fast, ship the same event stream to Prometheus/OTel so it stays fast across the fleet.

Profiling a run: profile_tasks, profile_roles, and timer

Ansible ships three profiling callbacks in the ansible.posix collection. They are aggregate callbacks — they ride alongside your normal output and print a summary at the end, so you can leave them on in dev with almost no downside:

Enable them in ansible.cfg (or via the ANSIBLE_CALLBACKS_ENABLED environment variable, which is what you set inside an AAP execution environment):

# ansible.cfg
[defaults]
callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.profile_roles, ansible.posix.timer

# Equivalent for an execution environment / one-off run
export ANSIBLE_CALLBACKS_ENABLED=ansible.posix.profile_tasks,ansible.posix.timer
# Tune what profile_tasks prints:
export PROFILE_TASKS_SORT_ORDER=descending     # slowest first (default)
export PROFILE_TASKS_TASK_OUTPUT_LIMIT=25       # show the top N tasks

A representative profile_tasks summary at the end of a run (output is representative, not from a live run here):

Sunday 19 July 2026  10:15:49 +0000 (0:00:00.048)       0:02:31.007 ***********
===============================================================================
Gathering Facts -------------------------------------------------------- 42.13s
geerlingguy.nginx : Ensure nginx is installed -------------------------- 28.02s
Deploy vhost templates ------------------------------------------------- 12.44s
Install base packages -------------------------------------------------- 09.88s
Wait for service to come up -------------------------------------------- 06.10s
Restart nginx ----------------------------------------------------------- 02.31s

How to read it: the right-hand number is each task’s duration; the header’s 0:02:31.007 is total run time. The lesson jumps out immediately — Gathering Facts is the single biggest cost at 42s. That is the number-one finding in almost every real playbook, and it is entirely avoidable with fact caching or by not gathering facts you don’t use. Tune the top of this list, re-run, and watch it fall — that before/after discipline is the whole game.

The timer callback simply appends the bottom line:

Playbook run took 0 days, 0 hours, 2 minutes, 31 seconds

Profiling callbacks are the local form of everything in the second half of this lesson. When you enable ansible.posix.opentelemetry instead, the exact same per-task timing becomes span durations in Tempo. (For the full menu of callback types, see Callback, connection & other plugins.)

The tuning levers, at a glance

Once profiling names the bottleneck, reach for the matching lever. Almost every Ansible slowdown is one of two things: too much fact gathering or too much SSH connection overhead. The levers:

A well-tuned ansible.cfg that combines the safe, universally-applicable levers:

# ansible.cfg — a fast, sane baseline
[defaults]
forks = 50
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /var/tmp/ansible_fact_cache
fact_caching_timeout = 7200          # seconds; re-gather after 2h

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o PreferredAuthentications=publickey
control_path_dir = ~/.ansible/cp

And in the plays themselves, gather only what you use:

- name: Configure web tier (facts scoped to what we actually reference)
  hosts: webservers
  gather_facts: true
  gather_subset:
    - '!all'
    - '!min'
    - network          # we only need ansible_default_ipv4, etc.
  tasks:
    - name: Template the vhost using a fact we actually gathered
      ansible.builtin.template:
        src: vhost.conf.j2
        dest: /etc/nginx/conf.d/app.conf
      notify: reload nginx

gathering has three modes worth knowing cold: implicit (the old default — gather every play unless gather_facts: false), explicit (never auto-gather; you must ask), and smart (gather once per host per run, then reuse — the sweet spot with a fact cache). Set gathering = smart plus a cache and the 42-second “Gathering Facts” line from our profile becomes a near-zero cache hit on every subsequent run.

On mitogen: the Mitogen-for-Ansible strategy plugin reuses Python interpreters and connections in memory and can deliver 1.5–7× speedups. It is a third-party, unofficial plugin: it is pinned to specific ansible-core versions, it can break with certain connection/become/strategy plugins, and Red Hat does not support it in AAP execution environments. Evaluate it for standalone ansible-core fleets, pin the exact versions, and never assume it in a supported-platform design.

The full mechanics of each lever (including benchmarks and the pipelining/requiretty interaction) live in the dedicated Performance tuning deep-dive. Here the point is the decision: profile → identify the lever → apply → re-measure.

Execution strategies: linear, free, host_pinned

A strategy plugin decides how Ansible steps hosts through the tasks of a play. There are three you should know:

Set it per play:

- name: Independent per-host bootstrap (order across hosts is irrelevant)
  hosts: all
  strategy: free          # or ansible.builtin.free (FQCN form)
  tasks:
    - name: Ensure baseline packages
      ansible.builtin.package:
        name: ["chrony", "rsync", "curl"]
        state: present

- name: Ordered rolling deploy (25% of the fleet at a time, lock-step)
  hosts: webservers
  strategy: linear
  serial: 25%             # batching is orthogonal to strategy
  tasks:
    - name: Deploy new release
      ansible.builtin.unarchive:
        src: "app-{{ release }}.tar.gz"
        dest: /opt/app

serial (batch size) is independent of strategy — you can batch a linear rolling update or a host_pinned fleet sweep. Strategies and serial/rolling patterns are covered end-to-end in Strategies, serial & rolling updates.

async and poll: long tasks that don’t hold the fleet hostage

Some tasks legitimately take a long time — a database import, a firmware flash, a full-system dnf upgrade. Under the default synchronous model, Ansible holds the SSH connection open and blocks the whole play on the slowest host. async breaks that: the task is launched on the remote, and Ansible either polls it periodically or fires-and-forgets.

Fire-and-forget (poll: 0) — launch, do other work, then explicitly wait with ansible.builtin.async_status:

tasks:
  - name: Kick off a long dataset import (don't block)
    ansible.builtin.command: /opt/scripts/import_dataset.sh
    async: 3600            # allow up to 1 hour of runtime
    poll: 0                # fire-and-forget: return immediately
    register: import_job

  - name: Do useful work while the import runs
    ansible.builtin.command: /opt/scripts/warm_cache.sh

  - name: Now wait for the import to finish
    ansible.builtin.async_status:
      jid: "{{ import_job.ansible_job_id }}"
    register: import_result
    until: import_result.finished
    retries: 60
    delay: 60             # poll every 60s, up to 60 times (1h ceiling)

Polled async (poll > 0) — put a hard time ceiling on a task that would otherwise hang forever, without writing the async_status loop yourself:

  - name: Patch everything, but cap it at 30 minutes
    ansible.builtin.dnf:
      name: '*'
      state: latest
    async: 1800           # max runtime: 30 min
    poll: 15              # Ansible checks every 15s and fails if it overruns

Two rules that trip people up: poll: 0 means launched, not finished — you must reconcile it later with async_status, or you have a fleet of orphaned background jobs you never checked. And async needs a connection that can leave a temp job file behind, so it does not work with the raw connection or with a task like ansible.builtin.reboot that kills the connection out from under it.

Scaling to thousands of hosts

At fleet scale the levers compound, and a few extra patterns come into play:

• Raise forks, but not blindly. Each fork is a full worker process on the control node. 500 forks can OOM a small controller. Raise forks until the control node’s CPU/RAM (not the network) becomes the limit, then scale horizontally instead.
• Cache facts centrally. With redis or memcached fact caching shared across control nodes, gathering = smart means you gather each host’s facts once and every subsequent play across the fleet reads from cache.
• Reuse connections. ControlPersist plus pipelining cut per-task SSH cost dramatically — the single biggest win when a play has many small tasks across many hosts.
• Batch with serial / throttle. Roll changes across the fleet in waves so a bad change hits 5% before 100%, and use throttle: N to cap concurrency on a task that hammers a shared backend.
• Slice the job at the platform layer. In AAP, a sliced job splits one job template across many execution nodes, each handling a fraction of the inventory in parallel — horizontal scale beyond a single control node’s forks.
• Consider pull mode at the extreme edge. Ten thousand intermittently-connected edge nodes are often better served by ansible-pull (each node pulls and applies its own config on a schedule) than by pushing from a central controller.

Every one of these tunes throughput. The second half of this lesson is how you prove, across all those hosts, that throughput and correctness are actually holding.

---

Observability: knowing whether the fleet is healthy

You have made the fleet fast. Now the harder question: across 50,000 runs a quarter, how do you know it is still fast and still correct? That is the observability half — the ticket rail and cameras over the whole kitchen. Everything below builds on the same callback-plugin event stream you just used for profiling; we now ship it to Prometheus, Loki, Tempo and Grafana instead of printing a local summary.

This is the final lesson in the Tier 5 wave, and it deliberately closes the loop. Across nine deep-dives we’ve built up an automation platform that is compliant (D1), disaster-resilient (D2), capable of bulk migrations (D3), able to operate in air-gapped enclaves (D4), integrated with complex stacks like SAP (D5), capable of fleet operation at edge scale (D6), governed by ITSM (D7), backed up by tested immutable backups (D8), and able to migrate databases without downtime (D9). All of that is meaningless without one final ingredient: the system has to know whether it is healthy.

A platform that runs correctly 99.97% of the time but cannot tell you which 0.03% failed is not a platform — it is a black box that occasionally surprises everyone. The thesis of this lesson is that automation observability is not a “nice to have” added later; it is the foundation that makes everything else trustworthy at scale. When your CHG-gated, evidence-bundled, ServiceNow-tracked, SLA-verified automation has 50,000 runs per quarter, the only way to know it is working is metrics, logs, and traces that aggregate into a single view answerable in seconds.

The four pillars of automation observability:

1. Metrics — counters, gauges, histograms about playbook runs, AAP control plane, hosts, and ITSM/CHG flow
2. Logs — structured stdout/stderr from every play, every task, indexed and queryable
3. Traces — distributed traces through multi-step orchestrations (workflow → job → host → task)
4. Events — discrete state-change events (CHG opened, job launched, EDA rule fired) correlated with the above

The toolchain we will assemble:

This is the canonical CNCF observability stack, with the caveat that you can substitute Datadog, New Relic, Splunk, or Elastic at the storage layer without changing the patterns in this lesson. The instrumentation contract (what to emit) is the durable part; the storage choice is replaceable.

---

1. The four golden signals, applied to automation

Google’s SRE book defines four golden signals for any service: latency, traffic, errors, saturation. Translated to an automation platform:

These four signals at the platform level give you the macro view. But automation has a fifth signal that conventional services don’t: convergence. Did the automation actually achieve its desired state, or did it merely “complete”?

A playbook that “succeeds” but fails to change anything because it was misconfigured is a successful run that produced a wrong outcome. Convergence means measuring not just job.status == 'successful' but job.changed_count > 0 AND desired_state == observed_state after the run. We’ll wire this in.

---

2. The Ansible callback plugin: the foundation of instrumentation

Every metric, log, and trace in this lesson originates from the same place: a callback plugin that fires on every Ansible event. Red Hat ships an official OpenTelemetry callback plugin in ansible.posix:

# ansible.cfg or AAP execution environment env
[defaults]
callbacks_enabled = ansible.posix.opentelemetry, ansible.posix.profile_tasks

[callback_opentelemetry]
otel_service_name = ansible-aap
enable_from_environment = OTEL_EXPORTER_OTLP_ENDPOINT
hide_task_arguments = true

# Environment variables (set in execution environment)
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.kv.local:4318
OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer <token>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
OTEL_SERVICE_NAME=ansible-aap

When this is enabled, every Ansible run produces a complete OpenTelemetry trace with this hierarchy:

playbook span (root, named after the playbook)
  └── play span (one per play)
       └── task span (one per task per host)
            ├── attributes: ansible.task.module=template, host=foo, status=ok/changed/failed
            └── events: stderr lines as span events

A nginx_install.yml playbook with 12 tasks running across 8 hosts will produce roughly 1 + 1 + 12*8 = 98 spans, all linked. In Tempo this is queryable as “show me all task failures for nginx_install in the last 7 days,” and you get back exact module name, host, task, exception text, and parent context.

The tradeoff is volume. A workflow that orchestrates 200 playbooks across 5,000 hosts produces hundreds of thousands of spans per run. Sample aggressively in production:

# OTel collector config
processors:
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: errors-always
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces-always
        type: latency
        latency: { threshold_ms: 30000 }
      - name: sample-others
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

This keeps every error trace and every slow trace, samples 5% of normal traces, and discards the rest. Storage cost drops by 95% with effectively zero loss of debugging value.

2.1 Custom callback for non-OTel workflows

Sometimes you need metrics or logs that don’t naturally fit into trace span attributes — for example, a fleet-wide compliance score, or the count of hosts behind on patches. For these, write a thin custom callback that emits to Prometheus pushgateway or directly to a metrics endpoint:

# callback_plugins/kv_metrics.py
from ansible.plugins.callback import CallbackBase
from prometheus_client import CollectorRegistry, Counter, Histogram, push_to_gateway

class CallbackModule(CallbackBase):
    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = 'aggregate'
    CALLBACK_NAME = 'kv_metrics'

    def __init__(self):
        super().__init__()
        self.registry = CollectorRegistry()
        self.task_counter = Counter(
            'ansible_task_total', 'Tasks executed',
            ['template', 'play', 'task', 'status'],
            registry=self.registry,
        )
        self.task_duration = Histogram(
            'ansible_task_duration_seconds', 'Task duration',
            ['template', 'task'],
            buckets=[0.1, 0.5, 1, 5, 10, 30, 60, 300, 1800],
            registry=self.registry,
        )

    def v2_runner_on_ok(self, result):
        self._record(result, 'ok')

    def v2_runner_on_failed(self, result, ignore_errors=False):
        self._record(result, 'failed')

    def v2_runner_on_skipped(self, result):
        self._record(result, 'skipped')

    def _record(self, result, status):
        labels = {
            'template': os.environ.get('TOWER_JOB_TEMPLATE_NAME', 'cli'),
            'play': result._task._role._role_name if result._task._role else 'no-role',
            'task': result._task.get_name(),
            'status': status,
        }
        self.task_counter.labels(**labels).inc()
        # ...

    def v2_playbook_on_stats(self, stats):
        push_to_gateway(
            os.environ['PROMETHEUS_PUSHGATEWAY'],
            job=os.environ.get('TOWER_JOB_TEMPLATE_NAME', 'cli'),
            registry=self.registry,
        )

Drop this in your execution environment’s callback_plugins/, list it in callbacks_enabled, set PROMETHEUS_PUSHGATEWAY=https://pushgateway.kv.local:9091, and every playbook run will emit task-level counters and histograms. This is the foundation for any custom metric you want.

---

3. AAP control plane metrics

AAP exposes a Prometheus-compatible metrics endpoint at /api/v2/metrics/. A minimal scrape config:

# prometheus.yml
scrape_configs:
  - job_name: aap-controller
    metrics_path: /api/v2/metrics/
    bearer_token: '{{ aap_metrics_token }}'
    scheme: https
    static_configs:
      - targets: ['aap.kv.local:443']
    relabel_configs:
      - source_labels: [__address__]
        target_label: aap_instance

The metrics AAP exposes natively (excerpts):

• awx_status_total{state="successful|failed|canceled|error"} — job result counters
• awx_running_jobs — current parallelism
• awx_pending_jobs — queued jobs (saturation signal)
• awx_instance_consumed_capacity — how busy each control node is
• awx_database_connections_total — Postgres connection pool usage
• awx_inventory_total / awx_organization_total — counts
• awx_subscription_total{type="hosts|nodes"} — license utilisation

The two metrics most worth alerting on:

# alert: control plane saturation
- alert: AAPControlPlaneSaturated
  expr: |
    avg by (aap_instance) (awx_instance_consumed_capacity)
      / avg by (aap_instance) (awx_instance_total_capacity)
      > 0.85
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "AAP control plane > 85% capacity for 15 minutes"
    description: "Schedule capacity scale-up; queue depth is rising."

# alert: job failure rate spike
- alert: AAPJobFailureRateSpike
  expr: |
    sum(rate(awx_status_total{state="failed"}[5m]))
      / sum(rate(awx_status_total[5m]))
      > 0.10
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "AAP job failure rate > 10% over 10 minutes"
    description: "Investigate template or environment regression."

These are the only two AAP-level alerts most teams need. Per-template alerts are usually too noisy and end up disabled within a quarter.

---

4. The unified dashboard taxonomy

A common failure mode is “we have 200 Grafana dashboards and nobody knows which one to open during an incident.” The fix is a strict three-layer dashboard taxonomy:

Every alert routes to a specific dashboard. The Slack message format is rigid:

🔴 AAP job failure rate > 10%
Severity: critical | Triggered: 14:03 | Active: 12m
Dashboard: https://grafana.kv.local/d/aap-l1-health (L1)
Investigation: https://grafana.kv.local/d/aap-l3-jobs (L3)
Runbook: https://wiki.kv.local/runbooks/aap-failure-spike

Every responder gets the same starting point. No “where do I look?” question.

4.1 The L1 platform-health dashboard

Twelve panels:

1. Job rate (5m): rate(awx_status_total[5m]) stacked by status — see traffic + errors at once
2. p95 job duration by template family: 95th percentile from histogram_quantile(0.95, rate(ansible_task_duration_seconds_bucket[5m]))
3. Control node CPU/memory: node_cpu_seconds_total, node_memory_* filtered to AAP nodes
4. Postgres health (AAP database): connection pool, replication lag, slow queries
5. Receptor mesh state: AAP’s internal mesh — node count, peer health, message queue depth
6. EDA rulebook activations: count of running rulebooks; up for each rulebook process
7. Inventory sync health: time since last successful sync per inventory; alert > 4h
8. Webhook receivers: HTTP rates and error rates on AAP webhook endpoints
9. Subscription / license: awx_subscription_total vs limit
10. Top 10 slowest jobs (last 24h): tabular drill-down
11. Top 10 most-failing templates (last 7d): tabular drill-down
12. CHG-compliance metric: % of production jobs that ran with a change_request_number extra var

That last panel is the most underrated. It’s a single number that answers “is governance actually working?” Healthy organisations keep this at 100% (excluding the read-only template list). Drop below 99% and it’s a P2 incident — someone has bypassed the gate.

---

5. Logs: Loki and structured AAP output

AAP emits two distinct log streams:

1. Job execution logs — stdout/stderr of every playbook run; written to disk and accessible via API
2. Service logs — control plane internals (web tier, task scheduler, callback receiver)

Both should land in Loki via promtail or Vector. The crucial discipline is structured logging: rather than free-form prose, use the community.general.log_plays callback plugin to emit JSON-per-task:

[defaults]
callbacks_enabled = ansible.posix.opentelemetry, community.general.log_plays
log_path = /var/log/ansible/play-{{ tower_job_id }}.json

Each JSON record contains ts, host, task, module, status, result.changed, result.msg. Loki labels include job_id, template_name, inventory, severity. The query language (LogQL) becomes precise:

# Find all failed `template` module tasks across all jobs in last hour
{template_name=~".+"} 
  | json 
  | status="failed" 
  | module="template"
  | line_format "{{.task}} on {{.host}}: {{.result_msg}}"

That single query, displayed in a Grafana log panel beside the L1 metrics, lets on-call instantly see “what’s failing right now” without clicking into individual jobs.

For ad-hoc operator debugging, a pre-built saved query for “show me everything from job 12345”:

{job_id="12345"} | json | line_format "{{.host}} | {{.task}} | {{.status}} | {{.result_msg}}"

Same data, different filter. The point is that every log query from operators should use the structured fields, not full-text grep. Free-text search of multi-GB log streams is prohibitively expensive at scale; field-indexed query is fast.

5.1 Log retention discipline

Operations logs typically need 30-90 days of hot storage; compliance often mandates 1-7 years for production change records. Structure your Loki tiering accordingly:

• 0-7 days: hot storage, full indexing, sub-second query
• 7-90 days: warm storage, ~1s query, retained for incident investigation
• 90 days - 7 years: cold storage in S3/object lock, queryable but slow, primarily for compliance

Most environments use Loki’s built-in compactor to handle this; Mimir / Cortex have the same pattern for metrics. Without these tiers, observability storage cost runs away within a quarter.

---

6. Traces and the multi-host correlation problem

The single most useful capability traces unlock is per-host execution timeline visualisation. AAP’s UI shows a job’s task list serially. A trace shows the same job as a Gantt chart across hosts: host1 ran task A from 14:03:00 to 14:03:08, then waited 4 seconds, then ran task B; host2 ran task A from 14:03:01 to 14:03:25 (slow!) — and immediately you can see which host is the long pole.

The default Tempo + Grafana visualisation gives you this for free once OTel is wired. The skills to use it well:

Find slow tasks across a fleet: trace search with service.name = ansible-aap AND duration > 30s shows every task that took longer than 30 seconds, grouped by task name. Discover that template render on host group X is consistently slow → investigate filesystem latency on those hosts.

Find failures correlated by module: trace search with service.name = ansible-aap AND status = error AND ansible.task.module = systemd shows all systemd-related failures across the fleet, last 24h. Discover a pattern (specific service name on specific OS version) and fix it once.

Cross-system correlation: this is where traces really earn their keep. Wire OTel into your AAP webhook receiver, into Event-Driven Ansible, into the application code that triggered the workflow. Now a single trace shows: “User clicked Slack button → Slack webhook received by EDA → EDA fired remediation rulebook → AAP launched job → Ansible ran on host → host’s metric came back to normal.” That entire causal chain in one trace, linked by traceparent headers passed at every boundary.

Implementing the full chain requires:

1. AAP webhook receiver propagates incoming traceparent into the launched job’s extra vars
2. The Ansible callback plugin reads traceparent from extra vars and uses it as the parent context
3. EDA’s rulebook engine, when triggering a job via API, propagates its current trace context
4. Slack/Teams bots, when invoking AAP, set traceparent from their incoming request

This is fiddly to set up but transformative once running. Mean time to root cause for “why did remediation fail?” drops from 30 minutes of cross-system investigation to one Grafana click.

---

7. The ServiceNow event correlation

Linking ITSM to observability is the final capstone wire. Two integration directions:

ServiceNow → metrics: Every CHG, INC, and PRB record event posts to a webhook that emits a Prometheus event. You get metrics like:

• servicenow_chg_total{state, type, environment} — change rate by state
• servicenow_inc_total{priority, assignment_group, category} — incident rate
• servicenow_chg_lead_time_seconds_histogram — time from CHG.opened to CHG.implement

Metrics → ServiceNow: Alertmanager’s webhook receiver creates ServiceNow incidents directly:

# alertmanager.yml
receivers:
  - name: servicenow
    webhook_configs:
      - url: 'https://aap.kv.local/api/v2/job_templates/snow-create-inc/launch/'
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/aap-token

The “snow-create-inc” job template runs a playbook that takes the alertmanager payload, derives priority/category/assignment, and creates an INC via servicenow.itsm.incident. Now every operationally significant alert has a ticket; every ticket auto-resolves when the alert clears.

The closed-loop pattern, end-to-end:

1. Host metric crosses threshold (e.g. disk > 90%)
2. Prometheus fires alert → Alertmanager → AAP webhook
3. AAP creates ServiceNow INC, priority computed from severity
4. EDA rulebook subscribed to "INC created with category=disk" fires
5. EDA launches "INC: Disk cleanup" job template
6. Job runs cleanup, verifies disk now < 80%
7. Job posts work note + resolves INC
8. Host metric returns to normal → Alertmanager fires resolved
9. AAP webhook closes any matching open INCs (idempotent)

In a healthy organisation this loop runs hundreds of times a day, with humans involved only on the long tail of cases the automation cannot handle. The metric to track is the auto-resolution rate — the percentage of incidents that closed without human intervention. Healthy mature platforms reach 60-80%; the remaining 20-40% are the genuinely novel issues humans should focus on.

---

8. SLOs as the contract

The thread that holds the whole observability story together is the Service Level Objective. For an automation platform, the SLOs that matter:

The discipline: every SLO has an error budget. If your platform availability SLO is 99.9%, you have 0.1% of “budget” to spend on outages, deploys, and changes per quarter. When the budget is consumed, you must freeze risky changes until the budget recovers (typically over the next 30 days).

This is the SRE playbook applied to automation. The error budget aligns the platform team’s incentives: they want to ship features, but every failure consumes budget; therefore quality and reliability work earn the right to ship the next feature. Without this, the platform team always picks features over reliability, and the platform degrades over time.

A Grafana SLO dashboard panel:

Availability SLO: 99.9% (target)
  Last 30 days: 99.94% (above target ✅)
  Error budget remaining: 73% (8h 12m of 11h 43m)
  Burn rate (1h): 0.4x (sustainable)
  Burn rate (24h): 1.1x (sustainable)

When burn rate exceeds 14.4x for an hour or 6x for six hours (Google’s recommended thresholds), page on-call. Otherwise the SLO panel is just a quiet, daily health check.

---

9. The closed feedback loop in practice

The thing that makes all of this actually transformative is when alerts trigger automation that closes the alert — and the loop is observable end-to-end. The full lifecycle:

Step 1: A node_exporter metric on prod-app-04 shows node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10.

Step 2: Prometheus fires DiskSpaceLow alert. Alertmanager routes to webhook receiver.

Step 3: AAP Create-INC template runs. Creates ServiceNow INC with category=disk, priority=2, cmdb_ci=prod-app-04. Trace ID abc123.

Step 4: EDA rulebook incident-remediation polls ServiceNow every 30s, sees the new INC, matches the rule for “disk full,” calls AAP API to launch INC-disk-cleanup template. Propagates abc123 as traceparent.

Step 5: AAP runs the cleanup playbook on prod-app-04. The OTel callback plugin uses abc123 as the trace root. Tasks run, logs flow to Loki tagged with traceparent=abc123.

Step 6: Cleanup succeeds. df shows disk now at 67%. Playbook posts work note, resolves INC, sets close_code=“Solved (Permanently)”.

Step 7: 60 seconds later, node_exporter scrape shows disk back below threshold. Prometheus fires DiskSpaceLow resolved.

Step 8: Alertmanager sends resolved notification. AAP webhook receives it, looks for any open INCs matching this CI + alert; the INC is already resolved, so this is a no-op (idempotent).

The Grafana dashboard for this single incident shows:

• A trace tree starting at the alert webhook, through INC creation, through job launch, through every Ansible task, ending in the resolved INC update — all under abc123
• Metric panels showing the disk utilisation falling exactly when the cleanup tasks ran
• Log panels filtered by traceparent=abc123 showing every line of every task
• ServiceNow INC linked from the trace root span

That single screen tells the story of one auto-healed incident with zero ambiguity. Operationally, this is gold — but it is also the evidence artefact an auditor wants when asking “show me an example of how your automation responds to incidents.” One trace ID, one Grafana link, one minute to walk through the full chain.

---

10. Operational rituals that keep observability healthy

A surprising failure mode: organisations build great observability, then it decays over months. Three rituals prevent this:

Weekly observability review (15 min): Platform team reviews:

• Top 10 noisiest alerts by volume — are they actionable? If not, fix or delete.
• Any dashboards that haven’t been opened in 30 days — delete (yes, really).
• Any metrics with cardinality > 100k — investigate; usually a label runaway bug.
• SLO burn rates — anything above 1.0x merits investigation.

Monthly chaos game day: Pick one specific failure mode (control plane node down, Postgres replica lagging, Loki ingest backlogged) and verify your alerting catches it within target latency. Failures here mean the alert is misconfigured; fix it before the real outage.

Quarterly dashboard pruning: Each domain owner reviews their L2 dashboards. If a panel has fired no useful insight in 90 days, either rewrite it to be useful or remove it. Dashboards bloat over time; aggressive pruning keeps them readable.

The principle: observability is a product, not a one-time build. It needs roadmap, ownership, and continuous quality work. The orgs that get this wrong end up with massive observability bills, dashboards no one looks at, alerts that fire constantly and are universally muted — and they then re-build the whole thing every two years. The orgs that get it right have a stable, slowly-evolving observability layer that sustains for a decade.

---

11. Cardinality discipline (the silent killer)

A specific failure mode worth its own section: metric cardinality explosion. Prometheus is fast and cheap if you keep cardinality bounded; it falls over hard if you don’t.

Examples of cardinality bombs:

# BAD — user_id has unbounded cardinality
rate(api_request_total{user_id="$user"}[5m])

# BAD — unique trace IDs as label
counter.add(traceparent=$traceparent)

# BAD — host names without bucketing
ansible_task_total{host="host-12345.cluster.local"}

Every unique label value combination is a separate time series. 10 templates × 100 hosts × 50 tasks × 4 statuses = 200k series. Add host_ip_address as a label and now it’s 200k × 1k IPs = 200M series. Prometheus crashes.

The discipline:

• Bucket high-cardinality fields: template_family (10 values) instead of template_name (1000 values); record the full template_name in the log, not the metric
• Never use IDs as labels: trace IDs, request IDs, user IDs, transaction IDs go in logs and traces, not metrics
• Drop labels you don’t query on: every label you add costs forever; if no PromQL query in your repo references it, delete it

A continuous monitoring metric:

# alert if any single metric exceeds 100k series
- alert: HighCardinalityMetric
  expr: |
    count by (__name__) (count by (__name__) ({__name__=~".+"})) > 100000
  for: 30m

This catches cardinality bombs within an hour of introduction, before they impact ingestion performance.

---

12. Budgeting and capacity

Final practical concerns. Observability is not free, and ungoverned observability costs grow faster than the workload they observe.

Rough annual costs at scale (industry typical, varies by vendor and region):

• Metrics: $0.15-1.00 per active series per year. 1M series ≈ $200k-1M/year. Aggressive aggregation cuts this 5-10x.
• Logs: $0.50-2.00 per GB ingested. 5TB/day ≈ $1-4M/year. Tiered storage and structured-log compression cut this 3-5x.
• Traces: $0.10-0.50 per million spans. With 10% sampling, 1B spans/day ≈ $400k-2M/year.

The most common pattern: an organisation spends $X on the cloud workload they’re observing and $0.5-2X on observing it. That’s normal. When it exceeds 2X you have a quality problem (label explosion, log spam, no sampling) — fix the discipline, not the budget.

Capacity planning for the observability stack itself:

• Prometheus / Mimir: scale by active series (target < 5M per node)
• Loki: scale by ingest rate (target < 50MB/s per ingester)
• Tempo: scale by spans per second (target < 100k spans/s per ingester)
• Grafana: scale by concurrent dashboards (target < 100 concurrent users per node)

Run synthetic load tests quarterly to verify the headroom. Never let any component exceed 70% utilisation in steady state — the spike on the day of an incident will push it over.

---

13. Where this leaves you

You have just completed Tier 5 of this Ansible course. Across ten lessons we’ve covered:

• D1 — Compliance: STIG, CIS, OpenSCAP, signed evidence
• D2 — Disaster recovery: hybrid topology, DR drills, RTO/RPO discipline
• D3 — Migrations: P2V, V2V, leapp, RHEL major upgrades
• D4 — Air-gap: soft / sneakernet / data-diode archetypes
• D5 — SAP: HANA, Netweaver, redhat.sap collections
• D6 — Edge/IoT: pull-mode, bootc, k3s+fleet
• D7 — ITSM/ChatOps: ServiceNow CMDB, CHG-gating, Slack/Teams approval
• D8 — Backup: 3-2-1-1-0, immutability, restore drills
• D9 — Database migrations: expand-contract, blue-green, online DDL
• D10 — Observability (this lesson): the closed feedback loop

What you should walk away with is the conviction that mature automation in a regulated enterprise is not a single tool or playbook. It is an interoperable system of disciplines: governance via ITSM, content via Ansible, evidence via signed bundles, recovery via tested DR, scale via fleet patterns, and visibility via observability. None of these alone is sufficient; together they form a platform that auditors trust, executives can defend, and engineers actually want to use.

The next steps from here depend on your role:

• Platform engineer: pick the lesson with the biggest gap in your environment. For most orgs that is D8 (backup automation) or D7 (ITSM integration). Ship one of these in the next quarter.
• SRE / Operations: implement D10 (this lesson) end-to-end. The other lessons amplify their value once the observability layer exists.
• Architect / Tech lead: use Tier 5 as a maturity assessment. For each pillar, ask “what is our current state? What is our 12-month target? What does the gap require?”
• Engineering leader: this curriculum maps to a 12-24 month maturity transformation. Treat it as a programme, not a project. The teams that deliver each pillar in 3-month iterations succeed; the teams that try to do all ten simultaneously fail.

The hardest lesson across this whole course is also the simplest: automation is a cultural and organisational artefact as much as a technical one. The playbooks are the easy part. The disciplines — change-management, evidence-bundling, restore-testing, SLO-budgeting — are what separate organisations whose automation actually works from those whose automation is a slide deck. This curriculum exists to give you the patterns. Whether they take root depends on the people, leadership, and engineering culture you build around them.

That’s what makes the journey worth it.

---

Going deeper

The connective idea of this whole lesson is that profiling, tuning, and observability all ride the same callback event stream. Here is what is actually happening under the hood, and the edge cases that bite at scale.

The callback plugin execution model. Ansible loads two kinds of callbacks. There is exactly one stdout callback (set by stdout_callback, default default) that owns the terminal — you cannot enable two, and if you try, the second is ignored. Then there are many aggregate/notification callbacks enabled via callbacks_enabled; profile_tasks, profile_roles, timer, opentelemetry, and log_plays are all of this second kind, which is why you can stack them. Each declares CALLBACK_NEEDS_ENABLED = True, so they stay dormant until explicitly listed. They fire on documented hook methods (v2_runner_on_ok, v2_runner_on_failed, v2_playbook_on_stats, …). Profiling and OTel are literally the same hooks doing different things with the timing: one prints a table, the other emits a span. Internalise that and the two halves of this lesson stop being separate topics.

Why profile_tasks and OTel spans can disagree. profile_tasks measures wall-clock between task-start and task-end on the controller, including the strategy plugin’s scheduling gaps. An OTel task span measures the module execution window. With the free strategy, hosts run out of lock-step, so a profile_tasks line can look inflated because it spans idle scheduling time. When numbers surprise you, trust the trace’s per-host Gantt over the aggregate table.

Fact caching internals. fact_caching is a cache plugin: jsonfile writes one file per host under fact_caching_connection; redis/memcached write to a shared store keyed by hostname with a TTL of fact_caching_timeout. gathering = smart checks the cache before running the setup module — a hit skips gathering entirely. The subtle bug: a stale cache serves old facts (an IP that changed, a disk that grew), and playbooks then template wrong values. Set a sane fact_caching_timeout, and for correctness-critical plays force a refresh with meta: clear_facts or an explicit ansible.builtin.setup. gather_subset narrows what setup collects (network, hardware, virtual, !all, !min); the hardware subset is the expensive one on bare metal, so excluding it is often the single biggest gather-time win.

Pipelining, become, and requiretty. Pipelining removes the “copy the module to a temp file, then execute it” round-trip by piping the module straight into the remote Python interpreter over the existing SSH session. It only works if the remote sudo config does not demand a TTY — a requiretty line in /etc/sudoers breaks it. That is why pipelining is off by default: it is a safe, large win once you’ve confirmed requiretty is disabled. Combined with ControlPersist (which keeps one SSH master socket alive under control_path_dir and multiplexes every task through it), you cut the per-task cost from “TCP + TLS + auth + two file copies” down to “write to an open pipe.”

The cost of forks. Each fork is a separate fork()ed worker process on the control node, each running its own Python interpreter and connection. Memory scales roughly linearly with forks; 500 forks on a controller with 8 GB RAM will OOM. The right ceiling is empirical: raise forks, watch control-node CPU/RAM under a real run, and stop when the controller — not the network or the targets — becomes the bottleneck. Beyond that, scale horizontally (AAP execution nodes, sliced jobs), not vertically.

free strategy pitfalls. Under free, a failure on one host does not stop the others (unlike linear, where any_errors_fatal can halt the batch at a task boundary). Handlers still run at the end of each host’s play, but hosts reach that point at different times, so notification timing is non-deterministic. Output interleaves, which makes raw logs harder to read — another reason structured logging into Loki (labelled by host) matters more with free. Reach for free/host_pinned only when hosts are genuinely independent; keep linear for anything ordered.

Mitogen, precisely. Mitogen replaces the strategy so that module execution reuses long-lived remote Python interpreters over a persistent connection, importing modules in-memory rather than shipping files. The speedup is real (roughly 1.5–7× on connection-heavy plays) but it is a third-party plugin pinned to specific ansible-core versions, it can conflict with certain connection/become/other strategy plugins, and it is not supported inside AAP execution environments. Treat it as an optimisation for self-managed ansible-core, version-locked and tested, never as a load-bearing assumption in a supported design.

OTel context propagation. The traceparent a run inherits is a W3C Trace Context header of the form 00-<32-hex-trace-id>-<16-hex-span-id>-<flags>. The whole cross-system chain in §6 works because each boundary forwards that string: the webhook receiver passes it as an extra var, the callback adopts it as the parent context, EDA re-emits it when launching the next job. Get the format wrong at any hop and the trace fragments into disconnected trees.

Exemplars: the metrics↔traces bridge. A Prometheus histogram bucket can carry an exemplar — a sampled trace_id attached to a specific observation. In Grafana, that renders as a clickable dot on the latency panel that jumps straight to the trace in Tempo. This is how you go from “p95 task duration spiked at 14:03” (metric) to “here is the exact slow run” (trace) in one click. It requires exemplar storage enabled on the Prometheus/Mimir side and instrumentation that records the trace ID with the observation.

Check mode and profiling. Running with --check still fires callbacks, so profile_tasks works in a dry run — useful for estimating a change’s task profile before you actually apply it. But check-mode timings under-report anything whose real cost is the change itself (a package install reports near-zero in check mode), so calibrate expectations.

---

Practice challenges

Work these in order — they escalate from “turn a knob and read the output” to “wire the full performance-plus-observability loop.” Each has a worked solution; try it before you open it.

1. (Beginner) Profile a playbook and name the slowest task. Enable profile_tasks and timer, run any multi-task playbook, and identify the single biggest time sink.

<details> <summary>Solution</summary>

# ansible.cfg
[defaults]
callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer

Run ansible-playbook site.yml. Read the summary table printed at the end; the top line (descending sort) is your slowest task, and timer prints total wall-clock. Why: in almost every real playbook the top line is Gathering Facts — you can’t tune what you haven’t measured, and measurement points straight at the fact-gathering lever. </details>

2. (Beginner) Cut fact-gathering cost on a play that doesn’t need it. You have a play that only installs packages and never references a fact. Make it stop gathering facts, then make a second play gather only network facts.

<details> <summary>Solution</summary>

- name: Package-only play — no facts needed
  hosts: all
  gather_facts: false
  tasks:
    - name: Ensure tools present
      ansible.builtin.package:
        name: ["vim", "curl", "git"]
        state: present

- name: Needs only network facts
  hosts: webservers
  gather_facts: true
  gather_subset:
    - '!all'
    - '!min'
    - network

Why: gather_facts: false removes the setup run entirely (biggest single win); gather_subset narrows collection so you skip the expensive hardware subset when you only need ansible_default_ipv4. </details>

3. (Intermediate) Build a fast, safe ansible.cfg. Raise parallelism, reuse SSH connections, enable pipelining, and turn on smart fact caching to a JSON file — the four universally-safe levers together.

<details> <summary>Solution</summary>

[defaults]
forks = 50
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /var/tmp/ansible_fact_cache
fact_caching_timeout = 7200

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
control_path_dir = ~/.ansible/cp

Why: forks lifts the default 5-host throttle; ControlPersist + pipelining collapse per-task SSH overhead; gathering = smart + a cache turns repeated fact gathering into cache hits. Confirm requiretty is off in remote sudoers or pipelining will fail on become tasks. </details>

4. (Intermediate) Run a long task without blocking the fleet. A dataset import takes ~40 minutes. Launch it fire-and-forget, do other work, then wait for completion with a one-hour ceiling.

<details> <summary>Solution</summary>

- name: Launch import, don't block
  ansible.builtin.command: /opt/scripts/import_dataset.sh
  async: 3600
  poll: 0
  register: import_job

- name: Meanwhile, warm the cache
  ansible.builtin.command: /opt/scripts/warm_cache.sh

- name: Wait for the import to finish
  ansible.builtin.async_status:
    jid: "{{ import_job.ansible_job_id }}"
  register: r
  until: r.finished
  retries: 60
  delay: 60

Why: poll: 0 returns immediately (launched, not finished), so the play does useful work in parallel; async_status with until: r.finished reconciles the background job — skip that step and you’d leave an untracked job behind. </details>

5. (Advanced) Choose and justify a strategy for two different plays. One play bootstraps 2,000 independent hosts (no cross-host ordering). Another does an ordered rolling deploy, 20% at a time. Pick the strategy for each and explain the risk you accept.

<details> <summary>Solution</summary>

- name: Independent bootstrap — race ahead per host
  hosts: all
  strategy: free
  tasks:
    - ansible.builtin.package: { name: chrony, state: present }

- name: Ordered rolling deploy
  hosts: webservers
  strategy: linear
  serial: 20%
  max_fail_percentage: 10
  tasks:
    - ansible.builtin.unarchive:
        src: "app-{{ release }}.tar.gz"
        dest: /opt/app

Why: free maximises throughput when hosts are independent — the accepted risk is interleaved output and non-lock-step handler timing. The deploy uses linear + serial so each 20% batch completes in order before the next, and max_fail_percentage halts the rollout if a batch goes bad — you trade speed for a controlled blast radius. </details>

6. (Advanced) Close the performance→observability loop with a PromQL SLO query. Your custom callback already emits the ansible_task_duration_seconds histogram. Write the PromQL that computes p95 task duration per task, and describe the alert that would fire if a task family regresses past 30s.

<details> <summary>Solution</summary>

# p95 task duration, per task, over the last 5 minutes
histogram_quantile(
  0.95,
  sum by (le, task) (rate(ansible_task_duration_seconds_bucket[5m]))
)

# Alert on a latency regression
- alert: AnsibleTaskLatencyRegression
  expr: |
    histogram_quantile(0.95,
      sum by (le, task) (rate(ansible_task_duration_seconds_bucket[5m]))) > 30
  for: 10m
  labels: { severity: warning }
  annotations:
    summary: "p95 for task {{ $labels.task }} > 30s for 10m"

Why: the same per-task timing you read locally in profile_tasks is now a fleet-wide histogram; histogram_quantile over the _bucket series gives p95, and an exemplar on that panel lets you click straight from “slow” to the exact Tempo trace. That is the whole lesson in two queries — profile locally, prove it globally. </details>

---

Common beginner mistakes

• “More forks always means faster.” Forks help only until the control node is the bottleneck. Each fork is a full worker process; push forks too high and the controller thrashes or OOMs, making runs slower and less reliable. Right model: raise forks while watching control-node CPU/RAM, and scale out (execution nodes, sliced jobs) instead of endlessly up.

• “I’ll just guess which task is slow and optimise that.” Optimising without profiling is superstition; the real cost is almost always fact gathering or connection overhead, not the module you suspected. Right model: profile_tasks first, tune the top line, re-measure.

• “Turning off gather_facts will break my playbook.” It only breaks if a task actually references a fact. Many plays gather a full fact set and use none of it. Right model: gather nothing by default, then add a narrow gather_subset for exactly the facts you template — or cache facts with gathering = smart.

• “Mitogen is just a supported speed setting.” Mitogen is a powerful but unofficial, version-pinned third-party plugin that Red Hat does not support in AAP. Treating it as a supported feature is how a platform upgrade silently breaks every job. Right model: use it only on self-managed ansible-core, version-locked and tested; never assume it in a supported-platform design.

• “The free strategy is just linear-but-faster.” free changes semantics, not just speed: a failure on one host doesn’t stop the others, handler timing is non-deterministic, and output interleaves. Right model: use free/host_pinned only for genuinely independent hosts; keep linear for anything ordered or where a failure should halt the batch.

• “poll: 0 means the task finished.” It means the task launched. Without a follow-up async_status, you have an untracked background job whose success you never verified. Right model: every fire-and-forget task gets a matching async_status wait (or is deliberately, documentedly fire-and-forget).

• “Profiling a run in production is fine — it’s just a summary.” The profiling callbacks themselves are cheap, but the mistake is emitting high-cardinality labels (per-host, per-trace-id) into Prometheus, which explodes series count and can take down ingestion. Right model: IDs and hostnames belong in logs/traces; metrics get bucketed labels like template_family.

• “One big Grafana dashboard shows everything.” A single mega-dashboard is unreadable during an incident and nobody knows where to look. Right model: the L1/L2/L3 taxonomy — platform health, workload domain, and per-run investigation — with every alert linking to the right layer.

---

Glossary

• Callback plugin — Ansible plugin that fires on run events (task ok/failed, stats). One stdout callback owns the terminal; many aggregate/notification callbacks (profiling, OTel, log_plays) can be enabled together via callbacks_enabled.
• profile_tasks / profile_roles / timer — the three ansible.posix profiling callbacks: per-task timing (sorted slowest-first), per-role timing, and total wall-clock, respectively.
• forks — how many hosts Ansible operates on in parallel ([defaults] forks, -f). Default 5; each fork is a worker process on the control node.
• pipelining — SSH optimisation that pipes a module straight into the remote interpreter instead of copying a temp file, cutting round-trips. Requires requiretty disabled in remote sudoers.
• ControlPersist / ControlMaster — OpenSSH connection multiplexing: keep one master socket alive and route every task through it, avoiding a fresh handshake per task. Set via ssh_args / control_path_dir.
• fact caching — storing gathered facts (jsonfile, redis, memcached) so subsequent runs skip re-gathering. Paired with gathering = smart and a fact_caching_timeout.
• gathering (implicit / explicit / smart) — when Ansible auto-gathers facts: always, never, or once-then-cached. smart + a cache is the tuned default.
• gather_subset — narrows which fact categories setup collects (network, hardware, virtual, !all, !min); excluding hardware is a common speed win.
• mitogen — third-party strategy plugin that reuses remote Python interpreters/connections for large speedups; unofficial, version-pinned, and unsupported in AAP.
• strategy (linear / free / host_pinned) — the plugin that steps hosts through a play: lock-step (linear, default), each host racing independently (free), or workers pinned to a host until it finishes (host_pinned).
• serial / throttle — batch size for rolling changes across a fleet, and a per-task concurrency cap; orthogonal to strategy.
• async / poll — run a long task in the background. poll > 0 polls with a hard time ceiling; poll: 0 is fire-and-forget, reconciled later with async_status via the returned ansible_job_id.
• async_status — module that checks a backgrounded async job’s finished state, typically in an until/retries/delay loop.
• sliced job (AAP) — splitting one job template across multiple execution nodes, each handling a fraction of the inventory in parallel — horizontal scale beyond a single controller’s forks.
• OpenTelemetry (OTel) — vendor-neutral standard for metrics/logs/traces; ansible.posix.opentelemetry emits a span tree (playbook → play → task) per run.
• span / trace / traceparent — a trace is a tree of spans (timed operations); traceparent is the W3C header (00-<trace-id>-<span-id>-<flags>) forwarded across systems to stitch one causal chain together.
• Prometheus — time-series metrics database with the PromQL query language; scrapes AAP’s /api/v2/metrics/ endpoint natively.
• counter / gauge / histogram — Prometheus metric types: monotonically increasing count, a value that goes up and down, and bucketed distributions (used for latency percentiles).
• PromQL / histogram_quantile — Prometheus query language; histogram_quantile(0.95, ...) over _bucket series computes p95 latency.
• exemplar — a sampled trace ID attached to a histogram observation, rendered in Grafana as a clickable dot that jumps from a metric spike to the exact trace.
• cardinality — the number of unique label-value combinations; each is a separate time series. Unbounded labels (IDs, hostnames) cause a “cardinality explosion” that can crash Prometheus.
• Loki / LogQL — Grafana’s label-indexed log store and its query language; pairs with structured JSON logs from community.general.log_plays.
• Tempo — Grafana’s distributed-tracing backend that stores and searches OTel spans.
• Mimir / Cortex / Thanos — long-term, horizontally-scalable, multi-tenant storage backends for Prometheus metrics.
• Grafana — visualisation layer unifying metrics, logs, and traces in one pane.
• Alertmanager — Prometheus’s alert router; deduplicates, groups, and dispatches alerts to Slack/Teams/PagerDuty/ServiceNow/webhooks.
• pushgateway — Prometheus component that batch/ephemeral jobs (like a finished playbook) push metrics to, since they can’t be scraped.
• golden signals — latency, traffic, errors, saturation; the four SRE signals, plus convergence (did the automation reach desired state) for automation platforms.
• SLO / error budget / burn rate — Service Level Objective (target reliability), the allowance of failure it implies, and how fast that allowance is being consumed; excessive burn rate pages on-call.
• AAP metrics endpoint — /api/v2/metrics/, the Prometheus-format endpoint exposing awx_* control-plane metrics (job status, capacity, queue depth, subscription).
• EDA (Event-Driven Ansible) — rulebook engine that subscribes to events (alerts, ITSM records) and launches remediation job templates, forming the closed loop.
• tail sampling — deciding whether to keep a trace after it completes (keep all errors and slow traces, sample the rest) to cut storage cost with minimal loss of debugging value.