DevOps Troubleshooting: Pipelines, Builds, Deployments, Runners & Artifacts

The engineer everyone pages when the pipeline is red is rarely the one who has memorised the most YAML keys. They are the one with a method. When the build that passed an hour ago suddenly fails, when a deploy halts at an approval that nobody can see, when a runner sits “idle” while jobs queue for twenty minutes, the strong engineer does not start blindly re-running the job and hoping — they run a short, fixed sequence that pins the failure to one stage (source, build, test, package/publish, or deploy) and one layer (the code/config, the pipeline definition, the runner/agent, the artifact registry, or the target environment), form a falsifiable hypothesis, prove it from logs, fix it with the smallest safe change, and verify with a clean run. Everyone else clicks Re-run jobs five times, then git commit --allow-empty -m "trigger CI", and burns an afternoon.

This lesson gives you that method and then turns it into playbooks: for each common failure you get a table of symptom → likely cause → the diagnostic that confirms it → the fix. We cover the failures you genuinely hit in production and that come up in interviews and on the AWS / Azure / GCP DevOps engineer exams — build failures (unpinned or unresolvable dependencies, a stale or missing cache, environment and toolchain drift, out-of-memory and disk-full, and flaky tests); pipeline and YAML errors (indentation and type-coercion traps, templating-expression mistakes, and the single biggest source of confusion — variable and secret scope); runner and agent problems (offline or unregistered, no capacity, missing permissions and labels, and the perennial Docker-in-Docker mess); artifact and registry issues (401/403 auth, pull-rate limits, and missing or immutable versions); and deployment failures (a stuck approval gate, OIDC/credential errors, ImagePullBackOff, and a rollout that never becomes healthy). We close with a dedicated flaky-pipeline playbook because intermittent failure is its own discipline. Examples are given for GitHub Actions, GitLab CI, Azure Pipelines and Jenkins, since the symptoms are universal even when the syntax differs.

Learning objectives

By the end of this lesson you can:

• Apply a repeatable troubleshooting loop — observe, isolate the stage and layer, reproduce, hypothesise, fix with the smallest safe change, verify, prevent — to any CI/CD failure.
• Diagnose and fix build failures: dependency resolution and lockfiles, cache hits/misses and cache poisoning, environment/toolchain drift (“works on my machine”), and resource exhaustion (OOM, disk-full).
• Read pipeline/YAML errors correctly — indentation and the Norway/octal coercion traps, templating-expression mistakes, and especially variable and secret scoping across triggers, jobs, environments and forks.
• Resolve runner/agent problems: offline/unregistered runners, capacity and concurrency limits, label/permission mismatches, and Docker-in-Docker vs the Docker socket.
• Fix artifact and registry failures: 401 vs 403, registry pull-rate limits, missing/yanked versions, and immutable-tag pushes.
• Triage deployment failures: approval/gate stalls, OIDC/credential and trust-policy errors, ImagePullBackOff/ErrImagePull, and a rollout stuck “progressing” — and run a flaky-test/flaky-pipeline quarantine-and-fix workflow.

Prerequisites & where this fits

You need a passing familiarity with a CI/CD system — a repository with a pipeline you can edit (GitHub Actions, GitLab CI, Azure Pipelines or Jenkins) and permission to view its logs and re-run jobs. The hands-on lab uses a free GitHub Actions workflow on a throwaway repository, so it costs nothing within the free tier. You should already have the core model from earlier in this course: what a pipeline is and how it is structured from the DevOps fundamentals — culture, CI/CD and the DevOps lifecycle, the YAML you need for pipelines — anchors, templates and the gotchas, how a pipeline is composed from stages, gates and artifacts, and the deployment strategies — rolling, blue/green, canary and rollback whose failures we triage here. We re-explain each failure from first principles, so it is fine if some of that is still fresh. This is the Troubleshooting lesson of the DevOps Zero-to-Hero track — the bridge between building pipelines and operating them under pressure. The next lesson steps up to the DevOps architecting ladder, from a single pipeline to an internal developer platform.

The method: a loop, not a re-run

Almost every pipeline failure yields to the same loop. The discipline is to follow it in order rather than hammering the re-run button — which, for a non-flaky failure, just wastes minutes and tells you nothing new.

1. Observe — read the log to the failing line, not the summary. CI surfaces a red ✗ and an exit code, but the cause is usually a specific line a few screens up. Open the failing step, expand it, and scroll to the first error (a later error is often a consequence of the first). Note the exit code (1 = generic, 137 = OOM/SIGKILL, 143 = SIGTERM/cancelled, 124 = timeout, 127 = command-not-found). Resist re-running until you have read the actual failure.
2. Isolate the stage and the layer. Every failure lives in one stage — source (checkout, submodules, LFS), build (compile/deps), test, package/publish (artifact/registry), or deploy (the target environment) — and one layer: the code/config, the pipeline definition (YAML/expressions/scope), the runner/agent, the registry, or the target environment. Naming both eliminates most of the search space: a YAML parse error is the pipeline layer at parse time; ImagePullBackOff is the deploy stage at the registry/environment boundary; “no runner available” is the runner layer before any of your steps run.
3. Reproduce — shrink the loop. A 12-minute pipeline is a terrible debugger. Reproduce the failing step locally where possible (run the exact build/test command in the same container image), or shrink CI feedback: run only the failing job, add a debug step (env | sort, docker version, whoami, ls -la), or re-run with debug logging (ACTIONS_RUNNER_DEBUG/ACTIONS_STEP_DEBUG for GitHub, CI_DEBUG_TRACE: "true" for GitLab, System.Debug=true for Azure Pipelines). The goal is the shortest path from change to signal.
4. Form one hypothesis. State it as a falsifiable sentence: “the build fails because the cache restored a node_modules built for a different lockfile, so a native module is missing.” A vague hunch (“CI is broken”) cannot be tested; a specific claim can — and points straight at the diagnostic (here: clear the cache and re-run).
5. Fix — the smallest, safest, reproducible change. Prefer pinning a version over loosening one, scoping a secret correctly over widening its scope, fixing a label over adding runs-on: self-hosted everywhere. Change one thing, and prefer a change that makes the pipeline more deterministic, not less. (Re-running until it passes is not a fix — it’s hiding a flake; see the flaky playbook.)
6. Verify, then prevent. A genuine fix is a green run from a clean state (fresh cache where relevant), not “the re-run happened to pass”. Then ask the prevention question: what guardrail — a committed lockfile, a cache key that includes the lockfile hash, a pinned action/image digest, a required status check, a timeout, a concurrency group — stops this recurring?

The decision tree above encodes step 2: start from the symptom at the top, branch on “which stage failed — build, pipeline parse, runner, artifact/registry, or deploy?”, and each leaf points you at the playbook below. Under pressure, walking the tree keeps you honest about which layer you are in before you change the pipeline.

The signals that answer everything

You can diagnose the large majority of failures from a handful of signals. Know precisely what each one tells you — this is what stops the flailing.

A few high-leverage habits: always read the first error, not the last; check the exit code before the message (it often names the class of failure instantly); when a value is “missing”, dump the environment rather than guessing whether the secret reached the job; reach for debug logging the moment the surface log is insufficient; and remember that a failure in a setup/checkout/login step is an infrastructure problem, while a failure in your script is a code/config problem — the boundary tells you which playbook to open.

Build failures: dependencies, cache, environment & resources

The build stage fails for four recurring reasons: it can’t resolve dependencies, the cache lied to it, the environment differs from where the code was written, or it ran out of a resource. Name which one before you touch the YAML — the fixes are completely different.

Two reflexes to burn in. First, make the build reproducible before you make it fast: a committed lockfile plus an install-from-lockfile command (npm ci, not npm install) plus a pinned toolchain (ideally a container image) eliminates the entire “works on my machine” class — most CI-only build failures are an environment difference, not a code bug. Second, treat the cache as a suspect, not a given: a cache key that doesn’t include the lockfile hash will happily restore stale, mismatched artifacts and produce baffling failures, so when a build fails in a way that “makes no sense”, bust the cache and re-run as your first cheap experiment — and then fix the key so it can’t poison again.

Pipeline & YAML errors: syntax, expressions & scope

These are pipeline-layer failures — the system rejects or mis-runs the workflow because of the YAML, an expression, or (most commonly) where a variable or secret is visible. They fail differently from build errors: often the job never starts, or a value is mysteriously empty.

The single most valuable idea here is variable and secret scope. A pipeline value is not “global” — it lives at a level (organisation, repository, environment, job, step) and is visible only where that level reaches, and secrets are deliberately withheld from fork PRs as a security boundary. So when a value is “missing”, the question is never “did I set it?” but “is it in scope here?” — and the fastest answer is a one-line env | sort debug step (secrets print masked but present, which instantly distinguishes “not in scope” from “wrong value”). The second idea is YAML coerces types: an unquoted no, 1.20, or 08 is not the string you think — quote ambiguous scalars by default, and run yamllint before you push so parse errors never cost you a pipeline run.

Runner & agent problems: offline, capacity, permissions & Docker

Runner problems are insidious because they fail before your steps run — the job sits queued, or dies in setup — so engineers waste time debugging code that never executed. The four buckets: there is no runner (offline/unregistered), there is no capacity (all busy / concurrency capped), the runner lacks permission or the right labels, or the build needs Docker and the runner can’t provide it.

Two reflexes here. First, “queued forever” and “idle runners” are almost always a label/capacity problem, not a code problem — before you read a single line of your build script, check that a runner is online and that its labels match what the job requested; a job that never started can’t have a code bug. Second, prefer ephemeral, daemonless container builds: long-lived self-hosted runners accumulate disk cruft, leak state between jobs, and turn Docker-in-Docker into a recurring support ticket — autoscaled ephemeral runners (e.g. GitHub’s Actions Runner Controller, or Azure DevOps scale-set agents) plus a daemonless builder (Kaniko/BuildKit/Buildah) removes most of this class of failure at the root.

Artifact & registry issues: auth, rate limits & versions

The package/publish boundary fails for three reasons: the runner can’t authenticate to the registry, it got rate-limited, or the version/tag it wants is absent, already present, or immutable. The 401 vs 403 distinction is the one to internalise.

The two reflexes. First, 401 means “I don’t know who you are”, 403 means “I know you, but you can’t do that” — 401 sends you to the login/credential step (did it run? is the token expired? right registry?), while 403 sends you to permissions/RBAC (does this identity have push/pull on this repo?); confusing them wastes the most time at this boundary. Second, stop pulling base images anonymously at scale — Docker Hub’s anonymous pull-rate limit will eventually 429 a busy pipeline, so authenticate pulls, cache base layers, and pin by digest so you pull a fixed image once; running your own Harbor or a cloud artifact registry with a pull-through cache removes the dependency on a shared public quota entirely.

Deployment failures: gates, OIDC, image pull & stuck rollouts

The deploy stage spans the boundary between your pipeline and the target environment, so it fails in the richest variety of ways: the pipeline waits on a gate that never clears, it can’t authenticate to the cloud (now usually OIDC), the cluster can’t pull the image, or the rollout never becomes healthy.

The single most valuable idea is that kubectl describe and --previous logs answer almost every Kubernetes deploy failure: ImagePullBackOff is a registry/auth problem the pod Events name exactly, while CrashLoopBackOff is an application problem the previous container’s logs explain — reading those two before changing anything tells you whether to fix the pipeline’s registry auth or the app’s config. The second idea is OIDC failures are trust-policy mismatches: the cloud rejected the federated token because its sub/aud claims don’t match what the role/credential trusts (wrong branch, environment, or repo, or a missing id-token: write permission) — so read the trust condition against the token’s claims rather than reaching for long-lived keys, which reintroduce the very secret-sprawl OIDC removes.

The flaky-pipeline playbook

A flaky pipeline passes and fails without any change to the code — the most corrosive failure mode, because it trains the team to ignore red and to re-run on faith, which lets real failures slip through. Treat flakiness as a first-class bug with its own workflow, not as background noise.

First, prove it’s flaky, not broken. Re-run the same commit with no changes. If it sometimes passes and sometimes fails, it’s flaky; if it always fails, it’s a real bug — go back to the playbooks above. Flakiness is, by definition, non-determinism.

Then find the source of non-determinism. Flaky failures nearly always trace to one of a small set of causes:

Then quarantine, fix, and re-admit — in that order. The workflow that actually clears flakiness:

1. Detect & measure. Track per-test/per-job pass rate over many runs (many CI platforms and test frameworks flag flaky tests automatically). You cannot fix what you don’t measure, and “it feels flaky” isn’t a metric.
2. Quarantine the worst offenders. Move proven-flaky tests to a non-blocking lane (a separate job, continue-on-error, or a quarantine tag) so they stop blocking everyone — but keep running them so they’re still visible. Quarantine is a holding pen, not a delete.
3. Fix the root cause, not the symptom. Resist the cheap “wrap it in a retry” fix for anything that isn’t a genuine async wait — a blanket retry hides real regressions. Fix the determinism (isolate state, wait on conditions, mock externals, control the clock/seed).
4. Re-admit with evidence. Only move a test back to the blocking lane once it has passed N consecutive runs. Then add the guardrail: a flaky-test detector in CI, a policy that new tests must be deterministic, and a budget that caps allowed flake rate.

The meta-rule: a retry is a painkiller, not a cure. Bounded retries are legitimate for genuinely unavoidable non-determinism (a network call, an eventually-consistent cloud API), but reaching for retries: 3 to silence a flaky unit test just buries a real bug and erodes trust in the pipeline. Every retry you add should come with a written reason for why the underlying operation is legitimately non-deterministic.

Hands-on lab: break it, diagnose it, fix it

You’ll plant several classic CI faults in a free GitHub Actions workflow on a throwaway repository, then walk each through the loop. (The symptoms are identical on GitLab/Azure/Jenkins; only the YAML differs.)

1. Create a throwaway repo and a first workflow.

mkdir ci-ts-lab && cd ci-ts-lab && git init -b main
mkdir -p .github/workflows
cat > .github/workflows/ci.yml <<'EOF'
name: ci
on: [push, workflow_dispatch]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: show env
        run: env | sort | grep -E '^(GITHUB_REF|RUNNER_OS)='
      - name: build
        run: echo "build ok"
EOF
git add . && git commit -m "ci: first workflow"
# Create the repo and push (requires the gh CLI, authenticated):
gh repo create ci-ts-lab --private --source=. --push

Open the Actions tab and confirm the run is green. (No gh? Create an empty repo in the UI and git push to it.)

2. Fault A — a YAML parse error (the pipeline layer at parse time). Introduce a tab and a bad indent:

# Break indentation deliberately:
printf '      - name: oops\n         run: echo nope\n' >> .github/workflows/ci.yml
git commit -am "break: bad indentation" && git push

The run fails before any step with “Invalid workflow file”. Diagnose by validating locally — yamllint .github/workflows/ci.yml points at the exact line. Fix by correcting the indentation (spaces, consistent depth), commit, push, and confirm green. Lesson: a parse error never reaches your build — it’s the pipeline layer, caught by yamllint before you push.

3. Fault B — secret scope (the most common “missing value”). Add a step that needs a secret you haven’t set:

cat >> .github/workflows/ci.yml <<'EOF'
      - name: needs a secret
        run: |
          env | sort | grep -i token || true
          test -n "$API_TOKEN" && echo "have token" || { echo "MISSING token"; exit 1; }
        env:
          API_TOKEN: ${{ secrets.API_TOKEN }}
EOF
git commit -am "add step needing API_TOKEN" && git push

It fails with MISSING token. Diagnose: the env | sort step shows API_TOKEN is absent (not masked-but-present) — it isn’t in scope because you never defined it. Fix: gh secret set API_TOKEN --body "dummy-value", re-run, and watch the same step now show the secret masked-but-present and pass. Lesson: “missing” almost always means “not in scope” — and the env dump distinguishes “absent” from “wrong value” in seconds.

4. Fault C — a non-zero exit hidden by a pipe, then a timeout. Show why set -euo pipefail and timeouts matter:

cat >> .github/workflows/ci.yml <<'EOF'
      - name: hidden failure
        run: |
          set -euo pipefail        # remove this line to see the bug hide
          false | cat              # without pipefail, the job stays GREEN despite 'false'
          echo "this line should NOT run"
EOF
git commit -am "demonstrate pipefail" && git push

With set -euo pipefail the job correctly fails at false | cat; delete that line, push again, and watch the job go green despite a failed command — the classic silent failure. Fix: keep set -euo pipefail in every multi-line run. Lesson: an exit code you don’t enforce is an error you won’t see.

5. Fault D — flakiness, and why blind retries lie. Add a deliberately flaky step and observe re-runs:

cat >> .github/workflows/ci.yml <<'EOF'
      - name: flaky
        run: |
          if [ $(( RANDOM % 2 )) -eq 0 ]; then echo "pass"; else echo "fail"; exit 1; fi
EOF
git commit -am "add a flaky step" && git push

Re-run the same commit several times (Re-run all jobs): it passes and fails without any change — the definition of flaky. The tempting “fix” is a retry wrapper; the real fix is removing the non-determinism (here, the RANDOM). Lesson: re-run the same commit to prove flakiness, then fix the source of non-determinism — don’t paper over it with retries.

Validation. A clean walk-through shows: Fault A caught by yamllint before push; Fault B’s env dump distinguishing absent-vs-present and the secret fixing it; Fault C going green-when-it-shouldn’t once pipefail is removed; and Fault D flapping across identical re-runs until the RANDOM is removed.

Cleanup (so nothing is left behind):

gh repo delete ci-ts-lab --yes     # remove the throwaway repo
cd .. && rm -rf ci-ts-lab

Cost note: free. GitHub Actions on a private repo includes a monthly free-minutes allowance; this tiny workflow uses seconds per run. Delete the repo when done to leave no trace.

Common mistakes & troubleshooting

A meta-table — the mistakes engineers make while troubleshooting CI/CD, which keep them stuck or hide real problems:

Best practices

• Make pipelines reproducible by default. Commit lockfiles and install from them; pin actions/images by tag and digest and the toolchain by version (or a fixed container); cache on a lockfile-hash key. Reproducibility is what makes failures debuggable — and most CI-only failures are an environment difference.
• Fail fast and loud. Put set -euo pipefail at the top of multi-line scripts, set a sensible timeout-minutes on every job, and make required checks actually required so a red build blocks merge rather than being ignored.
• Scope secrets tightly and never expose them to forks. Define secrets at the narrowest scope that works (environment over repo over org), pass them explicitly into reusable workflows, and rely on the fork-PR secret boundary — debug “missing value” with an env dump, not by widening scope.
• Prefer ephemeral, daemonless runners. Autoscaled ephemeral runners isolate jobs and shed disk cruft; daemonless builders (Kaniko/BuildKit/Buildah) remove the Docker-in-Docker support burden.
• Keep deploys observable and reversible. Read kubectl describe/--previous before changing anything; use backward-compatible (expand/contract) migrations; make every deploy strategy have a tested rollback; treat an aborted canary as a correct signal, not an obstacle.
• Hunt flakiness as a first-class bug. Measure per-test/per-job flake rate, quarantine proven flakes to a non-blocking lane (still running), fix the determinism, and re-admit only with evidence — a retry is a painkiller, not a cure.
• Use OIDC, not long-lived keys. Federate the pipeline’s identity to the cloud so a leaked log or artifact isn’t a standing breach, and so deploy-auth failures are trust-policy fixes, not key rotations.

Security notes

Troubleshooting a pipeline repeatedly puts you near its most sensitive assets — credentials, the registry, and the production environment — so do it safely. Never widen a secret’s scope to make a job pass: a “missing” value is almost always a scoping problem, and the fix is to grant it at the right level, not to expose it more broadly; in particular, secrets are deliberately withheld from fork PRs, and pull_request_target runs with secrets — never use it to check out or run untrusted code, which is a real exfiltration path. Debug logs can leak secrets: ACTIONS_STEP_DEBUG/CI_DEBUG_TRACE/System.Debug print far more, and masking only catches known values — capture debug output to a controlled location, scrub before sharing, and disable it again after. When a registry or cloud call returns 403, resist granting broad permissions (push/admin everywhere, or a wildcard IAM policy) just to make it pass — confirm who the identity is and grant the minimum missing scope; over-broad CI identities are a classic escalation path. Prefer OIDC/short-lived credentials over long-lived keys so a leaked artifact isn’t a standing breach, and pin actions/images by digest so a compromised upstream tag can’t silently change what your pipeline runs. Finally, treat self-hosted runners as security boundaries: they often hold cloud credentials and run arbitrary PR code, so isolate them (ephemeral, least-privilege, no shared state), and never run untrusted forks on a runner with production access. These themes are developed in the DevSecOps pipeline lesson and the keyless OIDC deploys lesson.

Interview & exam questions

1. A pipeline that passed an hour ago is now red with no code change. Walk me through your response. First, don’t re-run blindly — re-run the same commit once to test whether it’s flaky vs genuinely broken. Read the first error and the exit code; name the stage (build/test/publish/deploy) and layer (code/pipeline/runner/registry/environment). Common no-change culprits: a moved dependency or floating tag, an expired cache, an expired token, a registry rate limit, or a flaky test. Fix the specific cause and add the guardrail (pin the version/digest, fix the cache key) so it can’t recur.
2. A secret is empty in a job even though you “set it”. How do you debug this? It’s a scope problem, not a missing value. Add an env | sort debug step: a secret in scope prints masked but present (***), while one out of scope is absent — that instantly distinguishes the two. Then grant it at the right scope (environment/repo/org), pass it explicitly into reusable/called workflows, and remember fork PRs don’t receive secrets by design.
3. What do exit codes 137, 143, 124, and 127 tell you? 137 = SIGKILL, almost always OOM (out of memory) — reduce memory or use a bigger runner. 143 = SIGTERM, usually a cancelled/terminated job. 124 = a timeout (e.g. timeout wrapper). 127 = command not found (toolchain/PATH problem). Reading the code first often names the failure class before you read a line of the message.
4. A job sits “Queued” forever while the runners show “idle”. What’s happening? A label/tag mismatch: runners are online but none matches the job’s requested labels (runs-on/tags/demands). The job hasn’t started, so it can’t be a code bug. Fix by aligning labels — add the required label to a runner or correct the job’s requested label — and check capacity/concurrency if runners are genuinely all busy.
5. Distinguish 401 from 403 at a container registry, and how you’d fix each. 401 Unauthorized = not authenticated — no/expired token, not logged in, or wrong registry; fix the login/credential step. 403 Forbidden = authenticated but not permitted — the identity lacks push/pull on that repo; fix the RBAC/scope (least privilege). Confusing the two sends you to the wrong fix.
6. Your deploy fails with Not authorized to perform sts:AssumeRoleWithWebIdentity (or AADSTS70021). Cause and fix? An OIDC trust mismatch: the cloud rejected the federated token because the role/credential’s trust conditions don’t match the token’s sub/aud claims (wrong branch/environment/repo), or the workflow lacks id-token: write. Fix the trust policy to match the actual subject/audience and grant id-token: write — don’t fall back to long-lived keys.
7. A Kubernetes pod is ImagePullBackOff after deploy; another is CrashLoopBackOff. How do you tell them apart and fix each? ImagePullBackOff/ErrImagePull is a registry/auth problem — kubectl describe pod shows the exact pull error (wrong tag, missing imagePullSecret, unreachable registry); fix the image reference or pull credentials. CrashLoopBackOff is an application problem — kubectl logs --previous shows why it started and died (bad config/secret/migration); fix the app config and roll back while you fix forward.
8. What makes a pipeline “flaky”, and what’s the right workflow to deal with it? Flaky = passes/fails on the same commit with no change — non-determinism (test order/shared state, timing/races, external deps, resource contention, time/randomness). The workflow: measure flake rate, quarantine proven flakes to a non-blocking lane (still running, so still visible), fix the determinism (isolate state, wait on conditions, mock externals, control the clock/seed), and re-admit only after N green runs. A blanket retry on a unit test hides real regressions.
9. A build passes locally but fails in CI with a missing native module. Most likely cause? A cache restored against a different lockfile/OS/arch, or a toolchain difference between laptop and runner. Diagnose by comparing the cache key to the lockfile and printing the tool versions in CI. Fix by keying the cache on the lockfile hash + OS/arch (bust it once to recover) and pinning the toolchain/container so the environment matches.
10. docker: Cannot connect to the Docker daemon on a self-hosted runner. What are your options? The runner has no Docker (or the user isn’t in the docker group). Options: install/enable Docker and add the user to the docker group; mount the host Docker socket; use Docker-in-Docker (privileged, with DOCKER_HOST/TLS configured); or — best — a daemonless builder (Kaniko/BuildKit/Buildah) that needs no daemon at all.
11. Why are secrets withheld from fork pull requests, and what’s the danger of pull_request_target? Because a fork PR contains untrusted code; exposing secrets to it would let anyone exfiltrate them by editing the workflow/build. pull_request_target runs in the base repo’s context with secrets but checks out the base — so running the fork’s code under it (e.g. building/testing the PR) would hand secrets to untrusted code. Never run untrusted code with secrets in scope.
12. A deploy “succeeds” but users still hit the old version. What likely went wrong? The traffic switch didn’t complete — the slot swap, service selector, or ingress/LB weight wasn’t actually changed, or DNS/LB caching is serving the old target. Verify the router/ingress/slot points at the new version and account for DNS TTL / LB warm-up before declaring success; “deployed” is not “receiving traffic”.

Quick check

1. A job has been “Queued” for 15 minutes while the runners page shows two idle runners. Is this most likely a code bug, and what’s the first thing to check?
2. A step’s env | sort shows your secret as *** (masked). Is the secret in scope or out of scope, and what does that rule out?
3. A docker pull of a public base image starts failing intermittently with 429 toomanyrequests only when many jobs run at once. What’s the cause and two fixes?
4. A test passes when run alone but fails when the suite runs in parallel. What category of flake is this, and what’s the fix (not a retry)?
5. A deploy fails with 403 Forbidden after a successful docker login. Is this authentication or authorisation, and where do you look?

Answers

1. Not a code bug — a queued job never started, so your script can’t be the cause. First check is a label/tag mismatch: the idle runners’ labels don’t match the job’s runs-on/tags/demands. Align the labels (or check capacity/concurrency if they’re actually busy).
2. In scope — a masked *** means the secret reached the job (present but redacted). That rules out “not set / wrong scope / fork PR”; if auth still fails, the problem is the secret’s value/permission, not its visibility.
3. Docker Hub’s anonymous pull-rate limit, hit by many parallel pulls. Fixes: authenticate the pulls (higher limit) and cache/pin by digest (or use a pull-through-cache registry) so you pull the image once instead of per-job.
4. Test ordering / shared mutable state (it depends on isolation that parallelism breaks). Fix the determinism: give each test fresh fixtures and remove shared state; randomise order in CI to surface such bugs — don’t wrap it in a retry, which hides the real defect.
5. Authorisation — you authenticated (login succeeded), but the identity lacks permission (push/pull) on that repository. Look at the registry’s RBAC/scope for that principal and grant the minimum needed; a 401 (not 403) would have meant an authentication/credential problem instead.

Exercise

Build your own CI break-and-fix runbook (timed, free). On a fresh GitHub repo with a small Actions workflow, plant one fault per layer and prove you can diagnose each from observation alone — before you fix it.

1. Scaffold a workflow (checkout + a trivial build/test), confirm it’s green.

2. Plant five faults, one per layer:

   • Pipeline (parse): introduce a YAML indentation/tab error and capture how it fails before any step.
   • Pipeline (scope): reference a secret you haven’t set; capture the env-dump showing it absent, then set it and show it masked-but-present.
   • Build (reproducibility): use an unpinned dependency or a cache key without the lockfile hash; force a mismatch and capture the failure.
   • Runner (labels): request a runs-on/label that no runner provides; capture the job stuck “queued”.
   • Deploy/flake: add a RANDOM-based flaky step; re-run the same commit repeatedly and record the pass/fail flapping.

3. For each, write down the stage, the layer, the diagnostic signal (first error / exit code / env dump / runner status), the root cause, and the fix — before fixing. Time yourself: under 6 minutes per fault.

4. Fix each with the smallest, most deterministic change (correct indentation; grant the secret at the right scope; pin the dep + lockfile-hash cache key; fix the label; remove the non-determinism), and verify each yields a green run from a clean state.

5. Self-assess:

   Criterion	Target
   Identified the correct stage + layer before touching anything	All 5
   Found the root cause from the log/exit-code/env/runner status (not guessing)	All 5
   Fixed with the smallest, most deterministic change	All 5
   Verified a clean green run afterwards (fresh cache where relevant)	All 5
   Whole drill completed	Under 30 minutes

6. Cleanup: delete the throwaway repo.

Cost note: free — a tiny Actions workflow on a private repo uses seconds of the monthly free-minutes allowance; delete the repo when done.

Certification mapping

• AWS DevOps Engineer (DOP-C02), Azure DevOps Engineer (AZ-400), Google Cloud DevOps Engineer — these exams test CI/CD operationally, which is exactly this lesson: diagnosing a failed pipeline, fixing OIDC/identity trust for keyless deploys, resolving artifact-registry auth (401 vs 403) and rate limits, triaging a stuck/failed deployment (ImagePullBackOff, rollout health, rollback), and securing self-hosted runners/agents. The scenario questions (“a deploy fails with an OIDC error”, “a pod won’t pull its image”, “a build is flaky”) map directly to the playbooks here.
• DevOps Institute — DevOps Foundation / DevSecOps — the method (isolate the stage/layer, fix the root cause, prevent recurrence) and the secret-handling/least-privilege themes align with the foundation and DevSecOps syllabi.
• CKA / CKAD — the deployment-failure playbook is core Kubernetes troubleshooting: kubectl describe/logs --previous, ImagePullBackOff vs CrashLoopBackOff, rollout status/undo, probes and resource/quota issues are directly examinable.
• GitHub Actions / GitLab certifications — variable/secret scope, expressions and contexts, reusable workflows, runners/agents, and caching are tested; the pipeline-layer and runner playbooks here cover those objectives.

Glossary

• Stage / job / step — the nesting of a pipeline: a stage groups jobs, a job runs on one runner and contains ordered steps. Naming where a failure sits is half the diagnosis.
• Runner / agent — the machine (hosted or self-hosted) that executes a job. “Queued forever” usually means no runner with matching labels is available.
• Exit code — the numeric result of a command; 0 = success. Key non-zero codes: 137 (OOM/SIGKILL), 143/130 (terminated/cancelled), 124 (timeout), 127 (command not found), 126 (not executable).
• Cache (CI) — stored dependencies/artifacts reused between runs to speed builds; a key that omits the lockfile hash can restore stale artifacts and cause baffling failures.
• Lockfile — the pinned dependency manifest (package-lock.json, poetry.lock, etc.); installing from it (npm ci) makes builds reproducible.
• Secret scope — the level at which a secret is defined and visible (org / repo / environment / job). Secrets are deliberately withheld from fork PRs.
• Type coercion (YAML) — YAML interpreting unquoted scalars as booleans/numbers/null (the Norway problem: no→false; 1.20→1.2); quote ambiguous values.
• Docker-in-Docker (DinD) — running a Docker daemon inside a CI container to build images; needs privileged mode/TLS config. Daemonless builders (Kaniko/BuildKit/Buildah) avoid it.
• Docker socket — /var/run/docker.sock; mounting the host’s socket lets a job use the host daemon instead of DinD (with security trade-offs).
• OIDC (keyless) auth — federating a pipeline’s identity to a cloud via short-lived tokens instead of long-lived keys; failures are trust-policy (sub/aud) mismatches, not bad passwords.
• 401 vs 403 — 401 Unauthorized = not authenticated (login/token problem); 403 Forbidden = authenticated but not permitted (RBAC/scope problem).
• Pull-rate limit — a registry cap on pulls (notably Docker Hub for anonymous/free); causes intermittent 429 toomanyrequests under parallel load.
• Immutable tags — a registry policy forbidding overwriting a published tag; a re-push of the same version then fails with a conflict.
• ImagePullBackOff / ErrImagePull — Kubernetes can’t pull the image (wrong tag, missing imagePullSecret, unreachable registry); the pod Events name the cause.
• CrashLoopBackOff — the container starts then repeatedly crashes (bad config/secret/migration/command); kubectl logs --previous explains why.
• Rollout (stuck “progressing”) — a deployment that never becomes healthy — failing readiness probes, insufficient resources, or a quota/PDB blocking; resolved by fixing the probe/resources or rolling back.
• Flaky test/pipeline — passes/fails on the same commit with no change, due to non-determinism; quarantine and fix the root cause rather than masking with retries.
• Quarantine (flaky) — moving a proven-flaky test to a non-blocking lane (still running, still visible) until its determinism is fixed and it’s re-admitted with evidence.
• pull_request_target — a GitHub trigger that runs in the base repo’s context with secrets; dangerous if used to run untrusted fork code.

Next steps

You can now diagnose the everyday CI/CD failures across every stage and layer — red builds, pipeline/YAML and scope errors, runner/agent problems, artifact/registry auth, and deployment failures — and run a disciplined flaky-pipeline workflow. The next lesson steps up from “fix this failure” to “design a delivery platform that rarely produces these failures in the first place”:

• The DevOps Architecting Ladder: From a Single Pipeline to an Internal Developer Platform — the maturity rungs from one CI workflow to a governed, self-service platform.
• Deployment Strategies: Rolling, Blue/Green, Canary, Progressive Delivery & Rollback — the strategies (and rollbacks) whose failures this lesson triages.
• Building a DevSecOps Pipeline: SAST, DAST, SCA & Policy Gates — adding security gates without making the pipeline brittle.
• Keyless GitHub Actions Deployments with OIDC to AWS, Azure, and GCP — the deep dive on the OIDC trust that fixes most deploy-auth failures.
• Terraform Troubleshooting: State, Providers, Drift, Dependencies & Debugging — the IaC counterpart to this CI/CD playbook.