The DevOps Architecting Ladder: From a Single Pipeline to an Internal Developer Platform

The most expensive DevOps mistake is not a broken build; it is building the wrong altitude of delivery platform for the team in front of you. One organisation reads too many platform-engineering conference talks and stands up Backstage, golden paths, ephemeral environments and a self-service portal to serve eight engineers who deploy a single monolith twice a week — then spends more effort maintaining the platform than the platform ever saves. Another runs a revenue-bearing product on a hand-clicked deploy from a senior engineer’s laptop, with no tests in the path and no way to roll back, and discovers the gap the Friday afternoon that engineer is on leave and the release is on fire. Same root cause: the delivery architecture was chosen by aspiration or inertia, not derived from how the team actually ships and what failure it can tolerate.

This lesson teaches DevOps delivery as a ladder — six rungs, each adding rigour, safety, or self-service capability over the last. You start with a single continuous-integration workflow that builds and tests on every push, and you climb only as far as the team, the risk, and the rate of change push you. For every rung we walk the same five questions: the scenario and its requirements (team size, deploy frequency, risk tolerance, compliance), the design and its components, the key decisions and trade-offs, and the question most DevOps articles skip — “when is this rung genuinely enough?” — so you know when to stop. We close with an explicit method for choosing your rung, because the right answer for most teams is not the top of the ladder; it is the lowest rung that meets the requirement, operated with discipline.

In a nutshell

Picture building software delivery like running a coffee shop that might one day become a chain. On day one, a single barista tastes each cup before it reaches the customer — that is the only “process” you need. Grow to a few shops and you add written recipes and labelled batches so every cup is identical and traceable; grow again and you add a manager’s sign-off before a new drink hits the menu, then a shared recipe book every shop syncs to automatically, then a health inspector standing in the line, and finally a franchise kit that opens a fully-compliant new shop in a day. Each addition earns its place only when the business is big enough to need it. Nobody hires a franchise operations department to run one espresso machine — and nobody serves a thousand customers a day straight off a hand-scribbled sticky note.

This lesson is that story for software. A delivery pipeline is simply the automated path your code travels from “a developer changed a file” to “it is running for real users.” We lay that path out as a ladder of six rungs, and each rung adds one kind of safety, speed, or self-service over the rung below it. The whole skill is climbing only as high as your team size, your risk, and your rate of change actually push you — because the single most expensive mistake in DevOps is getting the altitude wrong: building a giant platform for a tiny team, or running a real product off a hand-clicked deploy from one person’s laptop.

Why should a beginner care about a topic that sounds senior? Because “which rung does this team need?” is exactly the question an interviewer or a lead will hand you, usually phrased as “design a delivery process for this situation.” Choose the altitude well and the rest is detail; choose it badly and you either drown in machinery you cannot maintain or ship outages you cannot roll back. Everything below teaches you to look at any team and name — and defend — the rung it belongs on.

Here is the entire ladder at a glance, lowest to highest. Skim it now; the rest of the lesson simply zooms into each row.

If you remember nothing else: start at the bottom, climb one rung only when a concrete pain forces you, and stop at the lowest rung that meets the requirement.

Five terms recur and are the levers that move you up the ladder. CI (Continuous Integration): every change is automatically built and tested against the mainline, fast and on every push. CD: Continuous Delivery keeps the mainline in a state that is always releasable (a human still clicks deploy), while Continuous Deployment removes even that click and ships every green build to production automatically. The DORA metrics: the four research-backed measures of delivery performance — deployment frequency, lead time for changes, change-failure rate, and time to restore service. GitOps: a model where the desired state of a system lives in Git and a controller continuously reconciles the running system to match it. IDP (Internal Developer Platform): a self-service layer that lets developers provision and ship through a paved road without filing tickets or learning the plumbing underneath.

Learning objectives

By the end of this lesson you will be able to:

• Map a team’s delivery requirements (size, deploy frequency, risk tolerance, regulatory constraints, number of services and teams) to a specific rung on the DevOps architecting ladder rather than guessing.
• Describe the component set, failure model, and capability of each of the six rungs, from a single CI workflow to a full Internal Developer Platform.
• Explain why each rung exists — the specific failure, risk, or bottleneck the previous rung could not address.
• Reason about the dominant trade-off at every step: engineering investment and operational toil versus delivery speed, safety, and developer leverage.
• Recognise when an organisation has over-built (a platform for one team) or under-built (revenue on a laptop deploy) and recommend the correct rung.
• Apply a repeatable decision method to choose the lowest rung that satisfies the requirement, and know the concrete signals that justify climbing higher.

Prerequisites & where this fits

You should already understand the building blocks the ladder assembles: what a CI/CD pipeline is (stages, jobs, steps), what an artifact and an artifact repository are, and the core deployment strategies — rolling, blue/green, and canary. The ladder is fundamentally about how many of those pieces you run, how they are wired together, and how much of the wiring developers have to see. If pipeline anatomy is not yet second nature, work through the DevOps fundamentals and CI/CD pipeline-design lessons first; if deployment strategies are hazy, read the deployment-strategies lesson before rung 3. It also helps to have built at least one pipeline yourself, because the lower rungs will feel familiar and the higher ones will make sense as additions to what you already run. This lesson sits in the Architecture module of the DevOps Zero-to-Hero course: the earlier lessons teach you to build each component well; this one teaches you how many to build, how to compose them, and where to stop.

Core concepts: what actually changes as you climb

Before the rungs, internalise the axes the ladder moves along. Every rung is a point in this space, and naming the axes lets you reason about a delivery setup you have never seen.

Two principles cut across all five axes and explain most of the ladder’s shape:

• Speed and safety are not opposites on this ladder — they rise together. The classic fear is that adding gates, approvals, and scans slows delivery down. The DORA research shows the reverse for teams that automate well: elite performers deploy more often and fail less, because automation makes each change small, tested, and reversible. Each rung that adds a safety mechanism should also shrink the cost of shipping, not inflate it. If a rung makes you slower without making you safer, you have implemented it wrong.
• Climbing is largely about moving humans off the critical path and replacing them with code. A manual deploy depends on a person being awake, available, and careful. Each rung up replaces a human judgement with an automated, repeatable, auditable mechanism — a test, a gate, a controller, a policy, a template. The cost of each rung is the engineering to build and operate that mechanism; the payoff is that delivery no longer depends on heroics. The top of the ladder is reached when even provisioning a new service needs no human in the loop.

With those in hand, climb.

Rung 1 — A single CI workflow: build and test on every push

Why you start here. Every team, whatever its ambitions, needs one thing before anything else: automatic proof that each change still builds and passes its tests, so a machine — not a person at release time — catches breakage.

Scenario & requirements. A small team — one to a handful of engineers — is building an application and currently integrates by hope: someone merges, something breaks, and an afternoon is lost to “it worked on my machine.” There is no deployment requirement on the table yet, or deployment is still a manual step done rarely. The single concrete need is to stop broken code reaching the mainline: every change must be automatically built and tested so breakage is caught in minutes, not at the next manual release. Deploy frequency is low or irregular, risk tolerance for delivery is high (there may be no production yet), and there is no compliance pressure.

The design & components. One CI workflow triggered on every push and pull request: check out the code, install dependencies (with caching), build/compile, run the unit-test suite and a linter, and report a single red/green status back on the commit and PR. This lives as pipeline-as-code in the repository — a .github/workflows/ci.yml, a .gitlab-ci.yml, an azure-pipelines.yml, or a Jenkinsfile — and runs on hosted runners so there is nothing to operate. A branch-protection rule makes the green check required to merge, which is what turns a passive script into an actual integration gate.

Key decisions & trade-offs. The defining decision is simply to automate integration at all — to make “does it build and pass the tests?” a machine’s answer on every change rather than a human’s at release time. The trade-off you deliberately accept is that this rung does nothing about deployment: getting the artifact to an environment is still manual, unversioned, and unsafe. That is fine when there is no production yet or releases are rare and low-stakes. The quiet trap is a slow or flaky CI: if the suite takes twenty minutes or fails randomly, developers disable the required check or merge around it, and you are back to integrating by hope with extra steps. Keep CI fast and deterministic from day one — it is the foundation every higher rung builds on, and a cracked foundation propagates upward.

When this rung is enough. Enough for a brand-new project before it has users, a learning or side project, a library whose “release” is a manual publish done rarely, or any codebase where the only current pain is broken merges. It is never enough the moment a real environment must be kept up to date from the mainline, or a release happens often enough that doing it by hand is error-prone or a bottleneck. The first time someone says “we need to ship this reliably and repeatedly,” you have left rung 1.

Rung 2 — Full CI/CD: quality gates, artifacts, and an automated deploy

Why you climb here. A real environment now has to be kept current, and hand-deploying it is slow, inconsistent, and impossible to reproduce or safely roll back.

Scenario & requirements. The application now has at least one real environment that must be kept current, and releases happen often enough that hand-deploying is slow, inconsistent, or risky. The team wants a single automated path from commit to a deployed environment, with confidence that what ships has passed more than unit tests, and with the ability to redeploy a known-good version if the latest one is bad. Requirements: deploy on the order of weekly-to-daily, a low-but-nonzero tolerance for delivery mistakes, and a need to know exactly what is running (which version, built from which commit).

The design & components. The rung-1 CI workflow is extended into a full CI/CD pipeline — source → build → test → scan → package → publish → deploy → verify:

• More quality gates beyond unit tests. Code-coverage thresholds, integration tests, and a first security scan (at minimum dependency/SCA and secret detection) become gates that can fail the build. Each gate is a place a bad change stops.
• Immutable, versioned artifacts. The build produces a versioned artifact — a container image, an npm/Maven/PyPI/NuGet package, a Helm chart — and publishes it to an artifact repository (GitHub Packages, Azure Artifacts, Nexus, Artifactory, a container registry). The artifact is built once and then promoted, never rebuilt per environment, so what you tested is byte-for-byte what you deploy.
• An automated deploy step. The pipeline deploys the published artifact to the environment — to a VM, an App Service, a container host, or a cluster — replacing the manual step. This is Continuous Delivery if a human clicks to release the final deploy, or Continuous Deployment if every green build ships automatically.
• Keyless cloud auth (OIDC). The pipeline authenticates to the cloud with short-lived OIDC tokens via a federated identity rather than long-lived stored secrets — fewer credentials to leak or rotate.
• Pipeline-as-code with reuse. Reusable workflows / templates / shared libraries keep the pipeline DRY as it grows.

Key decisions & trade-offs. The defining decision is introducing an immutable, promoted artifact and an automated deploy — the shift from “CI tells me it’s green” to “a pipeline gets a known artifact into an environment by itself.” A second decision is Continuous Delivery versus Continuous Deployment: auto-deploy every green build (fastest feedback, but demands real test confidence and easy rollback) or require a human click to release (a safety brake while confidence is still building). The trade-off you still accept is that this is typically one environment, or a couple wired ad hoc, with no formal promotion policy across many stages and no sophisticated deploy strategy — a deploy here is often a straightforward replace or rolling update. The classic trap is rebuilding the artifact per environment, which silently breaks the “test once” guarantee; build once, promote the same bits. Another is caring about scan reports but never gating on them — a scan that cannot fail the build is documentation, not a control.

When this rung is enough. Enough for a single application with one or two environments, a team shipping weekly-to-daily, where a straightforward automated deploy plus the ability to redeploy a previous artifact is sufficient safety. Most small products and internal services live here productively for a long time. The signal to climb is the arrival of multiple environments that need a governed promotion path (dev → test → staging → prod with approvals), or a risk profile that demands deploying without downtime or to a fraction of users first — both of which are rung 3.

Rung 3 — Multi-environment delivery: promotion, deploy strategies, and approvals

Why you climb here. Real users make a bad deploy expensive, so releases now need a governed, audited promotion path, sign-off before production, and a downtime-free strategy with a rollback you have actually tested.

Scenario & requirements. You now serve real users, and a bad deploy is costly — downtime, errors, or a rollback scramble. You have multiple environments (dev, test/UAT, staging, production) and changes must flow through them in a governed, auditable order, with the right humans signing off before production. The deploy itself must avoid downtime and limit blast radius: you cannot take the service offline to release, and you want to expose a new version to a fraction of traffic before all of it. Requirements: daily-or-more deploys, a real (often contractual) expectation of availability, and an audit trail of who approved what to production and when.

The design & components. A multi-stage delivery pipeline that promotes one artifact through a chain of environments, with deploy strategies and approval gates:

• Environment promotion of a single artifact. The same immutable artifact is promoted dev → test → staging → prod; each stage deploys that artifact and runs its own checks (smoke tests, integration tests, manual QA). Configuration is per-environment (config and secrets injected at deploy), but the code/artifact is identical across the chain.
• Deployment strategies that limit blast radius. Instead of a blunt replace, production deploys use rolling, blue/green, or canary releases (App Service slots, two target groups, or a progressive rollout), so a bad version is caught before it reaches everyone and a rollback is a fast, deterministic switch back to the last good version.
• Approval gates and protected environments. Lower environments may auto-deploy; staging and production require manual approval by designated reviewers, implemented as GitHub Environments with required reviewers, Azure DevOps approvals and checks, or equivalent. This is both a safety brake and the audit trail.
• Feature flags to decouple deploy from release. Where appropriate, code ships dark behind a flag and is released by flipping the flag rather than by a deploy — separating “the bits are in production” from “users see the feature.”
• Per-environment validation and automated rollback. Each stage gates on health (smoke/synthetic checks); a failed post-deploy check triggers an automatic rollback or halts the promotion.

Key decisions & trade-offs. The defining decision is how each environment deploys and who must approve the jump to production — the move from “a pipeline deploys” to “a pipeline promotes, safely and with sign-off.” A core sub-decision is the deployment strategy (rolling is simplest and cheapest; blue/green gives instant switch and rollback at the cost of double capacity during cutover; canary gives the smallest blast radius and metric-driven confidence at the cost of more tooling and traffic-shaping). The trade-off you deliberately accept is added latency and process at the production boundary — approvals and progressive rollouts make each release a little slower, which is the price of not breaking users; keep lower environments fully automatic so the friction sits only where the risk is. The traps here are approvals that are rubber-stamps (an unread “approve” is theatre, not a control — pair it with automated gates that catch what humans won’t) and a rollback path that has never been tested (a rollback you have not exercised is a hope, not a capability).

When this rung is enough. Enough for the large majority of production applications: a real service with several environments, governed promotion, a downtime-free deploy strategy, audited production approvals, and tested rollback. Many serious products run here for years and never need more. Do not climb for novelty. The signals that justify the next rungs are specific and different from each other: running on Kubernetes at a scale where push-based deploys and config drift become the problem points to GitOps (rung 4); a security or compliance mandate that the pipeline itself enforce supply-chain and policy controls, or a need to measure and improve delivery performance, points to DevSecOps + DORA (rung 5); and many teams needing to ship independently without the platform team as a bottleneck points to an IDP (rung 6).

Rung 4 — GitOps to Kubernetes: Git as the source of truth, a controller as the deployer

Why you climb here. You run Kubernetes, and the push model has started to bite — the cluster drifts from what anyone intended and the CI system holds cluster credentials — so Git should become the source of truth and a controller should do the deploying.

Scenario & requirements. Your workloads run on Kubernetes (one cluster or several), and the push-based model from rung 3 is starting to hurt: the CI system holds cluster credentials and pushes changes in, the actual cluster state drifts from what anyone intended (someone kubectl applyed a hotfix at 2 a.m.), and reconstructing “what is supposed to be running” means reading pipeline logs. You want the desired state of every environment to live in Git, the running cluster to be continuously reconciled to match it, drift to be detected and corrected automatically, and rollback to be a Git revert. Requirements: Kubernetes as the runtime, multiple environments and possibly clusters, and a need for an auditable, declarative, single source of truth.

The design & components. A GitOps delivery model layered on the rung-3 pipeline:

• Git as the single source of truth. A Git repository (or environment-specific directories/branches) holds the declarative desired state — Kubernetes manifests, Helm values, or Kustomize overlays — for each environment. Nothing is changed by hand on the cluster; you change Git.
• A reconciliation controller in the cluster. Argo CD or Flux runs inside the cluster, watches the Git repo, and continuously reconciles the live state to the declared state — applying changes, detecting drift, and (optionally) self-healing back to Git when something is changed out of band.
• A pull-based, credential-inverted flow. CI’s job shrinks to build, test, scan, publish the artifact, and update the image tag/manifest in Git; the controller pulls the change into the cluster. The cluster’s credentials never leave the cluster — a significant security improvement over CI holding cluster admin.
• Declarative rollback and environment promotion. Rolling back is reverting the Git commit; promoting an environment is a pull request that bumps the manifest in the next environment’s path. Progressive delivery (Argo Rollouts / Flux + Flagger) plugs in for canary/blue-green on top of GitOps.
• App-of-apps / fleet patterns. For many apps or clusters, an app-of-apps (Argo) or a Flux source/Kustomization inventory delivers config consistently across them from one source of truth.

Key decisions & trade-offs. The defining decision is inverting the deploy model from push to pull and making Git the single source of truth — the system now converges on what Git says rather than on whatever the last pipeline did. A real sub-decision is how strict to make reconciliation (auto-sync and self-heal give the strongest drift guarantees but mean all change must go through Git — no emergency kubectl edit; manual sync is gentler but lets drift persist). The trade-off you accept is a new component to operate and a model shift the team must learn — Argo CD or Flux is one more thing to run and secure, and “you change Git, not the cluster” is a genuine behaviour change. The traps are leaving a back door open (allowing direct kubectl writes alongside GitOps defeats the single-source-of-truth guarantee — drift creeps back) and treating GitOps as only for Kubernetes app manifests while infrastructure and policy still drift; the leverage compounds when everything the cluster needs is reconciled from Git. Note that GitOps is largely a Kubernetes-shaped tool — if you do not run Kubernetes, you typically stay on rung 3’s push model and reach the higher rungs through it.

When this rung is enough. Enough when you run Kubernetes and want declarative, drift-corrected, Git-audited delivery with credential-inverted security — which is most serious Kubernetes shops. It does not, on its own, add security supply-chain controls or delivery measurement, and it is orthogonal to the platform question: GitOps can serve one team or be the backend of a self-service platform. You climb to rung 5 when security/compliance demands that the pipeline enforce supply-chain and policy controls, or when you need to measure delivery to improve it; you climb to rung 6 when many teams need to self-serve.

Rung 5 — DevSecOps with observability and DORA: security in the path, performance measured

Why you climb here. Security is not yet enforced inside the path to production and you cannot tell whether delivery is getting better or worse — you need gates that block real risk and metrics that prove performance.

Scenario & requirements. Delivery is automated and (often) GitOps-driven, but two gaps now bite. First, security is not yet a first-class part of the path to production: scanning is ad hoc, the software supply chain is unverified, and there is no machine-enforced policy on what may ship — a real problem under regulatory pressure (SOC 2, ISO 27001, PCI, FedRAMP) or simply for a team that takes supply-chain attacks seriously. Second, you cannot tell whether delivery is getting better or worse, because you do not measure it. Requirements: shift security left into the pipeline as enforced gates, establish supply-chain integrity (signed, attested, provenance-tracked artifacts), enforce policy-as-code, and instrument both the pipeline and the running system so you can track the DORA metrics and SLOs and actually improve.

The design & components. A DevSecOps layer plus observability and DORA measurement woven through the rung-3/4 pipeline:

• Security gates across the pipeline. SAST (static analysis of your code), SCA (dependency/known-CVE scanning), secret detection, container image scanning (e.g. Trivy/Grype), IaC scanning (Checkov/tfsec/KICS), and DAST (dynamic testing against a running staging instance) — each placed where its signal-to-noise is best and gated by severity and reachability, so the build blocks on real exploitable risk without burying developers in noise. (Full wiring in Building a DevSecOps Pipeline: SAST, SCA, and Policy Gates.)
• Software-supply-chain integrity. Generate an SBOM (software bill of materials), sign artifacts with Sigstore/cosign, produce build provenance (SLSA attestations), and verify signatures at admission so only signed, attested images run. The pipeline becomes a trustworthy, tamper-evident producer of artifacts.
• Policy-as-code as an unbypassable gate. OPA/Conftest in the pipeline and Kyverno or OPA Gatekeeper at the cluster admission boundary enforce the rules every workload must obey (no privileged pods, required labels, allowed registries, signed images) in code, so they cannot be skipped.
• Observability of the running system. Metrics, logs, and traces (Prometheus/Grafana, OpenTelemetry, the cloud’s native stack) with SLOs and error budgets and alerting — you cannot operate to a reliability target you cannot see.
• DORA metrics instrumentation. Capture deployment frequency, lead time for changes, change-failure rate, and time to restore from your pipeline and incident data, and review them to drive improvement (the DORA metrics lesson shows how to compute them from pipeline events).

Key decisions & trade-offs. The defining decision is making security and measurement first-class, automated parts of delivery rather than bolt-ons — the pipeline now enforces trust and reports performance. The critical calibration is gate strictness and signal-to-noise: gate on severity and reachability, fail on real risk, and route the rest to a backlog — because the well-known failure mode of DevSecOps is so much false-positive noise that developers disable the security stage to ship (covered head-on in the DevSecOps lesson). The trade-off is up-front engineering and ongoing tuning — scanners, signing, SBOMs, policy bundles, and an observability stack are real investments that need maintenance, and badly-tuned gates do slow teams down. The traps: scanning without gating (advisory-only scans change nothing), collecting DORA as a vanity metric or a stick (DORA is a team diagnostic to find bottlenecks, not a leaderboard to punish individuals), and adding security gates without easy rollback and observability — security that finds problems but cannot recover from them is half a control.

When this rung is enough. Enough for security-conscious and regulated organisations that need enforced supply-chain integrity, policy-as-code, and the ability to prove and improve delivery performance — which is where most mature single- or few-team engineering organisations should sit. It is the right ceiling for many. The single signal that justifies the top rung is organisational scale: when many teams must each get a secure, observable, multi-environment, GitOps-or-CD pipeline, and replicating all of rung 1–5 by hand for every new team or service becomes the bottleneck. Productising that is rung 6.

Rung 6 — An Internal Developer Platform: golden paths, self-service, ephemeral environments

Why you climb here. Many teams are each hand-building the same secure pipeline, so the bottleneck is no longer any missing capability — it is the per-team toil of replication.

Scenario & requirements. The driver here is organisational scale and developer cognitive load, not a single missing capability. You have many teams and many services, and although you know how to build a secure, observable, GitOps-driven pipeline (rungs 1–5), doing it by hand for every new service is the bottleneck: every team reinvents the same pipeline, slightly differently; onboarding a new service is a multi-week ticket-driven slog; and developers spend their time wiring CI, registries, environments, and policy instead of writing features. The requirement is self-service with guardrails baked in — a developer should get a complete, conformant, production-ready service through a paved road in minutes, without the platform team in the loop and without being able to skip the controls. This rung is for platform-engineering at organisational scale; most organisations should be sure they belong here before climbing.

The design & components. An Internal Developer Platform that productises rungs 1–5 as self-service:

• A developer portal and service catalog. Backstage (or an equivalent IDP) gives a single front door: a catalog of all services with their owners, docs, and health, plus the entry point to the paved road. (Build details in Standing Up Backstage as an Internal Developer Portal.)
• Golden paths / software templates. A scaffolder template generates, from a short form, a complete new service: repository, a rung-2-through-5 pipeline (build, test, security gates, signing, artifact publishing), GitOps wiring, environments, observability, dashboards, and policy — all conformant by construction. The platform team curates the road; developers drive on it.
• Self-service infrastructure with guardrails. Developers request databases, queues, namespaces, and environments through the platform, which provisions them via Terraform/Crossplane behind policy-as-code so every resource is compliant, tagged, and within budget — no tickets, no drift from standard.
• Ephemeral / preview environments. Each pull request can spin up a short-lived environment (often a vCluster or a namespace, deployed via the same GitOps machinery) for review and testing, then tear it down automatically — fast feedback without permanent environment sprawl.
• The platform built on the lower rungs. Crucially, the IDP does not replace rungs 1–5; it packages and serves them. Every service the platform scaffolds gets the same CI, CD, GitOps, DevSecOps, and DORA instrumentation — the platform’s value is that teams get all of it by default and uniformly, with org-wide DORA and reliability dashboards as a natural by-product.

Key decisions & trade-offs. The defining decision is to productise delivery as a self-service platform — to treat the paved road as an internal product with the platform team as its owner and developers as its customers. The dominant trade-off is large up-front and ongoing investment for long-run leverage: an IDP is a real product needing a dedicated team, a roadmap, and maintenance, and it only pays off at scale — below roughly five or six teams (or a handful of services), the platform machinery costs more than the toil it removes, and you should stay on rung 5 and copy a good pipeline template by hand. The traps are building the platform nobody asked for (a portal demos beautifully and rots quietly into a catalog of stale entities if it does not solve real developer pain — let demand pull each capability), golden paths so rigid that teams route around them (a paved road must be genuinely easier than the alternative and allow escape hatches, or it becomes shelfware), and mistaking the portal for the platform (Backstage is a UI; the platform is the templates, the GitOps backend, the policy, and the provisioning underneath — the frontend is the least important part).

When this rung is enough — and when it is too much. The right rung only when many teams shipping many services makes the per-team cost of rungs 1–5 the binding constraint, and a self-service paved road demonstrably removes more toil than it creates. Otherwise it is over-engineering — which has its own failure mode: a platform a small organisation cannot staff or evolve becomes a half-built, abandoned tab that is worse than letting each team copy a solid pipeline. The honest test: if you cannot name the teams that will use the golden path next quarter and the specific toil it removes for them, you do not yet need rung 6.

How to choose your rung

The diagram above lays the six rungs side by side — what each one adds over the rung beneath it, the specific problem that justifies the climb, and roughly when it fits — so you can find the lowest rung that meets your situation at a glance instead of defaulting to the top.

The method is deliberately boring, because boring is how you avoid both expensive failure modes — over-building and under-building:

1. Describe how you actually ship today, with numbers, before designing anything. State your team and service count, deploy frequency, the manual steps still in the path, your tolerance for a bad release, your runtime (Kubernetes or not), and any compliance constraint. If you cannot describe the current path concretely, you are not ready to choose a rung — go map it.
2. Map each requirement to the lowest rung that satisfies it. “Stop broken merges” → rung 1; “one automated, traceable path to an environment” → rung 2; “governed multi-env promotion with safe deploys and approvals” → rung 3; “declarative, drift-corrected delivery on Kubernetes” → rung 4; “enforced security/supply-chain and measured delivery” → rung 5; “self-service for many teams” → rung 6. Take the highest rung any single hard requirement forces — but no higher.
3. Default to the lowest sufficient rung, operated with discipline. A rung-3 pipeline run well beats a rung-6 platform a team cannot maintain. The lowest sufficient rung minimises engineering cost, operational toil, and the complexity that is itself a source of delivery failures.
4. Separate the independent climbs. Three things move semi-independently: delivery rigour (rungs 1→2→3), the runtime model (rung 4 GitOps, which mostly applies if you run Kubernetes), and organisational scale (rung 6 IDP, driven by team count). Security and measurement (rung 5) layer onto whichever delivery model you run. Do not let “we should do GitOps” or “we need a platform” stampede you past the rigour you actually lack.
5. Climb only on a concrete signal. The signals are specific: broken merges → rung 1; a real environment to keep current → rung 2; multiple environments, a costly bad deploy, or an audit requirement → rung 3; Kubernetes plus painful drift and credential exposure → rung 4; a compliance/supply-chain mandate or a need to measure delivery → rung 5; many teams bottlenecked on hand-built pipelines → rung 6. Re-run the assessment on material change (a compliance regime, a move to Kubernetes, a jump in team count), not continuously.

The unifying idea is that the component set is stable and only the composition and ownership change as you climb. Build, test, artifact, deploy, gate, observe — the same primitives appear at every rung; what changes is how many environments they span, whether a controller or a human drives them, whether security and measurement are woven in, and whether developers self-serve the whole thing. That stability of shape is what makes climbing incremental rather than a rewrite.

Hands-on lab: feel the bottom two rungs locally

You cannot stand up a full IDP on a laptop, but you can viscerally experience the difference between rung 1 (build-and-test only) and the rung-2 idea of an immutable, versioned artifact that you build once and could then promote — using only Git, Docker, and a tiny app. This lab is free, runs in minutes, and makes the abstract concrete. Requirements: Docker and Git installed (no cloud account, no CI service needed — we simulate the CI logic locally).

Step 1 — Rung 1: a local CI gate (build + test)

Create a trivial project with a test, and a script that is a rung-1 CI workflow — it builds and tests, and exits non-zero on failure (exactly what a required status check enforces).

mkdir -p /tmp/ladder-lab && cd /tmp/ladder-lab && git init -q
cat > app.py <<'EOF'
def add(a, b):
    return a + b
EOF
cat > test_app.py <<'EOF'
from app import add
def test_add():
    assert add(2, 3) == 5
EOF
cat > ci.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
echo "== CI: build/syntax check =="
python -m py_compile app.py test_app.py
echo "== CI: unit tests =="
python -m pytest -q
echo "CI PASSED (green) — this is what a required status check enforces"
EOF
chmod +x ci.sh
./ci.sh

Expected output: the syntax check passes, pytest reports 1 passed, and you see CI PASSED (green). Now break the test (assert add(2, 3) == 6) and re-run ./ci.sh — it exits non-zero and prints the failure. That red/green exit code is rung 1’s entire value proposition: a machine, not a human, decides whether the change is safe to merge.

Step 2 — Rung 2: build an immutable, versioned artifact

Package the app into a versioned container image tagged with the Git commit — the rung-2 idea that you build once and then promote the same bits, with full traceability from running version back to commit.

cd /tmp/ladder-lab
git add -A && git commit -q -m "rung-2 lab app"
cat > Dockerfile <<'EOF'
FROM python:3.12-slim
WORKDIR /app
COPY app.py .
CMD ["python", "-c", "from app import add; print('add(2,3)=', add(2,3))"]
EOF

SHA=$(git rev-parse --short HEAD)
docker build -t ladder-app:"$SHA" -t ladder-app:latest .
echo "Built immutable artifact ladder-app:$SHA (also tagged :latest)"
docker images ladder-app

Expected output: a Docker build that ends in Successfully tagged ladder-app:<sha>, and docker images listing the image under two tags pointing at one image ID — the commit SHA (the immutable, promotable identity) and latest (a moving pointer). That single artifact, identified by commit, is what a real rung-2 pipeline would publish to a registry and then promote unchanged through environments.

Step 3 — Validation: prove “build once, run the same bits”

“Deploy” the artifact by running it, then confirm the tagged image is the exact thing built — the guarantee that what you tested is what runs.

cd /tmp/ladder-lab
SHA=$(git rev-parse --short HEAD)
echo "== 'Deploying' the promoted artifact by digest-identity =="
docker run --rm ladder-app:"$SHA"
echo "== Proof the SHA tag and latest are the SAME image (built once) =="
docker inspect --format '{{.Id}}' ladder-app:"$SHA"
docker inspect --format '{{.Id}}' ladder-app:latest

Validation criterion: the container prints add(2,3)= 5, and the two docker inspect image IDs are identical — proving the commit-tagged artifact and the deployed one are byte-for-byte the same image, built once. That is the rung-2 promise (test once, ship the same bits) made concrete; a rung-1 setup has no artifact at all, so this guarantee simply does not exist below rung 2.

Cleanup

docker rmi ladder-app:latest ladder-app:"$(cd /tmp/ladder-lab && git rev-parse --short HEAD)" 2>/dev/null || true
rm -rf /tmp/ladder-lab

Cost note. This lab is entirely free: it runs in local Docker with no cloud resources, no hosted CI minutes, no registry, and nothing to bill. The real rungs cost money and effort — hosted CI/CD minutes and a registry (rung 2), multiple environments and load balancers for deploy strategies (rung 3), an Argo CD/Flux controller to operate (rung 4), scanners and an observability stack (rung 5), and a dedicated platform team for an IDP (rung 6) — which is exactly why you climb only as far as the requirement demands.

Going deeper: the forces that actually push you up the ladder

The rungs are what you see — pipelines, controllers, portals. But a team never climbs because a rung exists; it climbs because an underlying force has crossed a threshold and made the current rung too painful. Naming those forces is what lets you predict the next climb instead of reacting to an outage, and it explains why the ladder is not a straight staircase everyone ascends in lockstep. Five forces do almost all of the work, and each rung is a response to one or more of them.

1. Team and service fan-out. The dominant driver of the top of the ladder is not capability but replication. One team can hand-build a superb rung-5 pipeline. The problem is the tenth team building the tenth slightly-different copy, and the coordination cost of keeping them consistent, which grows faster than the team count itself. Rung 6 exists only because per-team fan-out eventually makes “build it again by hand” more expensive than “productise it once.” The threshold signal is economic: when the marginal cost of onboarding the next service by hand exceeds the amortised cost of a platform that onboards it in minutes — empirically around five or six teams — the fan-out force has crossed the line.

2. Cognitive load. Borrowed from Team Topologies: the total amount a team must hold in its head to ship safely. Every rung adds machinery, and machinery cuts both ways — it can raise load (more tools to understand) or lower it (self-service hides the plumbing). The paved road (rung 6) is, at bottom, a cognitive-load transfer: it moves the extraneous load — wiring CI, registries, environments, policy — off every stream-aligned team and onto a single platform team that owns it as a product. The failure mode is a half-built platform that adds tools without hiding them, so teams carry both the new machinery and the old plumbing — the worst of both worlds, and the usual shape of a premature IDP.

3. Lead time and batch size. Lead time for changes — one of the four DORA metrics — falls as automation depth rises, and the mechanism is batch size. Automation makes each change small, and small changes are faster to verify and safer to ship, which is the real reason speed and safety rise together rather than trading off. Every rung that automates a step shrinks the batch and the blast radius at the same time. The counter-force is that gates and approvals add latency at the boundary where you place them; you defend against it by keeping lower environments fully automatic and spending human latency only at the production boundary, where the risk actually is.

4. Reliability and blast radius. As a service carries more revenue and more users, the cost of a single bad change rises, and that cost is what justifies investment in safety machinery. Rung 3’s deploy strategies (limit who sees a bad version) and rung 5’s observability, SLOs, and error budgets (see and bound failure) are both direct responses to rising blast radius. The error budget is the elegant formalisation of the trade-off: when you are burning budget you slow down and spend on safety; when you have budget to spare you spend it on speed. A pre-launch app has effectively infinite budget, which is precisely why rung 1 can ignore all of this.

5. Governance, compliance, and auditability. External mandates — SOC 2, ISO 27001, PCI-DSS, FedRAMP — convert controls from “nice to have” into “must be enforced and evidenced.” This is the force that turns an advisory scan into a blocking gate (rung 5), turns tribal deploy knowledge into an audited required-reviewer approval (rung 3), and turns “we think that is what is running” into Git history as the deploy log (rung 4). Under this force, auditability is a first-class output of the pipeline, not a side effect — you must be able to prove, after the fact, what shipped, who approved it, and that the controls actually ran.

Three advanced observations make the model sharper:

• The forces are semi-independent, which is why the ladder is not linear. Fan-out drives rung 6; runtime choice drives rung 4; blast radius and governance drive rungs 3 and 5. A two-person team building a regulated payments product can legitimately need rung-5 controls (governance force high) while being years from rung 6 (fan-out force near zero) and never touching rung 4 (not on Kubernetes). Diagnose each force separately; do not assume a high reading on one implies the others.
• Conway’s Law is quietly in charge. A system’s architecture tends to mirror the communication structure of the organisation that builds it — so your delivery architecture will come to reflect your org chart whether you plan it or not. An IDP is, in part, a deliberate inverse-Conway move: a platform team’s boundary is drawn precisely so that every stream team’s pipeline comes out shaped the same way. Reorganise the teams and the delivery architecture will drift to follow.
• Every rung has a cost curve, and complexity is itself a failure source. Each rung carries a fixed cost to build and an ongoing cost to operate and secure; it pays off only once cumulative toil-saved exceeds fixed + ongoing. Below its threshold, a rung’s fixed + ongoing cost dominates — which is the precise, unromantic reason a too-early platform is worse than a copied template. And because added complexity is one of the leading causes of delivery failures, the marginal safety each rung buys diminishes even as its complexity keeps rising. The optimum is therefore never the top of the ladder; it is the lowest rung whose failure model you can actually live with, operated with discipline.

The practical upshot: when you feel the itch to climb, name the force, not the tool. “We should do GitOps” is a tool talking; “our Kubernetes clusters are drifting and CI holds cluster-admin” is a force talking — and only the second is a reason to move.

Common mistakes & troubleshooting

Common beginner mistakes

These are misconceptions about the ladder itself — distinct from the symptom→cause→fix table above. Each is a wrong mental model a newcomer arrives with, and the corrected model to replace it.

“A higher rung is a better, more mature setup — we should aim for the top.” No. Each rung buys its capability with a jump in complexity, and complexity is itself a leading cause of delivery failures. The right model is the lowest sufficient rung, operated with discipline: a rung-3 pipeline run well beats a rung-6 platform a team cannot maintain. “Maturity” is choosing the correct altitude, not the highest one.

“Good engineering orgs build an Internal Developer Platform, so we should build one too.” This is the classic premature IDP, and it is expensive. Rung 6 is driven by team fan-out, not prestige — it pays off only when many teams each hand-building the same pipeline is the binding constraint, roughly north of five or six teams. Below that, the platform’s fixed and ongoing cost dwarfs the toil it removes, and you end up with a half-built portal nobody can staff. The right model: let real demand pull each capability; if you cannot name the teams that will use the golden path next quarter and the specific toil it removes, you do not yet need it.

“GitOps is the next level of maturity, so everyone should adopt it.” GitOps is a Kubernetes-shaped tool, not a universal upgrade. It is an orthogonal choice about your runtime and deploy model, not a rung of delivery rigour stacked above rung 3. If you do not run Kubernetes, you typically reach the higher rungs through rung 3’s push model and GitOps simply does not apply. Adopt it because your clusters are drifting and you want Git as the source of truth — not because it is fashionable.

“You must climb the rungs strictly in order, one at a time.” The rungs are semi-independent axes, not a single staircase. Delivery rigour (rungs 1→2→3), the runtime/GitOps model (rung 4), and organisational scale (rung 6) move independently, and security-and-measurement (rung 5) layers onto whichever delivery model you run. A tiny, heavily-regulated team can genuinely need rung-5 controls while never approaching rung 6. Diagnose each axis on its own.

“CI/CD is a tool you install — pick Jenkins or GitHub Actions and you are done.” The tool is the least important part. The design is what matters: build the artifact once and promote it, gate on real risk, decouple deploy from release, keep a rollback you have tested. The same primitives — build, test, artifact, deploy, gate, observe — appear at every rung; what changes is how they are composed and owned, and that is a design decision no product makes for you.

“We added a security scan / an approval step, so we are secure / governed now.” Only if it can actually block. A scan that cannot fail the build is documentation; an approval nobody reads is theatre. A control is defined by what it can stop, not by what it reports — gate on real, severity-and-reachability-ranked risk, and pair every human approval with an automated check that catches what a tired reviewer will not.

“The developer portal is the platform.” Backstage (or Port, or any portal) is a UI — the shop window. The platform is everything behind it: the software templates, the GitOps backend, the policy-as-code, and the self-service provisioning. Teams get value from the paved road underneath, not the catalog on top; a beautiful portal over no real automation is the most seductive way to build a platform that helps nobody.

Best practices

• Derive the rung from how you actually ship and how much a bad release costs — never from a conference talk or a résumé-driven urge to run a platform. The facts (team count, deploy frequency, risk tolerance, runtime, compliance) come first; the architecture is downstream.
• Default to the lowest sufficient rung, operated with discipline. Reliable, fast delivery comes more from operating a simple pipeline well than from owning a complex platform. Each rung up adds engineering cost and the complexity that itself causes delivery failures.
• Keep CI fast and deterministic from day one. It is the foundation of every higher rung; a slow or flaky CI gets routed around, and the rot propagates upward. Cache aggressively, parallelise, and quarantine flaky tests ruthlessly.
• Build the artifact once and promote it; never rebuild per environment. Immutable, versioned, traceable artifacts are what make “what we tested is what we ship” true — the guarantee the whole upper ladder depends on.
• Make every gate either enforce or be removed. A scan or approval that cannot stop a bad change is theatre. Gate on real risk, and pair human approvals with automated checks so the human is a backstop, not the only line.
• Keep speed and safety rising together. If a new gate, approval, or strategy makes you slower without making you safer, you implemented it wrong — small, tested, reversible changes are both faster and safer.
• Treat a platform as a product with customers, and let demand pull it. Build IDP capabilities only when real teams pull them; a paved road must be easier than the alternative, or it becomes an abandoned tab.
• Re-assess on material change, not continuously. A move to Kubernetes, a new compliance regime, or a jump in team count is the signal to re-run the choice; absent those, resist the itch to climb.
• Test the safety you claim. Exercise a rollback, fail a canary in staging, restore from a Git revert, kill a region in a game-day. An untested mechanism is a hypothesis, not a guarantee.

Security notes

The ladder has a security dimension that tracks its automation dimension, and it strengthens as you climb:

• Each rung should reduce standing credentials, not multiply them. Rung 2’s move to OIDC federated, short-lived tokens removes long-lived cloud keys from the CI system; rung 4’s GitOps pull model keeps cluster credentials inside the cluster rather than in the pipeline. Prefer keyless, short-lived auth at every rung — stored long-lived secrets are the credential an attacker hunts for. (And never re-commit credentials to Git; secrets belong in a manager, injected at deploy.)
• The pipeline is itself a high-value target and part of your supply chain. Anything that can build and deploy can also ship a backdoor. Rung 5’s SBOMs, signing (cosign), provenance (SLSA), and signature verification at admission make the pipeline a tamper-evident producer — treat unsigned, unattested artifacts as untrusted, and protect the CI/CD system as production infrastructure.
• Make policy unbypassable, not documented. Policy-as-code in the pipeline (OPA/Conftest) and at the cluster boundary (Kyverno/Gatekeeper) enforces the rules every workload must obey so they cannot be skipped under deadline pressure — a control that can be commented out is not a control.
• GitOps puts Git and the CD controller inside your trust boundary. Once a commit reaches production by itself, the repository, its review gates, and the Argo/Flux controller’s permissions are as sensitive as cluster-admin — protect branch protection and signed commits accordingly.
• A platform centralises both leverage and blast radius. A golden path that bakes in security controls hardens every service uniformly — but a compromised template or platform credential reaches every service it scaffolds. Secure the platform’s templates, provisioning credentials, and policy bundles as your highest-value assets, and audit changes to them.
• Scan, but gate on real risk. Severity- and reachability-based gating keeps security effective without the false-positive flood that makes teams disable it — an unmaintained scanner that everyone ignores is worse than a focused one that everyone trusts.

Interview & exam questions

Q1. What is the single most common DevOps delivery mistake, and how do you avoid it? Choosing the wrong altitude for the team — over-building (an IDP for one small team) or under-building (revenue deployed by hand from a laptop). Avoid it by describing how you actually ship today in concrete terms (team count, deploy frequency, manual steps, risk tolerance, runtime, compliance) and picking the lowest rung that satisfies those facts, operated well.

Q2. Distinguish Continuous Integration, Continuous Delivery, and Continuous Deployment. CI automatically builds and tests every change against the mainline, fast and on every push (rung 1). Continuous Delivery keeps the mainline always releasable and automates deploy up to a human click. Continuous Deployment removes the click and ships every green build to production automatically. Delivery requires real test confidence and easy rollback before it is safe; Deployment requires even more.

Q3. Why is an immutable, versioned artifact (built once, promoted) so important, and what breaks without it? It guarantees that what you tested is byte-for-byte what you deploy, with traceability from the running version back to a commit. Without it — if you rebuild per environment — production can silently differ from what passed the gates (different dependency versions, base images, build context), so your tests no longer mean what you think. Build once, publish, and promote the same artifact.

Q4. A scan runs in the pipeline but never fails the build. Is that a security control? No — it is documentation. A control must be able to stop a bad change. The fix is to gate on real risk (by severity and reachability) so exploitable findings block while low-signal ones go to a backlog. The same logic applies to rubber-stamp approvals: an unread “approve” is theatre, so pair human approval with automated gates.

Q5. What problem does GitOps solve that push-based CD does not, and what does it cost? GitOps makes Git the single source of truth and has an in-cluster controller continuously reconcile the cluster to it — giving drift detection/correction, git revert rollback, Git-as-audit-log, and credentials that stay inside the cluster (push CD holds cluster creds externally and lets drift go silent). It costs a new component to operate (Argo CD/Flux) and a behaviour change (“change Git, not the cluster”), and it leaks value if direct kubectl back doors remain open. It is largely a Kubernetes-shaped tool.

Q6. Name the four DORA metrics and explain how they should and should not be used. Deployment frequency, lead time for changes, change-failure rate, and time to restore service. They should be used as a team diagnostic to locate and remove delivery bottlenecks and to show that speed and stability rise together. They should not be used as a vanity number or as a stick to rank or punish individuals — that drives gaming and erodes the very behaviour you want.

Q7. Why is it a myth that adding gates and approvals necessarily slows delivery down? Because the DORA research shows elite performers deploy more often and fail less. Good automation makes each change small, tested, and reversible, which is both faster and safer; the gates catch risk cheaply rather than via slow manual review or production incidents. A gate only slows you down if it is implemented badly (noisy, blunt, or a rubber stamp) — well-built, safety and speed rise together.

Q8. Compare rolling, blue/green, and canary deployments and when you’d choose each. Rolling replaces instances incrementally — simplest and cheapest, but a bad version reaches some users and rollback is gradual. Blue/green runs two full environments and switches traffic at once — instant cutover and instant rollback, at the cost of roughly double capacity during the switch. Canary sends a small fraction of traffic to the new version and promotes on healthy metrics — smallest blast radius and metric-driven confidence, at the cost of more tooling and traffic-shaping. Choose by how costly a bad release is and how much infrastructure/tooling you can invest.

Q9. When does building an Internal Developer Platform pay off, and how is it different from climbing for delivery rigour? It pays off at organisational scale — many teams and services where hand-building a secure, observable pipeline per team is the bottleneck (roughly above five or six teams). It is a different axis from rigour: rungs 1–3 add delivery rigour, rung 4 is a runtime/GitOps choice, and the IDP (rung 6) productises rungs 1–5 as self-service. Team count drives it, not how good any single pipeline is — and below the threshold it costs more than it saves.

Q10. Why can a too-early platform or a fancy delivery setup be worse than a simpler one? Because its capability is bought with a large jump in complexity, and complexity is itself a leading cause of delivery failures. A small team that cannot staff or evolve an IDP ends up with a half-built, abandoned portal that is worse than a solid copied pipeline; a team that bolts on noisy security gates ends up disabling them. Reliability and speed come from operating a fit-for-purpose design well, not from owning a complex one.

Q11. How does the path to production change as you climb the ladder? Manual deploy (rung 1) → one automated pipeline to one environment with an immutable artifact (rung 2) → governed promotion of that artifact across multiple environments with deploy strategies and audited approvals (rung 3) → a Git commit reconciled into the cluster by a controller (rung 4) → the same path with enforced security/supply-chain gates and DORA measurement (rung 5) → a templated, self-service paved road any team can use (rung 6). The primitives are stable; what changes is automation depth, who drives the deploy, and self-service.

Q12. How do you decide when to stop climbing the ladder? Stop at the lowest rung whose capabilities and failure model satisfy your written situation. Climb only on a concrete signal — broken merges → rung 1; an env to keep current → rung 2; multiple envs / costly bad deploy / audit need → rung 3; Kubernetes + drift pain → rung 4; compliance/supply-chain or measurement need → rung 5; many teams bottlenecked → rung 6 — and re-assess on material change, not continuously. If you cannot name the requirement and the cost of not meeting it, you do not yet need the next rung.

Practice challenges

Six “read the requirements, choose the rung, defend it” exercises, escalating from the bottom of the ladder to the top. For each, decide the rung and the single most important decision or fix before opening the answer — the discipline is naming the lowest rung that satisfies the hard requirement, and no higher.

Challenge 1 (beginner). A two-person startup is building a pre-launch MVP with no users yet. They “deploy” by occasionally running a script by hand, and nobody is complaining about that. Their real pain: they keep merging changes that break each other’s work, and lose afternoons to “it worked on my machine.” Which rung, and what is the one thing to add?

<details> <summary>Model answer</summary>

Rung 1 — a single CI workflow. The only live pain is broken merges; there is no deployment requirement on the table (no users, rare manual deploys), so adding CD would be solving a problem they do not have. Add a CI workflow that builds, runs unit tests, and lints on every push and pull request, and make the green check required to merge via branch protection — that required status check is what turns a script into an enforced integration gate. Why not higher: nothing about their situation demands an automated deploy or environments yet; climb the day a real environment must be kept current. </details>

Challenge 2 (beginner–intermediate). A small internal tool now has a staging and a production environment. One engineer deploys by SSHing into each box and pulling the latest code, roughly twice a week. Last month, production behaved differently from staging and caused an incident, and only that one engineer knows the steps. Which rung, and what specifically fixes the incident’s root cause?

<details> <summary>Model answer</summary>

Rung 2 — full CI/CD with an immutable, versioned artifact. The prod-differed-from-staging incident is the textbook symptom of rebuilding per environment: pulling “latest” on each box produces subtly different bits. Fix it by building the artifact once (for example a container image tagged with the commit SHA), publishing it to a registry, and having an automated deploy step promote that same artifact to staging and then production — so what you tested is byte-for-byte what runs. This also removes the single-engineer, laptop-style deploy. Why not rung 3: only two environments and no stated approval/audit or blast-radius requirement yet — a straightforward automated promotion is enough. Climb to rung 3 when a bad deploy starts costing real users or an auditor asks who approved production. </details>

Challenge 3 (intermediate). A B2B SaaS with paying customers runs on Azure App Service (not Kubernetes). It has dev, test, staging, and production; it deploys daily; it has a contractual uptime SLA; and auditors have started asking, “who approved this production release, and when?” Releases occasionally cause brief downtime. Which rung, and which two mechanisms address the SLA and the audit question?

<details> <summary>Model answer</summary>

Rung 3 — multi-environment delivery with promotion, strategies, and approvals. Promote the one immutable artifact dev → test → staging → prod. For the SLA / downtime: use a downtime-free strategy — App Service deployment slots for blue/green (swap staging into prod, roll back by swapping straight back), or a canary. For the audit question: a protected production environment with required-reviewer approval, which is simultaneously the safety brake and the audit trail of who approved what and when. Also exercise the rollback so it is a capability, not a hope. Why not rung 4: they are not on Kubernetes, so GitOps largely does not apply. Why not rung 5 yet: no compliance/supply-chain mandate is stated — though the SLA hints they should start measuring DORA soon. </details>

Challenge 4 (intermediate–advanced). That same SaaS migrates onto Kubernetes — three clusters across regions. Now they hit a new class of pain: someone kubectl applyed a hotfix at 2 a.m. and nobody can reconstruct what production is supposed to run; the clusters drift from intent; and the security team has flagged that the CI system holds cluster-admin credentials. Which rung, and what is the headline security win?

<details> <summary>Model answer</summary>

Rung 4 — GitOps. Put the declarative desired state (manifests / Helm values / Kustomize overlays) in Git as the single source of truth, and run Argo CD or Flux inside each cluster to continuously reconcile live state to Git, with drift detection and optional self-heal. Rollback becomes git revert; promotion between environments becomes a pull request; Git history becomes the audit log. The headline security win is credential inversion: the flow becomes pull-based, so cluster credentials stay inside the cluster and CI no longer holds cluster-admin — it just updates an image tag in Git. The trap to avoid: leaving kubectl back doors open, which lets drift creep straight back and defeats the single-source-of-truth guarantee. Why this and not rung 5: drift and credential exposure are runtime/deploy-model problems, orthogonal to security gating and measurement. </details>

Challenge 5 (advanced). A fintech is pursuing SOC 2 and PCI-DSS. Delivery is already GitOps-driven, but security scans are advisory-only, artifacts are unsigned, there is no machine-enforced policy on what may run, and leadership cannot tell whether delivery is improving. Which rung, and name two non-negotiables plus the classic failure mode to avoid?

<details> <summary>Model answer</summary>

Rung 5 — DevSecOps with observability and DORA. Under regulatory pressure, controls must be enforced and evidenced. Two non-negotiables: (1) turn advisory scans into severity-and-reachability-gated checks (SAST, SCA, secret, container-image, IaC, DAST) that block real exploitable risk, and enforce policy-as-code (OPA/Conftest in the pipeline, Kyverno/Gatekeeper at admission) so rules cannot be skipped; (2) establish supply-chain integrity — SBOMs, cosign signing, SLSA provenance, and signature verification at admission so only signed, attested images run. Then instrument the four DORA metrics and SLOs to prove and improve delivery. The classic failure mode: false-positive noise so bad that developers disable the security stage to ship — calibrate by severity and reachability and route low-signal findings to a backlog. Also avoid treating DORA as a stick on individuals; it is a team diagnostic. Why not rung 6: the driver here is governance and measurement for one org, not many teams each hand-building pipelines. </details>

Challenge 6 (advanced — and a trap). A scale-up has grown to 45 engineers across 9 stream-aligned product teams. Every team hand-builds its own slightly-different rung-5 pipeline; onboarding a new service is a three-week ticket slog; developers spend more time wiring CI, registries, and policy than writing features. Leadership wants to “adopt platform engineering.” What rung — and what would flip your answer to the opposite advice?

<details> <summary>Model answer</summary>

Rung 6 — an Internal Developer Platform. The binding constraint is now fan-out, not any missing capability: nine teams above the ~5–6 threshold, each replicating rungs 1–5 by hand, with multi-week onboarding as the tell. Build a paved road — a service catalog plus software templates that scaffold a complete, conformant rung-2-through-5 service (pipeline, GitOps wiring, environments, observability, policy) in minutes, self-service, with guardrails baked in and escape hatches provided. Treat it as a product with the platform team as owner and let demand pull each capability; remember Backstage/Port is only the UI — the value is the templates, GitOps backend, and policy underneath. What flips the answer: if this were a 4-engineer, single-service team that had merely read a conference talk, the correct advice is the opposite — do not build a platform. Below the fan-out threshold the IDP’s fixed and ongoing cost dwarfs the toil it removes, and a platform they cannot staff becomes an abandoned, half-built portal that is worse than a solid pipeline template copied by hand. Also confirm the scale-up can actually staff a dedicated platform team; if not, ship an excellent template first and revisit. </details>

Quick check

1. Which three climbs on the ladder move semi-independently of each other, such that you can need one without the others?
2. At which rung do you first get an immutable, versioned artifact that is built once and promoted, and why does it matter?
3. What does GitOps (rung 4) change about who deploys and where the cluster credentials live, compared with push-based CD?
4. Name the four DORA metrics, and state the one rule for using them well.
5. Above roughly how many teams does building an Internal Developer Platform typically start to pay off, and what should you do below that threshold?

Answers

1. Delivery rigour (rungs 1→2→3), the runtime/GitOps model (rung 4, mostly relevant if you run Kubernetes), and organisational scale / self-service (rung 6, the IDP). Security and measurement (rung 5) layer onto whichever delivery model you run.
2. Rung 2. It guarantees that what you tested is byte-for-byte what you deploy (traceable from running version back to commit), because the artifact is built once and promoted unchanged rather than rebuilt per environment — the guarantee the entire upper ladder relies on.
3. GitOps inverts the model from push to pull: an in-cluster controller (Argo CD/Flux) deploys by reconciling the cluster to Git, instead of the CI system pushing changes in. Consequently the cluster credentials stay inside the cluster rather than being held by the external pipeline — a real security improvement.
4. Deployment frequency, lead time for changes, change-failure rate, and time to restore service. Use them as a team diagnostic to find and remove bottlenecks — never as a vanity number or a stick to rank or punish individuals.
5. Around five or six teams (or many services); below that, skip the platform and ship a solid rung-5 pipeline template that teams copy by hand — the IDP machinery would cost more than the toil it removes.

Exercise

Take a delivery setup you know — at work, a side project, or a hypothetical “B2B SaaS with three product teams” — and produce a one-page rung-selection memo:

1. Describe the current path first. State how a change reaches production today, the manual steps still in it, your team and service count, deploy frequency, your tolerance for a bad release, your runtime (Kubernetes or not), and any compliance constraint. If you have to guess, mark the guess — that itself is a finding.
2. Place it on the ladder. For each requirement, name the lowest rung that satisfies it, then take the highest rung any single hard requirement forces. Note separately whether the GitOps (rung 4) and IDP (rung 6) climbs apply, since they are independent axes.
3. Find the weakest link. Identify the single biggest gap between where you are and where you should be (e.g. “no immutable artifact,” “approvals are rubber stamps,” “no rollback path,” “security scans don’t gate”) and the one change that would close it.
4. Justify the stop. Write one paragraph on why you are not climbing higher — which next-rung capability you are deliberately forgoing and why the residual cost is acceptable.
5. Name the climb trigger. State the single concrete signal that would justify moving up one rung (e.g. “we move workloads to Kubernetes,” “a SOC 2 audit,” “team count crosses six”).

The goal is to practise the discipline the lesson teaches: facts first, lowest sufficient rung, fix the weakest link before adding altitude, an explicit reason to stop, and a defined trigger to climb.

Certification mapping

This lesson is delivery-architecture reasoning that underpins the practical DevOps exams rather than a single exam objective, and it maps across the cloud DevOps professional certifications:

• AWS Certified DevOps Engineer – Professional (DOP-C02): The exam’s domains — CI/CD (CodePipeline/CodeBuild/CodeDeploy), deployment strategies (in-place, rolling, blue/green, canary), monitoring and incident response, and resilience — are exactly rungs 2–5. Understanding why you promote an immutable artifact, when a canary beats a rolling deploy, and how observability and rollback bound failure is the judgement layer above the service mechanics.
• Microsoft Azure DevOps Engineer Expert (AZ-400): AZ-400 covers exactly this ladder in Azure terms — CI with Azure Pipelines, artifacts and feeds (rung 2), multi-stage YAML with environments and approvals (rung 3), GitOps/IaC delivery (rung 4), and securing the pipeline and supply chain (rung 5). The lesson’s “promote one artifact through approved environments” maps directly to AZ-400’s release and approval-gate objectives.
• Google Cloud Professional Cloud DevOps Engineer: Strongly SRE-flavoured, it leans on rung 5 — SLOs, error budgets, the four DORA metrics, and reducing toil — alongside CI/CD with Cloud Build/Deploy and progressive delivery. The ladder’s “speed and safety rise together” and the DORA framing are core to this exam’s philosophy.
• Adjacent: the GitOps rung touches the CNCF Kubernetes exams (CKA/CKAD), the supply-chain and policy content of rung 5 aligns with security-focused material, and the IaC underpinning the whole ladder maps to the HashiCorp Terraform Associate.

The exams test the services and mechanisms; this lesson teaches the delivery-architecture judgement that decides which mechanisms a given situation needs — exactly the kind of “design a delivery process for these constraints” question senior DevOps interviews open with.

Glossary

• Rung — one level of the architecting ladder; a delivery-architecture point defined by its automation depth, path to production, safety mechanisms, feedback/visibility, and degree of developer self-service.
• CI (Continuous Integration) — automatically building and testing every change against the mainline, fast and on every push, so breakage is caught in minutes.
• Continuous Delivery — keeping the mainline always in a releasable state, with deployment automated up to a final human approval.
• Continuous Deployment — automatically deploying every change that passes the pipeline to production, with no human gate.
• Quality gate — an automated check (tests, coverage, scan, policy) that can fail the pipeline and stop a bad change from progressing.
• Artifact — the immutable, versioned output of a build (a container image, package, or chart) that is published once and promoted unchanged through environments.
• Artifact repository — a store for versioned artifacts (a container registry, Nexus, Artifactory, GitHub Packages, Azure Artifacts) from which deployments pull.
• OIDC (keyless auth) — federated, short-lived identity tokens that let a pipeline authenticate to a cloud without long-lived stored secrets.
• Deployment strategy — how a new version is rolled out: recreate, rolling, blue/green, or canary, trading blast radius, cost, and tooling.
• Approval gate / protected environment — a required-reviewer sign-off (and audit record) before deploying to a sensitive environment such as production.
• Feature flag — a runtime toggle that decouples deploying code from releasing a feature, so the bits can ship dark and be turned on later.
• GitOps — a delivery model where the desired state lives in Git and an in-cluster controller (Argo CD/Flux) continuously reconciles the running system to it.
• Drift — divergence between the running system and its declared desired state; GitOps detects and can auto-correct it.
• DevSecOps — embedding security controls (SAST, SCA, secret/image/IaC scanning, DAST, policy-as-code) as gates within the delivery pipeline.
• SBOM (software bill of materials) — a machine-readable inventory of everything in an artifact, used for supply-chain transparency and vulnerability response.
• SLSA / provenance / signing — supply-chain integrity practices: attesting how an artifact was built (SLSA provenance) and signing it (cosign) so it can be verified before it runs.
• Policy-as-code — enforcing organisational rules in code (OPA/Conftest in the pipeline, Kyverno/Gatekeeper at admission) so they are unbypassable.
• DORA metrics — the four delivery-performance measures: deployment frequency, lead time for changes, change-failure rate, and time to restore service.
• SLO / error budget — a reliability target for a service and the allowable amount of unreliability before changes must slow down to protect it.
• IDP (Internal Developer Platform) — a self-service layer (e.g. Backstage + software templates + a GitOps backend + self-service infra) that lets developers provision and ship through a paved road without tickets.
• Golden path / paved road — a templated, opinionated, conformant-by-construction way to create and run a service that bakes in the organisation’s pipeline, security, and policy.
• Ephemeral / preview environment — a short-lived environment (often per pull request) spun up for review and testing and automatically torn down.
• Delivery pipeline — the automated path a change travels from a code commit to running in an environment; the object the whole ladder composes and re-composes.
• Blast radius — how many users, or how much of the system, a bad change can harm before it is caught or rolled back; deploy strategies and canaries exist to shrink it.
• Batch size — how much change ships at once; smaller batches are faster to verify and safer to reverse, which is why automation makes delivery both quicker and safer.
• Lead time for changes — the DORA metric measuring elapsed time from a change being committed to it running in production; it falls as automation depth rises.
• Reconciliation — a controller’s continuous loop of comparing live state to declared desired state and changing the live system to match; the mechanism at the heart of GitOps.
• Progressive delivery — releasing a new version gradually (canary, staged rollout) with automated promotion or rollback based on health metrics; Argo Rollouts or Flux + Flagger add it on top of GitOps.
• Admission control — a cluster gate (Kyverno, OPA Gatekeeper, a signature verifier) that inspects each workload as it is created and rejects any that violate policy or lack a valid signature.
• SAST / SCA / DAST — the main automated security scans: SAST analyses your own source code, SCA scans your dependencies for known vulnerabilities, and DAST tests a running instance from the outside.
• Cognitive load — the total a team must hold in its head to ship safely (from Team Topologies); a paved road lowers it by hiding plumbing, a half-built platform raises it by adding tools without hiding any.
• Stream-aligned team / platform team — Team Topologies roles: a stream-aligned team delivers product value to users; a platform team builds the internal self-service platform those teams ride on.
• Conway’s Law — the observation that a system’s architecture mirrors the communication structure of the organisation that builds it; an IDP is partly a deliberate attempt to shape delivery by shaping the platform team’s boundary.
• Demand-pull / platform-as-product — the discipline of building platform capabilities only when real teams pull them, and running the platform like a product with customers rather than a mandated framework.
• Backstage / Port — developer-portal products that provide the service catalog and the front door to golden paths; the visible UI over a platform, not the platform itself.

Next steps

You can now place a delivery requirement on the ladder and defend where you stopped. Turn that judgement into something hiring managers can see by building the matching portfolio: continue to Real-World DevOps Portfolio Projects: From a First Pipeline to a Platform, whose project ladder mirrors this one rung for rung.

Then deepen the rungs that matter most for your situation:

• Building a DevSecOps Pipeline: SAST, SCA, and Policy Gates — the full wiring behind rung 5’s security gates, calibrated to ship real risk-blocking without false-positive noise.
• Standing Up Backstage as an Internal Developer Portal — the catalog, golden-path software templates, and docs pipeline that make rung 6’s self-service real.
• Deployment Strategies: Rolling, Blue/Green, Canary, Progressive Delivery & Rollback — the strategies and rollback mechanics that rung 3 depends on.
• DORA Metrics from Pipeline Events: Deployment Frequency & Lead Time — how to compute and act on the four metrics that drive rung 5’s improvement loop.
• CI/CD Pipeline Design: Stages, Quality Gates, Artifacts & Security Scans — the end-to-end pipeline anatomy that rungs 2–3 assemble.