Securing the Software Supply Chain: SBOMs, Sigstore Signing, and SLSA Provenance in CI/CD

Supply-chain attacks do not break your code; they break the path your code travels from commit to cluster. This guide wires up the four controls that actually move the needle: a software bill of materials, keyless artifact signing, build provenance, and fail-closed verification at deploy time.

In a nutshell

Software reaches production the way food reaches a supermarket: it passes through many hands — your code, open-source libraries, a build server, a registry, a cluster — and any one of those hands can slip something into the box. A supply-chain attack does exactly that. It does not break into your source code; it tampers with the path your code travels after you push it. The famous breaches — SolarWinds, Codecov, the xz backdoor — all worked this way. The malware shipped wearing your pipeline’s own badge.

This lesson wires up four controls that make that tampering first visible and then refusable:

An SBOM (software bill of materials) is the ingredient label: every package that went into the artifact.
A signature is the tamper-evident seal: cryptographic proof of who produced this exact artifact.
Provenance is the receipt from the factory: a verifiable record of how and from where it was built.
A policy gate is the inspector at the loading dock: it refuses anything whose seal, label, or receipt is missing or wrong.

SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”) is the framework that ties these into a maturity ladder. Sigstore — the cosign CLI plus the Fulcio certificate authority and the Rekor transparency log — is the toolset that does the signing without you ever holding a signing key. By the end you will have a build that signs and attests itself, and a cluster that fails closed on anything unsigned.

Level: Advanced · Time: ~35 min

Prerequisites. You should be comfortable with a CI pipeline and container images. If GitHub Actions or OIDC keyless auth is new, read GitHub Actions fundamentals and OIDC keyless deploys first — the entire keyless-signing story rests on OIDC. A running Kubernetes cluster helps for the admission part but is not required to follow along.

After this lesson you can:

Explain the SLSA Build track (L1–L3) and the source → build → deploy threat model in plain language.
Generate an SBOM with Syft in both SPDX and CycloneDX, bound to an image digest.
Sign a container image keylessly with cosign and attach the SBOM and SLSA provenance as in-toto attestations.
Emit non-forgeable SLSA Build L3 provenance from the slsa-github-generator reusable workflow.
Enforce a fail-closed admission gate (Kyverno) that verifies signature, provenance, and a clean scan before a pod runs.

Trusted build chain: source through signed, attested, and verified deploy

Read it left to right: a known workflow builds an immutable image; three proofs are attached to that exact digest — an SBOM (what), SLSA provenance (how), and a keyless signature (who); everything is recorded in the Rekor transparency log; and a policy gate re-verifies all of it at deploy time and refuses anything that falls short.

1. Anatomy of a supply-chain attack and SLSA as a roadmap

The classic attacks (SolarWinds, Codecov, the event-stream npm hijack, and the 2024 xz backdoor) share a shape: the attacker compromises the build or distribution step, not the source. The malicious artifact is signed by your legitimate pipeline and ships looking exactly like the real thing. Source review never sees it because the tampering happens after git push.

The three places tampering happens. SLSA frames the supply chain as a pipeline with distinct attack surfaces, and it helps to name them before you pick controls:

Where	What the attacker does	Real example	The control that closes it
Source	Commit malicious code, or compromise the SCM to inject it	`event-stream` maintainer handoff	Branch protection, code review, signed commits
Build	Tamper with the build so the output differs from the source	SolarWinds build implant; Codecov uploader	Provenance + isolated/hermetic builds (this lesson)
Dependencies	Publish a poisoned package your build pulls in	`xz` backdoor; typosquats	SBOM + pinning + vulnerability scanning
Distribution / deploy	Swap the artifact between build and run, or repoint a tag	Registry compromise; tag re-push	Sign by digest + verify at admission

The uncomfortable truth is the second row: source review cannot catch a build-time attack, because the malicious bytes never appear in a commit. That is precisely the gap provenance and signing exist to close, and why the SLSA Build track is where most teams start.

SLSA (Supply-chain Levels for Software Artifacts) gives you a maturity ladder to close that gap. The Build track is what most teams target:

Build level	What it requires	What it stops
L1	Provenance exists and is distributed	“Where did this come from?” guesswork
L2	Provenance is signed; build runs on a hosted, authenticated service	Casual forgery; build run on a laptop
L3	Provenance generated in an isolated, non-falsifiable build; secrets unforgeable by the build steps	A compromised build step tampering with its own provenance

Read the ladder as cumulative trust: L1 gives you a provenance document you can read but must take on faith; L2 signs it so you can tell forgery from the real thing; L3 moves generation onto an isolated builder so even a hijacked build step cannot forge the record about itself. Each rung shrinks the set of parties you have to trust — which is the entire point of the exercise.

The deliverables map cleanly to tools: SBOMs answer what is inside, signatures answer who produced it, and provenance answers how and from where it was built. You need all three, plus an enforcement point that rejects anything missing them.

2. Generating SBOMs with Syft

A software bill of materials is the dependency inventory you query later when the next CVE drops. Syft scans source trees and container images and emits SPDX or CycloneDX. Generate it against the exact image digest you are about to ship, never a floating tag.

# Scan the built image by digest and emit CycloneDX JSON
syft "registry.example.com/app@${IMAGE_DIGEST}" \
  -o cyclonedx-json=sbom.cdx.json \
  -o spdx-json=sbom.spdx.json

Producing the SBOM is not the goal; binding it to the artifact is. There are two complementary patterns. Embed it at build time as an OCI attestation, and also attach it as a signed Sigstore attestation (covered in step 4). With BuildKit you can capture an SBOM as part of the build itself:

docker buildx build \
  --sbom=true \
  --provenance=mode=max \
  -t "registry.example.com/app:${GIT_SHA}" \
  --push .

Callout: SBOM accuracy depends on the cataloguers matching your ecosystem. Validate that Syft actually detects your language’s packages (Go modules, Python wheels, jars) before you trust the output. An empty or partial SBOM is worse than none because it looks complete.

SPDX or CycloneDX — which format? Syft emits both, so you rarely have to choose, but they come from different worlds:

	SPDX	CycloneDX
Steward	Linux Foundation	OWASP
Standard	ISO/IEC 5962:2021	ECMA-424
Original focus	License compliance & provenance	Security, vulnerability & risk
Strengths	Rich licensing metadata; regulatory acceptance (e.g. US EO 14028)	Compact; carries VEX, services & SaaS components; tight scanner tooling
Emit with Syft	`-o spdx-json`	`-o cyclonedx-json`

In practice, produce both — it is one Syft invocation — attach both as attestations, and let the consumer pick. If you must choose one, teams optimizing for vulnerability management lean CycloneDX; teams optimizing for license and compliance reporting lean SPDX. Both encode the same core dependency graph, so a downstream scanner such as Grype reads either without complaint.

3. Keyless signing with cosign, Fulcio, and Rekor

Long-lived signing keys are a liability: they leak, they expire unnoticed, and rotating them across a fleet is painful. Sigstore’s keyless flow removes the standing key entirely. cosign obtains an OIDC token from your CI’s workload identity, exchanges it at Fulcio for a short-lived (roughly 10-minute) X.509 certificate bound to that identity, signs, and records the signature in the Rekor transparency log. The private key never outlives the signing operation.

In GitHub Actions, the ambient OIDC token drives this. The job needs id-token: write, and you should pin the cosign version rather than tracking main.

jobs:
  sign:
    runs-on: ubuntu-latest
    permissions:
      id-token: write   # mint the OIDC token Fulcio verifies
      packages: write    # push signature/attestation to GHCR
    steps:
      - uses: sigstore/cosign-installer@v3
        with:
          cosign-release: "v2.4.1"
      - name: Sign by digest
        env:
          COSIGN_EXPERIMENTAL: "1"
        run: |
          cosign sign --yes \
            "ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}"

Always sign by digest, never by tag. Tags are mutable; a signature over :latest proves nothing once the tag is repointed. cosign in modern releases defaults to the public good Fulcio and Rekor instances, so no key material is configured.

Callout: The signed identity is the OIDC subject (for example, the workflow path plus ref), not a key fingerprint. Verification policy therefore asserts “signed by this workflow, via this issuer” rather than “signed by this key.” That is the whole point of keyless: you verify provenance of identity, not custody of a secret.

4. Attaching the SBOM as a signed attestation

With keyless wired up, bind the SBOM to the image as an in-toto attestation so verifiers can demand it. cosign wraps the SBOM in a DSSE envelope, signs it the same keyless way, and logs it to Rekor.

cosign attest --yes \
  --predicate sbom.cdx.json \
  --type cyclonedx \
  "ghcr.io/org/app@${IMAGE_DIGEST}"

Now the SBOM is not a loose file in an artifact store that can be swapped; it is a signed predicate attached to the exact digest and anchored in a transparency log.

What an in-toto attestation actually is. The word “attestation” hides a precise structure worth knowing, because verification policies assert on its fields. An in-toto attestation is a signed statement with three parts:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    { "name": "ghcr.io/org/app",
      "digest": { "sha256": "e3b0c44298fc1c14..." } }
  ],
  "predicateType": "https://cyclonedx.org/bom",
  "predicate": { "...the SBOM document...": "..." }
}

subject — what the statement is about: the artifact, pinned by digest. This anchor is what stops an attestation being replayed onto a different image.
predicateType — a URI naming the kind of claim (an SBOM, SLSA provenance, a test result, a vulnerability scan).
predicate — the claim’s body (the SBOM itself, or the provenance fields).

cosign wraps that statement in a DSSE envelope (Dead Simple Signing Envelope), signs the envelope keylessly, and pushes it to the registry as a .att artifact plus a Rekor log entry. When a verifier later runs cosign verify-attestation --type cyclonedx, it checks the DSSE signature, confirms the subject digest matches the image, and hands you the predicate to assert on. That structure is why you can write a policy such as “require an SBOM attestation and require the provenance predicate to show sourceRepo == our repo” — the fields are structured and signed, not free text.

5. Producing SLSA provenance

Provenance is the cryptographically verifiable record of how the artifact was built: the source repo and commit, the builder identity, and the build parameters. Generating it inside the same job that runs your build only gets you so far, because a compromised build step could lie about its own inputs. SLSA Build L3 requires the provenance to be generated by an isolated component the build steps cannot influence.

The slsa-framework/slsa-github-generator provides reusable workflows that run the provenance generation on a separate, isolated runner and produce non-forgeable provenance. For container images, call the generator and hand it the image and digest your build produced:

  provenance:
    needs: [build]
    permissions:
      actions: read     # read workflow run metadata
      id-token: write   # keyless signing of the provenance
      packages: write    # write the provenance attestation
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
    with:
      image: ghcr.io/org/app
      digest: ${{ needs.build.outputs.digest }}
    secrets:
      registry-password: ${{ secrets.GITHUB_TOKEN }}

Because the reusable workflow runs in its own job with its own token, the build job cannot tamper with the provenance it emits. The result is a signed SLSA provenance attestation attached to the image digest, verifiable later with cosign verify-attestation or slsa-verifier.

Callout: Build L3 hinges on isolation between the thing that builds and the thing that attests. If you generate provenance with an inline run: step in the same job as the build, you are at L2 at best, regardless of what the predicate claims.

6. Hardening the build environment

Signing a build that runs on a tampered machine just produces a trustworthy signature over malware. The build environment is itself part of the threat model.

Ephemeral runners. Each job gets a fresh VM that is destroyed afterwards, so a compromise cannot persist into the next run. GitHub-hosted runners are ephemeral by default; for self-hosted, use the --ephemeral flag or an autoscaling controller (Actions Runner Controller) that recycles pods per job.
Pin everything by digest. Pin Actions to a full commit SHA and base images to a digest. A mutable @v4 tag on a third-party action is an unguarded write into your build.
Hermetic builds. Disallow network access during the build so dependencies come only from a vetted, pre-fetched cache. This makes builds reproducible and removes the “fetch a fresh malicious dependency at build time” vector.
Least-privilege tokens. Set permissions: {} at the workflow root and grant scopes per job. The build job rarely needs id-token: write; only the signing and provenance jobs do.

permissions: {}   # deny-all default; opt in per job

jobs:
  build:
    permissions:
      contents: read   # nothing more

7. Admission-time verification with Kyverno

Producing all this metadata is wasted effort unless the cluster refuses to run anything that lacks it. Kyverno ships a verifyImages rule that performs cosign verification at admission and, critically, mutates the image reference to its digest so the verified bits are exactly the bits that run, defeating tag-swap races.

This policy demands a keyless signature from a specific GitHub Actions workflow and OIDC issuer:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce   # fail closed
  webhookTimeoutSeconds: 30
  rules:
    - name: verify-cosign-keyless
      match:
        any:
          - resources:
              kinds: ["Pod"]
      verifyImages:
        - imageReferences:
            - "ghcr.io/org/*"
          mutateDigest: true       # pin to verified digest
          required: true
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/org/app/.github/workflows/release.yml@refs/heads/main"
                    rekor:
                      url: "https://rekor.sigstore.dev"

You can extend the same rule with an attestations block to require the SLSA provenance predicate and assert fields inside it (for example, that the source repo matches), so an image signed by the right identity but built from the wrong repo is still rejected. The cosign policy controller (formerly Sigstore’s policy-controller) is a reasonable alternative if you prefer a Sigstore-native enforcement point over Kyverno.

Callout: validationFailureAction: Enforce is what makes this real. In Audit mode Kyverno only logs violations. Run in Audit first to find the unsigned images already in your cluster, then flip to Enforce once the backlog is clean.

8. Vulnerability gates: failing closed without locking yourself out

Signatures prove origin, not safety. A correctly signed image full of critical CVEs should still be blocked. Run Grype against the SBOM you already produced (faster and more accurate than re-scanning the image) and fail the pipeline on threshold.

grype "sbom:sbom.cdx.json" \
  --fail-on critical \
  --output table

“Fail closed” is correct, but a naive gate becomes an outage when a zero-day lands in a base image overnight and every deploy halts. Build the escape hatches in advance:

A reviewed, expiring allowlist (.grype.yaml ignore rules with a justification and a follow-up date), not a blanket bypass.
A documented break-glass path that still requires a signature, so the audit trail survives.
Threshold tuned by severity and fix availability. Failing on an unfixable low-severity finding just trains people to disable the gate.

The deploy gate is then a logical AND: a valid signature, the required provenance, and a clean (or explicitly waived) vulnerability scan. Any one missing is a hard stop.

Enterprise scenario

A fintech platform team rolled keyless cosign across ~140 services and flipped Kyverno to Enforce. Within hours, a handful of production Deployments wedged: pods stuck ImagePullBackOff with no matching signatures on images that were demonstrably signed and present in Rekor. The signatures were real; verification at admission still failed.

The cause was the signature discovery model. cosign stores a signature as a sibling tag derived from the image digest (sha256-<digest>.sig). The platform fronted GHCR with a pull-through cache (Harbor proxy) for egress control. The cache happily mirrored the image manifest on first pull but had never been asked for the .sig tag, so it returned a 404 and Kyverno read that as “unsigned.” Worse, mutateDigest: true had pinned the running pods to a digest whose signature the cache couldn’t serve.

The fix was to make the proxy treat signatures as first-class artifacts and warm them alongside the image, plus point Kyverno’s verification at the upstream registry for the cosign lookup rather than the cache:

verifyImages:
  - imageReferences: ["harbor.corp/ghcr-proxy/org/*"]
    mutateDigest: true
    attestors:
      - entries:
          - keyless:
              issuer: "https://token.actions.githubusercontent.com"
              subject: "https://github.com/org/app/.github/workflows/release.yml@refs/heads/main"
    repository: "ghcr.io/org"   # resolve .sig/.att from upstream, not the cache

The durable lesson: signatures and attestations are OCI artifacts with their own tags. Any registry mirror, replication rule, or retention policy that operates on “the image” will silently drop them unless it is explicitly artifact-aware. Audit your caching and GC paths for .sig and .att before you enforce.

Verify

Prove the chain end to end before you trust it.

# 1. Verify the keyless signature and identity
cosign verify \
  --certificate-identity-regexp "https://github.com/org/app/.+" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "ghcr.io/org/app@${IMAGE_DIGEST}"

# 2. Verify the SLSA provenance attestation
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity-regexp "https://github.com/org/app/.+" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "ghcr.io/org/app@${IMAGE_DIGEST}"

# 3. Confirm the entry is in the Rekor transparency log
cosign tree "ghcr.io/org/app@${IMAGE_DIGEST}"

# 4. Prove the cluster fails closed: an unsigned image must be rejected
kubectl run unsigned --image=nginx:latest --dry-run=server
#   expect: admission webhook "...kyverno..." denied the request

A passing signature check, a verified provenance predicate, a Rekor entry, and an admission denial for the unsigned image mean the controls are live, not theoretical.

Auditing and incident response

When a key identity or build is suspected compromised, the transparency log is your forensic record. Query Rekor for every artifact a given identity or workflow ever signed:

# Find all log entries signed by a given identity (email/SPIFFE/OIDC subject)
rekor-cli search --email "ci-bot@org.example"

# Inspect a specific entry by its log index
rekor-cli get --log-index 123456789

Because Rekor is append-only, you cannot delete a bad entry. You revoke trust instead: tighten the verification policy to exclude the compromised identity, repo, or time window, then rebuild and re-sign affected artifacts with a clean identity. Keyless makes this far less painful than rotating a leaked long-lived key, since there is no key to revoke across a fleet.

Going deeper

How the transparency log actually proves anything

Rekor is not a database you trust because it is Sigstore’s; it is an append-only Merkle tree, and that structure is what makes it trustworthy. When cosign uploads an entry, Rekor returns two things: an inclusion proof (the hash path showing your entry is committed to the tree at a specific position) and a Signed Entry Timestamp (SET) — Rekor’s countersignature over your entry. Because the tree is append-only, an entry cannot be quietly altered or removed without changing every hash above it, which independent monitors and witnesses would detect. Practically this buys you two things: proof that a signature existed at a point in time — so a short-lived Fulcio certificate that has since expired is still valid, because it was valid when it signed — and a public record an attacker cannot scrub after a compromise.

For offline or airgapped verification you do not want to phone rekor.sigstore.dev at deploy time. cosign can bundle the certificate, signature, and inclusion proof into a self-contained file and verify it later with no network:

# At sign time, capture everything needed to verify offline
cosign sign --yes --bundle app.bundle "ghcr.io/org/app@${IMAGE_DIGEST}"

# Later, in an airgapped environment, verify without contacting Rekor
cosign verify --bundle app.bundle --offline \
  --certificate-identity-regexp "https://github.com/org/app/.+" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "ghcr.io/org/app@${IMAGE_DIGEST}"

The root of trust: TUF

The public-good Fulcio and Rekor keys are themselves distributed through TUF (The Update Framework), which protects the key-distribution channel and lets keys rotate without every client hardcoding them. Regulated or airgapped organizations run their own Sigstore stack — private Fulcio, private Rekor, private TUF root — so no verification depends on a public internet service. When you do, pin the TUF root you initialize cosign with (cosign initialize --root ...); that pinned root, not any single key, is the thing you actually trust.

Version caveats that bite

cosign v1 vs v2. In v1, keyless required COSIGN_EXPERIMENTAL=1; in v2 keyless is the default and the env var is a no-op. The snippet in step 3 keeps it for readers still on mixed tooling — it does no harm, but you can drop it once every runner is on v2. Pin the cosign version in CI regardless.
SLSA provenance v0.2 vs v1.0. The predicate was restructured: v0.2’s flat invocation / materials became v1.0’s buildDefinition + runDetails, and the type URI changed from .../provenance/v0.2 to .../provenance/v1. Verify with the matching type — cosign verify-attestation --type slsaprovenance reads v0.2, --type slsaprovenance1 reads v1.0. A verifier asserting on predicate.builder.id against the wrong schema silently matches nothing.
OCI 1.1 referrers. How signatures attach to images is moving from the older “sibling .sig tag” scheme to the OCI 1.1 Referrers API. Registries that do not yet implement referrers fall back to the tag scheme — which is exactly the fan-out that broke the fintech team above. Know which scheme your registry serves before you enforce.

Reproducibility is the independent check

Provenance says how the build claims it ran; a reproducible build lets a third party rebuild from the same source and confirm they get the same bytes. That turns “trust the builder” into “verify the builder,” and it is where SLSA is heading beyond L3. Hermetic builds (no network, pinned inputs) are the prerequisite: nondeterminism — timestamps, file ordering, embedded build paths — is what breaks reproducibility, so tools such as SOURCE_DATE_EPOCH and BuildKit’s reproducible options exist to strip it.

Scale and latency at the gate

Admission verification runs on the hot path of every pod-scheduling event, inside a webhook with a timeout (30s in step 7). cosign verification fetches the signature and attestation and may hit Rekor, so at fleet scale you cache verification results, keep the Rekor lookup on a fast path (or use bundles for offline verify), and mind webhookTimeoutSeconds — a slow registry turns a signing policy into a cluster-wide scheduling stall. This is the operational cost of “verify everything,” and it is why you verify once at admission and pin to the digest rather than re-verifying on every image pull.

Common beginner mistakes

“An SBOM secures my software.” An SBOM is an inventory, not a control — it stops nothing by itself. Its value is response speed: when the next Log4Shell lands, you query the SBOM instead of grepping 140 repos. Right model: an SBOM is the map you read during an incident, not a wall that prevents one.
“A signed image is a safe image.” A signature proves origin and integrity (“this is the exact artifact that identity produced, unmodified”), not quality. Perfectly signed malware is still malware, and a signed image can be riddled with critical CVEs. Right model: signing answers who, scanning answers is it safe — you need both, ANDed at the gate.
“Keyless means there is no identity to verify.” Backwards. Keyless removes the long-lived key, not the identity; you verify a stronger thing — the OIDC subject and issuer (the exact workflow and ref). Right model: you stopped verifying custody of a secret and started verifying a cryptographically attested identity.
“Provenance from my build job is Build L3.” If the same job that builds also writes the provenance, a compromised build step can forge its own record — that is L2 at best. Right model: the thing that attests must be something the build cannot influence, which is why L3 uses an isolated reusable workflow.
“The transparency log will leak my source or secrets.” Rekor stores metadata — signatures, certificates, digests, hashes — never your artifact, source, or secrets, and it is public by design. Right model: what is public is the proof that something was signed, not the payload that was signed.
“Any SBOM is better than no SBOM.” A partial or empty SBOM is worse than none, because it looks authoritative and lulls the scanner into reporting “no known issues” on components it never saw. Right model: trust an SBOM only after you have confirmed the cataloguer actually detects your ecosystem’s packages.

Practice challenges

Work these in order; each builds on the last. Solutions are hidden — try first.

1. (Beginner) Prove your SBOM is not empty. Generate a CycloneDX SBOM for a public image and count the packages it found.