In a nutshell
DevSecOps means building automated security checks into the same CI/CD pipeline that already builds, tests, and ships your code — so every commit and pull request is scanned the moment it is made, instead of a security team auditing the app by hand months after it shipped. “Shift left” is the core idea: move each check as early as it can usefully run, because a flaw caught on a developer’s laptop costs minutes to fix, while the same flaw caught in production costs an incident.
Picture a factory assembly line. Your pipeline is the conveyor belt; each security scanner is an inspection station bolted onto it. One station checks the raw materials you bought in (your dependencies — SCA), one checks the blueprint (your infrastructure code — IaC scanning), one pokes the assembled product while it runs (DAST), and one makes sure nobody taped a master key to the crate (secret detection). The policy gate is the line supervisor who can halt the belt — but a good supervisor only stops the line for defects that will actually hurt a customer, because if they halt it for every cosmetic scratch, the workers quietly learn to wave everything through.
That last sentence is the whole game. Most DevSecOps rollouts do not fail because the scanners are bad; they fail because the gate is calibrated to block on everything, developers get buried in alerts they cannot act on, and within a week they route around the security stage entirely. This lesson teaches you to wire the scanners in at the right stage and — the harder part — to tune the gate so it blocks real, fixable, reachable risk and stays quiet about everything else.
The diagram traces one change left to right: fast deterministic checks (secrets, lint-grade SAST) run on the laptop, diff-aware SAST and SCA run on the pull request, the heavier image and dynamic (DAST) scans run once there is a built artifact and a running staging target, every finding is normalized to SARIF, and a single policy gate — not five separate scanners each with its own --exit-code — decides what turns the build red.
Level: Advanced · Time: ~35 min
Prerequisites & what you’ll be able to do
You will get the most from this lesson if you already know:
- How a CI/CD pipeline runs jobs on a pull request and on merge — if that is new, start with CI/CD pipeline design.
- Basic Git and the pull-request flow (branches, diffs, merge).
- What a CVE and a CVSS score are at a high level: a CVE is a public ID for a specific vulnerability, and CVSS is its 0–10 severity number.
- How to read a YAML workflow file.
- Helpful but not required: the secrets fundamentals in Secrets & configuration management.
After working through it you will be able to:
- Place each scan type (secret, SAST, SCA, IaC, image, DAST) at the pipeline stage where its signal-to-noise ratio is best.
- Write a diff-aware SAST job that only flags code the pull request actually changed.
- Gate SCA on fixable, reachable High/Critical instead of raw CVSS, and keep a non-blocking inventory of the rest.
- Layer three independent controls for secret detection, and know why removal is not remediation.
- Scan a Terraform plan (not just raw HCL) and record an intentional deviation as an attributed, expiring exception.
- Normalize every tool’s output to SARIF, centralize findings, and measure the program with escape rate, MTTR, and false-positive rate.
Most “DevSecOps” rollouts fail the same way: someone bolts five scanners onto the pipeline, sets every gate to fail-on-anything, and within a week developers are commenting out the security stage to ship. The goal is not maximum scanning. It is catching exploitable risk early while keeping the build green for everything else. This guide wires SAST, SCA, secret detection, and IaC scanning into CI with gates calibrated to severity and reachability, then closes the loop with aggregation and metrics.
1. Shift-left without shift-pain: where each scan belongs
Not every scan belongs in the inner loop. The cost of a finding rises the later you catch it, but so does the cost of a false finding interrupting flow. Place each control where its signal-to-noise ratio is best.
| Scan | Pre-commit | PR / CI | Main / nightly | Blocking by default? |
|---|---|---|---|---|
| Secret detection | Yes (fast, deterministic) | Yes (diff scan) | Yes (full history) | Yes |
| SAST (CodeQL/Semgrep) | Lightweight rules only | Yes (diff-aware) | Full database build | High/critical only |
| SCA (Trivy/Grype) | No | Yes (lockfile diff) | Yes (full SBOM) | Reachable high/critical |
| IaC (Checkov/tfsec) | Optional | Yes | Yes | High/critical only |
| Container image (Trivy/Grype) | No | Yes (on build) | Yes (re-scan published) | Fixable high/critical |
| DAST (OWASP ZAP) | No | Preview env only | Yes (staging) | Warn first, then high-confidence |
The principle: deterministic, sub-second checks (secrets, lint-grade Semgrep) run pre-commit. Anything that needs a build, a dependency graph, or a database (CodeQL) runs in CI on the pull request. Expensive full-history and full-image scans run nightly on the default branch where latency does not block a human. The last two rows — the container image and DAST — need a built artifact or a running target, so they sit later than the source-level checks; Section 8 and Section 9 cover them.
Callout: A gate that blocks a PR must produce a finding the author can act on today. If the only fix is upgrading a transitive dependency with no patched version, that is a tracked exception, not a red build. Conflating “risk exists” with “this PR is the place to fix it” is the fastest way to lose developer trust.
2. Static analysis with Semgrep and CodeQL
Run two layers. Semgrep gives fast, customizable, diff-aware results on every PR. CodeQL gives deeper dataflow analysis (taint tracking across functions) on a schedule and on the default branch.
Semgrep in CI, scoped to the diff so you only flag code the PR actually touches:
# .github/workflows/semgrep.yml
name: semgrep
on:
pull_request: {}
jobs:
semgrep:
runs-on: ubuntu-latest
container:
image: semgrep/semgrep
steps:
- uses: actions/checkout@v4
- name: Diff-aware scan
run: semgrep ci --sarif --output=semgrep.sarif
env:
SEMGREP_BASELINE_REF: ${{ github.event.pull_request.base.sha }}
- uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: semgrep.sarif
semgrep ci automatically scans only changed lines when a baseline ref is set, which is the single biggest false-positive reducer for legacy codebases. New rules apply to new code; the existing backlog does not light up every build.
For tuning, prefer suppressing at the source over disabling rules globally. An inline # nosemgrep: rule-id (with a comment justifying it) is reviewable in the diff; deleting a rule from config hides risk for the whole org silently.
CodeQL runs the heavy analysis. Use the default setup for most repos, or a custom workflow when you need specific query packs:
# .github/workflows/codeql.yml
name: codeql
on:
push:
branches: [main]
schedule:
- cron: "0 3 * * 1"
jobs:
analyze:
runs-on: ubuntu-latest
permissions:
security-events: write
strategy:
matrix:
language: [javascript-typescript, python]
steps:
- uses: actions/checkout@v4
- uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
queries: security-extended
- uses: github/codeql-action/autobuild@v3
- uses: github/codeql-action/analyze@v3
The security-extended suite adds higher-recall queries at the cost of more findings; start with the default query set, prove the program works, then opt into security-extended once triage is healthy.
3. Dependency and SCA scanning with Trivy and Grype
SCA is where noise goes to spawn. A typical Node or Python project pulls thousands of transitive packages, and a raw CVE list will report hundreds of “criticals” you can do nothing about. Two filters tame this: severity and reachability.
Scan the filesystem (lockfiles) in the PR, and the built image nightly. Trivy reads package-lock.json, go.sum, requirements.txt, and friends directly:
# Fail the build only on fixable high/critical, output SARIF for the dashboard
trivy fs \
--scanners vuln \
--severity HIGH,CRITICAL \
--ignore-unfixed \
--exit-code 1 \
--format sarif \
--output trivy.sarif \
.
--ignore-unfixed is the flag that matters most for sanity: it drops vulnerabilities with no released fix, so you only gate on things a developer can actually remediate by bumping a version. Unfixed criticals still get recorded (run a second non-blocking scan without the flag), but they become tracked work, not a blocked merge.
Reachability is the next layer. A vulnerable function buried in a dependency you never call is lower risk than one on your hot path. Trivy can be paired with reachability analysis for some ecosystems; Grype plus its SBOM workflow gives a similar second opinion. The pragmatic pattern is to enrich findings with EPSS (exploit prediction) and known-exploited status, then gate hardest on the intersection of high severity, fixable, and reachable or actively exploited:
# Generate an SBOM once, then scan it (decouples build from scan)
syft dir:. -o cyclonedx-json=sbom.json
grype sbom:sbom.json --fail-on high -o sarif > grype.sarif
Callout: Do not gate purely on CVSS. A CVSS 9.8 in a dev-only dependency that never reaches production is noise; a CVSS 7.5 on your auth path that appears in CISA’s Known Exploited Vulnerabilities catalog is an emergency. Severity is an input to risk, not risk itself.
If you would rather buy this layer than assemble it, a commercial SCA such as Snyk wraps reachability, fix advice, and gating into one action — see Integrating Snyk with GitHub Actions for SCA, container, and IaC gating. The tuning principles below are identical whichever tool you pick.
4. Secret detection with gitleaks, hooks, and push protection
Secrets are the one category where you want defense in depth, because a leaked credential is exploitable the instant it lands. Layer three controls: client-side pre-commit, server-side CI, and platform push protection.
Pre-commit catches most secrets before they ever leave the laptop:
# .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.4
hooks:
- id: gitleaks
pip install pre-commit
pre-commit install # installs the git hook into .git/hooks
CI is the backstop for anyone who skips the hook with --no-verify. Scan the full history on the default branch and the diff on PRs:
# .github/workflows/gitleaks.yml
name: gitleaks
on: [pull_request, push]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history so historical leaks are caught
- uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Finally, enable platform-level push protection so the forge itself rejects a push containing a recognized secret pattern. On GitHub that is secret scanning push protection, enabled at the org or repo level; on GitLab the equivalent is secret push protection. This is the only control that stops the secret from ever reaching the remote.
Critically: detection is step one. Any committed secret is compromised and must be rotated, even after you scrub history, because it existed in a clone, a fork, or a CI log somewhere. Bake rotation into the runbook, not just removal.
5. IaC scanning for Terraform and Kubernetes with Checkov and tfsec
Infrastructure misconfigurations (public S3 buckets, unencrypted disks, 0.0.0.0/0 security groups) are cheap to catch in the plan and expensive to catch in production. Checkov has the broadest policy coverage across Terraform, CloudFormation, Kubernetes, and Helm; tfsec (now folded into the Trivy project) is a fast Terraform-focused complement.
Scan Terraform in CI:
# Checkov over a Terraform directory, SARIF out, soft-fail handled by the gate logic
checkov \
--directory . \
--framework terraform \
--output sarif \
--output-file-path checkov_results \
--compact
For higher fidelity, scan the plan rather than raw HCL so conditional and variable-driven resources are evaluated as they will actually deploy:
terraform plan -out=tfplan.binary
terraform show -json tfplan.binary > tfplan.json
checkov -f tfplan.json --framework terraform_plan -o sarif --output-file-path checkov_plan
Trivy also covers IaC and Kubernetes manifests with a single binary, which is convenient if you already run it for SCA:
trivy config --severity HIGH,CRITICAL --format sarif --output trivy-iac.sarif ./infra
Suppress intentional deviations inline and in version control so they are reviewable. Checkov honors a skip comment directly above the resource:
# checkov:skip=CKV_AWS_18:Access logging handled centrally by the org log archive bucket
resource "aws_s3_bucket" "artifacts" {
bucket = "kv-build-artifacts"
}
The justification text is mandatory in review: a skip without a reason is a finding in its own right.
6. Risk-based gating: thresholds, allowlists, and time-boxed exceptions
This is the heart of the program. The gate decides what turns a build red. Get it wrong in either direction and you either ship vulnerabilities or train developers to ignore security.
A defensible default policy:
- Block on: new high/critical SAST findings, fixable high/critical SCA on reachable paths, any verified secret, high/critical IaC misconfig.
- Warn (annotate, do not fail) on: medium findings, unfixable dependencies, low-confidence SAST.
- Track (ticket, no build impact) on: everything else.
Implement the gate as explicit logic, not as each scanner’s own --exit-code, so the policy lives in one readable place:
enforce:
needs: [semgrep, trivy, gitleaks, checkov]
runs-on: ubuntu-latest
if: always()
steps:
- name: Evaluate risk gate
run: |
set -euo pipefail
fail=0
for s in semgrep trivy gitleaks checkov; do
n=$(jq '[.runs[].results[]
| select(.level=="error")] | length' "artifacts/${s}.sarif" 2>/dev/null || echo 0)
echo "${s}: ${n} blocking findings"
fail=$(( fail + n ))
done
if [ "$fail" -gt 0 ]; then
echo "::error::Risk gate failed with ${fail} blocking finding(s)"
exit 1
fi
Allowlists and exceptions must be explicit, attributed, and expiring. Encode them as data, not as a forgotten || true:
# .security/exceptions.yaml
exceptions:
- id: CVE-2025-12345
component: example-lib
reason: "No patched release; not reachable from our entrypoints (see analysis #4821)"
owner: platform-security
expires: "2026-08-01"
A nightly job that fails when an exception is past expires is what keeps the allowlist from becoming a graveyard. An exception with no expiry is a permanent hole; the expiry forces a re-decision. Policy-as-code engines (OPA/Conftest) can evaluate this file in the gate so the rules themselves are versioned and testable.
7. Aggregating findings: SARIF, dashboards, and ticket automation
Five tools emitting five report formats is unmanageable. Standardize on SARIF (Static Analysis Results Interchange Format) as the lingua franca; every tool above can emit it, and GitHub’s code scanning ingests it natively:
- uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: trivy.sarif
category: trivy-sca # category keeps tools separate in the UI
Uploaded SARIF populates the repo’s Security tab with deduplicated, line-anchored findings, including fix status across runs so you can see what is new in a PR versus pre-existing. Use a distinct category per tool so one tool’s results never overwrite another’s.
For cross-repo visibility, pull findings into a central system. The pragmatic options are DefectDojo (open-source vulnerability management that imports SARIF and many native formats) or a SIEM/data warehouse fed by the forge’s security API. Automate ticket creation so tracked findings do not rot: a scheduled job queries open code-scanning alerts above a threshold and opens or updates issues, deduplicating on the rule ID plus location so you do not spawn a new ticket every night for the same finding.
8. Dynamic testing (DAST) against the running app
The six scans in the title include one the sections above deferred on purpose: DAST. SAST, SCA, secret, and IaC scanning are all static — they read code, lockfiles, or config without running anything, which is exactly why they fit on a laptop or a fast PR job. DAST (Dynamic Application Security Testing) is the opposite: it needs a running instance of your application and probes it from the outside, the way an attacker would — sending real requests to real endpoints and watching how the deployed app responds. That buys you a class of findings static analysis structurally cannot see: an auth check that is present in the code but wired up wrong, a security header missing only in the production config, an injection reachable through the actual request path, an admin endpoint accidentally exposed by the load balancer.
The trade-off is placement. DAST is slow (it spiders and probes) and it needs a deployed target, so forcing it onto every PR is the fastest way to make the security stage the reason merges are slow. Run it against an ephemeral preview environment on deploy, or nightly against staging.
OWASP ZAP is the open-source workhorse. It has two modes: a baseline scan (passive — it spiders the app and applies passive rules, safe and quick) and a full/active scan (it actually fires attack payloads such as SQLi and XSS probes, which can mutate data — never point it at production without care, and ideally only at a throwaway environment).
# .github/workflows/dast.yml — nightly DAST against a running staging target
name: dast
on:
schedule:
- cron: "0 2 * * *"
workflow_dispatch: {}
jobs:
zap-baseline:
runs-on: ubuntu-latest
steps:
- name: ZAP baseline scan (passive, safe to run nightly)
uses: zaproxy/action-baseline@v0.14.0
with:
target: https://staging.example.com
fail_action: false
allow_issue_writing: false
cmd_options: "-a"
The choices that matter:
fail_action: false— a nightly, informational baseline scan should not page anyone at 02:00. Feed its report to the dashboard and triage in the morning; reserve build-failing behavior for a smaller, high-confidence rule set once you trust the signal.cmd_options: "-a"— include the alpha passive rules for higher recall while you calibrate.- The single biggest reason DAST “finds nothing” is authentication: an unauthenticated scan hits the login wall and reports a clean run against an app it never actually explored. For anything behind a login you must give ZAP an authentication context (a login script, or a bearer token injected as a header) so it can reach the pages that matter. For APIs,
zaproxy/action-api-scandrives the scan from your OpenAPI/Swagger spec so it hits every documented endpoint.
Like every other stage, ZAP can produce SARIF (directly or via conversion) so its findings land in the same aggregated view as SAST and SCA — one dashboard, not a separate ZAP report nobody opens.
Callout: SAST and DAST are complementary, not redundant. SAST reads the source and can point at the exact vulnerable line but cannot tell you whether that line is reachable at runtime; DAST proves a real request path is exploitable but hands you a symptom at the HTTP boundary, not a line number. A mature pipeline runs both and correlates them.
9. Container image scanning: base images, layers, and the registry
The SCA section scanned your lockfiles — the dependencies you chose. But the artifact you actually ship is a container image, and most of its attack surface is code you did not write directly: the base image. A python:3.12 or node:20 base drags in a whole operating system — glibc, OpenSSL, curl, apt — and that OS layer is where the majority of an image’s “criticals” live. Scanning package-lock.json and calling it done misses all of it.
So scan the built image, not just the source tree. Trivy (which you may already run for SCA) does both with one binary:
# .github/workflows/image-scan.yml — build, then scan the image you actually ship
name: image-scan
on:
pull_request: {}
jobs:
build-and-scan:
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t ghcr.io/kv/app:${{ github.sha }} .
- name: Scan image (base image + app deps; block fixable High/Critical)
uses: aquasecurity/trivy-action@0.28.0
with:
scan-type: image
image-ref: ghcr.io/kv/app:${{ github.sha }}
format: sarif
output: trivy-image.sarif
severity: HIGH,CRITICAL
ignore-unfixed: "true"
exit-code: "1"
- uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: trivy-image.sarif
category: trivy-image
Three habits keep image findings manageable:
- Shrink the base. A
-slim, Alpine, or distroless base carries a fraction of the OS packages, so it carries a fraction of the CVEs. The cheapest way to close a hundred image findings is often to change oneFROMline. - Pin by digest, rebuild on a cadence. A green image today is a red image next week when a new CVE lands in its base — the image did not change, the world did. Pin the base by digest (
FROM …@sha256:…) for reproducibility, and schedule a nightly re-scan of published images (or lean on registry-integrated scanning) so you learn about the new CVE before an attacker does. See Harbor artifact registry with vulnerability gating for the registry-side enforcement pattern. - Remember
--ignore-unfixedcuts differently here. A base-image CVE with no fixed package yet cannot be remediated by bumping your code — the fix is to rebuild on a patched base or wait for one. Gate on fixable so those do not block the merge, but keep them in the inventory pass so you rebuild the moment a patched base ships.
Also scan the Dockerfile itself for misconfiguration (running as root, ADD from a URL, unpinned versions). trivy config covers Dockerfiles alongside Terraform and Kubernetes, and hadolint is a focused Dockerfile linter if you want stricter style rules.
Enterprise scenario
A fintech platform team I worked with turned on Trivy SCA across ~140 service repos with --severity HIGH,CRITICAL and --exit-code 1. Within two days every PR in the monorepo-adjacent Node services was red, all pointing at the same culprit: a transitive glibc and OpenSSL chain pulled in through the base image, plus a lodash advisory deep in the build toolchain that had no patched release on their pinned major. Developers did the rational thing and started merging with the security check set to non-required. The gate was technically working and operationally dead.
The fix was two-fold. First, add --ignore-unfixed so only remediable findings could block, and split the scan into a blocking pass (fixable) and a non-blocking inventory pass (everything, uploaded to the Security tab). Second — the part that actually saved it — establish a baseline so the existing backlog stopped lighting up new PRs. Trivy supports this with a baseline file generated on main:
# One-time on main: snapshot current findings as the baseline
trivy fs --severity HIGH,CRITICAL --ignore-unfixed \
--format json --output .trivy-baseline.json .
# In PR CI: only fail on findings absent from the baseline
trivy fs --severity HIGH,CRITICAL --ignore-unfixed \
--exit-code 1 --baseline .trivy-baseline.json .
New code was held to the bar; the legacy debt became tracked, time-boxed exceptions with owners and expires dates. PR-blocking findings dropped from hundreds to low single digits, the check went back to required, and the backlog burned down on a schedule instead of in a panic. The lesson: a gate calibrated for a greenfield repo will be routed around the instant you point it at a real codebase with history.
Verify
Confirm each layer actually fires before you trust the green check.
# 1. SCA gate trips on a known-vulnerable lockfile
trivy fs --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 .; echo "exit=$?"
# 2. Secret detection catches a planted test credential
printf 'aws_secret_access_key = AKIAIOSFODNN7EXAMPLE\n' > /tmp/leak.tf
gitleaks detect --source /tmp/leak.tf --no-git -v; echo "exit=$?"
# 3. IaC scan flags a deliberately public bucket
checkov -f tfplan.json --framework terraform_plan --compact; echo "exit=$?"
# 4. SARIF parses and contains results
jq '.runs[].results | length' trivy.sarif
Then verify the gate logic itself: open a throwaway PR that introduces one fixable critical dependency and confirm the build goes red, the Security tab shows the finding, and a ticket is created. A gate you have never seen fail is a gate you cannot trust.
Rollout checklist
Measuring the program
A DevSecOps program you cannot measure is a faith-based one. Track at least four metrics:
- MTTR for security findings by severity. Trending up means triage is underwater.
- Escape rate: findings discovered in production (or by external researchers) that the pipeline should have caught. The single best measure of pipeline efficacy.
- False-positive rate per tool. If a tool’s suppressions outnumber its actionable findings, retune or retire it.
- Developer friction: median added PR latency from security stages, and how often developers bypass controls. If bypass is climbing, your gates are miscalibrated.
Going deeper
Inside SARIF: why the format is the linchpin
A SARIF file is just JSON with a fixed shape: a top-level runs[] array (one per tool invocation), each run carrying tool.driver.rules[] (the catalog of checks) and results[] (the actual findings). Each result has a ruleId, a level (error, warning, or note), and locations[] anchoring it to a file and line. Two fields do the heavy lifting:
levelis the field your gate reads. Mapping a scanner’s native severity (Trivy’s CRITICAL, Semgrep’s ERROR) onto SARIFlevelis itself a policy decision — it is where “CVSS 9.8” becomes “blocks the build” or “just annotates.”partialFingerprintslets a platform track the same finding across commits, so moving a function down twenty lines does not spawn a brand-new “alert.” Baseline and “new in this PR” logic ride on these fingerprints, and when two tools report the same issue, consistent fingerprints are what let a dashboard deduplicate instead of double-counting.
A minimal result looks like:
{
"ruleId": "python.lang.security.audit.dangerous-subprocess-use",
"level": "error",
"message": { "text": "Subprocess call with shell=True" },
"locations": [{ "physicalLocation": {
"artifactLocation": { "uri": "app/tasks.py" },
"region": { "startLine": 42 } } }],
"partialFingerprints": { "primaryLocationLineHash": "9f2b…" }
}
From severity to risk: EPSS, KEV, and reachability
CVSS is a static severity score assigned once when the CVE is published; it says nothing about whether this vulnerability is being exploited or whether your code even calls it. Three signals turn severity into risk:
- EPSS (Exploit Prediction Scoring System) is a daily-updated probability (0–1) that a CVE will be exploited in the wild in the next 30 days. A CVSS 9.8 with an EPSS of 0.02 is far less urgent than a CVSS 7.5 with an EPSS of 0.7.
- CISA KEV (Known Exploited Vulnerabilities) is a binary list: this CVE is being actively exploited, right now, in the real world. A KEV hit is an all-hands event regardless of CVSS.
- Reachability asks whether your code has a call path from an entrypoint to the vulnerable function. True reachability analysis builds a call graph and is expensive and ecosystem-limited; a cheap approximation is “is the vulnerable package even imported on a code path we ship?”
The defensible risk function is the intersection: gate hardest on High/Critical ∩ fixable ∩ (reachable ∨ KEV ∨ EPSS above a threshold), and treat everything else as tracked, non-blocking work.
The gate as policy-as-code
The shell gate in Section 6 is readable, but once the rules get interesting (per-team thresholds, KEV overrides, exception expiry) you want them versioned and unit-testable. OPA/Conftest evaluates the SARIF and the exceptions file with a single Rego policy:
# policy/gate.rego (conftest) — one versioned, testable place for the gate
package main
# block on any SARIF result the scanners flagged as an error…
deny[msg] {
result := input.runs[_].results[_]
result.level == "error"
not excepted(result.ruleId)
msg := sprintf("blocking finding: %s", [result.ruleId])
}
# …unless a non-expired, attributed exception covers that rule
excepted(rule) {
e := data.exceptions[_]
e.id == rule
time.parse_ns("2006-01-02", e.expires) > time.now_ns()
}
conftest test --policy policy/ --data .security/exceptions.yaml trivy.sarif now returns the exact blocking findings, and you can write Rego unit tests that assert an expired exception no longer suppresses its rule. The policy is reviewed like any other code.
Break-glass without breaking the audit trail
Sooner or later a production incident collides with a red gate: you must ship a hotfix now, and the build is blocked on a pre-existing finding unrelated to the fix. The wrong answer is commenting out the security stage — it is invisible, it has no expiry, and it tends to stay commented out. Break-glass is the documented version: a labelled override (say, a security-override PR label) that requires a second approver, records who invoked it and why, auto-files a ticket to review the bypass, and — crucially — expires, so the next build re-applies the gate. The bypass is not the problem; an unaudited, permanent bypass is. The audit trail is the entire point.
Securing the scanners themselves
Every scanner action is third-party code running inside your pipeline with access to your GITHUB_TOKEN and your source. Treat the supply chain of your security tooling with the same suspicion it is meant to enforce:
- Pin actions by commit SHA, not a floating tag —
uses: aquasecurity/trivy-action@<sha>— so a compromised tag cannot silently push new code into your build. - Give each job the least token scope it needs. A scan job needs
contents: readandsecurity-events: writefor SARIF upload — notwriteon everything. - Never run untrusted PR code with secrets.
pull_request_targetruns in the context of the base repo with secrets; combining it with a checkout of the fork’s code is a well-known token-exfiltration hole. Scan forks with the plainpull_requesttrigger, which has no secrets and a read-only token.
Tuning: the false-positive economy
A scanner’s value is precision × coverage, and precision is something you operate, not something you install. Measure suppressions per rule: when a rule’s justified suppressions start to outnumber its true positives, it is costing more attention than it saves — retune it (tighten the pattern) or retire it. Suppress at the source with a reason (# nosemgrep: rule-id — validated: input is an internal enum) so the suppression is reviewable in the diff and greppable later, never by deleting the rule from config where the whole org loses it silently. In a monorepo, add path filters so a change to one service does not trigger every service’s scans, and cache the CodeQL database so the deep analysis does not rebuild from scratch on every run.
Practice challenges
Work these in order — they escalate from beginner to advanced. Each has a worked solution; try it before you expand it.
Challenge 1 — Diff-aware SAST that only flags new code (beginner)
Add a Semgrep job to a repo’s pull-request workflow so it scans only the lines the PR changed and uploads its results to the Security tab.
<details> <summary>Solution</summary>
Use the semgrep.yml from Section 2. The load-bearing line is the environment variable:
env:
SEMGREP_BASELINE_REF: ${{ github.event.pull_request.base.sha }}
followed by github/codeql-action/upload-sarif@v3 pointed at semgrep.sarif.
Why: the baseline ref makes semgrep ci compare against the PR’s merge base, so new rules apply to new code only. Without it, every legacy finding lights up on every PR and developers stop reading the check.
</details>
Challenge 2 — Block on fixable, inventory everything else (beginner)
Configure SCA so it fails the build only on fixable High/Critical, yet still records the unfixable ones somewhere you can see them.
<details> <summary>Solution</summary>
Run Trivy twice — a blocking pass and a non-blocking inventory pass:
# Blocking: only remediable High/Critical can turn the build red
trivy fs --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \
--format sarif --output trivy-block.sarif .
# Inventory: everything (incl. unfixed), never fails, uploaded for visibility
trivy fs --severity HIGH,CRITICAL --exit-code 0 \
--format sarif --output trivy-all.sarif .
Upload both with distinct category values.
Why: you gate only on what a developer can fix today (a version bump), but you never lose sight of the unfixable backlog — it becomes tracked work, not a blocked merge. </details>
Challenge 3 — Three layers of secret detection, and prove the backstop fires (intermediate)
Wire pre-commit, CI, and platform push protection for secrets, then demonstrate that the CI layer still catches a secret when a developer bypasses the local hook.
<details> <summary>Solution</summary>
Install the gitleaks pre-commit hook (Section 4), add the gitleaks-action CI job with fetch-depth: 0, and enable org-level push protection. Then prove the backstop:
printf 'aws_secret_access_key = AKIAIOSFODNN7EXAMPLE\n' >> config.tf
git add config.tf
git commit -m "test" --no-verify # skips the local hook on purpose
git push # push protection or CI gitleaks trips
Why: the hook is a convenience the developer can skip with one flag; CI and push protection are the controls they cannot opt out of. Defense in depth means no single bypass leaks a live credential. </details>
Challenge 4 — Scan the Terraform plan, not the HCL, and skip one finding with a reason (intermediate)
Scan a Terraform configuration in a way that evaluates variable-driven and conditional resources as they will actually deploy, then intentionally suppress exactly one finding with an attributed justification.
<details> <summary>Solution</summary>
terraform plan -out=tfplan.binary
terraform show -json tfplan.binary > tfplan.json
checkov -f tfplan.json --framework terraform_plan -o sarif --output-file-path checkov_plan
Suppress one finding inline, above the resource:
# checkov:skip=CKV_AWS_18:Access logging handled centrally by the log-archive account
resource "aws_s3_bucket" "artifacts" {
bucket = "kv-build-artifacts"
}
Why: scanning the plan resolves variables and conditionals that raw HCL leaves ambiguous, so you catch what actually deploys; the mandatory skip reason turns a silent suppression into reviewable evidence. </details>
Challenge 5 — Make exceptions expire (advanced)
Encode a vulnerability exception as data, then add a scheduled job that fails the build when any exception is past its expiry date.
<details> <summary>Solution</summary>
Keep the .security/exceptions.yaml from Section 6, then add:
# .github/workflows/exception-expiry.yml — fail when an allowlist entry is stale
name: exception-expiry
on:
schedule:
- cron: "0 6 * * *"
workflow_dispatch: {}
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Fail the build on any expired exception
run: |
set -euo pipefail
today=$(date -u +%F)
fail=0
# process substitution keeps the loop in THIS shell so 'fail' survives
while read -r id exp; do
if [[ "$exp" < "$today" ]]; then
echo "::error::exception ${id} expired on ${exp}"
fail=1
fi
done < <(yq -r '.exceptions[] | .id + " " + .expires' .security/exceptions.yaml)
exit "$fail"
Why: an exception with no expiry is a permanent hole. The scheduled failure forces a re-decision on a cadence — and note the < <(…) process substitution: piping into while would run the loop in a subshell, so fail would reset to 0 and the job would never actually fail.
</details>
Challenge 6 — Authenticated nightly DAST against staging (advanced)
Add a nightly OWASP ZAP scan that runs against a deployed staging URL, reports without blocking, and actually gets past the login page.
<details> <summary>Solution</summary>
Start from the dast.yml in Section 8 (fail_action: false, on a nightly schedule). The advanced part is authentication — pass ZAP a context/auth so it explores past login, for example a bearer token as a request header:
with:
target: https://staging.example.com
fail_action: false
cmd_options: "-a -config replacer.full_list(0).replacement='Bearer ${{ secrets.DAST_TOKEN }}' -config replacer.full_list(0).matchtype=REQ_HEADER"
For form login or complex flows, provide a ZAP context file instead.
Why: without a running target and credentials, DAST spiders the login page and reports a false all-clear — an authenticated context is the difference between scanning your app and scanning your sign-in screen. The token is a placeholder secret, never a literal. </details>
Common beginner mistakes
- “More scanners means more security.” Turning on ten tools with no aggregation produces ten report formats and a wall of noise nobody triages — which is less secure than three well-tuned tools whose findings land in one place. Coverage without triage is theater. Start narrow, aggregate, then widen.
- “A green security check means the code is secure.” It means the gates you configured did not trip. Absence of findings is not absence of risk — a scanner only knows the rules it has. That is why you measure escape rate (issues found in production the pipeline should have caught), not just the pass rate.
- “Gate on everything High and Critical.” CVSS is a severity label, not a risk assessment. An unreachable, unfixable CVSS 9.8 in a dev-only dependency is noise; a KEV-listed 7.5 on your auth path is an emergency. Gate on fixable ∩ reachable ∩ exploited, and route the rest to the tracked backlog.
- “Pre-commit hooks enforce the policy.” A hook is a convenience the developer runs, and
git commit --no-verifyskips it in one flag. Enforcement lives server-side: the CI job and platform push protection are the controls a developer cannot opt out of. - “We removed the leaked secret from git history, so we’re safe.” The secret was compromised the instant it was committed — it lives in every clone, fork, and CI log that ever saw it. Scrubbing history removes the evidence, not the exposure. The only remediation is to rotate the credential; treat removal as cleanup, not a fix.
- “SAST covers the code, so we don’t need DAST.” SAST reads source and is blind to runtime — a mis-wired auth check, or a header missing only in the deployed config, looks fine on paper. DAST probes the running app and catches exactly those. Different tools, different blind spots.
- “Roll it out to every repo at once.” A gate calibrated for a greenfield project turns every real repo with history red on day one, and developers respond rationally by marking the check non-required. Baseline the existing backlog, gate only on new code, and widen coverage once triage and MTTR are healthy.
Pitfalls
The recurring failure modes are predictable. Gating on raw CVSS instead of fixability and reachability buries teams in unactionable criticals. Treating secret removal as remediation while skipping rotation leaves live credentials exposed. Letting allowlists accumulate without expiry turns every exception into a permanent hole. Running expensive full scans on every PR adds minutes of latency that pushes developers to bypass the stage. And scanning without aggregation produces noise no one owns. Start narrow, gate only on high-confidence high-impact findings, prove the loop works end to end, and widen coverage only once triage and MTTR are healthy. Security that developers route around protects nothing.
Glossary
- DevSecOps — building automated security checks into the CI/CD pipeline so every change is scanned as it is made, rather than audited manually after release.
- Shift-left — moving each check as early in the lifecycle as it can usefully run, because flaws are cheaper to fix the earlier they are caught.
- SAST (Static Application Security Testing) — analyzing source code without running it, to find insecure patterns and dataflow (e.g., Semgrep, CodeQL, SonarQube).
- DAST (Dynamic Application Security Testing) — probing a running instance of the app from the outside, the way an attacker would (e.g., OWASP ZAP).
- SCA (Software Composition Analysis) — scanning your dependencies for known vulnerabilities (e.g., Trivy, Grype, Snyk).
- IaC scanning — checking infrastructure-as-code (Terraform, Kubernetes, Helm) for misconfigurations before it deploys (e.g., Checkov, tfsec).
- Secret detection — finding credentials, keys, and tokens accidentally committed to code (e.g., gitleaks).
- Container image scanning — scanning a built image’s OS packages, layers, and baked-in dependencies, not just the source lockfiles.
- SBOM (Software Bill of Materials) — a machine-readable inventory of everything in a build; scanning it decouples the scan from the build (e.g., Syft → CycloneDX).
- CVE — a public identifier for one specific known vulnerability.
- CVSS — a 0–10 severity score attached to a CVE; a static label, not a measure of your risk.
- EPSS — a daily-updated probability (0–1) that a CVE will be exploited in the wild soon.
- KEV — CISA’s Known Exploited Vulnerabilities list; a binary “this is being exploited right now” flag.
- SARIF — the JSON interchange format every scanner can emit, so five tools land in one deduplicated dashboard.
- Policy gate (quality gate) — the single decision point that reads all findings and decides what turns the build red.
- Allowlist / exception — an explicit, attributed, expiring record that a specific finding is accepted for now.
- Break-glass — a documented, audited, time-boxed override for shipping under an incident without disabling the gate.
- Diff-aware / baseline — scanning (or gating on) only what changed relative to a reference, so legacy findings do not block new work.
- Reachability — whether your code actually has a call path to a vulnerable function; a vulnerability you never call is lower risk.
- Taint / dataflow analysis — tracing untrusted input through the code to a dangerous sink; the deeper analysis CodeQL performs.
- False positive / false negative — a flagged non-issue vs. a real issue the scanner missed; both erode trust, in opposite ways.
- Push protection — a platform control that rejects a git push containing a recognized secret before it reaches the remote.
- Pre-commit hook — a local git hook that runs checks before a commit is created; a convenience, not enforcement (
--no-verifyskips it). - Distroless — a minimal base image with no shell or package manager, shrinking the attack surface and the CVE count.
- OPA / Rego / Conftest — Open Policy Agent and its Rego language, run by Conftest, to express gate rules as versioned, testable policy-as-code.
- DefectDojo — an open-source vulnerability-management platform that imports SARIF and native formats for cross-repo triage.
- MTTR (Mean Time To Remediate) — how long, on average, a security finding stays open; a core health metric.
- Escape rate — the count of issues found in production that the pipeline should have caught; the single best measure of its efficacy.