In a nutshell
Picture a restaurant franchise with four hundred outlets. Every outlet serves an identical burger — not because four hundred managers each memorised the recipe, but because head office writes one recipe, tests it, and ships it. A store doesn’t reinvent the burger; it follows the card. When head office improves the recipe or pulls a recalled ingredient, every store changes on the next shift — no four-hundred-store scramble, no store secretly cooking its own version.
A Jenkins shared library is head office for your build pipelines. Instead of four hundred repos each carrying a copied-and-mutated Jenkinsfile (four hundred snowflakes, four hundred ways to break), one versioned library owns the real pipeline logic. Each repo keeps a tiny Jenkinsfile that essentially says “build me the standard way; my app is called payments-api” — and the library does the rest. Patch the library once and every repo picks it up on its next build, with zero pull requests to those repos.
The library is a plain Git repo with three magic folders: vars/ holds your custom pipeline commands (the recipe cards teams call by name), src/ holds real Groovy classes for logic that deserves testing (the test kitchen), and resources/ holds supporting files like Kubernetes pod templates. Repos opt into a version — @Library('lib@v3') — exactly like pinning a software dependency, so you can ship changes on the “v3” line while a frozen, regulated repo stays on v3.4.1 until it’s ready. And because the library is code, you unit-test it before its tag moves, and lock it down with Configuration-as-Code so no team can quietly swap in a fork.
Here’s the mental model to hold onto: a shared library is a product, and your repos are its customers. It has an API (the steps in vars/), versions (Git tags), tests (JenkinsPipelineUnit), and a governance boundary (a trusted global library that repos can call but not fork). Treat it like a copy-paste template and it rots into snowflakes; treat it like a platform and one engineer can patch a CVE across the whole estate before lunch.
Level: Advanced · Time: ~30 min
Prerequisites: You should already be comfortable authoring a declarative Jenkinsfile — pipeline, agent, stages, steps, post, and the credentials store — all covered in Jenkins fundamentals: declarative pipelines, Jenkinsfile & agents. A little Groovy helps (closures and maps especially), and the ephemeral-agent section builds directly on self-hosted, autoscaling ephemeral Kubernetes runners. This lesson is the org-wide governance sequel to that fundamentals lesson: it owns versioned shared libraries, Configuration-as-Code, agent tuning, secrets brokering (see Vault dynamic secrets for CI/CD), and testing the Groovy itself. The very same pattern in another tool is GitHub Actions reusable workflows as a platform — worth reading side by side.
After this lesson you will be able to:
- Lay out a shared library correctly across
vars/,src/, andresources/, and explain what Jenkins does with each directory. - Write a custom DSL step (
standardPipeline) so consuming repos need only a one-lineJenkinsfile. - Model config as a typed, validated
src/class instead of an untyped map — and know why it mustimplement Serializable. - Version the library with branches, moving major tags, and exact tags, and choose the right pin for a normal repo versus a frozen, regulated one.
- Distinguish global/trusted from folder/untrusted libraries and lock the golden library down in JCasC (
allowVersionOverride: false). - Unit-test
vars/steps andsrc/classes with JenkinsPipelineUnit on plain JVM CI, so a bad commit never reaches a moving tag.
Read the diagram left → right. Four hundred repos each carry a one-line Jenkinsfile that pins a library version and calls one step; that step lives in the shared library (vars/ custom commands, src/ typed classes, resources/ pod templates); the library composes a single golden pipeline — Build, Test, Scan, Publish — that every repo runs identically; and versioning plus unit tests plus a trusted, un-overridable global library make that pipeline safe to change without a four-hundred-PR campaign. Every numbered section below is a detail of one of those boxes.
Every Jenkins estate decays the same way: each team copies a Jenkinsfile from the repo next door, mutates it, and within a year you have four hundred snowflakes and no way to roll out a CVE patch without a four-hundred-PR campaign. The fix is a platform, not a template. One versioned shared library owns the pipeline logic, repos call a single entrypoint, and the controller is rebuildable from code. This guide builds that platform end to end: library layout, custom DSL, versioning, self-onboarding folders, JCasC, ephemeral Kubernetes agents, secrets, and unit tests for the Groovy itself.
1. Structure the shared library: vars, src, resources
A Jenkins shared library is a Git repo with a fixed, magic directory layout. Jenkins only recognizes three top-level directories, and each has a distinct role:
pipeline-library/
vars/ # global variables -> custom DSL steps
standardPipeline.groovy # exposes step standardPipeline(...)
standardPipeline.txt # help text shown in Snippet Generator
dockerBuild.groovy
notifySlack.groovy
src/ # Groovy classes on the classpath (org.foo.*)
com/kloudvin/ci/
BuildConfig.groovy
Semver.groovy
resources/ # non-Groovy files, loaded with libraryResource
com/kloudvin/ci/
pod-templates/jnlp-maven.yaml
sonar-project.properties.tmpl
The contract that matters: every .groovy file in vars/ becomes a global step named after the file. vars/standardPipeline.groovy defines a call() method, and pipelines invoke it as standardPipeline { ... }. That is the entire trick behind a custom DSL.
// vars/standardPipeline.groovy
def call(Map config = [:]) {
// config is the closure-populated map from the Jenkinsfile
def cfg = new com.kloudvin.ci.BuildConfig(config)
pipeline {
agent {
kubernetes {
yaml libraryResource("com/kloudvin/ci/pod-templates/jnlp-maven.yaml")
}
}
options {
timeout(time: cfg.timeoutMinutes, unit: 'MINUTES')
buildDiscarder(logRotator(numToKeepStr: '30'))
disableConcurrentBuilds()
}
stages {
stage('Build') { steps { container('maven') { sh 'mvn -B clean package' } } }
stage('Test') { steps { container('maven') { sh 'mvn -B test' } } }
stage('Scan') { when { expression { cfg.scanEnabled } }
steps { sonarScan(cfg) } }
stage('Publish') { when { branch 'main' }
steps { dockerBuild(cfg) } }
}
post {
always { junit testResults: '**/surefire-reports/*.xml', allowEmptyResults: true }
failure { notifySlack(status: 'FAILED', config: cfg) }
}
}
}
Keep
vars/files thin. They are orchestration glue; anything with real logic (parsing, version math, config validation) belongs insrc/as a unit-testable class. Avarsfile that grows past ~80 lines is asrcclass trying to escape.
The BuildConfig class lives in src/ and gives you a typed, validated config object instead of a bag of untyped map keys:
// src/com/kloudvin/ci/BuildConfig.groovy
package com.kloudvin.ci
class BuildConfig implements Serializable {
String appName
String registry = 'registry.kloudvin.internal'
Integer timeoutMinutes = 30
Boolean scanEnabled = true
BuildConfig(Map cfg) {
this.appName = cfg.appName ?: { throw new IllegalArgumentException('appName is required') }()
if (cfg.registry) this.registry = cfg.registry
if (cfg.timeoutMinutes) this.timeoutMinutes = cfg.timeoutMinutes as Integer
if (cfg.scanEnabled != null) this.scanEnabled = cfg.scanEnabled
}
}
implements Serializableis not optional. Pipeline state is persisted to disk across restarts and resumed; any object that survives across ashstep or stage boundary must serialize. Forgetting this throwsNotSerializableExceptionat the worst possible moment.
2. Write custom DSL steps and a one-line Jenkinsfile
The whole point is that consuming repos do not author pipeline logic. Their Jenkinsfile declares intent and nothing else:
// A consuming repo's entire Jenkinsfile
@Library('kloudvin-pipeline@v3') _
standardPipeline {
appName = 'payments-api'
timeoutMinutes = 45
scanEnabled = true
}
That trailing _ after the @Library annotation is required: the annotation must attach to something, and _ is the idiomatic no-op import. Now every supporting step is its own vars/ file, composed by standardPipeline:
// vars/sonarScan.groovy
def call(com.kloudvin.ci.BuildConfig cfg) {
withSonarQubeEnv('kloudvin-sonar') {
container('maven') {
sh "mvn -B sonar:sonar -Dsonar.projectKey=${cfg.appName}"
}
}
timeout(time: 10, unit: 'MINUTES') {
// qualityGate aborts the build if the gate fails
waitForQualityGate abortPipeline: true
}
}
// vars/dockerBuild.groovy
def call(com.kloudvin.ci.BuildConfig cfg) {
def tag = "${cfg.registry}/${cfg.appName}:${env.GIT_COMMIT.take(12)}"
container('kaniko') {
sh """
/kaniko/executor \
--context=`pwd` \
--dockerfile=Dockerfile \
--destination=${tag} \
--cache=true
"""
}
}
This composition is the source of the platform’s power. Patch dockerBuild.groovy once – add a scan, switch the builder, change the registry – and every repo on that library version gets it on the next build, zero PRs to product repos.
3. Version the library: tags, trusted vs untrusted
A platform you cannot version is a platform you cannot change safely. The @Library('name@ref') annotation pins to any Git ref: a branch, a tag, or a commit SHA. Use semantic tags and let teams opt into a major line:
| Reference style | Example | When to use |
|---|---|---|
| Floating branch | @Library('lib@main') |
Internal platform repos only; you accept breakage |
| Pinned major tag | @Library('lib@v3') |
Default for product repos; moving tag tracks v3.x |
| Exact tag | @Library('lib@v3.4.1') |
Repos that must freeze, e.g. during a compliance window |
Make v3 a moving tag you re-point to the latest v3.x release, so consumers pin to a major line and pick up backward-compatible fixes automatically:
# Cut a patch and advance the major-line pointer
git tag -a v3.4.2 -m "fix: kaniko cache key"
git tag -f v3 v3.4.2 # move the v3 alias forward
git push origin v3.4.2
git push -f origin v3 # consumers on @v3 get this on next build
The security model is the second axis. Libraries configured at the global/folder level by an admin run as trusted – they may call internal Jenkins APIs and @Grab dependencies. Libraries loaded dynamically by a Jenkinsfile via the library step are untrusted and run inside the Groovy sandbox. The rule for a platform:
Configure the golden library as a global trusted library in JCasC, marked implicit load off and allow default version override off. That stops a product repo from pinning an arbitrary fork or an older, unpatched tag. Treat anything a repo can self-declare as untrusted and sandboxed.
4. Template multibranch and organization folders for self-onboarding
You do not want to click “New Item” four hundred times. An Organization Folder (GitHub or Bitbucket) scans an org, and for every repo containing a Jenkinsfile it auto-creates a multibranch project – branches and PRs included. Onboarding a repo becomes “add a Jenkinsfile,” nothing more.
Define it in code through the Job DSL seed job so the folder itself is reproducible:
// jobs/seed-org-folder.groovy (Job DSL)
organizationFolder('kloudvin-services') {
description('Auto-onboards every repo with a Jenkinsfile')
organizations {
github {
repoOwner('kloudvin')
apiUri('https://api.github.com')
credentialsId('github-app-kloudvin')
traits {
gitHubBranchDiscovery { strategyId(1) } // branches
gitHubPullRequestDiscovery { strategyId(1) } // PRs from origin
}
}
}
projectFactories {
workflowMultiBranchProjectFactory { scriptPath('Jenkinsfile') }
}
orphanedItemStrategy {
discardOldItems { daysToKeep(7); numToKeep(20) }
}
triggers { periodicFolderTrigger { interval('1d') } }
}
The GitHub App credential (github-app-kloudvin) matters at scale: a personal token shares one rate-limit bucket across the whole estate and starves at a few hundred repos, while a GitHub App gets per-installation limits and finer scopes.
5. Manage the controller with JCasC and seed jobs
Configuration-as-Code (the configuration-as-code plugin) renders the controller’s entire configuration from YAML, replacing point-and-click setup. Set CASC_JENKINS_CONFIG to a file, directory, or URL; Jenkins applies it on boot and on a reload from Manage Jenkins -> Configuration as Code -> Reload.
# jenkins.yaml -- the controller, as code
jenkins:
systemMessage: "KloudVin CI -- managed by JCasC. Do not configure by hand."
numExecutors: 0 # controller runs no builds; agents only
authorizationStrategy:
roleBased:
roles:
global:
- name: "admin"
permissions: ["Overall/Administer"]
assignments: ["platform-team"]
clouds:
- kubernetes:
name: "k8s"
serverUrl: "https://kubernetes.default"
namespace: "jenkins-agents"
jenkinsUrl: "http://jenkins.jenkins.svc:8080"
containerCapStr: "50"
unclassified:
globalLibraries:
libraries:
- name: "kloudvin-pipeline"
defaultVersion: "v3"
implicit: false
allowVersionOverride: false # repos cannot pin a fork or old tag
retriever:
modernSCM:
scm:
git:
remote: "https://github.com/kloudvin/pipeline-library.git"
credentialsId: "github-app-kloudvin"
jobs:
- script: |
pipelineJob('seed') {
definition {
cps {
script(readFileFromWorkspace('jobs/seed-org-folder.groovy'))
sandbox(false)
}
}
}
Bootstrap order is the subtle part: JCasC applies first and creates the seed job; the seed job runs Job DSL and creates the org folders. Keep jenkins.yaml and jobs/ in one repo, mount it into the controller pod, and the entire Jenkins is a git revert away from any prior state.
6. Ephemeral agents on Kubernetes: pod template and resource tuning
A pool of static agents accrues state between builds and bills you while idle. The Kubernetes plugin instead launches one pod per build and deletes it on completion. The pod template – referenced earlier via libraryResource – defines the containers a build can container('name') { ... } into:
# resources/com/kloudvin/ci/pod-templates/jnlp-maven.yaml
apiVersion: v1
kind: Pod
spec:
containers:
- name: maven
image: maven:3.9-eclipse-temurin-21
command: ["sleep"]
args: ["infinity"]
resources:
requests: { cpu: "500m", memory: "1Gi" }
limits: { cpu: "2", memory: "2Gi" }
- name: kaniko
image: gcr.io/kaniko-project/executor:v1.23.2-debug
command: ["sleep"]
args: ["infinity"]
resources:
requests: { cpu: "500m", memory: "1Gi" }
limits: { cpu: "1", memory: "2Gi" }
Tuning notes from production:
- Always set requests and limits. Without requests the scheduler bin-packs blindly and OOM-kills agents under load; without limits one runaway build starves a node. Memory
limit == request(Guaranteed QoS) for build containers avoids eviction surprises. - Do not declare your own
jnlpcontainer unless you must. The plugin injects the inbound-agentjnlpcontainer automatically; redefining it with the wrong image silently breaks the connection back to the controller. containerCapandpodRetention. Cap concurrent pods (containerCapStr: "50"above) so a thundering herd cannot DoS the cluster, and setpodRetention: neverso failed pods do not pile up.- Idle timeout to zero. With ephemeral pods there is no warm pool to keep – agents scale to zero between builds, and you pay only for active build time.
7. Secure secrets: credentials binding and external Vault
Secrets never live in a Jenkinsfile or a library file. They live in the credentials store, and the platform binds them into the build environment only for the steps that need them, masked in the log. Wrap this in a vars/ step so consumers cannot fumble the binding:
// vars/withRegistryCreds.groovy
def call(Closure body) {
withCredentials([usernamePassword(
credentialsId: 'registry-push',
usernameVariable: 'REG_USER',
passwordVariable: 'REG_PASS')]) {
body() // $REG_USER / $REG_PASS exist only here and are masked in logs
}
}
For anything beyond low-stakes secrets, do not store them in Jenkins at all – broker them from HashiCorp Vault so rotation happens outside the CI system and Jenkins holds only short-lived leases. The HashiCorp Vault plugin authenticates the controller (AppRole or Kubernetes auth) and injects paths per build:
stage('Deploy') {
steps {
withVault(configuration: [vaultUrl: 'https://vault.kloudvin.internal',
vaultCredentialId: 'vault-approle'],
vaultSecrets: [[ path: 'secret/data/ci/payments',
secretValues: [[envVar: 'DB_PASSWORD', vaultKey: 'db_password']] ]]) {
sh 'deploy --db-pass "$DB_PASSWORD"'
}
}
}
Prefer Vault’s Kubernetes auth method over a long-lived AppRole secret-id: the agent pod’s ServiceAccount token becomes the Vault login, so there is no static credential to leak. Pair it with short TTL leases so a compromised build log buys an attacker minutes, not months.
8. Test the pipeline code with Jenkins Pipeline Unit
Pipeline logic is code, and untested code in vars/ fails in production at 2am. The JenkinsPipelineUnit framework mocks the pipeline DSL so you can unit-test vars/ steps and src/ classes on plain JVM CI – no Jenkins required. Wire it into a Gradle/Maven build that runs on every library PR:
// test/com/kloudvin/ci/StandardPipelineSpec.groovy
import com.lesfurets.jenkins.unit.BasePipelineTest
import org.junit.Before
import org.junit.Test
import static org.junit.Assert.assertEquals
class StandardPipelineSpec extends BasePipelineTest {
@Before void setUp() {
super.setUp()
// register mocks for any DSL step the library calls
helper.registerAllowedMethod('sh', [String]) { _ -> }
helper.registerAllowedMethod('libraryResource', [String]) { 'apiVersion: v1' }
helper.registerAllowedMethod('container', [String, Closure]) { _, c -> c() }
}
@Test void buildConfigRejectsMissingAppName() {
try {
new com.kloudvin.ci.BuildConfig([:])
assert false : 'expected IllegalArgumentException'
} catch (IllegalArgumentException e) {
assertEquals('appName is required', e.message)
}
}
@Test void scanStageRunsWhenEnabled() {
def script = loadScript('vars/sonarScan.groovy')
// assert callstack / step invocations via printCallStack()
assertJobStatusSuccess()
}
}
Run it in the library repo’s own pipeline so a bad commit never reaches a moving tag:
./gradlew test # JenkinsPipelineUnit specs, fast, no Jenkins
This closes the loop: the library that everything depends on is itself gated by tests before its tag moves.
Verify
Confirm each layer is wired correctly before declaring the platform live:
# 1. JCasC parsed cleanly (no boot errors, config visible)
curl -s -u "$JENKINS_USER:$JENKINS_TOKEN" \
https://jenkins.kloudvin.internal/configuration-as-code/ | grep -q "Reload"
# 2. The global library is registered at the expected version
curl -s -u "$JENKINS_USER:$JENKINS_TOKEN" \
"https://jenkins.kloudvin.internal/manage/configureTools/" | grep -q "kloudvin-pipeline"
# 3. Library unit tests are green
./gradlew test --console=plain
- A test repo with the one-line
Jenkinsfilebuilds end to end and produces a tagged image. - Moving the
v3tag to a new patch causes consuming repos to pick it up on their next build, with no PR to those repos. - An agent pod appears in
kubectl get pods -n jenkins-agentsduring a build and is gone within the retention window after it completes. - A secret bound via
withCredentialsshows as****in the build log, never plaintext.
Enterprise scenario
A platform team running ~600 microservice repos on a single Jenkins controller hit a hard wall during a Log4Shell-class incident. The vulnerable logging dependency was baked into the Docker build step that every team had copied into its own Jenkinsfile. There was no central step to patch – the “fix” was a 600-repo PR campaign that would have taken weeks while the window stayed open.
The constraint: they could not break in-flight releases, and several regulated repos were frozen under a change-control window and legally could not take the new behavior until their next window. A flag-day forced upgrade was off the table.
They solved it by collapsing the Docker logic into a single dockerBuild step in the shared library and switching every repo to the one-line standardPipeline entrypoint pinned to a moving major tag. The patched builder shipped behind that tag, so unfrozen repos picked it up on their next build automatically – no PRs. Frozen repos stayed safe by pinning an exact tag until their window opened:
// Frozen, change-controlled repos -- pinned to an exact patch, opt in later
@Library('kloudvin-pipeline@v3.4.1') _
standardPipeline { appName = 'ledger-core' }
Crucially, JCasC had set allowVersionOverride: false on the global library, so no repo could silently pin a stale fork and dodge the fix indefinitely – the platform team could see every repo’s effective version and drive the laggards. What would have been a multi-week, multi-hundred-PR scramble became a single tagged library release plus a short list of frozen repos to track. That is the entire economic argument for the platform.
Going deeper
The eight sections above are enough to run a shared-library platform. This section is for the person who has to operate one when it misbehaves — the internals that decide whether your library is a stable product or a flaky one.
1. The CPS transform: your pipeline Groovy isn’t quite Groovy
Pipeline code in vars/ and in a Jenkinsfile does not run like an ordinary Groovy script. The workflow-cps plugin rewrites it into Continuation-Passing Style so a running build can be paused, serialised to disk, and resumed — after a controller restart, mid-sh, hours later. That single capability is why Jenkins pipelines survive a reboot, and it is the source of every weird rule you will hit:
- Local variables must be
Serializable. Between one step and the next the entire local state is written toprogram.dat. APattern, aMatcher, an open stream, aJsonSlurperresult — none serialise, and you getNotSerializableExceptionat a stage boundary, not at the offending line. - Some ordinary Groovy idioms break or behave oddly under CPS — iterating with certain closures, streaming over a lazy sequence, calling deep into a third-party library mid-flow. The symptom is a
CpsCallableInvocationerror or a silently wrong result. @NonCPSis the escape hatch, with a catch. Annotate a pure helper (string munging, sorting a list, rendering a template) with@NonCPSand it runs as normal, fast Groovy — but it must not call any pipeline step (sh,echo,container) and its local state is not saved across a checkpoint. Use it for computation, never for orchestration.
The practical rule for library authors: keep vars/ steps as thin orchestration and put pure computation in @NonCPS methods on src/ classes. Your BuildConfig parsing, a semver bump, a properties-file render — all @NonCPS. The sh/container/withCredentials calls stay in CPS land.
2. A vars/ file is an object, not just a call()
vars/standardPipeline.groovy looks like a script with a lone call() method, but Jenkins exposes each vars/ file as a global variable — a singleton-like object named after the file. That buys you more than one entrypoint:
// vars/deploy.groovy -> a global variable named `deploy`
def call(Map cfg) { toEnv(cfg.env) } // deploy(env: 'prod')
def toStaging() { toEnv('staging') } // deploy.toStaging()
def toEnv(String e) { echo "deploying to ${e}" }
Now a pipeline can call deploy(env: 'prod') or deploy.toStaging() — the file is a small API, not a single function. The companion vars/deploy.txt (Markdown) is surfaced in the Snippet Generator and the Global Variables Reference page, so a well-documented library is self-describing inside Jenkins. And libraryResource('path') reads a file from resources/ as a string (that is how the pod template reaches the kubernetes agent) — it is a sandbox-safe step, so even untrusted callers can use it.
3. Trusted libraries bypass the sandbox — so they are a supply-chain target
The trusted/untrusted split from section 3 is not a nicety; it is a security boundary with teeth. Untrusted code — a Jenkinsfile, or a library loaded dynamically by a repo — runs inside the Groovy sandbox (the script-security plugin). Every method call is checked against an allow-list; a disallowed signature throws RejectedAccessException and waits for an admin in Manage Jenkins → In-process Script Approval. Trusted code — your global library, configured by an admin in JCasC — runs outside the sandbox, with full JVM and Jenkins-API access.
That is exactly what you want for a platform library (it needs to do real work), and exactly why the library repo becomes your single most sensitive piece of infrastructure. A merged pull request to it is arbitrary code execution on the controller, inherited by every one of the hundreds of repos on their next build. Treat it accordingly:
- Protected
main, mandatory review from the platform team, no direct pushes. - Signed, and ideally immutable, release tags; restrict who can move the
v3alias. - Pin the library’s own plugin and tool versions, and make its unit tests (section 8) a required status check.
A shared library is a force multiplier in both directions: one good fix reaches everyone — and so does one poisoned commit.
4. The controller runs pipeline Groovy single-threaded — keep it thin
Each build’s CPS program executes on the controller in a single-threaded flow-execution loop (the “CPS VM thread”). Ordinary Groovy you write directly in the pipeline — a big for loop, a regex over a 50 MB log, parsing a large JSON payload in-process — runs on that thread, on the controller, and blocks it. At hundreds of repos and dozens of concurrent builds, a handful of heavy Groovy loops can make the whole controller sluggish while agents sit idle.
Two levers keep the controller healthy:
- Push work to the agent. Anything heavy belongs in a
shstep running in a container, not in Groovy on the controller. Groovy decides what to run; the agent does the running. - Tune the durability hint.
MAX_SURVIVABILITY(the default) writes pipeline state to disk very frequently so a crash loses nothing — at an I/O cost. For high-throughput, easily-retried builds you can setoptions { durabilityHint('PERFORMANCE_OPTIMIZED') }(or a global default), which writes far less and runs noticeably faster, accepting that a hard controller crash may lose an in-flight build’s progress.
5. Library caching and the mutable-tag hazard
“Move the v3 tag and everyone gets it on the next build” has a footnote. Global libraries have a “cache fetched versions on the controller” option with a TTL. With caching on, a @Library('lib@v3') build may be served the previously cached checkout of v3 until the cache expires — so a freshly force-moved tag does not always propagate instantly. That is usually a feature (far fewer Git fetches at scale), but it ambushes teams debugging “why didn’t my fix ship?” Know your cache TTL, and disable caching for a library you iterate on rapidly.
There is a deeper, related hazard: a moving tag is mutable state in Git. Force-pushing v3 rewrites what “v3” means for everyone at once, and a compromised or fat-fingered move is a supply-chain event with no diff trail. A more robust pattern is immutable release tags (v3.4.2, never re-pointed) plus a thin, reviewed indirection — a defaultVersion in JCasC that you bump through a pull request, or a version file — so “advancing the line” becomes an auditable change instead of a silent git push -f.
6. Testing deeper: call-stack regression, and where unit tests stop
JenkinsPipelineUnit does more than assert that a bad config throws. It records every mocked DSL call your pipeline makes, in order, as a call stack you can print and snapshot:
@Test void standardPipelineCallStack() {
def script = loadScript('vars/standardPipeline.groovy')
script.call(appName: 'demo')
printCallStack() // prints the recorded step sequence
// Extend BaseRegressionTest and testNonRegression('name') writes that
// sequence to a .txt fixture and fails the build if the shape changes.
}
Extend BaseRegressionTest and the framework snapshots that call stack to a .txt fixture and fails the build when it changes — a regression test for pipeline shape: reorder a stage or drop a step and the diff catches it, no Jenkins required.
Know the boundary, though. JenkinsPipelineUnit mocks the DSL — it proves your Groovy composes the right calls, not that waitForQualityGate really talks to SonarQube or that a credential actually binds and masks. For that you need one integration smoke test: a throwaway controller stood up from your JCasC (or jenkinsfile-runner / a Testcontainers image) running the real one-line Jenkinsfile end to end against a test repo. The layered answer: unit tests (fast, every PR, catch logic and regressions) plus one real smoke pipeline (slow, catches plugin and wiring reality). Add the Declarative Linter (ssh <controller> declarative-linter, or the CLI) as a pre-commit syntax gate, and use Replay in the UI to iterate on a step without cutting a tag.
Practice challenges
Work these in order — each builds on the last, escalating from “write the one-liner a consuming team writes” to “lock the platform down and test it.” No Jenkins controller is required; every snippet is schema-correct Groovy/YAML you can reason about on paper, and the unit-test solutions run on plain JVM CI.
1. (Beginner) The consumer’s whole Jenkinsfile. A team owns a repo whose app is called orders-api. Write the complete Jenkinsfile that opts into the kloudvin-pipeline library’s v3 line and runs the standard pipeline — nothing more.
<details> <summary>Solution</summary>
@Library('kloudvin-pipeline@v3') _
standardPipeline {
appName = 'orders-api'
}
Why: the @Library annotation must attach to something, so the no-op _ import follows it; standardPipeline { ... } is the closure that populates the Map the library’s call(Map) consumes. The repo declares intent, not logic.
</details>
2. (Beginner) Add a custom step, and predict its name. Add a file vars/helloStep.groovy that echoes a greeting. What step name will pipelines call it by, and how do you invoke it with an argument?
<details> <summary>Solution</summary>
// vars/helloStep.groovy -> exposed as a step named `helloStep`
def call(String who = 'world') {
echo "hello, ${who}"
}
Invoke it as helloStep 'payments'. Why: every .groovy file in vars/ becomes a global step whose name is the filename minus .groovy, and its call() method is what runs — so helloStep.groovy is called as helloStep(...).
</details>
3. (Intermediate) A validated, serialisable config field. Extend BuildConfig (in src/) with a deployEnv field that must be either staging or prod, defaulting to staging, and rejects anything else with a clear message. Keep it unit-testable.
<details> <summary>Solution</summary>
package com.kloudvin.ci
class BuildConfig implements Serializable {
String appName
String deployEnv = 'staging'
private static final List<String> ENVS = ['staging', 'prod']
BuildConfig(Map cfg) {
this.appName = cfg.appName ?: { throw new IllegalArgumentException('appName is required') }()
if (cfg.deployEnv) {
if (!(cfg.deployEnv in ENVS)) {
throw new IllegalArgumentException("deployEnv must be one of ${ENVS}, got '${cfg.deployEnv}'")
}
this.deployEnv = cfg.deployEnv
}
}
}
Why: validating in the constructor fails a misconfigured repo at pipeline-parse time with a precise message, instead of deep inside a stage an hour later; implements Serializable lets the object survive the controller persisting and resuming the build across a sh step or a restart.
</details>
4. (Intermediate) Ship a CVE fix to 400 repos — and freeze one. A vulnerable transitive dependency lives in dockerBuild. You have fixed it. Write the tag commands that ship it on the v3 line, plus the two Jenkinsfiles: one for a normal repo (catalog-api, auto-adopts) and one for a change-frozen regulated repo (ledger-core, must stay put).
<details> <summary>Solution</summary>
# Ship the fix on the v3 line and advance the moving alias
git tag -a v3.4.2 -m "fix: bump vulnerable transitive dep in dockerBuild"
git tag -f v3 v3.4.2
git push origin v3.4.2 && git push -f origin v3
// catalog-api — tracks the moving major tag, adopts on next build
@Library('kloudvin-pipeline@v3') _
standardPipeline { appName = 'catalog-api' }
// ledger-core — frozen to an exact patch until its change window opens
@Library('kloudvin-pipeline@v3.4.1') _
standardPipeline { appName = 'ledger-core' }
Why: a moving major tag (v3) lets the fleet pick up backward-compatible fixes with zero PRs, while an exact tag (v3.4.1) freezes a regulated repo until it can formally adopt the change — one library release safely serves both populations.
</details>
5. (Advanced) Lock the library down in JCasC. Write the unclassified.globalLibraries YAML that registers kloudvin-pipeline at default version v3 so that a product repo cannot override the version or pin a fork, and the library is not auto-loaded into every build. Explain the two flags that do the work.
<details> <summary>Solution</summary>
unclassified:
globalLibraries:
libraries:
- name: "kloudvin-pipeline"
defaultVersion: "v3"
implicit: false
allowVersionOverride: false
includeInChangesets: false
retriever:
modernSCM:
scm:
git:
remote: "https://github.com/kloudvin/pipeline-library.git"
credentialsId: "github-app-kloudvin"
Why: allowVersionOverride: false stops a repo pinning @lib@some-fork or an unpatched old tag, so the platform team can see and drive every repo’s effective version; implicit: false means a repo must explicitly @Library-import it (no surprise auto-load), keeping the opt-in deliberate and the trusted blast radius visible.
</details>
6. (Advanced) Unit-test the library with no Jenkins. Write a JenkinsPipelineUnit spec that (a) asserts BuildConfig([:]) throws appName is required, and (b) loads vars/sonarScan.groovy and runs it, mocking every DSL step it calls so the test passes on plain JVM CI.
<details> <summary>Solution</summary>
import com.lesfurets.jenkins.unit.BasePipelineTest
import org.junit.Before
import org.junit.Test
import static org.junit.Assert.assertEquals
class LibrarySpec extends BasePipelineTest {
@Before void setUp() {
super.setUp()
helper.registerAllowedMethod('sh', [String]) { _ -> }
helper.registerAllowedMethod('container', [String, Closure]) { _, c -> c() }
helper.registerAllowedMethod('timeout', [Map, Closure]) { _, c -> c() }
helper.registerAllowedMethod('withSonarQubeEnv', [String, Closure]) { _, c -> c() }
helper.registerAllowedMethod('waitForQualityGate', [Map]) { [status: 'OK'] }
}
@Test void rejectsMissingAppName() {
try {
new com.kloudvin.ci.BuildConfig([:])
assert false : 'expected IllegalArgumentException'
} catch (IllegalArgumentException e) {
assertEquals('appName is required', e.message)
}
}
@Test void sonarScanRunsTheScanner() {
def scan = loadScript('vars/sonarScan.groovy')
scan.call(new com.kloudvin.ci.BuildConfig([appName: 'demo']))
printCallStack() // records the mocked sh 'mvn ... sonar:sonar' + waitForQualityGate
}
}
Why: unit-testing the library on plain JVM CI (no Jenkins) means a broken step is caught on the library’s own PR — before its tag moves and hundreds of repos inherit the bug. Every DSL step the code touches must be registered as a mock, or the test fails with “expected to call … but … was not declared.” </details>
Common beginner mistakes
These are the misconceptions underneath the bugs — the wrong mental model that produces the failure in the first place.
-
“I’ll just put the logic straight in
vars/— it’s simpler.” For three lines, sure; butvars/files can’t be unit-tested cleanly and become untestable orchestration blobs the moment they carry real logic. Right model:vars/is thin glue that composes steps; anything with branches, parsing, or math belongs in asrc/class you can test in isolation. Avarsfile past ~80 lines is asrcclass trying to escape. -
“It’s Groovy, so any Groovy I know will work.” Pipeline code is run through the CPS transform and (for repo-declared libraries) the sandbox, so many ordinary idioms break —
NotSerializableExceptionacross a stage,RejectedAccessExceptionon a “normal” method call, iterators that misbehave. Right model: pipeline Groovy is a restricted, resumable dialect. Keep steps simple, push heavy work toshon the agent, and confine pure computation to@NonCPShelpers. -
“A moving
v3tag means every repo always has the very latest, instantly.” Controller-side library caching can serve a stale checkout ofv3until its TTL expires, and force-moving a tag is itself a mutable, un-diffable change. Right model: understand your cache TTL, and treat the library’s Git repo as production infrastructure — prefer immutable release tags plus a revieweddefaultVersionbump over silentgit push -f. -
“Trusted just means an admin set it up — no big deal.” A trusted library runs outside the Groovy sandbox with full JVM and Jenkins-API access; a malicious or careless merge to it is remote code execution on your controller, inherited by every repo on its next build. Right model: the library is your single most sensitive repo — protected branch, mandatory review, signed tags, and its own test gate.
-
“I unit-tested the Groovy, so the pipeline is tested.” JenkinsPipelineUnit mocks the DSL; it can’t prove
waitForQualityGateactually talks to SonarQube or that a credential really binds and masks. Right model: unit tests catch logic and regressions cheaply on every PR; you still need one real end-to-end smoke pipeline (a test repo on a throwaway controller) to catch plugin and wiring reality. -
“Every repo needs its own tweaked
Jenkinsfilefor its edge cases.” That is exactly how you grow four hundred snowflakes again. Right model: edge cases belong as config passed to the entrypoint (a new flag, a map key, a validatedBuildConfigfield), not as forked pipeline code. Extend the library’s API; keep the repo’sJenkinsfileone line.
Glossary
- Shared library — a Git repo of reusable Jenkins pipeline code that many repos load, so pipeline logic lives in one versioned place instead of being copied into every
Jenkinsfile. vars/— the library directory whose every.groovyfile becomes a global step (custom DSL command) named after the file; holds thin orchestration glue.src/— the library directory holding ordinary Groovy classes on the classpath (e.g.com.kloudvin.ci.BuildConfig); the home for real, unit-testable logic.resources/— the library directory for non-Groovy files (pod templates, property templates), read into a build as a string withlibraryResource('path').- Global variable / custom step — the object Jenkins creates from a
vars/*.groovyfile; itscall()method makesmyStep(...)(ormyStep { ... }) a legal pipeline command, and it may expose extra methods too. - DSL (domain-specific language) — the vocabulary of custom steps your library exposes (
standardPipeline,dockerBuild,sonarScan) so aJenkinsfilereads as intent, not implementation. @Library('name@ref')— the annotation that imports a shared library and pins it to a Git ref: a branch, a tag, or a commit SHA. The trailing_is the required no-op import it attaches to.- Moving major tag — a Git tag (e.g.
v3) you force-re-point to the latestv3.xrelease, so consumers on@v3automatically pick up backward-compatible fixes on their next build. - Exact tag — an immutable pin (e.g.
@v3.4.1) that freezes a repo on a specific release, used for change-controlled or regulated repos that cannot take new behaviour yet. - Trusted vs untrusted library — a library configured by an admin at the global/folder level runs trusted (outside the sandbox, full JVM access); a library a
Jenkinsfileloads dynamically runs untrusted (inside the Groovy sandbox). - Groovy sandbox — the
script-securitymechanism that allow-lists method calls for untrusted code; a disallowed call throwsRejectedAccessExceptionand waits for admin approval in In-process Script Approval. - CPS (Continuation-Passing Style) — the transform (
workflow-cpsplugin) that rewrites pipeline Groovy so a build can be paused, serialised to disk, and resumed after a restart. It is why pipeline state must be serialisable. @NonCPS— an annotation marking a method to run as plain, un-transformed Groovy (fast, for pure computation); such a method must not call pipeline steps and its local state isn’t saved across a checkpoint.implements Serializable/NotSerializableException— any object that lives across a step or stage boundary must serialise, because Jenkins persists and resumes build state; a non-serialisable object throwsNotSerializableExceptionmid-build.- JCasC (Configuration-as-Code) — the
configuration-as-codeplugin that renders the controller’s entire configuration from YAML pointed to byCASC_JENKINS_CONFIG, replacing point-and-click setup. allowVersionOverride/implicit— JCasC flags on a global library:allowVersionOverride: falsestops repos pinning a fork or old tag;implicit: falsemeans a repo must explicitly@Library-import it rather than it auto-loading everywhere.- Organization Folder — a Jenkins item that scans a GitHub/Bitbucket org and auto-creates a multibranch project for every repo containing a
Jenkinsfile, so onboarding is just “add aJenkinsfile.” - Multibranch pipeline — a project that discovers a
Jenkinsfileper branch and pull request and builds each one, so pipeline changes review alongside the code. - Job DSL / seed job — a Groovy script (run by a “seed” job) that generates Jenkins jobs and folders from code, making the folder structure itself reproducible.
- Ephemeral agent / pod template — the Kubernetes-plugin model where each build runs in a fresh pod (defined by a pod template) that is deleted on completion, so agents scale to zero and hold no state between builds.
containerCap— the Kubernetes cloud cap on concurrent agent pods, so a thundering herd of builds cannot exhaust the cluster.withCredentials/ credentials store — the step and store that bind a secret into the build environment only for the wrapped block, masked as****in the log, so secrets never live in aJenkinsfile.- JenkinsPipelineUnit — a framework that mocks the pipeline DSL so
vars/steps andsrc/classes can be unit-tested on plain JVM CI with no running Jenkins;printCallStack()andBaseRegressionTestcatch shape regressions. - Durability hint — a per-pipeline or global setting (
MAX_SURVIVABILITYvsPERFORMANCE_OPTIMIZED) trading how often build state is written to disk against execution speed.