Building a Kubernetes Operator with Kubebuilder: CRDs, Reconciliation & Production Hardening

In a nutshell

Level: Expert, with a beginner on-ramp · Time: ~31 min

An operator is software that encodes an operations team’s knowledge for running an application, so a computer can do the job a human used to do by hand. Picture the senior engineer who knows exactly how to stand up your cache, keep an eye on it, resize it when traffic grows, rebuild it when a node dies, and clean it up when it’s decommissioned. An operator is that engineer’s runbook turned into a program that never sleeps — a robot SRE that watches your app around the clock and fixes drift the moment it appears.

It has two halves, and you build both. The first is a CRD (Custom Resource Definition): you teach the Kubernetes API server a brand-new object type — say Cache — so that kubectl get cache works exactly like kubectl get pod. An instance of that type (a custom resource, or CR) is where a user writes down what they want: “run a Redis-style cache on image redis:7 with 3 replicas.” That is the desired state. The second half is a controller: a loop that continuously compares desired state against the actual state of the cluster and does whatever it takes to close the gap. Deployment missing? Create it. Someone deleted it? Recreate it. Replica count changed? Scale it. This compare-and-correct loop is called reconciliation, and it is the entire heart of the pattern.

The mental model that makes everything else click: an operator is a thermostat, not a light switch. A light switch is edge-triggered — you flip it once and something happens once. A thermostat is level-triggered — it doesn’t care why the temperature is wrong or how many times it has been asked; it reads the current temperature, compares it to the target, and nudges. Call it a thousand times with nothing changed and nothing happens a thousand times. That property — doing the same safe thing no matter how often or why you are invoked — is idempotency, and it is the single most important idea in this lesson.

Kubebuilder is the tool that scaffolds all of this in Go so you never write the plumbing by hand. From a couple of commands you get three moving parts: the API (your CRD type, as a Go struct), the controller (an empty Reconcile function for you to fill in), and the manager (the runtime that wires everything together — it watches the API server, feeds changes to your controller, and provides leader election, metrics, and health checks). You supply the business logic; Kubebuilder and its library, controller-runtime, supply the machinery. The rest of this guide builds a real Cache operator and then hardens the parts that bite in production.

Read the loop clockwise. (1) A user applies a Cache custom resource that declares desired state (image, replicas); its CRD is what makes that kind legal in the first place. (2) The manager’s cache — backed by a shared informer — watches the CR and every object the operator owns, wired up with For(&Cache{}).Owns(&Deployment{}). (3) Any change enqueues the object’s key; a worker pulls it and runs Reconcile, which re-fetches current state and computes the diff between desired and actual — a no-op when they already match. (4) It applies only the difference to the managed resources (a Deployment and a Service, each stamped with an owner reference so garbage collection and watches work), using server-side apply so an unchanged object writes nothing. Finally it records observed reality on the status subresource and requeues — on error with exponential backoff, on a timer with RequeueAfter, or simply when the next watch event fires. The loop never ends; that is what “operator” means.

Prerequisites and what you will be able to do

Know this first: you should understand what a CRD, a controller, and the reconcile pattern are conceptually — if any of those is fuzzy, read Kubernetes CRDs, operators & the controller pattern first; this lesson is the hands-on build that follows it. You will want to be comfortable driving a cluster with kubectl, reading core objects (Deployment, Service), and reading basic Go — you do not need to be a Go expert, but you should recognise a struct and a method. You do not need a running cluster to follow the reasoning here: every command and manifest is real and current for Kubebuilder 4.x / controller-runtime, and any representative output is labelled as such. When you are ready to run it, a local kind cluster is enough.

After this lesson you will be able to:

• Explain the operator pattern as a level-triggered control loop, and say precisely why idempotency and re-fetching matter.
• Scaffold a project with Kubebuilder and design a CRD with a status subresource, validation markers, and printer columns.
• Write a Reconcile function that is idempotent, self-healing, and reports truth on status with conditions and observedGeneration.
• Use owner references for garbage collection, Owns() watches for self-healing, and server-side apply to avoid reconcile storms.
• Add a finalizer that guarantees external cleanup runs before the object is deleted, and keep that cleanup idempotent.
• Reason about the manager’s cache, informers, and rate-limited workqueue, and bound concurrency so a mass re-sync degrades gracefully.
• Decide when an operator is the right tool at all — versus a Helm chart — and place your operator on the OLM capability-level ladder.

An operator is not “a controller plus a CRD.” It is a promise: declare desired state in a custom resource, and the operator continuously drives the world toward it — surviving restarts, partial failures, and concurrent edits. This guide builds one for real with Kubebuilder, then hardens the parts that bite you in production: idempotency, finalizers, webhooks, and tests.

We’ll model a Cache resource that provisions a Redis-style Deployment plus Service, and (to make finalizers meaningful) registers itself with a hypothetical external metadata service on create and deregisters on delete.

1. Operator pattern fundamentals

The whole design rests on one idea: level-triggered reconciliation, not edge-triggered event handling.

Edge-triggered thinking (“on create, do X; on update, do Y”) is a trap. Events get coalesced, dropped on restart, and delivered out of order. Instead, your Reconcile function receives only a name and namespace, fetches current desired state, observes current actual state, and computes the diff. It must produce the same result whether it’s the first call or the thousandth.

            +-----------------------+
desired --> |  Reconcile(req)       | --> actual cluster state
(spec)      |  observe -> diff ->   |
            |  act -> update status |
            +-----------------------+
                    ^      |
                    +------+  re-queued on change, error, or interval

Three rules follow directly:

• Idempotent. Running twice with no spec change must be a no-op (modulo status).
• Self-correcting. If someone deletes the managed Deployment, the next reconcile recreates it.
• No assumptions about call cause. You never know why you were called. Always re-fetch.

2. Scaffold with Kubebuilder and design the CRD

Initialize the project and create the API. Pick a domain you own; the group/version/kind become your API surface.

mkdir cache-operator && cd cache-operator
go mod init github.com/acme/cache-operator

# Scaffold project layout, Makefile, manager main.go
kubebuilder init --domain acme.io --repo github.com/acme/cache-operator

# Create the API: group cache, version v1alpha1, kind Cache
kubebuilder create api --group cache --version v1alpha1 --kind Cache \
  --resource --controller

Now define the type. The key design decisions live in the marker comments above the struct, not in the fields themselves.

// api/v1alpha1/cache_types.go
package v1alpha1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type CacheSpec struct {
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=9
	// +kubebuilder:default=1
	Replicas int32 `json:"replicas"`

	// +kubebuilder:validation:Required
	Image string `json:"image"`

	// +optional
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
}

type CacheStatus struct {
	// ObservedGeneration is the .metadata.generation the status reflects.
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`

	ReadyReplicas int32 `json:"readyReplicas,omitempty"`

	// +listType=map
	// +listMapKey=type
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type Cache struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   CacheSpec   `json:"spec,omitempty"`
	Status CacheStatus `json:"status,omitempty"`
}

The +kubebuilder:subresource:status marker is load-bearing. It puts /status on its own endpoint, so a status write cannot accidentally clobber spec, and a spec edit does not bump nothing — it makes status.observedGeneration vs metadata.generation a reliable “have I caught up?” signal.

Regenerate the deepcopy code and CRD manifests after every type change:

make generate   # runs controller-gen object — DeepCopy methods
make manifests  # runs controller-gen crd,rbac,webhook -> config/crd, config/rbac

3. Implement an idempotent, level-triggered Reconcile

The reconcile body is a fixed skeleton. Memorize this shape; deviating from it is how bugs get in.

func (r *CacheReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := logf.FromContext(ctx)

	// 1. ALWAYS re-fetch. Never trust cached object state from the event.
	var cache cachev1alpha1.Cache
	if err := r.Get(ctx, req.NamespacedName, &cache); err != nil {
		// NotFound means it was deleted and finalizers (if any) already ran.
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 2. (Finalizer + deletion handling goes here — see section 5.)

	// 3. Reconcile owned objects toward desired state, idempotently.
	dep := r.desiredDeployment(&cache)
	if err := ctrl.SetControllerReference(&cache, dep, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}
	if err := r.applyDeployment(ctx, dep); err != nil {
		return ctrl.Result{}, err
	}

	// 4. Observe actual state and update status (on the status subresource).
	var live appsv1.Deployment
	if err := r.Get(ctx, client.ObjectKeyFromObject(dep), &live); err != nil {
		return ctrl.Result{}, err
	}

	cache.Status.ReadyReplicas = live.Status.ReadyReplicas
	cache.Status.ObservedGeneration = cache.Generation
	meta.SetStatusCondition(&cache.Status.Conditions, metav1.Condition{
		Type:    "Available",
		Status:  conditionStatus(live.Status.ReadyReplicas >= cache.Spec.Replicas),
		Reason:  "DeploymentReady",
		Message: fmt.Sprintf("%d/%d replicas ready", live.Status.ReadyReplicas, cache.Spec.Replicas),
	})
	if err := r.Status().Update(ctx, &cache); err != nil {
		return ctrl.Result{}, err
	}

	log.Info("reconciled", "ready", cache.Status.ReadyReplicas)
	return ctrl.Result{}, nil
}

A few non-obvious rules baked into that code:

• Return the error, let the queue back off. controller-runtime requeues failed reconciles with exponential backoff automatically. You almost never need RequeueAfter for error handling — return err and stop.
• client.IgnoreNotFound turns the post-deletion reconcile into a clean no-op.
• Status is a separate write via r.Status().Update. Never mutate spec and status in the same Update call.

4. Owner references, server-side apply, and avoiding reconcile storms

ctrl.SetControllerReference stamps the Deployment with an owner reference back to the Cache. This does two things: garbage collection deletes the Deployment automatically when the Cache is removed, and your watch on owned objects works.

Wire that up in SetupWithManager so a change to the managed Deployment triggers a reconcile of the owner:

func (r *CacheReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.Cache{}).
		Owns(&appsv1.Deployment{}).   // maps owned obj -> owner via ownerRef
		Owns(&corev1.Service{}).
		Complete(r)
}

For the actual write, server-side apply beats the classic get-then-update dance. SSA declares the fields you own; other controllers and humans can own other fields without a write-write conflict, and there is no read-modify-write race:

func (r *CacheReconciler) applyDeployment(ctx context.Context, dep *appsv1.Deployment) error {
	return r.Patch(ctx, dep, client.Apply,
		client.FieldOwner("cache-operator"),
		client.ForceOwnership)
}

The reconcile storm. The classic self-inflicted outage: your reconcile writes an object on every call, the write triggers a watch event, which triggers another reconcile, which writes again — a hot loop pinning a CPU. Server-side apply with a stable FieldOwner is inherently no-op when nothing changed (the resourceVersion doesn’t move, so no event fires). If you must use CreateOrUpdate, mutate only inside its callback and never set timestamps, random values, or re-ordered slices.

5. Finalizers for safe external cleanup

Owner references handle in-cluster GC. They do nothing for resources outside the cluster — the external registration in our example. For that you need a finalizer: a string in metadata.finalizers that blocks deletion until you remove it.

The deletion-handling block from section 3 expands to this. Order matters: add the finalizer before doing external work, run cleanup before removing the finalizer.

const finalizer = "cache.acme.io/finalizer"

// Inside Reconcile, after the Get:

if cache.DeletionTimestamp.IsZero() {
	// Not being deleted: ensure our finalizer is present.
	if !controllerutil.ContainsFinalizer(&cache, finalizer) {
		controllerutil.AddFinalizer(&cache, finalizer)
		if err := r.Update(ctx, &cache); err != nil {
			return ctrl.Result{}, err
		}
	}
} else {
	// Being deleted: run external cleanup, then drop the finalizer.
	if controllerutil.ContainsFinalizer(&cache, finalizer) {
		if err := r.deregisterExternal(ctx, &cache); err != nil {
			// Return the error — deletion stays blocked, we retry with backoff.
			return ctrl.Result{}, err
		}
		controllerutil.RemoveFinalizer(&cache, finalizer)
		if err := r.Update(ctx, &cache); err != nil {
			return ctrl.Result{}, err
		}
	}
	// Finalizer gone -> Kubernetes completes deletion. Stop here.
	return ctrl.Result{}, nil
}

Two failure modes worth internalizing:

• deregisterExternal must be idempotent. It will be retried. “Already deregistered” is success, not an error — otherwise the object is stuck in Terminating forever.
• Finalizer updates need RBAC on the finalizers subresource. Kubebuilder generates this when you mark the controller, but if you hand-edit RBAC, the update verb on caches/finalizers is mandatory.

6. Validating and defaulting with admission webhooks

CRD OpenAPI validation (the +kubebuilder:validation markers) handles structural rules — ranges, required fields, enums. For cross-field logic, immutability, or environment-aware defaults, you need webhooks.

Scaffold them:

kubebuilder create webhook --group cache --version v1alpha1 --kind Cache \
  --defaulting --programmatic-validation

Modern Kubebuilder generates the CustomDefaulter and CustomValidator interfaces (decoupled from the API type). The validator signatures return warnings plus an error:

// internal/webhook/v1alpha1/cache_webhook.go

func (v *CacheCustomValidator) ValidateUpdate(
	ctx context.Context, oldObj, newObj runtime.Object,
) (admission.Warnings, error) {
	oldC := oldObj.(*cachev1alpha1.Cache)
	newC := newObj.(*cachev1alpha1.Cache)

	// Enforce immutability that OpenAPI can't express.
	if oldC.Spec.Image != newC.Spec.Image {
		return nil, field.Forbidden(
			field.NewPath("spec", "image"),
			"image is immutable; delete and recreate the Cache")
	}
	return nil, nil
}

Webhooks require TLS and a CABundle wired into the webhook configuration. In real deployments let cert-manager issue and rotate the serving cert, and use Kubebuilder’s [WEBHOOK] and [CERTMANAGER] kustomize patches rather than managing certs by hand. Always set failurePolicy deliberately: Fail (default) blocks writes if the webhook is down — safe but can wedge your cluster; Ignore is more available but lets invalid objects through.

Conversion webhooks become relevant the moment you ship a v1beta1. Set one version as the storage version (+kubebuilder:storageversion) and implement Hub/Convertible so the API server can round-trip objects between versions. Get the conversion functions wrong and you silently corrupt stored data — treat them like a database migration.

7. Test with envtest, and emit events, conditions, and metrics

envtest runs a real kube-apiserver and etcd binary locally — no kubelet, no scheduler. You get genuine API-server validation, admission, and your CRD, which is exactly what controller logic depends on.

// suite_test.go (Ginkgo)
var _ = Describe("Cache controller", func() {
	It("creates an owned Deployment", func() {
		cache := &cachev1alpha1.Cache{
			ObjectMeta: metav1.ObjectMeta{Name: "c1", Namespace: "default"},
			Spec:       cachev1alpha1.CacheSpec{Image: "redis:7", Replicas: 3},
		}
		Expect(k8sClient.Create(ctx, cache)).To(Succeed())

		// Reconcile is async — poll with Eventually, never sleep.
		key := types.NamespacedName{Name: "c1", Namespace: "default"}
		Eventually(func() error {
			var dep appsv1.Deployment
			return k8sClient.Get(ctx, key, &dep)
		}, "10s", "200ms").Should(Succeed())
	})
})

# Downloads pinned apiserver/etcd binaries and runs the suite
make test

For observability, three signals matter and each has a built-in path:

controller-runtime already emits controller_runtime_reconcile_total, controller_runtime_reconcile_errors_total, and a reconcile-duration histogram. Alert on the error rate and a rising reconcile queue depth before writing any custom metric.

8. Package, generate RBAC, and distribute

RBAC is generated from markers, never written by hand. Annotate the reconciler with exactly the permissions it uses — least privilege is the default if you are honest in these comments:

// +kubebuilder:rbac:groups=cache.acme.io,resources=caches,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cache.acme.io,resources=caches/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=cache.acme.io,resources=caches/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch

make manifests regenerates config/rbac/role.yaml from these. Then build, push, and deploy:

make docker-build docker-push IMG=ghcr.io/acme/cache-operator:v0.1.0
make deploy IMG=ghcr.io/acme/cache-operator:v0.1.0   # applies CRDs + RBAC + manager

For distribution, pick based on audience:

• Helm chart — pragmatic for internal platform teams. Kubebuilder can scaffold one (kubebuilder edit --plugins=helm/v1-alpha); ship the CRD, RBAC, and Deployment together and let consumers helm upgrade.
• OLM bundle — the right choice if you publish to OperatorHub or target OpenShift. OLM manages install, upgrade graphs, and dependency resolution via a ClusterServiceVersion. More machinery, but it handles version-to-version upgrade edges a Helm chart leaves to you.

Ship CRDs out-of-band from the operator Deployment when you can. Helm’s handling of CRD upgrades is notoriously weak (it does not upgrade CRDs in the crds/ directory on helm upgrade), and a CRD is cluster-scoped shared state — treat its lifecycle more carefully than the workload.

Enterprise scenario

A payments platform ran a multi-tenant operator that provisioned a Cache per tenant. After a routine helm upgrade bumped the manager image, every reconcile across ~1,400 Cache objects fired at once on leader-election handover. The controller hammered the API server, the workqueue depth alert paged, and controller_runtime_reconcile_total spiked while apiserver_request_duration_seconds for PATCH deployments blew past 2s. Root cause was a non-idempotent write: the team had added a last-reconciled annotation set to time.Now() inside the deployment spec on every pass, so server-side apply saw a diff every call — a textbook reconcile storm, dormant until a mass re-sync exposed it.

Two fixes shipped. First, the timestamp moved out of the applied object and into status only. Second — the part most teams miss — they bounded concurrency and client throughput so a future re-sync degrades gracefully instead of self-DoSing:

// cmd/main.go — manager setup
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    Controller: config.Controller{MaxConcurrentReconciles: 4},
})
// and rate-limit the client itself
cfg.QPS, cfg.Burst = 30, 50

They also added a PrometheusRule alerting on rate(controller_runtime_reconcile_errors_total[5m]) and workqueue_depth, with the depth alert firing before saturation rather than after. The lesson: idempotency isn’t a code-review nicety — at fleet scale, one time.Now() in an applied field is a latent outage waiting for a leader change.

Going deeper

The core sections take you to an operator you can ship. This section is the advanced layer: the runtime machinery underneath mgr.GetClient() that decides how fast and how safely your loop runs, the three ways a reconcile gets re-invoked, the status semantics beginners skip, and the two decisions that separate a mature operator from a toy — how mature it is, and whether it should exist at all.

The manager, informers, caches, and the rate-limited workqueue

When you call mgr.GetClient(), reads come from an in-memory cache the manager populates from informers — long-lived watch connections on the API server that stream add/update/delete events and keep a local store in sync. This is why your reconcile reads are cheap: r.Get usually hits RAM, not the API server, so you can re-fetch on every call (as the level-triggered rule demands) without hammering the control plane. One caveat is worth burning in: the cache is eventually consistent. Immediately after you create an object you might not read it back on the very next line — write code that tolerates the miss (return and let the next reconcile see it), or reach for the manager’s APIReader when you truly need an uncached, straight-from-the-server read.

Informer events do not call Reconcile directly. They are mapped to a request — just a namespace/name, never the object itself, which is the whole point of level-triggered design — and pushed onto a rate-limited workqueue. Workers (one per MaxConcurrentReconciles, default 1) pull requests off that queue. The queue does three things that matter:

• De-duplicates. Ten rapid edits to one object collapse to a single queued key, so you reconcile once against the latest state, not ten times against stale snapshots.
• Rate-limits with per-item backoff. A key that keeps erroring is retried further and further apart (exponential), so a broken object cannot monopolise a worker.
• Spreads retries. Failures don’t all fire at the same instant, so a thundering herd doesn’t stampede the API server.

When you return err, the item goes back on the queue at the next backoff step — that is the machinery behind “return the error and let the queue back off.” Two levers tune throughput, and the enterprise scenario above shows why they matter at fleet scale: MaxConcurrentReconciles (how many objects reconcile in parallel) and the client’s QPS/Burst (how hard the controller may hit the API server). Turn concurrency up for a slow, IO-bound reconcile; cap QPS so a mass re-sync on leader handover degrades gracefully instead of self-DoSing.

RequeueAfter vs error requeue vs watch

There are exactly three ways a reconcile gets called again, and picking the wrong one is a common bug:

The distinctions in prose: return err for automatic exponential backoff — never log-and-swallow, because a swallowed error silently drops the retry and stalls convergence. RequeueAfter: d to poll slow external state — “check the cloud API again in 30s to see whether the load balancer finished provisioning” — because there is no watch on someone else’s system; never time.Sleep inside Reconcile to wait, as that blocks a worker for the whole cluster. A watch event is the normal, free path: the object or something it Owns changed, so the informer enqueues it without you asking. That is why you rarely need RequeueAfter for in-cluster state — the watch already tells you. (Note: the bare Requeue: true immediate re-queue is deprecated in current controller-runtime; return an error to mean “retry with backoff,” or RequeueAfter to mean “retry on a timer.”)

Status, conditions, and observedGeneration as a contract

A word on the semantics beginners skip, because they turn status from decoration into a machine-readable contract. metadata.generation is bumped by the API server every time spec changes — and only spec, because the status subresource keeps status writes from touching it. Your reconcile copies the generation it acted on into status.observedGeneration. A reader — a human, a kubectl wait, or another controller — compares the two: observedGeneration == generation means “the operator has seen and processed the current spec,” while a lag means “still converging.” Without that field you cannot distinguish “healthy and caught up” from “the operator hasn’t looked at my last edit yet.”

Conditions layer on top. The standard shape is a list keyed by type — commonly Available, Progressing, Degraded — where each entry carries a status (True/False/Unknown), a machine-readable reason, a human-readable message, and a lastTransitionTime. Use meta.SetStatusCondition (it handles the transition-time bookkeeping for you) and model each condition as an observation of reality, never as a command. The three-way Unknown matters: “I haven’t determined this yet” is genuinely different from False, and collapsing them hides the moment your operator lost visibility.

The OLM capability levels: how “mature” is your operator?

The Operator Framework defines five capability levels — the industry vocabulary for how much an operator actually does, and a useful definition of “done”:

1. Basic Install — provisions the app and its config from the CR. (Where this lesson’s Cache lands.)
2. Seamless Upgrades — upgrades the operand (and itself) without data loss.
3. Full Lifecycle — backups, restores, failover, scaling: the day-2 runbook, encoded.
4. Deep Insights — metrics, alerts, and conditions rich enough to run the operand blind.
5. Auto Pilot — auto-scaling, auto-tuning, and auto-remediation driven by those insights.

The Operator SDK is a superset of Kubebuilder — the same controller-runtime underneath — that adds the packaging and lifecycle tooling: it scaffolds an OLM (Operator Lifecycle Manager) bundle whose ClusterServiceVersion describes install, permissions, and, critically, the upgrade graph between versions. If you only distribute to your own platform team, a Helm chart is enough; if you publish to OperatorHub or target OpenShift, OLM is the path, because it resolves which version can upgrade to which — the edge a Helm chart leaves to you.

When NOT to write an operator: the Helm comparison

The honest advanced question is whether you should write one at all. A Helm chart is a templating tool: it renders manifests and applies them once, at install and upgrade time. It has no runtime — nothing watches your app after helm install returns. An operator is a running control loop that reconciles continuously. So:

• If your app is stateless and its lifecycle is install / upgrade / delete — a Deployment, a Service, some config — a Helm chart (or Kustomize) is the right tool. Writing an operator for it is over-engineering: you take on a Go codebase, a controller to run, and RBAC, all to replace a template.
• If your app needs continuous, stateful, domain-specific operations — a database that needs ordered failover, backups, point-in-time restore, re-sharding; a system where “someone deleted a Pod” must self-heal to a specific topology — that operational knowledge cannot live in a template. That is where an operator earns its cost.

Many teams use both: a Helm chart to install the operator, and the operator to run the workload. The test is simple — if a human would have to keep watching and adjusting the app after it’s installed, encode that human in an operator; if not, ship a chart. For the broader menu of extension mechanisms (aggregated API servers, conversion), see Kubernetes aggregated API server & CRD conversion; for the admission-webhook machinery your validating/defaulting hooks plug into, see Admission controllers: validating & mutating webhooks.

Practice challenges

Work these in order — they escalate from reading a CR to wiring self-healing and making the operator-vs-Helm call. Try each before opening the solution; every snippet is real Kubebuilder / controller-runtime.

Challenge 1 (beginner): name desired vs actual. Given this CR, and a cluster that currently has a Deployment sessions with 1 ready replica, state the desired state, the actual state, and what the next reconcile must do.

apiVersion: cache.acme.io/v1alpha1
kind: Cache
metadata:
  name: sessions
  namespace: default
spec:
  image: redis:7
  replicas: 3

<details> <summary>Solution</summary>

Desired state is the spec: a Cache backed by redis:7 at 3 replicas. Actual state is the live Deployment with 1 ready replica. The diff is “two replicas short,” so Reconcile applies the desired Deployment (which sets spec.replicas: 3) idempotently — via server-side apply, not a blind +2 — then reads back live.Status.ReadyReplicas and writes it to status.readyReplicas. It touches nothing that already matches, and nothing about why it was called enters the logic. That “read desired, observe actual, apply the difference” is the entire loop. </details>

Challenge 2 (beginner): sketch a reconcile. In five steps — no Go required — sketch the body of a Reconcile for a Website CRD that should own an nginx Deployment.

<details> <summary>Solution</summary>

1. Re-fetch the Website by req.NamespacedName; on error return client.IgnoreNotFound(err) so a post-deletion call is a clean no-op.
2. Build the desired Deployment from the spec and call ctrl.SetControllerReference(&website, dep, r.Scheme) so it’s owned.
3. Apply it with server-side apply and a stable FieldOwner — idempotent, so an unchanged object writes nothing.
4. Observe and report: read the live Deployment, copy readiness onto status, and persist with r.Status().Update.
5. Return ctrl.Result{}, nil — the Owns() watch will re-invoke you on the next change.

That five-step shape — fetch → build desired → apply → update status → return — is every reconcile you will ever write. </details>

Challenge 3 (intermediate): find the reconcile storm. This reconcile pins a CPU at 100%. Explain why, and give the one-line fix.

dep.Spec.Template.Annotations["last-sync"] = time.Now().String()
r.Patch(ctx, dep, client.Apply, client.FieldOwner("op"))

<details> <summary>Solution</summary>

The annotation changes on every call, so server-side apply always sees a diff and writes the Deployment; the write fires a watch event; the event enqueues another reconcile; which writes again — an infinite hot loop (a reconcile storm). The fix is to remove the timestamp from the applied object entirely — put it on status if you need it at all:

// delete the annotation line from the applied Deployment;
// if you must record it: cache.Status.LastSync = metav1.Now()

Rule: never write a value that changes every pass (timestamps, generated names, re-sorted lists) into a managed object. Diff before you write, and an unchanged object must produce zero writes. </details>

Challenge 4 (intermediate): add a finalizer. Add a finalizer named cache.acme.io/finalizer that deregisters an external metadata service before the Cache is deleted. Give both branches of the deletion logic and state the ordering rule.

<details> <summary>Solution</summary>

const finalizer = "cache.acme.io/finalizer"

if cache.DeletionTimestamp.IsZero() {
    // Not being deleted: ensure the finalizer is present BEFORE external work.
    if !controllerutil.ContainsFinalizer(&cache, finalizer) {
        controllerutil.AddFinalizer(&cache, finalizer)
        if err := r.Update(ctx, &cache); err != nil {
            return ctrl.Result{}, err
        }
    }
} else {
    // Being deleted: run cleanup BEFORE removing the finalizer.
    if controllerutil.ContainsFinalizer(&cache, finalizer) {
        if err := r.deregisterExternal(ctx, &cache); err != nil {
            return ctrl.Result{}, err // deletion stays blocked; retried with backoff
        }
        controllerutil.RemoveFinalizer(&cache, finalizer)
        if err := r.Update(ctx, &cache); err != nil {
            return ctrl.Result{}, err
        }
    }
    return ctrl.Result{}, nil
}

The ordering is the whole point: add the finalizer before the external resource exists, run cleanup before dropping the finalizer. And deregisterExternal must be idempotent — “already gone” is success — or the object wedges in Terminating forever. Finalizer updates also need the update verb on caches/finalizers in RBAC. </details>

Challenge 5 (advanced): restore self-healing and GC. A teammate’s operator creates its Deployment correctly, but when someone kubectl deletes that Deployment by hand it never comes back, and when the Cache is deleted the Deployment is orphaned. Name the two missing pieces and where each goes.

<details> <summary>Solution</summary>

Two independent bugs, both about ownership:

1. No Owns(&appsv1.Deployment{}) in SetupWithManager, so a change to the Deployment does not enqueue the owning Cache — hence no self-heal after a manual delete. Add it to the builder chain: For(&Cache{}).Owns(&appsv1.Deployment{}).Complete(r).
2. No ctrl.SetControllerReference(&cache, dep, r.Scheme) before the write, so the Deployment carries no owner reference — Kubernetes garbage collection has nothing to follow, hence the orphan when the Cache is deleted. Add it just before applying the Deployment.

They are the two halves of “the operator owns this object”: the owner reference gives you GC, the Owns() watch gives you self-healing. You need both. </details>

Challenge 6 (advanced): operator or Helm? For each, pick operator or Helm chart and give a one-line reason: (a) a stateless REST API with 3 replicas and an ingress; (b) PostgreSQL with automated failover, backups, and point-in-time restore; © installing that PostgreSQL operator itself.

<details> <summary>Solution</summary>

• (a) Helm. Install / upgrade / delete is the entire lifecycle; there are no continuous runtime operations, so a template is right and an operator would be over-engineering.
• (b) Operator. Ordered failover, backups, and PITR are continuous, stateful, domain-specific operations that cannot live in a static template — that is exactly the operational knowledge an operator encodes.
• © Helm. Installing the operator (its Deployment, RBAC, and CRDs) is itself a one-shot install/upgrade, so a chart is the standard way to ship it. The real-world combination is “a chart installs the operator, and the operator runs the workload.” </details>

Common beginner mistakes

These are misconceptions, not typos — each is a wrong mental model that produces an operator which looks fine and fails later.

• Thinking edge-triggered (“on create do X, on update do Y”). Why it’s wrong: events are coalesced, dropped on restart, and delivered out of order, so a handler keyed to “which event fired” acts on state it never received an event for. Right model: level-triggered — ignore why you were called, re-fetch current state, diff against desired, act. The Reconcile request is deliberately only a name so you cannot cheat.

• A non-idempotent reconcile. Why it’s wrong: writing a value that changes on every call (a timestamp, a random suffix, a re-sorted list) makes every apply a diff, every diff a write, and every write a fresh watch event — an infinite hot loop that pins a CPU and hammers the API server. Right model: reconcile must be a no-op when nothing changed; diff before you write and keep churny values on status.

• Editing status through the spec path (or spec through status). Why it’s wrong: they are separate subresources on purpose — mixing them clobbers one when you meant the other and breaks the observedGeneration signal readers depend on. Right model: spec is the user’s desired state (you read it); status is your report (you write it, only via r.Status().Update).

• Swallowing errors or sleeping to wait. Why it’s wrong: return nil after a failure silently drops the automatic retry, so convergence stalls with no signal; time.Sleep inside Reconcile blocks a shared worker for the whole controller. Right model: return err and let the workqueue back off, or RequeueAfter to poll — never sleep.

• Forgetting owner references. Why it’s wrong: without ctrl.SetControllerReference, the objects you create have no owner, so deleting the CR orphans them (leaked Deployments nobody cleans up) and Owns() watches never fire to self-heal. Right model: stamp every object you create with a controller owner reference — it is what makes both garbage collection and self-healing work.

• No finalizer for external cleanup. Why it’s wrong: owner references only garbage-collect in-cluster objects; an external registration, a cloud load balancer, or an S3 bucket is leaked on delete because nothing runs cleanup. Right model: a finalizer blocks deletion until your idempotent cleanup runs, then removes itself so Kubernetes can complete the delete.

Glossary

• Operator — software that encodes an ops team’s runbook for an application as a controller plus one or more CRDs, so a computer manages the app continuously.
• CRD (Custom Resource Definition) — an object that teaches the API server a new resource kind (e.g. Cache), making kubectl get cache work like any built-in type.
• Custom resource (CR) — an instance of a CRD’s kind; where a user declares desired state (spec) for the operator to act on.
• Controller — the code that runs the reconcile loop, comparing desired state to actual state and closing the gap.
• Reconciliation / reconcile loop — the compare-and-correct cycle at the heart of the pattern; your Reconcile function is one iteration.
• Desired state — what the user wants, written in the CR’s spec.
• Actual state — what the cluster (and any external systems) currently look like.
• Level-triggered — acting on current state, regardless of why or how often you were called (a thermostat). The correct model for controllers.
• Edge-triggered — reacting to events (“on create…”). Fragile for controllers because events drop, coalesce, and reorder.
• Idempotent — running with no change produces no change; running twice equals running once. The core reconcile property.
• Kubebuilder — the scaffolding tool that generates the API, controller, and manager for a Go operator.
• controller-runtime — the library underneath Kubebuilder that provides the manager, client, caches, and workqueue.
• Manager — the runtime that hosts controllers: it runs the caches/informers, wires watches, and provides leader election, metrics, and health checks.
• Informer — a long-lived watch on the API server that streams changes and keeps a local store in sync.
• Cache (controller-runtime) — the in-memory store the manager populates from informers; mgr.GetClient() reads from it, so reads are cheap but eventually consistent.
• Workqueue — the rate-limited, de-duplicating queue of reconcile requests that workers pull from; the source of automatic backoff.
• Reconcile request — the input to Reconcile: just a namespace/name, never the object itself (so you must re-fetch).
• Owner reference — a pointer on a child object back to its owner; drives garbage collection and Owns() watches.
• Garbage collection (GC) — Kubernetes deleting owned objects automatically when their owner is deleted.
• Owns() / watch — telling the manager to reconcile the owner whenever an owned object changes; the mechanism behind self-healing.
• Finalizer — a string in metadata.finalizers that blocks deletion until the operator runs cleanup and removes it; used for external cleanup.
• Status subresource — a separate /status endpoint so status writes can’t clobber spec; enabled with +kubebuilder:subresource:status.
• observedGeneration — the metadata.generation your status reflects; comparing it to the live generation tells readers “have I caught up?”
• Condition — a typed, machine-readable status entry (type, status True/False/Unknown, reason, message, lastTransitionTime); set via meta.SetStatusCondition.
• Server-side apply (SSA) — a patch where you declare only the fields you own (client.Apply + FieldOwner); no read-modify-write race, and a no-op when nothing changed.
• Reconcile storm — a self-inflicted hot loop where a non-idempotent write triggers a watch event that triggers another write, forever.
• RequeueAfter — ctrl.Result{RequeueAfter: d}: re-invoke this reconcile after a fixed delay, used to poll external state with no watch.
• envtest — a test harness that runs a real kube-apiserver and etcd locally (no kubelet/scheduler) so controller tests hit genuine API validation and admission.
• Admission webhook — an HTTPS callback the API server invokes to mutate (defaulting) or validate writes beyond what OpenAPI can express.
• CustomDefaulter / CustomValidator — the controller-runtime interfaces you implement for defaulting and validating webhooks, decoupled from the API type.
• Conversion webhook — converts a CR between API versions (e.g. v1alpha1 ↔ v1beta1) around a single storage version; treat like a data migration.
• OLM (Operator Lifecycle Manager) — the system that installs, upgrades, and resolves dependencies for operators via a ClusterServiceVersion.
• ClusterServiceVersion (CSV) — the OLM manifest describing an operator’s install, permissions, and version-to-version upgrade graph.
• Capability levels — the Operator Framework’s 1–5 maturity scale, from Basic Install to Auto Pilot.
• RBAC marker — a // +kubebuilder:rbac:... comment from which make manifests generates the operator’s least-privilege Role.
• Leader election — having only one operator replica act at a time (the manager default), so multiple replicas don’t fight over the same objects.

Verify

Confirm the operator works end to end against a real cluster (a kind cluster is fine):

# 1. CRD is registered
kubectl get crd caches.cache.acme.io

# 2. Create an instance and watch the operator converge
kubectl apply -f - <<'EOF'
apiVersion: cache.acme.io/v1alpha1
kind: Cache
metadata:
  name: demo
spec:
  image: redis:7
  replicas: 3
EOF

# 3. Owner-referenced Deployment appears and becomes ready
kubectl get deploy -l app=demo -o wide
kubectl get cache demo   # printer columns show Replicas / Ready

# 4. Status subresource reflects reality
kubectl get cache demo -o jsonpath='{.status.conditions[?(@.type=="Available")].status}{"\n"}'

# 5. Self-healing: delete the managed Deployment, it comes back
kubectl delete deploy demo
kubectl get deploy -l app=demo -w   # recreated by the next reconcile

# 6. Finalizer blocks deletion until external cleanup runs
kubectl delete cache demo            # blocks briefly in Terminating, then clears
kubectl get cache demo               # NotFound

# 7. Webhook rejects an immutable change
kubectl patch cache demo --type=merge -p '{"spec":{"image":"redis:6"}}'
# expect: admission webhook ... image is immutable

Check the operator’s own health:

kubectl logs -n cache-operator-system deploy/cache-operator-controller-manager -c manager
kubectl get --raw /metrics | grep controller_runtime_reconcile_errors_total

Production-readiness checklist

• CRD uses a status subresource and tracks observedGeneration
• Reconcile re-fetches the object every call and is idempotent
• Owned resources carry a controller owner reference for GC
• Writes use server-side apply (or mutate only inside CreateOrUpdate) to avoid reconcile storms
• Finalizer guards external cleanup; cleanup logic is idempotent
• Validating/defaulting webhooks cover what OpenAPI cannot; certs via cert-manager
• envtest suite covers create, update, self-heal, and deletion paths
• Conversion webhook in place before introducing a second API version
• Leader election enabled for the manager (default in scaffold — confirm it’s on)
• Resource limits, securityContext (non-root, read-only rootfs), and PodDisruptionBudget set on the manager
• Alerts on controller_runtime_reconcile_errors_total and workqueue depth

Pitfalls

The mistakes that cause real incidents, ranked by how often I see them:

1. Non-idempotent writes -> reconcile storms. Anything that changes on every call (timestamps, generated names, re-sorted lists) creates an infinite hot loop. Diff before you write.
2. Stuck Terminating. A finalizer whose cleanup function can permanently fail wedges deletion forever. Make cleanup idempotent and bounded; “not found externally” is success.
3. Mutating spec in the status update path (or vice versa). Keep the two writes strictly separate, and only ever write status via r.Status().
4. Webhook failurePolicy: Fail with no availability plan. If the webhook pod is down, all writes to that resource — including by the operator itself — are blocked. Run multiple replicas or scope the webhook narrowly.
5. Hand-edited RBAC drifting from code. Always regenerate with make manifests; a missing verb fails silently at runtime as a permission error mid-reconcile, not at deploy time.

Get the reconcile skeleton and idempotency right first — everything else is hardening on top of a loop that already converges correctly.