Containerization Lesson 75 of 113

Extending the Kubernetes API: Aggregated API Servers, CRD Conversion Webhooks, and Versioning Strategy

Jun 8, 2026 33 min read Expert Vinod

Most teams stop at CustomResourceDefinitions, and for good reason: a CRD plus a controller covers the overwhelming majority of operator use cases. But there is a second extension mechanism — the aggregation layer — that serves API groups from your own binary, with your own storage, validation, and admission. Knowing which to reach for, and how to evolve either one across versions without breaking stored objects, is the line between a platform team that ships an API once and one that maintains it for years.

This guide covers the hard parts: CRD multi-version lifecycle, writing a conversion webhook that round-trips between v1alpha1 and v1, structural schemas and the status subresource, and standing up a real aggregated API server behind the aggregation layer. Throughout, the failure mode to keep in mind is data corruption during an upgrade — etcd holds objects in exactly one encoding, and getting versioning wrong is silent until a get fails to decode.

In a nutshell

Level: Expert · Time: ~33 min

Kubernetes ships with a fixed menu of object kinds — Pods, Deployments, Services. Sooner or later you want your own kind: a Widget, a Tenant, a Fleet. There are exactly two ways to teach the cluster a new kind, and picking the right one is most of this lesson.

Way 1 — a CustomResourceDefinition (CRD). Think of renting a furnished room inside the API server’s own building. You hand the landlord (the kube-apiserver) a description of your furniture — a schema — and from then on it stores your objects in the shared basement (etcd), checks them against your schema, hands out keys, and answers the door. You write zero server code. This is the easy, declarative path, and it is the right answer the overwhelming majority of the time.

Way 2 — an aggregated (extension) API server. Think of building your own annex next door and running a private phone line into the main lobby. Visitors still walk in through the same front door (the kube-apiserver), but the receptionist forwards calls for your rooms to your annex, where you run your own filing system, your own storage, your own logic. Far more work — you build, deploy, and secure a real server — but you control everything, including not using etcd at all. This is how metrics-server serves live CPU/memory numbers that are computed on the fly and never stored.

Conversion webhooks are the third idea, and they apply only to the CRD path. Say version 1 of your object was written in one “language” (v1alpha1) and version 2 in another (v1). The building stores every object in exactly one language in the basement — the storage version. When an old client asks for its object in the old language, a translator at the front desk (the conversion webhook) converts it on the way out. The catch: that translator must translate perfectly in both directions, or information is lost the first time somebody upgrades — and because etcd holds each object in only one encoding, a bad translation corrupts data silently until a later read fails.

One sentence to carry through the lesson: a CRD lets the apiserver store and serve your kind for you; an aggregated API server is your own server behind the same front door; and conversion webhooks let a CRD’s kind evolve across versions without ever breaking what is already stored.

This is an expert lesson that assumes you have already met CRDs and controllers. If “reconcile loop” or “CustomResourceDefinition” is new, start with CRDs, Operators & the Controller Pattern; conversion webhooks are a close cousin of the admission webhooks covered in Admission Control. After this lesson you will be able to:

1. CRDs vs aggregated API servers: choosing the mechanism

Both mechanisms add new REST paths under /apis/<group>/<version>. The difference is who serves them.

A CRD is declarative. You kubectl apply a CustomResourceDefinition, and the kube-apiserver itself stores your objects in its own etcd, validates them against your OpenAPI v3 schema, and serves CRUD + watch. You write no API server code.

An aggregated API server (an “extension API server”) is a separate binary you build, deploy as a Deployment + Service, and register via an APIService object. The kube-apiserver proxies matching requests to it over TLS. You own storage, validation, admission, and the conversion of every request.

Concern CRD Aggregated API server
Code to write None (schema only) A full apiserver binary (Go)
Storage kube-apiserver’s etcd Your choice: etcd, a DB, or computed
Custom business logic on read/write Webhooks only Arbitrary, in-process
Non-etcd backing (proxy to external system) No Yes
Custom subresources beyond status/scale No Yes (e.g. /exec, /logs, arbitrary verbs)
Operational cost Trivial You run an HA, certificate-managed service
Protobuf serialization No (JSON only) Yes

Rule of thumb: reach for a CRD unless you have a concrete requirement a CRD cannot meet — a non-etcd backing store, computed/virtual resources, custom subresources or verbs, response sizes that demand protobuf, or imperative semantics. Metrics-server and apiregistration.k8s.io’s own metrics.k8s.io are the canonical aggregated APIs precisely because the data is computed, not stored.

The two mechanisms are not mutually exclusive. A common pattern is CRDs for declarative config plus a tiny aggregated API for a virtual subresource (think a custom /scale-like endpoint or a token-minting verb).

The whole decision, and the machinery behind each answer, fits in one picture:

Two ways to extend the Kubernetes API — a CRD backed by etcd with a conversion webhook, or an aggregated API server behind the aggregation layer

Read it left to right. Every request enters the one front door, the kube-apiserver, which routes by API group. Path A (a CRD) stays inside the apiserver: your objects are validated, stored in the apiserver’s own etcd, and served back — and when you run more than one version, a conversion webhook translates between them while etcd keeps exactly one storage version. Path B (the aggregation layer) leaves the apiserver: the built-in kube-aggregator proxies the request to your extension API server, which does its own storage (etcd, a database, or nothing at all — computed on the fly). The badges mark the five places this either forces a decision or breaks silently; we return to each below.

A one-sentence test for the choice: if your data can live in etcd as plain JSON objects and all you need is CRUD + watch + validation, a CRD is enough; the moment you need storage that is not etcd, a subresource or verb a CRD cannot express, protobuf on the wire, or genuinely imperative behaviour, you have crossed into aggregated-API-server territory.

2. CRD versioning: storage version, served versions, lifecycle

A CRD declares a list of versions. Each version has two independent booleans that people constantly conflate:

The distinction trips up almost everyone the first time, so make it concrete. served is about the front door: can a client read or write this version at its URL path right now? storage is about the basement: which single version’s schema is used to actually persist the bytes in etcd? The two are orthogonal, and there is always exactly one storage version, because a stored object has to be written in some single encoding.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.platform.acme.io
spec:
  group: platform.acme.io
  scope: Namespaced
  names:
    plural: widgets
    singular: widget
    kind: Widget
    listKind: WidgetList
  versions:
    - name: v1alpha1
      served: true
      storage: false          # still served for old clients, no longer stored
      schema:
        openAPIV3Schema: { ... }
    - name: v1
      served: true
      storage: true           # everything new is persisted as v1
      schema:
        openAPIV3Schema: { ... }
      subresources:
        status: {}

The four combinations of the two flags, and when you actually use each:

served storage What it means Typical use
true true Clients use it and objects persist in its schema The current version
true false Clients still read/write it, but objects persist as another version (conversion bridges them) An old version kept for compatibility, or a new version before the flip
false false Unreachable and not the storage encoding A retired version, kept only until status.storedVersions drains, then removed
false true Legal but unusual: the on-disk encoding is a version no client can reach Avoid in steady state — keep the storage version served

The mental model: a client may GET a v1alpha1/widgets/foo even though it is stored as v1. The apiserver decodes the stored v1 object and converts it to v1alpha1 to satisfy the request. With more than one served version whose schemas differ, that conversion is where a webhook becomes mandatory.

The upgrade lifecycle for adding v1 to an existing v1alpha1 CRD is strict:

  1. Add v1 as served: true, storage: false; keep v1alpha1 as the storage version. Ship the conversion webhook (Section 3) in the same change.
  2. Roll out clients that understand v1.
  3. Flip storage: true to v1 (and false on v1alpha1). New writes now persist as v1.
  4. Migrate stored objects so nothing remains physically encoded as v1alpha1 (Section 8).
  5. Only after migration completes: drop served: false on v1alpha1, then remove it from the versions list entirely.

Skipping step 4 is the classic foot-gun: you can never remove v1alpha1 from the CRD while a single object in etcd is still stored in it, because removing the version removes the schema needed to decode it.

3. Writing a conversion webhook (v1alpha1 <-> v1)

A conversion webhook is the translator from the In a nutshell analogy, made real: a small HTTPS server the apiserver calls whenever it must present a stored object in a different version than the one on disk. You need it only when two or more served versions have different schemas — if every served version is byte-identical (you renamed a version without changing its shape), the built-in None strategy is enough (see Going deeper).

When served versions diverge, set spec.conversion.strategy: Webhook. The apiserver POSTs a ConversionReview to your endpoint whenever it must translate between versions — on reads, on storage-version writes, and during migration.

spec:
  conversion:
    strategy: Webhook
    webhook:
      conversionReviewVersions: ["v1"]
      clientConfig:
        service:
          namespace: widget-system
          name: widget-conversion-webhook
          path: /convert
          port: 443
        caBundle: <base64 PEM of the serving CA>

The contract: the request carries a list of objects all in some desiredAPIVersion, and you return the same list converted into the requested target version, preserving metadata and crucially the annotation/label data you cannot lose. Conversion must be lossless and round-trippable — a classic approach is to stash fields with no home in the target version inside an annotation so the reverse conversion can restore them.

Here is the core handler in Go. Note the uid must be echoed and the response APIVersion must match the request.

func handleConvert(w http.ResponseWriter, r *http.Request) {
    body, _ := io.ReadAll(r.Body)
    review := &apix.ConversionReview{}
    if _, _, err := codecs.UniversalDeserializer().Decode(body, nil, review); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    req := review.Request
    resp := &apix.ConversionResponse{UID: req.UID}

    for _, raw := range req.Objects {
        cr := &unstructured.Unstructured{}
        if err := cr.UnmarshalJSON(raw.Raw); err != nil {
            resp.Result = metav1.Status{Status: metav1.StatusFailure, Message: err.Error()}
            break
        }
        if err := convert(cr, req.DesiredAPIVersion); err != nil {
            resp.Result = metav1.Status{Status: metav1.StatusFailure, Message: err.Error()}
            break
        }
        out, _ := cr.MarshalJSON()
        resp.ConvertedObjects = append(resp.ConvertedObjects, runtime.RawExtension{Raw: out})
    }
    if resp.Result.Status == "" {
        resp.Result = metav1.Status{Status: metav1.StatusSuccess}
    }
    review.Response = resp
    review.Request = nil
    writeJSON(w, review)
}

The convert function does the field surgery. Suppose v1alpha1 had a single spec.size string (“small”/“large”) and v1 replaced it with a structured spec.resources.replicas integer:

func convert(cr *unstructured.Unstructured, target string) error {
    switch target {
    case "platform.acme.io/v1":
        size, _, _ := unstructured.NestedString(cr.Object, "spec", "size")
        replicas := map[string]int64{"small": 1, "large": 5}[size]
        unstructured.RemoveNestedField(cr.Object, "spec", "size")
        _ = unstructured.SetNestedField(cr.Object, replicas, "spec", "resources", "replicas")
    case "platform.acme.io/v1alpha1":
        replicas, _, _ := unstructured.NestedInt64(cr.Object, "spec", "resources", "replicas")
        size := "small"
        if replicas >= 5 { size = "large" }
        unstructured.RemoveNestedField(cr.Object, "spec", "resources")
        _ = unstructured.SetNestedField(cr.Object, size, "spec", "size")
    }
    cr.SetAPIVersion(target)
    return nil
}

Two hard rules: never mutate anything outside spec/status and your own fields, and make the conversion total — every object the apiserver hands you must convert or the whole batch fails. Conversion runs on the hot path of every cross-version read, so keep it allocation-light and never call back into the apiserver.

It helps to know exactly when the apiserver invokes your webhook, because “it only runs on writes” is a common and costly misconception:

Trigger Direction Why
A client GETs/LISTs a version ≠ the storage version storage → requested Present stored bytes in the version asked for
A client writes (POST/PUT) a version ≠ the storage version requested → storage Persist in the single storage encoding
The storage-version migrator rewrites objects storage(old) → storage(new) Re-encode etcd during a storage-version flip
kubectl get --raw at an explicit version path storage → that version Same as a read at that version

Notice conversion fires on reads, not just writes — which is exactly why a webhook that is down or lossy breaks get, not just apply.

4. Structural schemas, validation, defaulting, status

Since apiextensions.k8s.io/v1, every CRD schema must be structural: each level specifies its type, no bare additionalProperties: true at the root, and value-validations (like oneOf) only inside properties/items. Structural schemas are the precondition for server-side defaulting, pruning, and the conversion machinery. In plain terms, structural means the schema is shaped like the objects it describes — every field has a declared type and lives somewhere the apiserver can find it — which is what lets the server safely default, prune, and convert without guessing. Non-structural schemas were allowed under the old apiextensions.k8s.io/v1beta1 (removed in 1.22) and are the reason so many pre-1.16 CRDs could not adopt these features.

openAPIV3Schema:
  type: object
  properties:
    spec:
      type: object
      required: ["resources"]
      properties:
        resources:
          type: object
          properties:
            replicas:
              type: integer
              minimum: 1
              maximum: 50
              default: 3              # server-side defaulting
        tier:
          type: string
          enum: ["bronze", "silver", "gold"]
          default: "bronze"
    status:
      type: object
      properties:
        readyReplicas: { type: integer }
      x-kubernetes-preserve-unknown-fields: false
  required: ["spec"]

Pruning is automatic with structural schemas: any field a client sends that is not in the schema is dropped, not stored. If you genuinely need to keep arbitrary keys (rare), opt in with x-kubernetes-preserve-unknown-fields: true on that node only.

Declaring the status subresource changes semantics: writes to /widgets/foo ignore the status stanza, and status is only mutated via /widgets/foo/status. This is what lets your controller update status without fighting the user’s spec edits, and it bumps metadata.generation only on spec changes — the signal your reconcile loop watches.

For validation beyond OpenAPI, prefer CEL validation rules (x-kubernetes-validations, GA since 1.29) over a validating webhook — they run in-process, need no certificates, and survive apiserver restarts:

replicas:
  type: integer
  x-kubernetes-validations:
    - rule: "self <= 10 || oldSelf > 10"
      message: "replicas above 10 can only be decreased, not increased"

That oldSelf transition rule enforces an invariant a static schema cannot — exactly the kind of policy that otherwise pushes people toward webhooks unnecessarily.

5. Standing up an extension API server (aggregation layer)

Everything so far kept you inside the kube-apiserver. An aggregated API server steps outside it. Physically, three things cooperate: (1) the aggregation layer (kube-aggregator), a component that lives inside the kube-apiserver and can proxy requests elsewhere; (2) an APIService object that tells the aggregator “requests for this group/version go to that Service”; and (3) your extension API server, an ordinary Deployment behind a Service, serving HTTPS. When a request arrives for a registered group, the kube-apiserver authenticates the user as usual, then opens a second, mutually-authenticated TLS connection to your server and forwards the request — adding headers that tell your server who the original user was.

When a CRD is not enough, you build an aggregated API server. The fastest correct path is to follow the structure of k8s.io/sample-apiserver, which wires k8s.io/apiserver’s GenericAPIServer to your scheme, or to scaffold with apiserver-builder (apiserver-boot), which generates that wiring plus storage and Makefiles.

# apiserver-builder approach
apiserver-boot init repo --domain acme.io
apiserver-boot create group version resource \
  --group platform --version v1 --kind Fleet
apiserver-boot build executables

The aggregation layer must be enabled on the cluster (it is by default on managed providers). The kube-apiserver needs these flags so it can authenticate to and trust your extension server:

--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
--requestheader-allowed-names=front-proxy-client
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
--enable-aggregator-routing=true

Those --requestheader-* and --proxy-client-* flags are the trust plumbing for that second hop. The kube-apiserver presents the proxy-client certificate to your server (so your server knows the caller is genuinely the apiserver), and it stamps the authenticated user’s identity into X-Remote-User/X-Remote-Group headers. Your server trusts those headers only because they arrive over a connection signed by the requestheader-client-ca. This is why your extension server reads the kube-system/extension-apiserver-authentication ConfigMap on startup: that is where the cluster publishes the requestheader CA it should trust. Get this wrong and you either reject the apiserver (every request 401s) or — far worse — trust forged headers from anyone who can reach your pod.

You then register the API group with an APIService. The kube-apiserver proxies /apis/platform.acme.io/v1 to your Service and verifies its serving cert against caBundle:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1.platform.acme.io
spec:
  group: platform.acme.io
  version: v1
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: fleet-apiserver
    namespace: fleet-system
    port: 443
  caBundle: <base64 PEM>            # omit and use cert-manager CA injection

Your extension server, on startup, must do delegated authn/authz: it does not re-implement auth. Using the genericapiserver recommended options, it calls TokenReview and SubjectAccessReview back against the kube-apiserver, and reads the kube-system/extension-apiserver-authentication ConfigMap to learn the request-header CA. This is what makes RBAC on your aggregated resources behave identically to built-in resources.

// In your server's options wiring (sample-apiserver style)
o.RecommendedOptions.Authentication.RemoteKubeConfigFileOptional = true
o.RecommendedOptions.Authorization.RemoteKubeConfigFileOptional = true
serverConfig := genericapiserver.NewRecommendedConfig(codecs)
if err := o.RecommendedOptions.ApplyTo(serverConfig); err != nil {
    return err   // applies delegated authn, authz, audit, openapi, etcd
}

6. Custom storage, RBAC, and admission for aggregated resources

The aggregation layer’s real power is custom storage. k8s.io/apiserver/pkg/registry/rest defines small interfaces — Getter, Lister, Creater, Updater, GracefulDeleter, Watcher — and whichever you implement determines which verbs your resource supports. Back them with the generic etcd Store for normal cases, or implement them by hand to project an external system as Kubernetes objects (the metrics-server pattern: Get/List only, computed on the fly, no storage).

// Minimal read-only REST storage backed by a live computation
type fleetREST struct{ source ExternalInventory }

func (r *fleetREST) New() runtime.Object          { return &v1.Fleet{} }
func (r *fleetREST) NewList() runtime.Object       { return &v1.FleetList{} }
func (r *fleetREST) NamespaceScoped() bool         { return true }
func (r *fleetREST) Get(ctx context.Context, name string, _ *metav1.GetOptions) (runtime.Object, error) {
    return r.source.Lookup(genericapirequest.NamespaceValue(ctx), name)
}
func (r *fleetREST) Destroy() {}

RBAC is automatic and unified: because your server delegates authorization via SubjectAccessReview, a normal Role/ClusterRole granting verbs on your group works without any special handling.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fleet-reader
rules:
  - apiGroups: ["platform.acme.io"]
    resources: ["fleets"]
    verbs: ["get", "list", "watch"]

Admission inside an aggregated server is in-process: register admission plugins on the GenericAPIServer chain rather than deploying external admission webhooks. You get ValidatingAdmission and MutatingAdmission interfaces and can short-circuit with full type information — no AdmissionReview round trip, no certificate to rotate.

7. Discovery, OpenAPI publishing, and client codegen

For your API to be first-class — kubectl explain fleet, kubectl get fleet, typed clients — three things must publish correctly.

Discovery. The kube-apiserver aggregates your /apis/platform.acme.io/v1 discovery document under the cluster’s discovery. Verify it surfaced:

kubectl get apiservice v1.platform.acme.io
# NAME                     SERVICE                      AVAILABLE
# v1.platform.acme.io      fleet-system/fleet-apiserver  True

kubectl api-resources --api-group=platform.acme.io

AVAILABLE: False with a FailedDiscoveryCheck message almost always means a TLS or networking problem reaching your Service, not a code bug.

OpenAPI. Serve OpenAPI v2 and v3 from the extension server (RecommendedConfig wires the endpoints; you supply generated definitions). This is what powers kubectl explain and server-side apply field management. For CRDs you get OpenAPI v3 for free from the structural schema.

Client codegen. Generate typed clients, listers, and informers with code-generator’s kube_codegen.sh (the modern entry point replacing the per-tool *-gen invocations). Tag your types so the generators know what to produce:

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// +kubebuilder:object:root=true
type Fleet struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec   FleetSpec   `json:"spec,omitempty"`
    Status FleetStatus `json:"status,omitempty"`
}

# vendor code-generator, then:
./hack/update-codegen.sh   # wraps kube_codegen.sh: deepcopy, client, lister, informer

For CRDs specifically, controller-gen does the equivalent: controller-gen object crd paths=./... emits deepcopy code and the CRD manifest with embedded schema, so the same Go types feed both your controller and the published API.

8. Rollout, deprecation, and migrating stored objects

The single most dangerous moment is the storage-version flip. After you set v1 as storage, existing objects are still physically encoded as v1alpha1 in etcd until something rewrites them. They are readable (conversion handles it) but they block ever deleting v1alpha1.

Force a rewrite by reading and writing every object back through the new storage version:

# Crude but effective: touch every object so it re-persists as the storage version
kubectl get widgets -A -o name | while read w; do
  kubectl annotate "$w" platform.acme.io/migrated="$(date -u +%FT%TZ)" --overwrite
done

For production scale and correctness, use the official storage-version-migrator (kube-storage-version-migrator), which drives the migration declaratively and tracks progress, instead of hand-rolling annotations:

apiVersion: migration.k8s.io/v1alpha1
kind: StorageVersionMigration
metadata:
  name: widgets-to-v1
spec:
  resource:
    group: platform.acme.io
    resource: widgets
    version: v1

Then track which versions clients still read using the CRD status, and mark old versions deprecated so kubectl warns users before you remove them:

versions:
  - name: v1alpha1
    served: true
    storage: false
    deprecated: true
    deprecationWarning: "platform.acme.io/v1alpha1 Widget is deprecated; use v1. Removal in operator 2.0."

Only when (a) migration reports complete and (b) status.storedVersions on the CRD no longer lists v1alpha1 should you set it served: false, and in a later release remove it entirely. The apiserver actively prevents you from dropping a version that still appears in storedVersions.

Going deeper

Everything above is enough to design and ship an extension. This section is for the reader who operates the aggregation layer and versioned CRDs at scale, where the internals decide whether an upgrade is a non-event or an outage.

How the aggregation layer actually routes a request

When a request for /apis/platform.acme.io/v1/... reaches the kube-apiserver, the kube-aggregator looks up a matching APIService. If it is a local group (built-ins and all CRDs are served by the apiserver itself), it is handled in-process. If it points at a Service, the aggregator proxies the request. Two facts make that proxy hop different from an ordinary client call:

There is a production nuance most people learn the hard way: an unavailable aggregated APIService degrades the whole cluster’s discovery. The apiserver must fetch each aggregated group’s discovery document to answer kubectl api-resources, kubectl get <anything>, and client-go’s discovery cache. If your extension server is down and its APIService is AVAILABLE: False, every such call waits for a timeout — the classic “metrics-server is broken and now kubectl is slow for everyone” symptom. Register with a sensible groupPriorityMinimum, run the server HA, and delete or disable the APIService if you decommission the server, so a dead endpoint does not tax discovery.

When you genuinely need an aggregated API server

The honest list of triggers, each with the real-world example that justifies the operational cost:

Against those benefits, weigh the cost you now own: an HA deployment, serving-cert rotation, tracking the k8s.io/apiserver library version across Kubernetes upgrades, and the discovery blast radius above. This is why the hybrid — a CRD for declarative config plus a tiny aggregated API for the one virtual verb you truly need — is so common: you pay the apiserver cost only for the sliver that requires it.

Conversion strategies: None vs Webhook, and the round-trip law

spec.conversion.strategy has two values, and choosing wrong is a data-integrity decision:

The round-trip law governs any webhook: conversion must be lossless and reversible. Formally, for any object o and versions a, b: convert(convert(o, a→b), b→a) == o. Because etcd stores exactly one encoding while clients may read or write any served version, a lossy conversion silently drops data on the first write-then-read at a different version. When the target version has no home for a field, the standard trick is to stash it under an annotation (e.g. platform.acme.io/v1alpha1-size: "large") so the reverse conversion restores the exact original rather than a reconstructed guess.

Two migrator options exist as of the current releases. The established, out-of-tree kube-storage-version-migrator (migration.k8s.io/v1alpha1, shown in Section 8) drives the rewrite and tracks progress. Kubernetes 1.30 also introduced an in-tree storagemigration.k8s.io/v1alpha1 StorageVersionMigration that folds this capability into the apiserver itself; check your cluster version’s docs for its current maturity before depending on it, and until then the out-of-tree tool remains the safe production choice. Either way, the rewrite is what drains status.storedVersions so you can finally remove the old version.

Structural schemas and the x-kubernetes-* vocabulary

Structural schemas unlock a set of Kubernetes-specific OpenAPI extensions worth knowing by name, because each changes server behaviour:

Defaulting has a subtlety operators must internalise: defaults apply on read as well as write, so adding a default to the storage version changes what already-stored objects appear to contain the next time they are fetched. That is usually what you want, but it means a default is not a purely additive, invisible change — it is observable retroactively.

API Priority and Fairness (APF)

Extension APIs make it easy to overload the control plane, so APF (flowcontrol.apiserver.k8s.io/v1, GA and on by default) matters here. APF classifies every inbound request through a FlowSchema into a PriorityLevelConfiguration, isolating concurrency so one noisy client cannot starve the apiserver. This bites extension authors two ways: the requests your controllers make are subject to APF, and a slow aggregated apiserver holds apiserver resources while it proxies, so bounding that matters.

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: platform-controllers
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 40
    limitResponse:
      type: Queue
      queuing:
        queues: 64
        queueLengthLimit: 50
        handSize: 6
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: platform-controllers
spec:
  priorityLevelConfiguration:
    name: platform-controllers
  matchingPrecedence: 900
  rules:
    - subjects:
        - kind: ServiceAccount
          serviceAccount:
            name: platform-operator
            namespace: platform-system
      resourceRules:
        - apiGroups: ["platform.acme.io"]
          resources: ["*"]
          verbs: ["*"]

Give a bulk operation — like a storage-version migration sweep over thousands of objects — its own low-share priority level so it yields, and give latency-sensitive controllers a dedicated level with guaranteed shares. Watch the apiserver_flowcontrol_* metrics and the X-Kubernetes-PF-FlowSchema-* response headers to confirm requests land where you intend.

Versioning discipline: never break the storage version

The cardinal rules that keep etcd decodable across years of API evolution:

Enterprise scenario

A fintech platform team ran a multi-tenant Tenant CRD across 40 clusters, originally shipped as v1alpha1 with a flat spec.quota string like "cpu=8,mem=32Gi". Two years of adoption meant ~6,000 stored Tenant objects and dozens of teams’ GitOps repos pinned to v1alpha1. They needed a real, validated spec.quota object for v1, but a hard cutover was impossible: flipping the storage version while old v1alpha1 objects sat in etcd, with no conversion path, would have made those objects undecodable the moment they touched v1.

The constraint that bit them: they initially shipped v1 as storage version without a conversion webhook, assuming “the schemas are close enough.” The first reconcile that read an old object failed with a decode error, because the apiserver had no way to turn the stored flat string into the structured v1 shape. Reads of unmigrated tenants started 500-ing in two clusters before they rolled back.

The fix was to follow the lifecycle in order. They shipped a conversion webhook first (with v1alpha1 still storage), proving round-trip correctness in CI by converting a corpus of real objects v1alpha1 -> v1 -> v1alpha1 and asserting equality. Then they flipped storage to v1, ran kube-storage-version-migrator cluster by cluster during change windows, and watched status.storedVersions drain to ["v1"] before touching served. The webhook’s reverse path kept every pinned GitOps repo working untouched throughout. The CI round-trip gate is the part they wish they had built first:

func TestConversionRoundTrips(t *testing.T) {
    for _, obj := range loadCorpus(t, "testdata/tenants_v1alpha1") {
        orig := obj.DeepCopy()
        if err := convert(obj, "platform.acme.io/v1"); err != nil { t.Fatal(err) }
        if err := convert(obj, "platform.acme.io/v1alpha1"); err != nil { t.Fatal(err) }
        if diff := cmp.Diff(orig.Object, obj.Object); diff != "" {
            t.Errorf("lossy conversion:\n%s", diff)
        }
    }
}

That single test would have caught the lossy assumption before any cluster did.

Verify

Run these to confirm an extension is healthy end to end.

# 1. CRD multi-version: confirm served vs storage and stored encodings
kubectl get crd widgets.platform.acme.io \
  -o jsonpath='{range .spec.versions[*]}{.name}{" served="}{.served}{" storage="}{.storage}{"\n"}{end}'
kubectl get crd widgets.platform.acme.io -o jsonpath='{.status.storedVersions}'

# 2. Conversion webhook actually fires: read the same object at both versions
kubectl get widget demo --output=yaml --raw \
  /apis/platform.acme.io/v1alpha1/namespaces/default/widgets/demo | grep -E 'apiVersion|size|replicas'
kubectl get widget demo --output=yaml --raw \
  /apis/platform.acme.io/v1/namespaces/default/widgets/demo | grep -E 'apiVersion|size|replicas'

# 3. Aggregated API server reachable and registered
kubectl get apiservice v1.platform.acme.io \
  -o jsonpath='{.status.conditions[?(@.type=="Available")].status} {.status.conditions[?(@.type=="Available")].message}{"\n"}'

# 4. Discovery, explain, and RBAC all resolve
kubectl api-resources --api-group=platform.acme.io
kubectl explain fleet.spec
kubectl auth can-i list fleets.platform.acme.io --as=system:serviceaccount:default:reader

# 5. Defaulting and pruning behave (apply minimal object, read back)
kubectl apply -f - <<'EOF'
apiVersion: platform.acme.io/v1
kind: Widget
metadata: { name: defaults-check }
spec: { resources: {} }
EOF
kubectl get widget defaults-check -o jsonpath='{.spec.resources.replicas} {.spec.tier}{"\n"}'  # expect: 3 bronze

A correct setup shows distinct fields per version in step 2 (proving conversion ran), Available: True in step 3, populated discovery in step 4, and defaulted values in step 5.

Common beginner mistakes

These are misconceptions, not symptoms — the wrong mental model that produces a whole class of bugs. Each pairs the belief with the correction.

Practice challenges

Work these top to bottom; they escalate from “pick the mechanism” to “protect the control plane”. Each solution is one click away — try first, then check. Where a live cluster is handy you can run the commands for real; where none is, the answers are still schema-correct.

Challenge 1 (Beginner) — CRD or aggregated? For each requirement, choose the mechanism and say why: (a) a Backup object teams create/list/watch, stored normally; (b) a NodeMetrics-style API returning live CPU/memory computed on demand; © a Certificate kind with a custom /approve action that is not create/update; (d) a high-throughput API returning 50k-item lists to many controllers that need protobuf.

<details> <summary>Solution</summary>

(a) CRD — declarative, etcd-backed CRUD + watch is precisely what a CRD is for. (b) Aggregated — computed, non-etcd data is the metrics-server pattern. © Aggregated — a custom verb beyond status/scale is impossible with a CRD. (d) Aggregated — protobuf and large lists are CRD non-starters (JSON only). Why: only (a) fits “objects live in etcd as JSON with standard verbs”; the moment any of storage, verbs, or wire format leaves that envelope, you need an aggregated server. </details>

Challenge 2 (Beginner) — Read the flags. Given v1alpha1 (served: true, storage: false) and v1 (served: true, storage: true): where is a newly created object stored? Can an old client still GET at v1alpha1? What must exist for that GET to work if the schemas differ?

<details> <summary>Solution</summary>

Stored as v1 (the storage version). Yes, v1alpha1 is still served, so old clients keep working. For the GET to succeed when schemas differ, a conversion webhook (strategy: Webhook) must translate the stored v1 object down to v1alpha1 on read. Why: served and storage are independent — conversion is what bridges the gap between them. </details>

Challenge 3 (Intermediate) — Spot the unsafe flip. A PR sets v1 storage: true, v1alpha1 storage: false, removes v1alpha1 from versions entirely, and ships no webhook. There are 6,000 v1alpha1 objects in etcd. List everything wrong and give the correct order of operations.

<details> <summary>Solution</summary>

Three faults: (1) schemas differ but there is no conversion webhook, so reads of old objects fail to decode; (2) removing v1alpha1 deletes the only decoder while objects are still stored in it; (3) the storage flip before migration leaves 6,000 objects encoded as v1alpha1 that now cannot be read as v1. Correct order: ship the webhook (with v1alpha1 still storage) → prove round-trip in CI → flip storage to v1 → run the storage-version-migrator until status.storedVersions == ["v1"] → set v1alpha1 served: false → in a later release remove it. Why: both the storage version and the versions list are contracts with etcd’s on-disk bytes. </details>

Challenge 4 (Intermediate) — Make the conversion lossless. v1alpha1 has spec.size (“small”/“large”); v1 has spec.resources.replicas (int). A team maps small→1, large→5, and reverses replicas>=5→large else small. Why is this lossy, and how do you fix it losslessly?

<details> <summary>Solution</summary>

It is lossy because the reverse mapping is not injective: replicas: 3 (a perfectly valid v1 value) reverses to "small", which forward-maps back to 1, so v1 → v1alpha1 → v1 silently turns 3 into 1. Fix: when converting v1 → v1alpha1, stash the exact original in an annotation (e.g. platform.acme.io/v1-replicas: "3"); when converting v1alpha1 → v1, if that annotation is present use it, otherwise fall back to the size mapping. Now the round-trip is exact. Why: a non-injective mapping must carry the lost information in a side channel or it cannot be reversed. </details>

Challenge 5 (Advanced) — Diagnose AVAILABLE: False. kubectl get apiservice v1.platform.acme.io shows AVAILABLE False with FailedDiscoveryCheck. Your Deployment is healthy and pod logs show it serving on :443. Give an ordered troubleshooting checklist.

<details> <summary>Solution</summary>

Cheapest first: (1) Endpoints — does the Service have Ready endpoints? kubectl get endpoints fleet-apiserver -n fleet-system. (2) caBundle — does it match the CA that signed the serving cert? A mismatched or expired CA is the #1 cause; prefer cert-manager CA injection. (3) Serving-cert SAN — must cover fleet-apiserver.fleet-system.svc. (4) NetworkPolicy — is apiserver → pod:443 allowed? (5) Delegated auth — can the server read kube-system/extension-apiserver-authentication (RBAC), and is RemoteKubeConfigFileOptional set? (6) Only then suspect the code. Why: FailedDiscoveryCheck is the apiserver failing to fetch /apis/... over TLS — a transport problem far more often than a logic one. Bonus: while it is False, cluster-wide discovery is degraded and kubectl is slow for everyone. </details>

Challenge 6 (Advanced) — Protect reconciliation with APF. During a storage-version migration sweep, normal controllers start getting 429s and reconcile lag spikes. Explain the cause and fix it with API Priority and Fairness.

<details> <summary>Solution</summary>

The migration rewrites thousands of objects, flooding a shared priority level; APF throttles everyone in that level, including latency-sensitive controllers. Fix: give the migration sweep its own low-share PriorityLevelConfiguration (via a FlowSchema matching the migrator’s identity) so it yields under contention, and/or give critical controllers a dedicated higher-share level with guaranteed concurrency. Confirm with the apiserver_flowcontrol_* metrics and the X-Kubernetes-PF-FlowSchema-* response headers. Why: APF isolates concurrency by flow, so a bulk client cannot starve steady-state reconciliation once they are in separate levels. </details>

Checklist

Glossary

kubernetesapi-extensioncrdoperatorswebhooks
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me
VH

Written by Vinod

Senior Cloud Architect with 22+ years of experience. Sharing practical knowledge from production environments.

Comments