Provisioning Production Kubernetes: kubeadm, HA Control Plane, etcd Backup & Upgrades

In a nutshell

Every Kubernetes cluster has a brain — the control plane — that records every decision (every Deployment, Secret, and Pod placement) into a single ledger called etcd. The worker nodes are the hands that run your containers; the brain tells them what to do. So far in this course you have been a user of that brain, handed to you ready-made by a cloud. This lesson is about building and running the brain yourself.

The tool for the job is kubeadm — think of it as the official setup crew that assembles one control plane correctly: it mints the security certificates, initialises the etcd ledger, and wires the components together, then hands you the keys with a single kubeadm init. Adding a worker later is one kubeadm join command.

But one brain is one power cut away from disaster. High availability (HA) means running three brains that keep an identical copy of the ledger and vote on every change, sitting behind a load balancer — a receptionist that routes callers to whichever brain is awake. As long as a majority (two of the three) agree, the cluster keeps working even when one node dies. That “majority of an odd number” rule is called quorum, and it is why you run 3 or 5 control-plane nodes, never 2 or 4. Lose the majority — or lose etcd with no backup — and the cluster is gone. That is why backing up etcd is the single most important skill in this entire lesson.

Level: Advanced — but written so a beginner can follow every step, with a “Going deeper” section and graded “Practice challenges” for experienced operators · Time: ~40 min · You’ll need: a terminal and, for the lab, Docker plus kind, kubectl, and etcdctl — everything runs free and locally.

Up to now in this course you have used clusters — deploying workloads, making them production-ready, troubleshooting them. This lesson is about building and running the cluster itself. When you let a cloud provider hand you a managed control plane (EKS, AKS, GKE), an enormous amount of careful engineering is hidden from you: how the API server is made redundant, where the cluster’s data actually lives, how the TLS certificates that hold the whole thing together are issued and rotated, and how you upgrade a live cluster without dropping a single request. The day you self-manage — on-prem, at the edge, in an air-gapped environment, or simply to understand what the managed service is doing for you — all of that becomes your job.

We will use kubeadm, the official, opinionated cluster-bootstrapping tool. kubeadm is not a full installer (it deliberately leaves CNI, OS prep, and infrastructure to you), but it is the reference for how a conformant cluster is assembled, and it is exactly what the CKA exam expects you to drive. By the end you will understand a highly available (HA) control plane built from three or more control-plane nodes behind a load balancer; the crucial choice between stacked and external etcd; how to back up and restore etcd (the single most important operational skill for a self-managed cluster); how to safely take nodes in and out of service; and how to perform a staged, version-skew-aware upgrade of a running cluster. You will do the hands-on parts free and locally so the commands are real, not abstract.

Learning objectives

By the end of this lesson you can:

Bootstrap a control plane with kubeadm init and join workers and additional control-plane nodes with kubeadm join, and explain every artefact kubeadm produces (certificates, kubeconfigs, static Pod manifests).
Design a highly available control plane and choose correctly between a stacked etcd and an external etcd topology, justifying the trade-off.
Explain etcd’s quorum requirement and why control-plane node counts are odd (3, 5, 7), and reason about blast radius.
Back up etcd with etcdctl snapshot save and restore a cluster from that snapshot — the skill that turns a disaster into an inconvenience.
Manage the cluster’s PKI: list certificate expiry, rotate certificates, and rotate the kubelet’s serving and client certs.
Run the node lifecycle safely — cordon, drain, uncordon, and taints/tolerations — for maintenance and decommissioning.
Apply the version-skew policy and perform a staged upgrade (control plane first, then kubelets), and decide when managed Kubernetes is the better call.

Prerequisites & where this fits

You should be comfortable with the control-plane components and the reconciliation loop from Kubernetes Architecture Deep-Dive, and with the workload concepts from Pods, Deployments & Services and the Day-2 readiness checklist. You need a terminal, and for the hands-on lab a free local cluster tool — kind, minikube, or k3d — plus kubectl and etcdctl. This is the Architecture tier of the Kubernetes Zero-to-Hero course: it is where you stop being a consumer of clusters and start being the person who can build, operate, recover, and upgrade one. It is also the densest single block of CKA exam content, so the commands here are worth committing to muscle memory.

A note on honesty before we start: you will almost never hand-build a production cluster the way this lesson does, command by command — you will use a higher-level installer (Cluster API, kubespray, Rancher, or a managed service). But you will be operating, recovering, and upgrading such clusters, and to do that well you must understand the anatomy underneath. That anatomy is the whole point of this lesson.

Core concepts: what a cluster actually is

Strip away the abstractions and a Kubernetes cluster is two things: a control plane that holds the desired state and makes decisions, and a set of worker nodes that run your Pods. The control plane is itself a small set of programs:

Component	Role	How kubeadm runs it
kube-apiserver	The front door — the only thing that talks to etcd; everything else talks to it	Static Pod on each control-plane node
etcd	The cluster’s database — all state lives here as key/value data	Static Pod (stacked) or separate machines (external)
kube-scheduler	Assigns Pods to nodes	Static Pod on each control-plane node (leader-elected)
kube-controller-manager	Runs the reconciliation controllers	Static Pod on each control-plane node (leader-elected)
kubelet	Node agent — talks to the container runtime via CRI; runs the static Pods above	systemd service on every node
kube-proxy	Programs Service networking (iptables/IPVS)	DaemonSet

Two ideas do most of the heavy lifting in this lesson:

Static Pods. The control-plane components are run as static Pods — Pods defined by manifest files in /etc/kubernetes/manifests/ that the kubelet watches and runs directly, with no API server or scheduler involved. This is the bootstrap chicken-and-egg solution: the kubelet can run the API server before the API server exists, because static Pods do not need the API server. Edit a file in that directory and the kubelet reacts within seconds — which is also how you will tweak etcd and apiserver flags later.
etcd is the cluster. Every object you have ever created — Deployments, Secrets, RBAC, the lot — is a key in etcd. The API server is stateless; lose every control-plane node but keep a good etcd snapshot and you can rebuild. Lose etcd with no backup and the cluster is gone. This is why etcd backup is the single most important operational task on a self-managed cluster, and why we spend a whole section on it.

Jargon check. Quorum is the minimum number of etcd members that must agree (a majority) for the cluster’s data store to accept writes. PKI (public-key infrastructure) is the set of certificate authorities and certificates that let cluster components prove who they are to each other over TLS. Version skew is the set of rules about how far apart component versions are allowed to drift during an upgrade.

kubeadm: what `init` and `join` actually do

kubeadm does the fiddly, security-critical bootstrap and nothing more. It is worth knowing exactly what each command produces, because half of CKA cluster questions are really “do you understand the artefacts kubeadm created?”

kubeadm init on the first control-plane node performs these phases (you can run them individually with kubeadm init phase ...):

Preflight checks — kernel modules, swap off, ports free, container runtime reachable, correct cgroup driver (systemd is the modern default).
Generate the PKI — a self-signed cluster CA at /etc/kubernetes/pki/, plus the etcd CA, the front-proxy CA, and the service-account signing keypair. Every component cert is signed by one of these.
Write kubeconfigs — admin.conf (your cluster-admin credential — this is what you copy to ~/.kube/config), plus controller-manager.conf, scheduler.conf, and the kubelet’s bootstrap config.
Write static Pod manifests to /etc/kubernetes/manifests/ for the apiserver, controller-manager, scheduler, and (in the stacked topology) etcd. The kubelet starts them.
Bootstrap tokens & add-ons — install the cluster DNS (CoreDNS) and kube-proxy, and print a kubeadm join command containing a token and the CA cert hash.

A minimal init for an HA cluster names the load balancer up front so the certificates and kubeconfigs point at the stable address, not at one node:

# On the FIRST control-plane node
sudo kubeadm init \
  --control-plane-endpoint "k8s-api.internal:6443" \  # the LB DNS:port — REQUIRED for HA
  --upload-certs \                                     # stash certs in a Secret so other CP nodes can pull them
  --pod-network-cidr "10.244.0.0/16" \                 # must match your CNI
  --kubernetes-version v1.30.2

--control-plane-endpoint is the make-or-break flag for HA: it bakes the load balancer’s address into every cert and kubeconfig. Omit it and you can never add a second control-plane node without regenerating certificates. --upload-certs puts the control-plane certificates into a short-lived (2-hour) Secret so additional control-plane nodes can fetch them with a --certificate-key instead of you copying files by hand.

kubeadm join comes in two flavours, and the difference is one flag:

# Join a WORKER node
sudo kubeadm join k8s-api.internal:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

# Join an additional CONTROL-PLANE node (note --control-plane + --certificate-key)
sudo kubeadm join k8s-api.internal:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <key-from-upload-certs>

The --discovery-token-ca-cert-hash is what stops a worker from being tricked into joining an imposter API server — it pins the cluster CA. Tokens expire after 24 hours by default; regenerate one any time with kubeadm token create --print-join-command.

Designing an HA control plane

A single control-plane node is a single point of failure: lose it and you cannot schedule, scale, or change anything (existing Pods keep running, but the cluster is “frozen”). HA means three or more control-plane nodes so the cluster keeps making decisions through the loss of one. Two things must be made redundant: the API server (stateless, easy — just run several behind a load balancer) and etcd (stateful, the hard part — covered next).

The non-negotiable pieces of an HA control plane:

Piece	Requirement	Why
Control-plane nodes	Odd number ≥ 3 (3 is standard; 5 for large/critical)	etcd needs a majority; odd counts maximise fault tolerance per node
Load balancer	TCP (layer 4) LB in front of all apiserver `:6443`	Gives one stable `--control-plane-endpoint`; health-checks out dead apiservers
etcd	3 or 5 members with quorum	The data store must survive a node loss
Spread	Across racks/AZs/failure domains	A correlated failure must not take a majority at once

Why odd numbers and why three? etcd uses the Raft consensus protocol and must have a majority (quorum) of members available to accept writes. The fault tolerance is (n-1)/2 rounded down:

etcd members	Quorum (majority)	Failures tolerated
1	1	0
2	2	0 (worse than 1 — you doubled risk for no gain)
3	2	1
4	3	1 (no better than 3, more overhead)
5	3	2
7	4	3

The killer insight in that table: two members tolerate zero failures (you need both, so either dying loses quorum), and four are no more resilient than three. Even numbers buy you nothing but cost and write latency, so you always run an odd number, and three is the sweet spot for most clusters. Beyond seven, write latency (every write must reach a majority) outweighs the extra resilience, so etcd recommends a maximum of seven members.

The load balancer should be layer 4 (TCP), not layer 7, because the API server speaks TLS end-to-end and you do not want the LB terminating it. Its health check should probe /healthz (or /livez) on :6443 so a wedged apiserver is taken out of rotation. On-prem this is commonly HAProxy + keepalived (a virtual IP that floats to a healthy node); in cloud it is a network load balancer.

Stacked vs external etcd: the architecture decision

This is the architecture choice the brief cares about most, and a classic interview question. kubeadm supports two topologies for where etcd runs.

Stacked etcd (the kubeadm default): each control-plane node also runs an etcd member, co-located as a static Pod on the same machine. Three control-plane nodes give you a three-member etcd cluster “for free.”

External etcd: etcd runs on its own dedicated set of machines, separate from the control-plane nodes. The API servers point at this external etcd cluster.

Stacked (3 nodes):                     External (3 + 3 nodes):
┌─────────────┐                        ┌─────────────┐   ┌──────────┐
│ CP1: api,   │                        │ CP1: api,   │   │ etcd-1   │
│   sched,    │                        │   sched,    │──▶│          │
│   ctrl,etcd │                        │   ctrl      │   │ etcd-2   │
├─────────────┤   etcd cluster         ├─────────────┤──▶│          │
│ CP2: ...etcd│   (co-located)         │ CP2: ...    │   │ etcd-3   │
├─────────────┤                        ├─────────────┤──▶│ (own     │
│ CP3: ...etcd│                        │ CP3: ...    │   │  machines)│
└─────────────┘                        └─────────────┘   └──────────┘

The comparison that matters:

Dimension	Stacked etcd	External etcd
Machines needed	3 (CP = etcd)	6+ (3 CP + 3 etcd)
Cost & ops complexity	Lower — fewer machines, kubeadm manages it	Higher — separate cluster to provision, secure, back up, upgrade
Blast radius	Losing a node loses an apiserver and an etcd member together	etcd and control-plane failures are decoupled
etcd I/O isolation	etcd competes with apiserver for CPU/disk on the same box	etcd gets dedicated, fast disks (it is latency-sensitive)
Scaling etcd independently	No — tied to control-plane node count	Yes
kubeadm default	Yes	No (you provision etcd first, then point kubeadm at it)
Best for	Most clusters; simplicity wins	Large/critical clusters; strict isolation; very high object counts

The right default is stacked — it is simpler, cheaper, and what most clusters run. Reach for external when you need to isolate etcd’s disk I/O (etcd is acutely sensitive to disk latency — slow disks cause leader elections and cluster-wide slowdowns), when your blast-radius requirements forbid coupling a control-plane failure with an etcd-member failure, or when a very large cluster pushes etcd hard enough to warrant its own tuned hardware. The trade-off is real operational weight: a second cluster to secure (its own TLS), back up, and upgrade.

For external etcd, you set up the etcd cluster yourself (or with kubeadm in etcd-only mode), then feed kubeadm a config file pointing at it:

# kubeadm-config.yaml (external etcd)
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.30.2
controlPlaneEndpoint: "k8s-api.internal:6443"
etcd:
  external:
    endpoints:
      - https://10.0.1.10:2379
      - https://10.0.1.11:2379
      - https://10.0.1.12:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

etcd backup and restore: the skill that saves your job

If you remember one thing from this lesson, make it this. An etcd snapshot is the difference between a five-minute recovery and a destroyed cluster. Managed Kubernetes does this for you invisibly; on a self-managed cluster it is your responsibility, and “we never took a backup” is how clusters die permanently.

A snapshot is a point-in-time copy of the entire key-value store — every object in the cluster. Take it with etcdctl, pointing at a live etcd member with that member’s client certificates:

# Back up etcd (run on a control-plane node; v3 API is required)
sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot is intact and see how many keys it holds
sudo ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-snapshot.db --write-out=table

In production you run that on a schedule (a CronJob or a systemd timer), copy the snapshot off the cluster to object storage, and test restoring it periodically — an untested backup is a rumour, not a backup.

Restore is the high-stakes operation. You restore the snapshot into a new data directory and point etcd at it. The exact safe sequence on a single-member (or one-at-a-time) restore:

# 1. STOP the control plane so nothing writes to etcd while you restore.
#    Move the static Pod manifests aside; the kubelet stops the Pods.
sudo mv /etc/kubernetes/manifests/*.yaml /tmp/

# 2. Restore the snapshot into a fresh data directory.
sudo ETCDCTL_API=3 etcdctl snapshot restore /var/backups/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restored

# 3. Point etcd's static Pod manifest at the new data dir
#    (edit the hostPath volume for etcd-data to /var/lib/etcd-restored),
#    then move the manifests back so the kubelet restarts the control plane.
sudo vi /tmp/etcd.yaml         # change the data dir volume path
sudo mv /tmp/*.yaml /etc/kubernetes/manifests/

# 4. Verify
kubectl get nodes
kubectl get pods -A

The non-obvious rules that trip people up in exams and incidents:

Restore is not “load into the running cluster.” It rebuilds a data directory. You must stop etcd, restore to a new dir, repoint, restart.
Restoring rolls the whole cluster back to the snapshot’s moment. Anything created after the snapshot is gone. This is fine for disaster recovery but is not a per-object undo.
In a multi-member HA etcd, a full restore is a cluster-rebuild operation: you restore on each member with matching --initial-cluster/--initial-advertise-peer-urls so they form one new cluster, rather than three lone members. The simplest reliable pattern is restore-to-one then let the others re-sync, but know that the official procedure restores every member from the same snapshot.

Certificate management and rotation

The cluster is held together by TLS. kubeadm generates a cluster CA and issues short-lived (one-year) certificates to the API server, controller-manager, scheduler, etcd, and the kubelets. Certificates that silently expire are one of the most common causes of a “the cluster died overnight” incident — the API server stops trusting its own components and everything grinds to a halt.

Check expiry at any time:

sudo kubeadm certs check-expiration

This prints every certificate, its expiry date, and whether the CA that signed it is externally managed. The good news: kubeadm auto-renews all control-plane certificates on every kubeadm upgrade. A cluster you upgrade at least yearly effectively never sees an expired cert. The trap is the cluster that runs untouched for over a year — those expire and you must renew manually:

# Renew all kubeadm-managed certs (then restart the control-plane static Pods)
sudo kubeadm certs renew all

# Or one at a time, e.g. just the apiserver
sudo kubeadm certs renew apiserver

After renewing you must restart the static Pods (move the manifests out and back, or reboot the kubelet) so they pick up the new certs, and refresh your admin.conf if that was renewed.

The kubelet has its own certificates and is the one place automatic rotation is the norm:

Client cert rotation (rotateCertificates: true, on by default): the kubelet automatically renews its client certificate before expiry via the CSR API. This is why worker nodes do not silently fall off after a year even without a cluster upgrade.
Serving cert rotation (serverTLSBootstrap: true): the kubelet requests a serving cert, but those CSRs require approval — by a controller (cloud providers run one) or manually with kubectl certificate approve. A common gotcha is metrics-server or kubectl logs failing because kubelet serving CSRs are sitting unapproved.

The one certificate kubeadm cannot auto-rotate is the cluster root CA itself (default ten-year life) — rotating a CA is a deliberate, disruptive operation because every component must re-trust the new CA. Plan it; do not let it surprise you a decade in.

Node lifecycle: cordon, drain, taints

Operating a cluster means routinely taking nodes out of service — to patch the OS, upgrade the kubelet, replace hardware, or decommission. Do it wrong and you cause an outage; do it right and users never notice. The vocabulary:

Operation	What it does	When
`kubectl cordon <node>`	Marks the node unschedulable — no new Pods land, existing Pods stay	First step before maintenance; “stop sending me work”
`kubectl drain <node>`	Cordons and evicts existing Pods (respecting PodDisruptionBudgets) so they reschedule elsewhere	Before reboot/upgrade/decommission
`kubectl uncordon <node>`	Marks the node schedulable again	After maintenance, to return it to the pool
Taint	A property on a node that repels Pods unless they carry a matching toleration	Reserve nodes (GPU, control plane); auto-applied on problems

The safe maintenance sequence is always cordon → drain → do the work → uncordon:

# Take node out of service safely (skip DaemonSets, tolerate emptyDir if you must)
kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data

# ... patch / upgrade / reboot node-3 ...

kubectl uncordon node-3      # put it back in the pool

drain honours PodDisruptionBudgets — if evicting a Pod would breach a PDB, the drain blocks and waits, which is exactly the protection you want (it is why the Day-2 lesson insisted on PDBs). --ignore-daemonsets is almost always required because DaemonSet Pods are managed per-node and are not drained.

Taints and tolerations are the complementary mechanism: a taint repels, a toleration grants permission to ignore that repulsion. Control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule so ordinary workloads stay off them. Taints have three effects: NoSchedule (don’t place new Pods), PreferNoSchedule (avoid if possible), and NoExecute (also evict running Pods that don’t tolerate it — this is how Kubernetes evacuates a NotReady node automatically).

kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule     # add
kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule-    # remove (trailing minus)

To decommission a node for good: drain it, then kubectl delete node <name> to remove it from the cluster’s records, then on the node itself kubeadm reset to clean up its kubeadm state.

The version-skew policy and a staged upgrade

Upgrading a live cluster is the operation that most rewards understanding and most punishes guesswork. Two rules govern it: the version-skew policy (how far components may differ) and the upgrade order (control plane before nodes, one minor version at a time).

The version-skew policy for the components you operate:

Relationship	Allowed skew (Kubernetes ≥ 1.28)	Plain English
kubelet vs kube-apiserver	kubelet may be up to 3 minor versions older, never newer	Nodes may lag the control plane, but never lead it
kube-controller-manager / scheduler vs apiserver	Same minor, or 1 older	Control-plane peers stay close to the apiserver
kubectl vs apiserver	Within 1 minor (newer or older)	Your client should be near the server version
Across the upgrade	One minor version at a time (1.29 → 1.30, not 1.29 → 1.31)	No skipping minors

The two load-bearing rules: the control plane is always upgraded first (the apiserver must never be older than the things talking to it), and you never skip a minor version — to go from 1.28 to 1.30 you upgrade to 1.29 first, then 1.30. Patch versions (1.30.1 → 1.30.2) are freely applied.

A safe staged upgrade with kubeadm, control plane first, then workers one node at a time:

# ── FIRST control-plane node ──
# 1. Upgrade the kubeadm binary to the target version (via your package manager), then:
sudo kubeadm upgrade plan                 # shows current vs available, checks skew
sudo kubeadm upgrade apply v1.30.2        # upgrades the control plane components + renews certs

# 2. Drain THIS node, then upgrade the kubelet + kubectl binaries, then:
kubectl drain cp-1 --ignore-daemonsets
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon cp-1

# ── OTHER control-plane nodes ──
sudo kubeadm upgrade node                 # NOT "apply" — "node" on the rest
# then drain → upgrade kubelet → restart → uncordon, as above

# ── WORKER nodes, one at a time ──
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
#   (on the node) upgrade kubeadm + kubelet + kubectl packages
sudo kubeadm upgrade node
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon worker-1

The pattern to internalise: kubeadm upgrade apply runs exactly once, on the first control-plane node (it upgrades the cluster’s control-plane components and renews certificates). Every other node — control-plane or worker — uses kubeadm upgrade node. And every node is drained before and uncordoned after its kubelet is restarted, so the rolling upgrade never takes capacity you need. Because the version-skew policy lets kubelets lag the apiserver by up to three minors, draining workers one at a time through a single-minor bump is always safe.

Before any upgrade: take an etcd snapshot. It is your rollback if kubeadm upgrade apply goes wrong.

Managed vs self-managed: choosing well

Everything above is your job only if you self-manage. The honest architect’s question is whether you should.

Concern	Self-managed (kubeadm/CAPI/kubespray)	Managed (EKS / AKS / GKE)
Control-plane HA & etcd	You build and operate it	Provider runs it, multi-AZ, with an SLA
etcd backup/restore	Your CronJob, your restore drill	Automatic, provider-managed
Upgrades	You stage them (this lesson)	One click / one API call; you still upgrade nodes
Cert rotation	Your responsibility	Handled by the provider
Cost	No control-plane fee (but your machines + your time)	A control-plane fee, but far less ops labour
Control & portability	Total — any infra, any version, air-gapped	Constrained to the provider’s offering
Best for	On-prem, edge, air-gapped, deep customisation, learning	Almost every cloud workload — let the provider carry the toil

The default recommendation for a cloud workload is managed: the control plane and etcd are precisely the parts that are hard to run well and have nothing to do with your application’s value. Self-manage when you have a reason the managed offering cannot meet — on-prem or edge with no managed option, an air-gapped environment, a regulatory or customisation requirement, or simply to learn the machinery so you can operate the managed service intelligently. Even then, prefer a higher-level tool (Cluster API, kubespray, Rancher) over raw kubeadm for fleets — but everything those tools do, they do on top of the kubeadm concepts you have just learned.

Provisioning an HA Kubernetes cluster

The diagram traces the whole picture: a load balancer fronting three control-plane nodes, the stacked-versus-external etcd split, the certificate authority issuing component certs, the etcd snapshot flowing out to backup storage, and the staged upgrade order arrowing from control plane down to workers.

Going deeper

The sections above give you a working cluster and the operational reflexes. This section is for the reader who wants to know what is happening under those commands — the internals that separate someone who can follow the runbook from someone who can debug it at 3 a.m.

kubeadm is a toolkit of phases, not a black box

Everything kubeadm init does is an individually addressable phase, and you can run any one of them alone. kubeadm init phase --help lists the lot: kubeadm init phase certs all regenerates only the PKI; kubeadm init phase control-plane apiserver rewrites just the apiserver static Pod manifest; kubeadm init phase kubeconfig all rebuilds the kubeconfigs. This is how you repair a half-broken control plane without a destructive kubeadm reset, and it is exactly how higher-level installers (kubespray, Cluster API’s bootstrap provider) drive kubeadm surgically rather than running the monolithic init. kubeadm join phase mirrors it for nodes, and kubeadm certs is the phase family for the PKI. The mental shift: kubeadm init is just a convenient ordering of phases you can also invoke à la carte.

Raft, quorum, and why you add etcd members one at a time

etcd’s quorum rule is not arbitrary — it falls out of the Raft consensus protocol. Only the elected leader accepts writes, and a write is committed only once a majority of members has persisted it to their replicated log. That is why write latency is bounded by your slowest majority member, and why a geographically stretched etcd is a trap: every write must round-trip to a quorum, so a member across an ocean drags the whole cluster down. It is also the deeper reason an even count buys nothing — six members still need four to commit, so you have added cost and latency without improving the failures you tolerate.

Modern etcd softens the riskiest moment — adding a member — with learner members. A new member joins as a non-voting learner, replays the leader’s log until it has caught up, and is only promoted to a voting member once it is in sync. Without learners, adding a fourth member to a three-member cluster momentarily required that new (empty, slow) member to participate in quorum, which could stall writes. The practical rule survives regardless: grow etcd one member at a time and let each fully sync before adding the next.

The control-plane load balancer: three real patterns

--control-plane-endpoint needs a stable address, but what sits behind it is an architecture choice:

Cloud network load balancer — a managed layer-4 LB (AWS NLB, Azure LB, GCP TCP LB) health-checking :6443. The default in cloud.
Off-cluster HAProxy + keepalived — on bare metal, a pair of HAProxy hosts with keepalived floating a virtual IP between them via VRRP, so the VIP survives one LB host dying.
On-cluster VIP (no external appliance) — kube-vip, or a static-Pod HAProxy/keepalived, running the virtual IP on the control-plane nodes themselves — elegant when you have no external LB to call. (This is the same family of tooling covered in MetalLB & kube-vip for bare-metal load balancing.)

Whichever you choose, it must be layer 4 (TCP) and health-check /healthz (or /readyz) on :6443. A layer-7 LB that terminates TLS breaks the client-certificate (mutual TLS) authentication the API server relies on — the LB would strip the client’s identity.

The certificate tree and external-CA mode

kubeadm’s PKI is a small tree, and kubeadm certs check-expiration walks all of it:

a root ca signs the apiserver serving cert, the apiserver→kubelet client cert, and the admin/component client certs;
a separate etcd/ca signs etcd’s server and peer certs and the apiserver-etcd-client cert the API server uses to reach etcd;
a front-proxy CA signs the aggregation-layer certs; and
a standalone service-account keypair (a signing key, not a cert) signs ServiceAccount tokens.

In regulated environments you run external-CA mode: you place the CA certificate in /etc/kubernetes/pki but withhold its private key. kubeadm detects the missing key, emits certificate signing requests (kubeadm certs generate-csr), your external CA signs them out-of-band, and kubeadm never touches the signing key. The trade-off is that renewals become your job — kubeadm certs renew cannot sign without the key.

What `kubeadm upgrade` actually changes

kubeadm upgrade apply does considerably more than bump the static-Pod image tags. In one run it: upgrades the cluster add-ons (CoreDNS and kube-proxy) to versions matching the new control plane; rewrites the kubeadm-config ConfigMap in kube-system (the cluster’s own stored configuration, which every later kubeadm upgrade node reads); renews every control-plane certificate; and reconciles feature gates. kubeadm upgrade plan reads that ConfigMap plus the live component versions to compute the valid next hops and flag anything that would violate the skew policy. On every other node, kubeadm upgrade node pulls that stored config and updates only that node’s kubelet configuration (and, on a control-plane node, its local static Pods) — which is precisely why apply runs once and node runs everywhere else.

etcd housekeeping: compaction, defragmentation, quota, and alarms

This is the operational depth managed Kubernetes hides completely. etcd is multi-version: it keeps a revision history of every key, so a busy cluster’s store grows without bound unless it is pruned.

Compaction discards old key revisions. kubeadm configures etcd’s auto-compaction, so this usually happens for you — but compaction frees only logical space.
Defragmentation (etcdctl defrag) is separate and manual: compaction leaves the on-disk file the same size (with holes inside it), and only a defrag rewrites the file compactly to return space to the filesystem. A defrag briefly blocks the member it runs on, so do it off-peak, one member at a time.
The space quota (--quota-backend-bytes, default 2 GiB, often raised to 8 GiB) is a hard ceiling. Cross it and etcd raises a NOSPACE alarm and goes read-only — the whole cluster freezes for writes. Recovery is: compact to the current revision, etcdctl defrag, then etcdctl alarm disarm to clear the latch. Simply freeing disk is not enough; the alarm stays raised until you disarm it.

etcd control	What it does	Default / typical
`--quota-backend-bytes`	Hard cap on DB size; exceeding it raises `NOSPACE` and stops writes	2 GiB default; 8 GiB common for large clusters
Auto-compaction	Discards old key revisions to bound history	On (kubeadm configures it)
`etcdctl defrag`	Rewrites the DB file to reclaim space that compaction freed	Manual, off-peak, one member at a time
`etcdctl alarm list` / `disarm`	Shows / clears a latched `NOSPACE` or `CORRUPT` alarm	Manual, after remediation

A production etcd therefore wants a deliberately sized quota, automatic compaction, an occasional off-peak defrag, and monitoring on DB size and alarm state. For the full backup-and-recovery drill that pairs with this, see Set up etcd snapshot backup & disaster restore.

CNI and CSI: the parts kubeadm deliberately leaves to you

Immediately after kubeadm init, your nodes sit NotReady and CoreDNS stays Pending — and that is correct. kubeadm ships no pod network. You must install a CNI (Container Network Interface) plugin — Calico, Cilium, Flannel — before the cluster is usable, and the --pod-network-cidr you passed at init must match the CNI’s configuration. This omission is deliberate: pod networking is a genuine architecture decision (network policy, eBPF versus iptables, encryption, IPAM), so kubeadm refuses to choose for you. (The mechanics are in The Kubernetes CNI & pod-networking model.) Storage is the identical story via CSI (Container Storage Interface): kubeadm installs no storage driver, so dynamic PersistentVolumes need a CSI driver (a cloud-disk driver, Ceph/Rook, or local-path) plus a matching StorageClass. Both live outside kubeadm because Kubernetes is intentionally pluggable at the network and storage boundaries.

Join tokens and the TLS bootstrap handshake

A kubeadm join is a carefully choreographed trust exchange that lets a brand-new node — which holds no cluster credentials yet — safely obtain its own. Two secrets do the work: the bootstrap token (short-lived, mapped to the system:bootstrappers group) proves the node was invited, and the --discovery-token-ca-cert-hash pins the cluster CA so the node cannot be lured into trusting an imposter API server. Using that bootstrap credential, the kubelet submits a certificate signing request; an approver auto-approves CSRs from the bootstrappers group, and the kubelet receives its own client certificate — this is TLS bootstrapping. From that point the kubelet rotates its client cert itself (which is why nodes do not silently fall off after a year). When the 24-hour token expires, kubeadm token create --print-join-command mints a fresh token and prints the exact join command with the current CA hash.

Hands-on lab

You will not hand-build a multi-node HA cluster in this lab — that needs several machines or VMs. Instead you will create a real multi-node cluster locally with kind (which runs each node as a container), then practise the operational skills that transfer directly to a kubeadm cluster: inspecting the control plane, taking an etcd snapshot, checking certificate expiry, and running the node lifecycle (cordon/drain/uncordon). Everything here is free and runs on your laptop.

Prerequisites: Docker (or Podman), kind, kubectl, and etcdctl installed. Install etcdctl from the etcd release if you do not have it.

Step 1 — Create a multi-node cluster (1 control plane + 2 workers). kind uses kubeadm under the hood, so the artefacts are the real thing.

cat <<'EOF' > kind-ha.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF

kind create cluster --name provlab --config kind-ha.yaml
kubectl get nodes -o wide

Expected: three nodes — one control-plane and two worker, all Ready within a minute or two.

Step 2 — See the control plane as static Pods. These are exactly what kubeadm wrote to /etc/kubernetes/manifests/.

kubectl get pods -n kube-system -o wide | grep -E 'etcd|apiserver|scheduler|controller'

Expected: etcd-provlab-control-plane, kube-apiserver-..., kube-scheduler-..., kube-controller-manager-... all Running. Note how etcd is stacked on the control-plane node.

Step 3 — Take an etcd snapshot from inside the control-plane container. This is the production backup command, run against the real etcd.

docker exec provlab-control-plane sh -c '\
  ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key'

# Verify the snapshot and count the keys it holds
docker exec provlab-control-plane sh -c '\
  ETCDCTL_API=3 etcdctl snapshot status /tmp/etcd-backup.db --write-out=table'

Expected: Snapshot saved at /tmp/etcd-backup.db, then a table showing a non-zero TOTAL KEYS — proof you captured live cluster state. In production you would copy this file off the node to object storage.

Step 4 — Check certificate expiry. Run kubeadm’s own check inside the node.

docker exec provlab-control-plane kubeadm certs check-expiration

Expected: a table of certificates (apiserver, etcd, front-proxy, etc.) with expiry roughly one year out and CERTIFICATE AUTHORITY entries about ten years out.

Step 5 — Practise the node lifecycle. Cordon and drain a worker, watch a workload move, then return the node.

kubectl create deployment web --image=nginx --replicas=4
kubectl rollout status deployment/web

kubectl cordon provlab-worker          # no new Pods will land here
kubectl get nodes                      # note SchedulingDisabled on that node

kubectl drain provlab-worker --ignore-daemonsets --delete-emptydir-data
kubectl get pods -o wide               # web Pods have moved off the drained node

kubectl uncordon provlab-worker        # back in the pool
kubectl get nodes

Expected: after drain, no web Pods remain on provlab-worker; after uncordon it is schedulable again.

Validation. You should have: a 3-node cluster, a verified etcd snapshot with a non-zero key count, a certificate-expiry report, and a worker you successfully drained and returned to service — the four operational skills of running a self-managed cluster.

Cleanup.

kind delete cluster --name provlab
rm -f kind-ha.yaml

Cost note. Zero — kind runs entirely in local containers. The identical commands (etcdctl snapshot save, kubeadm certs check-expiration, kubectl drain) are exactly what you run on a real kubeadm cluster; only the host changes.

Practice challenges

Work these top-to-bottom — they escalate from beginner to advanced. Try each before opening the solution. The command-based ones run against the local kind cluster from the lab (provlab); the design ones are pen-and-paper reasoning you should be able to defend in an interview.

Challenge 1 (beginner) — quorum arithmetic. You must size etcd for a cluster that has to survive two simultaneous node failures. How many etcd members do you run, and why is running one more than that a bad idea?