Containerization Fundamentals

Kubernetes ConfigMaps & Secrets, In Depth: Injection, Mounting, Immutability & Encryption

Almost every application needs two things from its environment that do not belong baked into the container image: configuration (the database hostname, a feature flag, a log level, an entire config file) and secrets (the database password, an API token, a TLS private key, a registry credential). Hard-coding either into the image is the classic anti-pattern — you cannot promote the same image from staging to production, and you end up rebuilding just to change a hostname or, worse, leaking a password into a registry layer.

Kubernetes gives you two purpose-built objects for this: the ConfigMap for non-sensitive configuration, and the Secret for sensitive data. They look almost identical on the surface — both are key/value stores that you inject into Pods — but they differ in how they are stored, how they are protected, and how you are expected to treat them. This lesson covers both, exhaustively: every field, every Secret type, every way to get the data into a container, what happens when you update them, how to make them tamper-resistant with immutability, and — the part most tutorials skip — how Secrets are actually stored in etcd and how to turn on encryption at rest so they are not sitting there in plain base64.

The single most important sentence to internalise up front: a Secret is not encrypted by default — its values are merely base64-encoded, which is encoding, not encryption. Anyone who can read the Secret object (through the API or by reading etcd) can decode it instantly. Everything in the security half of this lesson exists to close that gap.

Learning objectives

By the end of this lesson you can:

Prerequisites & where this fits

You need a terminal, a free local cluster (kind, minikube or k3d — the What Is Kubernetes? lesson walks you through installing one), and comfort with Pods and how a container’s environment and filesystem work — covered in Kubernetes Pods, In Depth. It also helps to have met Services, since one common use of a ConfigMap is to hand an app the address of a Service — see Kubernetes Services & Networking, In Depth. This is Lesson 5 of the Kubernetes Zero-to-Hero course (Foundation tier); it sits between Services and Namespaces, ResourceQuotas & LimitRanges, and it is the foundation for everything you will later do with the Secrets Store CSI driver, sealed secrets and GitOps.

Core concepts: config vs secrets, and the data model

Both objects are namespaced (they live in a namespace and can only be consumed by Pods in the same namespace) and both are decoupled from Pods — you change the object, and Pods that reference it pick up the change according to the rules below. The mental model:

A few hard limits and rules apply to both:

Property ConfigMap Secret Notes
Scope Namespaced Namespaced A Pod can only mount one from its own namespace.
Size limit 1 MiB total 1 MiB total The limit is etcd’s; it counts keys, values and metadata. Use a volume/PVC or an external store for anything larger.
Key name rules Must be a valid env-var name if used as env; otherwise alphanumerics, -, _, . Same Keys with . work for files but not as env-var names.
Value storage Plain string in etcd base64 in the manifest; plain or encrypted in etcd base64 is not security.
Can be immutable? Yes (immutable: true) Yes (immutable: true) Optional, recommended at scale.

Jargon check. etcd is the cluster’s key/value database — the single source of truth where every object, including every Secret, is stored. If a Secret is not encrypted at rest, it sits in etcd in plain base64, which is why protecting etcd (and turning on encryption) matters so much.

ConfigMaps: every field

A ConfigMap is deliberately simple. It has no spec — the data lives at the top level:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: demo
data:                       # UTF-8 string values
  LOG_LEVEL: "info"
  FEATURE_FLAGS: "beta,checkout-v2"
  app.properties: |        # a whole file as one value
    server.port=8080
    cache.ttl=300
binaryData:                 # base64-encoded binary values
  logo.png: iVBORw0KGgoAAA...   # raw bytes, base64-encoded
immutable: false            # optional; see the immutability section

The fields:

Field What it holds Values Default When to use Gotcha
data UTF-8 string key/value pairs Any UTF-8 string; the value can be multi-line (a whole file) empty Almost all config: flags, hostnames, full *.properties/*.yaml/*.conf files A value must be valid UTF-8 — if you put binary here it will be rejected. Use binaryData.
binaryData binary values, base64-encoded in the manifest Base64 of arbitrary bytes empty Binary config: a small icon, a .p12, a gzipped blob Keys in data and binaryData must not collide. When mounted, the file contains the decoded bytes.
immutable Lock the object against updates true/false false Config that should never change in place (see below) Once true you cannot edit data/binaryData — you must delete and recreate.

Creating ConfigMaps imperatively

kubectl create configmap is the fastest way, and it has four source flags you can mix:

# --from-literal: individual key=value pairs
kubectl create configmap app-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=FEATURE_FLAGS=beta,checkout-v2

# --from-file: the file's *name* becomes the key, its *contents* the value
kubectl create configmap app-config --from-file=app.properties
# rename the key:  --from-file=props=app.properties

# --from-file with a DIRECTORY: every file in it becomes one key
kubectl create configmap nginx-conf --from-file=./conf.d/

# --from-env-file: a KEY=VALUE file → one ConfigMap key per line
kubectl create configmap app-config --from-env-file=.env
Flag Each key becomes… Each value becomes… Typical use
--from-literal=K=V K V A handful of simple settings
--from-file=path the filename the file contents Inject a whole config file
--from-file=key=path key (renamed) the file contents Control the mounted filename
--from-file=dir/ each filename in the dir each file’s contents A directory of configs (e.g. nginx conf.d)
--from-env-file=.env the left side of each K=V line the right side Turn an existing .env into many keys

A pro habit: generate the YAML rather than creating live objects, so it goes into Git:

kubectl create configmap app-config \
  --from-literal=LOG_LEVEL=info \
  --dry-run=client -o yaml > configmap.yaml

Secrets: every type, in full

A Secret has the same data/binaryData/immutable shape as a ConfigMap, plus a type and a convenience field stringData:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: demo
type: Opaque
data:                       # values MUST be base64-encoded here
  username: YWRtaW4=        # "admin"
  password: czNjcjN0UGFzcw==
stringData:                 # write-only convenience: plain text in, base64 at rest
  connection-string: "postgres://admin:s3cr3tPass@db:5432/app"
immutable: false
Field What it does Gotcha
type Tells Kubernetes (and tools) what shape the Secret has and which keys are required Built-in types validate required keys; Opaque validates nothing.
data base64-encoded values If you hand-write YAML, you must base64-encode. echo -n value | base64.
stringData Plain-text values that the API base64-encodes for you on write Write-only — it never shows up on read; on read you see it merged into data. If a key exists in both, stringData wins.
binaryData base64-encoded binary values (same as ConfigMap) Use for genuinely binary secrets.
immutable Lock against updates Same semantics as ConfigMap.

The base64 trap. data values are base64, not encrypted. echo czNjcjN0UGFzcw== | base64 -d prints the password. base64 exists so that binary data survives being stored as a string — it provides zero confidentiality. Treat a Secret manifest exactly as you would treat the plaintext password.

The Secret types

type is the field that distinguishes Secret flavours. Each built-in type expects a specific set of keys, and some Kubernetes features only accept the right type (e.g. imagePullSecrets only accepts dockerconfigjson, Ingress TLS only accepts kubernetes.io/tls).

type Required keys What it is for How you usually create it
Opaque none (arbitrary) The default, general-purpose bucket for any sensitive key/value kubectl create secret generic
kubernetes.io/dockerconfigjson .dockerconfigjson Pulling images from a private registry (used by imagePullSecrets) kubectl create secret docker-registry
kubernetes.io/dockercfg .dockercfg Legacy registry-credential format rarely; superseded by the above
kubernetes.io/tls tls.crt, tls.key TLS cert + private key for Ingress / app TLS kubectl create secret tls
kubernetes.io/basic-auth username, password HTTP basic-auth credentials kubectl create secret generic --type=...
kubernetes.io/ssh-auth ssh-privatekey An SSH private key (e.g. for Git clone) kubectl create secret generic --type=...
kubernetes.io/service-account-token token (+ annotations) A long-lived ServiceAccount token (legacy) annotate with kubernetes.io/service-account.name
bootstrap.kubernetes.io/token token-id, token-secret, … Node bootstrap tokens (kubeadm join) cluster bootstrap; not app config

Creating each of the common ones imperatively:

# Opaque — the everyday case
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password='s3cr3tPass'

# docker-registry — for pulling private images
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=ci-bot \
  --docker-password='REDACTED' \
  --docker-email=ci@example.com
#  → produces a single key  .dockerconfigjson

# tls — cert + key
kubectl create secret tls web-tls \
  --cert=tls.crt --key=tls.key

# basic-auth and ssh-auth (note the explicit --type)
kubectl create secret generic basic-cred \
  --type=kubernetes.io/basic-auth \
  --from-literal=username=admin --from-literal=password='s3cr3tPass'

dockerconfigjson and imagePullSecrets

The dockerconfigjson Secret is what lets a node authenticate to a private registry. You reference it from a Pod (or, better, attach it to a ServiceAccount so every Pod using that SA gets it):

spec:
  imagePullSecrets:
    - name: regcred          # the docker-registry Secret above
  containers:
    - name: app
      image: registry.example.com/team/app:1.4.0
# Attach to a ServiceAccount so you never repeat it per-Pod:
kubectl patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

ServiceAccount tokens — the modern default

Historically every ServiceAccount got a long-lived kubernetes.io/service-account-token Secret minted automatically. Since Kubernetes 1.24 that no longer happens. The modern, more secure path is bound, projected tokens: short-lived, audience-scoped JWTs that kubelet refreshes and mounts via a projected volume (covered below). You only create a service-account-token Secret by hand when you genuinely need a long-lived token (e.g. for an external CI system), and even then a short-lived token from kubectl create token <sa> is preferred.

Consuming config and secrets in Pods

This is where most of the day-to-day knowledge lives. There are four ways to get data into a container, and the choice has real consequences (especially for whether updates reach a running Pod). The same four mechanisms work for both ConfigMaps and Secrets — the YAML keys just change (configMapKeyRefsecretKeyRef, configMapsecret).

Method What the container sees Updates live? Best for
Single env var (valueFrom) One environment variable No — env is set once at container start A few individual settings
envFrom Every key as an env var No Loading a whole ConfigMap/Secret of flags
Volume mount A directory of files, one file per key Yes (with delay; see propagation) Whole config files; values that change
Projected volume Multiple sources combined into one dir; SA tokens Yes Combining sources; bound SA tokens

1 & 2 — Environment variables

spec:
  containers:
    - name: app
      image: myapp:1.0
      env:
        # one specific key from a ConfigMap
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: LOG_LEVEL
        # one specific key from a Secret
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
      envFrom:
        # EVERY key in this ConfigMap becomes an env var
        - configMapRef:
            name: app-config
        # EVERY key in this Secret becomes an env var, with a prefix
        - secretRef:
            name: db-credentials
          prefix: DB_          # → DB_username, DB_password

Important env-var rules and gotchas:

Behaviour Detail
No live updates Environment variables are baked into the process at start. Editing the ConfigMap/Secret does not change a running container’s env — you must restart the Pod (kubectl rollout restart deployment/...).
Invalid env names are skipped With envFrom, any key that is not a valid shell identifier (e.g. app.properties, my-key) is silently skipped and an event is logged. Use volume mounts for file-like keys.
Missing reference = Pod won’t start If a configMapKeyRef/secretKeyRef points at something that doesn’t exist, the container fails to start with CreateContainerConfigErrorunless you mark it optional: true.
prefix Only valid on envFrom; prepends a string to every imported key name.
Secrets in env leak easily Env vars show up in kubectl describe pod references, crash dumps, /proc/<pid>/environ, and child processes. Prefer mounting Secrets over env for anything truly sensitive.

Mark a reference optional so a missing source does not block startup:

env:
  - name: OPTIONAL_FLAG
    valueFrom:
      configMapKeyRef:
        name: maybe-missing
        key: FLAG
        optional: true        # absent ⇒ var simply not set, Pod still starts
envFrom:
  - configMapRef:
      name: maybe-missing
      optional: true          # absent ⇒ no vars imported, Pod still starts

3 — Volume mounts (files)

Mounting turns each key into a file inside a directory. This is the right choice for whole config files and for any value you want to update without restarting:

spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: config-vol
          mountPath: /etc/app          # files appear here, one per key
          readOnly: true
        - name: secret-vol
          mountPath: /etc/app/secrets
          readOnly: true
  volumes:
    - name: config-vol
      configMap:
        name: app-config
        defaultMode: 0644              # file permissions for all keys
        items:                         # OPTIONAL: pick & rename specific keys
          - key: app.properties
            path: app.properties        # → /etc/app/app.properties
            mode: 0600                  # per-file override
    - name: secret-vol
      secret:
        secretName: db-credentials
        defaultMode: 0400              # secrets: read-only to owner only
        optional: false

The volume fields, exhaustively:

Field What it does Values / default Gotcha
configMap.name / secret.secretName Which object to mount Note the inconsistent field namesname for ConfigMap, secretName for Secret.
items Mount only some keys, optionally renamed List of {key, path, mode} If you specify items, only the listed keys are mounted; everything else is omitted.
items[].path Filename (relative to mountPath) Can include subdirs (tls/server.crt) Lets you reshape the layout.
items[].mode Per-file permission bits Octal (0600) Overrides defaultMode for that file.
defaultMode Permission bits for all files Octal; default 0644 (decimal 420 in YAML if not quoted) Always quote octal or write decimal — unquoted 0644 is parsed oddly. Combined with the process UID and fsGroup.
optional Don’t block the Pod if the object is missing true/false, default false With false, a missing object means the Pod stays Pending/ContainerCreating.
readOnly (on the mount) Mount the volume read-only default false Always set readOnly: true for config/secret mounts — the data is managed by Kubernetes, not the app.

Two subtleties that trip everyone up:

subPath — mount one file without hiding the directory

volumeMounts:
  - name: config-vol
    mountPath: /etc/nginx/nginx.conf   # a single file path
    subPath: nginx.conf                # mount just this key as that file
    readOnly: true
Behaviour Detail
What it does Mounts a single key as a single file, leaving the rest of the target directory intact.
The big trade-off A subPath mount does NOT receive updates. It is resolved once at mount time. If you edit the ConfigMap, a subPath-mounted file stays stale until the Pod restarts.
When to use Dropping a config file next to files that exist in the image (e.g. nginx.conf into /etc/nginx/).
Alternative for live updates Mount the whole volume at a dedicated dir (no subPath) and point the app there, or use subPathExpr only when you accept staleness.

4 — Projected volumes (combine sources + SA tokens)

A projected volume merges several sources into one directory. It is also how modern, short-lived ServiceAccount tokens are delivered:

volumes:
  - name: combined
    projected:
      defaultMode: 0440
      sources:
        - configMap:
            name: app-config
            items:
              - key: app.properties
                path: app.properties
        - secret:
            name: db-credentials
            items:
              - key: password
                path: db-password
        - downwardAPI:                 # pod metadata as files
            items:
              - path: pod-name
                fieldRef:
                  fieldPath: metadata.name
        - serviceAccountToken:         # short-lived, audience-bound token
            path: token
            expirationSeconds: 3600    # kubelet auto-rotates before expiry
            audience: vault            # who the token is valid for
Source type Provides Notes
configMap Keys as files Same items/optional semantics as a plain mount.
secret Keys as files Same as above.
downwardAPI Pod/container metadata as files Labels, annotations, name, namespace, resource limits.
serviceAccountToken A bound, expiring JWT The modern auth path; audience scopes it, kubelet refreshes it. This is what automountServiceAccountToken wires up under the hood.

Update propagation: who sees changes, and when

This is one of the most-asked interview questions because the answer is “it depends on how you consumed it.”

Consumption method Does a live Pod see an edit? Timing
Env var (valueFrom) Never Requires Pod restart.
envFrom Never Requires Pod restart.
Volume mount (no subPath) Yes kubelet refreshes on its sync loop — typically within ~60–90s (the kubelet syncFrequency plus cache TTL), not instant.
subPath mount No Resolved once at mount; needs restart.
Projected volume Yes (for configMap/secret sources) Same kubelet refresh timing.
Immutable ConfigMap/Secret N/A The object cannot be edited at all — you create a new one.

Two consequences:

  1. The application must re-read the file for a mounted update to take effect. Kubernetes updates the file; it cannot make your process reload. Apps either watch the directory (inotify) or you trigger a kubectl rollout restart.
  2. A very common production pattern is to make the change visible by triggering a restart. Tools like the “Reloader” controller, or a config-hash annotation in your Deployment template, change the Pod template whenever the ConfigMap/Secret changes, which forces a clean rolling restart — far more predictable than relying on in-place file refresh.
# config-hash pattern: annotation changes ⇒ rolling restart on every config edit
spec:
  template:
    metadata:
      annotations:
        checksum/config: "{{ sha256 of the configmap data }}"   # e.g. via Helm/Kustomize

Immutable ConfigMaps and Secrets

Set immutable: true and the object can no longer be edited — any attempt is rejected; to change it you must delete and recreate (typically under a new name). Two reasons to do this:

Benefit Why
Protection from accidental change An immutable Secret can’t be edited out from under running Pods by a fat-fingered kubectl edit.
Cluster performance at scale The kubelet stops watching immutable objects for changes. On large clusters with thousands of Pods/ConfigMaps, those watches are a real load on the API server and etcd; immutability removes them.
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v3       # version the NAME, since you can't edit in place
data:
  LOG_LEVEL: "info"
immutable: true

The natural pattern is versioned, immutable objects (app-config-v1, -v2, …): each release ships a new immutable ConfigMap/Secret and points the Deployment at it, which also gives you a clean rollback (point back at the old one) and automatic rolling restarts (the Pod template changed).

Gotcha. You cannot toggle immutable back to false, and you cannot add immutable: true and change data in the same edit. Plan to recreate.

Encryption at rest: how Secrets are really stored

By default, the API server writes Secrets to etcd as plain base64 — i.e. effectively plaintext. Anyone with read access to etcd (a backup file, a snapshot, a compromised etcd node) can read every Secret in the cluster. Encryption at rest fixes this by having the API server encrypt resources before writing them to etcd and decrypt on read.

You enable it with an EncryptionConfiguration file passed to the API server via --encryption-provider-config. It lists which resources to encrypt and, in order, which providers to use — the first provider encrypts new writes; all listed providers can decrypt (so you can roll keys).

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets               # encrypt Secrets (you can add configmaps, etc.)
    providers:
      - aescbc:               # the encrypting provider listed FIRST
          keys:
            - name: key2024-06
              secret: <base64-32-byte-key>
      - identity: {}          # MUST be present to read pre-encryption data

The providers, with their trade-offs:

Provider What it does Strength / speed When to use Gotcha
identity No encryption (plaintext) n/a The default; also needed last so old plaintext can still be read during migration Listing it first = nothing is encrypted.
secretbox XSalsa20-Poly1305 Fast, strong, modern Good local-key choice Keys live in the config file on the control plane.
aescbc AES-CBC with PKCS#7 Strong; older Widely used local-key option CBC has known padding-oracle concerns; many now prefer secretbox or KMS.
aesgcm AES-GCM Fast, authenticated Only with automated, frequent key rotation Nonce reuse is catastrophic — do not use with static keys.
kms (v2) Envelope encryption via an external KMS (cloud KMS / Vault) Strongest — root key never leaves the KMS Production, especially in the cloud Requires a KMS plugin; v2 is the current, performant version.

Key points to remember:

Note for managed clusters. On EKS/GKE/AKS you usually enable envelope encryption with the provider’s KMS through a checkbox/flag rather than hand-editing the API server — the managed control plane wires up the kms provider for you. The concept is identical; you just don’t own the manifest.

External secret stores (concept level)

Storing every secret as a Kubernetes Secret — even encrypted — has limits: secrets sprawl across clusters, rotation is manual, and your source of truth is etcd rather than a purpose-built vault. Two mature patterns pull secrets from an external store (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager):

Approach How it works You get Trade-off
Secrets Store CSI Driver A CSI volume that, at Pod start, fetches secrets from the external store and mounts them as files (optionally also syncing them into a native Secret) Secrets never need to live in etcd; values mount straight from the vault; auto-rotation supported Adds a CSI driver + a provider (Vault/AWS/Azure/GCP); secrets only present while a Pod mounts them.
External Secrets Operator (ESO) A controller watches an ExternalSecret custom resource and creates/refreshes a normal Kubernetes Secret from the external store on a schedule Apps consume an ordinary Secret (no app changes); central source of truth; periodic resync/rotation The materialised Secret does live in etcd (so still encrypt at rest); eventual-consistency on rotation.

The mental distinction: CSI driver = mount from the vault into the Pod (etcd optional); ESO = mirror the vault into a native Secret (etcd always). Both keep the authoritative secret in a real vault with audit logging and rotation, and both are vastly better than committing secrets to Git. (If you must keep secrets in Git for GitOps, Sealed Secrets or SOPS-encrypted manifests are the encrypt-before-commit answer — a topic for a later lesson.)

Kubernetes core objects, in one picture

Kubernetes core objects

ConfigMaps and Secrets sit alongside the workload and networking objects you have already met: a Deployment owns Pods, a Service gives them a stable address, and ConfigMaps and Secrets feed configuration and credentials into those Pods — by environment variable or by mounted volume. Keeping config and secrets as separate objects is exactly what lets you ship one image to every environment and change behaviour by swapping the attached ConfigMap/Secret.

Hands-on lab

Free, local, ~15 minutes. Works on kind, minikube or k3d. We will create a ConfigMap and a Secret, consume them as env vars and as files, prove the base64 point, prove that mounted updates propagate but env vars don’t, and clean up.

0. A namespace to keep it tidy

kubectl create namespace cms-lab
kubectl config set-context --current --namespace=cms-lab

1. Create a ConfigMap (literals + a file)

printf 'server.port=8080\ncache.ttl=300\n' > app.properties

kubectl create configmap app-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=GREETING=hello \
  --from-file=app.properties

kubectl get configmap app-config -o yaml

Expected: data contains LOG_LEVEL, GREETING, and an app.properties key whose value is the file’s contents.

2. Create a Secret and prove base64 is not encryption

kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password='s3cr3tPass'

# The stored value is base64 — trivially reversible:
kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d ; echo
# → s3cr3tPass

3. A Pod that consumes both — env vars AND files

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: consumer
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      env:
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef: { name: app-config, key: LOG_LEVEL }
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: { name: db-credentials, key: password }
      envFrom:
        - configMapRef: { name: app-config }
      volumeMounts:
        - name: cfg
          mountPath: /etc/app
          readOnly: true
        - name: sec
          mountPath: /etc/secret
          readOnly: true
  volumes:
    - name: cfg
      configMap: { name: app-config }
    - name: sec
      secret: { secretName: db-credentials, defaultMode: 0400 }
EOF

kubectl wait --for=condition=Ready pod/consumer --timeout=60s

4. Validate both injection styles

# env var from the Secret:
kubectl exec consumer -- printenv DB_PASSWORD          # → s3cr3tPass
# envFrom imported the whole ConfigMap:
kubectl exec consumer -- printenv GREETING LOG_LEVEL   # → hello / info
# the ConfigMap mounted as files:
kubectl exec consumer -- ls /etc/app                   # → LOG_LEVEL GREETING app.properties
kubectl exec consumer -- cat /etc/app/app.properties   # → the file contents
# the Secret mounted as files, owner-read-only:
kubectl exec consumer -- ls -l /etc/secret             # → -r-------- username, password

5. Prove the propagation rules

# Edit the ConfigMap's GREETING value:
kubectl patch configmap app-config --type merge -p '{"data":{"GREETING":"namaste"}}'

# The ENV VAR does NOT change (set at start):
kubectl exec consumer -- printenv GREETING             # → still "hello"

# The MOUNTED file DOES change — but allow ~60-90s for kubelet to refresh.
# Re-run until it flips:
kubectl exec consumer -- sh -c 'cat /etc/app/GREETING; echo'   # → eventually "namaste"

That contrast — env frozen, file updated — is the whole propagation lesson in two commands.

6. Make a versioned, immutable Secret (optional)

kubectl create secret generic db-credentials-v2 \
  --from-literal=username=admin --from-literal=password='newPass' \
  --dry-run=client -o yaml | \
  kubectl apply -f - --dry-run=client -o yaml      # inspect first
# Then add `immutable: true` in the manifest and apply for real.

Cleanup

kubectl delete namespace cms-lab          # removes Pod, ConfigMap, Secrets in one go
kubectl config set-context --current --namespace=default
rm -f app.properties

Cost note: entirely free — everything runs on your local kind/minikube/k3d cluster; no cloud resources are created.

Common mistakes & troubleshooting

Symptom Likely cause Fix
CreateContainerConfigError on the Pod A configMapKeyRef/secretKeyRef/secretName points at a key or object that doesn’t exist Create the object/key, or set optional: true; check kubectl describe pod.
Pod stuck ContainerCreating, event “couldn’t find key/secret” Mounted ConfigMap/Secret missing and optional is false Create it (it must be in the same namespace) or mark optional.
Edited a ConfigMap but the app didn’t change Consumed as env var, or via subPath, or app doesn’t re-read the file Restart the Pod (kubectl rollout restart), or switch to a non-subPath mount + an app that watches the dir.
Some envFrom keys never appear Keys aren’t valid env-var names (contain ./-) They’re silently skipped — mount those keys as files instead.
Mounted file has the wrong permissions defaultMode written unquoted, or overridden by items[].mode Quote octal ("0400") or use decimal; remember fsGroup/UID interaction.
kubectl get secret -o yaml shows my password “encrypted” but it’s readable It’s base64, not encryption This is expected; enable encryption at rest, restrict RBAC, don’t commit manifests.
Image pull fails with ImagePullBackOff on a private registry Missing/incorrect imagePullSecrets or wrong Secret type Create a docker-registry Secret and reference it (Pod or ServiceAccount).
Secret edits rejected outright Object is immutable: true Create a new (versioned) object and repoint the workload.

Best practices

Security notes

Interview & exam questions

  1. Is a Kubernetes Secret encrypted? What is base64 doing there? No — values are base64-encoded, which is reversible and provides no confidentiality. base64 only lets binary data be stored as a string. Real protection comes from encryption at rest, RBAC, and not leaking the manifest.

  2. A teammate edits a ConfigMap; the app’s behaviour doesn’t change. Why, and how do you fix it? It was consumed as an environment variable (or via subPath), which is set once at container start and never updates. Restart the Pods (kubectl rollout restart), or switch to a non-subPath volume mount and have the app watch the file/directory.

  3. How fast does a mounted ConfigMap update reach a running Pod? Not instantly — kubelet refreshes mounted ConfigMaps/Secrets on its sync loop, typically within ~60–90 seconds; the app must then re-read the file. subPath mounts and env vars never update.

  4. List the Secret types and when you’d use each. Opaque (general); kubernetes.io/dockerconfigjson (private-registry pulls / imagePullSecrets); kubernetes.io/tls (tls.crt+tls.key for Ingress/app TLS); kubernetes.io/basic-auth; kubernetes.io/ssh-auth; kubernetes.io/service-account-token (legacy long-lived SA token). Some features require the matching type.

  5. What does immutable: true give you, beyond preventing edits? The kubelet stops watching the object, cutting API-server/etcd load at scale, and it prevents accidental in-place changes to running workloads. You version the name and recreate to change it.

  6. Walk me through enabling encryption at rest and rotating the key. Provide an EncryptionConfiguration via --encryption-provider-config listing the encrypting provider first and identity last. Existing Secrets stay plaintext until rewritten (kubectl get secrets -A -o json | kubectl replace -f -). To rotate: add the new key first in the provider’s key list, keep the old key for decryption, restart API servers, re-encrypt all Secrets, then drop the old key.

  7. data vs stringData in a Secret? data must be base64-encoded; stringData takes plaintext that the API encodes for you and is write-only (never shown on read). If both define a key, stringData wins.

  8. How do you give a Pod credentials to pull from a private registry? Create a docker-registry Secret (type kubernetes.io/dockerconfigjson) and reference it via imagePullSecrets on the Pod or, better, attach it to the ServiceAccount.

  9. configMapKeyRef vs envFrom vs a volume mount — when each? configMapKeyRef/secretKeyRef for one specific key as one env var; envFrom to import every key as env vars (with optional prefix); a volume mount for whole files or any value you want to update without restarting.

  10. Why prefer mounting a Secret over injecting it as an env var? Env vars appear in /proc/<pid>/environ, crash dumps, describe references and child processes, and never live-update. Mounted secrets avoid that exposure and support rotation.

  11. What changed about ServiceAccount tokens in 1.24, and what’s the modern approach? Auto-creation of long-lived token Secrets stopped. The modern path is bound, projected tokens — short-lived, audience-scoped JWTs mounted via a projected volume and auto-rotated by kubelet; kubectl create token mints one on demand.

  12. Compare the Secrets Store CSI Driver and the External Secrets Operator. The CSI driver mounts secrets from an external vault straight into the Pod as files (etcd optional); ESO mirrors an external vault into a normal Kubernetes Secret via an ExternalSecret CR (etcd always, so still encrypt at rest). Both keep the source of truth in a real vault with rotation/audit.

Quick check

  1. True or false: editing a Secret consumed via envFrom updates the value in a running Pod.
  2. Which Secret type is required for an Ingress TLS certificate, and which two keys must it contain?
  3. You mount a ConfigMap with subPath. Will a later edit to that ConfigMap reach the Pod without a restart?
  4. In an EncryptionConfiguration, which provider must remain listed so you can still read previously unencrypted Secrets, and where in the list does the encrypting provider go?
  5. Name two concrete benefits of setting immutable: true on a ConfigMap.

Answers

  1. False. Environment variables (including those from envFrom) are set once at container start and never update — restart the Pod.
  2. kubernetes.io/tls, containing tls.crt and tls.key.
  3. No. subPath mounts are resolved once at mount time and do not receive updates — the Pod must restart.
  4. identity must remain listed (so plaintext/old data can still be read); the encrypting provider must be listed first (the first provider is used for writes, all providers can decrypt).
  5. (a) It prevents accidental in-place edits to running workloads, and (b) the kubelet stops watching it, reducing API-server/etcd load at scale. (Versioned rollbacks are a third.)

Exercise

Build a tiny “promote one image across environments” demo:

  1. Create two namespaces, staging and prod.
  2. In each, create a ConfigMap web-config with a different BANNER value (e.g. "STAGING" vs "PROD") and a Secret web-cred with a different API_TOKEN.
  3. Deploy the same nginx-based Pod (same image) into both namespaces, mounting BANNER as a file at /usr/share/nginx/html/banner.txt and injecting API_TOKEN as a mounted Secret (not an env var).
  4. Verify each Pod serves the correct banner and has the correct token file, with the token file readable only by the owner (0400).
  5. Make both web-config objects immutable, then attempt kubectl edit and observe the rejection. Recreate as web-config-v2 and repoint the Pod.
  6. Bonus: add a checksum/config annotation to a Deployment version of step 3 and confirm that changing the ConfigMap triggers a rolling restart.

Success criteria: identical image in both namespaces, environment-specific behaviour driven entirely by the attached ConfigMap/Secret, secret delivered as a 0400 file, and immutability enforced.

Certification mapping

Exam Objective area this supports
KCNA (Kubernetes and Cloud Native Associate) Kubernetes Fundamentals — ConfigMaps and Secrets as core objects; how applications consume configuration.
CKAD (Certified Kubernetes Application Developer) Application Environment, Configuration & Security — create and consume ConfigMaps/Secrets (env, envFrom, volumes, projected, subPath, optional, defaultMode), imagePullSecrets, ServiceAccount tokens, and SecurityContext interactions.
CKA (Certified Kubernetes Administrator) Workloads & Scheduling / Cluster Architecture — manage ConfigMaps/Secrets; understand storage in etcd and immutability’s effect on the control plane.
CKS (Certified Kubernetes Security Specialist) Cluster Hardening & Minimise Microservice Vulnerabilitiesencryption at rest (EncryptionConfiguration, providers, key rotation), Secret RBAC and least privilege, short-lived bound tokens, and external secret stores.

Glossary

Next steps

Continue the course with Kubernetes Namespaces, ResourceQuotas & LimitRanges, In Depth — now that you can inject config and credentials, learn how namespaces isolate them and how quotas and limit ranges keep tenants in their lane. Then go further with:

KubernetesConfigMapsSecretsSecurityEncryptionCKAD
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments

Keep Reading