Terraform on AWS EKS: the EBS CSI Driver, StorageClasses & Persistent Volumes for Stateful Workloads

A fresh EKS cluster can schedule a thousand stateless pods and lose none of them — but ask it to run a single Postgres, a Kafka broker, a Prometheus TSDB, or anything that must remember something after a restart, and you discover the cluster has no durable storage at all until you give it some. There is no default place for a pod’s data to live that survives the pod. On AWS the answer for single-writer, low-latency, block storage is Amazon EBS, and the bridge between “a pod wants a 10 GiB volume” and “AWS creates, attaches, and formats an EBS volume on the right node” is the EBS CSI driver — a controller that speaks the Kubernetes storage API on one side and the EC2 API on the other. This lesson builds that bridge with Terraform: the driver installed as a first-class EKS add-on, permissioned with IRSA (not node-role credentials), fronted by a gp3 StorageClass tuned the way production wants it, and exercised by a real StatefulSet whose PersistentVolumeClaim binds an actual EBS volume you can see in aws ec2 describe-volumes.

Storage on Kubernetes is where a lot of otherwise-competent platform engineers get quietly burned, because the failure modes are not loud. A misconfigured StorageClass does not error at apply time — it errors hours later when the first PVC hangs Pending with a cryptic event, or weeks later when a pod cannot reschedule because its EBS volume is trapped in an Availability Zone the pod can no longer be placed in. The two settings that prevent most of that pain — the IRSA role on the driver and volumeBindingMode: WaitForFirstConsumer on the StorageClass — are exactly the two settings people copy wrong from an old blog. We treat both as first-class topics, not footnotes, alongside the one physical fact that governs every stateful design on AWS: an EBS volume lives in a single Availability Zone and attaches to one node at a time. Internalise that and half of “why won’t my pod start” answers itself.

By the end you will have a complete, copy-pasteable configuration you run yourself: terraform init → plan → apply installs the add-on, the IRSA role, and a default gp3 StorageClass; a StatefulSet then binds two encrypted EBS volumes; you verify with kubectl get pv,pvc and the EC2 API, grow a volume online by editing the PVC, snapshot it through the CSI snapshot controller, and terraform destroy — with the ⚠️ PVC-and-reclaim gotcha spelled out so you do not leave orphaned volumes billing overnight. Every knob is laid out in reference tables you will come back to: the PV/PVC/StorageClass object model, the add-on-versus-Helm install matrix, the StorageClass parameter set, the volumeBindingMode decision, and a troubleshooting table for the failures that actually happen.

This lesson assumes the cluster already exists. If you need to build it, the companion Provisioning EKS: VPC, Node Groups & the Cluster lesson stands up the VPC and managed node groups, and EKS OIDC & IRSA: IAM Roles for Service Accounts builds the OIDC provider and the IRSA pattern this driver depends on. For shared, multi-writer, multi-AZ storage — the opposite trade-off from EBS — see the sibling EKS EFS CSI: Shared ReadWriteMany Storage lesson. Core Terraform (HCL, providers, variables, state, modules, for_each) is assumed from the course foundation tier, and the aws provider auth plus the S3/DynamoDB backend from Getting Started on AWS: Provider Auth & S3/DynamoDB Backend. We pin hashicorp/aws ~> 5.0 and hashicorp/kubernetes ~> 2.35, assume Terraform ≥ 1.6 (OpenTofu is a drop-in), and run in ap-south-1 (Mumbai) to keep the INR bill small.

What you’ll build

The scenario is the one every team hits the first time a “we’ll run it on Kubernetes” project needs a database: a stateful workload that must keep its data across pod restarts, node replacements, and rolling updates. Concretely, one terraform apply produces the EBS CSI driver as an EKS add-on, an IAM role for its controller ServiceAccount (IRSA) bearing AmazonEBSCSIDriverPolicy, and a gp3 StorageClass marked cluster-default with WaitForFirstConsumer, encrypted = true, and allowVolumeExpansion = true. Then you apply a two-replica StatefulSet whose volumeClaimTemplates mint one PVC per replica; the driver dynamically provisions two encrypted gp3 EBS volumes — each in the AZ where its pod landed — attaches them, and the pods start writing. You will see two Bound PVCs, two PersistentVolumes, and two real volumes in the EC2 console, then grow one from 10 GiB to 20 GiB with a single PVC edit and snapshot it.

Why Terraform rather than eksctl, kubectl apply, or raw Helm? Because the driver, its IAM role, and the StorageClass are infrastructure you provision once per cluster and change over time — precisely what declarative IaC is for. The add-on’s version, the IRSA trust policy, and the StorageClass parameters all drift if managed by hand, and drift in the storage layer is the kind you find out about during an incident. The comparison is worth pinning down:

Terraform’s edge is not that it is the only tool that can install a CSI driver — Helm and eksctl both can. It is that the same plan → apply → destroy workflow, the same state discipline, and the same CI pipeline cover the add-on, the IAM role that permissions it, the StorageClass that configures it, and the VPC and node groups underneath — one workflow across the whole stack. Here is the full build as a table of resources so you can see the moving parts before the code:

Read the diagram left to right: Terraform installs the driver as an add-on and hands it an IRSA role (badge 1) so its controller may call the EC2 API; a gp3 StorageClass (badge 3) with WaitForFirstConsumer (badge 2) waits for a pod before provisioning; the PVC binds (badge 5 — it is also expandable) and the driver carves an encrypted, single-AZ EBS volume (badge 4) whose reclaim policy (badge 6) decides its fate on delete. The six legend entries are the six decisions you make in code below.

Kubernetes storage on EKS: PV, PVC, StorageClass & dynamic provisioning

Kubernetes deliberately splits storage into three objects so that the person who needs a volume never has to know how it is made. That separation is the whole model, and getting the vocabulary exact prevents most confusion downstream:

The flow is: an app author writes a PVC naming a StorageClass; the StorageClass’s provisioner (the CSI driver) creates a real EBS volume and a matching PV; Kubernetes binds the PVC to that PV; the pod mounts the PVC. The app never names an AWS volume ID, an AZ, or a driver — it names a class and a size. That indirection is what lets the same Deployment run unchanged on EKS with EBS, on AKS with Azure Disk, or on-prem with Ceph; only the StorageClass differs.

There are two ways a PV comes into existence, and modern clusters use exactly one of them:

We use dynamic provisioning exclusively: the reader writes a PVC, and an EBS volume appears. Static provisioning still matters for importing a volume that already holds data (you create an aws_ebs_volume, then a PersistentVolume referencing its volumeHandle), but for greenfield workloads dynamic is correct.

Access modes are the next concept people trip on, because EBS supports only some of them — and the ones it does not support are exactly the ones a naive ReadWriteMany PVC asks for:

That table is the single most important reason to know both the EBS and EFS drivers. EBS is a block device: fast, low-latency, single-writer, one-AZ — perfect for a database’s data directory. EFS is an NFS file system: multi-writer, multi-AZ, higher latency — perfect for shared assets a fleet of pods all read and write. If a PVC asks for ReadWriteMany against a gp3 class it will hang Pending forever, because no EBS volume can satisfy it. That is the cross-over point to the EFS CSI lesson.

The CSI model: why in-tree is gone

For years, Kubernetes shipped cloud storage drivers in-tree — the AWS EBS provisioner (kubernetes.io/aws-ebs) was compiled into Kubernetes itself. That coupled storage releases to Kubernetes releases and forced every cloud’s code into the core binary, so the project moved everything to the Container Storage Interface (CSI): an out-of-tree, versioned, vendor-maintained plugin API. The consequences are concrete and current:

The migration has a sharp edge worth stating plainly: on any current EKS cluster you must install the EBS CSI driver even to use the old gp2 StorageClass. EKS still ships a default gp2 class that lists the legacy provisioner kubernetes.io/aws-ebs, but the in-tree code is gone; a shim called CSI migration transparently routes those calls to ebs.csi.aws.com — if the driver is installed. Skip the driver and even a plain gp2 PVC hangs Pending. So the driver is not optional plumbing you add for fancy features; it is the thing that makes dynamic block storage work at all.

The EBS CSI driver: EKS add-on vs Helm, and IRSA

The driver has two halves. A controller Deployment (in kube-system) watches PVCs and calls the EC2 API to create, attach, delete, snapshot, and resize volumes. A node DaemonSet runs on every node and does the local work — formatting the block device, mounting it into the pod, growing the filesystem. The controller is the half that needs AWS credentials, and how it gets them is the crux of this whole lesson.

First, how to install it. Three paths exist; they are not equivalent:

We use the managed add-on. The advantages are real: AWS validates each add-on version against each Kubernetes version, the manifests are maintained for you, upgrades are a single version bump, and the add-on integrates with EKS’s own health reporting. The aws_eks_addon arguments you will set:

⚠️ Deprecation note: older examples set a single resolve_conflicts argument. That is deprecated — use the split resolve_conflicts_on_create / resolve_conflicts_on_update pair. Copying the old single argument from a blog will throw a deprecation warning and, on newer provider versions, may be rejected.

IRSA: why the driver needs its own IAM role

The CSI controller must call ec2:CreateVolume, ec2:AttachVolume, ec2:DeleteVolume, and friends. There are two ways to give it those permissions, and only one is the least-privilege answer:

IRSA — IAM Roles for Service Accounts — is the mechanism that gives a specific Kubernetes ServiceAccount, and nothing else, a specific IAM role. It works because the cluster publishes an OIDC issuer; you register that issuer as an IAM OIDC identity provider; and you write an IAM role whose trust policy says “allow this role to be assumed via web identity only by the ServiceAccount ebs-csi-controller-sa in kube-system.” When the add-on runs with service_account_role_arn set, EKS annotates that ServiceAccount with the role ARN, a projected OIDC token is mounted into the pod, and the AWS SDK exchanges it for role credentials. No node-wide key, no secret, nothing to rotate. The pieces:

The trust policy is where correctness lives. Two StringEquals conditions must match exactly — the :sub (the ServiceAccount) and the :aud (always sts.amazonaws.com). Get the ServiceAccount name wrong and the assume fails silently; the controller logs WebIdentityErr and every PVC hangs Pending. The OIDC & IRSA lesson covers the mechanism end to end; here we consume it.

AmazonEBSCSIDriverPolicy is the AWS-managed policy purpose-built for this driver. It grants the EC2 volume verbs plus the ability to use the AWS-managed aws/ebs KMS key for default encryption. One gotcha: if you encrypt with a customer-managed KMS key (CMK), the managed policy is not enough — you must add kms:CreateGrant, kms:GenerateDataKeyWithoutPlaintext, kms:Decrypt, kms:ReEncrypt*, and kms:DescribeKey on that key to the IRSA role (and a matching key policy grant). The permissions the managed policy covers:

StorageClass: provisioner, parameters, volumeBindingMode & expansion

The StorageClass is the platform team’s recipe, and it is where you encode every decision an app author should not have to think about: which volume type, how fast, encrypted or not, what happens on delete, and — the subtle one — when the volume is created relative to where the pod is scheduled. In Terraform it is a kubernetes_storage_class resource. Here is the production-shaped default we build:

resource "kubernetes_storage_class" "gp3" {
  metadata {
    name = "gp3"
    annotations = {
      # Make this the cluster default so PVCs without a storageClassName use it.
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }

  storage_provisioner    = "ebs.csi.aws.com"
  volume_binding_mode    = "WaitForFirstConsumer" # ⚠️ the multi-AZ-correct setting
  allow_volume_expansion = true
  reclaim_policy         = "Delete"

  parameters = {
    type       = "gp3"
    iops       = "3000"  # gp3 baseline; up to 16000
    throughput = "125"   # MB/s baseline; up to 1000
    encrypted  = "true"
    # kmsKeyId = "arn:aws:kms:ap-south-1:<acct>:key/<id>"  # optional CMK
    # tagSpecification_1 = "team=payments"                  # tag the EBS volume
  }
}

The top-level arguments of kubernetes_storage_class:

Note every value in parameters is a string, even numbers and booleans ("3000", "true") — Kubernetes StorageClass parameters are stringly typed and Terraform will not coerce them for you. The EBS CSI driver’s parameter set:

gp3 is the current default choice, and choosing it deliberately matters. Its predecessor gp2 coupled performance to size — 3 IOPS per GiB — so the only way to get more IOPS was to over-provision capacity you did not need. gp3 decouples them: 3000 IOPS and 125 MB/s baseline at any size, dial-able independently, at roughly 20% lower cost per GiB. The volume-type landscape:

volumeBindingMode — the setting that prevents cross-AZ heartbreak

This is the most consequential single field on the StorageClass, and the default is the wrong choice for a multi-AZ cluster. It controls when the volume is provisioned and the PVC bound:

The failure Immediate produces is textbook and common: the driver creates the volume in, say, ap-south-1a; the scheduler then wants to run the pod in 1b (that is where CPU is free); but the volume’s PV carries a nodeAffinity pinning it to 1a; so the pod hangs Pending with volume node affinity conflict or 0/N nodes are available: had volume node affinity conflict. Nothing is broken — the volume and the pod simply disagree about which AZ they live in, because the volume was created before anyone knew where the pod would go. WaitForFirstConsumer inverts the order: schedule the pod first, then create the volume where the pod landed. On any cluster whose node groups span more than one AZ — i.e. every production EKS cluster — this must be WaitForFirstConsumer.

Default class, reclaim policy, and expansion

Three more decisions, each a one-liner with outsized consequences.

The default annotation. A PVC that omits storageClassName gets whatever class carries storageclass.kubernetes.io/is-default-class: "true". Our gp3 class claims it — but EKS also ships a default gp2 class, and two defaults is an error state: the admission controller picks arbitrarily and your PVCs may land on gp2. Part of the demo is stripping the default flag off the old gp2 class so gp3 is the sole default.

Reclaim policy decides what happens to the underlying EBS volume when its PVC is deleted:

Delete is convenient and is a data-loss trap in production: a stray kubectl delete pvc, or a terraform destroy that removes the namespace, silently wipes the volume. Precious data wants Retain, accepting that you must delete orphaned volumes manually later. You cannot change a bound PV’s reclaim policy through the StorageClass afterward — the PV copies it at creation — but you can patch an individual PV’s persistentVolumeReclaimPolicy in place.

allow_volume_expansion = true is what lets you grow a volume without recreating it. With it set, enlarging storage is a one-line edit to the PVC’s spec.resources.requests.storage; the driver expands the EBS volume via ec2:ModifyVolume and grows the filesystem online, no downtime, no pod restart. You can only ever grow, never shrink — a smaller value is rejected with field can not be less than previous value. Leave this on; a class you cannot expand is a class you will have to migrate off the day a disk fills.

PVC, the StatefulSet & the AZ-affinity trap

A PVC is the app author’s request. Minimal, it is three lines of intent — size, access mode, class:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp3
  resources:
    requests:
      storage: 10Gi

For a StatefulSet you rarely write the PVC by hand; you use volumeClaimTemplates, and the controller mints one PVC per replica with a stable name. That stability is the whole point of a StatefulSet: web-0 always gets data-web-0, which always binds the same PV, which is always the same EBS volume — so a restarted web-0 reattaches its own data. Deployment-plus-a-single-PVC does not give you that; every replica would fight over one RWO volume.

Now the trap the whole lesson circles back to. An EBS volume exists in exactly one Availability Zone. When the driver provisions it, the resulting PV carries a nodeAffinity requiring topology.ebs.csi.aws.com/zone = <that AZ>. Consequences that shape every stateful design on AWS:

So a stateful pod is married to its AZ. WaitForFirstConsumer solves the creation-time mismatch — the volume is born in the pod’s AZ — but once created, the pod cannot follow a reschedule into another AZ, because the data physically cannot go with it. The design answers are not Terraform tricks; they are architecture:

• Spread replicas one-per-AZ and let the application replicate (Postgres streaming replication, a Kafka replication factor of 3). Now an AZ loss costs you one replica, not the data. A StatefulSet across AZ-spanning node groups with WaitForFirstConsumer naturally places each replica in a different AZ.
• For shared, multi-AZ, multi-writer storage, use EFS, not EBS — that is the EFS CSI lesson, and RWX is the reason it exists.
• Do not fight it with cross-AZ hacks. There is no supported way to attach one EBS volume to nodes in two AZs; snapshot-and-restore into a new AZ is a copy, not a move.

Hands-on: build it with Terraform

Time to run it. This configuration assumes the EKS cluster from the cluster-provisioning lesson already exists (with its OIDC provider). Paste each file into a directory — say eks-ebs-csi/ — and follow the numbered steps. ⚠️ This provisions real, billable resources (EBS volumes, snapshots). Do the destroy at the end.

Step 1 — versions.tf (providers + backend + the cluster handoff)

The kubernetes provider is configured from the existing cluster via data sources — the read-only equivalent of the provider chaining you would do if you built the cluster in the same config. Because the cluster already exists, this is safe (the endpoint and token are known at plan time).

terraform {
  required_version = ">= 1.6"

  required_providers {
    aws        = { source = "hashicorp/aws", version = "~> 5.0" }
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.35" }
  }

  # Remote state — S3 with a DynamoDB lock (see the AWS "Getting Started" lesson).
  backend "s3" {
    bucket         = "kloudvin-tfstate-<account-id>" # globally unique
    key            = "eks/ebs-csi/dev.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "kloudvin-tflock" # or use_lockfile = true on TF >= 1.10
    encrypt        = true
  }
}

provider "aws" {
  region = var.region
}

# ---- Read the existing cluster --------------------------------------------
data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.cluster_name
}

# The IAM OIDC provider was created alongside the cluster (IRSA lesson).
data "aws_iam_openid_connect_provider" "this" {
  url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
}

# Configure the kubernetes provider FROM the live cluster.
provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

Step 2 — variables.tf

variable "region" {
  type    = string
  default = "ap-south-1"
}

variable "cluster_name" {
  description = "Name of the existing EKS cluster"
  type        = string
  default     = "kv-eks-dev"
}

variable "namespace" {
  description = "Namespace for the demo workload"
  type        = string
  default     = "storage-demo"
}

variable "tags" {
  type = map(string)
  default = {
    environment = "dev"
    managed_by  = "terraform"
    course      = "terraform-zero-to-hero"
  }
}

Step 3 — main.tf (IRSA role + add-on + StorageClass)

locals {
  # Strip the scheme so we can build the OIDC condition keys "<oidc>:sub" / ":aud".
  oidc_url = replace(data.aws_iam_openid_connect_provider.this.url, "https://", "")
}

# ---- IRSA role for the EBS CSI controller ---------------------------------
data "aws_iam_policy_document" "ebs_csi_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principal {
      type        = "Federated"
      identifiers = [data.aws_iam_openid_connect_provider.this.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_url}:sub"
      values   = ["system:serviceaccount:kube-system:ebs-csi-controller-sa"]
    }

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_url}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ebs_csi" {
  name               = "${var.cluster_name}-ebs-csi-driver"
  assume_role_policy = data.aws_iam_policy_document.ebs_csi_assume.json
  tags               = var.tags
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
  role       = aws_iam_role.ebs_csi.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

# ---- The EBS CSI driver as a managed EKS add-on ---------------------------
data "aws_eks_addon_version" "ebs_csi" {
  addon_name         = "aws-ebs-csi-driver"
  kubernetes_version = data.aws_eks_cluster.this.version
  most_recent        = true
}

resource "aws_eks_addon" "ebs_csi" {
  cluster_name  = data.aws_eks_cluster.this.name
  addon_name    = "aws-ebs-csi-driver"
  addon_version = data.aws_eks_addon_version.ebs_csi.version

  # Wire the controller ServiceAccount to the IRSA role — the whole point.
  service_account_role_arn = aws_iam_role.ebs_csi.arn

  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "PRESERVE"

  tags = var.tags

  depends_on = [aws_iam_role_policy_attachment.ebs_csi]
}

# ---- A production-shaped default gp3 StorageClass -------------------------
resource "kubernetes_storage_class" "gp3" {
  metadata {
    name = "gp3"
    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }

  storage_provisioner    = "ebs.csi.aws.com"
  volume_binding_mode    = "WaitForFirstConsumer"
  allow_volume_expansion = true
  reclaim_policy         = "Delete"

  parameters = {
    type       = "gp3"
    iops       = "3000"
    throughput = "125"
    encrypted  = "true"
  }

  depends_on = [aws_eks_addon.ebs_csi]
}

The default gp2 class EKS ships is still marked default too. Two defaults is an error; strip the flag off gp2 after apply (Step 6) so gp3 is the sole default. You can also do this in Terraform by importing the gp2 class and setting its annotation to "false", but a one-line kubectl patch is simpler for a class you did not create.

Step 4 — outputs.tf

output "ebs_csi_role_arn" {
  value = aws_iam_role.ebs_csi.arn
}

output "ebs_csi_addon_version" {
  value = aws_eks_addon.ebs_csi.addon_version
}

output "default_storage_class" {
  value = kubernetes_storage_class.gp3.metadata[0].name
}

Step 5 — init, plan, apply

Make sure your kubeconfig points at the cluster (so the provider token works), then initialise:

aws eks update-kubeconfig --name kv-eks-dev --region ap-south-1
terraform init

Initializing the backend...
Successfully configured the backend "s3"!
Initializing provider plugins...
- Installing hashicorp/aws v5.x.x...
- Installing hashicorp/kubernetes v2.x.x...
Terraform has been successfully initialized!

terraform plan -out=ebs.plan

Terraform will perform the following actions:

  # aws_iam_role.ebs_csi will be created
  # aws_iam_role_policy_attachment.ebs_csi will be created
  # aws_eks_addon.ebs_csi will be created
  + resource "aws_eks_addon" "ebs_csi" {
      + addon_name               = "aws-ebs-csi-driver"
      + addon_version            = "v1.35.0-eksbuild.1"
      + service_account_role_arn = (known after apply)
    }
  # kubernetes_storage_class.gp3 will be created
  + resource "kubernetes_storage_class" "gp3" {
      + storage_provisioner    = "ebs.csi.aws.com"
      + volume_binding_mode    = "WaitForFirstConsumer"
      + allow_volume_expansion = true
      + reclaim_policy         = "Delete"
    }

Plan: 4 to add, 0 to change, 0 to destroy.

terraform apply ebs.plan

aws_iam_role.ebs_csi: Creation complete after 3s
aws_iam_role_policy_attachment.ebs_csi: Creation complete after 1s
aws_eks_addon.ebs_csi: Still creating... [30s elapsed]
aws_eks_addon.ebs_csi: Creation complete after 48s
kubernetes_storage_class.gp3: Creation complete after 1s

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:
ebs_csi_addon_version = "v1.35.0-eksbuild.1"
default_storage_class = "gp3"

Step 6 — verify the driver, IRSA, and the default class

Confirm the controller is running, its ServiceAccount carries the role ARN (IRSA is wired), and gp3 is the sole default:

# Controller + node pods
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver
# ebs-csi-controller-...   6/6   Running
# ebs-csi-node-...         3/3   Running   (one per node, DaemonSet)

# IRSA annotation on the controller SA — proves the role is attached
kubectl get sa ebs-csi-controller-sa -n kube-system \
  -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'
# arn:aws:iam::<acct>:role/kv-eks-dev-ebs-csi-driver

# Strip the default flag off the legacy gp2 class so gp3 is the ONLY default
kubectl patch storageclass gp2 \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

kubectl get storageclass
# NAME            PROVISIONER       RECLAIMPOLICY  VOLUMEBINDINGMODE      ...
# gp2             ebs.csi.aws.com   Delete         WaitForFirstConsumer
# gp3 (default)   ebs.csi.aws.com   Delete         WaitForFirstConsumer

Step 7 — apply a StatefulSet that binds real EBS volumes

The workload is a separate concern from the platform, so we apply it with kubectl (an app team’s job) rather than folding it into the platform state. Save as workload.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: storage-demo
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: storage-demo
spec:
  serviceName: web
  replicas: 2
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: app
          image: public.ecr.aws/docker/library/busybox:1.36
          command: ["sh", "-c", "while true; do echo $(date) $(hostname) >> /data/log.txt; sleep 5; done"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests:
            storage: 10Gi

kubectl apply -f workload.yaml
kubectl get pv,pvc -n storage-demo

NAME                          CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM
persistentvolume/pvc-a1b2...  10Gi      RWO           Delete          Bound   storage-demo/data-web-0
persistentvolume/pvc-c3d4...  10Gi      RWO           Delete          Bound   storage-demo/data-web-1

NAME                                 STATUS  VOLUME        CAPACITY  STORAGECLASS
persistentvolumeclaim/data-web-0     Bound   pvc-a1b2...   10Gi      gp3
persistentvolumeclaim/data-web-1     Bound   pvc-c3d4...   10Gi      gp3

Two Bound PVCs, two dynamically provisioned PVs. Now prove the volumes are real EBS, encrypted, and in the pods’ AZs — this is the payoff:

# Pods land on nodes in (ideally) different AZs — WaitForFirstConsumer at work
kubectl get pods -n storage-demo -o wide

# The PV's node affinity shows the volume's AZ
kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}{"\n"}{end}'
# pvc-a1b2...  ap-south-1a
# pvc-c3d4...  ap-south-1b

# Ask EC2 directly — the driver tags volumes with the PVC name
aws ec2 describe-volumes \
  --filters "Name=tag:kubernetes.io/created-for/pvc/name,Values=data-web-0" \
  --query 'Volumes[].{ID:VolumeId,AZ:AvailabilityZone,Type:VolumeType,IOPS:Iops,Enc:Encrypted,Size:Size}' \
  --output table

-------------------------------------------------------------------------
|                           DescribeVolumes                             |
+------------+----------------+--------+-------+---------+------+--------+
|  AZ        |  ID            | Type   | IOPS  | Enc     | Size |        |
+------------+----------------+--------+-------+---------+------+--------+
| ap-south-1a| vol-0abc123... | gp3    | 3000  | True    | 10   |        |
+------------+----------------+--------+-------+---------+------+--------+

Finally, prove persistence survives a pod restart — the reattach that justifies the whole StatefulSet:

kubectl exec -n storage-demo web-0 -- tail -2 /data/log.txt   # note the last line
kubectl delete pod web-0 -n storage-demo                       # pod is recreated
kubectl exec -n storage-demo web-0 -- tail -4 /data/log.txt    # SAME file, continues

The recreated web-0 reattached data-web-0 — the same EBS volume, on a node in the same AZ — and the log continued. That is durable state. Run the full smoke test so you know every piece landed:

Step 8 — grow a volume online by editing the PVC

Because allow_volume_expansion = true, enlarging is a one-line PVC edit — no restart:

kubectl patch pvc data-web-0 -n storage-demo \
  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

kubectl get pvc data-web-0 -n storage-demo   # CAPACITY climbs to 20Gi
kubectl describe pvc data-web-0 -n storage-demo | grep -A2 Conditions
# FileSystemResizePending → then clears as the node driver grows the fs online

aws ec2 describe-volumes \
  --filters "Name=tag:kubernetes.io/created-for/pvc/name,Values=data-web-0" \
  --query 'Volumes[].Size'   # 20

Try to shrink it and Kubernetes refuses: spec.resources.requests.storage: Forbidden: field can not be less than previous value. Growth only.

Step 9 — take a CSI snapshot (controller + VolumeSnapshotClass)

⚠️ The EBS CSI add-on does not include the snapshot controller — that is a separate component and its absence is the #1 snapshot gotcha. Install the external snapshotter’s CRDs and controller once (Helm shown; kustomize also works), then a VolumeSnapshotClass:

# Snapshot CRDs + controller (once per cluster)
helm repo add piraeus-charts https://piraeus.io/helm-charts/
helm install snapshot-controller piraeus-charts/snapshot-controller \
  -n kube-system

# snapshotclass.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-vsc
driver: ebs.csi.aws.com
deletionPolicy: Delete
---
# snapshot.yaml — snapshot data-web-0
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: web-0-snap
  namespace: storage-demo
spec:
  volumeSnapshotClassName: ebs-vsc
  source:
    persistentVolumeClaimName: data-web-0

kubectl apply -f snapshotclass.yaml -f snapshot.yaml
kubectl get volumesnapshot -n storage-demo
# NAME         READYTOUSE   SOURCEPVC     RESTORESIZE
# web-0-snap   true         data-web-0    20Gi

To restore, create a new PVC whose dataSource points at the snapshot — the driver provisions a fresh EBS volume from the EBS snapshot:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-restored
  namespace: storage-demo
spec:
  storageClassName: gp3
  accessModes: ["ReadWriteOnce"]
  resources:
    requests: { storage: 20Gi } # >= snapshot restoreSize
  dataSource:
    name: web-0-snap
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Step 10 — destroy & clean up

⚠️ This is where people leak money. Deleting a StatefulSet does NOT delete its PVCs — volumeClaimTemplates PVCs are retained by design — and with reclaimPolicy: Delete the EBS volume only disappears when the PVC is deleted. So tear down in this order:

# 1. The workload objects
kubectl delete -f workload.yaml            # deletes the StatefulSet + namespace
# 2. The PVCs the StatefulSet left behind (this deletes the EBS volumes)
kubectl delete pvc -n storage-demo --all   # only if the ns still exists; else:
#    kubectl get pvc -A  → delete any leftover data-web-* explicitly
# 3. Snapshots (they persist and bill as EBS snapshots)
kubectl delete volumesnapshot -n storage-demo --all
# 4. The platform (add-on, IRSA role, StorageClass)
terraform destroy -auto-approve

kubernetes_storage_class.gp3: Destroying...
aws_eks_addon.ebs_csi: Destroying...
aws_iam_role_policy_attachment.ebs_csi: Destroying...
aws_iam_role.ebs_csi: Destroying...
Destroy complete! Resources: 4 destroyed.

Then confirm no orphaned volumes remain — the check that saves you a surprise bill:

aws ec2 describe-volumes \
  --filters "Name=tag:kubernetes.io/created-for/pvc/namespace,Values=storage-demo" \
  --query 'Volumes[].VolumeId'
# []   ← empty means clean. Any IDs here are orphans; delete them.

If you had used reclaimPolicy: Retain, those volumes would still be listed after the PVC delete, and you would remove them with aws ec2 delete-volume by hand — the price of keeping data safe.

Variables, outputs & making it reusable

The demo hard-codes one StorageClass, but a real platform offers a menu — a fast gp3 default, a high-IOPS io2 class for databases, maybe a cheap st1 class for logs. That is a textbook for_each over a map, the same technique the modules-authoring lesson teaches, applied to StorageClasses:

variable "storage_classes" {
  description = "Map of StorageClasses to create"
  type = map(object({
    type          = string
    iops          = optional(string)
    throughput    = optional(string)
    is_default    = optional(bool, false)
    reclaim       = optional(string, "Delete")
    expandable    = optional(bool, true)
    binding_mode  = optional(string, "WaitForFirstConsumer")
  }))
  default = {
    gp3 = { type = "gp3", iops = "3000", throughput = "125", is_default = true }
    io2 = { type = "io2", iops = "10000", reclaim = "Retain" }
    st1 = { type = "st1", binding_mode = "WaitForFirstConsumer" }
  }
}

resource "kubernetes_storage_class" "this" {
  for_each = var.storage_classes

  metadata {
    name = each.key
    annotations = {
      "storageclass.kubernetes.io/is-default-class" = tostring(each.value.is_default)
    }
  }

  storage_provisioner    = "ebs.csi.aws.com"
  volume_binding_mode    = each.value.binding_mode
  allow_volume_expansion = each.value.expandable
  reclaim_policy         = each.value.reclaim

  parameters = merge(
    { type = each.value.type, encrypted = "true" },
    each.value.iops == null ? {} : { iops = each.value.iops },
    each.value.throughput == null ? {} : { throughput = each.value.throughput },
  )

  depends_on = [aws_eks_addon.ebs_csi]
}

Adding a class is now a map entry, not a new resource block. Wrap the add-on, IRSA role, and this for_each into a module with clear inputs and you have a reusable “EBS storage layer” building block for every cluster in the fleet.

Should you roll your own or use a registry module? The community terraform-aws-modules/eks module and the AWS EKS Blueprints Addons module (aws-ia/eks-blueprints-addons/aws) both install the EBS CSI add-on and build its IRSA role for you from a single flag. The trade-off:

The honest recommendation: build it yourself once so you understand the add-on, the IRSA trust policy, and the StorageClass, then decide whether a Blueprints module’s convenience is worth the abstraction. You cannot debug a module that hides mechanics you have never seen. See Module Sources & Composition for pinning and consuming registry modules safely.

Common mistakes and troubleshooting

The failures below are the ones that actually page people. Scan the table, then read the prose on the five nastiest.

PVC Pending is almost always one of three things, and the order to check them is fixed: (1) Is the driver installed? kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver — no pods means no provisioner, so install the add-on. (2) Is IRSA wired? Check the SA annotation and the controller logs; AccessDenied/WebIdentityErr means the trust policy :sub does not match the controller ServiceAccount. (3) Is it just waiting for a pod? With WaitForFirstConsumer, a PVC with no consuming pod should sit Pending showing waiting for first consumer to be created before binding — that is not a bug, it is the design; it binds the moment a pod uses it.

The AZ trap is the one that looks like a scheduling bug and is really a physics fact. A pod that ran fine yesterday is Pending today with volume node affinity conflict, because its node died and the only spare capacity is in another AZ where its EBS volume cannot follow. There is no in-place fix — the volume is single-AZ. The real fix is architectural (per-AZ replicas + application-level replication, or EFS for shared data), and the prevention is WaitForFirstConsumer plus node groups in every AZ so the scheduler always has same-AZ capacity to fall back to.

Volume not expanding has two distinct causes people conflate. If the PVC edit is rejected, you tried to shrink — EBS only grows. If the edit is accepted but nothing happens, either allowVolumeExpansion was false on the class (mutable — flip it, then re-edit the PVC), or you are hitting the once-per-6-hours EBS ModifyVolume limit (AWS rate-limits volume modifications; wait and retry). Watch kubectl describe pvc for the FileSystemResizePending condition to clear.

The snapshot controller gap trips everyone once. The EBS CSI driver knows how to snapshot, but the Kubernetes snapshot controller and its CRDs (VolumeSnapshot, VolumeSnapshotContent, VolumeSnapshotClass) are a separate install that AWS deliberately leaves out of the add-on. Symptom: kubectl apply of a VolumeSnapshot fails no matches for kind "VolumeSnapshot" (CRDs missing), or the object is created but never becomes readyToUse (controller missing). Install the external snapshotter once per cluster.

gp2 vs gp3 is a slow-burn cost mistake. Clusters that predate gp3, or that never stripped the default gp2 class, keep provisioning gp2 volumes that cost more and cap performance at 3 IOPS/GiB. New workloads should default to gp3; existing gp2 volumes can be modified in place at the AWS level (aws ec2 modify-volume --volume-type gp3) without detaching, though Kubernetes still records the PV as gp2 — a cosmetic mismatch. The clean, K8s-native migration is snapshot-and-restore into a gp3 PVC:

Cost, cleanup & production notes

EBS bills on provisioned capacity, not usage — a 10 GiB volume with 1 GiB written costs the full 10 GiB — plus provisioned IOPS/throughput above the gp3 baseline, plus snapshot storage. Rough ap-south-1 figures for this demo, to make the “destroy it” case concrete:

The volumes are cheap; the trap is leaving them running — an orphaned 100 GiB io2 volume from a forgotten Retain PVC quietly bills every month with no pod attached. The single biggest hygiene lever is the post-destroy describe-volumes sweep from Step 10. Snapshots are incremental (only changed blocks bill) but accumulate; delete old ones. Five production-hardening notes beyond the demo:

1. Encrypt by default, and consider a CMK. encrypted = "true" on the class covers most needs with the AWS-managed aws/ebs key; a customer-managed key gives you key rotation control and its own audit trail — remember the extra KMS permissions on the IRSA role.
2. Retain for anything precious, and back it with snapshots. Reclaim Delete is fine for reproducible data; production databases want Retain plus scheduled snapshots (via a CronJob creating VolumeSnapshot objects, or AWS Backup) so a fat-fingered kubectl delete pvc is survivable.
3. Size for growth and turn on expansion. Start smaller with allowVolumeExpansion = true and grow on demand rather than over-provisioning; you pay for provisioned capacity whether or not it is used.
4. Keep the driver on IRSA, never node-role. Attaching AmazonEBSCSIDriverPolicy to the node role gives every pod EC2 volume permissions via IMDS. IRSA scopes it to the controller alone. Consider EKS Pod Identity as the newer alternative to IRSA for the same least-privilege outcome.
5. Pin and manage the add-on version; watch drift. Pin addon_version, use resolve_conflicts_on_update = "PRESERVE", and run scheduled terraform plan in CI so an out-of-band console upgrade or a hand-edited StorageClass shows up before it bites.

The EFS equivalent of this build swaps single-AZ block storage for a multi-AZ NFS file system with ReadWriteMany — same CSI/IRSA/StorageClass shape, opposite storage trade-off — and lives in the EKS EFS CSI lesson. The IRSA mechanism both drivers lean on is built once in the OIDC & IRSA lesson.

Cheat-sheet

Interview and exam questions

1. Walk through what happens from kubectl apply of a StatefulSet to a running pod with an EBS volume. The volumeClaimTemplates create a PVC per replica; with WaitForFirstConsumer the PVC stays Pending until the scheduler places the pod; the CSI controller then calls ec2:CreateVolume in the pod’s AZ, creates a matching PV with zone nodeAffinity, binds the PVC; the node DaemonSet attaches and formats the volume and mounts it into the pod.

2. Why does the EBS CSI driver need IRSA, and what breaks without it? The controller must call the EC2 API to create/attach/delete volumes; IRSA gives its ServiceAccount a scoped IAM role (AmazonEBSCSIDriverPolicy) via the cluster’s OIDC provider. Without it (or with a wrong trust policy) the controller gets AccessDenied/WebIdentityErr and every PVC hangs Pending. The alternative — node-role credentials — works but gives every pod on the node those permissions.

3. What does volumeBindingMode: WaitForFirstConsumer fix, and what is the symptom of getting it wrong? It delays volume provisioning until a pod is scheduled, so the volume is created in the pod’s AZ (honouring node selectors, taints, topology). With the default Immediate, the volume can be born in an AZ where the pod cannot run, and the pod hangs Pending with had volume node affinity conflict.

4. An EBS-backed pod won’t reschedule after its node fails. Why, and what are the design fixes? An EBS volume is single-AZ and ReadWriteOnce; its PV pins the pod to that AZ, so if the AZ has no capacity the pod stays Pending. Fixes are architectural: spread replicas one-per-AZ with application-level replication, or use EFS (RWX, multi-AZ) for shared data. You cannot move an EBS volume across AZs.

5. Why must the EBS CSI driver be installed even to use the old gp2 StorageClass on a current EKS cluster? The in-tree kubernetes.io/aws-ebs provisioner is deprecated and its code was removed upstream (~v1.27). CSI migration transparently routes the legacy gp2 class to ebs.csi.aws.com — but only if the CSI driver is installed. No driver, no dynamic provisioning, even for gp2.

6. How do you expand an EBS-backed PVC, and what are the two limits? Set allowVolumeExpansion = true on the StorageClass, then edit the PVC’s spec.resources.requests.storage upward; the driver grows the volume and filesystem online. Limits: you can never shrink, and EBS rate-limits ModifyVolume to roughly once per six hours per volume.

7. What is the difference between reclaimPolicy: Delete and Retain, and which does a production database want? Delete destroys the EBS volume when the PVC is deleted; Retain keeps it (leaving a Released PV) so you reclaim it by hand. A production database wants Retain so an accidental PVC delete does not wipe data — accepting that you must clean up orphaned volumes manually.

8. Your VolumeSnapshot is created but never becomes readyToUse. What’s missing? The external snapshot controller and its CRDs. The EBS CSI add-on does not bundle them; install the snapshotter (Helm or kustomize) once per cluster, then define a VolumeSnapshotClass with driver: ebs.csi.aws.com.

9. Why do EBS access modes matter, and what happens to a ReadWriteMany PVC on a gp3 class? EBS supports only ReadWriteOnce / ReadWriteOncePod — one node/pod at a time. A ReadWriteMany PVC against an EBS class hangs Pending forever because no EBS volume can satisfy it; RWX needs EFS.

10. (Terraform Associate) Why configure the kubernetes provider from data.aws_eks_cluster data sources instead of building the cluster in the same config? Because a provider configured from a resource created in the same apply depends on values unknown until apply, which is fragile and can break plan/destroy. Reading an existing cluster via data sources means the endpoint, CA, and token are known at plan time — the safe pattern for layering platform add-ons onto a pre-built cluster.

11. (Terraform Associate) You changed resolve_conflicts on aws_eks_addon and got a deprecation error. What’s the fix? Replace the single resolve_conflicts argument with the pair resolve_conflicts_on_create and resolve_conflicts_on_update (values OVERWRITE/NONE, plus PRESERVE on update to keep field edits through upgrades).

12. After terraform destroy the plan is clean but EBS volumes still exist. Why, and how do you prevent the leak? StatefulSet volumeClaimTemplates PVCs are retained when the StatefulSet is deleted, and Retain-policy volumes survive PVC deletion — so Terraform (which never owned the PVCs) leaves them. Delete PVCs explicitly and sweep with aws ec2 describe-volumes filtered on the PVC-namespace tag; delete any orphans.

Key takeaways

• A fresh EKS cluster has no durable storage until you install a CSI driver. For single-writer block storage the answer is the EBS CSI driver, installed as a managed EKS add-on and permissioned with IRSA — never node-role credentials.
• IRSA is the security control of the whole lesson. The controller ServiceAccount ebs-csi-controller-sa federates through the cluster’s OIDC issuer to an IAM role carrying AmazonEBSCSIDriverPolicy; a wrong trust-policy :sub is the most common cause of PVCs stuck Pending.
• volumeBindingMode: WaitForFirstConsumer is non-negotiable on a multi-AZ cluster — it provisions the volume in the pod’s AZ instead of blind, avoiding volume node affinity conflict.
• An EBS volume is one-AZ, ReadWriteOnce, and pins its pod to that AZ. That single physical fact drives stateful design: replicas per-AZ plus application replication, or EFS for shared multi-AZ RWX storage.
• gp3 is the default choice — decoupled IOPS/throughput, ~20% cheaper than gp2 — set via StorageClass parameters (all stringly typed), with encrypted = "true" and allow_volume_expansion = true.
• Reclaim policy decides data fate. Delete for reproducible dev data, Retain for production — and remember StatefulSet PVCs and Retain volumes survive terraform destroy, so sweep for orphans or pay for them.
• Snapshots need a separate controller. The add-on ships the driver, not the snapshot controller/CRDs; install the external snapshotter, then use VolumeSnapshotClass/VolumeSnapshot and restore via a PVC dataSource.