Terraform on AWS EKS: Assembling a Production Platform — the End-to-End Capstone

Every EKS lesson in this track taught one thing in isolation — how to provision the cluster, how to wire the Kubernetes and Helm providers, how to autoscale with Karpenter, how to isolate environments. Each was a demo you could run. This lesson is the moment they stop being demos and become a platform: one coherent, production-shaped estate where the pieces are composed in the right order, in the right state boundaries, with the security and reliability that lets an on-call engineer sleep. It is the capstone, and its real subject is not any single resource but the layering and repository structure that keeps thirty-odd moving parts maintainable when three teams are changing them at once.

There is a failure mode this lesson exists to prevent, and it is worth naming up front because you will recognise it. A team stands up EKS with the community module, feels the rush of kubectl get nodes returning, and then — in the same Terraform configuration — bolts on the Load Balancer Controller, ExternalDNS, the CSI drivers, Prometheus and Argo CD with a pile of helm_release blocks. It works on the first apply on a good day. Then the day comes when the cluster must be recreated, or the endpoint changes, or someone runs destroy, and the whole thing detonates: the Helm provider can’t reach a cluster that doesn’t exist yet, a helm_release refuses to plan because its provider configuration is “unknown”, and the state is a tangle where a routine add-on bump proposes to touch the VPC. Nothing in the HCL was wrong. What was missing was structure — separate state per layer, a deliberate apply order, IRSA instead of node permissions, and a clean seam between what Terraform owns and what GitOps owns.

You already have the parts. Here we assemble the reference production EKS platform from them and, just as importantly, lay out the live/{env}/{network,cluster,platform,apps} repository that makes it survivable. Read the prose once for the reasoning; the tables — the platform-capability catalogue, the state-layering map, the apply-order and upgrade-order tables, the per-environment sizing grid, the SRE/security-controls catalogue, and the operations troubleshooting map — are the reference you keep open while you build. By the end you will be able to stand up an EKS platform that an auditor, an SRE and a new hire can all read, that upgrades without drama, and whose blast radius is designed into the state keys.

What you’ll build

The running example is Kestrel, a mid-size SaaS company standing up its first governed production platform on EKS in ap-south-1 (Mumbai) to run a customer-facing product plus a fleet of internal services. Kestrel’s platform team owns the cluster and the add-ons; product teams own the apps that run on top. We build that split as four state layers, applied left to right, each with its own remote state:

• network — a tagged multi-AZ VPC (public, private and intra subnets across three Availability Zones), NAT for egress, and the subnet tags EKS, the Load Balancer Controller and Karpenter all discover by. It changes rarely and underpins everything.
• cluster — the EKS control plane (private API endpoint, KMS envelope encryption for Secrets, control-plane audit logs, access entries instead of aws-auth surgery), a small managed node group for system add-ons, plus Karpenter’s cluster-scoped IAM (node role, instance profile, interruption queue). It changes on the upgrade cadence.
• platform — the add-ons layer, the heart of this lesson: AWS Load Balancer Controller, ExternalDNS, metrics-server, the EBS and EFS CSI drivers, CloudWatch Observability (or kube-prometheus-stack), External Secrets Operator, Karpenter’s Helm release and NodePools, and Argo CD — every controller delivered by Helm with its own IRSA role and no permissions on the node. It changes weekly.
• apps — application delivery, handed to Argo CD via an app-of-apps that syncs from Git. Terraform installs Argo CD; Argo CD runs everything above it.

The architecture in words, read as a request travels it: a user hits app.kestrel.example, Route 53 resolves it — the record was created by ExternalDNS watching an Ingress — to an Application Load Balancer that the AWS Load Balancer Controller provisioned from that same Ingress. The ALB forwards to pods on Karpenter-provisioned worker nodes (mostly spot, right-sized, consolidated when idle) in the private subnets. Those pods reach AWS APIs — S3, Secrets Manager, Route 53 — not through a node role or a static key but through IRSA: each service account assumes its own narrowly-scoped IAM role via the cluster’s OIDC provider. Secrets arrive through the External Secrets Operator pulling from Secrets Manager; storage is EBS (gp3) and EFS through the CSI drivers; metrics flow to Prometheus/CloudWatch; and the whole platform is upgraded control-plane-first on a schedule and backed up by Velero. That is the shape, and it is the same shape at three nodes or three hundred.

Why Terraform for all of this rather than eksctl, a stack of kubectl apply, or clicking through the console? Because a platform’s whole value is that it is reproducible, reviewable and enforced. eksctl is excellent for a quick cluster but stops at the cluster edge and keeps no desired-state model of the add-ons or the IAM. A pile of kubectl and aws commands is a script with no notion of drift. The console produces snowflakes nobody can recreate. Terraform’s provider model lets the same tool and the same state discipline govern the AWS resources (VPC, EKS, IAM, KMS), the in-cluster resources (via the kubernetes and helm providers), and the DNS zone together — which is exactly what a platform team needs. And Terraform’s module + remote-state model is the cleanest expression of the two things a platform must have: reuse (one module, many environments) and isolation (one state per layer per environment).

Here is the leap this lesson is about — the difference between “a cluster with some Helm on it” and a platform, stated as symptoms you can recognise on your own estate:

By the end you can build every row of the right-hand column with real HCL.

Learning objectives

By the end of this lesson you will be able to:

• Compose the reference EKS platform — tagged VPC → EKS (private endpoint, KMS, control-plane logs, access entries) → managed system node group + Karpenter → the platform add-ons → Argo CD — and reason about the request path and the identity path.
• Lay out the platform as separate state per layer (network → cluster → platform → apps), wire the layers with terraform_remote_state or Terragrunt dependency, and justify the split by blast radius, the provider-ordering problem, and team ownership.
• State and enforce the apply order (and its reverse, the destroy order), and the upgrade order with version-skew rules.
• Deliver every add-on via Helm with its own IRSA role — Load Balancer Controller, ExternalDNS, metrics-server, EBS/EFS CSI, observability, External Secrets Operator, Argo CD — with no node-role permissions and no static keys.
• Size the platform per environment (dev/staging/prod) — spot ratio, replicas, endpoint exposure, retention, budgets — with per-env state.
• Bake in security and SRE — private endpoints, KMS envelope + Secrets encryption, network policies, Pod Security Standards, cost via spot + consolidation + right-sizing, observability and alerts, cluster upgrades in order, Velero backup/DR, and an OIDC CI gate.
• Read a module map and make build-vs-buy calls — which terraform-aws-modules/* and which community Helm charts to adopt, and where to own the code.

Prerequisites & where this fits

This is the EKS capstone of the Terraform Zero-to-Hero course. It assumes the EKS building blocks are already familiar and pulls them into one estate. You will get the most from it having already provisioned an EKS cluster with its VPC and node groups, wired the Kubernetes and Helm providers to deploy apps and GitOps, set up the Cluster Autoscaler and Karpenter, driven a multi-environment estate with Terragrunt and approval gates, and built the AWS 3-tier platform with reusable modules, SRE and remote state. Where those lessons each teach one plane, this one shows all of them driven by a single production EKS platform at once.

A note on versions: everything targets Terraform ≥ 1.6 (the 1.9/1.10 line current in 2026) with the aws provider ~> 5.0, the kubernetes provider ~> 2.35, the helm provider ~> 2.17, the terraform-aws-modules/eks/aws module ~> 20.0 (the v20 line uses access entries and drops aws-auth by default), and the terraform-aws-modules/vpc/aws module ~> 5.8. OpenTofu is a drop-in for the CLI throughout; the module and state model are identical. Assume you have aws credentials working (SSO or an assumed role) in an account where you can create a VPC, an EKS cluster, IAM roles and KMS keys, and that kubectl and the aws CLI (for aws eks get-token) are on your PATH.

Because this platform pins those providers and uses the S3 backend, the handful of choices that bite an upgrader or a first-timer are worth having in front of you — every one is reflected in the HCL below:

Here is the map of companion lessons and what each carries, so you know where to go deep on any one plane:

The reference EKS platform: the “50-demo” synthesis

The signature of this capstone is that it is a synthesis: nearly every capability you built in a standalone lesson has a home in this one platform, delivered by a specific module or Helm chart, in a specific layer. Before any HCL, here is the catalogue — the roughly three dozen platform capabilities, which prior lesson taught each, and the module or chart that delivers it. This is the table you scan to answer “where does that live?” and it is the real table of contents for the build:

That is the “50-demo” made concrete: no capability is invented here; each is a lesson you already ran, now placed in a layer and wired to the rest. The single most important column is Layer, because the layer is the state boundary, and the state boundary is what makes the platform maintainable.

The diagram traces the whole platform left to right: the four Terraform/Terragrunt state layers on the left provision, in dependency order, the tagged VPC, the EKS cluster with its system nodes and Karpenter, the platform add-ons (every one under an IRSA role, shipped by Helm, observed by Prometheus/CloudWatch), and finally Argo CD syncing the apps with Route 53 records from ExternalDNS. The six badges are the six load-bearing decisions of the lesson: state per layer, IRSA everywhere, Karpenter + spot, the separate platform add-ons layer (the provider-ordering fix), GitOps for apps, and the fixed upgrade order.

The request path and the identity path are the two mental models to hold. The request path is how traffic reaches a pod; the identity path is how a pod reaches AWS — and the second is the one juniors skip, then spend a day debugging an AccessDenied:

State layering: network → cluster → platform → apps

The repository is the architecture, and on EKS the layering is not a nicety — it is forced on you by a hard technical constraint, then justified three more ways. Get the layers right and the platform is maintainable; get them wrong and you meet the detonation from the opening.

The provider-ordering problem — the reason the layers are not optional. Terraform resolves provider configuration at plan time, before most resources exist. The kubernetes and helm providers need the cluster’s API endpoint and CA to talk to it. If you configure those providers from module.eks.cluster_endpoint and create helm_release resources in the same configuration, you have asked a provider to depend on a resource being created in its own apply. On a clean first apply this is “unknown at plan time”; on a rebuild or a destroy it fails outright, because the provider can’t reach a cluster that is gone or not yet there. The robust fix is structural: the cluster layer creates the cluster and outputs its identity; the platform layer reads an already-created cluster via data "aws_eks_cluster" and only then configures the kubernetes/helm providers. A provider must point at infrastructure that already exists in another state — that single rule is why cluster and platform are separate layers.

The other three justifications reinforce it:

Here is the canonical layout for the Kestrel platform. Read it by state boundaries, because that is what matters at 2 a.m. — every leaf under live/ is a root module with its own state key, and that key is the unit of plan, apply and blast radius:

kestrel-platform/
├── modules/                         # the library — reusable, versioned, never applied directly
│   ├── eks-platform-addons/         # composes IRSA + helm for all add-ons (the star of this lesson)
│   │   ├── main.tf                  # for_each over add-ons: irsa role + helm_release
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   └── karpenter-nodepool/          # EC2NodeClass + NodePool manifests
│
├── live/                            # per-env ROOT configs — each LAYER has its own state key
│   ├── dev/
│   │   ├── network/                 # key: dev/network/terraform.tfstate   (vpc)
│   │   ├── cluster/                 # key: dev/cluster/terraform.tfstate    (eks + system NG + karpenter IAM)
│   │   ├── platform/                # key: dev/platform/terraform.tfstate   (IRSA + helm add-ons + argocd)
│   │   └── apps/                    # key: dev/apps/terraform.tfstate       (argocd Application → Git)
│   ├── staging/                     # identical structure, staging inputs
│   └── prod/
│       ├── network/
│       ├── cluster/
│       │   ├── versions.tf          # required_providers + backend "s3" {}
│       │   ├── main.tf              # module "eks" + module "karpenter" + access entries
│       │   ├── variables.tf
│       │   ├── outputs.tf           # cluster_name, endpoint, oidc_provider_arn, karpenter_*
│       │   ├── prod.auto.tfvars
│       │   └── backend.hcl          # key = "prod/cluster/terraform.tfstate"
│       ├── platform/
│       │   ├── versions.tf          # aws + kubernetes + helm providers
│       │   ├── main.tf              # data.terraform_remote_state.cluster → providers → add-ons
│       │   └── backend.hcl          # key = "prod/platform/terraform.tfstate"
│       └── apps/
│
├── gitops/                          # the manifests repo Argo CD syncs (apps live HERE, not in Terraform)
├── tests/                           # native terraform test (*.tftest.hcl)
├── .tflint.hcl
└── .checkov.yaml

The one rule that saves you: never terraform apply inside modules/ — modules are consumed by source, not run. And the second: apps live in gitops/, not in Terraform — Terraform installs Argo CD; Argo CD syncs the app manifests. The state-layering table is the reference:

Wiring the layers. The lower layer reads the upper layer’s outputs. In native Terraform that is a terraform_remote_state data source; in Terragrunt it is a first-class dependency block with mock_outputs so a plan runs before the dependency exists. The cross-layer wiring map:

The apply order is fixed — and its reverse is the destroy order. You cannot apply platform before cluster (no cluster to talk to) or cluster before network (no VPC to place it in). ⚠️ Getting the destroy order wrong is worse than getting apply wrong: destroy network before platform and you strand ENIs, load balancers and security groups that the add-ons created, and the VPC delete hangs for an hour.

In Terragrunt, terragrunt run-all apply walks this graph for you from the dependency edges; in native Terraform you apply the folders in order (a thin wrapper script, or the CI pipeline, enforces it). Either way, the order is a property of the dependency graph, not a thing you remember — which is the whole point of encoding it in state layers.

The cluster layer: VPC + EKS + system nodes + Karpenter IAM

The cluster layer reads the network layer and stands up the control plane, the small system node group, and Karpenter’s cluster-scoped IAM. It is deliberately thin on Helm — nothing in this layer uses the kubernetes or helm provider, precisely so that the cluster can always be created and destroyed cleanly.

Start with the provider and the read of the network layer:

# live/prod/cluster/versions.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
  backend "s3" {}   # completed at init: key = prod/cluster/terraform.tfstate
}

provider "aws" {
  region = var.region
  default_tags { tags = local.common_tags }   # every AWS resource inherits these
}

data "terraform_remote_state" "network" {
  backend = "s3"
  config  = { bucket = "kestrel-tfstate-apsouth1", key = "prod/network/terraform.tfstate", region = "ap-south-1" }
}

Now the EKS cluster itself, via the community module — private endpoint, KMS envelope encryption, control-plane logs, access entries, and a small managed node group tainted for system add-ons only:

# live/prod/cluster/main.tf
locals { cluster_name = "eks-${var.workload}-${var.environment}" }   # eks-kestrel-prod

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = local.cluster_name
  cluster_version = var.cluster_version                 # e.g. "1.30" — bumped on upgrades

  # Private API endpoint; public reachable only from office/CI CIDRs (tighten to false in prod-locked)
  cluster_endpoint_private_access      = true
  cluster_endpoint_public_access       = var.endpoint_public_access
  cluster_endpoint_public_access_cidrs = var.api_allowed_cidrs

  # Control-plane audit + component logs to CloudWatch
  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  # KMS envelope encryption for Kubernetes Secrets (encryption at rest, above EBS-level)
  create_kms_key            = true
  cluster_encryption_config = { resources = ["secrets"] }

  # Access entries API — no aws-auth ConfigMap surgery
  authentication_mode                      = "API_AND_CONFIG_MAP"
  enable_cluster_creator_admin_permissions = true

  # Core add-ons managed by EKS (kept in the CLUSTER layer, before any workload)
  cluster_addons = {
    coredns                = { most_recent = true }
    kube-proxy             = { most_recent = true }
    vpc-cni                = { most_recent = true }
    eks-pod-identity-agent = { most_recent = true }
  }

  vpc_id     = data.terraform_remote_state.network.outputs.vpc_id
  subnet_ids = data.terraform_remote_state.network.outputs.private_subnets

  # A SMALL managed node group for system add-ons only; workloads come from Karpenter
  eks_managed_node_groups = {
    system = {
      instance_types = var.system_instance_types      # ["m6i.large"]
      capacity_type  = "ON_DEMAND"                     # system add-ons on stable capacity
      min_size       = 2
      max_size       = 3
      desired_size   = 2
      labels = { role = "system" }
      taints = {
        addons = { key = "CriticalAddonsOnly", value = "true", effect = "NO_SCHEDULE" }
      }
    }
  }

  tags = local.common_tags
}

Two design decisions here are worth their own sentence. Why a managed system node group and Karpenter? Karpenter itself, CoreDNS and the CSI drivers need somewhere to run before Karpenter can provision anything — a chicken-and-egg. So a tiny, boring, on-demand managed group runs the system add-ons (tainted CriticalAddonsOnly so app pods don’t land on it), and Karpenter provisions everything else. Why access entries? Because the old aws-auth ConfigMap was an un-versioned, easy-to-lock-yourself-out-of blob; authentication_mode + aws_eks_access_entry makes cluster access a first-class, Terraform-managed IAM object.

An access entry granting a platform-admin SSO role cluster-admin, the modern replacement for an aws-auth line:

resource "aws_eks_access_entry" "platform_admins" {
  cluster_name  = module.eks.cluster_name
  principal_arn = var.platform_admin_role_arn      # e.g. an SSO permission-set role
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "platform_admins" {
  cluster_name  = module.eks.cluster_name
  principal_arn = var.platform_admin_role_arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
  access_scope { type = "cluster" }
}

Karpenter’s cluster-scoped IAM — the node role, instance profile, interruption SQS queue and controller IRSA policy — belongs in the cluster layer (it is IAM, not Helm). The community submodule builds it in one block; its outputs feed the platform layer’s Helm install:

module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> 20.0"

  cluster_name          = module.eks.cluster_name
  enable_v1_permissions = true
  namespace             = "kube-system"
  node_iam_role_name    = "karpenter-node-${local.cluster_name}"

  # Let the controller assume its role via Pod Identity (or set up IRSA — either works)
  create_pod_identity_association = true
  tags                            = local.common_tags
}

The cluster layer’s outputs are the contract the platform layer consumes. Emit exactly what the providers, the IRSA trust and Karpenter need:

# live/prod/cluster/outputs.tf
output "cluster_name"      { value = module.eks.cluster_name }
output "cluster_endpoint"  { value = module.eks.cluster_endpoint }
output "cluster_ca"        { value = module.eks.cluster_certificate_authority_data }
output "cluster_version"   { value = module.eks.cluster_version }
output "oidc_provider_arn" { value = module.eks.oidc_provider_arn }   # ← IRSA trust anchor
output "node_security_group_id" { value = module.eks.node_security_group_id }

output "karpenter_queue_name"   { value = module.karpenter.queue_name }
output "karpenter_node_role"    { value = module.karpenter.node_iam_role_name }
output "karpenter_irsa_arn"     { value = module.karpenter.iam_role_arn }

The EKS cluster options that matter for a production cluster, and what each buys you:

The platform add-ons layer: IRSA + Helm, the star of the show

This is the heart of the capstone. The platform layer reads the already-created cluster, configures the kubernetes and helm providers against it, and installs every add-on — each with its own IRSA role and no permission on the node. First the providers, authenticated by exec so no token is ever written to state:

# live/prod/platform/versions.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws        = { source = "hashicorp/aws",        version = "~> 5.0" }
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.35" }
    helm       = { source = "hashicorp/helm",       version = "~> 2.17" }
  }
  backend "s3" {}   # key = prod/platform/terraform.tfstate
}

data "terraform_remote_state" "cluster" {
  backend = "s3"
  config  = { bucket = "kestrel-tfstate-apsouth1", key = "prod/cluster/terraform.tfstate", region = "ap-south-1" }
}

# Read the LIVE cluster the cluster layer created — never create it here
data "aws_eks_cluster" "this" {
  name = data.terraform_remote_state.cluster.outputs.cluster_name
}

provider "aws" { region = var.region }

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name]
  }
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name]
    }
  }
}

The IRSA + Helm pattern, once. Every add-on is the same two-part shape: an IAM role scoped to exactly what that controller needs, trusted by the cluster’s OIDC provider for one namespace/service-account pair, and a helm_release that annotates its service account with the role ARN. The terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks module ships the canonical policy for each well-known controller behind a boolean, so you don’t hand-write the JSON. Here is the AWS Load Balancer Controller in full:

# live/prod/platform/main.tf
locals {
  oidc_provider_arn = data.terraform_remote_state.cluster.outputs.oidc_provider_arn
  cluster_name      = data.aws_eks_cluster.this.name
}

module "irsa_lb_controller" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name                              = "irsa-lbc-${local.cluster_name}"
  attach_load_balancer_controller_policy = true          # canonical AWS LBC policy, maintained upstream
  oidc_providers = {
    main = {
      provider_arn               = local.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-load-balancer-controller"]
    }
  }
}

resource "helm_release" "aws_lb_controller" {
  name       = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-load-balancer-controller"
  version    = var.lbc_chart_version           # e.g. "1.8.1"
  namespace  = "kube-system"

  set{
    name = "clusterName"
    value = local.cluster_name
  }
  set{
    name = "serviceAccount.create"
    value = "true"
  }
  set{
    name = "serviceAccount.name"
    value = "aws-load-balancer-controller"
  }
  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.irsa_lb_controller.iam_role_arn      # ← the IRSA wire-up
  }
}

ExternalDNS is the same shape, scoped to just the one hosted zone it may edit:

module "irsa_external_dns" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name                     = "irsa-extdns-${local.cluster_name}"
  attach_external_dns_policy    = true
  external_dns_hosted_zone_arns = [var.hosted_zone_arn]     # ← least-privilege: one zone
  oidc_providers = {
    main = { provider_arn = local.oidc_provider_arn, namespace_service_accounts = ["kube-system:external-dns"] }
  }
}

resource "helm_release" "external_dns" {
  name       = "external-dns"
  repository = "https://kubernetes-sigs.github.io/external-dns/"
  chart      = "external-dns"
  version    = var.external_dns_chart_version
  namespace  = "kube-system"

  set{
    name = "provider"
    value = "aws"
  }
  set {                                                   # create AND delete records it owns
    name  = "policy"
    value = "sync"
  }
  set{
    name = "domainFilters[0]"
    value = var.public_domain
  }
  set{
    name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.irsa_external_dns.iam_role_arn
  }
}

Eight add-ons written out longhand is eight near-identical blocks — the thing modules exist to kill. In a real library you extract the shape into a small eks-platform-addons module driven by a for_each map, so the root declares add-ons as data and the module builds the IRSA role + helm_release for each. That is the reusability payoff, and the module registry equivalent is exactly the iam-role-for-service-accounts-eks submodule you already saw. The full add-on catalogue — what each does, the chart, and the IRSA policy it needs — is the reference you build the map from:

The IRSA wire-up is the single most important security property of the platform, so state it as a rule: one role per controller, trusted for one service account, granting exactly that controller’s permissions. The node role, by contrast, holds almost nothing — just the managed policies EKS requires (AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, ECR read). The IRSA-per-add-on catalogue:

IRSA vs Pod Identity. IRSA (the OIDC-federation model) is the mature, universal path and what the iam-role-for-service-accounts-eks module builds. EKS Pod Identity (an agent add-on + an association API) is the newer, simpler model — no OIDC provider per cluster, no SA annotation, an association object instead. Use IRSA when a chart hard-codes the annotation or you need cross-account; prefer Pod Identity for new, in-account add-ons where you control the SA. Karpenter above uses Pod Identity via create_pod_identity_association; the rest use IRSA — mixing is normal, and the platform layer is where both live.

Argo CD and the Terraform/GitOps seam. Terraform installs Argo CD as one more Helm release; from there, Argo CD owns application delivery. The boundary is deliberate and it is the answer to “why aren’t the apps in Terraform?”: infrastructure changes go through plan/apply with an IAM audit trail; application changes go through a PR to the manifests repo with Argo CD’s sync and rollback. The seam is drawn at the platform/app line so Terraform and kubectl never fight over the same Deployment’s replica count.

resource "helm_release" "argocd" {
  name             = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = var.argocd_chart_version
  namespace        = "argocd"
  create_namespace = true
  values           = [yamlencode({ server = { ingress = { enabled = true } } })]
}

# The ONE app-of-apps that points Argo CD at the Git repo; everything else syncs from there
resource "kubernetes_manifest" "root_app" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata   = { name = "root", namespace = "argocd" }
    spec = {
      project     = "default"
      source      = { repoURL = var.gitops_repo, path = "envs/${var.environment}", targetRevision = "main" }
      destination = { server = "https://kubernetes.default.svc", namespace = "argocd" }
      syncPolicy  = { automated = { prune = true, selfHeal = true } }
    }
  }
  depends_on = [helm_release.argocd]
}

Multi-environment: dev, staging, prod

The layers are identical across environments; only the inputs differ. dev runs cheap and spot-heavy with the API endpoint open to the office CIDR; prod runs prod-sized, endpoint-locked, Multi-add-on-replica, and Velero-backed. Each environment has its own state key per layer (the <env>/ prefix), so a dev apply is physically incapable of touching prod. The per-environment sizing that carries every difference — the only thing that changes, because the module and add-on code is identical:

Promotion walks a change dev → staging → prod: prove the add-on chart bump or NodePool change in dev, run the same code against staging with staging inputs, then prod behind an approval. Nothing is hand-edited between environments; only the sizing variables change. This is the promotion discipline from the multi-environment Terragrunt lesson applied to a whole platform rather than a single stack:

Security & SRE as code

Reliability and security are properties of the platform, guaranteed by the layers and the IRSA discipline, not bolted on after the first incident. The controls, densest part of the lesson, each a small standard pattern applied everywhere:

Four of these deserve prose because they are where EKS platforms actually fail.

Pod Security Standards and network policy are namespace-level, and belong to apps, not the platform. Terraform (or Argo CD) labels each app namespace pod-security.kubernetes.io/enforce: restricted so a pod that runs as root or asks for privilege is rejected at admission; a default-deny NetworkPolicy per namespace forces east-west traffic to be allow-listed. Start in audit mode in dev (label warn/audit) and flip to enforce in staging/prod so you learn what breaks before it blocks a deploy.

Cost is a first-class control, and Karpenter is the lever. The largest EKS surprises are (1) the flat control-plane fee (~₹6,000/month per cluster, unavoidable — fold small teams into fewer clusters with namespaces, not a cluster each), (2) idle on-demand nodes, and (3) NAT and cross-AZ transfer. Karpenter attacks the second: a broad instance set, spot preferred with on-demand fallback, and consolidationPolicy: WhenEmptyOrUnderutilized so under-used nodes are drained and replaced with cheaper/smaller ones. Set a NodePool CPU limit so a runaway workload can’t provision unbounded capacity, and watch it with a budget alarm.

Upgrades are ordered, and the order is not negotiable. Upgrade the control plane first, then the managed node group and Karpenter nodes, then the add-ons — never nodes ahead of the API server. Kubelet may lag the control plane (EKS tolerates several minors behind on the extended path) but must never lead it, and you upgrade one minor at a time — no skipping. After the control plane bump, roll nodes (a new AMI/launch template triggers the managed-NG roll; Karpenter drift-replaces its nodes), then match each add-on’s chart/version to the new cluster version.

Backup and DR is Velero, and it needs its own IRSA. Velero backs up namespaced objects and, via the CSI/EBS integration, the persistent volumes; schedules push to an S3 bucket (copied cross-region for DR), and its IRSA role grants S3 read/write plus EBS snapshot. Test the restore, not just the backup — a backup you have never restored is a hope, not a plan.

The CI/CD gate that ties it together is the OIDC pipeline (covered in the GitHub Actions OIDC lesson): a PR runs fmt/validate/tflint/checkov and a plan per changed layer under a read-only role assumed via GitHub OIDC (no static keys); a merge assumes a scoped deploy role — one per layer, able to touch only that layer’s state key prefix — and applies the approved plan in apply order. Reference it as the enforcement plane; the platform above is what it enforces.

Hands-on: assemble the platform

Now assemble the platform end to end — the four layers, in order, with the cross-layer reads, the IRSA add-ons and Argo CD. ⚠️ This creates real, billable AWS resources — an EKS control plane (~₹6,000/month, billed from creation), NAT gateways, EC2 worker nodes, EBS volumes and one or more ALBs. Destroy in reverse order at the end.

Step 0 — one-time state backend (bootstrap). The S3 bucket and lock table must exist before any layer can init. Create them once by hand:

aws s3api create-bucket --bucket kestrel-tfstate-apsouth1 \
  --region ap-south-1 --create-bucket-configuration LocationConstraint=ap-south-1
aws s3api put-bucket-versioning --bucket kestrel-tfstate-apsouth1 \
  --versioning-configuration Status=Enabled
aws dynamodb create-table --table-name kestrel-tf-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST --region ap-south-1

Step 1 — apply network. The VPC layer stands up subnets across three AZs with the discovery tags EKS, the LB Controller and Karpenter need (kubernetes.io/role/elb on public, kubernetes.io/role/internal-elb on private, karpenter.sh/discovery = <cluster> on private):

cd live/prod/network
terraform init -backend-config=backend.hcl     # key = prod/network/terraform.tfstate
terraform apply

Step 2 — apply cluster. ⚠️ This creates the EKS control plane (billing starts now) and takes ~12–15 minutes:

cd ../cluster
terraform init -backend-config=backend.hcl     # key = prod/cluster/terraform.tfstate
terraform plan  -out=cluster.tfplan
terraform apply cluster.tfplan
aws eks update-kubeconfig --name eks-kestrel-prod --region ap-south-1   # for verification

Step 3 — apply platform. This is the layer that would have failed had you folded it into cluster: it reads the live cluster via the data source, configures the kubernetes/helm providers, and installs every add-on with its IRSA role. Representative plan output — note the IRSA modules and helm_releases, and that Terraform reads the cluster (not creates it):

cd ../platform
terraform init -backend-config=backend.hcl     # key = prod/platform/terraform.tfstate
terraform plan  -out=platform.tfplan
terraform apply platform.tfplan

data.terraform_remote_state.cluster: Reading...
data.aws_eks_cluster.this: Reading...

Terraform will perform the following actions:
  # module.irsa_lb_controller.aws_iam_role.this[0] will be created
  # module.irsa_external_dns.aws_iam_role.this[0] will be created
  # helm_release.aws_lb_controller will be created
  # helm_release.external_dns will be created
  # helm_release.metrics_server will be created
  # helm_release.karpenter will be created
  # helm_release.external_secrets will be created
  # helm_release.argocd will be created
  # kubernetes_manifest.root_app will be created

Plan: 18 to add, 0 to change, 0 to destroy.

Step 4 — verify the platform properties, not just that pods are running: that IRSA is wired (a pod assumes its role, not the node role), the LB Controller is up, and ExternalDNS owns records:

# Every controller Running, each with its IRSA-annotated service account
kubectl get pods -n kube-system | egrep 'aws-load-balancer|external-dns|karpenter|metrics-server'

# IRSA proof: the SA carries the role-arn annotation
kubectl get sa aws-load-balancer-controller -n kube-system \
  -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}{"\n"}'

# Argo CD came up and the app-of-apps is syncing
kubectl get applications -n argocd

Step 5 — hand off to Argo CD (the apps layer). The apps layer is thin: it is the app-of-apps Application pointing at the Git repo; from there Argo CD syncs the actual workloads. You do not helm_release each app in Terraform — that is the seam.

Step 6 — destroy in reverse. ⚠️ Destroy apps → platform → cluster → network. Destroying network first strands ALBs/ENIs the add-ons created and the VPC delete hangs:

cd ../platform && terraform destroy    # removes helm releases + IRSA roles first
cd ../cluster  && terraform destroy    # then the cluster (stops the control-plane billing)
cd ../network  && terraform destroy    # finally the VPC
# only if fully finished with the backend:
aws s3 rb s3://kestrel-tfstate-apsouth1 --force
aws dynamodb delete-table --table-name kestrel-tf-locks --region ap-south-1

Variables, outputs & making it reusable: the module map

The whole lesson is about reuse, so the last mile is the module map and the build-vs-buy call. The honest answer is not “always community” — it depends on how much of the surface is commodity versus a platform opinion you must own and defend. The map Kestrel settled on, a reasonable default for a mid-size estate:

The decision grid, and the trap: community modules and charts change on someone else’s schedule. An unpinned source or chart version means a future init/upgrade pulls a new major and blows up an environment you didn’t touch. Always pin (~> for modules, an exact chart version), read the CHANGELOG before a bump, and upgrade dev first — the same promotion flow you use for infrastructure applies to the library and the charts.

The reusability payoff is that the root declares add-ons as data — a map of { name, chart, version, irsa_policy } — and the eks-platform-addons module builds the IRSA role + helm_release per entry with for_each. Adding an add-on becomes one map entry, reviewed in a PR, promoted dev → prod, never a copy-paste of forty lines.

Common mistakes and troubleshooting

Platform failures cluster around layer ordering, IRSA wiring, Karpenter discovery, upgrade skew, and the Terraform/GitOps seam. This is the operations map:

Five gotchas cost real hours and deserve prose:

The provider-ordering problem is the whole reason for the cluster/platform split. If you take one thing from this lesson: a kubernetes or helm provider must be configured from a cluster that already exists in another state, read via data "aws_eks_cluster". Fold the add-ons into the cluster config and it works until the first rebuild or destroy, then fails because a provider can’t reach a cluster that isn’t there. Separate the states; the platform layer reads, never creates, the cluster.

IRSA fails silently, then loudly. The classic is a controller that seems fine until it makes its first AWS call and gets AccessDenied. Debug it in order: is the SA annotated with the role ARN? Does the role’s trust policy condition on the exact system:serviceaccount:<namespace>:<name>? For Pod Identity, is the eks-pod-identity-agent add-on installed and the association created? kubectl exec a pod and run aws sts get-caller-identity — if it returns the node role, the wire-up is wrong.

Karpenter discovery is tag-driven. Karpenter finds subnets and security groups by tag (karpenter.sh/discovery = <cluster-name>), and its EC2NodeClass references them by that tag selector. Forget the tag (it lives in the network layer) and Karpenter provisions nothing, with a quiet “no matching subnets” in its logs. Tag in the network layer, select by tag in the NodePool.

Upgrade order is not advice, it is a constraint. Kubelet must never lead the API server. Bump cluster_version one minor, let the control plane go green, upgrade the core add-ons to match, then roll the managed node group and let Karpenter drift-replace its nodes, then the platform charts. Skip a minor or upgrade nodes first and you get NotReady nodes or admission failures.

Draw the Terraform/GitOps line and defend it. The instant both Terraform and Argo CD (or a human with kubectl) manage the same object, you get an infinite reconcile loop or drift. Terraform’s job ends at installing Argo CD and the platform add-ons; everything above — the apps — is Argo CD’s. If you find Terraform planning to revert an app change, the object is on the wrong side of the seam.

Cost, cleanup & production notes

The platform bills whether or not traffic flows. The rough monthly cost if left running in ap-south-1, and the levers:

The dominant surprises are the control-plane fee (per cluster, so prefer fewer clusters with namespaces over a cluster per team), the NAT gateway (hourly even idle — single NAT in non-prod), and on-demand nodes (which is exactly what Karpenter spot + consolidation attacks). Destroy in reverse layer order; keep the bootstrap bucket and lock table unless you are fully finished.

Five production-hardening notes to carry beyond the demo:

1. State is the crown jewels — remote, locked (DynamoDB or use_lockfile on TF 1.10+), versioned and encrypted, one key per env per layer. A CI deploy role is scoped to a single key prefix, so a leaked credential is contained to one layer.
2. IRSA everywhere, nothing on the node — every controller and app that touches AWS assumes its own role; the node role holds only what EKS requires. No static keys in any Secret; app secrets arrive via the External Secrets Operator.
3. Private where it counts — private API endpoint in prod (public off), private subnets for nodes, KMS envelope encryption for Secrets, and default-deny NetworkPolicy with Pod Security Standards restricted.
4. Upgrade in order, on a cadence — control plane → core add-ons → nodes → platform charts, one minor at a time, with the skew rule; rehearse it in dev/staging before prod, and keep Velero restores tested.
5. Draw the platform/app seam — Terraform owns the cluster and add-ons; Argo CD owns the apps. The day someone kubectl edits a Terraform-managed object is the day drift begins.

Cheat-sheet

The dense quick-reference for assembling an EKS platform with Terraform.

Layers & state

Core modules & charts

Providers on EKS

Commands

Interview and exam questions

1. Why split the EKS platform into separate cluster and platform states? The provider-ordering problem: the kubernetes/helm providers are configured at plan time from the cluster’s endpoint, and a provider cannot depend on a resource created in its own apply. Folding add-ons into the cluster config works on a clean first apply but fails on rebuild/destroy. The platform layer reads an already-created cluster via data "aws_eks_cluster" and only then configures the providers. Blast radius, team ownership and apply time reinforce the split.

2. What is IRSA and why prefer it over node-role permissions or static keys? IRSA (IAM Roles for Service Accounts) federates the cluster’s OIDC provider to IAM so a service account assumes its own role via sts:AssumeRoleWithWebIdentity. A permission on the node role is granted to every pod on the node; a static key in a Secret can leak and never rotates. IRSA scopes each controller to exactly its permissions, trusted for one namespace/SA, with no long-lived credential.

3. IRSA vs EKS Pod Identity — when each? IRSA is the mature, universal path (per-cluster OIDC provider + SA annotation), needed when a chart hard-codes the annotation or you go cross-account. Pod Identity is newer and simpler — an agent add-on plus an association object, no OIDC provider or annotation. Prefer Pod Identity for new in-account add-ons you control; mixing is normal.

4. State the apply order and the destroy order, and why the destroy order matters. Apply network → cluster → platform → apps; destroy in reverse apps → platform → cluster → network. Destroy order matters because add-ons create AWS objects (ALBs, ENIs, security-group rules) inside the VPC; tearing down network first strands them and the VPC delete hangs. The order is a property of the dependency graph, not memory — Terragrunt run-all walks it.

5. Give the cluster upgrade order and the version-skew rule. Control plane first (one minor, never skip), then core add-ons to match, then the managed node group and Karpenter nodes, then the platform charts. Kubelet may lag the control plane (several minors on the extended path) but must never lead it. Match each add-on version to the new cluster version.

6. Why run a small managed node group and Karpenter? Chicken-and-egg: Karpenter, CoreDNS and the CSI drivers need somewhere to run before Karpenter can provision. A tiny on-demand managed group (tainted CriticalAddonsOnly) runs the system add-ons; Karpenter provisions right-sized, mostly-spot workload nodes and consolidates them when idle.

7. How does the helm provider authenticate to EKS, and why not a static token? Via an exec block that calls aws eks get-token at apply time, yielding a short-lived token. A data "aws_eks_cluster_auth" token would be baked into state and expire; exec fetches a fresh token per run and keeps nothing sensitive in state.

8. Where is the Terraform/GitOps boundary, and why draw it there? Terraform owns the cluster and platform add-ons and installs Argo CD; Argo CD owns application delivery from Git. Drawn at the platform/app seam, it keeps infra changes on plan/apply with an IAM audit trail and app changes on PRs with Argo CD sync/rollback — and stops Terraform and kubectl fighting over the same object.

9. How does Karpenter reduce cost, and what guardrail prevents a runaway? A broad instance set with spot preferred and on-demand fallback, plus consolidationPolicy: WhenEmptyOrUnderutilized to drain under-used nodes onto cheaper capacity. The guardrail is a NodePool CPU limit so a misbehaving workload can’t provision unbounded nodes, watched by a budget alarm.

10. What does KMS envelope encryption on EKS protect, above EBS encryption? cluster_encryption_config { resources = ["secrets"] } encrypts Kubernetes Secrets in etcd with a KMS key (envelope encryption), so a Secret is protected at the application layer, not just by the underlying EBS volume encryption of the control-plane storage. It closes the gap where a Secret would otherwise sit in etcd in plaintext.

11. (Associate-style) A backend "s3" block sets key = "${var.env}/cluster/terraform.tfstate". What happens? It fails at init — the backend is read before variables/locals are evaluated, so no interpolation is allowed there. Move the value to partial config: terraform init -backend-config=backend.hcl (or -backend-config="key=prod/cluster/terraform.tfstate").

12. How do the layers pass data, and how does Terragrunt improve on the native approach? Natively, a lower layer reads an upper layer’s outputs with a data "terraform_remote_state" source keyed to the other layer’s state. Terragrunt replaces that with a first-class dependency block (with mock_outputs so a plan runs before the dependency exists) and can run-all across the dependency graph in the correct order automatically.

Key takeaways

• A platform is the layers, not the resources. network → cluster → platform → apps, each with its own state key, is what turns “a cluster with Helm on it” into a maintainable estate — separate blast radius, separate owners, separate cadence.
• The cluster/platform split is forced by the provider-ordering problem. A kubernetes/helm provider must read an already-created cluster via a data source; the platform layer reads, never creates, the cluster — which is also why it can’t detonate on rebuild or destroy.
• IRSA everywhere, nothing on the node. Every controller and app assumes its own OIDC-federated role scoped to exactly its permissions; the node role holds only what EKS requires, and no static key ever lands in a Secret.
• A small managed node group plus Karpenter. System add-ons run on a tiny on-demand group; Karpenter provisions right-sized, mostly-spot workload nodes and consolidates them — the cost lever, guarded by a NodePool limit and a budget.
• Order is a constraint, not advice. Apply network→cluster→platform→apps and destroy in reverse; upgrade control plane → add-ons → nodes → charts, one minor at a time, kubelet never ahead of the API server.
• Draw the Terraform/GitOps seam. Terraform installs Argo CD and the platform; Argo CD syncs the apps. The moment both manage one object, you get drift or a reconcile loop.
• Adopt community for commodity, own your opinions, pin everything. terraform-aws-modules/{vpc,eks,iam,karpenter} and upstream charts for the building blocks; your own eks-platform-addons and NodePool modules for the platform’s decisions — and promote every version bump dev → staging → prod.
• Security and SRE are properties of the platform — private endpoints, KMS Secrets encryption, network policy + Pod Security Standards, observability with alerts, Velero DR you have actually restored, and an OIDC CI gate scoped to one state key per layer.