Terraform on AWS EKS: the kubernetes & helm Providers, App Deployment & the GitOps Handoff

You have an EKS cluster. terraform apply finished, kubectl get nodes shows three green nodes, and you feel done. You are roughly half done. A Kubernetes cluster with nothing running on it is an expensive way to heat a data centre; the value is in what runs inside it — the ingress controller that gives your services a load balancer, the secrets operator that pulls credentials from AWS Secrets Manager, the DNS controller that publishes records, and eventually the applications themselves. The moment you try to make Terraform reach inside the cluster to install those things, you cross into the single most error-prone territory in all of Terraform: the kubernetes and helm providers, configured from a cluster that Terraform itself is managing. Get the authentication wrong and your apply succeeds today and fails Unauthorized in fifteen minutes. Get the ordering wrong and a clean apply dies with Provider configuration ... cannot be determined until apply. Get the boundary wrong and you build a platform where every application deploy is a terraform apply that app teams are terrified to run.

This lesson is about doing all three correctly, and then knowing where to stop. The strong industry pattern — the one every mature platform team converges on — is a division of labour: Terraform provisions the platform (the cluster, its add-ons, the controllers, the IAM/IRSA wiring), and then hands off application delivery to GitOps — Argo CD or Flux, bootstrapped by Terraform but thereafter driven by Git. Terraform is superb at building the pieces of a cluster and terrible at continuously reconciling twenty microservices that deploy forty times a day; Argo CD is the reverse. The art is drawing the line between them in the right place, and then having Terraform bootstrap the thing that owns the other side of the line. That is the whole arc of this lesson: providers, ordering, app deployment, and the handoff.

By the end you will have run a complete, copy-pasteable, two-stack demo yourself: a cluster stack (from the companion lesson) whose outputs feed an apps stack that configures the kubernetes and helm providers over an exec block, installs a real controller and a sample app with helm_release, then bootstraps Argo CD plus a root Application pointing at a Git repository — the handoff, live. You will run init → plan → apply → verify → destroy, and you will leave with four reference tables you will come back to for years: exec-vs-data-source auth, the ordering patterns, helm_release versus kubernetes_manifest, and the Terraform-versus-GitOps boundary. Every failure that actually pages people — provider-unknown-at-plan, token-expiry drift, CRD-at-plan-time, secret-in-state, a Helm release stuck pending, and destroy ordering — gets its own row in the troubleshooting table and a paragraph on how to escape it.

What you’ll build

The scenario is the one every team hits the week after the cluster goes up: “great, now put the platform on it.” Concretely you will build a second Terraform stack — separate from the one that created the cluster — that does four jobs. First, it configures the kubernetes and helm providers to authenticate to your existing EKS cluster using the cluster’s API endpoint, its base64-decoded certificate authority, and an exec credential plugin that shells out to aws eks get-token. Second, it installs a platform controller (the AWS Load Balancer Controller) and a sample application (podinfo) as helm_release resources, wiring the controller’s IAM permissions through IRSA. Third, it bootstraps Argo CD with another helm_release. Fourth — the punchline — it creates a single root Argo CD Application that points at a Git repository, after which every future application is delivered by Argo CD from Git, not by Terraform. Terraform builds the platform; Git drives the apps.

Why split it this way rather than cram everything into one giant terraform apply that builds the cluster and installs everything and deploys the apps? Because the three layers have wildly different change rates and blast radii, and because the providers physically cannot be configured reliably from a cluster created in the same run. The comparison below is the argument in one table — it is the reasoning you will repeat to every engineer who asks “why can’t I just do it all in main.tf”:

The last two rows are the journey of this lesson: you start by proving Terraform can deploy apps (two stacks), then you deliberately hand that job to Argo CD because Terraform is the wrong tool for continuous app delivery. Here is the full inventory of what the apps stack builds, so you can see the moving parts before the code:

And here is the same picture as a division of labour between the two stacks — the mental model to keep as you read the code:

Read the diagram left to right: Terraform (badge 2 — the cluster and apps live in two separate stacks) configures the kubernetes and helm providers with an exec block that mints a fresh token, not a stale one (badge 1); those providers reach the EKS API using its endpoint and CA; Terraform installs the platform controllers with helm_release (badge 3) and wires External Secrets so credentials never land in state (badge 4); then it bootstraps Argo CD (badge 6) and crosses the Terraform → GitOps boundary (badge 5), after which Argo CD reconciles every application from Git. The six legend entries are the six decisions the rest of this lesson makes in code. This lesson assumes the cluster itself already exists, built the way the EKS cluster provisioning lesson lays it out (VPC, managed node groups, the OIDC provider that IRSA depends on), and that your AWS provider auth and S3/DynamoDB backend are configured per the Getting Started on AWS lesson.

The kubernetes and helm providers: authenticating to EKS

The kubernetes and helm providers are, at bottom, Kubernetes API clients wearing Terraform costumes. To do anything they need what any kubectl needs: the API server’s URL, the CA certificate to trust it, and a credential to prove who you are. EKS hands you the first two as cluster attributes and expects you to produce the third yourself — and how you produce it is the decision that separates a config that works forever from one that breaks every fifteen minutes.

Start with what the cluster exposes. Whether you read the cluster with a data "aws_eks_cluster" source or straight off the resource, three attributes matter:

The base64 trap catches everyone once. certificate_authority[0].data is base64-encoded PEM, and the cluster_ca_certificate provider argument wants decoded PEM. Feed it the raw value and TLS fails with x509: certificate signed by unknown authority or a decode error; you must wrap it in base64decode(...). That is not optional and it is the single most common copy-paste bug in EKS provider configs.

Those three cluster attributes feed a small, fixed set of provider arguments — the same names on both the kubernetes and helm providers:

The last two rows exist mostly so you can rule them out: config_path couples the run to a kubeconfig on disk (fine on a laptop, a landmine in CI), and insecure = true disables the very TLS check the CA is there to perform. That leaves the real choice between token and exec.

Now the credential — the real decision. There are two ways to give the provider a token, and they are not equivalent:

The provider-auth decision: exec vs the aws_eks_cluster_auth data source

Option A — the exec credential plugin (preferred). You give the provider an exec block that Terraform runs every time it needs to talk to the cluster, shelling out to aws eks get-token --cluster-name <name>, which returns a short-lived token. The crucial property: the token is generated at connection time and is never stored in Terraform state or the plan. It cannot go stale between plan and apply because it is minted fresh for each.

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args = [
      "eks", "get-token",
      "--cluster-name", data.aws_eks_cluster.this.name,
      "--region", var.region,
    ]
    # Optional: pin a profile or role so CI and laptops agree.
    # env = { AWS_PROFILE = "platform" }
  }
}

Each field of that exec block earns its place:

Option B — the aws_eks_cluster_auth data source (the trap). This data source computes a token by presigning an STS GetCallerIdentity request. It works, and you will see it in older blog posts and modules:

# ⚠️ WORKS, BUT THE TOKEN EXPIRES IN ~15 MINUTES.
data "aws_eks_cluster_auth" "this" {
  name = data.aws_eks_cluster.this.name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token   # <-- computed at plan, baked into state
}

The token from aws_eks_cluster_auth is valid for roughly 15 minutes. It is computed during plan (or during the refresh phase) and written into the plan and state as a literal string. If more than fifteen minutes pass between plan and apply — a code review on the PR, a slow CI queue, a coffee — the token is dead when apply runs and you get Unauthorized. It also produces perpetual “drift”: every refresh/plan recomputes a new token, so the data source is always “changing.” It is not wrong so much as fragile, and there is a strictly better option, so use exec unless you have a specific reason not to (for example, an execution environment where you cannot run the aws binary). The full comparison:

The exec protocol version is worth a word: use api_version = "client.authentication.k8s.io/v1beta1". The older v1alpha1 is long gone, and aws eks get-token speaks v1beta1. If you see exec plugin: invalid apiVersion, that mismatch is why.

Configuring the helm provider (and the 2.x vs 3.x syntax split)

The helm provider wraps Helm and needs the exact same connection settings — but where they go changed between major versions, and this trips people who copy a snippet that assumes the other version. In the widely-deployed helm ~> 2.x provider, the Kubernetes connection is a nested kubernetes { ... } block:

# helm provider 2.x — nested kubernetes { } block
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name, "--region", var.region]
    }
  }
}

The helm ~> 3.x provider (shipped 2025) flattens this to a single kubernetes = { ... } attribute (note the =), and exec becomes a nested attribute too:

# helm provider 3.x — kubernetes = { } attribute (note the equals sign)
provider "helm" {
  kubernetes = {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name, "--region", var.region]
    }
  }
}

The demo below pins ~> 2.17 for the classic block syntax you will see in 90% of modules today, but the table tells you how to translate if you are on 3.x:

Both providers, on both versions, run identically on OpenTofu — the kubernetes, helm, and aws providers are the same binaries. Nothing in this lesson is Terraform-vs-OpenTofu specific; assume Terraform ≥ 1.6 or OpenTofu ≥ 1.6 throughout. Pin every provider in required_providers with ~> so a surprise major (like helm 2 → 3) never lands on you mid-sprint:

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws        = { source = "hashicorp/aws",        version = "~> 5.60" }
    kubernetes = { source = "hashicorp/kubernetes",  version = "~> 2.31" }
    helm       = { source = "hashicorp/helm",        version = "~> 2.17" }
  }
}

The ordering problem: why cluster + apps in one apply is fragile

Here is the failure that defines this whole topic. It is tempting — obvious, even — to write one configuration that creates the EKS cluster with aws_eks_cluster and, in the same files, configures the kubernetes provider from that cluster and drops a helm_release on top. One apply, cluster and app together. It looks clean. It is a trap, and understanding exactly why is the difference between a platform engineer and someone who will spend a Saturday on it.

The root cause is Terraform’s evaluation model. Provider configurations are evaluated before resources are created, and a provider block cannot depend on values that only exist after an apply. When you write host = aws_eks_cluster.this.endpoint and the cluster does not exist yet, endpoint is an unknown value at plan time. Terraform cannot configure a provider from an unknown, so on a clean apply you get the notorious error. Even when it seems to work (because the values happen to be known from a prior state), it is fragile: the provider is configured from resources in the same graph, so destroys invert badly and any change that touches the cluster ripples into every in-cluster resource. The specific ways it breaks:

There are three ways to deal with ordering, and only one of them is a real production answer. Know all three because you will meet all three in other people’s code:

The winning pattern is two root modules with two state files. The cluster stack builds the cluster (and often the core VPC and node groups) and outputs the handful of values the apps stack needs. The apps stack reads those, configures the providers against an already-existing cluster — so every attribute is known — and installs everything in-cluster. Their lifecycles are independent: you can redeploy every app forty times without ever re-planning the cluster, and a terraform destroy of the apps stack cleanly removes workloads while the cluster is still up to receive the API calls.

The join is terraform_remote_state, reading the cluster stack’s outputs from its S3 backend:

# apps stack — data.tf
data "terraform_remote_state" "cluster" {
  backend = "s3"
  config = {
    bucket = "kv-tf-state-ap-south-1"
    key    = "eks/cluster/terraform.tfstate"
    region = "ap-south-1"
  }
}

# Re-read the cluster directly for its live endpoint/CA/name.
data "aws_eks_cluster" "this" {
  name = data.terraform_remote_state.cluster.outputs.cluster_name
}

You have two ways to get the cluster’s endpoint and CA into the apps stack, and it is worth knowing the trade-off:

A common and robust compromise — the one the demo uses — is to take the cluster name from remote state (a stable string) and then read the endpoint and CA fresh from data "aws_eks_cluster", so the providers always have current connection details even if the cluster stack’s outputs lag. The remote state at scale lesson goes deeper on wiring stacks together with terraform_remote_state and the alternatives (SSM parameters, a data source per resource) when you have many stacks.

Deploying apps: helm_release, kubernetes_manifest, namespaces & secrets

With the providers configured against a live cluster, Terraform can now create objects inside it. You have two families of resources for this, and choosing between them correctly is most of the skill.

helm_release — the workhorse

helm_release installs (and upgrades, and deletes) a Helm chart. It is the right tool for anything that ships as a chart: controllers, operators, add-ons, and most off-the-shelf apps. Its important arguments:

There are three ways to feed values into a chart, and idiomatic Terraform uses them in this order of preference:

templatefile is the one you will lean on for controllers, because a controller almost always needs a value that Terraform computed — an IAM role ARN, the cluster name, a VPC id. Here is the sample app first, kept trivial to prove the providers work, then the real controller:

resource "kubernetes_namespace" "demo" {
  metadata { name = "demo" }
}

resource "helm_release" "podinfo" {
  name       = "podinfo"
  repository = "https://stefanprodan.github.io/podinfo"
  chart      = "podinfo"
  version    = "6.7.1"
  namespace  = kubernetes_namespace.demo.metadata[0].name

  set {
    name  = "replicaCount"
    value = "2"
  }
  set {
    name  = "resources.requests.cpu"
    value = "50m"
  }
}

kubernetes_manifest and the raw-object family

helm_release covers charts; for raw Kubernetes objects Terraform has the kubernetes_* resource family — typed resources for the common objects, plus the generic kubernetes_manifest for anything else:

The typed resources are unremarkable; kubernetes_manifest is the powerful one, and it has a sharp edge that defines how you use it:

⚠️ kubernetes_manifest talks to the cluster at plan time. To compute the plan it fetches the resource’s OpenAPI schema from the live API server (a server-side dry-run). Two consequences: (1) the cluster must already exist and be reachable during plan — which is another reason for the two-stack split; and (2) if the object is a custom resource whose CRD does not yet exist at plan time, the plan fails with no matches for kind "X" in group "Y". You cannot install a CRD and a custom resource of that CRD in the same apply with kubernetes_manifest.

That CRD-at-plan-time constraint is the reason people reach for alternatives when applying custom resources (like an Argo CD Application, which is a CR of the argoproj.io CRDs). The comparison you will use to choose:

The practical rule: prefer helm_release for anything chart-shaped, and use kubernetes_manifest sparingly for one-off raw objects whose CRDs already exist. When you need to apply a CR whose CRD is installed in the same run, either (a) use a chart that bundles both, or (b) reach for the community kubectl_manifest resource (from alekc/kubectl or gavinbunney/kubectl), which applies raw YAML at apply time without the plan-time schema fetch — the standard escape hatch for the CRD ordering problem.

The secrets problem: why they land in state, and what to do instead

Here is a footgun with real security consequences. Any secret material you hand to Terraform — a kubernetes_secret’s data, a set_sensitive Helm value, a password in a values file — is written into the Terraform state file in base64, which is not encryption. Anyone who can read state can read the secret. set_sensitive only redacts it from console output; it does nothing about state.

# ⚠️ ANTI-PATTERN — the password is now plaintext-equivalent in state.
resource "kubernetes_secret" "db" {
  metadata{
    name = "db-creds"
    namespace = "demo"
  }
  data = {
    password = var.db_password   # <-- lands base64 in terraform.tfstate
  }
}

The options, worst to best:

The strong pattern is External Secrets Operator (ESO): Terraform installs the operator (a helm_release) and grants it an IRSA role that can read AWS Secrets Manager; then a SecretStore and ExternalSecret (delivered by Git, via Argo CD) tell ESO which secret to sync into which Kubernetes Secret. The actual secret value flows AWS Secrets Manager → ESO → Kubernetes, entirely outside Terraform. Terraform provisions the capability (operator + IAM) and never sees the value. That is the model to internalise: Terraform owns the plumbing for secrets, never the secrets themselves.

Where Terraform should stop: the GitOps handoff

Everything so far proves Terraform can deploy applications. The senior lesson is that for most applications it should not — and knowing where to stop is what makes a platform maintainable. Terraform and a GitOps controller (Argo CD or Flux) are good at opposite things, and the industry has converged on a division of labour that plays to each.

Terraform is a plan-and-apply tool: it computes a diff when you run it, applies it, and then stops thinking about the world until you run it again. That is exactly right for infrastructure that changes rarely and deliberately — a cluster, a node group, an IAM role, an ingress controller. It is exactly wrong for a fleet of applications that deploy dozens of times a day, where you want continuous reconciliation: if someone kubectl edits a Deployment at 3 a.m., you want it snapped back to the declared state within minutes, not “on the next terraform apply, whenever that is.” That continuous reconciliation is precisely what Argo CD and Flux do: they run in the cluster, watch a Git repository, and continuously make the cluster match Git — syncing, pruning removed resources, and self-healing drift.

So the boundary is drawn by change rate and by who owns the change. The reference table you will pin to the wall:

The head-to-head, so the trade-off is unmistakable:

Bootstrapping Argo CD with Terraform (the handoff, in code)

The elegant part: Terraform doesn’t just stop at the boundary — it installs the thing that owns the other side. Terraform bootstraps Argo CD (a helm_release) and creates one root object — the App-of-Apps root Application — that points at a Git repository. After that single apply, Terraform’s job with applications is done forever; Argo CD reads the repo and manages every child Application declared there. New app? Add a manifest to Git; Argo CD deploys it; Terraform never runs.

# 1) Bootstrap Argo CD itself — a normal helm_release.
resource "helm_release" "argocd" {
  name             = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = "7.7.7" # chart version — pin it
  namespace        = "argocd"
  create_namespace = true
  atomic           = true
  timeout          = 600

  values = [yamlencode({
    global = { domain = "argocd.${var.base_domain}" }
    configs = { params = { "server.insecure" = true } } # TLS terminated at the ALB
  })]
}

Now the App-of-Apps. An Argo CD Application is a custom resource of a CRD that the Argo CD chart just installed — so applying it with kubernetes_manifest would hit the CRD-at-plan-time wall on a clean apply (the CRD does not exist at plan). The clean way is to create the root Application with the argocd-apps Helm chart, which templates the Application object and applies it at apply time (after Argo CD’s CRDs exist), sequenced with depends_on:

# 2) The single root App-of-Apps — points Argo CD at your Git repo.
resource "helm_release" "root_app" {
  name       = "root"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argocd-apps"
  version    = "2.0.2"
  namespace  = helm_release.argocd.namespace

  values = [yamlencode({
    applications = {
      root = {
        namespace = "argocd"
        project   = "default"
        source = {
          repoURL        = var.gitops_repo_url        # e.g. https://github.com/acme/eks-apps.git
          targetRevision = "main"
          path           = "apps"                     # folder of child Application manifests
        }
        destination = {
          server    = "https://kubernetes.default.svc"
          namespace = "argocd"
        }
        syncPolicy = {
          automated   = { prune = true, selfHeal = true }
          syncOptions = ["CreateNamespace=true"]
        }
      }
    }
  })]

  depends_on = [helm_release.argocd] # CRDs must exist before the CR is applied
}

The fields of that root Application are the entire contract between Terraform and Git — after this, Terraform touches none of it again:

automated = { prune = true, selfHeal = true } is the whole philosophy in two flags: prune deletes cluster objects removed from Git, and selfHeal reverts manual changes back to Git’s declared state — continuous reconciliation Terraform structurally cannot do. From here, the workflow for every application is: commit a child Application manifest under apps/ in the Git repo; Argo CD notices, creates it, and keeps it in sync. Terraform’s involvement with applications ended at helm_release.root_app.

A note on Argo CD vs Flux, since the boundary is identical for both:

Both are bootstrapped by Terraform the same way and both take over app delivery at the same boundary; pick on team preference, not capability. The ALB controller lesson covers the controller we install below in depth, and the production platform capstone assembles the full picture — cluster, platform add-ons, IRSA, and this GitOps handoff — into one shippable platform.

Hands-on: build it with Terraform

Time to run it. This demo assumes the cluster stack already exists (from the EKS provisioning lesson) with an S3 backend at s3://kv-tf-state-ap-south-1/eks/cluster/terraform.tfstate, exporting cluster_name, oidc_provider_arn, and oidc_provider_url as outputs, and that its OIDC provider is created (IRSA depends on it). We build the apps stack in its own directory with its own state. Region is ap-south-1 (Mumbai). Everything is copy-pasteable.

⚠️ This spends real money. An EKS cluster (~$0.10/hr control plane), node-group EC2, and any ALB the controller creates all bill by the hour. Run the destroy step when you finish.

Step 1 — lay out the apps stack

eks-apps/
├── versions.tf        # providers + backend
├── providers.tf       # aws, kubernetes, helm — the exec auth
├── data.tf            # remote_state + aws_eks_cluster lookups
├── variables.tf
├── platform.tf        # ALB controller (IRSA) + External Secrets Operator
├── apps.tf            # podinfo sample app
├── gitops.tf          # Argo CD + root App-of-Apps
└── outputs.tf

Step 2 — versions.tf and the backend

# versions.tf
terraform {
  required_version = ">= 1.6"

  required_providers {
    aws        = { source = "hashicorp/aws",       version = "~> 5.60" }
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.31" }
    helm       = { source = "hashicorp/helm",       version = "~> 2.17" }
  }

  backend "s3" {
    bucket       = "kv-tf-state-ap-south-1"
    key          = "eks/apps/terraform.tfstate" # NOTE: a different key from the cluster
    region       = "ap-south-1"
    use_lockfile = true                          # S3-native locking (TF 1.10+); or dynamodb_table
    encrypt      = true
  }
}

The apps stack shares the bucket but uses a different state key from the cluster stack — that separation is the whole point of the two-stack pattern.

Step 3 — providers.tf (the exec auth) and data.tf

# data.tf
data "terraform_remote_state" "cluster" {
  backend = "s3"
  config = {
    bucket = "kv-tf-state-ap-south-1"
    key    = "eks/cluster/terraform.tfstate"
    region = "ap-south-1"
  }
}

data "aws_eks_cluster" "this" {
  name = data.terraform_remote_state.cluster.outputs.cluster_name
}

# providers.tf
provider "aws" {
  region = var.region
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name, "--region", var.region]
  }
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name, "--region", var.region]
    }
  }
}

# variables.tf
variable "region" {
  type    = string
  default = "ap-south-1"
}

variable "base_domain" {
  type    = string
  default = "kloudvin.internal"
}

variable "gitops_repo_url" {
  type = string
}

Step 4 — platform.tf: the ALB controller with IRSA

The AWS Load Balancer Controller needs AWS permissions (to create ALBs/target groups), delivered keylessly via IRSA: an IAM role whose trust policy federates the controller’s Kubernetes service account through the cluster’s OIDC provider. Terraform builds the role and passes its ARN into the chart via templatefile.

# platform.tf
locals {
  oidc_provider_url = data.terraform_remote_state.cluster.outputs.oidc_provider_url # e.g. oidc.eks.ap-south-1.amazonaws.com/id/ABC123
  oidc_provider_arn = data.terraform_remote_state.cluster.outputs.oidc_provider_arn
}

# IAM role the controller's service account assumes (IRSA trust).
data "aws_iam_policy_document" "alb_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"
    principals {
      type        = "Federated"
      identifiers = [local.oidc_provider_arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${local.oidc_provider_url}:sub"
      values   = ["system:serviceaccount:kube-system:aws-load-balancer-controller"]
    }
    condition {
      test     = "StringEquals"
      variable = "${local.oidc_provider_url}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "alb" {
  name               = "eks-alb-controller"
  assume_role_policy = data.aws_iam_policy_document.alb_trust.json
}

# The official controller policy (download it, or manage inline). Attach it.
resource "aws_iam_policy" "alb" {
  name   = "AWSLoadBalancerControllerIAMPolicy"
  policy = file("${path.module}/policies/alb-controller.json")
}

resource "aws_iam_role_policy_attachment" "alb" {
  role       = aws_iam_role.alb.name
  policy_arn = aws_iam_policy.alb.arn
}

resource "helm_release" "alb_controller" {
  name       = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-load-balancer-controller"
  version    = "1.8.1"
  namespace  = "kube-system"

  values = [templatefile("${path.module}/values/alb.yaml.tftpl", {
    cluster_name = data.aws_eks_cluster.this.name
    role_arn     = aws_iam_role.alb.arn
    region       = var.region
    vpc_id       = data.terraform_remote_state.cluster.outputs.vpc_id
  })]
}

# values/alb.yaml.tftpl
clusterName: ${cluster_name}
region: ${region}
vpcId: ${vpc_id}
serviceAccount:
  create: true
  name: aws-load-balancer-controller
  annotations:
    eks.amazonaws.com/role-arn: ${role_arn}   # <-- IRSA binding

And External Secrets Operator, so no application secret ever enters Terraform (the operator gets its own IRSA role the same way — omitted here for length; identical trust-policy shape scoped to external-secrets:external-secrets):

resource "helm_release" "external_secrets" {
  name             = "external-secrets"
  repository       = "https://charts.external-secrets.io"
  chart            = "external-secrets"
  version          = "0.10.4"
  namespace        = "external-secrets"
  create_namespace = true
  set {
    name  = "installCRDs"
    value = "true"
  }
}

Step 5 — apps.tf and gitops.tf

apps.tf is the kubernetes_namespace + helm_release "podinfo" from earlier. gitops.tf is the helm_release "argocd" + helm_release "root_app" from the handoff section. Put them in those files as shown.

Step 6 — init, plan, apply

cd eks-apps
terraform init

Initializing the backend...
Initializing provider plugins...
- Installed hashicorp/aws v5.60.0
- Installed hashicorp/kubernetes v2.31.0
- Installed hashicorp/helm v2.17.0
Terraform has been successfully initialized!

terraform plan -var="gitops_repo_url=https://github.com/acme/eks-apps.git"

data.terraform_remote_state.cluster: Reading...
data.aws_eks_cluster.this: Read complete after 1s [name=kv-eks-dev]
...
Plan: 9 to add, 0 to change, 0 to destroy.

Note what did not happen: no Provider configuration ... cannot be determined error, because the cluster already exists and every provider input is a known value. That is the two-stack split paying off. Apply:

terraform apply -var="gitops_repo_url=https://github.com/acme/eks-apps.git" -auto-approve

helm_release.external_secrets: Creation complete after 41s [id=external-secrets]
aws_iam_role.alb: Creation complete after 3s
helm_release.alb_controller: Creation complete after 55s [id=aws-load-balancer-controller]
helm_release.podinfo: Creation complete after 22s [id=podinfo]
helm_release.argocd: Creation complete after 2m18s [id=argocd]
helm_release.root_app: Creation complete after 9s [id=root]

Apply complete! Resources: 9 added, 0 changed, 0 destroyed.

Step 7 — verify

Verify the providers reached the cluster and every layer is live:

# The sample app proves kubernetes + helm providers authenticated.
kubectl get pods -n demo
# NAME                       READY   STATUS    RESTARTS   AGE
# podinfo-7c9b8d6f5-2xk4p    1/1     Running   0          2m
# podinfo-7c9b8d6f5-9tzll    1/1     Running   0          2m

# The platform controller is up.
kubectl get deploy -n kube-system aws-load-balancer-controller
# NAME                           READY   UP-TO-DATE   AVAILABLE
# aws-load-balancer-controller   2/2     2            2

# Argo CD is up and the root App-of-Apps is registered.
kubectl get pods -n argocd
kubectl get applications -n argocd
# NAME   SYNC STATUS   HEALTH STATUS
# root   Synced        Healthy

That last command is the handoff, confirmed: the root Application exists, is Synced to Git, and Healthy. From now on, anything you commit under apps/ in the Git repo, Argo CD deploys — no Terraform. Grab the Argo CD admin password and open the UI to watch it reconcile:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d && echo
kubectl -n argocd port-forward svc/argocd-server 8080:443
# open https://localhost:8080  (user: admin)

Step 8 — destroy & clean up

⚠️ Destroy the apps stack first, while the cluster is still up to receive the delete calls — this is exactly why the split matters. Argo-managed child apps live in the cluster, so let Argo prune them (or delete the root Application) before you tear down the platform:

# Optional: let Argo prune its children first for a clean teardown.
kubectl -n argocd delete application root

cd eks-apps
terraform destroy -var="gitops_repo_url=https://github.com/acme/eks-apps.git" -auto-approve

helm_release.root_app: Destroying...
helm_release.argocd: Destroying...
helm_release.podinfo: Destroying...
helm_release.alb_controller: Destroying...
...
Destroy complete! Resources: 9 destroyed.

If you destroy the cluster stack while the apps stack still has helm_release resources, those releases error with Kubernetes cluster unreachable because the API server they target is gone — the destroy-ordering trap. Apps stack down first, then the cluster stack. Only then is nothing billing.

Variables, outputs & making it reusable

The demo is already parameterised on region, base_domain, and gitops_repo_url. To make the platform reusable across dev/staging/prod, the two levers are a small platform module and for_each over the add-ons.

The for_each-over-releases pattern turns the fleet of controllers into data. Define them once as a map and loop:

variable "platform_addons" {
  description = "Helm releases that make up the platform layer."
  type = map(object({
    repository = string
    chart      = string
    version    = string
    namespace  = string
    values     = optional(list(string), [])
  }))
  default = {
    metrics-server = {
      repository = "https://kubernetes-sigs.github.io/metrics-server/"
      chart      = "metrics-server"
      version    = "3.12.1"
      namespace  = "kube-system"
    }
    external-dns = {
      repository = "https://kubernetes-sigs.github.io/external-dns/"
      chart      = "external-dns"
      version    = "1.15.0"
      namespace  = "external-dns"
    }
  }
}

resource "helm_release" "addon" {
  for_each         = var.platform_addons
  name             = each.key
  repository       = each.value.repository
  chart            = each.value.chart
  version          = each.value.version
  namespace        = each.value.namespace
  create_namespace = true
  values           = each.value.values
}

Adding a controller is now a map entry, not a copy-pasted resource block. Wrap the whole apps stack in a module and each environment becomes a thin root that passes region, gitops_repo_url (pointing at that env’s Git path), and its platform_addons map. Expose what the next layer needs:

# outputs.tf
output "argocd_namespace"    { value = helm_release.argocd.namespace }
output "alb_controller_role" { value = aws_iam_role.alb.arn }
output "gitops_repo"         { value = var.gitops_repo_url }

You do not have to roll your own for the well-trodden pieces. The community modules are excellent and worth using for the platform layer:

The iam-role-for-service-accounts-eks module is the biggest time-saver: it has a pre-baked policy for the ALB controller (and ExternalDNS, ESO, cert-manager, Karpenter, etc.), so the entire platform.tf IAM block collapses to a few lines. Roll your own only when your policy is genuinely bespoke — the production platform capstone shows the blueprint-addons approach at full scale.

Common mistakes and troubleshooting

These are the failures that actually page people on EKS provider work. Scan the table; the prose below unpacks the six nastiest.

Provider-unknown-at-plan is the defining failure of this topic, and its fix is architectural, not a flag. If your provider block references aws_eks_cluster.this.endpoint and that cluster is in the same configuration, a clean apply has no value for endpoint at plan time and Terraform refuses. depends_on and -target are band-aids that do not address the core issue that provider configs cannot consume unknowns. The only durable answer is two stacks: the apps stack reads an already-applied cluster, so every provider input is known. Internalise it as a rule — never configure the kubernetes/helm providers from a cluster resource in the same root module.

Token-expiry drift is subtle because everything looks fine until it doesn’t. With aws_eks_cluster_auth the token is computed at plan and stored, valid ~15 minutes — so in a pipeline with an approval gate between plan and apply, or on a busy runner, it dies and apply fails Unauthorized intermittently, the worst kind of bug. The exec plugin eliminates it entirely: no token in state, none to expire.

CRD-at-plan-time bites when you apply a custom resource (an Argo CD Application, a cert-manager ClusterIssuer, an ESO ExternalSecret) with kubernetes_manifest before its CRD exists. Because kubernetes_manifest fetches the object’s schema from the live API at plan, and the CRD is not there yet, the plan fails no matches for kind. Three escapes: install the CRD-bearing chart and the CR in separate applies; use a chart that bundles both (like argocd-apps); or use the kubectl_manifest resource, which applies raw YAML at apply time with no plan-time schema fetch. The demo uses the argocd-apps chart precisely to sidestep this for the root Application.

Secret-in-state is a security incident waiting to happen. terraform.tfstate stores kubernetes_secret data and set_sensitive values in base64 — trivially decodable, not encrypted. Anyone with read access to the state backend can read every secret. The fix is to keep secrets out of Terraform entirely: install External Secrets Operator, grant it an IRSA role to read AWS Secrets Manager, and let ExternalSecret manifests (delivered by Git) sync values into the cluster. Terraform provisions the capability and never sees the value. If you must put a secret through Terraform, encrypt it with SOPS so only ciphertext is in git and state.

Helm release stuck means the chart’s resources never became Ready within the timeout, or a prior failed install left the release in pending-install/pending-upgrade, and Terraform eventually errors context deadline exceeded. Two habits fix most of it: set atomic = true so a failed install rolls back instead of leaving a half-installed pending release, and raise timeout for slow charts (Argo CD, Istio). To clear an inherited stuck release, helm -n <ns> uninstall <name>, then re-apply.

Destroy ordering inverts the create order and is the reason the two-stack split matters as much on the way down as on the way up. helm_release resources need to reach the cluster’s API to uninstall; if the cluster is already gone, they error Kubernetes cluster unreachable and the destroy hangs with orphaned finalizers. Always tear down the apps stack (which talks to the cluster) before the cluster stack (which removes it). Within the apps stack, let Argo CD prune its children — or delete the root Application — before destroying, so Argo-managed workloads with finalizers don’t block the teardown.

Cost, cleanup & production notes

The apps stack itself is nearly free — Helm releases and IAM roles cost nothing. What costs money is the cluster this all runs on and anything the controllers create (notably an ALB when you deploy an Ingress). Rough Mumbai (ap-south-1) figures to make the “destroy it” case concrete:

The single biggest lever is not leaving it running: terraform destroy the apps stack, then the cluster stack. The subtle cost trap is orphaned ALBs — if the controller created a load balancer for an Ingress and you delete the Ingress out from under it (or destroy in the wrong order), the ALB can leak and keep billing; verify in the EC2 console after teardown. Five production-hardening notes beyond the demo:

1. State is sensitive — treat it accordingly. Even with ESO keeping app secrets out, the apps stack’s state can hold IAM ARNs and, if you slip, kubernetes_secret data. Keep state in an encrypted, access-controlled, versioned S3 bucket with locking; never local, never in git.
2. Draw the Terraform/GitOps boundary and enforce it. Terraform owns cluster + platform; Git owns apps. Resist the pull to terraform apply an application “just this once” — it erodes the boundary and reintroduces drift Terraform won’t reconcile.
3. Pin every chart version and every provider. Unpinned helm_release.version means a re-apply can silently jump chart major versions; unpinned providers can land helm 2 → 3 on you. Pin with explicit versions and ~>, and bump deliberately.
4. Use IRSA (or EKS Pod Identity) for every controller — never node-role permissions. Scope each controller’s IAM to exactly its service account via the OIDC trust sub. Broad node-instance-profile permissions are a lateral-movement risk.
5. Bootstrap Argo CD, then let it manage itself and drift-detect the platform. After bootstrap, move the platform charts’ desired state into Git so Argo reconciles them too, and run scheduled terraform plan (drift detection) on the cluster stack to catch out-of-band changes.

Cheat-sheet

Interview and exam questions

1. Why is exec/aws eks get-token preferred over the aws_eks_cluster_auth data source for the kubernetes provider? The exec plugin mints a fresh token at connect time and never writes it to state; aws_eks_cluster_auth computes a ~15-minute token at plan time and bakes it into the plan/state, so it goes stale between plan and apply (intermittent Unauthorized) and shows as perpetual drift.

2. You forgot one function on the CA and TLS fails. Which? base64decode. certificate_authority[0].data is base64-encoded PEM; the cluster_ca_certificate argument needs it decoded, or you get x509: certificate signed by unknown authority.

3. Why can’t you create the EKS cluster and a helm_release on it in the same apply? Provider configurations are evaluated before resources exist and cannot consume unknown values; on a clean apply the cluster’s endpoint/CA are unknown at plan, so the provider config is invalid (cannot be determined until apply). Split into two stacks.

4. What is special about kubernetes_manifest at plan time? It contacts the live API server for a server-side dry-run/schema fetch during plan, so the cluster must already exist and the resource’s CRD must already be installed at plan time — otherwise no matches for kind. helm_release has neither requirement.

5. Where do secrets go when you use kubernetes_secret or set_sensitive, and what’s the fix? Into terraform.tfstate, base64-encoded (not encrypted) — readable by anyone with state access. Fix: External Secrets Operator + IRSA (values flow AWS Secrets Manager → cluster, never through Terraform), or SOPS for cipher-only-in-git.

6. Draw the Terraform vs GitOps boundary. What does each own? Terraform owns the cluster, VPC, node groups, IAM/IRSA, and bootstraps the platform controllers and Argo CD/Flux. GitOps owns application Deployments/Services/Ingress/config and reconciles them continuously from Git. Terraform = low-cadence infra; GitOps = high-cadence apps.

7. How does Terraform “hand off” to GitOps in code? It installs Argo CD with a helm_release, then creates a single root App-of-Apps Application (via the argocd-apps chart) pointing at a Git repo with syncPolicy.automated { prune, selfHeal }. Thereafter Argo CD manages every app from Git; Terraform doesn’t run for app changes.

8. Why bootstrap the root Application with the argocd-apps chart instead of kubernetes_manifest? The Application is a CR of a CRD that the Argo CD chart just installed; kubernetes_manifest needs the CRD at plan time and would fail no matches for kind on a clean apply. The argocd-apps chart templates and applies the Application at apply time (after CRDs exist), sequenced with depends_on.

9. (Terraform Associate) Why must the apps stack use a different backend key from the cluster stack? So they have separate state files and independent lifecycles/locks. Sharing a key would merge their states, defeating the two-stack split and coupling their blast radius.

10. (Terraform Associate) A helm_release is stuck at pending-install after a timeout. What happened and how do you prevent it? The chart’s resources didn’t become ready within timeout, leaving the release pending. Set atomic = true (roll back on failure) and raise timeout; clear a stuck release with helm uninstall before re-applying.

11. In what order do you destroy, and why? Apps stack first, then the cluster stack. helm_release uninstalls must reach the cluster API; if the cluster is gone first they error Kubernetes cluster unreachable. Delete the Argo root Application (to prune children) before destroying.

12. How does a platform controller get AWS permissions without static keys? IRSA: an IAM role whose trust policy federates the controller’s Kubernetes service account through the cluster’s OIDC provider (condition on <oidc>:sub = system:serviceaccount:<ns>:<sa>), and the SA annotated with eks.amazonaws.com/role-arn. The pod assumes the role via a projected token — no stored credentials.

Key takeaways

• Configure the kubernetes/helm providers from the cluster’s endpoint + base64decode(certificate_authority[0].data) + an exec block calling aws eks get-token. The exec token is fresh per operation and never stored — unlike the aws_eks_cluster_auth data source’s ~15-minute token, which drifts and dies between plan and apply.
• Never build the cluster and its in-cluster resources in one apply. Provider configs can’t consume unknown values, so split into two stacks: a cluster stack, and an apps stack that reads a live cluster via terraform_remote_state/data.aws_eks_cluster. depends_on and -target are band-aids, not the fix.
• helm_release is the workhorse; kubernetes_manifest is the sharp-edged special case that contacts the API at plan time and needs CRDs to pre-exist. Prefer charts; for a CR whose CRD is installed in the same run, use a bundling chart or kubectl_manifest.
• Secrets handed to Terraform land in state, base64, not encrypted. Keep them out entirely with External Secrets Operator + IRSA; Terraform provisions the capability, never the value.
• Terraform provisions the platform; GitOps delivers the apps. Draw the boundary at change rate: cluster/controllers/IAM in Terraform, applications in Git. Terraform then bootstraps Argo CD and one root App-of-Apps — and app delivery leaves Terraform for good.
• Argo CD reconciles continuously (prune + selfHeal); Terraform reconciles only at apply. That difference is the entire reason for the handoff — continuous drift correction is Argo’s job, not Terraform’s.
• Destroy the apps stack before the cluster stack, and let Argo prune its children first, or helm_release uninstalls fail Kubernetes cluster unreachable.