Terraform on AWS EKS: Metrics Server, Horizontal & Vertical Pod Autoscaling

Your EKS cluster survives the quiet hours fine on two replicas. Then marketing sends the email, a product goes viral, or the 9 a.m. batch fires — and the same two pods pin CPU at 100%, p99 latency triples, and the pods that should have absorbed the surge never appeared because nothing was watching. Adding nodes doesn’t help: Karpenter and the Cluster Autoscaler only add nodes when there are unschedulable pods, and nothing is creating pods. The missing piece is the workload scaling layer — the controllers that watch each Deployment’s live resource use and change how many pods run (and how big each one is), which in turn creates the scheduling pressure that makes node autoscaling do its job.

This lesson builds that layer, end to end, in Terraform. Three controllers, three axes. Metrics Server publishes the resource-metrics API (metrics.k8s.io) that both kubectl top and every CPU/memory HPA depend on — install it and nothing scales, so it comes first. The Horizontal Pod Autoscaler (HPA, autoscaling/v2) changes the number of pods to hold a target utilisation, with behavior policies that decide how fast to add and how slowly to remove them. The Vertical Pod Autoscaler (VPA) changes the size of each pod — its CPU/memory requests — so you stop guessing at resources.requests. You will meet the two traps that sink most teams: resource requests are mandatory for a CPU-percent HPA (skip them and the HPA target reads <unknown> forever), and HPA and VPA fighting over the same metric oscillates a workload into the ground.

Everything is real HCL and YAML you can paste. We install Metrics Server and the VPA controller with helm_release, define a Deployment with proper requests, attach a native kubernetes_horizontal_pod_autoscaler_v2 with a full behavior block, and apply a recommend-only VPA — deliberately using kubectl_manifest for the VPA custom resource so you feel the plan-time CRD problem that breaks kubernetes_manifest. Then we drive a load generator and watch kubectl get hpa climb, new pods schedule, and — because pods go Pending — Karpenter provision a node underneath. That last hop is the whole point: HPA adds pods → the scheduler runs out of room → the node autoscaler adds capacity. For the node half of the story, this lesson pairs directly with the Cluster Autoscaler & Karpenter lesson; for the custom-metric path it builds on the EKS observability lesson; and the provider wiring assumes the Kubernetes/Helm providers lesson.

What you’ll build

The scenario is a single stateless web Deployment that you want to scale on CPU, plus a recommender that tells you whether your requests are right. You already have an EKS cluster (managed node group or Karpenter — this layer sits on top of either). Onto it, Terraform lays down four things: Metrics Server (so a resource-metrics API exists), the workload and its Service, an HPA that holds 50% average CPU by adding pods between 2 and 20, and a VPA in Off mode that watches the same workload and prints right-sized request recommendations without touching anything. You then generate load, watch the horizontal scale-out, and — if the cluster runs out of allocatable CPU — watch the node autoscaler react.

Why Terraform rather than kubectl apply or a Helm umbrella chart? Because this scaling policy is infrastructure, not a one-off: the HPA bounds, the behavior windows, the VPA guardrails and the Metrics Server version are exactly the sort of thing that drifts between environments and gets “fixed” live at 2 a.m. never to be written down. In Terraform they are reviewed in a pull request, pinned by version, and identical in dev and prod. The one subtlety — and it is the source of half the confusion in this topic — is that the HPA owns the Deployment’s replica count, so Terraform must be told not to manage replicas, or every apply will fight the HPA back down to its baseline. We handle that with lifecycle { ignore_changes }, and it is one of the most important lines in the whole lesson.

Here is the whole build at a glance, and who owns each moving part:

A one-line way to keep the three autoscalers straight: HPA changes how many, VPA changes how big, Karpenter changes how much cluster. Get all three wrong and you over-provision three times over; get them composed and the cluster breathes with load and cost tracks demand.

The two axes of pod autoscaling (and the third, nodes)

Autoscaling in Kubernetes is not one feature; it is three controllers operating on three independent axes, and the single most common design mistake is conflating them. The horizontal axis (HPA) answers “how many replicas?” The vertical axis (VPA) answers “how big should each replica’s requests be?” The cluster/node axis (Cluster Autoscaler or Karpenter) answers “do we have enough nodes to place the pods the other two produced?” They do not replace each other — a mature platform runs all three, each on the signal it is good at.

The composition is a chain, and the diagram below traces it left to right. Metrics Server publishes usage; the HPA reads it and drives the Deployment’s replica count; if the new pods can’t fit on the current nodes they go Pending; Karpenter (or the Cluster Autoscaler) sees Pending pods and provisions nodes; meanwhile the VPA watches the same workload and reports (or applies) better request sizes. Each hop is a separate controller with its own reconcile loop, and the numbered badges call out the six decisions that most often go wrong.

Read it as a pipeline: install → measure → decide → place. The failure modes cluster at the seams — no Metrics Server means the “measure” stage is empty and the HPA reads <unknown>; missing requests means the “decide” stage can’t compute a percentage; and a VPA in Auto mode fighting the HPA turns “decide” into an oscillation. The rest of the lesson is those seams in detail.

Metrics Server: the resource-metrics API everything depends on

Metrics Server is not installed on EKS by default, and without it CPU/memory HPAs and kubectl top simply do not work. It is a cluster-wide component that scrapes the kubelet Summary API on every node (CPU and memory, sampled ~every 15s), keeps only the latest value in memory (it is not a monitoring database — no history, no disk), and serves it through the Kubernetes aggregation layer as the metrics.k8s.io API. The HPA controller and kubectl top are clients of exactly that API. Prometheus is a different thing entirely: it stores history and powers dashboards and custom metrics, but the resource HPA and kubectl top want metrics.k8s.io, which only Metrics Server (or an equivalent) provides.

There are three metrics APIs in Kubernetes, and knowing which one a given HPA uses tells you which component must be healthy:

Install it with helm_release. The chart is first-party (kubernetes-sigs), and on EKS you almost always need --kubelet-insecure-tls because the kubelet’s serving certificate is self-signed and not approved by the cluster CA — without the flag Metrics Server logs x509: certificate signed by unknown authority and never becomes ready. This is the single most common EKS-specific Metrics Server failure.

resource "helm_release" "metrics_server" {
  name       = "metrics-server"
  repository = "https://kubernetes-sigs.github.io/metrics-server/"
  chart      = "metrics-server"
  version    = "3.12.2" # pin it — never track "latest"
  namespace  = "kube-system"

  # EKS kubelet serving certs are self-signed; allow Metrics Server to scrape them.
  set {
    name  = "args[0]"
    value = "--kubelet-insecure-tls"
  }

  # Sensible defaults; the chart already sets a resource-metrics resolution of 15s.
  set {
    name  = "replicas"
    value = "2" # HA: two replicas so a node roll doesn't blind your HPAs
  }
}

Prefer the chart’s set blocks over hand-rolling a Deployment: the chart wires the APIService registration, RBAC and the aggregation-layer plumbing correctly. The Helm arguments worth knowing:

Verification is three commands, and you should treat “the APIService reports Available” as the gate before you even look at an HPA:

# 1. The aggregated API is registered and healthy?
kubectl get apiservice v1beta1.metrics.k8s.io
# NAME                    SERVICE                      AVAILABLE   AGE
# v1beta1.metrics.k8s.io  kube-system/metrics-server   True        2m

# 2. Node-level metrics flow?
kubectl top nodes
# NAME                          CPU(cores)  CPU%  MEMORY(bytes)  MEMORY%
# ip-10-0-1-23.ec2.internal     142m        7%    712Mi          19%

# 3. Pod-level metrics flow (namespace of your workload)?
kubectl top pods -n demo
# NAME                   CPU(cores)  MEMORY(bytes)
# web-6c8f...-abcde      248m        41Mi

Horizontal Pod Autoscaler (autoscaling/v2)

The HPA is a control loop in the controller-manager that, on a fixed interval (15s on upstream Kubernetes; on EKS the controller-manager flags are AWS-managed, so you cannot change the global sync period or tolerance — you tune per-HPA behavior instead), compares a live metric to a target and resizes the Deployment. The core formula is worth memorising because it explains every scaling decision you will ever debug:

desiredReplicas = ceil[ currentReplicas × ( currentMetricValue / desiredMetricValue ) ]

If eight pods average 80% CPU and the target is 50%, ceil(8 × 80/50) = ceil(12.8) = 13 replicas. A 10% tolerance (default) means the controller ignores ratios within 1 ± 0.1, so it doesn’t churn on noise. Always use autoscaling/v2 — v1 only supported CPU and had no behavior block; v2 supports multiple metrics, memory, custom/external metrics, and the scale-velocity policies that keep production sane.

v2 supports four metric types, and choosing the right one is most of the design:

Resource requests are mandatory for a CPU-percent HPA ⚠️

This trap catches nearly everyone once. A Resource metric with target.type: Utilization is computed as currentUsage / requestedAmount. If the container has no resources.requests.cpu, there is no denominator, the HPA cannot compute a percentage, and kubectl get hpa shows the target as <unknown>/50% — forever. It never scales. The fix is not on the HPA; it is on the Deployment: set resources.requests.cpu (and .memory if you scale on memory). Utilization HPAs require requests; only the AverageValue variants can live without them.

The native HCL resource models v2 faithfully — this is the HPA for our demo, holding 50% average CPU between 2 and 20 replicas, with a behavior block we dissect next:

resource "kubernetes_horizontal_pod_autoscaler_v2" "web" {
  metadata {
    name      = "web"
    namespace = kubernetes_namespace.demo.metadata[0].name
  }

  spec {
    min_replicas = 2
    max_replicas = 20

    scale_target_ref {
      api_version = "apps/v1"
      kind        = "Deployment"
      name        = kubernetes_deployment.web.metadata[0].name
    }

    metric {
      type = "Resource"
      resource {
        name = "cpu"
        target {
          type                = "Utilization"
          average_utilization = 50
        }
      }
    }

    behavior {
      scale_up {
        stabilization_window_seconds = 0    # react immediately to spikes
        select_policy                = "Max"
        policy {
          type           = "Percent"
          value          = 100              # allow doubling...
          period_seconds = 15
        }
        policy {
          type           = "Pods"
          value          = 4                # ...or +4 pods, whichever is larger
          period_seconds = 15
        }
      }
      scale_down {
        stabilization_window_seconds = 300  # wait 5 min of calm before shrinking
        select_policy                = "Max"
        policy {
          type           = "Percent"
          value          = 50               # remove at most 50% of pods/min
          period_seconds = 60
        }
      }
    }
  }
}

The behavior block: scale-up fast, scale-down slow

behavior (a v2 feature) is how you stop an HPA from flapping. It has two sides — scaleUp and scaleDown — each with a stabilization window and a list of rate policies; selectPolicy (Max/Min/Disabled) picks among the policies. The universal production shape is asymmetric: scale up eagerly, scale down reluctantly, because a false scale-down during a lull followed by a real spike is far more expensive than holding a few extra pods.

The defaults matter because if you omit behavior you inherit them — and the default 5-minute downscale stabilization is why an HPA “won’t scale down” for people who never configured it:

Flapping (rapid up/down/up) is almost always a stabilization problem: widen scaleDown.stabilizationWindowSeconds, lower the scaleDown Percent, or set scaleDown.selectPolicy = "Disabled" for a workload that should only ever grow within a window (then reset on a schedule). Over-aggressive scale-up thrash is rarer but is tamed by a small scaleUp stabilization window or a Pods-capped step.

Beyond CPU: custom, external and KEDA

CPU and memory are proxies. The metric that actually reflects a web service’s load is requests per second or queue depth, and for those you go past resource metrics. Two paths:

KEDA (Kubernetes Event-Driven Autoscaling) is the advanced path most EKS teams reach for, because it scales on the queue or stream that is the work, and it can scale to zero when idle (a plain HPA has a floor of minReplicas ≥ 1). Under the hood KEDA creates and manages an HPA for the 1→N range and handles 0→1 itself. Its scalers cover the AWS event surface:

A worker that scales on an SQS backlog, from zero to thirty, driven by KEDA and authenticated with IRSA (so no static AWS keys live in the pod), looks like this — apply it with kubectl_manifest for the same CRD-timing reason we cover shortly:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker
  namespace: demo
spec:
  scaleTargetRef:
    name: worker            # the Deployment
  minReplicaCount: 0        # scale to zero when the queue is empty
  maxReplicaCount: 30
  cooldownPeriod: 120
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.ap-south-1.amazonaws.com/123456789012/jobs
        queueLength: "5"    # target ~5 messages per replica
        awsRegion: ap-south-1
      authenticationRef:
        name: keda-aws-irsa # a TriggerAuthentication bound to an IRSA role

For the RPS-on-Prometheus route, wire the EKS observability lesson’s Prometheus, add the Prometheus Adapter via helm_release, and give the HPA a Pods metric named after your recording rule (e.g. http_requests_per_second).

Vertical Pod Autoscaler (VPA)

Where the HPA answers “how many pods?”, the VPA answers “how big should each pod be?” — it observes historical CPU/memory usage and recommends (or sets) the container’s requests. That solves the request-guessing problem that plagues every cluster: requests set too high waste money and starve the scheduler; too low and pods get CPU-throttled or OOM-killed. The VPA is not built into Kubernetes — you install its three controllers (usually the Fairwinds chart) plus its CRDs.

The behaviour hinges entirely on updateMode, and the safe default for learning and for HPA-managed workloads is Off (recommend only — nothing is evicted or changed):

Install the controller with Helm, then declare a recommend-only VPA for the workload. Note the guardrails — minAllowed/maxAllowed cap what the recommender may suggest so a runaway sample can’t ask for a 32-core pod:

resource "helm_release" "vpa" {
  name       = "vpa"
  repository = "https://charts.fairwinds.com/stable"
  chart      = "vpa"
  version    = "4.5.0"
  namespace  = "vpa"
  create_namespace = true

  set {
    name  = "recommender.enabled"
    value = "true"
  }
  # Keep the updater OFF cluster-wide while learning; per-VPA updateMode still governs.
  set {
    name  = "updater.enabled"
    value = "false"
  }
}

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web
  namespace: demo
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  updatePolicy:
    updateMode: "Off"          # recommend only — safe alongside the CPU HPA
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed: { cpu: 50m,  memory: 64Mi }
        maxAllowed: { cpu: "1",  memory: 512Mi }
        controlledResources: ["cpu", "memory"]

Read the recommendation with kubectl describe vpa web -n demo — the Target is what the VPA would set, Lower/Upper Bound the range it’s confident in. In Off mode you take that number and update the Deployment’s requests yourself (ideally in the same Terraform), getting VPA’s right-sizing without its disruption.

The HPA + VPA conflict — and how to combine them ⚠️

Do not run a VPA in Auto/Recreate mode on the same resource an HPA scales on. The reason is mechanical: the HPA’s Utilization target is usage / request. If the VPA changes the request (the denominator) at the same time the HPA is reacting to usage (the numerator), the two loops chase each other — the VPA raises requests, utilisation drops, the HPA removes pods, per-pod load rises, the VPA raises requests again — and the workload oscillates. The official guidance is blunt: VPA (in an updating mode) and a CPU/memory HPA must not target the same metric.

So the two production-safe patterns are: (1) HPA on CPU (or a custom metric) for count, VPA in Off mode for guidance you apply deliberately; or (2) HPA on a business metric like RPS or queue depth (via Prometheus Adapter or KEDA) for count, and VPA in Auto for size — because now the two controllers read genuinely independent signals. Reach for pattern (2) when you’ve outgrown CPU-as-a-proxy and want each axis fully automated.

Managing Kubernetes objects from Terraform

You have three ways to push a Kubernetes object from Terraform, and picking wrong is why VPA and KEDA installs mysteriously fail on the first apply. The decision is really about CRDs and timing:

The plan-time CRD problem ⚠️

kubernetes_manifest performs a server-side lookup of the resource’s GVK during terraform plan. If the CRD does not yet exist — because the Helm release that installs it is also in this plan — the plan fails with no matches for kind "VerticalPodAutoscaler" / cannot create REST mapping. You cannot install a CRD and create a custom resource of that CRD in a single kubernetes_manifest-based apply. Three ways out:

1. Two-stage apply — terraform apply -target=helm_release.vpa first (installs the CRD), then a normal apply for the CR. Correct but breaks single-command automation.
2. kubectl_manifest (gavinbunney/kubectl) — it defers the API call to apply time, so a CRD created earlier in the same apply is visible when the CR is created. This is why our demo uses it for the VPA custom resource.
3. Bundle in one Helm chart — ship the controller and the CRs together so Helm orders them.

This is the concrete reason the demo mixes providers: native resource for the HPA (a built-in kind, best diff), helm for Metrics Server and the VPA controller, and kubectl_manifest for the VPA custom resource (its CRD is born in the same apply). It’s not inconsistency — it’s each tool on the job it’s correct for.

Hands-on: build it with Terraform

We now assemble the whole layer and run it end to end against a real EKS cluster, then generate load and watch it scale. ⚠️ This runs real pods and may trigger EC2 node provisioning — small but real spend. Destroy when done.

The layout — one root module, provider auth to EKS, four workload files:

versions.tf — pin everything; the S3 backend + DynamoDB lock is the standard pattern from the AWS getting-started lesson:

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws        = { source = "hashicorp/aws",        version = "~> 5.60" }
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.31" }
    helm       = { source = "hashicorp/helm",       version = "~> 2.14" }
    kubectl    = { source = "gavinbunney/kubectl",  version = "~> 1.14" }
  }
  backend "s3" {
    bucket         = "kloudvin-tfstate"
    key            = "eks/autoscaling/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "kloudvin-tf-lock" # or S3-native lockfile on TF 1.10+
    encrypt        = true
  }
}

providers.tf — authenticate the Kubernetes-family providers to the EKS API. Two patterns: the aws_eks_cluster_auth token (simple, but the token expires in ~15 min — fine for local runs, risky for long CI applies) or the exec plugin (calls aws eks get-token at apply time — the CI-safe choice). We show the exec form; the providers lesson covers the trade-offs in depth.

data "aws_eks_cluster" "this"      { name = var.cluster_name }

locals {
  eks_host = data.aws_eks_cluster.this.endpoint
  eks_ca   = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  eks_exec = {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", var.cluster_name, "--region", var.region]
  }
}

provider "kubernetes" {
  host                   = local.eks_host
  cluster_ca_certificate = local.eks_ca
  exec {
    api_version = local.eks_exec.api_version
    command     = local.eks_exec.command
    args        = local.eks_exec.args
  }
}

provider "helm" {
  kubernetes {
    host                   = local.eks_host
    cluster_ca_certificate = local.eks_ca
    exec {
      api_version = local.eks_exec.api_version
      command     = local.eks_exec.command
      args        = local.eks_exec.args
    }
  }
}

provider "kubectl" {
  host                   = local.eks_host
  cluster_ca_certificate = local.eks_ca
  load_config_file       = false
  exec {
    api_version = local.eks_exec.api_version
    command     = local.eks_exec.command
    args        = local.eks_exec.args
  }
}

variables.tf:

variable "cluster_name" { type = string }
variable "region"       {
  type = string
  default = "ap-south-1"
}
variable "namespace"    {
  type = string
  default = "demo"
}

variable "hpa" {
  type = object({
    min_replicas       = number
    max_replicas       = number
    target_cpu_percent = number
  })
  default = { min_replicas = 2, max_replicas = 20, target_cpu_percent = 50 }
}

metrics.tf — Metrics Server and the VPA controller (the helm_release blocks shown earlier). workload.tf — the workload; note the two load-bearing details: resources.requests.cpu (without it the HPA reads <unknown>) and lifecycle { ignore_changes = [spec[0].replicas] } (without it every apply reverts the HPA’s replica count):

resource "kubernetes_namespace" "demo" {
  metadata { name = var.namespace }
}

resource "kubernetes_deployment" "web" {
  metadata {
    name      = "web"
    namespace = kubernetes_namespace.demo.metadata[0].name
    labels    = { app = "web" }
  }
  spec {
    replicas = var.hpa.min_replicas          # initial only — HPA owns it after
    selector { match_labels = { app = "web" } }
    template {
      metadata { labels = { app = "web" } }
      spec {
        container {
          name  = "web"
          image = "registry.k8s.io/hpa-example" # classic php-apache CPU burner
          port { container_port = 80 }
          resources {
            requests = { cpu = "250m", memory = "128Mi" } # ⚠️ mandatory for CPU% HPA
            limits   = { cpu = "500m", memory = "256Mi" }
          }
        }
      }
    }
  }
  lifecycle {
    ignore_changes = [spec[0].replicas] # ⚠️ let the HPA, not Terraform, set replicas
  }
}

resource "kubernetes_service" "web" {
  metadata{
    name = "web"
    namespace = kubernetes_namespace.demo.metadata[0].name
  }
  spec {
    selector = { app = "web" }
    port{
      port = 80
      target_port = 80
    }
  }
}

autoscaling.tf — the native HPA (shown in full earlier) plus the VPA custom resource via kubectl_manifest, with depends_on the controller so the CRD exists at apply time:

resource "kubectl_manifest" "vpa_web" {
  depends_on = [helm_release.vpa] # CRD is installed by the chart in this same apply
  yaml_body  = yamlencode({
    apiVersion = "autoscaling.k8s.io/v1"
    kind       = "VerticalPodAutoscaler"
    metadata   = { name = "web", namespace = var.namespace }
    spec = {
      targetRef    = { apiVersion = "apps/v1", kind = "Deployment", name = "web" }
      updatePolicy = { updateMode = "Off" } # recommend-only: safe next to the CPU HPA
      resourcePolicy = {
        containerPolicies = [{
          containerName       = "*"
          minAllowed          = { cpu = "50m", memory = "64Mi" }
          maxAllowed          = { cpu = "1",   memory = "512Mi" }
          controlledResources = ["cpu", "memory"]
        }]
      }
    }
  })
}

Run it: init → plan → apply → verify → load test → destroy

After apply, confirm the HPA is healthy — a real target, not <unknown>:

kubectl get hpa -n demo
# NAME  REFERENCE        TARGETS       MINPODS  MAXPODS  REPLICAS  AGE
# web   Deployment/web   cpu: 1%/50%   2        20       2         40s

Now drive load. The classic hpa-example image burns CPU on every request, so a tight wget loop from a throwaway pod pushes utilisation up fast. Open two more terminals to watch the HPA and the pods/nodes:

# terminal 1 — generate load
kubectl run -n demo load --image=busybox:1.36 --restart=Never -- \
  /bin/sh -c "while true; do wget -q -O- http://web; done"

# terminal 2 — watch the HPA decide
kubectl get hpa web -n demo -w
# web   ...  cpu: 1%/50%    2   20   2
# web   ...  cpu: 210%/50%  2   20   4     <- spike; scale_up (Percent 100 / +4)
# web   ...  cpu: 158%/50%  2   20   8
# web   ...  cpu:  74%/50%  2   20   11
# web   ...  cpu:  49%/50%  2   20   11    <- settled near target

# terminal 3 — watch pods, and nodes follow if the cluster fills
kubectl get pods,nodes -n demo -w

What you are watching is the whole chain fire. The HPA reads the spike and adds pods per the scaleUp policy (double, or +4, whichever is more). As replicas climb, the scheduler may run out of allocatable CPU; those pods sit Pending, and Karpenter (or the Cluster Autoscaler) sees the unschedulable pods and provisions a node — the composition the whole lesson builds toward. Delete the load pod (kubectl delete pod load -n demo) and, after the 5-minute scaleDown stabilization window, the HPA walks replicas back toward 2, and shortly after Karpenter consolidates the now-empty node.

Variables, outputs & making it reusable

The demo is one root module; the reusable form is a small module you stamp per workload so every team gets the same HPA guardrails. Parameterise the bounds, the target, and the behavior windows; expose the HPA and VPA names as outputs so callers can assert on them.

module "web_autoscaling" {
  source              = "./modules/pod-autoscaling"
  name                = "web"
  namespace           = "demo"
  min_replicas        = 2
  max_replicas        = 20
  target_cpu_percent  = 50
  vpa_update_mode     = "Off" # recommend-only while an HPA owns CPU
}

output "hpa_name" { value = module.web_autoscaling.hpa_name }

For the controllers (Metrics Server, VPA, KEDA), prefer community modules over hand-rolled helm_release when you want IRSA, sane defaults and lifecycle handled for you — the terraform-aws-modules/eks/aws ecosystem and its eks-blueprints-addons module install Metrics Server, KEDA and friends as first-class add-ons with the IRSA roles wired. Roll your own helm_release (as here) when you need tight control over chart values or are pinning an exact version for a compliance baseline. A for_each over a map of workloads turns the module into a fleet-wide policy — every Deployment in a list gets the same HPA shape, reviewed in one place.

Common mistakes and troubleshooting

The nastiest real gotchas, in prose. The <unknown> target is a two-suspect mystery and you must eliminate both: no CPU request on the container, or Metrics Server not serving. Check kubectl top pods -n <ns> first — if that fails it’s Metrics Server; if top works but the HPA still reads <unknown>, it’s the missing request. The ignore_changes line is not optional. Because the HPA writes spec.replicas and Terraform also thinks it owns that field, without ignore_changes = [spec[0].replicas] every plan shows a spurious diff and every apply yanks the workload back to min_replicas, undoing the HPA between runs — a classic “why did prod suddenly drop to 2 pods during the deploy” incident. HPA+VPA on the same metric is a footgun, not a feature: if you want both fully automated, move the HPA onto a business metric (RPS via Prometheus Adapter, or a queue via KEDA) so the two controllers read independent signals; otherwise keep VPA in Off. VPA Auto mode evicts — it right-sizes by killing and recreating pods, which is disruptive for anything without a PodDisruptionBudget and graceful shutdown; on singletons it causes visible downtime. And the plan-time CRD failure is not a bug in your YAML — kubernetes_manifest genuinely cannot see a CRD that this same run installs, so use kubectl_manifest (defers to apply) or split the apply.

Cost, cleanup & production notes

The scaling controllers are nearly free — Metrics Server and the VPA recommender are small pods (tens of millicores, ~50–100Mi each). The cost is the workload they scale and the nodes underneath. An HPA that runs 2 pods at rest and 20 under load, on m6i capacity, is the difference between one small node idle and several nodes busy; the whole point is that spend tracks demand instead of being provisioned for peak 24/7. The line-item risks: a runaway maxReplicas with no upper node bound (an HPA + Karpenter with no ceiling will happily scale a bad loop into a large EC2 bill), and a VPA in Auto that ratchets requests up and forces bigger nodes.

Destroy cleanly: terraform destroy removes the HPA, VPA CR, Service, Deployment and both Helm releases. ⚠️ If a load-test pod is still running (kubectl run load ...), delete it first — it isn’t in Terraform state and will keep pods (and nodes) warm. After destroy, confirm kubectl get hpa,vpa,deploy -n demo is empty and that Karpenter has consolidated any nodes the test spun up.

Production hardening: (1) pin every chart version and every provider ~> — an unpinned Metrics Server upgrade can change the metrics pipeline under your HPAs. (2) Always cap maxReplicas and the node autoscaler’s limits so a bug can’t scale into a five-figure bill. (3) Give every HPA a behavior block — never ship the defaults into prod, because the 5-minute default downscale and the doubling scale-up are opinions you should make explicit. (4) Set resources.requests deliberately (use VPA Off recommendations as the input) — they are the denominator of every Utilization HPA and the currency of the scheduler. (5) Run Metrics Server with 2 replicas so a node roll doesn’t blind every HPA at once. (6) Keep the HPA’s replica ownership clean with ignore_changes, and prefer exec-based provider auth so long CI applies don’t die on an expired token.

Cheat-sheet

Resources & providers

HPA spec quick-ref

Verify & load-test commands

kubectl get apiservice v1beta1.metrics.k8s.io   # Metrics Server AVAILABLE=True
kubectl top nodes ; kubectl top pods -n demo    # metrics flowing?
kubectl get hpa -n demo                          # TARGETS not <unknown>
kubectl describe hpa web -n demo                 # events + current metrics
kubectl describe vpa web -n demo                 # Target recommendation
kubectl run -n demo load --image=busybox:1.36 --restart=Never -- \
  /bin/sh -c "while true; do wget -q -O- http://web; done"   # generate load
kubectl get hpa web -n demo -w                   # watch it scale
kubectl delete pod load -n demo                  # stop load

Interview and exam questions

1. Why does an HPA show <unknown> for its CPU target, and what are the two possible causes? The Utilization target is usage / request; it reads <unknown> if there is no CPU request on the container (no denominator) or if Metrics Server isn’t serving metrics.k8s.io. Diagnose with kubectl top pods — if that fails it’s Metrics Server, else it’s the missing request.

2. Is Metrics Server installed on EKS by default? What does it provide? No. It serves the resource-metrics API (metrics.k8s.io) — CPU/memory — that kubectl top and CPU/memory HPAs consume. It keeps only the latest sample in memory; it is not a time-series store.

3. State the HPA scaling formula. desiredReplicas = ceil[currentReplicas × (currentMetricValue / desiredMetricValue)], subject to a ~10% tolerance and the min/maxReplicas bounds and behavior rate limits.

4. What does the behavior block do, and what’s the standard shape? It rate-limits and stabilizes scaling. Standard shape is asymmetric: scaleUp fast (0s window, double or +4 pods), scaleDown slow (300s stabilization, ≤50%/min) — cheap to hold spare pods, expensive to shed too early.

5. HPA vs VPA vs Cluster Autoscaler/Karpenter — one line each. HPA changes replica count on live utilisation; VPA changes per-pod requests on historical usage; Karpenter/CA changes node count in response to unschedulable (Pending) pods.

6. Why can’t you run VPA Auto and a CPU HPA on the same workload? The VPA changes the request (the HPA’s Utilization denominator) while the HPA reacts to usage (the numerator); the loops chase each other and oscillate. Combine safely by keeping VPA in Off, or by scaling the HPA on a custom/external metric so the signals are independent.

7. Explain the plan-time CRD problem with kubernetes_manifest. kubernetes_manifest resolves the resource’s GVK during plan; if the CRD is installed by a helm_release in the same run, the CRD doesn’t exist at plan time and the plan fails. Use kubectl_manifest (defers to apply), a two-stage -target apply, or bundle CRD+CR in one chart.

8. Why must Terraform ignore the Deployment’s replicas? Because the HPA owns spec.replicas. Without lifecycle { ignore_changes = [spec[0].replicas] }, every apply reverts the workload to min_replicas, fighting the HPA. (Terraform Associate-style: which meta-argument prevents a resource attribute from causing drift? → lifecycle.ignore_changes.)

9. When would you choose KEDA over a plain CPU HPA? When the true load signal is an event source (SQS depth, Kafka lag) rather than CPU, or when you need scale-to-zero (a plain HPA floors at minReplicas ≥ 1). KEDA serves external.metrics.k8s.io and manages an HPA for 1→N while handling 0→1 itself.

10. How does workload scaling compose with node scaling? HPA adds pods → some pods can’t be scheduled → they go Pending → Karpenter/CA sees unschedulable pods and provisions nodes → pods schedule. Node scaling is driven by the pods the HPA (or VPA) creates.

11. What are VPA’s three components and the four update modes? Recommender (advises), Updater (evicts to resize), Admission controller (rewrites requests on creation). Modes: Off (recommend only), Initial (set at creation), Recreate/Auto (evict + recreate to apply).

12. Terraform Associate-style: which provider and resource model a Kubernetes HPA idiomatically? hashicorp/kubernetes provider, resource kubernetes_horizontal_pod_autoscaler_v2 (use v2, not the deprecated v1, to get memory/custom metrics and behavior).

Key takeaways

• Install Metrics Server first — CPU/memory HPAs and kubectl top are clients of metrics.k8s.io, which EKS does not ship; on EKS you almost always need --kubelet-insecure-tls.
• Resource requests are mandatory for a Utilization HPA — no request means no denominator means a permanent <unknown> target and zero scaling.
• Let the HPA own replicas — lifecycle { ignore_changes = [spec[0].replicas] } on the Deployment, or every apply fights the autoscaler back to the floor.
• Always set behavior — scale up fast, scale down slow (a wide scaleDown stabilization window is the cure for flapping); the defaults are opinions, so make them yours.
• VPA sizes, HPA counts, Karpenter capacitates — three axes, three controllers; keep VPA in Off next to a CPU HPA, or move the HPA to a custom metric before letting VPA run Auto.
• Pick the right Terraform tool per object — native resources for built-in kinds, helm_release for controllers and their CRDs, kubectl_manifest for custom resources whose CRD is born in the same apply (the plan-time CRD trap).
• The chain is the product — HPA adds pods, the scheduler runs out of room, Karpenter adds nodes; wire the Karpenter lesson to close the loop and cap maxReplicas and node limits so cost can’t run away.