Terraform on AWS EKS: Observability — CloudWatch Container Insights, Prometheus & Grafana

A cluster you cannot see into is a cluster you are operating on faith. The moment real workloads land on EKS, the questions stop being “did it apply?” and become “which pod is pinning a node, why did the p99 double at 14:02, and where did those 500s come from” — and none of those have an answer unless the metrics, logs and traces were wired up before the incident, not during it. Observability is infrastructure exactly like the VPC and the node group are infrastructure, which means it belongs in Terraform next to them: the add-on, the IAM role that lets an agent write to CloudWatch, the Helm release that installs Prometheus, the PVC its TSDB lives on, the alert rules and the SNS topic they page — all of it versioned, reviewed, and stamped identically into dev and prod from the same code.

This lesson builds that plane end to end, both ways, because EKS gives you a real choice. On the AWS-native side you install the CloudWatch Observability EKS add-on — one aws_eks_addon that drops the CloudWatch agent and Fluent Bit onto every node and lights up Container Insights with almost no moving parts, at the cost of CloudWatch’s per-GB log bill. On the CNCF side you install kube-prometheus-stack with a single helm_release — the Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics in one shot — and get the full open-source metrics ecosystem, at the cost of running (and storing, and scaling) it yourself. Most real platforms run both: Container Insights for the AWS-integrated view and CloudWatch alarms, kube-prometheus-stack for rich dashboards and PromQL. You will build each, then see the managed middle path — Amazon Managed Prometheus and Amazon Managed Grafana — that keeps Prometheus’s ergonomics while handing the HA storage to AWS.

This is the provider-specific, Kubernetes-flavoured layer of the KloudVin Terraform course. It assumes you already know core Terraform — HCL, providers, resources, variables, state and modules — and that you have an EKS cluster to point at (built in the earlier EKS lessons). It applies Terraform to EKS observability relentlessly, with copy-pasteable .tf files and a terraform init → plan → apply → verify → destroy you actually run.

What you’ll build

The scenario is the one every platform team hits the week after the cluster goes live: a handful of services are running on EKS, and the on-call rotation needs to see them — CPU and memory per pod and per node, container logs searchable in one place, the golden signals (latency, traffic, errors, saturation) on a dashboard, and a page when any of it degrades, through the same channel in every environment, with no one clicking around a console. That means an agent on every node collecting metrics and logs, a metrics database scraping the apps, dashboards the team actually opens, alert rules that fire on user impact, and a single notification path — an SNS topic — that fans a firing alert out to email, Slack or a pager.

Every piece of that is a Terraform resource. The AWS-native collector is aws_eks_addon (amazon-cloudwatch-observability) with an IAM role built from aws_iam_role + aws_iam_openid_connect_provider (IRSA) or aws_eks_pod_identity_association; the retained logs are aws_cloudwatch_log_group; the CNCF stack is one helm_release of kube-prometheus-stack; its storage is a gp3 StorageClass on the EBS CSI driver; the sample app’s scrape target is a kubernetes_manifest ServiceMonitor; the managed stores are aws_prometheus_workspace (AMP) and aws_grafana_workspace (AMG); and the alert fan-out is aws_sns_topic wired to a CloudWatch alarm or an Alertmanager receiver. Wiring this by hand — eksctl here, helm install there, an IAM role clicked in the console — gives you a snowflake nobody can reproduce; wiring it in Terraform gives you the dependency graph, a plan that shows the exact diff, and a module you apply to the next cluster by changing one variable.

Why Terraform rather than helm install and the console? Because the observability plane is precisely the kind of cross-referencing, multi-service configuration that rewards code: the add-on’s IAM role references the cluster’s OIDC issuer, the Helm values reference the AMP remote_write endpoint, the ServiceMonitor’s labels must match the Prometheus the Helm release created, and the alarm references the SNS topic ARN. Terraform orders all of that and shows it to you before it happens. And when the cluster is rebuilt — as clusters are, for upgrades and blue-green swaps — the entire observability plane comes back identically instead of being re-clicked from memory.

Reading that diagram left to right is reading the two paths you are about to build. Terraform provisions both planes onto the same EKS cluster whose pods and nodes emit metrics and logs. On the AWS-native path, the CloudWatch agent + Fluent Bit (installed by the add-on) push metrics and container logs into Container Insights log groups and the ContainerInsights metric namespace. On the CNCF path, kube-prometheus-stack installs Prometheus, which scrapes ServiceMonitor targets into a TSDB on an EBS PVC and feeds Grafana dashboards. Both paths end at alerting: a PrometheusRule through Alertmanager, or a CloudWatch alarm, publishes to an SNS topic that fans out to humans — while AMP + AMG stand ready as the managed swap-in that keeps Prometheus’s ergonomics without you running the storage.

Here is the full inventory a single run creates in the hands-on, and roughly what each part costs if you leave it running (ap-south-1 / Mumbai, on-demand, indicative July 2026):

The EKS cluster itself (control plane $0.10/hr ≈ ₹6,000/mo, plus node EC2) is the standing cost and is assumed to already exist; what this lesson adds is the collector, the Helm stack and its EBS volumes, and the CloudWatch log bill — the two lines that actually move the needle. It is still a build it, verify it, destroy it lesson, and every costly or destructive step below is marked ⚠️.

Where this fits: the PVCs the Helm stack needs are backed by the gp3 StorageClass and the EBS CSI driver stood up in the EKS EBS CSI & storage classes lesson; exposing Grafana on a real hostname with TLS uses the ALB Ingress controller, ExternalDNS and ACM from the EKS Ingress with ALB, SSL & ExternalDNS lesson; and the CloudWatch alarms → SNS pattern the AWS-native path leans on is covered in depth (composite alarms, metric filters, treat_missing_data) in the CloudWatch alarms, dashboards & SNS lesson.

The three pillars on EKS, and the AWS-native vs CNCF choice

Observability is conventionally three pillars — metrics (numeric time-series: CPU, request rate, latency), logs (discrete events: an error line, an access log), and traces (the path of one request across services) — and EKS gives you a distinct set of tools for each, split down an AWS-native vs open-source line. Getting the mental model straight first saves you from bolting on the wrong tool later.

Two clarifications that trip people up. First, kubectl top is not your monitoring system — it reads the lightweight Resource Metrics API served by metrics-server, which keeps only a few minutes of in-memory data to drive the HPA and top; it has no history and no alerting. It is a useful smoke test (and you install it as an add-on too), not a replacement for Container Insights or Prometheus. Second, the collector and the store are separate concerns: on the AWS path the CloudWatch agent + Fluent Bit collect and CloudWatch stores; on the CNCF path Prometheus both scrapes and stores (until you remote_write elsewhere). Confusing collector with store is why people ask “do I still need Prometheus if I have Fluent Bit?” — Fluent Bit is a log shipper, Prometheus is a metrics database; they are not alternatives.

Now the decision that shapes the whole build — Container Insights vs a self-run Prometheus stack:

There is no universally right answer, and the honest production pattern is both, scoped: Container Insights for cheap, no-ops cluster and node visibility plus CloudWatch alarms wired to the same SNS topic as everything else AWS; kube-prometheus-stack for application metrics, PromQL, rich Grafana dashboards and Alertmanager routing. The rest of this lesson builds each in turn so you can pick — or run both, which is what the hands-on does.

What actually emits the signals matters too, because a metric that no one produces cannot be scraped:

CloudWatch Container Insights: the observability EKS add-on

The AWS-native path is refreshingly small: one add-on. The CloudWatch Observability EKS add-on (amazon-cloudwatch-observability) installs the CloudWatch agent (as a DaemonSet, managed by a small operator) and Fluent Bit (also a DaemonSet) onto every node. The agent collects performance metrics and publishes them to the ContainerInsights CloudWatch namespace; Fluent Bit ships container logs to CloudWatch Logs. You get per-pod and per-node CPU/memory/network/disk, the Container Insights console maps, and (in enhanced mode) Kubernetes object state — without running a single collector pod yourself.

resource "aws_eks_addon" "cloudwatch_observability" {
  cluster_name  = data.aws_eks_cluster.this.name
  addon_name    = "amazon-cloudwatch-observability"
  addon_version = "v4.4.0-eksbuild.1"   # pin; list with `aws eks describe-addon-versions`

  # IRSA: the agent's service account assumes this role
  service_account_role_arn = aws_iam_role.cw_agent.arn

  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "PRESERVE"

  configuration_values = jsonencode({
    containerLogs = { enabled = true }
    agent = {
      config = {
        logs = {
          metrics_collected = {
            kubernetes = { enhanced_container_insights = true }
          }
        }
      }
    }
  })

  tags = local.tags
}

The add-on’s arguments are the same shape as any EKS add-on, with the observability specifics in configuration_values:

The one non-trivial dependency is permissions, because the agent needs to write to CloudWatch and it runs as a pod, which means a pod-scoped credential — either IRSA (IAM Roles for Service Accounts, the OIDC-federation classic) or EKS Pod Identity (the newer association-based model). Both attach the AWS-managed CloudWatchAgentServerPolicy to a role and bind it to the add-on’s service account (cloudwatch-agent in the amazon-cloudwatch namespace).

Here is the IRSA wiring — the OIDC provider (one per cluster) and a role whose trust policy only lets the agent’s service account assume it:

# 1) Register the cluster's OIDC issuer as an IAM identity provider (once per cluster)
data "tls_certificate" "oidc" {
  url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "oidc" {
  url             = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.oidc.certificates[0].sha1_fingerprint]
}

# 2) Trust policy: only cloudwatch-agent in amazon-cloudwatch may assume this role
data "aws_iam_policy_document" "cw_agent_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.oidc.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:amazon-cloudwatch:cloudwatch-agent"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc.url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "cw_agent" {
  name               = "${var.cluster_name}-cw-agent"
  assume_role_policy = data.aws_iam_policy_document.cw_agent_assume.json
}

resource "aws_iam_role_policy_attachment" "cw_agent" {
  role       = aws_iam_role.cw_agent.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

The trust policy is where every “the agent runs but no metrics appear” bug lives, so read it carefully. The two condition blocks are load-bearing: the :sub condition pins the exact service account (amazon-cloudwatch:cloudwatch-agent) so no other pod can borrow the role, and the :aud condition (sts.amazonaws.com) is required by the EKS OIDC flow — omit it and STS refuses the token exchange. If you would rather not manage an OIDC provider per cluster, the Pod Identity alternative drops it entirely:

resource "aws_eks_pod_identity_association" "cw_agent" {
  cluster_name    = data.aws_eks_cluster.this.name
  namespace       = "amazon-cloudwatch"
  service_account = "cloudwatch-agent"
  role_arn        = aws_iam_role.cw_agent.arn   # trust: pods.eks.amazonaws.com
}

with the role’s trust policy targeting the pods.eks.amazonaws.com service principal (sts:AssumeRole + sts:TagSession) and the eks-pod-identity-agent add-on installed on the cluster.

Where the signals land. The add-on creates and writes to a fixed set of log groups and one metric namespace. This is the map you keep next to you when something is missing:

And these are the metrics you actually alarm on — a small, high-value subset of the ContainerInsights namespace:

The ⚠️ retention cost. Fluent Bit is a firehose — a chatty cluster can push tens of GB a day of container logs — and the log groups the add-on creates default to never expire, which is the single most common surprise line on an EKS CloudWatch bill. The add-on manages the log-group creation, so the clean fix is either a retention_in_days in the add-on configuration_values, or you pre-create the groups with retention set (and let the add-on adopt them):

CloudWatch Logs bills on ingestion (~$0.50/GB, ~half for the Infrequent Access class), storage (~$0.03/GB-month), and Logs Insights analysis (scan-billed). Container Insights metrics are custom metrics (priced each). On a busy cluster the two levers that matter are turning off container-log collection you do not need (containerLogs.enabled = false, or scope it) and setting retention — both one-line decisions in Terraform, invisible in the console until the bill.

The CNCF stack: kube-prometheus-stack via helm_release

The open-source path installs the whole Prometheus ecosystem in one Helm chart. kube-prometheus-stack (from the prometheus-community repo) bundles the Prometheus Operator (which turns ServiceMonitor/PodMonitor/PrometheusRule custom resources into Prometheus config), Prometheus itself, Alertmanager, Grafana (pre-wired with a large library of cluster dashboards), node-exporter (a DaemonSet for machine metrics) and kube-state-metrics (cluster object state) — the complete stack, in one helm_release.

resource "helm_release" "kps" {
  name             = "kps"
  repository       = "https://prometheus-community.github.io/helm-charts"
  chart            = "kube-prometheus-stack"
  version          = "65.5.1"                # ⚠️ pin the chart version
  namespace        = "monitoring"
  create_namespace = true
  timeout          = 600                     # CRDs + many objects; give it room

  values = [templatefile("${path.module}/values/kps.yaml.tftpl", {
    storage_class   = var.storage_class      # gp3 from the EBS CSI lesson
    prom_pvc_size   = "50Gi"
    grafana_pvc     = "10Gi"
    amp_remote_write = var.amp_remote_write_url  # "" = disabled
    aws_region      = var.region
  })]

  # Secret value never goes through the values file / state as plaintext-in-template
  set_sensitive {
    name  = "grafana.adminPassword"
    value = random_password.grafana.result
  }
}

The chart’s own arguments on helm_release are the release plumbing; the content lives in values:

Managing Helm values from Terraform is its own small discipline, because a real values file is dozens of nested keys and you want it templated, reviewable and secret-safe. Three techniques, and when each wins:

Keep the big, static structure in a templatefile YAML you can lint and diff; inject the few Terraform-computed values (the StorageClass name, the AMP endpoint) as template variables; and route any secret through set_sensitive so it never appears in plan output or the values file. The template (values/kps.yaml.tftpl) carries the values that make the stack production-shaped — persistence, scrape scope, and remote-write:

# values/kps.yaml.tftpl  (rendered by templatefile)
grafana:
  persistence:
    enabled: true
    storageClassName: ${storage_class}
    size: ${grafana_pvc}
  service:
    type: ClusterIP            # exposed via Ingress, not a LoadBalancer per pod
prometheus:
  prometheusSpec:
    retention: 15d
    # Honour ServiceMonitors/PodMonitors in ALL namespaces, not only labelled ones
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: ${storage_class}
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: ${prom_pvc_size}
%{ if amp_remote_write != "" ~}
    remoteWrite:
      - url: ${amp_remote_write}
        sigv4:
          region: ${aws_region}
%{ endif ~}
alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: ${storage_class}
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

The values worth knowing by name, because these are the ones you tune on every real install:

Persistence is not optional, and it ties straight to the EBS CSI driver. Prometheus’s TSDB and Grafana’s state are stateful; the volumeClaimTemplate/persistence blocks above create PersistentVolumeClaims that a StorageClass must satisfy. On EKS that StorageClass is gp3 backed by the EBS CSI driver — and if the driver is not installed, or there is no default gp3 StorageClass, the PVCs sit Pending forever and the Prometheus and Grafana pods never start.

The gp3 StorageClass and the EBS CSI add-on are exactly what the EKS EBS CSI & storage classes lesson builds — this lesson assumes they exist and simply names the class in the values.

Scraping and ServiceMonitors. You do not hand-edit prometheus.yml; you create a ServiceMonitor and the Operator generates the scrape config. The one rule that catches everyone is two-level selection: the Prometheus resource has a serviceMonitorSelector (which we set to honour all monitors via the ...NilUsesHelmValues: false values), and each ServiceMonitor selects the Services it scrapes. A ServiceMonitor whose labels no Prometheus selects is silently ignored. Here is one for a sample app, authored as a Terraform-managed manifest:

resource "kubernetes_manifest" "sample_servicemonitor" {
  manifest = {
    apiVersion = "monitoring.coreos.com/v1"
    kind       = "ServiceMonitor"
    metadata = {
      name      = "sample-app"
      namespace = "monitoring"
      labels    = { release = "kps" }   # match the Prometheus serviceMonitorSelector
    }
    spec = {
      selector      = { matchLabels = { app = "sample-app" } }
      namespaceSelector = { matchNames = ["demo"] }
      endpoints = [{
        port     = "metrics"   # the Service port NAME, never a number
        path     = "/metrics"
        interval = "15s"
      }]
    }
  }
  depends_on = [helm_release.kps]   # CRDs must exist before this manifest
}

Exposing Grafana is the last mile, and on EKS the idiomatic path is an Ingress handled by the AWS Load Balancer Controller, with ExternalDNS creating the Route 53 record and ACM terminating TLS on the ALB — exactly the machinery from the EKS Ingress with ALB, SSL & ExternalDNS lesson. You turn it on in the Grafana values:

Managed metrics: Amazon Managed Prometheus and Grafana

Running Prometheus and Grafana yourself means owning their storage, HA and scaling — real work at real scale. The managed middle path keeps Prometheus’s ergonomics (PromQL, the exposition format, ServiceMonitors) while handing the hard parts to AWS. Amazon Managed Service for Prometheus (AMP) is a managed, horizontally-scaling Prometheus-compatible store; your in-cluster Prometheus keeps scraping but remote_writes the samples to it. Amazon Managed Grafana (AMG) is managed Grafana with SSO, using AMP (or CloudWatch, or X-Ray) as a data source.

# AMP: a managed metrics workspace
resource "aws_prometheus_workspace" "this" {
  alias = "${var.cluster_name}-metrics"
  tags  = local.tags
}

# AMG: a managed Grafana workspace (IAM Identity Center auth)
resource "aws_grafana_workspace" "this" {
  name                     = "${var.cluster_name}-grafana"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["AWS_SSO"]
  permission_type          = "SERVICE_MANAGED"
  data_sources             = ["PROMETHEUS", "CLOUDWATCH", "XRAY"]
  role_arn                 = aws_iam_role.grafana.arn
}

output "amp_remote_write_url" {
  value = "${aws_prometheus_workspace.this.prometheus_endpoint}api/v1/remote_write"
}

The amp_remote_write_url output is what you feed into the Helm values’ remoteWrite.url (shown earlier) — the in-cluster Prometheus then signs each write request with SigV4 (region-scoped, using the Prometheus service account’s IRSA role, which needs aps:RemoteWrite). The three AMP resources and what they hold:

The full self-vs-managed decision, the table you actually make the call from:

The pragmatic middle that many teams land on: run kube-prometheus-stack for scraping and Grafana dashboards, but remote_write to AMP for durable, HA long-term storage — you keep the rich local experience and offload the storage you least want to operate. AMG then reads AMP for the dashboards leadership looks at, with SSO you didn’t have to build.

Traces, and alerting to SNS

Traces are the third pillar, and on EKS the collector is ADOT — the AWS Distro for OpenTelemetry — available as its own EKS add-on (adot) or run as an OpenTelemetry Collector you deploy. Apps emit OTLP spans; the collector batches and exports them to X-Ray (AWS-native) or to Tempo/Jaeger (CNCF), and can also export OTLP metrics to AMP — one collector, all three pillars. It is a lesson of its own; the shape to remember:

Alerting is where both paths converge on the same SNS topic, and the SNS + alarm mechanics (composite alarms, treat_missing_data, topic policy) are covered in depth in the CloudWatch alarms & SNS lesson. The two ways an EKS alert reaches a human:

A CloudWatch alarm on the AWS path is the exact same resource you already know, just pointed at a Container Insights metric:

resource "aws_cloudwatch_metric_alarm" "node_disk" {
  alarm_name          = "${var.cluster_name}-node-disk-high"
  namespace           = "ContainerInsights"
  metric_name         = "node_filesystem_utilization"
  statistic           = "Maximum"
  period              = 60
  evaluation_periods  = 5
  datapoints_to_alarm = 3
  threshold           = 85
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "breaching"          # a silent node metric IS the failure
  dimensions          = { ClusterName = var.cluster_name }
  alarm_actions       = [aws_sns_topic.alerts.arn]
  ok_actions          = [aws_sns_topic.alerts.arn]
}

On the CNCF side, a PrometheusRule expresses the same intent in PromQL and Alertmanager routes it — to Slack, PagerDuty, or SNS via sns_configs (which signs with SigV4 using the Alertmanager pod’s IRSA role and sns:Publish). The kube-prometheus-stack chart ships a broad default rule set (node pressure, pod crash-loops, PVC filling, API latency) out of the box, so you get meaningful alerts on install and add your app-specific PrometheusRules on top.

Hands-on: build it with Terraform

Now the centrepiece — a complete, self-contained configuration you run end to end against a pre-existing EKS cluster. It builds both paths so you can compare them: the AWS-native Container Insights add-on (with its IRSA role), and the CNCF kube-prometheus-stack (with EBS-backed persistence, a Grafana password from random_password, a ServiceMonitor scraping a sample app, and a Grafana dashboard). You will apply, verify with kubectl top and by opening Grafana, then destroy. ⚠️ This creates EBS volumes and CloudWatch log ingestion — do the whole loop in one sitting.

Prerequisites. An EKS cluster you can reach (aws eks update-kubeconfig --name <cluster> works and kubectl get nodes returns nodes), the EBS CSI driver add-on installed with a gp3 StorageClass (from the EBS CSI lesson), and metrics-server (for kubectl top). We read the cluster as data sources, so this config never risks the cluster itself.

Step 0 — layout.

eks-observability/
├── versions.tf
├── variables.tf
├── providers.tf
├── cloudwatch-addon.tf     # AWS-native path
├── kube-prometheus.tf      # CNCF path
├── sample-app.tf           # a scrape target + ServiceMonitor
├── alerts.tf               # SNS + a CloudWatch alarm
├── outputs.tf
├── values/kps.yaml.tftpl
└── terraform.tfvars

Step 1 — versions.tf (pins; the helm/kubernetes versions match the rest of the course).

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws        = { source = "hashicorp/aws",        version = "~> 5.60" }
    helm       = { source = "hashicorp/helm",       version = "~> 2.17" }
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.35" }
    tls        = { source = "hashicorp/tls",        version = "~> 4.0" }
    random     = { source = "hashicorp/random",     version = "~> 3.6" }
  }
  backend "s3" {
    bucket         = "kloudvin-tfstate-apsouth1"
    key            = "labs/eks-observability/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "kloudvin-tf-locks"
    encrypt        = true
  }
}

Step 2 — variables.tf.

variable "region"        {
  type = string
  default = "ap-south-1"
}
variable "cluster_name"  { type = string }                         # your EKS cluster
variable "storage_class" {                                         # from the EBS CSI lesson
  type    = string
  default = "gp3"
}
variable "alarm_email"   { type = string }                         # SNS confirmation target
variable "amp_remote_write_url" {                                  # "" disables remote_write
  type    = string
  default = ""
}

Step 3 — providers.tf (the EKS auth chain — the exec block gets a fresh token via aws eks get-token).

provider "aws" {
  region = var.region
  default_tags { tags = local.tags }
}

locals {
  tags = { Project = "eks-obs", ManagedBy = "Terraform", Lesson = "eks-observability" }
}

data "aws_eks_cluster" "this"      { name = var.cluster_name }
data "aws_eks_cluster_auth" "this" { name = var.cluster_name }

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name]
  }
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.this.name]
    }
  }
}

⚠️ helm provider version note: the nested kubernetes { ... } block shown is the helm 2.x syntax the rest of this course uses. The helm 3.x provider (2025) flattens this — if you pin ~> 3.0, move the connection settings to the provider’s top level and switch set/set_sensitive blocks to the new set = [{...}] form. Both run identically on OpenTofu.

Step 4 — cloudwatch-addon.tf (the AWS-native path: OIDC provider + IRSA role + the add-on).

data "tls_certificate" "oidc" {
  url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "oidc" {
  url             = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.oidc.certificates[0].sha1_fingerprint]
}

data "aws_iam_policy_document" "cw_agent_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.oidc.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:amazon-cloudwatch:cloudwatch-agent"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc.url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "cw_agent" {
  name               = "${var.cluster_name}-cw-agent"
  assume_role_policy = data.aws_iam_policy_document.cw_agent_assume.json
}

resource "aws_iam_role_policy_attachment" "cw_agent" {
  role       = aws_iam_role.cw_agent.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

resource "aws_eks_addon" "cloudwatch_observability" {
  cluster_name                = var.cluster_name
  addon_name                  = "amazon-cloudwatch-observability"
  service_account_role_arn    = aws_iam_role.cw_agent.arn
  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "PRESERVE"

  configuration_values = jsonencode({
    containerLogs = { enabled = true }
    agent = { config = { logs = { metrics_collected = {
      kubernetes = { enhanced_container_insights = true }
    } } } }
  })

  depends_on = [aws_iam_role_policy_attachment.cw_agent]
}

# Cap the log bill: set retention on the groups the add-on writes to
resource "aws_cloudwatch_log_group" "ci" {
  for_each          = toset(["application", "host", "dataplane", "performance"])
  name              = "/aws/containerinsights/${var.cluster_name}/${each.key}"
  retention_in_days = 14
}

Step 5 — kube-prometheus.tf (the CNCF path: a Grafana password secret + the Helm release).

resource "random_password" "grafana" {
  length  = 20
  special = true
}

resource "aws_secretsmanager_secret" "grafana" {
  name = "${var.cluster_name}/grafana-admin"
}

resource "aws_secretsmanager_secret_version" "grafana" {
  secret_id     = aws_secretsmanager_secret.grafana.id
  secret_string = random_password.grafana.result
}

resource "helm_release" "kps" {
  name             = "kps"
  repository       = "https://prometheus-community.github.io/helm-charts"
  chart            = "kube-prometheus-stack"
  version          = "65.5.1"
  namespace        = "monitoring"
  create_namespace = true
  timeout          = 600
  atomic           = true

  values = [templatefile("${path.module}/values/kps.yaml.tftpl", {
    storage_class    = var.storage_class
    prom_pvc_size    = "50Gi"
    grafana_pvc      = "10Gi"
    amp_remote_write = var.amp_remote_write_url
    aws_region       = var.region
  })]

  set_sensitive {
    name  = "grafana.adminPassword"
    value = random_password.grafana.result
  }
}

with values/kps.yaml.tftpl exactly as shown in the CNCF section above.

Step 6 — sample-app.tf (a Deployment + Service that exposes /metrics, plus a ServiceMonitor scraping it and a Grafana dashboard). The app is prom/prometheus’s own demo target — any image exposing /metrics works; here we use a tiny instrumented sample.

resource "kubernetes_namespace" "demo" {
  metadata { name = "demo" }
}

resource "kubernetes_deployment" "sample" {
  metadata{
    name = "sample-app"
    namespace = kubernetes_namespace.demo.metadata[0].name
  }
  spec {
    replicas = 2
    selector { match_labels = { app = "sample-app" } }
    template {
      metadata { labels = { app = "sample-app" } }
      spec {
        container {
          name  = "app"
          image = "ghcr.io/stefanprodan/podinfo:6.7.1"   # exposes /metrics on 9797
          port{
            name = "metrics"
            container_port = 9797
          }
        }
      }
    }
  }
}

resource "kubernetes_service" "sample" {
  metadata {
    name      = "sample-app"
    namespace = kubernetes_namespace.demo.metadata[0].name
    labels    = { app = "sample-app" }
  }
  spec {
    selector = { app = "sample-app" }
    port{
      name = "metrics"
      port = 9797
      target_port = "metrics"
    }
  }
}

resource "kubernetes_manifest" "sample_servicemonitor" {
  manifest = {
    apiVersion = "monitoring.coreos.com/v1"
    kind       = "ServiceMonitor"
    metadata   = { name = "sample-app", namespace = "monitoring", labels = { release = "kps" } }
    spec = {
      selector          = { matchLabels = { app = "sample-app" } }
      namespaceSelector = { matchNames = ["demo"] }
      endpoints         = [{ port = "metrics", path = "/metrics", interval = "15s" }]
    }
  }
  depends_on = [helm_release.kps]
}

# A minimal Grafana dashboard, provisioned via the sidecar ConfigMap convention
resource "kubernetes_config_map" "dashboard" {
  metadata {
    name      = "sample-app-dashboard"
    namespace = "monitoring"
    labels    = { grafana_dashboard = "1" }   # the Grafana sidecar imports these
  }
  data = {
    "sample-app.json" = file("${path.module}/dashboards/sample-app.json")
  }
  depends_on = [helm_release.kps]
}

Step 7 — alerts.tf (one SNS topic + email subscription + a Container Insights alarm).

resource "aws_sns_topic" "alerts" { name = "${var.cluster_name}-obs-alerts" }

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alarm_email          # confirmed OUT OF BAND (Terraform can't)
}

resource "aws_cloudwatch_metric_alarm" "pod_restarts" {
  alarm_name          = "${var.cluster_name}-pod-restarts"
  namespace           = "ContainerInsights"
  metric_name         = "pod_number_of_container_restarts"
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 5
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "notBreaching"
  dimensions          = { ClusterName = var.cluster_name, Namespace = "demo" }
  alarm_actions       = [aws_sns_topic.alerts.arn]
  depends_on          = [aws_eks_addon.cloudwatch_observability]
}

Step 8 — outputs.tf.

output "grafana_admin_secret" { value = aws_secretsmanager_secret.grafana.name }
output "cw_agent_role_arn"    { value = aws_iam_role.cw_agent.arn }
output "sns_topic_arn"        { value = aws_sns_topic.alerts.arn }
output "container_insights_url" {
  value = "https://${var.region}.console.aws.amazon.com/cloudwatch/home?region=${var.region}#container-insights:infrastructure"
}

Step 9 — init and plan. Put your values in terraform.tfvars (cluster_name, alarm_email), then:

terraform init
terraform plan -out tf.plan

Representative tail:

Plan: 21 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cw_agent_role_arn      = (known after apply)
  + grafana_admin_secret   = "<cluster>/grafana-admin"
  + sns_topic_arn          = (known after apply)

Step 10 — apply. ⚠️ Creates the IRSA role, the add-on (agent + Fluent Bit on every node), the Helm stack with EBS PVCs, the sample app, and the SNS topic.

terraform apply tf.plan
# helm_release.kps: Still creating... [3m00s elapsed]
# helm_release.kps: Creation complete after 3m40s

Step 11 — confirm the SNS email subscription (the step Terraform can’t do). Click Confirm subscription in the “AWS Notification” email, then verify:

aws sns list-subscriptions-by-topic --topic-arn "$(terraform output -raw sns_topic_arn)" \
  --query 'Subscriptions[].SubscriptionArn' --output text   # a real ARN = confirmed

Step 12 — verify the AWS-native path. The agent and Fluent Bit should be Running on every node, and Container Insights metrics should appear:

kubectl get pods -n amazon-cloudwatch          # cloudwatch-agent-* and fluent-bit-* per node
kubectl top nodes                              # metrics-server: proves the cluster reports usage
aws logs describe-log-groups \
  --log-group-name-prefix "/aws/containerinsights/$(terraform output -raw grafana_admin_secret | cut -d/ -f1)"
# open the Container Insights console URL and watch the map populate (~2-3 min)

Step 13 — verify the CNCF path. Every kube-prometheus-stack pod Running, the PVCs Bound, and the ServiceMonitor scraped:

kubectl get pods -n monitoring                 # prometheus-kps-*, kps-grafana-*, alertmanager-*, node-exporter-*
kubectl get pvc  -n monitoring                 # all Bound (not Pending) — proves the EBS CSI/gp3 path
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090 &
# In the Prometheus UI → Status → Targets, look for serviceMonitor/monitoring/sample-app/0 with both pods UP

Step 14 — open Grafana and log in.

kubectl -n monitoring port-forward svc/kps-grafana 3000:80 &
aws secretsmanager get-secret-value --secret-id "$(terraform output -raw grafana_admin_secret)" \
  --query SecretString --output text            # the admin password (user: admin)
# open http://localhost:3000 → the bundled "Kubernetes / Compute Resources" dashboards
# and your imported "sample-app" dashboard should be present

You now have both planes live: Container Insights in the AWS console (node/pod maps, CloudWatch alarms), and Grafana with PromQL dashboards, from the same terraform apply.

Step 15 — destroy (do not skip). ⚠️

terraform destroy

Confirm kubectl get ns monitoring is gone, kubectl get pvc -A | grep monitoring returns nothing (the EBS volumes are deleted with their PVCs), the add-on is removed (aws eks list-addons --cluster-name <c>), and the log groups are gone. The EKS cluster itself is untouched — we only ever read it.

Variables, outputs and making it reusable

Copy-pasting this stack into the next cluster is the click-ops we set out to kill, in HCL. The fix is a small module that takes a cluster name and a few toggles and stamps the whole observability plane — so a new cluster gets identical monitoring by adding one block:

module "eks_observability" {
  source       = "./modules/eks-observability"
  cluster_name = module.eks.cluster_name

  enable_container_insights = true
  enable_kube_prometheus    = true
  storage_class             = "gp3"
  log_retention_days        = 30
  grafana_ingress_host      = "grafana.dev.kloudvin.com"   # ALB + ExternalDNS
  amp_remote_write_url      = ""                            # "" = self-hosted only
  alarm_email               = "sre@kloudvin.com"
}

Gate each half behind a count = var.enable_* ? 1 : 0 (or the module’s own bool) so a team can run Container Insights only, Prometheus only, or both — the same module, three shapes. Stamp it across a fleet with for_each over your clusters.

Roll your own vs the registry. The community modules are worth knowing: terraform-aws-modules/eks/aws provisions the cluster and its add-ons (including a clean IRSA sub-module, iam-role-for-service-accounts-eks, that builds exactly the OIDC-trust role shown above), and the aws-observability/observability-accelerator modules wrap AMP/AMG/ADOT with sensible defaults. Use them once your needs stabilise; roll your own (as here) when you want an opinionated house standard — every cluster gets exactly these log groups, this retention, this Grafana, wired to this SNS topic — which is often more valuable to a platform team than raw flexibility. The two compose: your house module can call the registry IRSA sub-module internally.

Common mistakes and troubleshooting

Observability fails silently — the metric that should have paged simply isn’t there — so these are worth memorising. Symptom → cause → fix:

The three that eat the most hours, in prose. The IRSA trust policy is the sneakiest AWS-native failure: the agent pods are green, kubectl logs shows the agent starting, and yet no metrics appear — because the service account cannot assume the role, so every CloudWatch API call is silently denied. The tell is AccessDenied in the agent logs; the fix is always the trust policy’s two conditions (:sub pinning amazon-cloudwatch:cloudwatch-agent, :aud = sts.amazonaws.com) and the managed policy attachment — not the agent config. The Pending PVC is the CNCF equivalent: the Helm release “succeeds,” but the Prometheus and Grafana pods sit Pending because their PVCs have no provisioner — this is entirely an EBS-CSI/StorageClass prerequisite, not a Prometheus problem, and it is why the storage lesson comes first. And the un-scoped ServiceMonitor is the classic “my app isn’t in Prometheus”: the monitor exists, but its labels don’t match the Prometheus instance’s serviceMonitorSelector, so it is ignored — set serviceMonitorSelectorNilUsesHelmValues: false (honour all) and label the monitor release: kps, then check Status → Targets in the Prometheus UI.

Cost, cleanup and production notes

What it costs left running. The standing cost is the cluster (control plane ~₹6,000/mo + node EC2), which predates this lesson. What this adds: the EBS volumes for the TSDB and Grafana (~60 GB gp3 ≈ ₹500/mo), the CloudWatch log ingestion + storage from Fluent Bit (volume-dependent — the real variable, and the one that surprises people), the Container Insights custom metrics (priced each — enhanced insights raises the count), a couple of Secrets Manager secrets (~₹35/mo each), and, if enabled, AMP (per-sample ingested) and AMG (~$9/editor/mo). The two cost levers that matter are log retention/scope and metric cardinality.

Clean up with terraform destroy; the EBS volumes go with their PVCs (StorageClass reclaimPolicy: Delete), the add-on and its DaemonSets are removed, and the log groups are destroyed (no skip_destroy). Production hardening notes:

• Split the state, or at least the applies. The cluster, its add-ons, and the Helm workloads have different blast radii and change cadences. A common shape is cluster + core add-ons (EBS CSI, CloudWatch observability) in one config, and the Helm/app-layer observability (kube-prometheus-stack, ServiceMonitors, dashboards) in a second config that reads the cluster as a data source — so a Grafana upgrade never risks the cluster.
• Pin everything, upgrade deliberately. Pin the addon_version, the Helm chart version, and the provider versions; upgrade in a PR with a plan diff, not implicitly. An un-pinned chart re-templates on every apply.
• Least privilege on the agent and on remote_write. The agent role gets CloudWatchAgentServerPolicy and nothing more; the Prometheus remote_write role gets only aps:RemoteWrite to the specific workspace. Scope the SNS topic policy with a SourceArn condition.
• Set retention on day one. The single biggest EKS observability cost mistake is Fluent Bit shipping to never-expire log groups. Bake retention_in_days into the module so a new cluster can’t leak.
• Route by severity and guard drift. Two SNS topics — warnings (email/Slack) and pages (composites/critical only) — keep the pager meaningful; a scheduled terraform plan in CI surfaces the threshold someone widened in the console at 2 a.m.

Cheat-sheet

Resources and the argument that matters most on each:

Verification commands:

The service account the add-on uses: cloudwatch-agent in namespace amazon-cloudwatch — that string is what the IRSA :sub condition (or the Pod Identity association) must name.

Interview and exam questions

1. What does the amazon-cloudwatch-observability EKS add-on install, and what permission does it need? The CloudWatch agent (a DaemonSet) plus Fluent Bit — the agent publishes performance metrics to the ContainerInsights namespace and Fluent Bit ships container logs to CloudWatch Logs. It needs the CloudWatchAgentServerPolicy managed policy, granted to the cloudwatch-agent service account (in amazon-cloudwatch) via IRSA or EKS Pod Identity.

2. Walk through IRSA for the agent. Register the cluster’s OIDC issuer as an aws_iam_openid_connect_provider; create an IAM role whose trust policy allows sts:AssumeRoleWithWebIdentity from that Federated provider, conditioned on :sub = system:serviceaccount:amazon-cloudwatch:cloudwatch-agent and :aud = sts.amazonaws.com; attach CloudWatchAgentServerPolicy; pass the role ARN as the add-on’s service_account_role_arn.

3. IRSA vs EKS Pod Identity — when each? IRSA federates through the cluster’s OIDC provider (per-cluster issuer in the trust). Pod Identity trusts the pods.eks.amazonaws.com service principal and binds the SA→role with an aws_eks_pod_identity_association, so the same role is reusable across clusters and there’s no OIDC provider to manage. Prefer Pod Identity for new clusters; IRSA remains everywhere and is still required by some tools.

4. Container Insights vs kube-prometheus-stack — the trade? Container Insights is one add-on, AWS-run, natively integrated (CloudWatch alarms, X-Ray), but AWS-only and priced per-GB/per-metric (cardinality is expensive). kube-prometheus-stack is one Helm release, self-run (you own storage, upgrades, scaling), PromQL + a huge Grafana library, portable, cheap on cardinality. Most teams run both, scoped.

5. Your Grafana and Prometheus pods are Pending. Most likely cause on EKS? Their PVCs can’t be provisioned — the EBS CSI driver isn’t installed or there’s no matching gp3 StorageClass. Install the aws-ebs-csi-driver add-on (with IRSA) and a StorageClass; the PVCs then bind and the pods start. It’s a storage prerequisite, not a Prometheus bug.

6. A ServiceMonitor exists but no target appears in Prometheus. Two causes. Its labels don’t match the Prometheus instance’s serviceMonitorSelector (so it’s ignored — fix with release: kps and/or serviceMonitorSelectorNilUsesHelmValues: false), or the endpoint port is a number instead of the Service port name (or the Service doesn’t expose /metrics).

7. How do you keep Prometheus data across pod restarts? Give prometheusSpec a storageSpec.volumeClaimTemplate pointing at an EBS-backed StorageClass — without it, Prometheus uses emptyDir and loses its TSDB on every restart. Size the PVC to the retention window.

8. What is AMP, and how does data get into it from an in-cluster Prometheus? Amazon Managed Service for Prometheus is a managed, HA, Prometheus-compatible store. Your Prometheus keeps scraping and remote_writes to the workspace’s .../api/v1/remote_write endpoint, signing each request with SigV4 (region-scoped) using an IRSA role that has aps:RemoteWrite.

9. How do you manage secret Helm values (the Grafana password) from Terraform safely? Generate it with random_password, pass it through set_sensitive on the helm_release (so it’s redacted in plan), and store it in Secrets Manager — never in the values file or a plaintext template variable.

10. How does an EKS alert reach a human on each path? AWS-native: an aws_cloudwatch_metric_alarm on a ContainerInsights metric fires alarm_actions to an SNS topic. CNCF: a PrometheusRule fires into Alertmanager, whose sns_configs receiver (SigV4-signed) publishes to the same SNS topic. Both fan out to email/Slack/PagerDuty.

11. (Terraform Associate style) Your kubernetes_manifest for a ServiceMonitor fails at plan with “no matches for kind ServiceMonitor”. Why, and the fix? The CRD doesn’t exist yet at plan time — kubernetes_manifest needs a live API that knows the type. Install the CRDs first (the kube-prometheus-stack helm_release) and depends_on it; in practice, apply the stack before the manifests, or split them into a second config.

12. (Terraform Associate style) Why pin both the addon_version and the Helm chart version? Un-pinned, each apply may select a newer version and silently change infrastructure (or re-template the whole release), producing drift and surprise upgrades. Pinning makes upgrades explicit, reviewable plan diffs.

Key takeaways

• Observability is infrastructure: the collector add-on, its IAM role, the Helm stack, the PVCs, the alert rules and the SNS topic all belong in Terraform next to the cluster — versioned, reviewed, and stamped identically into every environment from one module.
• EKS gives you two paths, and both are one resource: the AWS-native CloudWatch Observability add-on (aws_eks_addon + IRSA/Pod Identity + CloudWatchAgentServerPolicy) drops the agent and Fluent Bit onto every node; the CNCF kube-prometheus-stack (helm_release) installs the whole Prometheus ecosystem. Most platforms run both, scoped.
• The AWS-native failure is always the trust policy: green agent pods with no metrics means the cloudwatch-agent service account can’t assume the role — check the :sub/:aud conditions and the policy attachment, not the agent.
• The CNCF failure is always the PVC: Prometheus and Grafana are stateful, so their volumeClaimTemplate/persistence needs a gp3 StorageClass on the EBS CSI driver — no driver, Pending pods, no history. Storage comes first.
• ServiceMonitors obey two-level selection: the monitor’s release label must match the Prometheus serviceMonitorSelector, and the port must be the Service port name — get either wrong and the target is silently never scraped.
• Manage Helm values from Terraform with templatefile/yamlencode for structure and set_sensitive for secrets; pin the add-on and chart versions so upgrades are explicit diffs, not surprises.
• Managed (AMP + AMG) keeps Prometheus’s ergonomics and offloads the HA storage: your Prometheus remote_writes to AMP over SigV4, AMG reads it with SSO — a common pragmatic middle for teams that want PromQL without operating the database.
• Set log retention on day one — Fluent Bit into never-expire Container Insights log groups is the classic silent EKS cost leak — and always finish the loop: apply, verify with kubectl top and an open Grafana, then destroy.