Terraform on AWS: CloudWatch Alarms, Dashboards & Logs, SNS Alerting & SRE Observability as Code

An alarm you clicked together in the console is an alarm nobody else can see, review, or reproduce. It has no history in git, no owner, no diff when someone widens the threshold at 2 a.m. to stop the pages, and no way to be identical on the other forty instances in the fleet. That is the whole reason observability-as-code exists: the alarm thresholds, the SNS topics, the log-retention policy, the dashboard JSON and the auto-remediation wiring are infrastructure, and infrastructure belongs in Terraform next to the thing it watches — versioned, peer-reviewed, and stamped identically into every environment from one module. This lesson builds that plane end to end on AWS.

By the end you will have stood up, from an empty directory, a real aws_cloudwatch_metric_alarm on an EC2 instance’s CPU, an SNS topic with a confirmed email subscription wired to the alarm’s alarm_actions, a log group with a sane retention_in_days and a metric filter that turns an ERROR log line into its own alarm, a composite alarm that only pages when two conditions are both true, and a CloudWatch dashboard rendered from a jsonencode widget body — then you will deliberately trigger the alarm and watch the notification arrive, before tearing it all down with terraform destroy. You will leave knowing the arguments that decide whether an alarm is trustworthy or noisy (evaluation_periods, datapoints_to_alarm, treat_missing_data), the exact metric to alarm on for each service, and the fistful of gotchas — an unconfirmed email subscription, a KMS-encrypted topic CloudWatch can’t publish to, a log group billing you forever — that eat an afternoon the first time.

This is the provider-specific, SRE-flavoured layer of the KloudVin Terraform course. It assumes you already know core Terraform — HCL, providers, resources, variables, state and modules from the Foundation and Intermediate tiers — and applies it to AWS relentlessly, with copy-pasteable .tf files and a terraform init → plan → apply → verify → destroy you actually run.

What you’ll build

The scenario is the one every on-call rotation lives inside: a small web service on AWS — an EC2 instance (or an Auto Scaling group) behind an Application Load Balancer, with an RDS database behind it — and a requirement that the team find out when any of it degrades, through the same channel, with the same thresholds, in dev and in prod, without anyone clicking around a console. That means metrics turned into alarms, application logs shipped to a retained log group, a couple of patterns in those logs promoted to alarms of their own, one dashboard the team actually looks at, and a single notification path — an SNS topic — that fans a firing alarm out to email and Slack, and can hand the nastier ones to a Lambda that fixes the problem before a human wakes up.

Everything in that paragraph is a Terraform resource. The metric alarm is aws_cloudwatch_metric_alarm; the composite is aws_cloudwatch_composite_alarm; the notification topic is aws_sns_topic with aws_sns_topic_subscription; the retained logs are aws_cloudwatch_log_group; the log-pattern-to-alarm bridge is aws_cloudwatch_log_metric_filter; the dashboard is aws_cloudwatch_dashboard; the event-driven ops rule is aws_cloudwatch_event_rule. Wiring them by hand in the console is slow and invisible; wiring them in Terraform gives you the dependency graph (the alarm references the topic ARN, the metric filter references the log-group name, and Terraform orders all of it), a plan that shows the exact diff before anything changes, and a module you apply to fifty instances by changing one for_each.

Why Terraform rather than the console, the AWS CLI, or CloudFormation? Because monitoring is precisely the kind of sprawling, repetitive, cross-referencing configuration that rewards code and punishes clicking. A fleet needs the same CPU alarm on every host; a console gives you fifty subtly different ones. An audit needs to know who changed a threshold and when; git has the answer, the console does not. CloudFormation can express all of this too, but you lose Terraform’s plan preview, its for_each ergonomics over a list of services, and the single state model that already holds the EC2, ALB and RDS resources these alarms point at.

Reading that diagram left to right is reading the signal path you are about to build: your resources emit metrics and logs into CloudWatch; a metric alarm (or a metric filter promoted from a log line) evaluates the numbers and, when the breach survives the M-of-N test, fires its alarm_actions at an SNS topic; the topic fans the message out to humans (email, Slack) or to a Lambda that auto-remediates. Two things sit off to the side but matter as much as the happy path: treat_missing_data, which decides what an alarm does when the metric goes silent, and retention_in_days, which is the difference between a predictable log bill and a runaway one.

Here is the full inventory a single terraform apply creates in the hands-on, and roughly what each part costs if you leave it running (ap-south-1 / Mumbai, on-demand, indicative July 2026):

The EC2 instance is the only line item with a real hourly charge; everything CloudWatch and SNS here is effectively free at demo volume. It is still a build it, verify it, destroy it lesson — an instance left running for a month is a bill you did not mean to pay — and every costly or destructive step below is marked ⚠️.

Where this fits: the CPU alarm here targets a single instance so the lesson stays about observability, but in production you alarm on an Auto Scaling group (AutoScalingGroupName dimension) and let a scaling policy react — that compute layer is the subject of the AWS Auto Scaling & launch templates lesson. The ALB and RDS this plane watches are stood up properly in the 3-tier architecture, modules & SRE remote-state lesson, which is also where the S3 + DynamoDB backend these alarms live beside comes from. And when you outgrow raw CloudWatch alarms into SLOs, error budgets and synthetic browser checks, the Datadog monitors, SLOs & synthetics lesson picks up exactly there.

Why observability as code (and not click-ops)

Before the resources, the argument — because it governs every choice below. Monitoring drifts faster than almost any other infrastructure: someone bumps a threshold to stop a page, someone adds an alarm for one incident and never removes it, someone forgets the new service entirely. Code fixes each of those failure modes structurally, not by discipline.

The SRE reframing is the important one: an alarm is a contract about what “healthy” means for a service, and a contract belongs in version control where it can be reviewed, tested and evolved — not in a UI where it can be silently edited. Everything that follows treats CloudWatch and SNS as declarative infrastructure, exactly like a VPC or an IAM role.

Metrics and alarms: aws_cloudwatch_metric_alarm

A CloudWatch alarm watches a single metric (or a metric-math expression) and moves between three states — OK, ALARM, and INSUFFICIENT_DATA — based on how the metric compares to a threshold over a window of time. The whole art is in the window: a good alarm fires on a real, sustained problem and stays quiet through a one-second blip. Here is the canonical resource — a CPU alarm on an EC2 instance:

resource "aws_cloudwatch_metric_alarm" "ec2_cpu_high" {
  alarm_name          = "${var.project}-ec2-cpu-high"
  alarm_description   = "EC2 CPUUtilization >= 80% for 3 of the last 5 minutes"
  namespace           = "AWS/EC2"
  metric_name         = "CPUUtilization"
  statistic           = "Average"
  period              = 60          # seconds per datapoint
  evaluation_periods  = 5           # look back over 5 datapoints
  datapoints_to_alarm = 3           # fire if 3 of those 5 breach (M-of-N)
  comparison_operator = "GreaterThanOrEqualToThreshold"
  threshold           = 80
  treat_missing_data  = "missing"
  dimensions          = { InstanceId = aws_instance.app.id }

  alarm_actions             = [aws_sns_topic.alerts.arn]
  ok_actions                = [aws_sns_topic.alerts.arn]
  insufficient_data_actions = []
  tags                      = local.tags
}

Read it as a sentence: average CPU, sampled every 60 seconds, alarms if it is at or above 80% in at least 3 of the last 5 samples, and when it does it publishes to the SNS topic. Every one of those words maps to an argument, and getting them right is the difference between a trustworthy alarm and pager fatigue.

The comparison operator picks the direction and, for anomaly detection, the band form:

statistic vs extended_statistic trips people up: the flat statistics are cheap aggregates; percentiles need the extended argument, and you cannot set both:

evaluation_periods + datapoints_to_alarm is the single most important tuning knob — the “M of N” rule that decides how twitchy the alarm is. period × evaluation_periods is the window; datapoints_to_alarm is how much of that window must breach:

treat_missing_data is the argument that produces the most “why is my alarm stuck on INSUFFICIENT_DATA?” tickets. A metric can simply stop arriving — the instance was replaced, the ALB got no traffic, the Lambda wasn’t invoked — and this setting decides what the alarm does with the gap:

The rule of thumb: for a liveness signal (“this thing should always be emitting”), breaching — silence is an outage. For a bursty metric that legitimately goes quiet (an ALB with no night traffic), notBreaching — silence is fine. Leaving it default missing is why so many alarms sit forever in INSUFFICIENT_DATA and never page.

The alarm catalog: which metric per service

The other half of “a good alarm” is choosing the right metric and dimension per service. Get the dimension wrong and the alarm watches nothing (it stays INSUFFICIENT_DATA forever). This is the table to keep next to you when you write alarms — the metric, its namespace, the statistic, a sane starting threshold, and the dimension that scopes it:

Two dimension gotchas worth their own line. First, ALB and Target Group dimensions want the arn_suffix, not the ARN — in Terraform that is aws_lb.this.arn_suffix and aws_lb_target_group.this.arn_suffix. Pass the full ARN and the alarm silently watches nothing. Second, ASG-aggregated CPU lives in AWS/EC2 keyed by AutoScalingGroupName (a roll-up across the group), while the group’s own metrics (GroupInServiceInstances, GroupDesiredCapacity) live in AWS/AutoScaling — and the latter must be explicitly enabled on the ASG (enabled_metrics) before they exist to alarm on.

Composite alarms, metric math and anomaly detection

Three techniques turn a pile of single-metric alarms into something an SRE actually trusts.

Composite alarms (aws_cloudwatch_composite_alarm) combine other alarms with boolean logic so you page on user impact, not on any single twitchy signal. The classic use: only page when CPU is high and 5xx is elevated — high CPU alone might just be a batch job; high CPU plus errors is a real incident. The composite has no metric of its own; it references other alarms by name in an alarm_rule:

resource "aws_cloudwatch_composite_alarm" "service_degraded" {
  alarm_name        = "${var.project}-service-degraded"
  alarm_description = "Page only when CPU high AND 5xx elevated (real user impact)"

  alarm_rule = join(" AND ", [
    "ALARM(${aws_cloudwatch_metric_alarm.ec2_cpu_high.alarm_name})",
    "ALARM(${aws_cloudwatch_metric_alarm.alb_5xx_rate.alarm_name})",
  ])

  alarm_actions   = [aws_sns_topic.pager.arn]   # this one pages
  ok_actions      = [aws_sns_topic.pager.arn]
  actions_enabled = true

  depends_on = [
    aws_cloudwatch_metric_alarm.ec2_cpu_high,
    aws_cloudwatch_metric_alarm.alb_5xx_rate,
  ]
}

The child alarms keep their (quieter) email actions; only the composite talks to the pager. That is how you cut alarm noise without going blind. The alarm_rule grammar is small:

Metric math lets an alarm compute a value that no single metric gives you — the most common being an error rate (5xx ÷ requests × 100) instead of a raw count, so a traffic spike doesn’t false-alarm. You express it with metric_query blocks: the raw metrics carry return_data = false (they feed the expression), and the expression carries return_data = true (it is what the alarm watches):

resource "aws_cloudwatch_metric_alarm" "alb_5xx_rate" {
  alarm_name          = "${var.project}-alb-5xx-rate"
  alarm_description   = "Target 5xx error rate > 5% for 2 of 3 minutes"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  datapoints_to_alarm = 2
  threshold           = 5            # percent
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  metric_query {
    id          = "e1"
    expression  = "(m_5xx / m_req) * 100"
    label       = "5xx error rate %"
    return_data = true
  }
  metric_query {
    id          = "m_5xx"
    return_data = false
    metric {
      metric_name = "HTTPCode_Target_5XX_Count"
      namespace   = "AWS/ApplicationELB"
      period      = 60
      stat        = "Sum"
      dimensions  = { LoadBalancer = aws_lb.this.arn_suffix }
    }
  }
  metric_query {
    id          = "m_req"
    return_data = false
    metric {
      metric_name = "RequestCount"
      namespace   = "AWS/ApplicationELB"
      period      = 60
      stat        = "Sum"
      dimensions  = { LoadBalancer = aws_lb.this.arn_suffix }
    }
  }
}

Anomaly detection replaces a hand-picked threshold with a model of the metric’s normal band. You point the alarm at an ANOMALY_DETECTION_BAND expression and set threshold_metric_id (not threshold) plus a band-aware comparison operator:

resource "aws_cloudwatch_metric_alarm" "cpu_anomaly" {
  alarm_name          = "${var.project}-ec2-cpu-anomaly"
  alarm_description   = "EC2 CPU outside its expected band (2 stddev)"
  comparison_operator = "GreaterThanUpperThreshold"   # spikes only
  evaluation_periods  = 3
  datapoints_to_alarm = 2
  threshold_metric_id = "e1"
  treat_missing_data  = "missing"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  metric_query {
    id          = "e1"
    expression  = "ANOMALY_DETECTION_BAND(m1, 2)"
    label       = "CPU (expected band)"
    return_data = true
  }
  metric_query {
    id          = "m1"
    return_data = true
    metric {
      metric_name = "CPUUtilization"
      namespace   = "AWS/EC2"
      period      = 300
      stat        = "Average"
      dimensions  = { InstanceId = aws_instance.app.id }
    }
  }
}

Use static thresholds for anything with a real SLO (a latency budget, an error-rate ceiling) and anomaly detection for shape-shifting metrics — request volume, queue depth — where “normal” is a moving target and no fixed number is ever right.

SNS alerting: topics, subscriptions and alarm_actions

An alarm on its own only changes colour in a console. The alarm_actions list is what turns a state change into a notification, and the near-universal target is an SNS topic — a pub/sub fan-out that lets one firing alarm reach many places at once. You create the topic, subscribe endpoints to it, and put its ARN in the alarm’s alarm_actions:

resource "aws_sns_topic" "alerts" {
  name = "${var.project}-alerts"
  tags = local.tags
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alarm_email        # confirmed OUT OF BAND — see below
}

Subscriptions are where the fan-out happens — the same topic can notify a human inbox, a phone, a webhook, an SQS queue for buffering, or a Lambda for action. The protocol decides both the endpoint format and, crucially, whether Terraform can finish the job:

The single biggest SNS surprise for Terraform users: an email subscription cannot be confirmed by Terraform. apply creates the subscription in PendingConfirmation state and AWS sends a confirmation email; until a human clicks the link, no alarm notification is delivered. This is by design (anti-spam) and there is no argument to bypass it — you confirm out of band, once. For https/http/lambda/sqs, confirmation is automatic and the wiring is complete on apply.

Slack and PagerDuty ride on top of these protocols. The clean, code-only path to Slack is AWS Chatbot (aws_chatbot_slack_channel_configuration) subscribed to the topic — no webhook secret in state; alarms render as rich cards in a channel. The alternative is an https subscription pointed at a Slack Incoming Webhook, or at PagerDuty’s SNS integration URL, both of which auto-confirm. For anything that needs to reshape the message (SNS’s JSON is ugly in Slack), subscribe a small Lambda and let it format and forward.

Topic policy is the gotcha that produces “the alarm fires but nothing arrives.” An SNS topic’s default access policy allows the owning account’s principals, but a CloudWatch alarm publishes as the cloudwatch.amazonaws.com service principal — and if you have tightened the topic policy at all, you must explicitly allow it (with a SourceArn condition so only your alarms can publish):

data "aws_caller_identity" "me" {}

data "aws_iam_policy_document" "alerts_policy" {
  statement {
    sid       = "AllowCloudWatchAlarmsToPublish"
    effect    = "Allow"
    actions   = ["SNS:Publish"]
    resources = [aws_sns_topic.alerts.arn]
    principals {
      type        = "Service"
      identifiers = ["cloudwatch.amazonaws.com"]
    }
    condition {
      test     = "ArnLike"
      variable = "aws:SourceArn"
      values   = ["arn:aws:cloudwatch:${var.region}:${data.aws_caller_identity.me.account_id}:alarm:*"]
    }
  }
  # EventBridge → SNS needs events.amazonaws.com the same way
  statement {
    sid       = "AllowEventBridgeToPublish"
    effect    = "Allow"
    actions   = ["SNS:Publish"]
    resources = [aws_sns_topic.alerts.arn]
    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
  }
}

resource "aws_sns_topic_policy" "alerts" {
  arn    = aws_sns_topic.alerts.arn
  policy = data.aws_iam_policy_document.alerts_policy.json
}

Auto-remediation is the payoff of a Lambda subscription: instead of paging a human, the alarm’s SNS message triggers code that fixes the problem — reboot a wedged instance, bump ASG desired capacity, clear a queue, flush a cache. You subscribe the function and grant SNS permission to invoke it:

resource "aws_sns_topic_subscription" "remediate" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "lambda"
  endpoint  = aws_lambda_function.remediate.arn
}

resource "aws_lambda_permission" "allow_sns" {
  statement_id  = "AllowExecutionFromSNS"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.remediate.function_name
  principal     = "sns.amazonaws.com"
  source_arn    = aws_sns_topic.alerts.arn
}

Forget the aws_lambda_permission and the subscription exists but SNS is denied at invoke time — the function never runs and, worse, the failure is silent unless you also alarm on SNS NumberOfNotificationsFailed.

Logs: log groups, retention, metric filters and shipping

Metrics tell you that something is wrong; logs tell you what. CloudWatch Logs organises log streams into log groups, and the single most important argument on a log group is the one Terraform makes you set on purpose: retention_in_days.

resource "aws_cloudwatch_log_group" "app" {
  name              = "/app/${var.project}"
  retention_in_days = 14           # ⚠️ omit this and logs are kept FOREVER
  kms_key_id        = var.logs_kms_key_arn   # optional CMK encryption
  tags              = local.tags
}

The default behaviour — retain logs forever — is the most common silent AWS cost leak there is: a chatty app ships gigabytes a day into a log group nobody set retention on, and years later it is a five-figure storage line. Always set retention_in_days to a real value. The allowed values are a fixed set; here is the cost intuition and the common choices:

CloudWatch Logs bills on ingestion (roughly $0.50/GB, or ~half that for the Infrequent Access class), storage (~$0.03/GB-month), and analysis (Logs Insights queries scan-billed). Retention caps the storage term; the log-group class caps the ingestion term. Both are one-line decisions in Terraform and invisible in the console until the bill.

Metric filters are the bridge back to alarming: they scan a log group for a pattern and emit a custom metric every time it matches — so you can alarm on “count of ERROR lines” exactly like a native metric. This is how you page on a log signal (a stack trace, an OOM, a specific error string) that has no metric of its own:

resource "aws_cloudwatch_log_metric_filter" "app_errors" {
  name           = "${var.project}-app-errors"
  log_group_name = aws_cloudwatch_log_group.app.name
  pattern        = "?ERROR ?Exception ?\"Traceback\""   # OR of terms

  metric_transformation {
    name          = "AppErrorCount"
    namespace     = "Custom/${var.project}"
    value         = "1"
    default_value = "0"       # emit 0 when no match → alarm can leave INSUFFICIENT_DATA
    unit          = "Count"
  }
}

resource "aws_cloudwatch_metric_alarm" "app_error_rate" {
  alarm_name          = "${var.project}-app-errors"
  alarm_description   = ">= 5 ERROR log lines per minute"
  namespace           = "Custom/${var.project}"
  metric_name         = "AppErrorCount"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 1
  threshold           = 5
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.alerts.arn]
}

Two subtleties make or break this. Setting default_value = "0" means the filter emits a zero when nothing matches, so the metric is continuous and the alarm can sit in OK instead of INSUFFICIENT_DATA — without it, a quiet period looks like missing data. And the pattern grammar has two dialects:

For structured JSON logs the { $.field = "x" } form is far more robust than string matching — it survives log-format changes and lets you threshold numeric fields directly.

Subscription filters (aws_cloudwatch_log_subscription_filter) are the other direction — they stream matching log events out of CloudWatch in near-real-time to another service for processing or centralisation:

Getting logs into the group is the last mile. Lambda writes to /aws/lambda/<fn> automatically. For EC2 and on-host app logs you run the CloudWatch agent, whose config (usually stored in SSM Parameter Store and pulled at boot) both ships log files and collects the metrics EC2 does not emit by default — memory and disk:

resource "aws_ssm_parameter" "cw_agent" {
  name = "/${var.project}/cloudwatch-agent/config"
  type = "String"
  value = jsonencode({
    agent   = { metrics_collection_interval = 60 }
    metrics = { namespace = "CWAgent", metrics_collected = {
      mem  = { measurement = ["mem_used_percent"] }
      disk = { measurement = ["used_percent"], resources = ["*"] }
    } }
    logs = { logs_collected = { files = { collect_list = [{
      file_path         = "/var/log/app/app.log"
      log_group_name    = aws_cloudwatch_log_group.app.name
      log_stream_name   = "{instance_id}"
      retention_in_days = 14
    }] } } }
  })
}

The memory/disk gap catches everyone: teams alarm on CPU, an instance runs out of memory, and there was never a metric because the agent was never installed.

Dashboards as code

A dashboard is the one artifact the team looks at without being paged, and aws_cloudwatch_dashboard renders it from a JSON body — which jsonencode lets you write as native HCL instead of a brittle string. Each widget is an object with a type, a position on a 24-column grid, and properties:

resource "aws_cloudwatch_dashboard" "overview" {
  dashboard_name = "${var.project}-overview"
  dashboard_body = jsonencode({
    widgets = [
      {
        type = "metric", x = 0, y = 0, width = 12, height = 6,
        properties = {
          title   = "EC2 CPU"
          region  = var.region
          view    = "timeSeries"
          metrics = [["AWS/EC2", "CPUUtilization", "InstanceId", aws_instance.app.id]]
          yAxis   = { left = { min = 0, max = 100 } }
          annotations = { horizontal = [{ label = "alarm", value = 80 }] }
        }
      },
      {
        type = "log", x = 12, y = 0, width = 12, height = 6,
        properties = {
          title  = "Recent errors"
          region = var.region
          query  = "SOURCE '${aws_cloudwatch_log_group.app.name}' | fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20"
        }
      },
      {
        type = "alarm", x = 0, y = 6, width = 24, height = 3,
        properties = {
          title  = "Alarm status"
          alarms = [aws_cloudwatch_metric_alarm.ec2_cpu_high.arn]
        }
      }
    ]
  })
}

For a dashboard reused across environments, move the body into a templatefile("dashboard.json.tpl", { instance_id = ..., region = ... }) so the same layout renders with per-environment IDs — the file stays valid JSON you can lint, and Terraform injects the values. CloudWatch bills $3/dashboard-month after the first three, so generate a few meaningful dashboards, not one per resource.

EventBridge for event-driven ops

Alarms watch metrics; EventBridge (aws_cloudwatch_event_rule + aws_cloudwatch_event_target) watches events — an instance changing state, an ECS task dying, a scheduled tick — and routes them to a target. It is the event-driven complement to metric alarms, and the same SNS topic is a natural target:

resource "aws_cloudwatch_event_rule" "ec2_stopped" {
  name        = "${var.project}-ec2-stopped"
  description = "Notify when an instance stops or terminates"
  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail        = { state = ["stopped", "terminated"] }
  })
}

resource "aws_cloudwatch_event_target" "to_sns" {
  rule      = aws_cloudwatch_event_rule.ec2_stopped.name
  target_id = "notify-sns"
  arn       = aws_sns_topic.alerts.arn
}

The same “let the service publish” rule applies: an EventBridge → SNS target needs the topic policy to allow events.amazonaws.com (shown earlier); an EventBridge → Lambda target needs an aws_lambda_permission for events.amazonaws.com. For pure cron use schedule_expression; note that AWS now steers new scheduling toward EventBridge Scheduler (aws_scheduler_schedule), but rules remain the right tool for event patterns.

SLOs and synthetics are the layer above all of this — turning these raw signals into error budgets and running scripted browser/API checks from outside your infrastructure. CloudWatch has Synthetics canaries and Application Signals for that; when you want first-class SLOs, burn-rate alerts and multi-step browser tests as code, the Datadog monitors, SLOs & synthetics lesson covers the same ideas with a dedicated observability provider.

Hands-on: build it with Terraform

Now the centrepiece — a complete, self-contained stack you run end to end: an EC2 instance, a retained log group with a metric filter, a CPU alarm and an error-count alarm both wired to an SNS email topic, a composite alarm, and a dashboard. You will apply, confirm the email, deliberately trigger the alarm to watch the notification land, then destroy. ⚠️ This creates a billable EC2 instance — do the whole loop in one sitting.

Step 0 — layout. Create a directory with these files:

cw-observability/
├── versions.tf
├── variables.tf
├── main.tf
├── outputs.tf
└── terraform.tfvars

Step 1 — versions.tf (provider pin + remote backend). The S3 bucket and DynamoDB lock table are assumed to exist (they come from the 3-tier / remote-state lesson); for a throwaway run you can delete the backend block and use local state.

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
  backend "s3" {
    bucket         = "kloudvin-tfstate-apsouth1"
    key            = "labs/cw-observability/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "kloudvin-tf-locks"
    encrypt        = true
  }
}

Step 2 — variables.tf.

variable "region" {
  type    = string
  default = "ap-south-1"
}
variable "project" {
  type    = string
  default = "cwlab"
}
variable "alarm_email" {
  type        = string
  description = "Address that must confirm the SNS subscription"
}
variable "instance_type" {
  type    = string
  default = "t3.micro"
}

Step 3 — main.tf (the whole plane). It uses the latest Amazon Linux 2023 AMI, a locals block for tags, and wires every resource discussed above.

provider "aws" {
  region = var.region
  default_tags { tags = local.tags }
}

locals {
  tags = {
    Project   = var.project
    ManagedBy = "Terraform"
    Lesson    = "cloudwatch-sns-observability"
  }
}

data "aws_caller_identity" "me" {}

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }
}

# --- The thing we watch ------------------------------------------------------
resource "aws_instance" "app" {
  ami           = data.aws_ami.al2023.id
  instance_type = var.instance_type
  tags          = { Name = "${var.project}-app" }
}

# --- SNS: topic + email subscription + policy --------------------------------
resource "aws_sns_topic" "alerts" {
  name = "${var.project}-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alarm_email
}

data "aws_iam_policy_document" "alerts_policy" {
  statement {
    sid       = "AllowCloudWatchAlarmsToPublish"
    effect    = "Allow"
    actions   = ["SNS:Publish"]
    resources = [aws_sns_topic.alerts.arn]
    principals {
      type        = "Service"
      identifiers = ["cloudwatch.amazonaws.com"]
    }
    condition {
      test     = "ArnLike"
      variable = "aws:SourceArn"
      values   = ["arn:aws:cloudwatch:${var.region}:${data.aws_caller_identity.me.account_id}:alarm:*"]
    }
  }
}

resource "aws_sns_topic_policy" "alerts" {
  arn    = aws_sns_topic.alerts.arn
  policy = data.aws_iam_policy_document.alerts_policy.json
}

# --- Logs: retained group + metric filter ------------------------------------
resource "aws_cloudwatch_log_group" "app" {
  name              = "/app/${var.project}"
  retention_in_days = 14
}

resource "aws_cloudwatch_log_metric_filter" "app_errors" {
  name           = "${var.project}-app-errors"
  log_group_name = aws_cloudwatch_log_group.app.name
  pattern        = "?ERROR ?Exception"
  metric_transformation {
    name          = "AppErrorCount"
    namespace     = "Custom/${var.project}"
    value         = "1"
    default_value = "0"
    unit          = "Count"
  }
}

# --- Alarms ------------------------------------------------------------------
resource "aws_cloudwatch_metric_alarm" "ec2_cpu_high" {
  alarm_name          = "${var.project}-ec2-cpu-high"
  alarm_description   = "EC2 CPU >= 80% for 3 of 5 minutes"
  namespace           = "AWS/EC2"
  metric_name         = "CPUUtilization"
  statistic           = "Average"
  period              = 60
  evaluation_periods  = 5
  datapoints_to_alarm = 3
  threshold           = 80
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "missing"
  dimensions          = { InstanceId = aws_instance.app.id }
  alarm_actions       = [aws_sns_topic.alerts.arn]
  ok_actions          = [aws_sns_topic.alerts.arn]
}

resource "aws_cloudwatch_metric_alarm" "app_error_rate" {
  alarm_name          = "${var.project}-app-errors"
  alarm_description   = ">= 5 ERROR log lines per minute"
  namespace           = "Custom/${var.project}"
  metric_name         = "AppErrorCount"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 1
  threshold           = 5
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.alerts.arn]
}

resource "aws_cloudwatch_composite_alarm" "degraded" {
  alarm_name        = "${var.project}-degraded"
  alarm_description = "CPU high AND errors elevated"
  alarm_rule = join(" AND ", [
    "ALARM(${aws_cloudwatch_metric_alarm.ec2_cpu_high.alarm_name})",
    "ALARM(${aws_cloudwatch_metric_alarm.app_error_rate.alarm_name})",
  ])
  alarm_actions = [aws_sns_topic.alerts.arn]
  depends_on    = [aws_cloudwatch_metric_alarm.ec2_cpu_high, aws_cloudwatch_metric_alarm.app_error_rate]
}

# --- Dashboard ---------------------------------------------------------------
resource "aws_cloudwatch_dashboard" "overview" {
  dashboard_name = "${var.project}-overview"
  dashboard_body = jsonencode({
    widgets = [
      { type = "metric", x = 0, y = 0, width = 12, height = 6, properties = {
        title = "EC2 CPU", region = var.region, view = "timeSeries",
        metrics = [["AWS/EC2", "CPUUtilization", "InstanceId", aws_instance.app.id]],
        yAxis = { left = { min = 0, max = 100 } },
        annotations = { horizontal = [{ label = "alarm", value = 80 }] } } },
      { type = "log", x = 12, y = 0, width = 12, height = 6, properties = {
        title = "Recent errors", region = var.region,
        query = "SOURCE '${aws_cloudwatch_log_group.app.name}' | fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20" } }
    ]
  })
}

Step 4 — outputs.tf.

output "sns_topic_arn"   { value = aws_sns_topic.alerts.arn }
output "instance_id"     { value = aws_instance.app.id }
output "cpu_alarm_name"  { value = aws_cloudwatch_metric_alarm.ec2_cpu_high.alarm_name }
output "log_group_name"  { value = aws_cloudwatch_log_group.app.name }
output "dashboard_url" {
  value = "https://${var.region}.console.aws.amazon.com/cloudwatch/home?region=${var.region}#dashboards:name=${aws_cloudwatch_dashboard.overview.dashboard_name}"
}

Step 5 — init and plan. Put your email in terraform.tfvars (alarm_email = "you@example.com"), then:

terraform init
terraform plan -out tf.plan

Representative tail of the plan:

Plan: 12 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cpu_alarm_name  = "cwlab-ec2-cpu-high"
  + dashboard_url   = (known after apply)
  + instance_id     = (known after apply)
  + log_group_name  = "/app/cwlab"
  + sns_topic_arn   = (known after apply)

Step 6 — apply. ⚠️ Billable EC2 instance from here.

terraform apply tf.plan

Step 7 — confirm the email subscription (the step Terraform can’t do). Check your inbox for “AWS Notification - Subscription Confirmation” and click Confirm subscription. Verify it took:

aws sns list-subscriptions-by-topic \
  --topic-arn "$(terraform output -raw sns_topic_arn)" \
  --query 'Subscriptions[].SubscriptionArn' --output text
# a real ARN = confirmed;  "PendingConfirmation" = you haven't clicked yet

Step 8 — verify the alarms exist and their state.

aws cloudwatch describe-alarms \
  --alarm-names "$(terraform output -raw cpu_alarm_name)" \
  --query 'MetricAlarms[0].[AlarmName,StateValue,Threshold]' --output text
# cwlab-ec2-cpu-high  INSUFFICIENT_DATA  80.0   (new alarm, no datapoints yet)

Step 9 — trigger the alarm and watch the email arrive. The fastest, cost-free way to prove the wiring (alarm → SNS → inbox) is to force the state with set-alarm-state:

aws cloudwatch set-alarm-state \
  --alarm-name "$(terraform output -raw cpu_alarm_name)" \
  --state-value ALARM \
  --state-reason "Manual test of SNS wiring"

Within seconds you should receive an email titled “ALARM: cwlab-ec2-cpu-high”. Because ok_actions is also set, it will auto-recover on the next real datapoint and send an “OK:” email too. To prove the metric filter path, push a matching log line and let the alarm evaluate:

aws logs create-log-stream --log-group-name "$(terraform output -raw log_group_name)" --log-stream-name test
aws logs put-log-events --log-group-name "$(terraform output -raw log_group_name)" \
  --log-stream-name test \
  --log-events "timestamp=$(( $(date +%s) * 1000 )),message=ERROR something broke x6"
# repeat / send 5+ within a minute; the app_error_rate alarm flips to ALARM

For a real CPU alarm (not forced), SSH in and run stress-ng --cpu 2 --timeout 300s (or sudo dnf install -y stress-ng first) and watch the alarm flip after ~3 minutes on the dashboard.

Step 10 — see the dashboard. terraform output -raw dashboard_url and open it — CPU on the left, recent ERROR logs on the right.

Step 11 — destroy (do not skip). ⚠️

terraform destroy

Confirm with aws cloudwatch describe-alarms --alarm-name-prefix cwlab returning empty and the EC2 instance gone. The log group is destroyed too (no skip_destroy), so no lingering storage cost.

Variables, outputs and making it reusable

Copy-pasting that CPU alarm onto every instance is exactly the click-ops we set out to kill — just in HCL. The fix is a small module that takes a resource identity and a set of thresholds and emits a standard alarm set, so a new service gets consistent monitoring by adding one block. Structure:

modules/standard-alarms/
├── variables.tf
├── main.tf
└── outputs.tf

modules/standard-alarms/variables.tf:

variable "name"          { type = string }                 # alarm name prefix
variable "sns_topic_arn" { type = string }
variable "dimensions"    { type = map(string) }            # e.g. { InstanceId = "i-…" }
variable "namespace"     {
  type = string
  default = "AWS/EC2"
}
variable "cpu_threshold" {
  type = number
  default = 80
}
variable "extra_alarms" {                                  # optional, per-metric
  type = map(object({
    metric_name         = string
    statistic           = string
    threshold           = number
    comparison_operator = string
    period              = optional(number, 60)
    evaluation_periods  = optional(number, 5)
    datapoints_to_alarm = optional(number, 3)
    treat_missing_data  = optional(string, "missing")
  }))
  default = {}
}

modules/standard-alarms/main.tf — a baseline CPU alarm plus a for_each over any extra alarms the caller declares:

resource "aws_cloudwatch_metric_alarm" "cpu" {
  alarm_name          = "${var.name}-cpu-high"
  namespace           = var.namespace
  metric_name         = "CPUUtilization"
  statistic           = "Average"
  period              = 60
  evaluation_periods  = 5
  datapoints_to_alarm = 3
  threshold           = var.cpu_threshold
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "missing"
  dimensions          = var.dimensions
  alarm_actions       = [var.sns_topic_arn]
  ok_actions          = [var.sns_topic_arn]
}

resource "aws_cloudwatch_metric_alarm" "extra" {
  for_each            = var.extra_alarms
  alarm_name          = "${var.name}-${each.key}"
  namespace           = var.namespace
  metric_name         = each.value.metric_name
  statistic           = each.value.statistic
  period              = each.value.period
  evaluation_periods  = each.value.evaluation_periods
  datapoints_to_alarm = each.value.datapoints_to_alarm
  threshold           = each.value.threshold
  comparison_operator = each.value.comparison_operator
  treat_missing_data  = each.value.treat_missing_data
  dimensions          = var.dimensions
  alarm_actions       = [var.sns_topic_arn]
}

Now every service is one call — and stamping the same alarm set across a fleet is a for_each over your instances:

module "app_alarms" {
  source        = "./modules/standard-alarms"
  for_each      = toset(["i-0abc", "i-0def", "i-0ghi"])
  name          = "app-${each.key}"
  sns_topic_arn = aws_sns_topic.alerts.arn
  dimensions    = { InstanceId = each.key }
  extra_alarms = {
    status-check = { metric_name = "StatusCheckFailed", statistic = "Maximum",
                     threshold = 1, comparison_operator = "GreaterThanOrEqualToThreshold" }
  }
}

Roll your own vs the registry. The community terraform-aws-modules/cloudwatch/aws module ships submodules for metric alarms, log groups and metric filters and is worth using once your needs stabilise — it handles the argument surface and edge cases for you. Roll your own (as above) when you want an opinionated house standard — every service gets exactly these alarms, named this way, wired to this topic — which is often the more valuable thing for an SRE team than raw flexibility. The two compose: your standard module can call the registry module internally.

Common mistakes and troubleshooting

Monitoring has a cruel failure mode: when it is broken, it fails silently — the page that should have fired simply doesn’t. These are the ones that bite, symptom → cause → fix:

The three that cost the most hours, in prose. The KMS-encrypted topic is the sneakiest: everything looks correct, the alarm goes red, and no message arrives — because the alarm publishes as cloudwatch.amazonaws.com and the default SNS encryption key (alias/aws/sns) does not grant that service principal kms:GenerateDataKey. The fix is a customer-managed CMK whose key policy allows cloudwatch.amazonaws.com (and events.amazonaws.com if EventBridge publishes), then kms_master_key_id on the topic. The unconfirmed email is the most common: Terraform reports success, the subscription resource exists, but it sits in PendingConfirmation and delivers nothing until a human clicks — so a freshly-applied stack looks monitored but is deaf until someone confirms. And treat_missing_data is the quiet killer of liveness alarms: leave it default and an alarm that should scream when a heartbeat stops instead drifts into INSUFFICIENT_DATA and never pages — set it to breaching for anything whose absence is the failure.

Cost, cleanup and production notes

What it costs left running. The EC2 t3.micro is the only meaningful charge (~₹625/$7.5 a month). CloudWatch here is nearly free: standard alarms are $0.10 each per month, the composite is $0.50, custom metrics from the filter $0.30 each, and the first three dashboards are free. The real cost risks in production are two: log ingestion/storage on a chatty app with no retention (this is the one that produces surprise five-figure lines), and high-resolution alarms (period 10 or 30) at $0.30 each versus $0.10 — only pay for sub-minute resolution where you genuinely act on it.

Clean up with terraform destroy; the only thing that survives a destroy is a log group flagged skip_destroy (which you would only set for audit logs you must keep). Production hardening notes:

• Remote state, always. These alarms reference EC2/ALB/RDS IDs; keep them in the same S3 + DynamoDB backend as the resources they watch so a rename can’t orphan an alarm. The backend itself comes from the 3-tier / remote-state lesson.
• Tag every alarm and topic. default_tags on the provider plus a Team/Service tag makes cost allocation and ownership queryable — and lets you find the stale alarms nobody deletes.
• Least privilege on the notification path. Scope the SNS topic policy with a SourceArn condition; give the remediation Lambda’s role only the exact actions it needs (reboot this ASG, not ec2:*).
• Guard against drift. Someone will edit a threshold in the console during an incident; a scheduled terraform plan in CI surfaces that drift as a diff so it gets folded back into code (or reverted) instead of silently diverging.
• Route by severity. Two topics — one for email/Slack (warnings), one for the pager (composites only) — keep the pager meaningful. actions_enabled = false lets you ship an alarm in “observe” mode before it can wake anyone.

Cheat-sheet

Resources and the argument that matters most on each:

Verification and trigger commands:

Alarm states: OK (metric within bounds), ALARM (threshold breached per M-of-N), INSUFFICIENT_DATA (not enough datapoints — governed by treat_missing_data).

Interview and exam questions

1. What do evaluation_periods and datapoints_to_alarm do together? They define the “M of N” rule: evaluation_periods (N) is the window of datapoints CloudWatch inspects; datapoints_to_alarm (M) is how many of them must breach for the alarm to fire. 3 of 5 tolerates a one-datapoint blip while still catching a sustained problem. Omitting M makes it N-of-N.

2. An alarm is stuck in INSUFFICIENT_DATA. Give two causes. Either no datapoints are arriving (wrong dimensions, so it’s watching a resource that emits nothing — e.g. an ALB alarm given the ARN instead of arn_suffix), or the metric is legitimately silent and treat_missing_data is left at missing. Fix the dimension, or set treat_missing_data to breaching/notBreaching per intent, and set default_value=0 on metric filters.

3. Why might an alarm fire but no email arrive? Most often the email subscription is unconfirmed (PendingConfirmation — Terraform can’t confirm it). Otherwise the topic is KMS-encrypted with a key CloudWatch can’t use, or the topic policy doesn’t allow cloudwatch.amazonaws.com to SNS:Publish.

4. When would you use a composite alarm? To page on correlated conditions and cut noise — e.g. only page when CPU is high AND 5xx is elevated (real user impact), leaving the individual alarms on quieter email actions. aws_cloudwatch_composite_alarm references child alarms by name in alarm_rule.

5. How do you alarm on p99 latency? Use extended_statistic = "p99" (not statistic) on the TargetResponseTime metric — the two are mutually exclusive, and averaging latency hides the tail you actually care about.

6. What’s the difference between a metric filter and a subscription filter? A metric filter counts log-pattern matches into a CloudWatch metric you can alarm on (log → metric → alarm). A subscription filter streams matching log events out to Lambda/Kinesis/Firehose in near-real-time for processing or centralisation (log → external pipeline).

7. Why set retention_in_days? The default is never expire — logs accumulate and bill for storage forever. Setting retention caps storage cost; the INFREQUENT_ACCESS log-group class further cuts ingestion price.

8. How does an alarm trigger auto-remediation? alarm_actions publishes to an SNS topic that has a Lambda subscription; the function runs the fix (reboot, scale, flush). You must add aws_lambda_permission allowing sns.amazonaws.com to invoke it, or the invoke is denied silently.

9. Static threshold vs anomaly detection — when each? Static for hard SLOs where a fixed “bad” number exists (latency ≤ 1 s, error rate ≤ 1%). Anomaly detection for seasonal/shape-shifting metrics (request volume) where no fixed threshold is ever right; it needs history to learn the band and uses threshold_metric_id with a band comparison operator.

10. (Terraform Associate style) You add an email subscription and apply succeeds, but notifications don’t arrive. Why? The subscription is created in PendingConfirmation; SNS email subscriptions require out-of-band confirmation that Terraform cannot perform. Notifications flow only after the recipient clicks the confirmation link.

11. (Terraform Associate style) How do you emit the same alarm set across 30 instances without repeating HCL? Author a module that takes the resource identity + thresholds and emits the alarms, then call it with for_each over the instance IDs — one source of truth, consistent alarms, one place to change a threshold.

12. Which EC2 signals are not available by default, and how do you get them? Memory and filesystem disk usage — the hypervisor doesn’t see them. Install the CloudWatch agent, which publishes them under the CWAgent namespace and can also ship app log files to a log group.

Key takeaways

• Observability belongs in code: alarms, topics, retention and dashboards are infrastructure — versioned, reviewed, and stamped identically across the fleet from one module, never clicked together per host.
• The alarm’s honesty lives in three arguments: evaluation_periods + datapoints_to_alarm (M-of-N tames flapping) and treat_missing_data (decides what silence means). Get these wrong and the alarm is either noise or deaf.
• Pick the right metric and dimension per service — the alarm catalog is the reference; ALB/Target-Group dimensions want arn_suffix, and a wrong dimension leaves the alarm watching nothing.
• SNS is the fan-out: alarm_actions → topic → email/Slack/PagerDuty or a Lambda that auto-remediates. Email needs manual confirmation; a KMS-encrypted topic needs a CMK CloudWatch can use; a tightened topic policy must allow cloudwatch.amazonaws.com.
• retention_in_days is not optional — the default keeps logs forever and is the classic silent AWS cost leak. Metric filters promote a log pattern to an alarm (with default_value=0 so it stays continuous).
• Composite alarms, metric math and anomaly detection turn raw single-metric alarms into signals an on-call actually trusts: page on user impact, on error rate, and on deviation from normal.
• Always finish the loop: apply, confirm the subscription, trigger the alarm to prove the wiring, then destroy — a stack that was never triggered is a stack you don’t know is monitored.