Terraform on AWS at Scale: A 3-Tier Web Architecture, Reusable Modules, Multi-Environment & SRE

There is a moment on every serious AWS programme where a single main.tf stops being infrastructure and becomes a liability. For one retail client it arrived the morning an availability-zone brown-out took half their fleet down and the on-call engineer discovered three things at once: prod, staging and dev were “the same” only in the sense that someone had copy-pasted the files months ago and they had quietly drifted apart; the one state file lived on a build agent nobody wanted to touch; and there was no alarm on the thing that had actually failed because alarms had been added by hand, resource by resource, and this one had been forgotten. Nothing in the HCL was wrong. What was missing was a platform: a reusable module library, per-environment isolation, guardrails, and observability that comes for free with every resource. This lesson is how you build that platform on AWS, and it is the capstone that ties the whole course together.

You already know the parts. You can write a resource, a variable, a module; you know what remote state and locking are; you have met Terragrunt and community modules. Here we compose them into the classic three-tier web architecture done properly — Route 53 → ACM → ALB in public subnets → an Auto Scaling Group of EC2 in private subnets → RDS Multi-AZ in isolated database subnets, with NAT for egress, tightly tiered security groups, S3 for assets and logs, and CloudWatch/SNS for reliability — and we assemble it from a versioned modules/ library consumed by per-environment root configs. Every environment gets its own remote-state key on S3 with a DynamoDB (or native S3) lock, promotion walks a change dev → staging → prod, per-env sizing lives in variables, and SRE is baked into the code: default_tags everywhere, standardized alarms and dashboards, budgets, least-privilege IAM scoped per environment and per state, drift detection, and blast radius designed as separate state per layer.

This is deliberately opinionated, because a platform is a set of decisions, not a pile of options. Read the prose once for the reasoning; then the tables — the 3-tier mapping, the repo layout, the modules-vs-Terragrunt matrix, the per-env sizing grid, the SRE-controls catalogue, and the troubleshooting map — are the reference you keep open while you build. By the end you will be able to stand up a three-tier AWS platform that survives instance loss, AZ loss and database failover, that an auditor, an SRE and a new hire can all read, and answer the “prove dev and prod are the same” question with a git log and a terraform plan.

What you’ll build

The running example is Meridian, a company standing up its first governed AWS estate in ap-south-1 (Mumbai) to run a customer-facing storefront. We build the three-tier web architecture as a platform with three moving planes. The module plane is a modules/ directory holding five reusable, versioned building blocks: vpc (network across two or three AZs — public, private and isolated database subnets, an internet gateway and NAT), alb (a public Application Load Balancer, target group, HTTPS listener and its security group), asg-app (a launch template and EC2 Auto Scaling Group in the private subnets, with target-tracking scaling and its security group), rds (a Multi-AZ managed database with encrypted storage and its security group), and observability (SNS topic, standardized CloudWatch alarms and a dashboard). Each module emits its own alarms, so reliability is a property of the module, not an afterthought. The environment plane is a set of root configurations — dev, staging, prod — each of which composes the modules, passing one module’s outputs (subnet IDs, security-group IDs, target-group ARNs) into the next, and each of which owns a separate remote-state key so a mistake in dev is physically incapable of touching prod. The reliability plane — tags, alarms, budgets, IAM and drift detection — is stamped across everything by the code itself.

The architecture in words, read as a request travels it: a user hits shop.meridian.example, Route 53 resolves it (an alias A record) to a public Application Load Balancer whose HTTPS listener carries an ACM certificate. The ALB lives in public subnets across two Availability Zones and forwards to an Auto Scaling Group of EC2 instances in private subnets — the app tier, which has no public IP and reaches the internet only through a NAT gateway for patches and outbound calls. The app tier talks to an RDS Multi-AZ MySQL database in isolated database subnets that have no route to the internet at all. Static assets and access logs live in S3. Between every tier sits a security group that references the previous tier’s security group rather than a CIDR block, so the rules survive every scale event. CloudWatch alarms watch every tier and page an SNS topic; a budget watches the spend. That is the whole shape, and it is the same shape whether you have three instances or three hundred.

Why Terraform rather than the console, the aws CLI, or CloudFormation? Because a platform’s whole value is that it is reproducible, reviewable and enforced. The console produces snowflakes nobody can recreate; a pile of aws commands is a script with no notion of desired state or drift; CloudFormation is excellent AWS-native IaC but stops at the AWS boundary — Terraform’s provider ecosystem lets the same tool and the same state model govern AWS, the DNS zone, GitHub or Datadog together, which is what a platform team needs. And Terraform’s module + remote-state model is the cleanest expression of the two things a platform must have: reuse (one module, many environments) and isolation (one state per environment per layer).

Here is the leap this lesson is about — the difference between a config and a platform, stated as symptoms you can recognise on your own estate:

By the end you can build every row of the right-hand column with real HCL.

Learning objectives

By the end of this lesson you will be able to:

• Map the classic three-tier architecture to AWS services and Terraform resources — Route 53/ACM → ALB → ASG of EC2 → RDS Multi-AZ, with NAT, tiered security groups, S3 and CloudWatch/SNS — and reason about the request path and the subnet tiers.
• Lay out a platform repository — a modules/ library (vpc, alb, asg-app, rds, observability) and per-environment root configs — and justify what lives where and why each layer gets its own state.
• Compose modules in a root config, passing outputs (subnet IDs, SG IDs, target-group ARN) from one module into the next, and reason about the implicit ordering that creates.
• Isolate environments two ways — tfvars-per-env with an S3 backend and Terragrunt — and drive promotion dev → staging → prod, with per-env sizing (instance types, Multi-AZ on/off, counts) in variables.
• Decide when to use community modules (terraform-aws-modules/{vpc,alb,rds,autoscaling}) versus roll-your-own, and pin them safely.
• Bake SRE into code — default_tags, standardized alarms and dashboards, budgets, least-privilege IAM per env/state, drift detection, and blast radius as separate state per layer.
• Test the library with fmt, validate, tflint, checkov and native terraform test, and gate it in a CI pipeline.

Prerequisites & where this fits

This is the AWS capstone of the Terraform Zero-to-Hero course. It assumes the platform-building primitives are already familiar and pulls them together into one real estate. You will get the most from it having already authored a reusable module with typed inputs, outputs and versioning, built the multi-environment 3-tier centerpiece with Terragrunt and approval gates, wired a GitHub Actions Terraform pipeline with OIDC, plan-on-PR automation, and stood up the network in AWS VPC: subnets, IGW, NAT and routing. Where those lessons teach a technique in isolation, this one shows all of them driven by a real three-tier AWS platform at once.

A note on versions: everything targets Terraform ≥ 1.6 (the 1.9/1.10 line current in 2026) with the aws provider ~> 5.0. Two AWS-specific backend details matter and are reflected below — the S3 backend has long paired with a DynamoDB table for state locking, and Terraform 1.10+ adds native S3 lockfile locking (use_lockfile = true) that can replace DynamoDB entirely. OpenTofu is a drop-in for the CLI throughout; the module and state model are identical. Assume you have aws credentials working (SSO or an assumed role) in an account where you can create a VPC, EC2, RDS and IAM.

Because this lesson pins aws ~> 5.0 and uses the S3 backend, the handful of choices that bite an upgrader or a first-timer are worth having in front of you — every one of them is reflected in the HCL below:

Here is the map of companion lessons and what each carries, so you know where to go deep on any one plane:

The classic three-tier architecture on AWS: what maps to what

The three-tier pattern — a stateless web/edge tier that terminates TLS and load-balances, a stateless app tier that runs your code, and a stateful data tier that holds everything durable — is the reference architecture because each tier can fail and heal independently. On AWS it maps onto specific services and, in turn, onto specific Terraform resources. The single most important idea is that state is pushed all the way down: the web and app tiers are stateless and disposable (an instance can die and be replaced with no data loss), so only the data tier needs the expensive durability guarantees. Get that right and scaling, patching and failure become routine.

This is the mapping you keep open while you build — tier, the AWS service, the Terraform resource that models it, and which subnet it lives in:

The request path and the ports it uses are the other half of the mental model — and the ports are exactly what the tiered security groups will enforce:

The tiered security groups are the heart of least-privilege networking, and the rule that makes them robust is: reference the previous tier’s security group ID, never a CIDR range. When the app tier scales from 2 to 20 instances, the DB security group does not need to change — it already allows the app security group, and every new instance is a member of it. This is the single most common thing juniors get wrong (they open the DB to the whole VPC CIDR “to make it work”), and it is the single control that most reduces blast radius inside the VPC:

The diagram traces the whole platform left to right: the Terraform plane on the left (the modules/ library plus per-env S3 state and its lock) provisions the real three-tier architecture — Route 53/ACM into a public ALB, on to the private-subnet app ASG, down to the Multi-AZ RDS in isolated DB subnets — while CloudWatch/SNS observes every tier. The six badges are the six load-bearing decisions of the lesson: module composition, per-env state key, blast-radius separate state, the three subnet tiers, the tiered security groups, and SRE alarms per environment.

From a single config to a platform: the repo that scales

The repository is the architecture. Before a line of HCL, the folder layout decides your reuse story (one module library or three copies?), your isolation story (how many state files, and what is the blast radius of each?), and your safety story (can a routine dev apply reach prod’s database?). Get the tree right and the rest of the lesson is filling it in.

Here is the canonical layout for the Meridian platform. It separates the modules library (reused, versioned, never applied directly) from per-environment roots (applied, each with its own state), and — critically for blast radius — it splits each environment’s roots into network / app / data layers, each with its own state key:

meridian-platform/
├── modules/                        # the library — reusable, versioned, never applied directly
│   ├── vpc/                        # or wrap terraform-aws-modules/vpc/aws
│   ├── alb/                        # aws_lb + target group + listener + alb SG
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── asg-app/                    # launch template + ASG + scaling policy + app SG
│   ├── rds/                        # aws_db_instance (Multi-AZ) + subnet group + db SG
│   └── observability/              # sns topic + standardized alarms + dashboard
│
├── environments/                   # per-env ROOT configs — each LAYER has its own state key
│   ├── dev/
│   │   ├── network/                # state key: dev/network/terraform.tfstate
│   │   ├── app/                    # state key: dev/app/terraform.tfstate  (composes alb+asg-app+obs)
│   │   └── data/                   # state key: dev/data/terraform.tfstate  (rds)
│   ├── staging/                    # identical structure, staging.auto.tfvars
│   └── prod/                       # identical structure, prod.auto.tfvars
│       ├── app/
│       │   ├── versions.tf         # required_providers + backend "s3" {}
│       │   ├── locals.tf           # naming + common_tags
│       │   ├── main.tf             # composes modules/*  ← the root module
│       │   ├── variables.tf
│       │   ├── outputs.tf
│       │   ├── prod.auto.tfvars    # env-specific values (sizes, counts, Multi-AZ)
│       │   └── backend.hcl         # partial backend: key = "prod/app/terraform.tfstate"
│       ├── network/
│       └── data/
│
├── tests/                          # native terraform test (*.tftest.hcl)
├── .tflint.hcl
└── .checkov.yaml

Read the tree by state boundaries, because that is what matters at 2 a.m. Every leaf directory under environments/ is a root module with its own state key — the unit of plan/apply and the unit of blast radius. Splitting each environment into network, app and data means a data migration can never touch the VPC, and a bad app apply can never drop the database. The modules/ directory has no state of its own; it is pure library code roots reference by source. The one rule that saves you: never terraform apply inside modules/ — modules are consumed, not run.

Two structural decisions deserve emphasis. First, state is split per layer, not just per environment. The network changes rarely and is a dependency of everything; the data tier is the crown jewels and changes under a DBA’s eye; the app tier changes daily. Giving each its own state means the daily churn of the app tier can never plan a change to the VPC or the database. The app layer reads the network layer’s outputs (subnet IDs, VPC ID) via a terraform_remote_state data source, and the data layer’s endpoint is read the same way. Second, environments are sibling folders, not a shared config with a workspace switch. Directory-per-environment is the right default for a platform because each env (and each layer) is an independently reviewable folder whose state key is explicit in a backend.hcl you can read — no terraform.workspace interpolation to reason about under pressure.

The modules library: vpc, alb, asg-app, rds, observability

A platform module is not “a wrapper around one resource”. It is a coherent, named unit of infrastructure with a contract: typed inputs, validated where it matters, sensitive outputs where secrets flow, a README, and — the Meridian house rule — its own CloudWatch alarms so that consuming the module automatically means the resource is observable. The library is versioned (git tags, or a private registry) so an environment pins alb v1.4.0 and upgrades on its own schedule; dev can trial v1.5.0 while prod stays on v1.4.0, and because they share the module’s shape, the upgrade is a reviewable diff, not a rewrite.

Here is the alb module — small enough to read, real enough to run. It creates the public-facing security group (443 from the internet), the load balancer across the public subnets, a target group with a real health check, and an HTTPS listener bound to an ACM certificate; it also emits its own 5xx alarm:

# modules/alb/main.tf
resource "aws_security_group" "alb" {
  name_prefix = "sg-${var.name_prefix}-alb-"
  description = "ALB: 443 from the internet"
  vpc_id      = var.vpc_id

  ingress {
    description = "HTTPS from the internet"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  tags = var.tags
}

resource "aws_lb" "this" {
  name                       = "alb-${var.name_prefix}"
  load_balancer_type         = "application"
  internal                   = false
  subnets                    = var.public_subnet_ids   # ← from the vpc module
  security_groups            = [aws_security_group.alb.id]
  enable_deletion_protection = var.deletion_protection
  drop_invalid_header_fields = true
  tags                       = var.tags
}

resource "aws_lb_target_group" "app" {
  name        = "tg-${var.name_prefix}"
  port        = var.app_port
  protocol    = "HTTP"
  vpc_id      = var.vpc_id
  target_type = "instance"

  health_check {
    path                = "/healthz"
    healthy_threshold   = 3
    unhealthy_threshold = 2
    interval            = 15
    matcher             = "200"
  }
  tags = var.tags
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.this.arn
  port              = 443
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = var.certificate_arn
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

# Observability is a property of the module, not a later chore.
resource "aws_cloudwatch_metric_alarm" "alb_5xx" {
  alarm_name          = "alarm-${var.name_prefix}-alb-5xx"
  namespace           = "AWS/ApplicationELB"
  metric_name         = "HTTPCode_ELB_5XX_Count"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 5
  threshold           = var.alb_5xx_threshold
  comparison_operator = "GreaterThanThreshold"
  dimensions          = { LoadBalancer = aws_lb.this.arn_suffix }
  alarm_actions       = [var.alarm_topic_arn]        # ← SNS from observability module
  ok_actions          = [var.alarm_topic_arn]
  treat_missing_data   = "notBreaching"
  tags                = var.tags
}

# modules/alb/outputs.tf
output "alb_dns_name"     { value = aws_lb.this.dns_name }
output "alb_zone_id"      { value = aws_lb.this.zone_id }
output "target_group_arn" { value = aws_lb_target_group.app.arn }
output "alb_sg_id"        { value = aws_security_group.alb.id }
output "alb_arn_suffix"   { value = aws_lb.this.arn_suffix }

The asg-app module is the same shape one tier down. Note the app security group whose ingress source is the ALB’s security group (var.alb_sg_id), IMDSv2 forced on the launch template, and target-tracking scaling that holds CPU at 50%:

# modules/asg-app/main.tf
resource "aws_security_group" "app" {
  name_prefix = "sg-${var.name_prefix}-app-"
  description = "App: ${var.app_port} from the ALB only"
  vpc_id      = var.vpc_id

  ingress {
    description     = "App port from the ALB security group"
    from_port       = var.app_port
    to_port         = var.app_port
    protocol        = "tcp"
    security_groups = [var.alb_sg_id]     # ← source SG, never a CIDR
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]           # outbound via the NAT gateway
  }
  tags = var.tags
}

resource "aws_launch_template" "app" {
  name_prefix            = "lt-${var.name_prefix}-"
  image_id               = var.ami_id
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.app.id]

  iam_instance_profile { arn = var.instance_profile_arn }
  metadata_options { http_tokens = "required" }   # IMDSv2 only

  user_data = base64encode(var.user_data)
  tag_specifications {
    resource_type = "instance"
    tags          = merge(var.tags, { Name = "ec2-${var.name_prefix}" })
  }
}

resource "aws_autoscaling_group" "app" {
  name                = "asg-${var.name_prefix}"
  vpc_zone_identifier = var.private_subnet_ids    # ← private subnets from vpc module
  target_group_arns   = [var.target_group_arn]    # ← from the alb module
  health_check_type   = "ELB"
  min_size            = var.min_size
  max_size            = var.max_size
  desired_capacity    = var.desired_capacity

  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }

  dynamic "tag" {
    for_each = var.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_autoscaling_policy" "cpu" {
  name                   = "tt-cpu-${var.name_prefix}"
  autoscaling_group_name = aws_autoscaling_group.app.name
  policy_type            = "TargetTrackingScaling"
  target_tracking_configuration {
    predefined_metric_specification { predefined_metric_type = "ASGAverageCPUUtilization" }
    target_value = 50
  }
}

The rds module hardens the data tier and — the strongest single SRE line in the whole lesson — sets manage_master_user_password = true so the master credential is generated and rotated in Secrets Manager and never appears in tfvars or state:

# modules/rds/main.tf
resource "aws_security_group" "db" {
  name_prefix = "sg-${var.name_prefix}-db-"
  description = "DB: ${var.db_port} from the app tier only"
  vpc_id      = var.vpc_id

  ingress {
    description     = "DB port from the app security group"
    from_port       = var.db_port
    to_port         = var.db_port
    protocol        = "tcp"
    security_groups = [var.app_sg_id]     # ← source SG, never a CIDR
  }
  tags = var.tags
}

resource "aws_db_instance" "this" {
  identifier              = "rds-${var.name_prefix}"
  engine                  = "mysql"
  engine_version          = var.engine_version
  instance_class          = var.instance_class      # per-env sizing
  allocated_storage       = var.allocated_storage
  storage_type            = "gp3"
  storage_encrypted       = true
  kms_key_id              = var.kms_key_arn
  multi_az                = var.multi_az             # ← false in dev, true in staging/prod
  db_subnet_group_name    = var.db_subnet_group_name # isolated DB subnets
  vpc_security_group_ids  = [aws_security_group.db.id]

  username                    = var.master_username
  manage_master_user_password = true                 # ← Secrets Manager; no password in state

  backup_retention_period = var.backup_retention
  deletion_protection     = var.deletion_protection
  skip_final_snapshot     = var.skip_final_snapshot
  performance_insights_enabled = true
  tags                    = var.tags
}

The observability module owns the shared SNS topic and email subscription that every module’s alarms publish to, plus a per-env dashboard the root fills in; every tier module takes its alarm_topic_arn as an input and creates its own alarms (as the ALB module did above). That is what makes an alarm-less resource impossible to create.

The catalogue below is the library contract at a glance — what each module wraps, its key inputs, and the outputs downstream modules consume:

The load-bearing idea is composition by output passing. A root doesn’t instantiate modules side by side; it wires them — the observability module’s alarm_topic_arn feeds every tier module’s alarms, the vpc module’s subnet IDs feed alb, asg-app and rds, and each tier’s security-group ID feeds the next. That wiring is also the dependency graph: Terraform sees asg-app reference alb.alb_sg_id and orders them automatically — no depends_on needed. This table is the wiring map you keep in your head:

How a root references a module — its source and version pin — is the other half of “versioned library”. The four forms, and when each is right:

The rule that makes upgrades safe: always pin (a ?ref tag or a version constraint), and let dev adopt a new version before staging and prod — the same promotion flow you use for infrastructure applies to the library itself. For the deep mechanics of module contracts — semver, optional() object attributes, sensitive propagation, publishing — see the authoring reusable modules lesson; here the point is how the library is consumed and wired into a three-tier platform.

Community modules: terraform-aws-modules vs roll-your-own

You do not have to write every module. The terraform-aws-modules organisation on the public registry maintains excellent, widely-used modules for exactly the building blocks in this architecture — vpc, alb, rds, autoscaling — and the real engineering call is which to adopt and which to own. The honest answer is not “always community” or “always your own”: it depends on how much the module’s surface is commodity versus bespoke, and on who you want maintaining its hardening.

The vpc module is the poster child for adopt community. A correct VPC across three AZs — public/private/database subnets, route tables, an internet gateway, NAT gateways (one per AZ in prod, a single one in dev), and the database subnet group — is a lot of fiddly, well-understood resources that terraform-aws-modules/vpc/aws gets right and maintains for you. Writing it yourself is a rite of passage (do it once in the VPC lesson to understand every route), but in a platform you consume it:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.8"

  name = "vpc-${local.name_prefix}"
  cidr = var.vpc_cidr

  azs              = var.azs                     # e.g. ["ap-south-1a","ap-south-1b"]
  public_subnets   = var.public_subnet_cidrs
  private_subnets  = var.private_subnet_cidrs
  database_subnets = var.database_subnet_cidrs

  enable_nat_gateway     = true
  single_nat_gateway     = var.single_nat_gateway      # true in dev (1 NAT), false in prod
  one_nat_gateway_per_az = !var.single_nat_gateway     # one per AZ in prod
  create_database_subnet_group = true
  enable_dns_hostnames   = true

  tags = local.common_tags
}

By contrast, the alb, asg-app and rds modules in Meridian’s library are rolled by hand — not because the community modules are bad, but because these carry the platform’s opinions: the exact health-check path, the SG-referencing-SG chain, the IMDSv2 requirement, the Secrets-Manager-managed password, the alarm set. When a module encodes decisions you must own and defend to an auditor, owning it is worth the maintenance. The decision grid:

The per-building-block call Meridian made, which is a reasonable default for a mid-size estate:

The trap with community modules is version pinning. A community module is someone else’s code that changes on their schedule; an unpinned source (or a loose >=) means a future terraform init can pull a new major version with breaking input changes and blow up an environment you didn’t touch. Always pin with ~> (allow patch/minor, forbid the breaking major), read the CHANGELOG before a bump, and upgrade dev first. The community-module version is promoted through environments exactly like your own module versions.

Multi-environment: tfvars-per-env vs Terragrunt

You now have a library and a root that composes it. The multi-environment problem is: how do you get dev, staging and prod from that root without duplicating the composition and without their states ever colliding? There are two mainstream answers, and choosing between them is the central decision of this section.

Answer 1 — tfvars-per-env (native Terraform). Keep a root config per environment/layer folder (they are near-identical), and drive the differences with a per-env *.auto.tfvars and a per-env partial backend config. The backend block is left empty in code and completed at init time, so each environment/layer gets the same bucket but a different key:

# environments/prod/app/versions.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  backend "s3" {}          # ← empty: completed by -backend-config at init
}

provider "aws" {
  region = var.region
  default_tags { tags = local.common_tags }   # every resource inherits these tags
}

# environments/prod/app/backend.hcl  →  terraform init -backend-config=backend.hcl
bucket         = "meridian-tfstate-apsouth1"
key            = "prod/app/terraform.tfstate"   # dev/staging + network/data differ ONLY here
region         = "ap-south-1"
dynamodb_table = "meridian-tf-locks"            # or, on TF 1.10+: use_lockfile = true
encrypt        = true

# environments/prod/app/prod.auto.tfvars
region             = "ap-south-1"
app_instance_type  = "m6i.large"   # dev: t3.small
asg_min            = 3             # dev: 1
asg_desired        = 3             # dev: 1
asg_max            = 9             # dev: 2
rds_multi_az       = true          # dev: false
monthly_budget     = 60000         # INR; dev much smaller

The strength is that it is just Terraform — no extra tool, every folder is self-contained and reviewable, and juniors already understand it. The weakness is repetition: the versions.tf, provider block and main.tf composition are copy-pasted across environments and layers, so a change is a multi-place edit and the folders can drift.

Answer 2 — Terragrunt (DRY). Terragrunt keeps the composition in one place and generates the backend and provider per environment, so the only thing in each env folder is its inputs. A root terragrunt.hcl defines the backend once (with a per-unit key derived from the path); each child includes it, points at a stack, declares its dependency on other units, and supplies inputs:

# live/terragrunt.hcl  (root — defined ONCE)
remote_state {
  backend = "s3"
  generate = {
    path = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "meridian-tfstate-apsouth1"
    key            = "${path_relative_to_include()}/terraform.tfstate"  # per-unit key
    region         = "ap-south-1"
    dynamodb_table = "meridian-tf-locks"
    encrypt        = true
  }
}

# live/prod/app/terragrunt.hcl  (a unit)
include "root" { path = find_in_parent_folders() }
terraform { source = "${get_repo_root()}/modules//app-stack" }

dependency "network" {
  config_path  = "../network"
  mock_outputs = {                                # lets `plan` run before network exists
    vpc_id          = "vpc-mock"
    public_subnets  = ["subnet-mock"]
    private_subnets = ["subnet-mock"]
  }
}

inputs = {
  vpc_id             = dependency.network.outputs.vpc_id
  public_subnet_ids  = dependency.network.outputs.public_subnets
  private_subnet_ids = dependency.network.outputs.private_subnets
  app_instance_type  = "m6i.large"
  asg_desired        = 3
}

Now dev/staging/prod differ only in their inputs; the backend, provider and stack source are DRY, and dependency blocks wire the layers with first-class mock_outputs so a plan runs before the network exists. The cost is the extra tool and the wrapper indirection. The Terragrunt mechanics — generate, dependency, run-all, mocks — and the approval gates are covered in the multi-environment 3-tier lesson.

Choose with this matrix, not by fashion:

The decision rule Meridian used: start native while there are three environments and three layers and one platform team; adopt Terragrunt when the number of (environment × layer × account) combinations makes the copy-paste unmanageable, or when you split into per-team AWS accounts and need generated per-account backends.

Either way, remote state is per-environment-per-layer, and the S3 + lock backend is what makes it safe. The key scheme is the single thing that stops a dev apply from planning to destroy prod:

DynamoDB provides the lock: Terraform writes a lock item keyed by the state path before an operation and deletes it after, so two applies against the same key serialise while different keys apply concurrently. On Terraform 1.10+ you can drop the table entirely and set use_lockfile = true, which uses a native .tflock object in S3 (leaning on S3’s conditional writes) — one fewer resource to run. Enable S3 bucket versioning on the state bucket either way, so a corrupt write is recoverable.

Promotion — dev → staging → prod. The point of shared modules and per-env tfvars is that a change is promoted, not rewritten. You prove it in dev, then run the same module code against staging with staging’s tfvars, then prod behind an approval. Nothing about the resources is hand-edited between environments; only the sizing variables change:

The per-environment sizing that carries those differences — the only thing that changes between environments, because the module code is identical — is the promotion story made concrete:

SRE as code: tags, alarms, dashboards, budgets, IAM & blast radius

The difference between infrastructure and a platform is that reliability is a property of every resource, guaranteed by the code, not something SRE bolts on after the first incident. This is the SRE plane, and it is the densest part of the lesson because there are several controls and each is a small, standard pattern you apply everywhere.

Standardized tags via default_tags. AWS lets you set default_tags once on the provider and every resource that supports tagging inherits them — no more forgetting a tag on the one resource that matters for cost allocation:

# environments/prod/app/locals.tf
locals {
  environment = "prod"
  workload    = "meridian-shop"
  name_prefix = "${local.workload}-${local.environment}"   # e.g. meridian-shop-prod

  common_tags = {
    environment = local.environment
    workload    = local.workload
    managed_by  = "terraform"
    cost_center = var.cost_center
    owner       = var.owner_email
    repo        = "meridian/platform"
    layer       = "app"
  }
}

Alarms and a dashboard. Each tier module emits its own aws_cloudwatch_metric_alarm (we saw the ALB 5xx alarm); the observability module owns the SNS topic they all publish to and a dashboard. The standard alarm set per tier:

# modules/observability/main.tf (excerpt)
resource "aws_sns_topic" "alarms" {
  name = "sns-${var.name_prefix}-alarms"
  tags = var.tags
}
resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alarms.arn
  protocol  = "email"
  endpoint  = var.oncall_email
}
output "alarm_topic_arn" { value = aws_sns_topic.alarms.arn }

Budgets. Cost is a reliability signal; a runaway apply that spins up an r6g.24xlarge should page someone. aws_budgets_budget fires at thresholds of actual and forecast spend:

resource "aws_budgets_budget" "monthly" {
  name         = "budget-${local.name_prefix}"
  budget_type  = "COST"
  limit_amount = tostring(var.monthly_budget_usd)
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = [var.oncall_email]
  }
  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = [var.oncall_email]
  }
}

Least-privilege IAM — per env and per state. Access is IAM roles at the tightest scope that works, and — the platform-specific twist — the CI role that applies an environment is scoped so it can only touch that environment’s state object and lock. The app instances get an instance profile that can read one S3 prefix and one secret, nothing more:

# The CI deploy role for prod/app can only touch THIS env+layer's state + lock
data "aws_iam_policy_document" "prod_app_state" {
  statement {
    actions   = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
    resources = ["arn:aws:s3:::meridian-tfstate-apsouth1/prod/app/*"]  # only its key prefix
  }
  statement {
    actions   = ["s3:ListBucket"]
    resources = ["arn:aws:s3:::meridian-tfstate-apsouth1"]
    condition {
      test     = "StringLike"
      variable = "s3:prefix"
      values   = ["prod/app/*"]
    }
  }
  statement {
    actions   = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:DeleteItem"]
    resources = ["arn:aws:dynamodb:ap-south-1:*:table/meridian-tf-locks"]
  }
}

The role the pipeline assumes is created via GitHub OIDC (no long-lived keys), which the OIDC pipeline lesson covers end to end. The least-privilege catalogue you assign from most:

Blast radius — separate state per layer. We split each environment into network, app and data states precisely so a failure is contained. The app layer reads the network layer’s outputs through a terraform_remote_state data source rather than sharing state:

# environments/prod/app/main.tf — read the network layer's outputs, don't share its state
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "meridian-tfstate-apsouth1"
    key    = "prod/network/terraform.tfstate"
    region = "ap-south-1"
  }
}
# ...then: vpc_id = data.terraform_remote_state.network.outputs.vpc_id

Drift detection. Drift is when reality diverges from state — someone hotfixes a security group in the console, or an AWS-side change mutates a field. The SRE practice is a scheduled terraform plan (per env, per layer) in CI: an empty plan means no drift; a non-empty plan is an alert to investigate and reconcile (re-apply to correct, or import/moved/update code to accept the change).

The SRE-controls catalogue — every control, the Terraform resource, and the standard we enforce:

Hands-on: build it with Terraform

Now assemble a minimal but real slice end to end: a prod/app root that composes observability, alb, asg-app and reads the network layer’s outputs, plus the data layer’s rds, all with default_tags, per-env sizing and per-env remote state. This is the composition, the output-passing, the S3+DynamoDB state, the tiered security groups, the alarms and the SRE wiring in one runnable stack. ⚠️ This creates real AWS resources — an ALB, EC2 instances, a NAT gateway and RDS all cost money by the hour. Destroy at the end.

Step 0 — one-time state backend (bootstrap). The S3 bucket and DynamoDB lock table must exist before any root can use them. Create them once by hand (the one thing you can’t Terraform-with-Terraform on day one):

aws s3api create-bucket --bucket meridian-tfstate-apsouth1 \
  --region ap-south-1 --create-bucket-configuration LocationConstraint=ap-south-1
aws s3api put-bucket-versioning --bucket meridian-tfstate-apsouth1 \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket meridian-tfstate-apsouth1 \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}'
aws dynamodb create-table --table-name meridian-tf-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST --region ap-south-1

Step 1 — the module files. Create modules/observability, modules/alb, modules/asg-app and modules/rds (shown above). Each is self-contained with variables.tf/outputs.tf; the tier modules take alarm_topic_arn and emit their own alarms.

Step 2 — the prod/app root composition. This is the heart: it reads the network layer’s outputs, then composes observability → alb → asg-app, passing subnet IDs, the ACM cert, and the SG/target-group chain between them:

# environments/prod/app/main.tf
data "terraform_remote_state" "network" {
  backend = "s3"
  config  = { bucket = "meridian-tfstate-apsouth1", key = "prod/network/terraform.tfstate", region = "ap-south-1" }
}

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]
  filter{
    name = "name"
    values = ["al2023-ami-*-x86_64"]
  }
}

module "observability" {
  source       = "../../../modules/observability"
  name_prefix  = local.name_prefix
  oncall_email = var.oncall_email
  tags         = local.common_tags
}

module "alb" {
  source            = "../../../modules/alb"
  name_prefix       = local.name_prefix
  vpc_id            = data.terraform_remote_state.network.outputs.vpc_id
  public_subnet_ids = data.terraform_remote_state.network.outputs.public_subnets
  certificate_arn   = var.certificate_arn                 # ACM cert (see note below)
  app_port          = var.app_port
  alarm_topic_arn   = module.observability.alarm_topic_arn
  deletion_protection = var.deletion_protection
  tags              = local.common_tags
}

module "asg_app" {
  source               = "../../../modules/asg-app"
  name_prefix          = local.name_prefix
  vpc_id               = data.terraform_remote_state.network.outputs.vpc_id
  private_subnet_ids   = data.terraform_remote_state.network.outputs.private_subnets
  alb_sg_id            = module.alb.alb_sg_id             # ← SG chain
  target_group_arn     = module.alb.target_group_arn      # ← register instances
  instance_type        = var.app_instance_type
  min_size             = var.asg_min
  desired_capacity     = var.asg_desired
  max_size             = var.asg_max
  ami_id               = data.aws_ami.al2023.id
  instance_profile_arn = var.app_instance_profile_arn
  alarm_topic_arn      = module.observability.alarm_topic_arn
  tags                 = local.common_tags
}

# Route 53 alias → the ALB (ACM cert validated separately)
resource "aws_route53_record" "app" {
  zone_id = var.hosted_zone_id
  name    = var.domain_name
  type    = "A"
  alias {
    name                   = module.alb.alb_dns_name
    zone_id                = module.alb.alb_zone_id
    evaluate_target_health = true
  }
}

Note on the front door: the HTTPS listener needs an ACM certificate (aws_acm_certificate + aws_acm_certificate_validation via a Route 53 record), which requires a real hosted zone/domain. For a no-domain demo, change the ALB module’s listener to protocol = "HTTP" on port 80 and skip the cert/Route 53 record — the rest of the stack is identical.

Every resource that gets created, which module owns it, and why:

Step 3 — init, plan, apply for prod/app. From the layer folder, initialise with the per-env backend config, then plan and apply:

cd environments/prod/app
terraform init -backend-config=backend.hcl        # binds key = prod/app/terraform.tfstate
terraform fmt -check && terraform validate         # cheap gates first
terraform plan  -out=prod-app.tfplan
terraform apply prod-app.tfplan

Representative plan output — note the module addressing and that Terraform ordered observability and alb before asg-app on its own, because of the alarm_topic_arn and alb_sg_id references:

Terraform will perform the following actions:

  # aws_route53_record.app will be created
  # module.observability.aws_sns_topic.alarms will be created
  # module.observability.aws_sns_topic_subscription.email will be created
  # module.alb.aws_security_group.alb will be created
  # module.alb.aws_lb.this will be created
  # module.alb.aws_lb_target_group.app will be created
  # module.alb.aws_lb_listener.https will be created
  # module.alb.aws_cloudwatch_metric_alarm.alb_5xx will be created
  # module.asg_app.aws_security_group.app will be created
  # module.asg_app.aws_launch_template.app will be created
  # module.asg_app.aws_autoscaling_group.app will be created
  # module.asg_app.aws_autoscaling_policy.cpu will be created

Plan: 12 to add, 0 to change, 0 to destroy.

Step 4 — verify. Confirm the platform properties, not just that resources exist: that the tags landed, the SG chain is right, and state went to the correct key:

# default_tags applied to the ALB?
aws elbv2 describe-tags --resource-arns $(terraform output -raw alb_arn) \
  --query "TagDescriptions[0].Tags"

# app SG allows 8080 ONLY from the ALB SG (source group, not a CIDR)?
aws ec2 describe-security-groups --group-ids $(terraform output -raw app_sg_id) \
  --query "SecurityGroups[0].IpPermissions[0].UserIdGroupPairs"

# state landed on the prod/app key?
aws s3 ls s3://meridian-tfstate-apsouth1/prod/app/

Step 5 — promote to staging/prod other layers (conceptual). The identical module code runs for the next environment or layer; you change directory, init with that folder’s backend config (a different key), and apply with that folder’s tfvars. Nothing in modules/ or the composition changes:

cd ../../staging/app
terraform init -backend-config=backend.hcl   # key = staging/app/terraform.tfstate
terraform plan -out=staging-app.tfplan        # same modules, staging.auto.tfvars values
terraform apply staging-app.tfplan

In real life the staging and prod applies run in CI behind the approval gate from the promotion table — prod applies the already-approved plan artifact, it does not re-plan.

Step 6 — destroy and clean up. ⚠️ Tear down each layer you stood up (data last-created is destroyed first if you reverse order; in practice destroy app before data before network only matters for cross-layer reads — destroy each layer’s folder):

cd environments/prod/app  && terraform destroy    # repeat per layer/env you applied
cd environments/prod/data && terraform destroy
# only if you are fully done with the backend:
aws s3 rb s3://meridian-tfstate-apsouth1 --force
aws dynamodb delete-table --table-name meridian-tf-locks --region ap-south-1

Testing the modules & the CI gate

A module library that many environments depend on must be tested before it ships, or a bad v1.5.0 breaks every environment that bumps to it. The testing ladder runs cheapest-first, and all of it belongs in CI on every PR to modules/:

terraform fmt -check -recursive        # style: fails if any file isn't canonical
terraform validate                     # syntax + internal consistency (needs init)
tflint --recursive                     # lint: aws ruleset — invalid instance types, bad refs
checkov -d . --framework terraform     # security: open SGs, unencrypted RDS, public S3
terraform test                         # behaviour: assertions on planned/applied values

Native terraform test is the one that proves behaviour. A *.tftest.hcl file sets inputs and asserts on the result of a plan (no cloud needed) or an apply. This test proves the asg-app module’s least-privilege SG contract — that the app port is sourced from the ALB SG, never a CIDR:

# tests/asg-app.tftest.hcl
variables {
  name_prefix        = "test-app"
  vpc_id             = "vpc-123"
  private_subnet_ids = ["subnet-1", "subnet-2"]
  alb_sg_id          = "sg-alb-123"
  target_group_arn   = "arn:aws:elasticloadbalancing:ap-south-1:0:targetgroup/x/1"
  instance_type      = "t3.small"
  ami_id             = "ami-123"
  app_port           = 8080
}

run "app_sg_sources_from_alb_not_cidr" {
  command = plan
  assert {
    condition     = length(aws_security_group.app.ingress[0].cidr_blocks) == 0
    error_message = "App SG must not use CIDR ingress — it must reference the ALB SG"
  }
}

run "imdsv2_required" {
  command = plan
  assert {
    condition     = aws_launch_template.app.metadata_options[0].http_tokens == "required"
    error_message = "Launch template must require IMDSv2 (http_tokens = required)"
  }
}

The command = plan tests run without AWS credentials for the assertions on planned values, so they are fast CI gates. Wire this into the plan-on-PR pipeline from the GitHub Actions + OIDC lesson: on a PR, CI assumes a read-only role via OIDC and runs fmt/validate/tflint/checkov/test + plan, posting the plan as a comment; on merge it assumes the scoped deploy role and applies the approved plan. Run checkov as a required status check so a module can never merge with a regressed control — an open database security group should fail the PR, not the production incident.

Variables, outputs & making it reusable

The whole lesson has been about reuse, so this section is the last mile: parameterise cleanly and know when to stop rolling your own. The environments already share modules and differ only by tfvars — that is the for_each-over-environments idea expressed as folders. For a platform, the explicit folder-per-env-per-layer is usually clearer and safer than a clever single-map loop, because each unit stays independently reviewable and independently applied.

The reuse decision that matters most at scale is your module vs a community module, covered earlier — the tell is maintenance burden. If you find yourself re-implementing what terraform-aws-modules/vpc/aws already ships (route tables, NAT-per-AZ, subnet groups), swap to it and spend your effort on the composition that is actually yours. Keep the module interface stable even when you swap the implementation: if alb exposes target_group_arn and alb_sg_id, the environments consuming it do not care whether inside it is your aws_lb or a community module — which is exactly why a good output contract is the most valuable thing in the library.

Common mistakes and troubleshooting

The failures on a platform are different from the failures on a single config — they cluster around cross-module wiring, per-layer state, security-group references, community-module versions, and drift across environments. This is the map:

Five gotchas deserve prose because they are the ones that cost hours:

Cross-module dependencies are implicit — trust the graph. New platform engineers reflexively add depends_on between modules “to be safe”. Don’t. When module.asg_app takes module.alb.alb_sg_id, Terraform already knows the ALB must come first; adding depends_on on top can create false ordering or cycles. Wire modules by passing outputs, and let the reference be the dependency. Reserve depends_on for genuinely hidden dependencies (e.g. an IAM policy that must exist before an instance uses it).

State-per-layer is a discipline, not a default. The single most dangerous moment on a platform is terraform init binding to the wrong state. If two folders ever init with the same key, they share a state and the second apply plans to destroy the first’s resources. Make the key explicit per folder (backend.hcl), template it from the path in CI so it cannot be wrong by hand, and make “confirm the key in the plan header” a step in your runbook. Splitting network/app/data means the daily-churning app layer can never propose a change to the VPC or the database — that is blast-radius thinking made concrete in the state keys.

Reference security groups, never CIDRs — and prove it in a test. The most common VPC mistake is opening the DB security group to the whole VPC CIDR “to make it work”. It works, and it also means every compromised instance in the VPC can reach the database. Source the DB SG’s ingress from the app SG id and the app SG’s ingress from the ALB SG id; the rules then survive every scale event without change, and the terraform test that asserts “no CIDR ingress on the app SG” stops a regression at the PR.

Community-module version pinning is promotion, not a one-off. A community module is code you don’t control that changes on someone else’s schedule. Pin it (~> 5.8), and when you bump it, treat it exactly like your own module upgrade: bump dev first, read the module’s CHANGELOG for input/behaviour changes, let it soak, then promote the same version to staging and prod. An unpinned community module is a time bomb that goes off on the next unrelated init.

Drift across environments is normal signal — reconcile deliberately. Because each env/layer has its own state, drift shows up per unit, and it is usually one that got hand-touched (a dev hotfix in the console that never made it to code, or a prod emergency change). The scheduled terraform plan per unit surfaces it; the discipline is to reconcile — re-apply to bring reality back to code, or bring code up to reality with import/moved/an edit — and to trace who changed it out of band, because unmanaged change is the thing a platform exists to eliminate.

Cost, cleanup & production notes

The hands-on slice is not free — several of its resources bill by the hour whether or not traffic flows. The rough monthly cost if you leave it running in ap-south-1, and how to kill it:

The dominant surprises are the NAT gateway (which bills hourly even idle — use a single NAT in non-prod) and RDS Multi-AZ (which doubles the database compute — keep it off in dev, on in staging/prod). Both are controlled by the per-env sizing variables, which is exactly why sizing lives in tfvars. Destroy each layer with terraform destroy in its folder; keep the bootstrap S3 bucket and DynamoDB table unless you are fully finished.

Five production-hardening notes to carry beyond the demo:

1. State is the crown jewels — remote, locked (DynamoDB or use_lockfile), versioned and encrypted (SSE-KMS on the bucket), access-controlled. One state per env per layer keeps a loss to one blast radius, and versioning makes a bad write recoverable.
2. Least-privilege to the state itself — the CI role for prod/app can touch only the prod/app/* key prefix and the lock table; no role has AdministratorAccess. Combine with OIDC so there are no long-lived keys anywhere.
3. Reliability is a module property — every tier module emits its own alarms to a shared SNS topic; a resource with no alarm should be impossible to create, not a thing you discover mid-incident. RDS manages its own password in Secrets Manager so no credential ever enters state.
4. Design blast radius in the tree — separate state per layer, tiered security groups referencing SG ids not CIDRs, subscriptions/accounts per environment where you can. The failure of one unit costs exactly that unit.
5. Promote, never hand-edit prod — the same module code walks dev → staging → prod; prod applies an approved plan artifact from CI. The day someone fixes prod in the console is the day drift begins.

Cheat-sheet

The dense quick-reference for building a three-tier AWS platform with Terraform.

Core resources

Backend & state (s3)

Commands

Community modules

Interview and exam questions

1. Why push all state down to the data tier in a three-tier architecture? Because the web and app tiers become stateless and disposable — an instance can die and be replaced by the ASG with no data loss — so only the data tier needs expensive durability and failover. That is what makes scaling, patching and failure routine: you replace stateless instances freely and reserve Multi-AZ, backups and deletion protection for RDS.

2. Why reference a security group id instead of a CIDR block between tiers? Because the membership is dynamic. When the app tier scales from 2 to 20 instances, a rule that allows the app security group automatically covers every new instance; a CIDR rule would need editing (or, worse, be opened to the whole VPC). SG-referencing-SG rules survive every scale event and keep the DB reachable only from the app tier.

3. How does a root config order module creation without depends_on? By reference: when module.asg_app uses module.alb.alb_sg_id and target_group_arn, Terraform’s graph orders the ALB first automatically. Passing outputs is declaring the dependency; explicit depends_on between modules is usually wrong and can create cycles.

4. Why split each environment into network/app/data states instead of one state per env? Blast radius. The network changes rarely and underpins everything; the data tier is the crown jewels; the app tier churns daily. Separate state per layer means the daily app apply can never plan a change to the VPC or drop the database, and a data migration is isolated to the DBA-gated data state. Layers read each other via terraform_remote_state.

5. tfvars-per-env vs Terragrunt — when do you switch? Start native (tfvars + partial S3 backend) for ≤3 environments and a few layers with one team. Switch to Terragrunt when the (env × layer × account) count makes copy-paste unmanageable, when you need generated per-account backends, or when you want first-class dependency wiring and run-all across many units.

6. When do you adopt terraform-aws-modules/* versus rolling your own? Adopt for commodity, fiddly, well-maintained surfaces (VPC is the poster child). Roll your own when the module encodes a platform opinion you must own and defend — the health-check path, the SG chain, IMDSv2, a Secrets-Manager-managed password, the alarm set. Mixing is normal: community VPC, your own ALB/ASG.

7. How do you keep the RDS master password out of state? Set manage_master_user_password = true so RDS generates and rotates the credential in Secrets Manager; the password never appears in tfvars or state. The app reads it from Secrets Manager at runtime via its instance-profile permission to that one secret.

8. Your dev plan proposes to destroy prod resources. Diagnose. Two folders are sharing a state key — dev init’d against prod’s key, or they were never separated. Stop, do not apply, fix the key per env/layer, re-init, and confirm the plan header points at the right state.

9. (Associate-style) The backend "s3" block references var.bucket. What happens? It fails at init — the backend is read before variables/locals are evaluated, so no interpolation is allowed there. Move the values to partial config: terraform init -backend-config=backend.hcl (or -backend-config="key=prod/app/terraform.tfstate").

10. (Associate-style) How does the S3 backend prevent two concurrent applies from corrupting state? With a lock: historically a DynamoDB table (a LockID item written before the operation and deleted after), and on Terraform 1.10+ a native S3 lockfile (use_lockfile = true) using S3 conditional writes. Two applies on the same key serialise; different keys apply concurrently. Enable bucket versioning so a bad write is recoverable.

11. What is the blast-radius argument for a CI deploy role scoped to one state key? State is a failure domain and a privilege boundary. A role that can only read/write the prod/app/* key prefix and the lock table can only ever damage the prod app layer — not the network, not the data, not another environment. Combined with OIDC (no static keys) it means a leaked CI credential is contained to one unit.

12. How do you detect and handle drift across a multi-env, multi-layer estate? Run a scheduled terraform plan per environment per layer in CI; an empty plan means no drift, a non-empty plan is an alert. Reconcile deliberately — re-apply to restore code-as-truth, or bring code to reality via import/moved/an edit — and trace the out-of-band change so it stops recurring.

Key takeaways

• A platform is reuse plus isolation. One versioned modules/ library (vpc, alb, asg-app, rds, observability) consumed by per-env, per-layer roots, each with its own S3 state key and a lock — that combination is the whole game.
• The three-tier shape is disciplined statelessness. Public ALB → private ASG → isolated RDS Multi-AZ, with NAT for egress and state pushed all the way down, so the web/app tiers are disposable and only the data tier carries durability.
• Compose by passing outputs. The root wires modules (subnet IDs into every tier, the SG chain ALB→app→DB, the target-group ARN into the ASG); the reference is the dependency, so skip depends_on.
• Tier security by referencing security groups, not CIDRs — ALB SG from the internet, app SG from the ALB SG, DB SG from the app SG — so the rules survive every scale event, and prove it with a terraform test.
• Isolate and promote deliberately — tfvars-per-env for a small estate, Terragrunt when (env × layer × account) grows — and move the same code dev → staging → prod, changing only sizing variables, never hand-editing prod.
• Adopt community modules for commodity, own them for opinions — terraform-aws-modules/vpc/aws for the network, your own alb/asg-app/rds for the platform’s decisions — and pin every version, promoting bumps like any other change.
• SRE is code, on every resource — default_tags, standardized CloudWatch alarms to one SNS topic, budgets, a Secrets-Manager-managed DB password, least-privilege IAM scoped even to the state key, and a scheduled drift plan.
• Blast radius is designed in the state keys — network / app / data isolated per environment, the CI role scoped to its own key, so no single failure or mistake can reach the whole estate.