Every workload that has to survive a busy Monday and a quiet Sunday eventually needs the same thing on AWS: a pool of identical instances that grows when load arrives, shrinks when it leaves, replaces anything that dies, and rolls out a new build without a maintenance window. That is the job of the Auto Scaling Group (ASG) — and the ASG is only ever as good as the Launch Template that tells it what an instance should look like. Click these together in the console and you get a snapshot no one can reproduce; worse, you get a Launch Configuration, the immutable, feature-frozen predecessor AWS has spent years steering everyone off. This lesson builds the whole elastic-compute tier in Terraform, the way you would run it in production: a versioned Launch Template, an ASG spread across two Availability Zones, registered into an Application Load Balancer’s target group, scaling itself against CPU, and deploying new AMIs with a rolling instance refresh.
By the end you will have stood up, from an empty directory, a real Launch Template (AMI, instance type, user_data, IMDSv2, encrypted gp3 root volume, tag specifications), an Auto Scaling Group across two AZs (min/max/desired, health_check_type = "ELB", target_group_arns, instance_refresh), an ALB target group the group self-registers into, and a target-tracking scaling policy that holds average CPU at 50% by managing its own CloudWatch alarms. You will curl the load balancer to watch it serve, drive a load test to watch desired_capacity climb, read the truth from aws autoscaling describe-auto-scaling-groups, then tear it all down with terraform destroy. Above all you will learn the two patterns that separate a toy ASG from a production one: the create_before_destroy + name_prefix dance that makes Launch Template changes zero-downtime, and the instance_refresh block that turns “deploy a new AMI” into a safe, batched, health-gated rollout.
This is the provider-specific, hands-on layer of the course. It assumes you already know core Terraform — HCL, providers, resources, variables, state and modules from the Foundation and Intermediate tiers — and applies it to a real cloud, relentlessly, with copy-pasteable .tf files and a terraform init → plan → apply → verify → destroy you actually run.
What you’ll build
The scenario is the one you meet on day one of almost any AWS project that runs code on EC2: a stateless web tier that must be reachable behind a public load balancer, survive a single instance (or a whole AZ) dying, scale on demand, and take a new build without downtime. That is an Auto Scaling Group behind an Application Load Balancer, end to end. The instances are cattle, not pets — every one is stamped from the same Launch Template, boots the same user_data, and is interchangeable. The ASG owns their lifecycle: it keeps desired_capacity instances healthy across two AZs, replaces any that fail an ELB health check, registers each new one into the ALB’s target group, and drains and terminates them on scale-in.
Reading the diagram below left to right is reading exactly what a single terraform apply wires together. Terraform renders the Launch Template (the versioned blueprint: which AMI, which instance type, the user_data, the security groups, the IAM instance profile). The Auto Scaling Group references that template and spreads instances across two AZs via vpc_zone_identifier. Every instance the group launches self-registers into the ALB target group through target_group_arns — you never attach targets by hand. And a target-tracking policy watches average CPU on CloudWatch, adding instances when the fleet runs hot and removing them when it cools, holding the target you set.
The six badges call out the decisions that matter: (1) a Launch Template rather than the deprecated Launch Configuration; (2) the ASG spanning two AZs; (3) target_group_arns self-registration; (4) the target-tracking policy that manages its own alarms; (5) instance_refresh for rolling deploys; and (6) the create_before_destroy + name_prefix pattern for zero-downtime template swaps. Each is a section below.
Why Terraform rather than the console, a CLI script, or CloudFormation? Because this tier is a graph of tightly-coupled resources — a Launch Template, an ASG, an ALB, a target group, a listener, two security groups, an IAM role and instance profile, a scaling policy — where every one holds an ID or ARN another references. Terraform’s dependency graph wires those references for you, shows the exact diff in plan before touching anything, and lets you stamp the identical fleet into dev, staging and prod from one module with different variables. The console gives you none of that reproducibility; CloudFormation gives you the graph but not the plan preview ergonomics, the multi-cloud state model, or the for_each patterns you already know from the core tiers.
Here is the full inventory a single terraform apply creates, and roughly what each part costs if you leave it running (Mumbai ap-south-1, on-demand, indicative July 2026):
| Resource (Terraform) | AWS object | Role in the build | Rough cost if left up |
|---|---|---|---|
aws_launch_template |
Launch Template | The versioned instance blueprint | Free |
aws_autoscaling_group |
Auto Scaling Group | Owns instance lifecycle across 2 AZs | Free (you pay for instances) |
aws_instance (via ASG) ×2 |
2× t3.micro EC2 |
The web fleet | ~₹1,700/mo for two |
aws_lb (application) |
Application Load Balancer | Public L7 entry, spreads traffic | ~₹1,600/mo + LCU |
aws_lb_target_group |
Target group | The registration + health target | Free |
aws_lb_listener |
Listener :80 | Forwards to the target group | Free |
aws_security_group ×2 |
ALB SG + instance SG | Allow 80 in; instance only from ALB | Free |
aws_iam_role + aws_iam_instance_profile |
Instance role | SSM access, no SSH keys needed | Free |
aws_autoscaling_policy |
Target-tracking policy | Holds 50% CPU; makes its own alarms | Free (alarms ~₹0) |
The ALB and the two instances are the only real line items, and both are modest — this is a build it, verify it, destroy it lesson you can run for well under ₹100 if you tear it down the same hour. Every costly or destructive step below is marked ⚠️.
Where this fits: the ALB here is deliberately minimal so the lesson stays about scaling. The load-balancing tier in full — ALB vs NLB vs Gateway, listeners, rules, path routing, and the target-group mechanics — is the subject of the ELB: ALB, NLB & Target Groups lesson; this lesson consumes a target group from it. The single-instance building blocks — aws_instance, security groups, key pairs and the IAM instance profile — are built in the Security Groups, EC2 & Key Pairs lesson. And the CloudWatch alarms and SNS notifications a production ASG hangs off are covered in the CloudWatch, SNS & Observability lesson; here we let the target-tracking policy create its alarms implicitly.
Launch Templates: the versioned instance blueprint
An Auto Scaling Group does not know what an instance is. It knows how many it should have and where to put them; the Launch Template supplies the what — the AMI, the instance type, the network and security configuration, the storage, the user_data, the IAM role, the tags. Every instance the ASG launches is stamped from one version of one template, which is exactly why the fleet is homogeneous and disposable.
The first decision is not an argument at all — it is which resource. AWS has two blueprint resources, and one of them is a trap. The Launch Configuration (aws_launch_configuration) is the old one: immutable (you cannot edit it — you replace it), single instance type, no versioning, and frozen out of every feature added since ~2017. The Launch Template (aws_launch_template) is the current one: versioned, supports multiple instance types (via the ASG’s mixed-instances policy), spot, IMDSv2, multiple network interfaces, T-instance credit specification, placement, licensing, and tag specifications. AWS recommends Launch Templates for all new work and the console no longer offers Launch Configurations for new accounts.
| Capability | Launch Configuration (aws_launch_configuration) |
Launch Template (aws_launch_template) |
|---|---|---|
| Status | Legacy, no new features | Current, recommended |
| Versioning | None (immutable, replace-only) | Numbered versions + $Latest/$Default |
| Multiple instance types | No | Yes (with ASG mixed-instances) |
| Spot + On-Demand mix | No | Yes (mixed-instances policy) |
| IMDSv2 enforcement | Limited | Yes (metadata_options) |
| Multiple block devices | Basic | Full block_device_mappings |
| Tag on launch | No | Yes (tag_specifications) |
| Terraform replace on edit | Always (forces new) | Only for immutable fields; else new version |
| Use it when | Never, for new builds | Always |
The practical upshot: never write aws_launch_configuration for anything new. If you inherit one, migrating to a Launch Template is a small, mechanical change that unlocks versioning and everything above.
Here is a Launch Template with the arguments you actually set in production. Note base64encode on user_data, the IMDSv2 enforcement, the encrypted gp3 root volume, tags applied to the instances at launch, and the lifecycle block we will justify in the ASG section:
resource "aws_launch_template" "web" {
name_prefix = "kv-web-" # name_prefix, not name (see create_before_destroy)
image_id = data.aws_ami.al2023.id # a data source, not a hard-coded AMI
instance_type = var.instance_type # e.g. t3.micro
key_name = var.key_name # optional; we use SSM instead in the demo
vpc_security_group_ids = [aws_security_group.instance.id]
iam_instance_profile {
arn = aws_iam_instance_profile.web.arn
}
# user_data MUST be base64-encoded; the OS runs it once on first boot.
user_data = base64encode(local.user_data)
block_device_mappings {
device_name = "/dev/xvda" # root device for Amazon Linux 2023
ebs {
volume_size = 8
volume_type = "gp3" # gp3 > gp2: cheaper, decoupled IOPS
encrypted = true
delete_on_termination = true
}
}
metadata_options {
http_tokens = "required" # IMDSv2 only — blocks SSRF-to-creds
http_endpoint = "enabled"
http_put_response_hop_limit = 1
}
monitoring { enabled = true } # 1-minute EC2 metrics → faster scaling
tag_specifications {
resource_type = "instance"
tags = { Name = "kv-web", role = "web" }
}
tag_specifications {
resource_type = "volume"
tags = { Name = "kv-web-vol" }
}
tags = { project = "tf-course", lesson = "asg" } # tags on the template itself
lifecycle {
create_before_destroy = true
}
}
The top-level aws_launch_template arguments, and why each matters:
| Argument | Purpose | Notes / gotcha |
|---|---|---|
image_id |
The AMI to boot | Use a data.aws_ami lookup, not a literal — AMIs are region-specific and rotate |
instance_type |
Default size | Overridable per-instance-type in the ASG mixed-instances policy |
key_name |
SSH key pair name | Optional; prefer SSM Session Manager (keyless) — omitted in the demo |
vpc_security_group_ids |
SGs in a VPC | Use this in a VPC; security_group_names is EC2-Classic only |
iam_instance_profile |
Instance role (by arn or name) |
How the app gets AWS creds without keys |
user_data |
First-boot script | Must be base64encode(...); runs once |
block_device_mappings |
Root + extra EBS | Set gp3, encrypted = true, delete_on_termination |
metadata_options |
IMDS config | http_tokens = "required" forces IMDSv2 |
monitoring |
Detailed (1-min) metrics | On = faster, more responsive scaling |
instance_market_options |
Spot request | Makes all instances spot; for a mix use the ASG |
tag_specifications |
Tags applied at launch | Per resource_type (instance, volume, …) |
network_interfaces |
ENI config | Public IP, multiple ENIs, security groups per-ENI |
credit_specification |
T-instance CPU credits | "unlimited" to avoid throttling under burst |
update_default_version |
Bump $Default on change |
Handy when consumers pin $Default |
Block device mappings deserve a table of their own, because a wrong device_name or a forgotten encrypted is a common production miss:
ebs {} argument |
Purpose | Sane default |
|---|---|---|
volume_size |
Size in GiB | 8–30 for a stateless web root |
volume_type |
gp3, gp2, io2, … |
gp3 (cheaper, tunable IOPS/throughput) |
iops |
Provisioned IOPS | Only for gp3/io2; gp3 baseline is 3000 |
throughput |
MB/s (gp3 only) | 125 default; raise for logs/DB |
encrypted |
Encrypt at rest | true — always |
kms_key_id |
CMK for encryption | Omit for the AWS-managed key |
delete_on_termination |
Delete the volume with the instance | true for a stateless root |
Versions are the whole reason Launch Templates exist. Every change produces a new numbered version (1, 2, 3, …), and two symbolic aliases let the ASG track them: $Latest always points at the newest version, and $Default points at whichever version you nominate as default. Which one the ASG references changes the deploy story completely:
| Version reference | Meaning | Deploy behaviour | Use when |
|---|---|---|---|
$Latest |
Newest version, always | New launches use the new template immediately; existing instances unchanged until refreshed | You drive rollouts with instance_refresh |
$Default |
The nominated default | New launches use the default; bump default_version to promote |
You want an explicit promote step |
"3" (pinned) |
One exact version | Frozen; nothing changes until you edit the number | You need a hard pin / rollback target |
The idiomatic production pattern is version = "$Latest" on the ASG plus an instance_refresh block: you edit the template (say, a new AMI), Terraform creates a new version, and the refresh rolls the fleet onto it in batches. Pinning to a literal version is your rollback lever — set it to the last-known-good number and re-apply.
Spot instances can be requested directly in the template via instance_market_options — but understand the blast radius: this makes every instance the template launches a spot instance. That is fine for a fault-tolerant batch fleet, but a web tier usually wants a mix of on-demand (for a stable floor) and spot (for cheap burst), which is the ASG’s mixed_instances_policy, not the template. Show the template form so you know it exists:
# In the Launch Template — makes ALL instances spot. Prefer the ASG mix for web tiers.
instance_market_options {
market_type = "spot"
spot_options {
max_price = "0.005" # cap; omit to pay up to on-demand
spot_instance_type = "one-time" # ASGs use one-time, not persistent
}
}
The Auto Scaling Group
The ASG is the control loop. You tell it a minimum, a maximum and a desired count; it keeps desired_capacity healthy instances running, never fewer than min_size, never more than max_size, and lets scaling policies move desired between those rails. It spreads instances across the subnets you give it, replaces unhealthy ones, and — the point of this lesson — registers each instance into a load balancer’s target group so traffic only reaches healthy members.
Here is the ASG that consumes the Launch Template above, spread across two AZs and wired to an ALB target group:
resource "aws_autoscaling_group" "web" {
name_prefix = "kv-web-asg-" # name_prefix + create_before_destroy
min_size = var.min_size # 2
max_size = var.max_size # 6
desired_capacity = var.desired_capacity # 2
vpc_zone_identifier = local.subnet_ids # 2+ subnets in different AZs
health_check_type = "ELB" # trust the LB's health check, not just EC2
health_check_grace_period = 300 # seconds to boot before health counts
target_group_arns = [aws_lb_target_group.web.arn] # self-register every instance
launch_template {
id = aws_launch_template.web.id
version = "$Latest"
}
instance_refresh {
strategy = "Rolling"
preferences {
min_healthy_percentage = 90
instance_warmup = 120
}
triggers = ["tag"] # also refresh when a tag changes
}
wait_for_capacity_timeout = "10m" # how long apply waits for healthy capacity
tag {
key = "Name"
value = "kv-web"
propagate_at_launch = true
}
lifecycle {
create_before_destroy = true
}
}
The aws_autoscaling_group arguments that carry the weight:
| Argument | Purpose | Notes / gotcha |
|---|---|---|
min_size / max_size |
Hard rails on capacity | Policies move desired only between these |
desired_capacity |
Target count now | Omit to let policies own it and avoid plan churn |
vpc_zone_identifier |
Subnets (⇒ AZs) to launch in | List 2+ subnets in different AZs for HA |
launch_template |
{ id, version } |
Use $Latest with instance_refresh |
target_group_arns |
ALB/NLB target groups | Self-registers instances — no manual attach |
health_check_type |
EC2 or ELB |
ELB catches app failures; EC2 only hardware |
health_check_grace_period |
Boot grace (seconds) | Must exceed boot-to-healthy or new instances get killed |
instance_refresh |
Rolling replace on change | The safe way to ship a new AMI |
mixed_instances_policy |
Spot + on-demand + types | Replaces the top-level launch_template block |
wait_for_capacity_timeout |
Apply wait for healthy | "0" disables the wait; default "10m" |
default_cooldown |
Seconds between simple-scaling actions | Ignored by target-tracking |
termination_policies |
Which instance to kill on scale-in | OldestLaunchTemplate, Default, … |
suspended_processes |
Pause Launch, Terminate, AZRebalance |
For controlled maintenance |
enabled_metrics |
Group metrics to CloudWatch | e.g. GroupInServiceInstances |
health_check_type is the single most consequential toggle and the cause of the classic “booted but broken” incident. With EC2, the ASG only replaces an instance the hypervisor reports as failed — a crash-looping app on a perfectly healthy VM stays in rotation, serving errors. With ELB, the ASG also honours the load balancer’s health check, so an instance whose app returns non-200 is marked unhealthy and replaced. Almost every web tier wants ELB — paired with a health_check_grace_period long enough for the app to boot, or the ASG kills new instances before they finish starting.
| Aspect | health_check_type = "EC2" |
health_check_type = "ELB" |
|---|---|---|
| Detects hardware/hypervisor failure | Yes | Yes |
| Detects app-level failure (5xx, crash) | No | Yes |
| Requires a target group | No | Yes (target_group_arns) |
| Grace period matters | Somewhat | Critically (kills new instances if too short) |
| Right for a web tier | Rarely | Almost always |
Spreading across AZs is vpc_zone_identifier — a list of subnet IDs. Give it subnets in two (or three) different Availability Zones and the ASG balances instances across them and rebalances after a zone recovers. Give it one subnet and you have a single point of failure that scales. This is badge (2) in the diagram: two subnets, two AZs, no exceptions.
mixed_instances_policy is how a web tier gets cheap burst without betting the floor on spot. It combines the Launch Template with a list of instance-type overrides and a distribution that says “keep N on-demand as a base, make the rest mostly spot.” Note you do not also set the top-level launch_template block when you use this — the mixed policy replaces it:
resource "aws_autoscaling_group" "web" {
name_prefix = "kv-web-asg-"
min_size = 2
max_size = 10
desired_capacity = 4
vpc_zone_identifier = local.subnet_ids
target_group_arns = [aws_lb_target_group.web.arn]
health_check_type = "ELB"
mixed_instances_policy {
launch_template {
launch_template_specification {
launch_template_id = aws_launch_template.web.id
version = "$Latest"
}
override { instance_type = "t3.micro" }
override { instance_type = "t3a.micro" } # AMD — different capacity pool
override { instance_type = "t2.micro" } # older gen — more spot capacity
}
instances_distribution {
on_demand_base_capacity = 1 # always ≥1 on-demand
on_demand_percentage_above_base_capacity = 25 # 25% on-demand above the base
spot_allocation_strategy = "price-capacity-optimized" # best practice
}
}
lifecycle { create_before_destroy = true }
}
instances_distribution argument |
Meaning | Sane value |
|---|---|---|
on_demand_base_capacity |
On-demand instances guaranteed first | 1–2 (the stable floor) |
on_demand_percentage_above_base_capacity |
% on-demand above the base | 20–30 for cost; 100 = no spot |
spot_allocation_strategy |
How spot pools are chosen | price-capacity-optimized (fewest interruptions) |
spot_instance_pools |
Pools to spread across (lowest-price only) | 2–4; ignored by capacity-optimized |
spot_max_price |
Cap per spot instance | Omit to pay up to on-demand |
⚠️ The create_before_destroy + name_prefix pattern. This is badge (6) and the gotcha that catches everyone. Some changes to a Launch Template or ASG force Terraform to replace the ASG rather than update it in place. If your ASG has a fixed name, Terraform tries to create the new ASG with the same name before destroying the old one, and AWS rejects the duplicate — the apply fails, and you can be left mid-replace. The fix is a two-part pattern: use name_prefix (so AWS generates a unique suffix and two ASGs can coexist for a moment) and lifecycle { create_before_destroy = true } (so Terraform stands the new group up, waits for healthy capacity, then destroys the old one). Apply the same pattern to the Launch Template. Without it, a template change that forces replacement is a downtime window; with it, it is seamless.
| Pattern element | Without it | With it |
|---|---|---|
name (fixed) vs name_prefix |
Replace fails: “already exists” | Unique suffix lets old + new coexist |
| Default (destroy-before-create) | Old ASG gone before new is healthy → downtime | New ASG healthy first, then old destroyed |
| Combined | Fragile, error-prone replaces | Zero-downtime template/ASG swaps |
Scaling policies: target-tracking, step, scheduled, predictive
An ASG with no scaling policy is just a fixed fleet that self-heals. Scaling policies are what make it elastic — they move desired_capacity in response to load. AWS gives you four kinds, and choosing correctly is most of the skill:
| Policy type | Terraform policy_type |
How it decides | Best for |
|---|---|---|---|
| Target tracking | TargetTrackingScaling |
Holds a metric at a target (like a thermostat); auto-manages alarms | The default choice — CPU, request count |
| Step scaling | StepScaling |
You define steps by alarm breach size; you own the alarm | Fine-grained, non-linear response |
| Simple scaling | SimpleScaling |
One adjustment per alarm + cooldown | Legacy; avoid — step scaling supersedes it |
| Scheduled | aws_autoscaling_schedule |
Time-based (cron); sets min/max/desired | Predictable daily/weekly patterns |
| Predictive | PredictiveScaling |
ML forecast from history; scales ahead of load | Regular cyclical load, warm-ahead |
Target tracking is the one you reach for first. You pick a metric and a target value, and AWS keeps the metric at the target by adding or removing instances — creating and managing the CloudWatch alarms for you. It is the thermostat model: “hold average CPU at 50%,” and the policy figures out the rest. This is badge (4):
resource "aws_autoscaling_policy" "cpu" {
name = "kv-web-cpu-tt"
autoscaling_group_name = aws_autoscaling_group.web.name
policy_type = "TargetTrackingScaling"
target_tracking_configuration {
predefined_metric_specification {
predefined_metric_type = "ASGAverageCPUUtilization"
}
target_value = 50.0
disable_scale_in = false # true = only scale out; scale-in handled elsewhere
}
}
The four predefined target-tracking metrics cover most needs; for anything else you supply a customized_metric_specification:
predefined_metric_type |
Tracks | Needs resource_label? |
Use for |
|---|---|---|---|
ASGAverageCPUUtilization |
Mean CPU across the group | No | CPU-bound apps |
ASGAverageNetworkIn |
Bytes in per instance | No | Network-bound ingest |
ASGAverageNetworkOut |
Bytes out per instance | No | Network-bound egress |
ALBRequestCountPerTarget |
Requests per target via the ALB | Yes (ALB+TG ARN suffixes) | Web tiers — scales on real traffic |
ALBRequestCountPerTarget is often the better web-tier signal than CPU, because it scales on the thing you actually care about — requests — and needs a resource_label pointing at the ALB and target-group ARN suffixes:
resource "aws_autoscaling_policy" "reqcount" {
name = "kv-web-req-tt"
autoscaling_group_name = aws_autoscaling_group.web.name
policy_type = "TargetTrackingScaling"
target_tracking_configuration {
predefined_metric_specification {
predefined_metric_type = "ALBRequestCountPerTarget"
resource_label = "${aws_lb.web.arn_suffix}/${aws_lb_target_group.web.arn_suffix}"
}
target_value = 1000 # requests per target per minute
}
}
Step scaling hands you the alarm and the steps. You define a CloudWatch alarm and a policy whose adjustment grows with how far the metric breaches the threshold — add 1 instance for a small breach, 2 for a bigger one. It is more work than target tracking but gives you non-linear, tuned control:
resource "aws_autoscaling_policy" "step_up" {
name = "kv-web-step-up"
autoscaling_group_name = aws_autoscaling_group.web.name
policy_type = "StepScaling"
adjustment_type = "ChangeInCapacity"
metric_aggregation_type = "Average"
step_adjustment {
scaling_adjustment = 1 # +1 instance
metric_interval_lower_bound = 0 # from threshold to +20 over
metric_interval_upper_bound = 20
}
step_adjustment {
scaling_adjustment = 2 # +2 instances
metric_interval_lower_bound = 20 # 20+ over the threshold
}
}
resource "aws_cloudwatch_metric_alarm" "cpu_high" {
alarm_name = "kv-web-cpu-high"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = 2
metric_name = "CPUUtilization"
namespace = "AWS/EC2"
period = 60
statistic = "Average"
threshold = 70
dimensions = { AutoScalingGroupName = aws_autoscaling_group.web.name }
alarm_actions = [aws_autoscaling_policy.step_up.arn]
}
The adjustment_type decides how scaling_adjustment is interpreted — a frequent source of confusion:
adjustment_type |
scaling_adjustment means |
Example |
|---|---|---|
ChangeInCapacity |
Add/remove this many instances | +2 → desired + 2 |
ExactCapacity |
Set desired to exactly this | 4 → desired = 4 |
PercentChangeInCapacity |
Change by this percent of current | +50 on 4 → 6 |
Scheduled scaling ignores metrics entirely and changes capacity on a cron schedule — perfect for known patterns like “scale up at 8am on weekdays, down at 8pm.” It sets min/max/desired at the scheduled time:
resource "aws_autoscaling_schedule" "business_up" {
scheduled_action_name = "scale-up-mornings"
autoscaling_group_name = aws_autoscaling_group.web.name
min_size = 4
max_size = 10
desired_capacity = 4
recurrence = "0 8 * * MON-FRI" # cron, in time_zone
time_zone = "Asia/Kolkata"
}
resource "aws_autoscaling_schedule" "business_down" {
scheduled_action_name = "scale-down-evenings"
autoscaling_group_name = aws_autoscaling_group.web.name
min_size = 2
max_size = 10
desired_capacity = 2
recurrence = "0 20 * * MON-FRI"
time_zone = "Asia/Kolkata"
}
Scheduled and dynamic (target-tracking/step) policies compose: the schedule sets the floor for the day, and target tracking handles the within-day variation above it. Set desired_capacity in a schedule but be aware it fights a desired_capacity you also hard-code on the ASG — a reason many teams omit desired_capacity from the ASG resource and let policies own it.
Predictive scaling uses machine-learning forecasts from up to 14 days of history to scale ahead of predictable load — it provisions capacity before the daily spike rather than reacting to it. It is a separate policy type, and ForecastAndScale mode actually acts on the forecast (ForecastOnly just reports):
resource "aws_autoscaling_policy" "predictive" {
name = "kv-web-predictive"
autoscaling_group_name = aws_autoscaling_group.web.name
policy_type = "PredictiveScaling"
predictive_scaling_configuration {
metric_specification {
target_value = 50
predefined_metric_pair_specification {
predefined_metric_type = "ASGCPUUtilization"
}
}
mode = "ForecastAndScale" # or "ForecastOnly" to observe first
scheduling_buffer_time = 300 # provision 5 min ahead
max_capacity_breach_behavior = "IncreaseMaxCapacity"
max_capacity_buffer = 10 # allow 10% over max_size
}
}
The rule of thumb across all four: start with target tracking; add scheduled for known patterns; reach for step when you need tuned, non-linear response; and layer predictive on top when your load is cyclical and reacting is too late.
Lifecycle hooks & warm pools
Two features handle the edges of an instance’s life — the moment it launches and the moment it terminates. Lifecycle hooks pause an instance in a Pending:Wait (launching) or Terminating:Wait (terminating) state so something can act before it enters or leaves service: run a warm-up, register with an external system, or — most commonly — drain connections before termination so scale-in and spot reclaims don’t drop in-flight requests. The instance stays paused until you call complete-lifecycle-action (or the heartbeat_timeout expires):
resource "aws_autoscaling_lifecycle_hook" "drain" {
name = "drain-on-terminate"
autoscaling_group_name = aws_autoscaling_group.web.name
lifecycle_transition = "autoscaling:EC2_INSTANCE_TERMINATING"
default_result = "CONTINUE" # proceed if the heartbeat times out
heartbeat_timeout = 300 # seconds to drain — must exceed TG dereg delay
}
| Lifecycle hook setting | Purpose | Gotcha |
|---|---|---|
lifecycle_transition |
...EC2_INSTANCE_LAUNCHING or ...TERMINATING |
Two hooks for both ends |
default_result |
CONTINUE or ABANDON on timeout |
ABANDON kills a stuck launch |
heartbeat_timeout |
Seconds the instance waits | Must exceed the target-group deregistration delay or you drop connections |
notification_target_arn |
SNS/SQS to notify | Where your automation listens |
The critical rule: a terminate hook’s heartbeat_timeout must be longer than the target group’s deregistration_delay (default 300s), or the instance is torn down before connections finish draining — the source of intermittent 5xx on every scale-in and spot reclaim.
Warm pools attack scale-out latency. If your instances take minutes to boot (a fat AMI, a slow app start), a traffic spike outruns cold launches. A warm pool keeps pre-initialised instances in a Stopped (or Running/Hibernated) state, so scaling out is a fast start rather than a slow launch:
resource "aws_autoscaling_group" "web" {
# ... min_size, max_size, launch_template, etc. ...
warm_pool {
pool_state = "Stopped" # cheapest: no compute charge while stopped
min_size = 2 # always keep 2 pre-warmed
max_group_prepared_capacity = 4
instance_reuse_policy {
reuse_on_scale_in = true # return scaled-in instances to the pool
}
}
}
warm_pool setting |
Purpose | Note |
|---|---|---|
pool_state |
Stopped / Running / Hibernated |
Stopped = cheapest (no compute charge) |
min_size |
Warm instances always held | Size to your surge rate |
max_group_prepared_capacity |
Cap on warm + in-service prepared | Bounds the pre-warm cost |
instance_reuse_policy |
Reuse scaled-in instances | reuse_on_scale_in = true recycles them |
instance_refresh: rolling deploys
When you bump the AMI in the Launch Template, existing instances don’t change — the new version only applies to future launches. instance_refresh is how you roll the fleet onto the new template safely: it terminates and replaces instances in batches, honouring a minimum healthy percentage so capacity never dips below your floor, and waiting an instance_warmup for each new instance to become healthy before moving on. This is badge (5). It is declared as a block on the ASG and triggered by changes to the launch template (and any extra fields you list in triggers):
instance_refresh {
strategy = "Rolling"
preferences {
min_healthy_percentage = 90 # never drop below 90% of desired during rollout
instance_warmup = 120 # wait 120s for each new instance to warm
checkpoint_percentages = [50, 100] # pause at 50% for a canary check
checkpoint_delay = 600 # hold 10 min at each checkpoint
auto_rollback = true # roll back if the refresh fails
}
triggers = ["tag"] # also refresh when a tag changes, not just the LT
}
instance_refresh preference |
Purpose | Trade-off |
|---|---|---|
min_healthy_percentage |
Floor of healthy instances during rollout | 100 = zero dip but needs surge headroom; 90 replaces faster |
max_healthy_percentage |
Ceiling (surge above desired) | Set 110+ to replace before terminating (no dip) |
instance_warmup |
Seconds before a new instance counts healthy | Too low = rolls onto not-yet-ready instances |
checkpoint_percentages |
Pause points (canary gates) | [20, 100] = 20% canary, then the rest |
checkpoint_delay |
Seconds to hold at each checkpoint | Your window to observe before proceeding |
auto_rollback |
Revert on failure | Needs a prior known-good version |
strategy |
Currently Rolling |
The only strategy today |
The mental model: min_healthy_percentage below 100 means the ASG can terminate before launching (capacity dips but no surge cost); max_healthy_percentage above 100 means it launches before terminating (surge cost but no dip). For a fleet that must never lose capacity mid-deploy, set max_healthy_percentage = 110 and let it surge; for cost, accept a small dip at min_healthy_percentage = 90. A warm pool makes either far faster.
Hands-on: build it with Terraform
⚠️ This provisions real, billable AWS resources — an Application Load Balancer and two EC2 instances. They are modest (well under ₹100 for an hour), but follow it end to end and run the destroy step. Do not leave it up.
We now assemble everything above into one working project: a Launch Template, an ASG across two AZs registered into an ALB target group, a target-tracking policy at 50% CPU, and an instance refresh. To keep the network out of scope we use the account’s default VPC via data sources; in production you would point subnet_ids at your own private subnets. Lay out the files:
mkdir -p asg-demo && cd asg-demo
touch versions.tf provider.tf variables.tf data.tf \
security.tf iam.tf alb.tf compute.tf scaling.tf outputs.tf
1. Pin Terraform and the provider (versions.tf). Pin aws with ~> so a plan in CI never silently changes behaviour, and use a remote backend — for AWS that is an S3 bucket for state plus a DynamoDB table for the lock:
# versions.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.60"
}
}
# Remote state (AWS = S3 + DynamoDB lock). Create these once, out of band.
backend "s3" {
bucket = "kv-tfstate-2026"
key = "asg-demo/terraform.tfstate"
region = "ap-south-1"
dynamodb_table = "kv-tf-locks"
encrypt = true
}
}
2. Configure the provider (provider.tf). Authenticate ahead of time with aws configure, aws sso login, an assumed role, or OIDC in CI — never put keys in HCL. Default tags applied here land on every taggable resource:
# provider.tf
provider "aws" {
region = var.region
default_tags {
tags = {
project = "tf-course"
lesson = "aws-asg"
owner = "vinod"
}
}
}
3. Variables (variables.tf). Parameterise region, sizes and capacity so the same code stamps any environment:
# variables.tf
variable "region" {
type = string
default = "ap-south-1"
}
variable "name" {
type = string
default = "kv-web"
}
variable "instance_type" {
type = string
default = "t3.micro"
}
variable "min_size" {
type = number
default = 2
}
variable "max_size" {
type = number
default = 6
}
variable "desired_capacity" {
type = number
default = 2
}
4. Data sources: default VPC, two AZs’ subnets, and the AMI (data.tf). We look up the latest Amazon Linux 2023 AMI (never hard-code an AMI ID — they are region-specific and rotate), the default VPC, and slice two of its subnets so the ASG spans exactly two AZs:
# data.tf
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-2023.*-x86_64"]
}
filter {
name = "state"
values = ["available"]
}
}
data "aws_vpc" "default" {
default = true
}
data "aws_subnets" "default" {
filter {
name = "vpc-id"
values = [data.aws_vpc.default.id]
}
}
locals {
vpc_id = data.aws_vpc.default.id
# Default VPC has one subnet per AZ; take two → two AZs.
subnet_ids = slice(tolist(data.aws_subnets.default.ids), 0, 2)
user_data = <<-EOF
#!/bin/bash
dnf install -y httpd stress-ng
echo "<h1>KloudVin ASG — $(hostname -f)</h1>" > /var/www/html/index.html
systemctl enable --now httpd
EOF
}
5. Security groups (security.tf). The ALB accepts 80 from the internet; the instances accept 80 only from the ALB’s security group — the tightest correct pattern:
# security.tf
resource "aws_security_group" "alb" {
name_prefix = "${var.name}-alb-"
vpc_id = local.vpc_id
ingress {
description = "HTTP from anywhere"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
lifecycle { create_before_destroy = true }
}
resource "aws_security_group" "instance" {
name_prefix = "${var.name}-inst-"
vpc_id = local.vpc_id
ingress {
description = "HTTP from the ALB only"
from_port = 80
to_port = 80
protocol = "tcp"
security_groups = [aws_security_group.alb.id] # SG reference, not a CIDR
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
lifecycle { create_before_destroy = true }
}
6. IAM instance profile for SSM (iam.tf). Attaching AmazonSSMManagedInstanceCore lets us connect with SSM Session Manager — no SSH key, no port 22 open — which is how we will drive the load test:
# iam.tf
resource "aws_iam_role" "web" {
name_prefix = "${var.name}-role-"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "ec2.amazonaws.com" }
}]
})
}
resource "aws_iam_role_policy_attachment" "ssm" {
role = aws_iam_role.web.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
resource "aws_iam_instance_profile" "web" {
name_prefix = "${var.name}-"
role = aws_iam_role.web.name
}
7. The ALB and target group (alb.tf). A minimal public ALB with an HTTP listener forwarding to a target group the ASG will register into. The target group’s health check is what makes health_check_type = "ELB" meaningful:
# alb.tf
resource "aws_lb" "web" {
name = "${var.name}-alb"
load_balancer_type = "application"
security_groups = [aws_security_group.alb.id]
subnets = local.subnet_ids # 2 AZs — ALB requires ≥2
}
resource "aws_lb_target_group" "web" {
name = "${var.name}-tg"
port = 80
protocol = "HTTP"
vpc_id = local.vpc_id
health_check {
path = "/"
protocol = "HTTP"
matcher = "200"
healthy_threshold = 2
unhealthy_threshold = 3
interval = 15
timeout = 5
}
}
resource "aws_lb_listener" "http" {
load_balancer_arn = aws_lb.web.arn
port = 80
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.web.arn
}
}
8. The Launch Template and ASG (compute.tf). The heart of the lesson — the versioned template and the group that spreads it across two AZs and registers it into the target group:
# compute.tf
resource "aws_launch_template" "web" {
name_prefix = "${var.name}-"
image_id = data.aws_ami.al2023.id
instance_type = var.instance_type
user_data = base64encode(local.user_data)
vpc_security_group_ids = [aws_security_group.instance.id]
iam_instance_profile {
arn = aws_iam_instance_profile.web.arn
}
block_device_mappings {
device_name = "/dev/xvda"
ebs {
volume_size = 8
volume_type = "gp3"
encrypted = true
delete_on_termination = true
}
}
metadata_options {
http_tokens = "required" # IMDSv2 only
http_endpoint = "enabled"
}
monitoring { enabled = true }
tag_specifications {
resource_type = "instance"
tags = { Name = var.name }
}
lifecycle { create_before_destroy = true }
}
resource "aws_autoscaling_group" "web" {
name_prefix = "${var.name}-asg-"
min_size = var.min_size
max_size = var.max_size
desired_capacity = var.desired_capacity
vpc_zone_identifier = local.subnet_ids # two AZs
health_check_type = "ELB"
health_check_grace_period = 120
target_group_arns = [aws_lb_target_group.web.arn] # self-register
launch_template {
id = aws_launch_template.web.id
version = "$Latest"
}
instance_refresh {
strategy = "Rolling"
preferences {
min_healthy_percentage = 90
instance_warmup = 120
}
triggers = ["tag"]
}
wait_for_capacity_timeout = "10m"
tag {
key = "Name"
value = var.name
propagate_at_launch = true
}
lifecycle { create_before_destroy = true }
}
9. The scaling policy (scaling.tf). Target tracking at 50% average CPU — it creates and manages its own CloudWatch alarms:
# scaling.tf
resource "aws_autoscaling_policy" "cpu" {
name = "${var.name}-cpu-tt"
autoscaling_group_name = aws_autoscaling_group.web.name
policy_type = "TargetTrackingScaling"
target_tracking_configuration {
predefined_metric_specification {
predefined_metric_type = "ASGAverageCPUUtilization"
}
target_value = 50.0
}
}
10. Outputs (outputs.tf). Emit the ALB DNS name to curl and the ASG name to inspect:
# outputs.tf
output "alb_dns_name" {
value = aws_lb.web.dns_name
}
output "asg_name" {
value = aws_autoscaling_group.web.name
}
11. Init. Downloads the provider and wires the backend:
terraform init
# Initializing the backend...
# Initializing provider plugins...
# - Installing hashicorp/aws v5.60.x ...
# Terraform has been successfully initialized!
12. Plan. Read the summary line — it must create the whole graph and change nothing unexpected:
terraform plan
# ...
# Plan: 12 to add, 0 to change, 0 to destroy.
# Changes to Outputs:
# + alb_dns_name = (known after apply)
# + asg_name = (known after apply)
13. Apply. ⚠️ Billing starts here. The ALB and instances come up in a couple of minutes; the apply waits (wait_for_capacity_timeout) for the ASG to report healthy capacity in the target group:
terraform apply -auto-approve
# aws_launch_template.web: Creation complete after 2s
# aws_lb.web: Still creating... [1m0s elapsed]
# aws_autoscaling_group.web: Still creating... [2m0s elapsed] # waiting for ELB health
# aws_autoscaling_group.web: Creation complete after 2m30s
# Apply complete! Resources: 12 added, 0 changed, 0 destroyed.
# Outputs:
# alb_dns_name = "kv-web-alb-123456789.ap-south-1.elb.amazonaws.com"
# asg_name = "kv-web-asg-20260709xxxxxx"
14. Verify — curl the ALB, then read the group and target health. Repeat the curl and watch the hostname alternate between the two instances as the ALB spreads requests:
ALB=$(terraform output -raw alb_dns_name)
ASG=$(terraform output -raw asg_name)
curl http://$ALB/
# <h1>KloudVin ASG — ip-172-31-x-x.ap-south-1.compute.internal</h1>
curl http://$ALB/
# <h1>KloudVin ASG — ip-172-31-y-y.ap-south-1.compute.internal</h1> # load balanced!
# The ASG's own view: capacity, AZs, and each instance's health.
aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names "$ASG" \
--query 'AutoScalingGroups[0].{Min:MinSize,Max:MaxSize,Desired:DesiredCapacity,
AZs:AvailabilityZones,Instances:Instances[].{Id:InstanceId,AZ:AvailabilityZone,
Health:HealthStatus,State:LifecycleState}}'
# "Desired": 2, "AZs": ["ap-south-1a","ap-south-1b"], each instance Healthy / InService
# The target group's view: are both instances registered and healthy?
TG=$(aws elbv2 describe-target-groups --names kv-web-tg \
--query 'TargetGroups[0].TargetGroupArn' --output text)
aws elbv2 describe-target-health --target-group-arn "$TG" \
--query 'TargetHealthDescriptions[].TargetHealth.State'
# [ "healthy", "healthy" ]
Two healthy targets across two AZs is the whole build working: the ASG launched instances in ap-south-1a and ap-south-1b, each registered itself into the target group via target_group_arns, and each passes the ALB health check. The verification checklist:
| Step | Command | Expect |
|---|---|---|
| ALB resolves | terraform output -raw alb_dns_name |
An *.elb.amazonaws.com name |
| Endpoint serves | curl http://$ALB/ |
The “KloudVin ASG” page |
| Load balancing works | repeat curl | Hostname alternates between instances |
| Two AZs | describe-auto-scaling-groups … AvailabilityZones |
Two distinct AZs |
| Targets registered + healthy | describe-target-health |
healthy for each instance |
15. Load test — watch it scale out. ⚠️ Drive CPU on the fleet so target tracking reacts. Connect to an instance with SSM Session Manager (no SSH key needed thanks to the instance profile) and run stress-ng, then watch desired_capacity climb:
# Grab one instance id from the group and open an SSM shell:
IID=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names "$ASG" \
--query 'AutoScalingGroups[0].Instances[0].InstanceId' --output text)
aws ssm start-session --target "$IID"
# Inside the instance — peg all CPUs for 10 minutes:
sudo stress-ng --cpu 0 --timeout 600s &
exit
# Back on your machine, watch the group grow (CPU > 50% → target tracking scales out):
watch -n 30 "aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names $ASG \
--query 'AutoScalingGroups[0].DesiredCapacity'"
# 2 ... 2 ... 3 ... 4 (climbs toward max_size as CPU stays high)
# The audit trail of every scaling decision:
aws autoscaling describe-scaling-activities --auto-scaling-group-name "$ASG" \
--query 'Activities[].{Time:StartTime,Desc:Description,Cause:Cause}' --max-items 5
# "Launching a new EC2 instance ... at 50.0 CPUUtilization ... breaching the alarm threshold"
Within a few minutes the target-tracking policy notices average CPU above 50%, its auto-created CloudWatch alarm fires, and the ASG raises desired_capacity and launches instances (up to max_size = 6). When stress-ng ends and CPU falls, the policy scales back in — more slowly and gently, by design, to avoid flapping. If you had used the ALBRequestCountPerTarget metric instead, a load generator (hey, ab, wrk) against the ALB URL would trigger the same scale-out on request volume rather than CPU.
16. Destroy. ⚠️ Do this — the ALB and instances bill by the hour.
terraform destroy -auto-approve
# aws_autoscaling_group.web: Destroying... (terminates instances first)
# aws_lb.web: Destruction complete
# Destroy complete! Resources: 12 destroyed.
Confirm the group is gone (aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG returns an empty list) and no stray instances linger (aws ec2 describe-instances --filters "Name=tag:project,Values=tf-course" "Name=instance-state-name,Values=running"). Terraform terminates the ASG’s instances as part of destroying the group — you do not clean them up separately.
Variables, outputs & making it reusable
The demo hard-codes one fleet. Real platforms run this shape many times — a web tier, an API tier, a worker tier — each an ASG behind a target group with its own sizing. Two patterns turn the demo into something reusable: wrap it in a module and drive many fleets with for_each, or reach for the mature community module.
Wrapped as a module, the inputs worth exposing are the ones that vary per environment and per tier:
| Variable | Type | Why expose it |
|---|---|---|
name |
string | Distinguish web/api/worker fleets |
instance_type |
string | Size per tier/env (t3.micro in dev, m6i in prod) |
min_size / max_size |
number | Capacity rails per env |
desired_capacity |
number (optional) | Often omitted so policies own it |
subnet_ids |
list(string) | Bring-your-own private subnets |
target_group_arns |
list(string) | Register into an existing ALB |
cpu_target |
number | Target-tracking value per tier |
ami_id |
string | Pin or float the AMI per env |
on_demand_base |
number | Spot/on-demand mix per env (0 in dev) |
Then a for_each over a map of tiers stamps them all:
variable "tiers" {
type = map(object({
instance_type = string
min_size = number
max_size = number
cpu_target = number
}))
default = {
web = { instance_type = "t3.micro", min_size = 2, max_size = 8, cpu_target = 50 }
worker = { instance_type = "t3.medium", min_size = 1, max_size = 4, cpu_target = 70 }
}
}
module "fleet" {
source = "./modules/asg"
for_each = var.tiers
name = "kv-${each.key}"
instance_type = each.value.instance_type
min_size = each.value.min_size
max_size = each.value.max_size
cpu_target = each.value.cpu_target
subnet_ids = local.private_subnet_ids
}
Before writing your own, know the registry standard. terraform-aws-modules/autoscaling/aws is the mature, widely-used community module — it builds the Launch Template and ASG together, handles the create_before_destroy and name_prefix wiring, and exposes scaling policies and instance refresh as inputs:
| Need | Registry module | Roll your own when |
|---|---|---|
| ASG + Launch Template | terraform-aws-modules/autoscaling/aws |
Highly bespoke lifecycle/mixed-instances logic |
| The ALB in front | terraform-aws-modules/alb/aws |
Unusual listener/routing topology |
| The VPC + subnets | terraform-aws-modules/vpc/aws |
— |
Pin the module version (version = "~> 8.0") — a floating module version is as dangerous as a floating provider. Use the registry module when your fleet is conventional and you value the maintenance; roll your own when your mixed-instances, warm-pool or lifecycle-hook logic is genuinely bespoke.
Common mistakes and troubleshooting
Auto Scaling failures cluster into a handful of signatures, and each has a precise confirmation command. This is the symptom → cause → fix table to keep open during an incident:
| Symptom | Likely cause | Fix |
|---|---|---|
Instances Running but 0 registered in the target group |
ASG missing target_group_arns, or you attached targets manually |
Set target_group_arns on the ASG; delete any aws_lb_target_group_attachment |
Instances register then go unhealthy and get replaced |
Target-group health check path 404s / wrong port, or grace too short | Point health_check.path at a real 200; raise health_check_grace_period above boot time |
| Health-check flapping (constant launch/terminate) | health_check_type = "ELB" + grace shorter than boot-to-healthy |
Increase health_check_grace_period; make the app healthy sooner |
| App crash-loops but instances stay in rotation | health_check_type = "EC2" — only hardware is checked |
Switch to health_check_type = "ELB" |
| Scaling policy never triggers | Not enough metric data, wrong metric, or already at max_size |
Enable detailed monitoring; verify the metric; check max_size headroom |
| Scales out but never scales in | disable_scale_in = true, or scale-in is just slow (by design) |
Remove disable_scale_in; target-tracking scale-in is deliberately gradual |
apply fails: ASG “already exists” on replace |
Fixed name + destroy-before-create |
Use name_prefix + lifecycle { create_before_destroy = true } |
| New AMI in the template but instances don’t update | $Latest only affects new launches; no refresh triggered |
Add instance_refresh, or bump a triggers field / new template version |
| Spot instances repeatedly interrupted, capacity drops | lowest-price strategy, few pools, no rebalancing |
Use price-capacity-optimized + more instance-type overrides |
| Connections dropped on scale-in / spot reclaim | No drain; terminate hook shorter than dereg delay | Set a terminate lifecycle hook heartbeat_timeout > target-group deregistration_delay |
instance_refresh dips capacity mid-rollout |
min_healthy_percentage too low, or no surge headroom |
Raise min_healthy_percentage, or set max_healthy_percentage = 110 to surge |
Instances launch but user_data never ran |
Not base64-encoded, or a script error | user_data = base64encode(...); read /var/log/cloud-init-output.log |
Because the “unhealthy target” family is the most common, here is the decision matrix that maps what describe-target-health reports to the specific misconfiguration — walk it top to bottom:
| Target health state | Meaning | Where the bug is |
|---|---|---|
healthy |
Passing the ALB health check | Working as intended |
unhealthy |
Registered, but the health check fails | Health-check path/port/matcher, or the app returns non-200 |
initial (stuck) |
Registered, still in the grace window | Wait; if it never clears, boot time > grace period |
draining |
Deregistering (scale-in / refresh) | Normal during scale-in; watch deregistration_delay |
unused — no registered targets |
ASG not registering | Missing target_group_arns; VPC/subnet mismatch between ASG and TG |
Beyond the table, the traps that cost real time:
Instances not registering to the target group. The number-one ASG-with-ALB failure. It is almost always one of three things: the ASG has no target_group_arns (so it registers nothing); the target group and the ASG’s subnets are in different VPCs (a target group is VPC-scoped); or someone added a manual aws_lb_target_group_attachment that fights the ASG’s own registration. The ASG owns registration end to end — set target_group_arns and never attach by hand.
Health-check flapping. With health_check_type = "ELB", the ASG replaces any instance the target group calls unhealthy. If your health_check_grace_period is shorter than your boot-to-healthy time, the ASG kills each new instance before its app finishes starting, launches a replacement, and repeats forever — a launch/terminate storm that never converges and burns money. Measure your real boot-to-first-200 and set the grace period comfortably above it (a fat app image with a slow start can need 300s+). The complementary bug is health_check_type = "EC2", where a crash-looping app stays in rotation because the hypervisor is fine — the “booted but broken” instance serving 5xx.
Scaling not triggering. Three usual causes. First, no metric data — an ASG without detailed monitoring emits CPU every 5 minutes, so target tracking reacts slowly; enable monitoring { enabled = true } on the template for 1-minute data. Second, the wrong metric — CPU tracking does nothing for an I/O-bound app; use the metric that actually moves under load (often ALBRequestCountPerTarget). Third, no headroom — if desired_capacity already equals max_size, there is nowhere to scale; check describe-scaling-activities for “reached max capacity.”
Launch Template version pinning. version = "$Latest" means new launches use the newest version — it does not touch running instances. Teams edit the template, see a new version, and are baffled that the fleet is unchanged. Either drive the rollout with instance_refresh (the right way) or, for an emergency rollback, pin version to the last-good number and re-apply so new launches use it (existing instances still need a refresh or manual cycle). And beware pinning to $Default while another process bumps default_version out from under you — that is a surprise deploy on the next launch.
The create_before_destroy chain. create_before_destroy is contagious: if an ASG that has it references a Launch Template that does not, Terraform can deadlock trying to satisfy the ordering. The rule is to put lifecycle { create_before_destroy = true } on the Launch Template, the ASG, and the security groups they depend on — the whole replacement chain must agree. And always pair it with name_prefix (never a fixed name) so two copies can briefly coexist during the swap.
Cost, cleanup & production notes
The ASG and Launch Template are free — you pay only for the instances they run and the ALB in front. Indicative ap-south-1, on-demand, July 2026:
| Resource | Rough monthly if left up | Notes |
|---|---|---|
| Application Load Balancer | ~₹1,600 (~$19) + LCU | Fixed hourly + Load Balancer Capacity Units |
2× t3.micro EC2 |
~₹1,700 (~$20) | The demo fleet; t3.micro is inexpensive |
| EBS (2× 8 GiB gp3) | ~₹130 (~$1.60) | Tiny; deleted on termination |
| Data transfer | Usage-based | Egress to the internet is the variable line |
| This demo, one hour | < ₹100 (~$1) | Which is why you destroy it |
Cleanup is terraform destroy. Two things to watch: destroying the ASG terminates its instances first (so a destroy can take a couple of minutes while instances drain and terminate), and if a lifecycle hook is holding an instance in Terminating:Wait, the destroy can stall until the hook heartbeat times out — call complete-lifecycle-action or shorten the heartbeat if you get stuck.
Production hardening, the five that matter:
- Remote, locked state. The
backend "s3"block (S3 for state + DynamoDB for the lock) shown inversions.tfis non-negotiable for a team — local state on a scaling fleet is how two engineers clobber each other’s ASG. - Own your network. The demo uses the default VPC for brevity; production ASGs launch into private subnets (
vpc_zone_identifier) with a NAT gateway or VPC endpoints for egress, and the ALB in public subnets. Never run a web fleet in the default VPC. ELBhealth checks with an honest path.health_check_type = "ELB"plus a health-check endpoint that returns 200 only when the app is truly ready (dependencies reachable) is what makes the ASG self-heal on app failures, not just hardware. Set the grace period above real boot time.- Zero-downtime deploys by construction.
create_before_destroy+name_prefixon the template and ASG, plus aninstance_refreshwith a sensiblemin_healthy_percentage(and a warm pool if boots are slow), turns every AMI change into a safe rolling deploy. Addauto_rollbackand a canarycheckpoint_percentagesfor extra safety. - Tag, monitor, and watch drift. Use
default_tagsfor cost attribution, send group metrics (GroupInServiceInstances,GroupDesiredCapacity) to CloudWatch, and runterraform planon a schedule — someone will “quickly” changedesired_capacityin the console, and drift detection catches it before it surprises you. The CloudWatch, SNS & Observability lesson wires the alarms and notifications a production ASG hangs off.
Cheat-sheet
The dense reference for this tier — resources, the arguments you reach for most, and the verification commands:
| Resource | Purpose | Must-set arguments |
|---|---|---|
aws_launch_template |
Versioned instance blueprint | image_id, instance_type, user_data (base64), name_prefix |
aws_autoscaling_group |
Instance lifecycle across AZs | min/max_size, vpc_zone_identifier, launch_template, target_group_arns |
aws_autoscaling_policy |
Dynamic scaling | policy_type, target_tracking_configuration |
aws_autoscaling_schedule |
Time-based scaling | recurrence, min/max/desired |
aws_autoscaling_lifecycle_hook |
Pause on launch/terminate | lifecycle_transition, heartbeat_timeout |
aws_lb_target_group |
Registration + health target | port, protocol, vpc_id, health_check |
aws_cloudwatch_metric_alarm |
Trigger for step scaling | metric_name, threshold, alarm_actions |
| Key argument | On | Sets |
|---|---|---|
version = "$Latest" |
ASG launch_template |
Track the newest template version |
health_check_type = "ELB" |
ASG | Replace on app failure, not just hardware |
target_group_arns |
ASG | Self-register instances into the ALB |
vpc_zone_identifier |
ASG | Subnets ⇒ AZ spread (list 2+) |
instance_refresh {} |
ASG | Rolling replace on template change |
create_before_destroy + name_prefix |
LT + ASG | Zero-downtime replaces |
metadata_options.http_tokens = "required" |
LT | Enforce IMDSv2 |
predefined_metric_type |
policy | ASGAverageCPUUtilization / ALBRequestCountPerTarget |
| Verify with | Command |
|---|---|
| Group + instance health | aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <asg> |
| Scaling decisions | aws autoscaling describe-scaling-activities --auto-scaling-group-name <asg> |
| Target registration | aws elbv2 describe-target-health --target-group-arn <tg-arn> |
| Launch Template versions | aws ec2 describe-launch-template-versions --launch-template-id <lt> |
| Refresh status | aws autoscaling describe-instance-refreshes --auto-scaling-group-name <asg> |
| Policies | aws autoscaling describe-policies --auto-scaling-group-name <asg> |
Interview and exam questions
1. Why a Launch Template over a Launch Configuration? Launch Templates are versioned, support multiple instance types and spot via the ASG’s mixed-instances policy, enforce IMDSv2, allow multiple block devices and tag-on-launch, and receive all new features. Launch Configurations are immutable, single-type, unversioned and frozen. AWS recommends Launch Templates for everything new; never write a Launch Configuration for new work.
2. How does an ASG register instances into an ALB? Set target_group_arns on the ASG to the target group’s ARN. The ASG then registers every instance it launches into that target group automatically and deregisters them on termination — you never use aws_lb_target_group_attachment. Pair it with health_check_type = "ELB" so the ASG also acts on the load balancer’s health check.
3. What’s the difference between health_check_type = "EC2" and "ELB"? EC2 only replaces instances the hypervisor reports as failed, so a crash-looping app on a healthy VM stays in rotation serving errors. ELB also honours the target group’s health check, replacing instances whose app fails. Web tiers want ELB, with a health_check_grace_period longer than boot-to-healthy time.
4. Explain the create_before_destroy + name_prefix pattern and why it exists. Some Launch Template/ASG changes force replacement. With a fixed name, Terraform’s default destroy-before-create either causes downtime or fails because the new ASG can’t share the old name. name_prefix lets AWS generate a unique name so two ASGs briefly coexist, and lifecycle { create_before_destroy = true } stands the new one up (waiting for healthy capacity) before destroying the old — a zero-downtime swap.
5. When would you use target tracking vs step scaling? Target tracking for the common case — pick a metric and a target and let AWS manage the alarms and adjustments (thermostat model). Step scaling when you need tuned, non-linear response: different adjustment sizes for different breach magnitudes, with a CloudWatch alarm you own. Start with target tracking; reach for step scaling only when target tracking’s single-target model isn’t enough.
6. You changed the AMI in the Launch Template and applied. Nothing happened to the running instances. Why? version = "$Latest" only affects new launches; it never touches running instances. To roll the fleet onto the new template you need an instance_refresh (triggered by the template change), or you manually cycle instances. This is by design so a template edit doesn’t cause an uncontrolled fleet-wide replace.
7. How do you mix spot and on-demand in one ASG? Use mixed_instances_policy: a launch_template with multiple override { instance_type } entries (more types = more spot capacity pools) and an instances_distribution with on_demand_base_capacity (a guaranteed floor), on_demand_percentage_above_base_capacity, and spot_allocation_strategy = "price-capacity-optimized" for the fewest interruptions. Don’t also set the top-level launch_template block — the mixed policy replaces it.
8. Your ASG scales out but connections drop during scale-in. Fix it? The instance is terminated before in-flight connections drain. Add a terminate lifecycle hook (autoscaling:EC2_INSTANCE_TERMINATING) whose heartbeat_timeout exceeds the target group’s deregistration_delay, so the instance drains before it dies; complete the hook when draining finishes. The same protects against spot-reclaim connection loss.
9. (Terraform Associate 003) The ASG references aws_launch_template.web.id and aws_lb_target_group.web.arn. What guarantees creation order? Terraform’s implicit dependency graph: because the ASG references attributes of the template and target group, Terraform creates both before the ASG automatically — no depends_on needed. depends_on is only for hidden dependencies with no attribute reference.
10. (Terraform Associate 003) You set desired_capacity = 2 on the ASG, but a scaling policy grew it to 4. On the next terraform plan, what happens? Terraform sees actual capacity (4) drifting from the configured desired_capacity (2) and plans to reset it to 2 — fighting your own autoscaling. The fix is to omit desired_capacity from the resource (or use lifecycle { ignore_changes = [desired_capacity] }) so scaling policies own it and Terraform stops reverting it.
11. What does instance_refresh with min_healthy_percentage = 90 do during a deploy? It replaces instances in rolling batches while keeping at least 90% of desired capacity healthy at all times — so capacity dips at most 10% during the rollout, and each new instance waits instance_warmup before counting healthy. Setting max_healthy_percentage = 110 instead makes it surge (launch before terminate) so capacity never dips.
12. Why enforce IMDSv2, and how, in a Launch Template? IMDSv1’s unauthenticated metadata endpoint is a classic SSRF-to-credentials vector — a server-side request forgery can read the instance role’s temporary credentials. metadata_options { http_tokens = "required" } forces IMDSv2’s session-token handshake, closing that path. Set it on every Launch Template.
Key takeaways
- Template first, group second. The Launch Template is the versioned blueprint (
aws_launch_template); the ASG (aws_autoscaling_group) references it and owns the fleet’s lifecycle. Always a Launch Template, never the deprecated Launch Configuration. target_group_arnsis how the ASG meets the ALB. Set it and every launched instance self-registers into the target group; never attach targets by hand. Pair withhealth_check_type = "ELB"so the group heals on app failures.- Spread across AZs, always.
vpc_zone_identifierwith 2+ subnets in different AZs is the difference between a fleet that survives a zone outage and one that scales a single point of failure. - Target tracking is the default scaling policy. Pick a metric (CPU or
ALBRequestCountPerTarget) and a target; AWS manages the alarms. Add scheduled for known patterns, step for tuned response, predictive for cyclical load. create_before_destroy+name_prefix= zero-downtime replaces. Put the lifecycle block andname_prefixon the template, the ASG, and their security groups so a forced replacement stands the new fleet up before tearing the old one down.instance_refreshturns “new AMI” into a safe rollout.$Latestonly affects new launches; the refresh rolls the running fleet onto the new version in health-gated batches — withmin_healthy_percentage,instance_warmup, and optional canary checkpoints.- Grace period and health type cause most incidents. Too short a
health_check_grace_periodwithELBhealth checks makes new instances get killed before they boot (flapping);EC2health checks leave crash-looped apps in rotation. Measure boot-to-healthy and set the grace above it. - Build it, verify it, destroy it. The ALB and instances bill by the hour —
terraform destroyis part of the exercise, and destroying the ASG terminates its instances for you.