Production Amazon ECS on Fargate: Task Networking, Auto Scaling, and Safe Rolling Deployments

In a nutshell

If you have ever run docker run on your laptop, you already know how to start a container. AWS Fargate is the version of that where you never touch the machine underneath: you describe the container you want (a task definition) and how many copies to keep alive (a service), and AWS finds the hardware, runs it, patches it, and replaces it when it dies. There is no server for you to log into, size, or reboot.

Think of it like a valet parking service for containers. You hand over the keys (your image plus a little config) and the valet parks it, watches it, and brings up a fresh one if it stalls. The catch is the paperwork you fill in at the desk: which lane the car goes in (the subnet and security group), how quickly the valet should swap it out on a busy night (auto scaling and deployments), and what to do when a car has to leave the lot without cutting off a passenger mid-ride (graceful shutdown). Get the paperwork wrong and the valet still parks the car — it just drops your passengers, runs out of spaces, or refuses new cars at the worst possible moment. This lesson is that paperwork, field by field.

Every task on Fargate is its own little computer on the network: it gets its own IP address and its own firewall, it can be told to make copies of itself when traffic rises, it can be replaced with a new version without downtime, and it can be told to stop politely so no request is cut in half. Those four ideas — networking, scaling, deploying, and stopping — are the whole lesson. Everything after them is just the exact knob that makes each one behave in production instead of only in a demo.

Level: Intermediate · Time: ~75 min

Before you start, be comfortable with: what a container image and a Dockerfile are; that ECS nests as cluster → service → task → container; VPC basics (subnets, availability zones, security groups); and reading JSON and running an aws command. If any of those is shaky, do the Amazon ECS & ECR Fundamentals and Your First Container Deployment on ECS Fargate lessons first, then come back — the Prerequisites section just below links both.

After this lesson you will be able to: size a task with a valid CPU/memory pair and pin its image to a digest; place tasks in private subnets with enough IPs to survive a deploy surge; scale on a signal that actually leads your load; ship a new version with an automatic rollback safety net; and shut a task down without dropping a single in-flight request. The dense reference tables below are built to stay open on a second screen during an incident — read the prose once, then treat the tables as a lookup.

A docker run on a laptop and an ECS service on AWS Fargate share almost no operational concerns. Fargate is the serverless launch type for Amazon Elastic Container Service — you hand AWS a task definition and a desired count, and it runs your containers on capacity it owns and patches, with no EC2 instances for you to size, drain, or reboot. That removes the host layer, but it does not remove the decisions that decide whether a deploy is safe at 3am: how each task gets an IP and a security group, when a deployment rolls back on its own, and what happens to in-flight requests when a task is told to stop. Those are the knobs that separate a service which drains cleanly on every release from one that drops connections, leaks IPs, and pages you on a Friday.

This guide walks the pieces I actually wire up for a production Fargate service, the same way I’d brief a new engineer joining the on-call rotation. Every section goes option-by-option: the valid CPU/memory matrix, the awsvpc ENI/IP planning math, the choice between target-tracking and step scaling and exactly which metric to scale on, the deployment circuit breaker that auto-rolls-back a bad image, the SIGTERM → stopTimeout → deregistration-delay triad that makes shutdown graceful, the execution-role vs task-role split that is the most common IAM mistake on ECS, and the Fargate Spot + Graviton + right-sizing levers that move the bill. Because this is a reference you’ll return to mid-incident, the options, limits, error codes and the deploy playbook itself are laid out as scannable tables — read the prose once, then keep the tables open when the rollout is stuck.

Assume the AWS provider/region is set, an Application Load Balancer (ALB) exists, and you’re on a recent CLI (aws --version >= 2.x). I use the Linux platform version LATEST throughout, which today resolves to Fargate platform version 1.4.0, and I default to ARM64 (Graviton) because it is the cheapest change you can make. By the end you’ll be able to register a correctly-sized task, place it in private subnets with per-task security groups, scale it on the right signal, deploy it with an automatic safety net, and shut it down without severing a single request.

What problem this solves

ECS on Fargate hides the fleet so you can ship a container without owning servers. That abstraction is a gift until a deploy goes sideways, and then the failure modes are not in your application code — they’re in the wiring between the ALB, the task ENI, the scaling policy, and the lifecycle hooks. The defaults are tuned for “it runs”, not for “it survives a release under load”, and almost every production ECS incident I’ve seen traces back to one of a small set of mis-set knobs.

What breaks without this knowledge, concretely: a service scaled on CPU that is actually I/O-bound scales out after p99 has already tripled, because CPU never crossed the target while request queues grew. A container launched via sh -c "java -jar app.jar" where the shell is PID 1 swallows SIGTERM, so the JVM is SIGKILLed on every task stop and every in-flight request dies. An ALB target group left at the default 300-second deregistration delay keeps routing to tasks ECS has already begun stopping. A bad image with no circuit breaker leaves ECS replacing failing tasks forever, draining the subnet’s IP pool until new tasks can’t even be placed. And the single most common IAM mistake — conflating the execution role (used by the agent to pull the image and read secrets before the container starts) with the task role (used by your code at runtime) — silently grants your application code permissions it should never have.

Who hits this: every team running containers on Fargate behind a load balancer. It bites hardest on services with chatty downstreams (the scaling-metric trap), services that were lifted from EC2 without revisiting PID 1 and signal handling (the dropped-connection trap), large services in small subnets (the IP-exhaustion trap), and anyone who has never actually seen their circuit breaker fire — because a safety net you’ve never tested is a configuration you don’t really have.

To frame the whole field before the deep dive, here is every failure class this article covers, the question it forces, and the one place to look first:

Learning objectives

By the end of this article you can:

• Pick a valid Fargate cpu/memory combination for the whole task (app + sidecars), choose ARM64 vs X86_64, and pin images to an immutable digest so two tasks in one deployment never run different code.
• Plan awsvpc networking: one ENI and security group per task, the subnet IP-consumption math during a rolling deploy, and reaching ECR/Secrets Manager/CloudWatch via VPC interface endpoints instead of a NAT gateway.
• Register a service as a scalable target and choose between target-tracking (and which predefined metric) and step scaling, layer multiple policies safely, and get the ResourceLabel right so the policy isn’t silently a no-op.
• Configure a rolling deployment with minimumHealthyPercent/maximumPercent, a deployment circuit breaker with rollback, and a health-check grace period — and know when to reach for native blue/green instead.
• Make shutdown graceful by coordinating SIGTERM, PID 1, stopTimeout, and the ALB deregistration delay so in-flight requests always drain.
• Separate the execution role from the task role, inject secrets via the secrets block (never environment), and scope each role to specific ARNs.
• Wire Container Insights, structured JSON logs (awslogs vs FireLens), and ADOT/X-Ray tracing, then pull the cost levers: Fargate Spot via a capacity-provider strategy, Graviton, and right-sizing.
• Read a symptom → root cause → confirm → fix deploy playbook and localize any Fargate deploy/scale failure to one hop.

Prerequisites & where this fits

You should already understand the container basics: an image is built and pushed to a registry (Amazon ECR), a task definition is the immutable, versioned spec of what to run, a task is one running instance of that spec, and a service keeps a desired number of tasks running and registered behind a load balancer. You should know how to run aws in a shell, read JSON output, and that a VPC has subnets spread across Availability Zones, security groups (stateful, allow-only), and route tables. Familiarity with HTTP status codes, basic Linux process/signal concepts, and IAM policy JSON helps.

This sits in the Containers track and assumes the fundamentals from Amazon ECS & ECR Fundamentals: Task Definitions, Services & Fargate and the first deploy in Your First Container Deployment on ECS Fargate. It pairs tightly with Elastic Load Balancing Deep Dive: ALB, NLB & GWLB (the ALB and target group are half the deploy story) and VPC Deep Dive: Subnets, Routing, IGW, NAT & Endpoints (where the per-task ENIs and VPC endpoints live). For the choice of whether Fargate is even the right runtime, Choose Your Container Path: ECS vs EKS vs Fargate is upstream of this.

A quick map of who owns what during a Fargate incident, so you page the right person fast:

Core concepts

Six mental models make every later decision obvious.

A task is a first-class network citizen, not a process sharing a host. On Fargate the network mode is always awsvpc: each task gets its own elastic network interface (ENI) with a private IP from the subnet you place it in, and its own security group(s). You get per-task security groups, per-task VPC Flow Logs, and clean blast-radius isolation — at the cost of consuming one subnet IP (and one ENI) per running task. That IP consumption is the planning trap, because during a rolling deploy you briefly run more tasks than steady state.

The task definition is immutable and versioned; the service points at one revision. Every register-task-definition produces a new revision (family:N). The service runs whatever revision you set, and a deploy is “make the service converge from revision N to revision N+1”. This is why pinning the image to a digest matters: if the task definition says :latest, ECS resolves the tag at each task launch, so two tasks in the same deployment can pull different code. Immutability of the task definition doesn’t help if the image tag moves underneath it.

A deploy is a controlled overlap of two task sets. ECS brings up tasks from the new revision before draining the old ones, bounded by two percentages of desired count: minimumHealthyPercent (the floor it keeps healthy) and maximumPercent (the ceiling it may temporarily exceed). The overlap is what gives zero-downtime — and what consumes extra IPs and extra Fargate vCPU-seconds for the duration. The deployment circuit breaker watches for a run of failed task launches and, if rollback is on, reverts to the last known-good revision instead of replacing failing tasks forever.

Scaling is a separate control loop on top of the service. ECS services scale through Application Auto Scaling, registered against a scalable target with a min and max. A policy adjusts desiredCount based on a CloudWatch metric. The hard part is not the mechanics — it’s choosing a metric that leads load. For a request-driven service, request count per target leads; CPU often lags because the work is I/O-bound.

Shutdown is a negotiation between three timers. When ECS stops a task it sends SIGTERM to each container’s PID 1, waits up to stopTimeout (default 30s, max 120s), then SIGKILL. Simultaneously it deregisters the task from the ALB target group, and the ALB waits the deregistration delay for in-flight connections to finish. Graceful shutdown means PID 1 receives SIGTERM, the app drains (stop accepting, finish in-flight, exit) inside stopTimeout, and the deregistration delay is long enough to cover the drain but no longer.

A task has two identities, and conflating them is the classic mistake. The execution role is assumed by the ECS agent before your container starts — to pull the image from ECR, write to the log group, and resolve secrets references. The task role is assumed by your application code at runtime to call AWS APIs (S3, DynamoDB, SQS). They are different principals doing different things at different times; secrets-reading belongs to the execution role, not the task role.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

1. Task definition: sizing, platform, and the CPU/memory matrix

A Fargate task definition declares the container(s), the CPU/memory envelope, the network mode (awsvpc), and two distinct IAM roles. The CPU/memory pair is not free-form: Fargate only accepts specific combinations, and the valid memory range is constrained by the CPU value you pick. The whole task shares this budget — a sidecar’s usage comes out of the same pool — so size the task for the sum, then optionally cap individual containers with container-level cpu/memory.

The container-level fields that shape sizing and lifecycle, each with its default and the trade-off:

The platform/runtime choices, where the biggest cost decision (ARM64) hides:

A correct task definition, ARM64, digest-pinned, with a sane health check and bounded non-blocking logs:

{
  "family": "checkout-api",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "runtimePlatform": { "cpuArchitecture": "ARM64", "operatingSystemFamily": "LINUX" },
  "executionRoleArn": "arn:aws:iam::111122223333:role/checkout-execution",
  "taskRoleArn": "arn:aws:iam::111122223333:role/checkout-task",
  "containerDefinitions": [
    {
      "name": "app",
      "image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/checkout@sha256:9b2c…e41",
      "essential": true,
      "user": "10001:10001",
      "readonlyRootFilesystem": true,
      "portMappings": [{ "containerPort": 8080, "protocol": "tcp" }],
      "stopTimeout": 60,
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost:8080/healthz || exit 1"],
        "interval": 15, "timeout": 5, "retries": 3, "startPeriod": 30
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/checkout-api",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "app",
          "mode": "non-blocking",
          "max-buffer-size": "25m"
        }
      }
    }
  ]
}

Register it:

aws ecs register-task-definition --cli-input-json file://checkout-api.task.json

The same in Terraform, with the digest passed in from CI so it’s never :latest:

resource "aws_ecs_task_definition" "checkout" {
  family                   = "checkout-api"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "512"
  memory                   = "1024"
  execution_role_arn       = aws_iam_role.exec.arn
  task_role_arn            = aws_iam_role.task.arn

  runtime_platform {
    cpu_architecture        = "ARM64"
    operating_system_family = "LINUX"
  }

  container_definitions = jsonencode([{
    name        = "app"
    image       = var.image_digest        # "...checkout@sha256:..."
    essential   = true
    user        = "10001:10001"
    portMappings = [{ containerPort = 8080, protocol = "tcp" }]
    stopTimeout = 60
  }])
}

The two choices worth repeating: ARM64 (Graviton) is typically ~20% cheaper per vCPU-hour and usually performs as well or better on typical web workloads — covered in depth in Graviton/ARM64 Migration: Multi-Arch Builds & Benchmarking. And pin a digest — a moving tag is a correctness bug, not a convenience.

The other task-definition fields you’ll touch on a real service, with their defaults and the trade-off:

2. awsvpc networking: one ENI and IP per task, and the deploy-surge math

On Fargate the network mode is always awsvpc, so each task is a first-class network citizen with its own ENI, private IP, and security group(s). This is the single most important networking fact about Fargate, and it has two consequences you must plan for: IP consumption and egress routing.

The IP-consumption trap is the rolling deploy. During a deploy you briefly run more tasks than steady state, and each consumes a subnet IP. Plan subnets for the peak, not the average:

Peak task IPs during a deploy ≈ desired_count × (maximumPercent / 100). For a 40-task service at maximumPercent: 200, plan for up to 80 task IPs across your subnets during the deploy, on top of everything else (other services, ENIs, reserved addresses) in those subnets.

A subnet also reserves 5 addresses (network, router, DNS, future, broadcast), so the usable count is smaller than the raw CIDR size. The math by subnet size:

Spread tasks across at least two private subnets in different AZs, and give each a /24 or larger for any sizeable service. The service network config disables public IPs and references security groups by ID:

{
  "awsvpcConfiguration": {
    "subnets": ["subnet-0aaa1111", "subnet-0bbb2222"],
    "securityGroups": ["sg-0task55555"],
    "assignPublicIp": "DISABLED"
  }
}

assignPublicIp must be DISABLED for tasks in private subnets — they reach AWS services through a NAT gateway or, better, VPC interface endpoints. The egress choices, side by side:

The minimum endpoint set to pull an image and read secrets without a NAT gateway is: com.amazonaws.<region>.ecr.api, ecr.dkr, secretsmanager, logs, sts (interface), plus an S3 gateway endpoint (ECR stores layers in S3). The security-group rules — reference SGs by ID, never CIDR:

Terraform for the two essential ECR-related endpoints (S3 gateway is free and mandatory):

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = var.vpc_id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = var.private_route_table_ids   # ECR layer pulls go via S3
}

resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true
}

The Fargate limits and quotas that actually shape a production design (many are soft/adjustable via Service Quotas — confirm against your account):

Networking details — subnet design, route tables, endpoint policies — are covered end to end in VPC Deep Dive: Subnets, Routing, IGW, NAT & Endpoints, and the inbound/outbound rule model in Security Groups & NACLs Deep Dive.

3. Service Auto Scaling: target tracking vs step scaling

ECS services scale through Application Auto Scaling, registered as a scalable target against the ecs:service:DesiredCount dimension with a min and max. The mechanics are easy; the metric choice is the whole game. For a request-driven service behind an ALB, ALBRequestCountPerTarget is the cleanest signal — it scales on actual load per task, independent of how CPU-bound the work is, and reacts before CPU saturates.

# Register the service as a scalable target (min/max bound on desiredCount)
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/prod-cluster/checkout-api \
  --scalable-dimension ecs:service:DesiredCount \
  --min-capacity 4 --max-capacity 40

The predefined target-tracking metrics, and exactly when each is the right one:

A request-count target-tracking policy. The ResourceLabel is <ALB full name>/<target group full name> — the portion after loadbalancer/ and targetgroup/ in the respective ARNs. Get it wrong and the policy silently does nothing:

{
  "TargetValue": 1000.0,
  "PredefinedMetricSpecification": {
    "PredefinedMetricType": "ALBRequestCountPerTarget",
    "ResourceLabel": "app/checkout-alb/50dc6c495c0c9188/targetgroup/checkout-tg/6d0ecf831eec9f09"
  },
  "ScaleInCooldown": 300,
  "ScaleOutCooldown": 60
}

aws application-autoscaling put-scaling-policy \
  --service-namespace ecs --resource-id service/prod-cluster/checkout-api \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-name reqcount-tt --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration file://reqcount-tt.json

The target-tracking knobs and how to reason about each:

Reach for step scaling when you need asymmetric or aggressive reactions — for example, add capacity hard when a queue-depth alarm crosses a threshold (a worker draining SQS, not a web service):

aws application-autoscaling put-scaling-policy \
  --service-namespace ecs --resource-id service/prod-cluster/worker \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-name queue-step --policy-type StepScaling \
  --step-scaling-policy-configuration '{
    "AdjustmentType": "ChangeInCapacity",
    "MetricAggregationType": "Maximum",
    "StepAdjustments": [
      { "MetricIntervalLowerBound": 0,   "MetricIntervalUpperBound": 1000, "ScalingAdjustment": 2 },
      { "MetricIntervalLowerBound": 1000,                                   "ScalingAdjustment": 5 }
    ]
  }'

Target tracking vs step scaling, decided:

The CloudWatch metrics worth alerting on for a Fargate service (leading indicators, not just “service down”), with a starting threshold:

You can attach multiple policies to one service. A common pattern: request-count target tracking for steady state, plus a CPU target-tracking policy as a safety net so a CPU-heavy code path can’t starve before request count reacts. When policies disagree, Application Auto Scaling takes the largest desired count — so layering scale-out policies is safe; combining aggressive scale-in policies is what gets you into flapping. A decision table for picking the primary signal:

4. Deployments: rolling updates and the circuit breaker

ECS rolling deployments are governed by two knobs on the service. minimumHealthyPercent is the floor of healthy tasks ECS keeps during a deploy; maximumPercent is the ceiling it may temporarily exceed desired count to bring up replacements. For a zero-downtime rolling deploy on an even-sized service, 100/200 is the safe default: never drop below desired count, allow a full extra set while rolling.

The deploy-config matrix — every knob, its default, and the trade-off:

The piece people skip is the deployment circuit breaker. Without it, a bad image that never passes health checks leaves the service replacing failing tasks indefinitely — draining your IP pool and paging you. With it, ECS watches for a run of failed task launches and, if rollback is on, automatically reverts to the last known-good task definition.

aws ecs update-service \
  --cluster prod-cluster --service checkout-api \
  --task-definition checkout-api:87 \
  --deployment-configuration '{
    "minimumHealthyPercent": 100,
    "maximumPercent": 200,
    "deploymentCircuitBreaker": { "enable": true, "rollback": true }
  }' \
  --health-check-grace-period-seconds 60

resource "aws_ecs_service" "checkout" {
  name            = "checkout-api"
  cluster         = aws_ecs_cluster.prod.id
  task_definition = aws_ecs_task_definition.checkout.arn
  desired_count   = 4
  launch_type     = "FARGATE"
  health_check_grace_period_seconds = 60

  deployment_circuit_breaker { enable = true, rollback = true }
  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200

  load_balancer {
    target_group_arn = aws_lb_target_group.checkout.arn
    container_name   = "app"
    container_port   = 8080
  }
  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.task.id]
    assign_public_ip = false
  }
}

--health-check-grace-period-seconds tells ECS to ignore ALB health-check failures for the first N seconds after a task starts, so a slow-booting app isn’t killed before it’s ready. Set it slightly above your real cold-start time. The circuit breaker counts failures relative to desired count (it scales the threshold with service size, with a floor), so it behaves sensibly for both a 3-task and a 300-task service.

The deployment-controller and strategy options, decided:

The rollout states you watch during a deploy, and what each means:

The most common task stopped reasons you’ll read in describe-tasks during a failed rollout, and what each points at:

5. Graceful shutdown: SIGTERM, stopTimeout, and deregistration

When ECS stops a task — a deploy, a scale-in, or a Spot interruption — it sends SIGTERM to each container’s entrypoint process (PID 1), waits up to stopTimeout (default 30s on Fargate, max 120s), then sends SIGKILL. Two failure modes hide here, and they are the number-one cause of deploy-time errors.

First: PID 1 must actually receive and handle SIGTERM. If your container starts the app via a shell (sh -c "node server.js"), the shell is PID 1 and may not forward the signal — your app gets SIGKILLed with in-flight requests. Either run the app as PID 1 directly (exec form CMD, or ENTRYPOINT ["node", "server.js"]) or set "initProcessEnabled": true in linuxParameters to get a tini-style init that reaps zombies and forwards signals. The combinations:

Second: drain before you exit. On SIGTERM the app should stop accepting new work, finish in-flight requests, then exit — inside stopTimeout:

const server = app.listen(8080);

process.on('SIGTERM', () => {
  console.log('SIGTERM received, draining');
  server.close(() => {                 // stop accepting, finish in-flight
    console.log('drained, exiting');
    process.exit(0);
  });
  // safety net well under stopTimeout (60s here)
  setTimeout(() => process.exit(1), 50_000).unref();
});

Coordinate three timers so they nest correctly. The order that must hold: deregistration delay ≥ app drain grace ≤ stopTimeout, and stopTimeout ≥ drain grace. ECS deregisters the task from the target group on stop; the ALB stops sending new connections and waits the deregistration delay for existing ones to finish. If stopTimeout is shorter than the drain, SIGKILL cuts the app mid-drain; if the deregistration delay is shorter than the drain, the ALB cuts connections the app is still serving.

The target group itself must register tasks by IP, not instance, because each Fargate task is its own ENI:

resource "aws_lb_target_group" "checkout" {
  name                 = "checkout-tg"
  port                 = 8080
  protocol             = "HTTP"
  target_type          = "ip"          # REQUIRED for awsvpc/Fargate tasks
  vpc_id               = var.vpc_id
  deregistration_delay = 30            # drain fast, well inside stopTimeout

  health_check {
    path                = "/healthz"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    interval            = 15
    timeout             = 5
    matcher             = "200"
  }
}

6. Secrets, config, and least-privilege roles

Fargate tasks have two roles, and conflating them is the most common IAM mistake on ECS. The split, in one table:

Keep them separate and minimal. Inject secrets via the secrets block so plaintext never lands in the task definition or in describe-tasks output, and keep non-sensitive config in environment:

"secrets": [
  { "name": "DB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/checkout/db-AbCdEf" }
],
"environment": [
  { "name": "LOG_LEVEL", "value": "info" }
]

The execution role needs secretsmanager:GetSecretValue (and kms:Decrypt if the secret uses a customer-managed key) on exactly those secret ARNs — not * — plus the ECR and Logs actions:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/checkout/*"
}

The task role carries only the runtime permissions your code uses. If your app writes to one bucket, scope it to that bucket’s ARN and the s3:PutObject action — nothing more. Static environment entries are visible in plaintext via the API, so never put credentials there; that’s what secrets is for. The config-injection options compared:

Secrets-rotation patterns and Parameter Store vs Secrets Manager are covered in Secrets Manager & Parameter Store Deep Dive; scoping roles tightly is the subject of IAM Least Privilege & Permission Boundaries.

7. Observability: Container Insights, structured logs, tracing

Turn on Container Insights at the cluster level for per-task/service CPU, memory, and network metrics plus curated dashboards. Enable the enhanced observability tier for container-level granularity:

aws ecs update-cluster-settings \
  --cluster prod-cluster \
  --settings name=containerInsights,value=enhanced

The three observability pillars and how to wire each on Fargate:

For logs, the awslogs driver is the simplest path; set mode=non-blocking with a bounded max-buffer-size so a slow log backend can’t block your application threads. The log-driver options:

When you need routing — duplicate to S3 and a SIEM, parse, or sample — use FireLens with a Fluent Bit sidecar:

{
  "name": "log_router",
  "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable",
  "essential": true,
  "firelensConfiguration": { "type": "fluentbit" },
  "memoryReservation": 50
}

Then the app container’s logConfiguration uses "logDriver": "awsfirelens" with output options. Emit logs as JSON from the app so they’re queryable in CloudWatch Logs Insights:

fields @timestamp, level, msg, latency_ms
| filter level = "error"
| sort @timestamp desc
| limit 50

For distributed tracing, add the AWS Distro for OpenTelemetry (ADOT) collector as a sidecar and grant the task role AWSXRayDaemonWriteAccess; instrument the app with OTel and export to X-Ray for end-to-end spans. The full tracing setup is in AWS X-Ray: Service Map, Segments & ADOT Tracing; the metrics/logs foundation in CloudWatch & CloudTrail Observability Deep Dive.

8. Cost levers: Fargate Spot, capacity providers, right-sizing

Three levers move the bill, in order of impact.

Capacity providers + Fargate Spot. Fargate Spot runs the same tasks at a steep discount but can reclaim them with a ~2-minute SIGTERM warning. Run a mixed strategy via a capacity-provider strategy: a base of on-demand FARGATE for a guaranteed floor, then FARGATE_SPOT for the elastic, interruption-tolerant remainder:

aws ecs put-cluster-capacity-providers \
  --cluster prod-cluster \
  --capacity-providers FARGATE FARGATE_SPOT \
  --default-capacity-provider-strategy \
    capacityProvider=FARGATE,base=2,weight=1 \
    capacityProvider=FARGATE_SPOT,weight=4

This keeps 2 tasks always on-demand, then splits additional tasks 1:4 on-demand:Spot. The capacity-provider parameters:

Only do this for stateless services that handle SIGTERM cleanly — Spot reclamation uses the same graceful-stop path, so a service that drains correctly tolerates it. The three cost levers ranked:

Graviton (ARM64) — already covered in Section 1, the cheapest change you can make for compatible images. Right-sizing — use Container Insights and Compute Optimizer’s ECS recommendations to find tasks provisioned at 4 vCPU that peak at 0.8. Fargate bills per vCPU-second and GB-second from pull to stop, so an oversized task definition costs you on every running replica, every hour. Resize the task definition, redeploy, re-measure. Spot interruption handling at scale is the subject of EC2 Spot & Mixed Instances with ASG Interruption Handling — the same draining discipline applies.

Going deeper

The eight numbered sections above are the working spec. This section is for the reader who already ships Fargate services and wants the internals, the sharp edges, and the facts that only bite in production.

Platform versions: what LATEST actually pins, and why tasks get retired under you

LATEST is not a version — it is a pointer that today resolves to Linux platform version 1.4.0. Fargate platform versions bundle the agent, kernel, and runtime behavior, and the jump most teams still carry scar tissue from is 1.3.0 → 1.4.0, which changed enough to break old assumptions:

Two production consequences fall out of this. First, pinning platformVersion: "1.4.0" buys reproducibility but freezes you out of fixes shipped to LATEST; pin to debug a version-sensitive issue, not as a permanent setting. Second, AWS periodically retires running tasks to patch the platform under LATEST — you do not opt out. A retired task is stopped with the same SIGTERM → stopTimeout → SIGKILL sequence as a deploy, and a service simply launches a replacement. So the graceful-shutdown discipline from Section 5 is not only a deploy concern: if your app swallows SIGTERM, an AWS-initiated maintenance retirement will drop requests on a random Tuesday with nothing in your change log to blame. Retirements arrive as an AWS Health event and an EventBridge ECS Task State Change — wire an alert so a mystery restart is explained, not investigated. Windows containers run on their own platform-version line (1.0.0) with fewer sizes and AZs.

The task ENI lifecycle, and why awsvpcTrunking is not your knob

Each Fargate task attaches a dedicated ENI before the image is pulled — which is exactly why a cold task spends time in PROVISIONING, and why RESOURCE:ENI appears when the subnet is out of addresses. The ENI is created, attached, given its IP, and wired to your security groups; only then does the pull begin. A few internals worth holding:

The practical takeaway: your only levers over the IP-exhaustion trap are subnet size, number of AZs/subnets, maximumPercent, and (increasingly) IPv6 dual-stack — not any ENI-density account setting.

Ephemeral storage: where it goes and how it’s protected

Every task gets ephemeral storage (20 GiB free on 1.4.0, configurable to 200 GiB via ephemeralStorage.sizeInGiB) used for the image’s writable layer and any scratch the app writes outside a mounted volume. Three facts people miss:

• It is encrypted at rest by default on 1.4.0 with AES-256 using keys AWS manages — there is no toggle to turn it on and no customer-managed KMS key to attach to it. If a compliance regime demands CMK-controlled data at rest, that data belongs on EFS (which does take your KMS key) or S3, never in ephemeral scratch.
• It is ephemeral in the literal sense: created at task start, destroyed at task stop. Nothing survives a task replacement, so never treat it as a cache that outlives a deploy.
• You are billed for provisioned ephemeral storage above the 20 GiB free, per GB-month, for the task’s lifetime — so bumping to 200 GiB “just in case” is a standing cost on every replica.

Scheduled scaling, and the thing that does not scale on Fargate

Section 3 covered target-tracking and step scaling — both reactive. For load you can predict (a batch window, a 9am login surge, a nightly quiet period), add a scheduled action to the same scalable target so capacity is in place before the traffic arrives:

aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --resource-id service/prod-cluster/checkout-api \
  --scalable-dimension ecs:service:DesiredCount \
  --scheduled-action-name morning-prewarm \
  --schedule "cron(45 8 * * ? *)" \
  --scalable-target-action MinCapacity=12,MaxCapacity=40

Two clarifications that trip people up. There is no predictive scaling for ECS services — that feature exists only for EC2 Auto Scaling groups; on ECS you get target-tracking, step, and scheduled. And on Fargate there is no cluster capacity to scale — the whole “scale the EC2 fleet behind the tasks” problem (managed cluster auto scaling with capacity providers on EC2) simply does not exist, because there is no fleet you own. You scale tasks; AWS finds the capacity. Scheduled actions layer with target tracking: the schedule moves the floor/ceiling, target tracking moves desiredCount within it.

East-west traffic: Service Connect vs an internal ALB vs raw Cloud Map

The ALB in this lesson is north-south (clients → service). Once two Fargate services call each other, you need east-west discovery, and there are three models:

Service Connect is usually the right modern default for service-to-service on ECS: it adds client-side retries and per-call metrics without standing up an internal ALB per callee. The full comparison is in ECS Service Connect vs Load Balancers.

The circuit-breaker threshold, and the one failure it can’t save you from

The deployment circuit breaker does not trip on the first failed task. It counts consecutive failed launches against a threshold that scales with service size — AWS computes it as roughly half the desired count, floored at 3 and capped at 200. A 4-task service tolerates a couple of failures before tripping; a 300-task service tolerates ~150. That is deliberate: one flaky task should not abort a large rollout, and a tiny service should not loop forever.

The failure it cannot save you from is a bad dependency change that also breaks the last known-good revision. Rollback reverts the task definition, not the world. If the deploy fails because you deleted the secret the exec role reads, or tightened a security group the task needs, then rolling back to revision N-1 lands you on a revision that now also can’t start — and the breaker has nowhere good to go. It protects against a bad image or task-def; it does not protect against a bad change underneath a good task-def. That is why testing the breaker once in staging (Section 5) matters — you want to have watched rolloutState: FAILED resolve into a healthy prior revision before you trust it at 3am. Spot reclaims, task retirements, and this rollback path all lean on the same graceful-stop plumbing, so the interruption-handling discipline generalizes; EC2 Spot & Mixed Instances walks the reclaim side end to end.

Architecture at a glance

Follow a single request left to right and the whole system falls into place. A client hits the ALB on 443; the ALB terminates TLS and forwards to a healthy target on the container port (8080). Because the tasks run awsvpc, the target group is target_type = ip and the ALB routes straight to a task ENI’s private IP inside two private subnets across two AZs — each task its own ENI, its own security group that only accepts the ALB’s SG, no public IP. The task itself runs the app container plus any sidecars on a valid CPU/memory envelope (here 512/1024, ARM64), with two IAM roles: the execution role pulled the digest-pinned image from ECR (via a VPC endpoint, layers over the free S3 gateway endpoint) and read secrets before start, while the task role is what the app’s SDK uses at runtime. Two control loops sit beside the data path: Application Auto Scaling watches ALBRequestCountPerTarget and moves desiredCount between 4 and 40, and the rollout runs at 100/200 with the deployment circuit breaker armed to roll back a bad revision. Downstream, the task reaches Secrets Manager + KMS and ships logs and traces to CloudWatch / X-Ray.

The five numbered badges mark exactly where a deploy or scale event breaks if a knob is wrong: a deploy-time 502 when the target type isn’t ip or the deregistration delay is still 300s (badge 1); IP/ENI exhaustion when the subnets can’t absorb the deploy surge (badge 2); swallowed SIGTERM dropping in-flight requests when a shell is PID 1 (badge 3); late or flapping scaling from the wrong metric (badge 4); and a bad deploy that never rolls back when the circuit breaker is off (badge 5). Read the diagram once with the legend, and the troubleshooting playbook below maps one-to-one onto these hops.

Real-world scenario

Lumio Pay, a fintech platform team, ran a payment-authorization service on Fargate behind an ALB, scaled on CPU target tracking, 6 tasks steady. It worked until a Friday evening release: under a traffic spike, p99 latency tripled and the team saw a steady trickle of 502s on every deploy and every scale-in event — even though CPU never crossed the 70% target. The on-call engineer’s first instinct was to scale up the task size, which did nothing, then to roll back, which also threw 502s on the way down.

Three root causes, none of them the application logic. First, the service used CPU target tracking, but the workload was I/O-bound on a downstream HSM — CPU stayed low while request queues grew, so scaling reacted late, after latency had already spiked. Second, and worse, the app was launched via sh -c "java -jar app.jar": the shell was PID 1, swallowed SIGTERM, and the JVM was SIGKILLed on every task stop, severing in-flight authorizations the instant a task drained. Third, the ALB target group still had the default 300-second deregistration delay, so during deploys the ALB kept routing new connections to tasks ECS had already begun stopping — a second source of cut connections layered on top of the first.

They confirmed each in minutes. The scaling lag showed up as CloudWatch CPU flat at ~40% while the ALB target-response-time and request-count climbed. The PID 1 problem was visible in the task definition ("command": ["sh","-c","java -jar app.jar"]) and in the stopped-task pattern — every deploy logged tasks SIGKILLed, not gracefully exited. The deregistration delay was a one-line describe-target-group.

The fix was three coordinated changes, no new infrastructure. They switched the primary scaling signal to ALBRequestCountPerTarget (keeping a CPU policy as a backstop), changed the container entrypoint to exec the JVM as PID 1 with a real SIGTERM handler that drained the in-flight queue, and aligned the timers: deregistration delay to 30s, stopTimeout to 60s, drain grace to ~45s.

"deploymentConfiguration": {
  "minimumHealthyPercent": 100,
  "maximumPercent": 200,
  "deploymentCircuitBreaker": { "enable": true, "rollback": true }
}

resource "aws_lb_target_group" "auth" {
  name                 = "auth-tg"
  port                 = 8080
  protocol             = "HTTP"
  target_type          = "ip"          # required for awsvpc/Fargate tasks
  vpc_id               = var.vpc_id
  deregistration_delay = 30

  health_check {
    path                = "/healthz"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    interval            = 15
    timeout             = 5
    matcher             = "200"
  }
}

Note target_type = "ip" — Fargate tasks register by IP, not instance, because each task is its own ENI. After the change, deploy-time 502s went to zero, and the service scaled out ahead of the latency curve instead of behind it. While they were in there, they also enabled the circuit breaker with rollback (they’d never had one) and tested it by deploying a deliberately-broken revision in staging — it flipped to FAILED and restored the prior revision in under two minutes. The lesson the team took away: on Fargate, “graceful shutdown” is not one setting — it’s PID 1, stopTimeout, and the target-group deregistration delay all agreeing with each other, and “scaling” only works if the metric you chose actually leads your load.

Advantages and disadvantages

The serverless-container model both removes real toil and introduces failure modes that live in the wiring rather than your code. Weigh it honestly:

Fargate is the right default when you want to ship containers, not operate servers, and your services are stateless and ALB-fronted. It bites hardest on chatty/I/O-bound services scaled on the wrong metric, services lifted from EC2 without revisiting PID 1 and signal handling, large services packed into small subnets, and anyone who deploys with the defaults (no circuit breaker, 300s deregistration delay) and never tunes them. The disadvantages are all manageable — but only if you know they exist, which is the entire point of this article. When the constraints argue for self-managed nodes (GPU, daemons, very high task density, specialized kernels), Choose Your Container Path: ECS vs EKS vs Fargate is the decision to revisit.

Hands-on lab

Stand up a minimal Fargate service, deploy a deliberately-broken revision, and watch the circuit breaker roll it back — then prove graceful drain. Free-tier-friendly-ish (Fargate has no free tier, but a 512/1024 task for under an hour is a few rupees; tear down at the end). Run in any shell with the AWS CLI configured.

Step 1 — Variables and a cluster.

REGION=us-east-1
CLUSTER=lab-cluster
aws ecs create-cluster --cluster-name $CLUSTER --region $REGION \
  --settings name=containerInsights,value=enhanced

Step 2 — A log group and a minimal task definition (good image). Use a public sample that listens on 80:

aws logs create-log-group --log-group-name /ecs/lab-web --region $REGION
cat > lab-web.task.json <<'JSON'
{
  "family": "lab-web",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256", "memory": "512",
  "executionRoleArn": "arn:aws:iam::ACCOUNT:role/ecsTaskExecutionRole",
  "containerDefinitions": [{
    "name": "web",
    "image": "public.ecr.aws/nginx/nginx:stable",
    "essential": true,
    "portMappings": [{ "containerPort": 80, "protocol": "tcp" }],
    "stopTimeout": 30,
    "logConfiguration": { "logDriver": "awslogs", "options": {
      "awslogs-group": "/ecs/lab-web", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "web" } }
  }]
}
JSON
aws ecs register-task-definition --cli-input-json file://lab-web.task.json --region $REGION

Expected: a taskDefinition JSON with "revision": 1 and "status": "ACTIVE".

Step 3 — Create the service with the circuit breaker armed. Use two private subnets and a task SG you already have (or a default-VPC subnet + SG for the lab):

aws ecs create-service --cluster $CLUSTER --service-name lab-web \
  --task-definition lab-web:1 --desired-count 2 --launch-type FARGATE \
  --deployment-configuration '{"deploymentCircuitBreaker":{"enable":true,"rollback":true},"minimumHealthyPercent":100,"maximumPercent":200}' \
  --network-configuration 'awsvpcConfiguration={subnets=[subnet-AAA,subnet-BBB],securityGroups=[sg-XXX],assignPublicIp=ENABLED}' \
  --region $REGION

Expected: a service with "rolloutState": "IN_PROGRESS" that reaches COMPLETED once 2 tasks are running.

Step 4 — Prove each task has its own ENI + private IP (awsvpc).

aws ecs list-tasks --cluster $CLUSTER --service-name lab-web --query 'taskArns' --output text --region $REGION \
  | xargs aws ecs describe-tasks --cluster $CLUSTER --region $REGION --tasks \
  --query 'tasks[].attachments[].details[?name==`privateIPv4Address`].value' --output text

Expected: two distinct private IPs — one per task.

Step 5 — Register a deliberately-broken revision and deploy it. A task def pointing at an image that will never become healthy (a non-existent tag):

sed 's#nginx/nginx:stable#nginx/nginx:THIS-TAG-DOES-NOT-EXIST#' lab-web.task.json > lab-web.broken.json
aws ecs register-task-definition --cli-input-json file://lab-web.broken.json --region $REGION
aws ecs update-service --cluster $CLUSTER --service lab-web --task-definition lab-web:2 --region $REGION

Step 6 — Watch the circuit breaker fire and roll back.

aws ecs describe-services --cluster $CLUSTER --services lab-web --region $REGION \
  --query 'services[0].deployments[].{status:status,rollout:rolloutState,reason:rolloutStateReason,desired:desiredCount,running:runningCount,failed:failedTasks}'

Expected: the new deployment moves to rolloutState: FAILED with a rolloutStateReason mentioning the circuit breaker, and the service converges back onto revision 1 (the last known-good) — running stays at 2 throughout.

Validation checklist. You created a service with the breaker armed, proved per-task ENIs, deployed a broken revision, and watched ECS automatically restore the good one without you touching it. The steps mapped to what each proves:

Cleanup (avoid lingering Fargate charges).

aws ecs update-service --cluster $CLUSTER --service lab-web --desired-count 0 --region $REGION
aws ecs delete-service --cluster $CLUSTER --service lab-web --force --region $REGION
aws ecs delete-cluster --cluster $CLUSTER --region $REGION
aws logs delete-log-group --log-group-name /ecs/lab-web --region $REGION

Cost note. Two 256/512 tasks for under an hour is well under ₹40; deleting the service stops the per-second billing immediately. Container Insights enhanced adds a small ingestion cost — fine for a lab, watch it at scale.

Practice challenges

Work these in order — they climb from “read the matrix” to “design the surge”. Each solution states the why in one line, not just the answer.

1. (Beginner) Is this a valid task size? A teammate sets cpu: "512" and memory: "512". Will register-task-definition accept it? If not, give the smallest valid memory for that cpu.

<details> <summary>Solution</summary>

Rejected. At cpu: "512" (0.5 vCPU) the valid memory range is 1024–4096 MiB in 1 GiB steps, so the smallest valid value is "1024". 512 MiB is only valid at cpu: "256".

Why: Fargate’s CPU/memory pairs are a fixed matrix, not free-form — the memory range is gated by the CPU tier, and an invalid pair fails registration before anything runs. </details>

2. (Beginner) How many tasks fit a /26? You place a service in a single /26 subnet. How many usable IPs are there, and what is the largest steady task count that still survives a maximumPercent: 200 deploy?

<details> <summary>Solution</summary>

A /26 has 64 addresses; AWS reserves 5, leaving 59 usable. A 200% deploy peaks at 2 × steady, so steady must satisfy 2 × steady ≤ 59 → steady ≤ 29 tasks — and that assumes the subnet holds nothing else. In practice, spread the service across two /24s in two AZs.

Why: the rolling deploy, not steady state, sets the IP requirement — each task is an ENI holding one IP, and the surge briefly doubles the count. </details>

3. (Intermediate) Wire request-count scaling. Register a service as a scalable target (min 4, max 30) and attach a target-tracking policy that holds 800 requests/min per task. Sketch the two commands.

<details> <summary>Solution</summary>

aws application-autoscaling register-scalable-target \
  --service-namespace ecs --resource-id service/prod-cluster/checkout-api \
  --scalable-dimension ecs:service:DesiredCount --min-capacity 4 --max-capacity 30

aws application-autoscaling put-scaling-policy \
  --service-namespace ecs --resource-id service/prod-cluster/checkout-api \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-name reqcount-tt --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 800.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ALBRequestCountPerTarget",
      "ResourceLabel": "app/checkout-alb/50dc6c495c0c9188/targetgroup/checkout-tg/6d0ecf831eec9f09"
    },
    "ScaleInCooldown": 300, "ScaleOutCooldown": 60
  }'

Why: the scalable target only sets the min/max bounds — nothing scales until a policy is attached, and the ResourceLabel must be the exact <ALB full name>/<TG full name> or the policy silently no-ops. </details>

4. (Intermediate) Fix PID 1. A Dockerfile ends with CMD node server.js and the app is SIGKILLed on every deploy. Give two independent fixes.

<details> <summary>Solution</summary>

Fix A — exec form: CMD ["node", "server.js"] so node is PID 1 and receives SIGTERM directly. Fix B — init shim: keep the shell form but set "initProcessEnabled": true in the container’s linuxParameters, inserting a signal-forwarding, zombie-reaping init as PID 1. Either way, add a SIGTERM handler that stops accepting connections and drains in-flight requests inside stopTimeout.

Why: in shell form the shell is PID 1 and commonly does not forward SIGTERM, so the app never hears the stop and is killed with requests still in flight. </details>

5. (Advanced) Size the subnets for a real surge. A service runs 40 steady tasks at maximumPercent: 200, spread across two AZs, and those subnets also hold ~30 other ENIs total. What is the minimum subnet size per AZ, with headroom?

<details> <summary>Solution</summary>

Peak task IPs = 40 × 2 = 80, split across two AZs = 40 per AZ, plus ~15 other ENIs per AZ ≈ 55, plus AWS’s 5 reserved. A /26 (59 usable) is right at the edge and will fail under any drift — step up to a /25 (123 usable) per AZ for real headroom, or a /24.

Why: you size for desired × maximumPercent at peak and everything else already in the subnet, then leave slack — a subnet that is exactly full deploys fine until the day it doesn’t. </details>

6. (Advanced) Design a Spot mix with a floor. You want at least 3 tasks always on-demand for a guaranteed baseline, then the elastic remainder split roughly 1 on-demand to 3 Spot. Write the capacity-provider strategy and explain what a 12-task service looks like.

<details> <summary>Solution</summary>

aws ecs put-cluster-capacity-providers --cluster prod-cluster \
  --capacity-providers FARGATE FARGATE_SPOT \
  --default-capacity-provider-strategy \
    capacityProvider=FARGATE,base=3,weight=1 \
    capacityProvider=FARGATE_SPOT,weight=3

At 12 tasks: 3 go on-demand first (base), leaving 9 to split by weight 1:3 → ~2 more on-demand and ~7 Spot, so roughly 5 on-demand and 7 Spot. Only do this if the service drains cleanly on the ~2-minute Spot SIGTERM.

Why: base guarantees a floor on the reliable provider before weights apply; weight then splits only the remainder — so the reliability floor and the cost split are two independent dials. </details>

Common beginner mistakes

These are conceptual traps — wrong mental models a beginner carries in, not incident symptoms (those live in the troubleshooting playbook below). Fix the model and the symptoms never appear.

Three of these deserve a sentence more, because they cause the most 3am confusion:

• “One role per task.” The single most common ECS IAM bug is putting secretsmanager:GetSecretValue on the task role. Secrets are resolved by the ECS agent before your code runs, so that permission belongs on the execution role. Put it on the task role and image-pull-time secret resolution fails while your running code quietly holds a permission it should never have — wrong on both ends.
• “Scaling out fixes OOM.” OOM kills (exit code 137) come from exceeding the task’s memory value, which is identical on every replica. Ten copies of an under-memoried task OOM ten times. The fix is a larger valid CPU/memory combo (scale up) or a fixed leak — never more copies.
• “The breaker is automatic.” New engineers assume ECS rolls back bad deploys on its own. It does not, until you enable the circuit breaker with rollback. A service created from console defaults or a bare create-service call has no safety net at all.

Common mistakes & troubleshooting

This is the playbook — the part you bookmark. First as a scannable table you read mid-incident, then the entries that bite hardest with full confirm-command detail.

The expanded form for the entries that bite hardest:

1. 502s on every deploy and every scale-in; fine at steady state. Root cause: The ALB target group isn’t draining gracefully — either target_type isn’t ip (so registration is wrong for awsvpc) or the deregistration delay is the default 300s, longer than ECS’s stop sequence, so the ALB keeps sending new connections to a task ECS is stopping. Confirm: aws elbv2 describe-target-groups --target-group-arns <arn> --query 'TargetGroups[].{type:TargetType,dereg:Attributes}' (or read the deregistration-delay attribute). Inspect stopped tasks for SIGKILL vs graceful exit. Fix: target_type=ip, deregistration_delay=30, and make sure that delay sits under stopTimeout (e.g. 60) so both the ALB and ECS finish draining together.

2. In-flight requests error exactly when a task stops. Root cause: PID 1 is a shell (sh -c "...") that swallows SIGTERM, so the app never gets the signal and is SIGKILLed after stopTimeout with requests still in flight. Confirm: Inspect the task definition’s command/entryPoint for a shell wrapper; stopped tasks show no graceful-exit log line, just an abrupt stop. Fix: Run the app as PID 1 via the exec form (CMD ["node","server.js"]) or set "initProcessEnabled": true in linuxParameters; implement a SIGTERM handler that stops accepting and finishes in-flight inside stopTimeout.

3. p99 climbs before scale-out; service thrashes on scale-in. Root cause: Wrong scaling metric — CPU target tracking on an I/O-bound service, so CPU stays low while queues grow and scaling reacts late; and/or a too-short ScaleInCooldown causing flapping. Confirm: CloudWatch shows CPU flat (e.g. 40%) while the ALB’s RequestCountPerTarget and TargetResponseTime climb. Fix: Make ALBRequestCountPerTarget the primary signal (keep CPU as a backstop), and raise ScaleInCooldown to stop thrash.

5. Tasks stuck PROVISIONING; stopped reason RESOURCE:ENI or IP-not-available. Root cause: The subnet(s) ran out of free IPs/ENIs during the deploy surge — desired × maximumPercent exceeded usable addresses (often a /26 or smaller hosting a 30+ task service at maximumPercent: 200). Confirm: Compare each subnet’s free-IP count to desired_count × (maximumPercent/100); describe-tasks shows stoppedReason with RESOURCE:ENI. Fix: Move tasks to /24-or-larger subnets across ≥2 AZs; as an immediate unblock, lower maximumPercent (e.g. to 150) so the surge is smaller.

6. A bad image leaves failing tasks replaced forever, IP pool draining. Root cause: The deployment circuit breaker is off (or on without rollback), so ECS keeps launching tasks from a revision that never becomes healthy. Confirm: aws ecs describe-services --query 'services[0].deployments[].{rollout:rolloutState,failed:failedTasks}' shows IN_PROGRESS with failedTasks climbing. Fix: update-service --deployment-configuration '{"deploymentCircuitBreaker":{"enable":true,"rollback":true}}'; redeploy and test it once in non-prod so you’ve actually seen it fire.

8. ResourceInitializationError: unable to pull secrets or registry auth. Root cause: The execution role lacks secretsmanager:GetSecretValue (or kms:Decrypt for a CMK), or the task has no network route to the Secrets Manager / ECR endpoints. Confirm: describe-tasks → stoppedReason; check the exec-role policy and whether a secretsmanager interface endpoint (or NAT) exists. Fix: Grant the exec role the exact secret ARN and KMS key; add the secretsmanager (and ECR) VPC endpoints or a NAT route.

9. The app can read secrets or buckets it never references. Root cause: Secret-read or broad runtime permissions were attached to the task role (the one your code assumes), or both roles use *. Your application now holds privileges it should never have. Confirm: Diff the taskRoleArn policy against what the code actually calls; look for secretsmanager:* or s3:* on the task role. Fix: Move secrets-reading to the execution role; scope the task role to only the specific actions and ARNs the code uses.

Best practices

• Pin an immutable image digest, never :latest. ECS resolves a tag at each task launch, so a moving tag means two tasks in one deployment run different code. Pass the digest from CI.
• Size the task for the whole task (app + sidecars) and pick a valid CPU/memory combination; cap individual containers only when a sidecar needs bounding.
• Default to ARM64 (Graviton) for compatible images — ~20% cheaper for the same size, usually equal or better performance.
• Spread tasks across ≥2 private subnets in different AZs with IP headroom for maximumPercent. Plan subnets for the deploy surge, not steady state.
• Disable public IPs; reach AWS via VPC endpoints (ECR api/dkr, secretsmanager, logs, sts interface + S3 gateway) — cheaper than NAT per-GB and keeps pulls on the AWS network.
• Scale on a metric that leads load — ALBRequestCountPerTarget for web, queue depth for workers, CPU/memory only when work maps to them. Layer a CPU backstop; avoid aggressive scale-in policies that flap.
• Always enable the deployment circuit breaker with rollback: true and a healthCheckGracePeriodSeconds above your cold-start time. Test it once so you’ve seen it fire.
• Make shutdown graceful as a triad: PID 1 receives SIGTERM (exec form or initProcessEnabled), the app drains inside stopTimeout, and the ALB deregistration delay (≈30s) covers the drain but no longer.
• Register targets by IP (target_type = ip) — Fargate tasks are ENIs, not instances.
• Keep the execution role and task role separate and minimal, each scoped to specific ARNs; inject secrets via secrets, never environment.
• Turn on Container Insights, emit structured JSON logs (non-blocking), and wire ADOT/X-Ray tracing from day one — diagnosis is a lookup, not an archaeology dig.
• Use Fargate Spot via a capacity-provider strategy (with an on-demand base) only for stateless services that drain on SIGTERM; right-size with Compute Optimizer and re-measure.

Security notes

• Two-role least privilege. The execution role pulls images, writes logs, and reads secrets before start; the task role is what your code uses at runtime. Scope each to specific ARNs — never * — and never put secrets-reading on the task role.
• Secrets out of plaintext. Inject via the secrets block (Secrets Manager or SSM SecureString), encrypted with a CMK where it matters; environment is visible in describe-tasks and the API, so it’s for non-secret config only.
• Network isolation by default. Tasks run in private subnets with assignPublicIp: DISABLED, security groups that accept only the ALB’s SG on the container port, and least-privilege egress (DB SG, endpoint SG) — reference SGs by ID, never CIDR.
• Private AWS access. VPC interface/gateway endpoints keep ECR pulls, secret reads, and log writes on the AWS backbone instead of traversing a NAT to the public internet.
• Harden the container. Run as a non-root user, set readonlyRootFilesystem: true (write only to mounts/tmpfs), and scan images in ECR; pin digests so a tampered or moved tag can’t slip in.
• Lock down ECS Exec. If you enable ECS Exec for debugging, gate it with IAM and log sessions; it’s a shell into a running task and should not be broadly granted.
• Front with a WAF where it’s internet-facing. Put the ALB behind AWS WAF and restrict the ALB SG to your CDN/edge ranges so tasks are never directly reachable.

The security controls that also prevent these incidents — secure and resilient pull the same direction:

Cost & sizing

The bill drivers and how they interact with the fixes:

• vCPU-seconds and GB-seconds dominate. Fargate bills per vCPU-second and GB-second from image pull to task stop, per running task. An oversized task definition (4 vCPU peaking at 0.8) costs you on every replica, every hour — right-sizing is often the biggest single saving.
• ARM64 is ~20% off the same size for one flag, on compatible images — the cheapest change you can make.
• Fargate Spot cuts the elastic portion up to ~70% but reclaims with a ~2-minute warning; only for stateless, SIGTERM-clean services, with an on-demand base for the floor.
• Networking adds up quietly. A NAT gateway charges per-GB on every ECR pull; VPC endpoints (interface hourly + per-GB, S3 gateway free) are usually cheaper at scale and keep traffic on AWS.
• Observability is per-GB / per-metric. Container Insights enhanced and CloudWatch ingest are worth it, but sample high-volume logs/traces so a traffic spike doesn’t spike the telemetry bill.

A rough monthly picture for a small production API (steady ~6 tasks, bursting to ~12), us-east-1, indicative — confirm against the live pricing page:

What exactly Fargate meters, so you know which knob each line item responds to:

Right-sizing workflow: read Container Insights / Compute Optimizer’s ECS recommendations, find tasks over-provisioned versus their peak, resize the task definition to the next valid combo down, redeploy, and re-measure after a full traffic cycle. Lumio’s post-incident bill dropped once they right-sized back down after fixing the scaling metric — the fix is usually configuration, not a bigger task.

Interview & exam questions

1. Why must a Fargate ALB target group use target_type = ip? Because every Fargate task runs awsvpc networking and has its own ENI and private IP — there’s no shared EC2 instance to register. instance target type registers EC2 instance IDs, which don’t exist on Fargate; ip registers each task’s private IP directly. Mapping to the SAA/DVA container objectives.

2. A service throws 502s on every deploy and scale-in but is fine at steady state. What’s the cause? The ALB target group isn’t draining gracefully — typically the default 300-second deregistration delay keeps routing new connections to tasks ECS is stopping, and/or PID 1 swallows SIGTERM so the app is SIGKILLed mid-request. Fix: deregistration_delay≈30 aligned under stopTimeout, and a real SIGTERM handler with the app as PID 1.

3. What does the deployment circuit breaker do, and what happens without it? It watches for a run of failed task launches during a deploy and, with rollback: true, automatically reverts to the last known-good task definition. Without it, a bad image that never passes health checks leaves ECS replacing failing tasks indefinitely, draining the subnet IP pool and paging on-call. It scales its failure threshold with service size.

4. Difference between the execution role and the task role? The execution role is assumed by the ECS agent before the container starts — to pull the image from ECR, write to the log group, and resolve secrets. The task role is assumed by your application code at runtime to call AWS APIs (S3, DynamoDB). Secrets-reading belongs to the execution role; the task role carries only runtime permissions. Conflating them is the classic ECS IAM mistake.

5. Which scaling metric should a web API behind an ALB use, and why not CPU? ALBRequestCountPerTarget — it scales on actual per-task load and reacts before CPU saturates. CPU target tracking lags for I/O-bound services because CPU stays low while request queues grow, so scaling reacts after latency has already spiked. Keep a CPU policy as a backstop, not the primary.

6. How do you plan subnet sizing for a Fargate deploy? Each task consumes one subnet IP via its ENI, and during a rolling deploy you run up to desired × (maximumPercent/100) tasks. For a 40-task service at maximumPercent: 200, plan for ~80 IPs during the deploy, plus AWS’s 5 reserved addresses per subnet and anything else in those subnets — so a /24 or larger across ≥2 AZs.

7. A task is stuck in PROVISIONING with stopped reason RESOURCE:ENI. What happened? The subnet ran out of free IPs/ENIs during the deploy surge — the task can’t get an ENI. Confirm by comparing free IPs to desired × maximumPercent. Fix by using larger subnets (/24+) across more AZs, or temporarily lowering maximumPercent to shrink the surge.

8. Why pin an image digest instead of a tag? ECS resolves the image reference at each task launch. A moving tag like :latest means two tasks in the same deployment can pull different code, producing nondeterministic behavior that’s brutal to debug. A @sha256: digest is immutable, so every task in a deployment runs identical bits.

9. How does graceful shutdown work on Fargate, and what are the three timers? On stop, ECS sends SIGTERM to PID 1, waits stopTimeout (default 30s, max 120s), then SIGKILL — while the ALB deregisters the task and waits its deregistration delay for in-flight connections. The three timers must nest: deregistration delay ≥ app drain grace ≤ stopTimeout. PID 1 must actually receive SIGTERM (exec form or initProcessEnabled).

10. When would you choose blue/green over rolling deployments? Rolling with a circuit breaker is the right default for most services. Reach for blue/green (native ECS or CodeDeploy) when you need a full parallel environment with instant cutover and rollback, or canary/linear traffic shifting — high-stakes changes where you want to validate the green environment before sending it real traffic. The cost is running a full second environment during the shift.

11. How does Fargate Spot save money, and what’s the precondition? It runs the same tasks at up to ~70% off but can reclaim them with a ~2-minute SIGTERM warning. The precondition is that the service is stateless and drains cleanly on SIGTERM — Spot reclamation uses the same graceful-stop path. Use a capacity-provider strategy with an on-demand base for the guaranteed floor and Spot for the elastic remainder.

12. Your scaling policy seems to do nothing. What’s the most likely silent cause? A malformed ResourceLabel on an ALBRequestCountPerTarget policy. It must be <ALB full name>/<target group full name> — the portions after loadbalancer/ and targetgroup/ in the ARNs. Get it wrong and Application Auto Scaling can’t read the metric, so the policy silently never acts.

These map to AWS Certified Solutions Architect – Associate (SAA-C03) and Developer – Associate (DVA-C02) for ECS/Fargate, task definitions, IAM roles, and deployments; the networking depth (awsvpc, endpoints, SGs) touches Advanced Networking – Specialty (ANS-C01). A compact cert-mapping for revision:

Quick check

1. A Fargate service throws 502s on every deploy and scale-in but is healthy at steady state. Name the two most likely causes and the one target-group setting you check first.
2. Your container starts via sh -c "node server.js". Why might in-flight requests be dropped on every task stop, and what are two fixes?
3. True or false: scaling out to more tasks fixes a service whose tasks are getting OOM-killed.
4. A web API behind an ALB is scaling late under load even though CPU stays at 40%. What metric should it scale on instead, and why?
5. A bad image is deployed and ECS keeps replacing failing tasks until the subnet runs out of IPs. What one feature would have prevented this, and how do you turn it on?

Answers

1. Cause A: the ALB target group’s deregistration delay is the default 300s, longer than ECS’s stop sequence, so the ALB keeps sending new connections to a stopping task. Cause B: PID 1 swallows SIGTERM so the app is SIGKILLed mid-request. First setting to check: deregistration_delay (set it to ≈30s and align it under stopTimeout). Also confirm target_type = ip.
2. The shell is PID 1 and may not forward SIGTERM, so the app never gets the signal and is SIGKILLed after stopTimeout with requests in flight. Fixes: run the app as PID 1 via the exec form (CMD ["node","server.js"]), or set "initProcessEnabled": true in linuxParameters to get a signal-forwarding init — plus implement a SIGTERM handler that drains.
3. False. OOM is against the per-task memory cap; every scaled-out task hits the same ceiling and OOMs. Fix by raising the task memory to a valid CPU/memory combination (scale up) and/or fixing the leak — scaling out doesn’t change the per-task limit.
4. ALBRequestCountPerTarget. It scales on actual per-task request load and reacts before CPU saturates; CPU target tracking lags for I/O-bound work because CPU stays low while request queues grow. Keep CPU as a backstop policy.
5. The deployment circuit breaker with rollback: true. Enable it via update-service --deployment-configuration '{"deploymentCircuitBreaker":{"enable":true,"rollback":true}}' (and set a healthCheckGracePeriodSeconds); it auto-detects the failing deploy and reverts to the last known-good revision.

Glossary

• AWS Fargate — the serverless launch type for ECS (and EKS); you run containers without provisioning or managing EC2 hosts, billed per vCPU-second and GB-second.
• Task definition — the immutable, versioned (family:revision) blueprint of containers, CPU/memory, network mode, and the two IAM roles.
• Task — one running instance of a task definition; on Fargate it gets its own ENI and private IP.
• Service — the controller that keeps a desired number of tasks running, registered behind a load balancer, and owns deployments and scaling.
• awsvpc network mode — the Fargate-mandatory mode giving each task its own ENI, IP, and security group(s).
• ENI (elastic network interface) — the virtual NIC attached to each task; finite per subnet, which constrains the deploy surge.
• Execution role — the IAM role the ECS agent assumes before the container starts (ECR pull, log writes, resolving secrets).
• Task role — the IAM role your application code assumes at runtime to call AWS APIs.
• minimumHealthyPercent / maximumPercent — the floor of healthy tasks ECS keeps and the ceiling it may temporarily exceed during a deploy.
• Deployment circuit breaker — the deploy safeguard that detects a run of failed task launches and (with rollback) reverts to the last known-good revision.
• stopTimeout — the grace period (default 30s, max 120s on Fargate) between SIGTERM and SIGKILL when a container is stopped.
• Deregistration delay — the time the ALB waits for in-flight connections to finish before fully removing a target (default 300s; set ≈30s for fast services).
• Application Auto Scaling — the service that adjusts a scalable target’s desiredCount based on a CloudWatch metric via target-tracking or step policies.
• ALBRequestCountPerTarget — the predefined target-tracking metric that scales on requests-per-task; the leading signal for ALB-fronted web services.
• ResourceLabel — the <ALB full name>/<target group full name> string a request-count policy needs; if malformed, the policy silently does nothing.
• Capacity provider — FARGATE (on-demand) or FARGATE_SPOT; a strategy with base/weight mixes them for cost.
• Fargate Spot — discounted capacity (up to ~70% off) that can be reclaimed with a ~2-minute SIGTERM warning; for stateless, drain-clean services only.
• VPC endpoint — an interface (ECR, Secrets Manager, Logs, STS) or gateway (S3, DynamoDB) endpoint that keeps AWS API traffic on the backbone instead of via NAT.
• PID 1 / init — the entrypoint process that must receive SIGTERM to shut down gracefully; initProcessEnabled adds a signal-forwarding, zombie-reaping init.
• Container Insights — the CloudWatch feature giving per-task/service (and, in enhanced mode, per-container) metrics and dashboards.
• Platform version — the versioned Fargate runtime (agent + kernel + behavior); LATEST currently resolves to Linux 1.4.0. Pinning freezes behavior; LATEST gets fixes but can retire tasks to patch under you.
• Ephemeral storage — the per-task scratch disk (20 GiB free, up to 200 GiB), encrypted at rest by default, created at task start and destroyed at stop.
• Task retirement — an AWS-initiated stop-and-replace of a running task to patch the platform underneath it; uses the same SIGTERM/stopTimeout path as a deploy.
• awsvpcTrunking — an account setting that raises ENI density on EC2 container instances; irrelevant on Fargate, where AWS manages the task ENI.
• Scheduled action — an Application Auto Scaling entry that changes a scalable target’s min/max on a cron/at schedule for predictable load; complements (does not replace) target tracking.
• Step scaling — a scaling policy that adds or removes a defined amount when a CloudWatch alarm crosses graduated thresholds; for asymmetric or queue-driven bursts.
• Service Connect — an ECS feature giving service-to-service (east-west) discovery, client-side retries, and per-call metrics via a managed Envoy sidecar and a Cloud Map namespace.
• Cloud Map — AWS service-discovery that registers DNS records for services; the namespace layer Service Connect builds on.
• Health-check grace period — the seconds after a task starts during which ECS ignores ALB health-check failures, so a slow-booting app is not killed before it is ready.
• FireLens — the ECS log-routing integration (a Fluent Bit / Fluentd sidecar) that parses, filters, and fans out container logs to multiple destinations.
• ADOT — AWS Distro for OpenTelemetry, the collector sidecar that exports traces/metrics (e.g. to X-Ray) once the app is instrumented with OpenTelemetry.
• initProcessEnabled — a linuxParameters flag that runs a tiny init as PID 1 to forward signals (like SIGTERM) and reap zombie processes.
• Image digest — the immutable @sha256: content hash of an image; pinning it guarantees every task in a deployment runs identical bits, unlike a moving tag.

Next steps

You can now wire a production Fargate service: correctly sized, isolated per-task, scaled on the right signal, deployed with a tested safety net, and shut down without dropping a request. Build outward:

• Next: Elastic Load Balancing Deep Dive: ALB, NLB & GWLB — the target groups, health checks, and listeners that are half of every Fargate deploy.
• Related: VPC Deep Dive: Subnets, Routing, IGW, NAT & Endpoints — design the subnets and endpoints your per-task ENIs live in.
• Related: ECS Service Connect vs Load Balancers: Discovery & Resilience — service-to-service discovery once you have more than one Fargate service.
• Related: Graviton/ARM64 Migration: Multi-Arch Builds & Benchmarking — make the ~20% ARM64 saving real with multi-arch images.
• Related: Secrets Manager & Parameter Store Deep Dive — inject and rotate the secrets your execution role resolves at launch.
• Related: AWS X-Ray: Service Map, Segments & ADOT Tracing — add the distributed tracing sidecar referenced in the observability section.