Private-Subnet Internet Egress: NAT Gateway vs NAT Instance, Hands-On & Cost

Quick take: A private subnet is one whose route table has no door to the internet. But the servers in it still need to reach out — to pull OS patches, hit a package repo, call an AWS API, phone a payment gateway. A NAT gateway is the managed component that lets them initiate outbound IPv4 connections while remaining completely unreachable from the internet. It lives in a public subnet, wears an Elastic IP, and the private route table points 0.0.0.0/0 at it. It is also one of the most reliably surprising line items on an AWS bill, because it charges both by the hour and per gigabyte it processes — including traffic to S3 that had no business going through it. Learn the mechanism, the limits, and the two cost traps, and NAT stops being either a mystery or a monthly shock.

A team ships a fleet of workers into a private subnet — correctly, no public IPs, nothing reachable from outside. Deploy succeeds. Two hours later the workers are all failing health checks because yum update and pip install time out: the subnet has no path to the internet at all. Someone adds a NAT gateway, egress works, everyone moves on. Three weeks later the finance channel lights up: the NAT gateway processed 8 TB that month — mostly nightly backups and log shipping to S3, in the same region — and the “add a NAT gateway” fix quietly added a few hundred dollars. Nothing here was exotic. It was two ordinary gaps: not knowing that a private subnet needs an explicit egress path, and not knowing that a NAT gateway meters every byte, so the fix for a free problem (S3 is reachable privately for nothing) was routed through a paid one.

This article is the complete, practitioner-grade treatment of private-subnet egress on AWS. We define why a private subnet needs managed outbound at all, then take the NAT gateway apart attribute by attribute — the Elastic IP, the public-subnet placement, the 0.0.0.0/0 route, the per-AZ high-availability pattern versus a single shared NAT and its cross-AZ charge and single-point-of-failure trade-off, the 55,000-connections-per-destination ceiling, and the bandwidth that auto-scales from 5 up to 100 Gbps. We contrast it with the self-managed NAT instance (cheaper at low scale, but you disable its source/destination check, you patch it, and it is a single point of failure) in a full side-by-side. We cover the egress-only internet gateway for IPv6, because IPv6 has no NAT. And we go hard at the cost reality — the hourly-plus-per-GB shape that produces the classic surprise bill — and show exactly how a free S3/DynamoDB gateway endpoint removes that traffic from the NAT entirely.

By the end you can build a correct, highly-available egress topology from scratch with aws CLI and Terraform, get a shell on a truly private instance through SSM Session Manager with no bastion and no open port, prove that S3 traffic bypasses the NAT once a gateway endpoint exists, read the CloudWatch metrics that tell you a NAT is close to port exhaustion, and diagnose the six failure modes that put a private instance offline or a NAT on the front page of your bill. This maps directly to the networking domains of SAA-C03, SOA-C02, and the ANS-C01 advanced-networking specialty.

What problem this solves

Every non-trivial workload needs to reach out even when nothing should be allowed to reach in. An application server pulls security patches. A container host pulls images from a registry. A batch job downloads input from a partner API. A Lambda in a VPC calls Secrets Manager. A monitoring agent ships metrics to a SaaS endpoint. None of these require the instance to be reachable from the internet — they are all outbound-initiated — but all of them fail silently if the private subnet has no egress path. The instinct to “just give it a public IP” is exactly wrong: that makes it reachable, which is the thing you were trying to avoid. What you want is asymmetry — outbound yes, inbound no — and that asymmetry is precisely what a NAT device provides.

What breaks without a correct egress design is a spectrum, from “nothing works” to “everything works but the bill is wrong.” At one end, a private instance with no route to a NAT can’t resolve a yum mirror and every package operation hangs for the full TCP timeout. In the middle, a NAT placed in one AZ becomes a single point of failure whose loss takes down egress for every AZ routed through it, and silently bills cross-AZ data transfer the whole time it’s working. At the other end, everything functions perfectly and you discover months later that terabytes of in-region S3 traffic — backups, logs, data-lake writes — have been flowing through the NAT’s per-GB meter when a free gateway endpoint would have carried them for nothing.

Who hits this: essentially every team that runs private workloads, which is every team doing AWS properly. It bites hardest on three groups. First, people new to VPC design who don’t yet know that “private” is a property of the route table, not the subnet, so they can’t explain why their instance is offline. Second, teams optimizing for availability who put one NAT in one AZ and don’t realize they’ve built both a SPOF and a cross-AZ tax. Third, teams optimizing for nothing in particular whose S3 and DynamoDB traffic quietly runs up the NAT bill. The mechanism is small — one managed box, one route, one Elastic IP — but the failure modes around it are the ones that generate 2 a.m. pages and end-of-month arguments.

Here is the whole field of private-subnet egress options in one table — the thing you’re choosing between before we go deep on the two that matter most:

Learning objectives

By the end of this article you can:

• Explain why a private subnet needs a managed egress path, and why giving an instance a public IP is the wrong way to provide it.
• Build a NAT gateway correctly — Elastic IP, public-subnet placement, the private route table’s 0.0.0.0/0 → nat entry — with both aws CLI and Terraform.
• Design the per-AZ high-availability pattern and articulate the exact trade-off against a single shared NAT: cross-AZ data-transfer charges plus a multi-AZ single point of failure.
• Enumerate every NAT gateway limit that matters in production — the 5→100 Gbps auto-scaling bandwidth, the 55,000 connections per unique destination ceiling, the 350-second idle timeout — and read the CloudWatch metrics (ErrorPortAllocation, ActiveConnectionCount) that warn you before they bite.
• Stand up and reason about a NAT instance: disable its source/destination check, size it, attach a security group for egress filtering, and understand why it’s a single point of failure you have to babysit.
• Use an egress-only internet gateway for IPv6 egress and explain why NAT doesn’t exist for IPv6.
• Model the cost of a NAT gateway (hourly + per-GB + the public-IPv4 charge + cross-AZ transfer) and cut it hard with S3/DynamoDB gateway endpoints.
• Get a shell on a truly private instance through SSM Session Manager with no bastion, and diagnose the six canonical egress failures from route table to port exhaustion.

Prerequisites & where this fits

You should be comfortable with VPC fundamentals: that a VPC is a region-scoped private network, that subnets are per-AZ slices of its CIDR, that a route table decides where a subnet’s traffic can go, and that an internet gateway (IGW) is the VPC’s only door to the public internet. You should know that a security group is a stateful, allow-only firewall on an instance’s network interface, and that 0.0.0.0/0 means “every address.” You need an AWS account, the AWS CLI v2 installed and configured (aws configure), and enough comfort with JSON to read describe-* output. Terraform is used throughout; you don’t need to be an expert, but you should recognize a resource block. If any of that is shaky, start with the ground-up build in AWS VPC from Scratch: Subnets, Route Tables and Internet Gateway and the firewall model in AWS VPC, Subnets and Security Groups Explained.

Concretely, here is the assumed knowledge, why each piece matters for egress, and where to brush up:

This topic sits one layer above raw VPC construction and one layer below multi-VPC connectivity. Egress is the piece that makes a private subnet actually usable — without it, “private” just means “broken.” Once you’re comfortable here, the natural next moves are reaching AWS services without any internet path at all, covered in AWS PrivateLink and VPC Endpoints: Interface vs Gateway, and centralizing egress across many VPCs through a hub, in AWS Transit Gateway Hub-and-Spoke, Hands-On. The reason those two matter for cost and security is exactly the reason this article exists: a NAT gateway per spoke VPC is expensive and hard to inspect, so mature networks either centralize NAT behind a transit gateway or eliminate the traffic with endpoints.

Core concepts

The whole subject rests on one idea: network address translation (NAT) lets many private addresses share one public address for outbound connections, and because the translation device tracks each flow, only replies to connections the private side started can come back. Nothing outside can initiate. That is the asymmetry a private subnet wants. Everything else — gateway versus instance, one AZ versus per-AZ, the cost, the endpoints — is detail hung on that single mechanism.

Here is the vocabulary you need, defined once, precisely:

And here are the route-table targets relevant to egress specifically — the entries you’ll actually add, what direction they carry, and what they cost:

A NAT gateway is routinely confused with the other boxes that live near the VPC edge. Getting the distinctions straight is half the battle, because the wrong mental model leads to the wrong route:

The mental model to lock in: a NAT device is a thing in a public subnet that private subnets route to. It straddles the boundary. Its own default route goes to the IGW (that’s how it reaches the internet); the private subnets’ default route goes to it (that’s how they reach the NAT). Get those two routes backwards and nothing works, so we’ll come back to them repeatedly.

Why private subnets need egress at all

It’s worth being concrete about what reaches out, because it explains both why you can’t skip egress and where the per-GB bill comes from. Every item below is an outbound-initiated flow — the instance starts it, the internet only replies — which is exactly what NAT is built to allow and exactly what an inbound-blocking design must still permit.

Two rows there are quiet cost lessons. Object storage I/O and AWS API calls are the traffic that most often blows up a NAT bill, and both can be moved onto free or cheap endpoints — the whole reason the cost section later leans so hard on endpoints. And the last two rows are a common red herring: NTP and the instance-metadata service (IMDS) live on link-local addresses (169.254.x.x) that never traverse the NAT or the internet, so “my instance can reach IMDS but not the internet” is a normal, expected state, not a working egress path.

NAT gateway, attribute by attribute

A NAT gateway is a fully-managed, horizontally-scaled, redundant-within-an-AZ NAT appliance. You don’t size it, patch it, or scale it — AWS does. You give it four things and it works. Here is exactly what it needs and why:

The single most important structural fact — the one that trips up almost everyone once — is the two-route rule. The NAT gateway sits in a public subnet so that its default route reaches the IGW, and the private subnets point their default route at the NAT. Two different route tables, two different default routes, pointing in what feels like opposite directions:

# The NAT lives in a PUBLIC subnet — that subnet's route table points 0/0 at the IGW
aws ec2 describe-route-tables --route-table-id rtb-public \
  --query "RouteTables[].Routes[?DestinationCidrBlock=='0.0.0.0/0']"
# → GatewayId: igw-0abc...

# The workloads live in a PRIVATE subnet — that table points 0/0 at the NAT
aws ec2 describe-route-tables --route-table-id rtb-private-a \
  --query "RouteTables[].Routes[?DestinationCidrBlock=='0.0.0.0/0']"
# → NatGatewayId: nat-0def...

Connectivity type: public vs private

A NAT gateway has a connectivity type chosen at creation and immutable afterwards. Most people only ever use public. The private type exists for a specific case: giving overlapping or many VPCs a way to talk to each other or to on-prem through NAT (to avoid IP conflicts) without any internet access.

Bandwidth, connections and timeouts — the real limits

You never configure a NAT gateway’s throughput, but you must know its ceilings because they define when you need a second one or a redesign:

The 55,000-connections-per-destination number is the one that surfaces as a real incident. It is per unique destination tuple, so a fleet making millions of connections to many endpoints is fine, but a fleet hammering one endpoint (a single API IP:port) can exhaust source ports to that destination and start failing. The tell is the CloudWatch ErrorPortAllocation metric going non-zero. The fixes: spread across more destination IPs (many APIs sit behind multiple addresses), add secondary EIPs to the NAT for more source ports, or add NAT gateways and split the fleet.

High availability: one NAT per AZ, and why

A NAT gateway is AZ-scoped. It’s redundant inside its AZ (AWS runs it across hardware there), but if the entire AZ has a problem, that NAT is gone. This creates the single most consequential design decision in the whole topic: do you run one NAT per AZ, or one shared NAT for the whole VPC?

The shared-NAT option looks like it saves two-thirds of the hourly cost, and for a dev VPC that’s a fine trade. But in production it’s a trap for two compounding reasons. First, availability: if the NAT’s AZ has an issue, the instances in your other, healthy AZs lose all egress — you’ve coupled three AZs’ fate to one. Second, cost: every byte an instance in AZ-b or AZ-c sends through a NAT in AZ-a crosses an AZ boundary, and AWS bills cross-AZ data transfer (~$0.01/GB each direction) on top of the NAT’s own per-GB processing. At any real volume the cross-AZ charge erases the hourly saving and then some. The correct production pattern is one NAT gateway per AZ, and each AZ’s private route table points at its own NAT — which is exactly what the Terraform below does with a for_each over AZs.

Here’s the whole per-AZ build. Note how each private route table gets a default route to the NAT in its own AZ:

# One NAT gateway per AZ, each in that AZ's public subnet, each with its own EIP.
locals { azs = ["a", "b", "c"] }

resource "aws_eip" "nat" {
  for_each = toset(local.azs)
  domain   = "vpc"
  tags     = { Name = "nat-eip-${each.key}" }
}

resource "aws_nat_gateway" "this" {
  for_each          = toset(local.azs)
  allocation_id     = aws_eip.nat[each.key].id
  subnet_id         = aws_subnet.public[each.key].id   # PUBLIC subnet
  connectivity_type = "public"
  tags              = { Name = "nat-${each.key}" }
  depends_on        = [aws_internet_gateway.this]
}

# Each private subnet's route table -> the NAT in the SAME AZ (no cross-AZ traffic).
resource "aws_route_table" "private" {
  for_each = toset(local.azs)
  vpc_id   = aws_vpc.this.id
  tags     = { Name = "rt-private-${each.key}" }
}

resource "aws_route" "private_default" {
  for_each               = toset(local.azs)
  route_table_id         = aws_route_table.private[each.key].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.this[each.key].id
}

The NAT instance: cheaper, self-managed, and a SPOF

Before the NAT gateway existed, you ran a NAT instance — a plain EC2 instance that forwards traffic with Linux iptables masquerading. It still has a place: at low, non-critical scale (a dev VPC, a hobby project) it can cost a tenth of a NAT gateway, and unlike the gateway you can attach a security group to it for egress filtering. The catch is that everything the gateway does for you, you now do yourself: sizing, patching, and — the big one — availability.

The mechanics that make a NAT instance work (and the one that most often defeats a first attempt):

The source/destination check is the concept to internalize. By default, an EC2 instance drops any packet whose destination isn’t itself (an anti-spoofing safeguard). A NAT instance’s entire job is to handle packets addressed to other hosts, so you must turn that check off or it silently drops every forwarded packet — the classic “my NAT instance does nothing” symptom:

# Disable source/dest check — without this the NAT instance forwards nothing.
aws ec2 modify-instance-attribute --instance-id i-0natinst --no-source-dest-check

# Point the private subnet's default route at the NAT instance's ENI.
aws ec2 create-route --route-table-id rtb-private \
  --destination-cidr-block 0.0.0.0/0 \
  --network-interface-id eni-0natinst

resource "aws_instance" "nat" {
  ami                    = data.aws_ami.al2023_arm.id
  instance_type          = "t4g.nano"
  subnet_id              = aws_subnet.public["a"].id
  source_dest_check      = false            # THE critical line
  vpc_security_group_ids = [aws_security_group.nat.id]
  user_data              = <<-EOF
    #!/bin/bash
    sysctl -w net.ipv4.ip_forward=1
    iptables -t nat -A POSTROUTING -o $(ip route show default | awk '{print $5}') -j MASQUERADE
  EOF
  tags = { Name = "nat-instance" }
}

resource "aws_route" "private_via_nat_instance" {
  route_table_id         = aws_route_table.private["a"].id
  destination_cidr_block = "0.0.0.0/0"
  network_interface_id   = aws_instance.nat.primary_network_interface_id
}

Instance sizing matters because, unlike the gateway, a NAT instance’s throughput is bounded by its instance type’s network performance — and NAT is CPU-bound on packet processing, not just bandwidth:

NAT gateway vs NAT instance — the full comparison

This is the decision table to keep. Row by row, what you’re actually trading:

The rule of thumb: use a NAT gateway unless you have a specific, budget-driven reason not to. The data-processing charge is the gateway’s only real weakness, and it’s usually cheaper to attack that with endpoints (below) than to take on the operational burden of a self-managed NAT. The NAT instance earns its place in exactly two situations — a genuinely cost-sensitive dev environment with trivial egress, or a case where you need a security group on the egress path (the gateway can’t do that, so some teams run a NAT instance or a proper firewall appliance for outbound filtering).

IPv6 egress: the egress-only internet gateway

There is no such thing as a NAT gateway for IPv6, and the reason is fundamental: IPv6 addresses are globally routable. There’s no address shortage, so there’s no address translation — every IPv6 instance already has a public, internet-routable address. But “publicly addressed” would mean “reachable from the internet,” which breaks the private-subnet promise. To keep IPv6 instances outbound-only, AWS provides the egress-only internet gateway (EIGW): a stateful gateway that allows outbound IPv6 and its return traffic while blocking any inbound-initiated connection. It is the IPv6 analogue of a NAT’s asymmetry, minus the address translation.

# IPv6 outbound-only in three commands: EIGW + a ::/0 route.
aws ec2 create-egress-only-internet-gateway --vpc-id vpc-0abc \
  --query 'EgressOnlyInternetGateway.EgressOnlyInternetGatewayId' --output text
# → eigw-0123456789abcdef0
aws ec2 create-route --route-table-id rtb-private \
  --destination-ipv6-cidr-block ::/0 \
  --egress-only-internet-gateway-id eigw-0123456789abcdef0

resource "aws_egress_only_internet_gateway" "this" {
  vpc_id = aws_vpc.this.id
}
resource "aws_route" "v6_egress" {
  route_table_id              = aws_route_table.private["a"].id
  destination_ipv6_cidr_block = "::/0"
  egress_only_gateway_id      = aws_egress_only_internet_gateway.this.id
}

A dual-stack private subnet therefore carries two default routes: 0.0.0.0/0 → nat-… for IPv4 and ::/0 → eigw-… for IPv6. Forget the second and your IPv6 traffic silently has no path even though IPv4 works — a confusing “half the internet is down” symptom on dual-stack hosts.

The cost reality — and how to kill it

The NAT gateway’s pricing shape is what makes it infamous. It’s not one charge, it’s a stack of them, and the per-GB component is the one nobody sees coming because it applies to traffic you didn’t think of as “internet” — like in-region S3.

Prices vary by region — Mumbai (ap-south-1) runs a touch higher (roughly $0.056/hr and $0.056/GB) — so treat these as the shape, not the exact figure, and confirm on the pricing page. What matters is the shape: a fixed hourly floor per NAT per AZ, plus a variable per-GB that scales with everything your private fleet sends out. Model a few volumes and the surprise becomes obvious:

Look at the last two rows. The only change is attaching a free S3 gateway endpoint so the 7 TB of backup/log traffic stops flowing through the NAT — and the bill drops by ~$315/month. That is the single highest-leverage NAT optimization, and it costs nothing to implement:

The gateway-versus-interface distinction matters for the math: gateway endpoints (S3, DynamoDB only) are free, so they are pure savings with no downside. Interface endpoints cost ~$0.01/hr per AZ plus ~$0.01/GB, so they’re worth it when the traffic they remove from the NAT is large or when you want the AWS API path to be fully private — but at trivial volume a handful of interface endpoints across three AZs can cost more than the NAT traffic they save, so measure first. The deep treatment of that trade-off is in AWS PrivateLink and VPC Endpoints: Interface vs Gateway.

Architecture at a glance

The diagram below is the whole system, read left to right. A private EC2 instance in AZ-a has no public IP and no inbound rule, so nothing on the internet can reach it — yet you still get a shell via SSM (badge 1). Its private route table sends 0.0.0.0/0 to the NAT gateway (badge 2); miss or mispoint that route and the instance is offline. The NAT gateway lives in a public subnet, wears an Elastic IP, and meters every GB it processes (badge 3) — one NAT per AZ is the HA unit. The NAT reaches the internet only because its public subnet routes 0.0.0.0/0 to the internet gateway (badge 4), the VPC’s single free door. Out on the internet, the ~55,000-connections-per-destination ceiling is where a hot endpoint causes port exhaustion (badge 5). And the S3 gateway endpoint (badge 6) is the free bypass: a prefix-list route peels S3 traffic off the NAT entirely, so it never hits the per-GB meter.

The one-line summary of which route lives where — the thing to verify first in any egress incident:

Real-world scenario

Meridian Ledger, a fintech running a reconciliation platform in ap-south-1 (Mumbai), moved from a single flat VPC to a proper three-tier design ahead of an audit. The app tier — 40 c6i.large workers — went into private subnets across three AZs. To get them online fast, an engineer created one NAT gateway in AZ-a and pointed all three private route tables at it. Everything worked in testing, the audit passed on the network-isolation criteria, and it shipped.

Two problems surfaced over the next six weeks. First, the bill. The workers wrote reconciliation outputs and gzipped audit logs to S3 continuously — about 9 TB/month — and every byte flowed through the NAT’s per-GB meter plus a cross-AZ charge for the two-thirds of workers not in AZ-a. The NAT line item came to roughly ₹42,000/month, most of it S3 traffic that should have been free and cross-AZ traffic that shouldn’t have existed. Second, the outage. During an AZ-a networking event, the workers in AZ-b and AZ-c — perfectly healthy, in unaffected AZs — all lost egress simultaneously because their only path out ran through the dead AZ’s NAT. Reconciliation stalled for 25 minutes until someone spun up NATs in the other AZs by hand.

The fix was two changes, both from this playbook. They deployed one NAT gateway per AZ and repointed each private route table at its local NAT, which eliminated the cross-AZ transfer entirely and contained any future AZ failure to that one AZ. Then they attached a free S3 gateway endpoint and added the pl-S3 → vpce route to all three private tables, which lifted the entire 9 TB of S3 traffic off the NAT. The before/after told the story:

Tripling the NAT count cut the bill by three-quarters, because the per-GB and cross-AZ charges dwarfed the extra hourly cost — and they got real HA as a bonus. The lesson the team wrote into their standards doc: “one NAT per AZ is not more expensive, it’s less; and an S3 gateway endpoint is not optional.”

Advantages and disadvantages

The NAT gateway trade-off in one view:

When each side matters: the advantages dominate for anything production — the operational simplicity and hands-off HA are worth real money, and the per-GB weakness is fixable with endpoints. The disadvantages dominate only at the extremes: a hobby/dev VPC where $35/month is meaningful (use a NAT instance), or an egress-filtering requirement where you need a security group or deep-packet inspection on outbound (use a NAT instance or a firewall appliance behind a transit gateway, per AWS Transit Gateway Hub-and-Spoke, Hands-On).

Hands-on lab

You’ll build a private subnet with real egress, prove an instance with no public IP and no inbound rule can still reach the internet outbound and be administered by you via SSM Session Manager (no bastion, no SSH key), then attach an S3 gateway endpoint and confirm S3 traffic bypasses the NAT. Everything is aws CLI; a Terraform equivalent follows. Region is us-east-1; adjust as needed.

⚠️ Cost warning: A NAT gateway bills ~$0.045/hour + ~$0.045/GB and the Elastic IP bills ~$0.005/hour the entire time it exists. This lab is not free-tier. Expect a few cents per hour; the EC2 instance and endpoints are cheap or free. Do the teardown at the end — a forgotten NAT gateway + EIP is ~$36/month.

Step 1 — VPC, subnets, IGW

VPC=$(aws ec2 create-vpc --cidr-block 10.20.0.0/16 \
  --query Vpc.VpcId --output text)
aws ec2 create-tags --resources $VPC --tags Key=Name,Value=nat-lab

PUB=$(aws ec2 create-subnet --vpc-id $VPC --cidr-block 10.20.1.0/24 \
  --availability-zone us-east-1a --query Subnet.SubnetId --output text)
PRIV=$(aws ec2 create-subnet --vpc-id $VPC --cidr-block 10.20.11.0/24 \
  --availability-zone us-east-1a --query Subnet.SubnetId --output text)

IGW=$(aws ec2 create-internet-gateway --query InternetGateway.InternetGatewayId --output text)
aws ec2 attach-internet-gateway --vpc-id $VPC --internet-gateway-id $IGW

Step 2 — Public route table (so the NAT can reach the internet)

RT_PUB=$(aws ec2 create-route-table --vpc-id $VPC --query RouteTable.RouteTableId --output text)
aws ec2 create-route --route-table-id $RT_PUB \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
aws ec2 associate-route-table --route-table-id $RT_PUB --subnet-id $PUB
# Expected: { "AssociationState": { "State": "associated" } }

Step 3 — Allocate an EIP and create the NAT gateway

EIP=$(aws ec2 allocate-address --domain vpc --query AllocationId --output text)
NAT=$(aws ec2 create-nat-gateway --subnet-id $PUB --allocation-id $EIP \
  --query NatGateway.NatGatewayId --output text)

# NAT creation takes ~1-2 minutes. Wait for it:
aws ec2 wait nat-gateway-available --nat-gateway-ids $NAT
echo "NAT $NAT is available"

Step 4 — Private route table pointing at the NAT

RT_PRIV=$(aws ec2 create-route-table --vpc-id $VPC --query RouteTable.RouteTableId --output text)
aws ec2 create-route --route-table-id $RT_PRIV \
  --destination-cidr-block 0.0.0.0/0 --nat-gateway-id $NAT
aws ec2 associate-route-table --route-table-id $RT_PRIV --subnet-id $PRIV

Step 5 — IAM role for SSM (so you can get a shell with no bastion)

cat > trust.json <<'JSON'
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow",
  "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" }] }
JSON
aws iam create-role --role-name nat-lab-ssm --assume-role-policy-document file://trust.json
aws iam attach-role-policy --role-name nat-lab-ssm \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam create-instance-profile --instance-profile-name nat-lab-ssm
aws iam add-role-to-instance-profile --instance-profile-name nat-lab-ssm --role-name nat-lab-ssm

Step 6 — Launch a truly private instance (no public IP, no key pair)

AMI=$(aws ssm get-parameters --names \
  /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \
  --query 'Parameters[0].Value' --output text)

INST=$(aws ec2 run-instances --image-id $AMI --instance-type t3.micro \
  --subnet-id $PRIV --no-associate-public-ip-address \
  --iam-instance-profile Name=nat-lab-ssm \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=nat-lab-private}]' \
  --query 'Instances[0].InstanceId' --output text)
aws ec2 wait instance-running --instance-ids $INST

Step 7 — Get a shell via SSM and prove outbound works

Give the SSM agent ~2 minutes to register, then:

aws ssm start-session --target $INST
# Inside the session — NO SSH, NO bastion, NO open port 22:
sh-5.2$ curl -s https://checkip.amazonaws.com     # → your NAT's Elastic IP
sh-5.2$ sudo dnf -y check-update >/dev/null && echo "egress OK"   # → egress OK
sh-5.2$ exit

The public IP curl returns is the NAT’s Elastic IP — proof that outbound is source-NAT’d through the NAT gateway, while nothing outside can start a connection back. That’s the asymmetry working.

Step 8 — Add an S3 gateway endpoint and prove S3 skips the NAT

aws ec2 create-vpc-endpoint --vpc-id $VPC \
  --service-name com.amazonaws.us-east-1.s3 \
  --vpc-endpoint-type Gateway \
  --route-table-ids $RT_PRIV
# This adds a pl-…(S3) → vpce-… route to RT_PRIV automatically.

aws ec2 describe-route-tables --route-table-id $RT_PRIV \
  --query "RouteTables[].Routes[].{Dest:DestinationPrefixListId,Target:GatewayId}"
# → an entry like { "Dest": "pl-63a5400a", "Target": "vpce-0abc..." }

Back in an SSM session, an aws s3 ls now travels the gateway endpoint over the AWS backbone, not the NAT. Confirm it on the NAT’s own meter: pull BytesOutToDestination from CloudWatch before and after an S3 copy and watch it not move for S3 traffic (while a curl to the internet does move it).

aws cloudwatch get-metric-statistics --namespace AWS/NATGateway \
  --metric-name BytesOutToDestination --dimensions Name=NatGatewayId,Value=$NAT \
  --start-time $(date -u -v-15M +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum

Step 9 — Teardown (do this, it’s costing money)

aws ec2 terminate-instances --instance-ids $INST
aws ec2 wait instance-terminated --instance-ids $INST
aws ec2 delete-vpc-endpoints --vpc-endpoint-ids $(aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=$VPC --query 'VpcEndpoints[].VpcEndpointId' --output text)
aws ec2 delete-nat-gateway --nat-gateway-id $NAT
aws ec2 wait nat-gateway-deleted --nat-gateway-ids $NAT
aws ec2 release-address --allocation-id $EIP          # ⚠️ the EIP bills until released
# delete routes/subnets/RTs, detach+delete IGW, then delete the VPC, then the IAM role/profile

Here is the whole lab as Terraform for a repeatable, one-command build and terraform destroy teardown:

resource "aws_vpc" "lab" { cidr_block = "10.20.0.0/16" }
resource "aws_internet_gateway" "lab" { vpc_id = aws_vpc.lab.id }

resource "aws_subnet" "public" {
  vpc_id = aws_vpc.lab.id
  cidr_block = "10.20.1.0/24"
  availability_zone = "us-east-1a"
}
resource "aws_subnet" "private" {
  vpc_id = aws_vpc.lab.id
  cidr_block = "10.20.11.0/24"
  availability_zone = "us-east-1a"
}

resource "aws_eip" "nat" { domain = "vpc" }
resource "aws_nat_gateway" "lab" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public.id
  depends_on    = [aws_internet_gateway.lab]
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.lab.id
  route { cidr_block = "0.0.0.0/0"  gateway_id = aws_internet_gateway.lab.id }
}
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.lab.id
  route { cidr_block = "0.0.0.0/0"  nat_gateway_id = aws_nat_gateway.lab.id }
}
resource "aws_route_table_association" "public" {
  subnet_id = aws_subnet.public.id  route_table_id = aws_route_table.public.id
}
resource "aws_route_table_association" "private" {
  subnet_id = aws_subnet.private.id  route_table_id = aws_route_table.private.id
}

# Free S3 gateway endpoint — its route makes S3 traffic bypass the NAT.
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.lab.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

Use these checkpoints to confirm each stage did what you expected before moving on — most lab failures are one of these being skipped:

Common mistakes & troubleshooting

This is the section you’ll come back to mid-incident. The playbook first — symptom, root cause, the exact command or console path to confirm, and the fix — then an error reference and a decision table.

The connection/port-exhaustion failure (rows 9–10) is the nastiest because it’s load-dependent and intermittent — it passes every test and fails in production at peak. The mechanism: a NAT gateway can hold ~55,000 simultaneous connections to each unique destination (destination IP + port + protocol). A fleet that fans out to many destinations never hits it, but a fleet that stampedes one endpoint — a single API IP, a single database proxy — runs out of source ports to that destination and new connections fail while the endpoint itself is healthy. The confirmation is unambiguous: ErrorPortAllocation is zero in normal operation and non-zero the instant you’re exhausting ports.

The CloudWatch metrics that tell you what a NAT is doing — keep these on a dashboard for any production NAT:

And the error/status reference you’ll see around NAT and its routes:

Finally, the fast decision table — from what you observe to the likely cause:

Best practices

• One NAT gateway per AZ. It’s cheaper than a shared NAT at real volume (no cross-AZ transfer) and gives you AZ-independent egress. Never let AZ-b route through AZ-a’s NAT.
• Attach S3 and DynamoDB gateway endpoints to every private route table. They’re free and they remove the single biggest source of surprise NAT charges. Do this by default, not after the bill.
• Add interface endpoints for the AWS APIs you call constantly (SSM, STS, Secrets Manager, ECR, CloudWatch) when their NAT-processed volume exceeds the endpoint’s hourly cost — and always when you want a fully private path.
• Alarm on ErrorPortAllocation ≥ 1. It’s the early-warning for port exhaustion; a non-zero value is never normal.
• Alarm on BytesOutToDestination with a monthly budget in mind — it’s the bill driver, and a spike is your first sign of a cost regression or a misrouted flow.
• Reserve a subnet as truly isolated (no NAT route) for databases and anything that must never reach the internet; give it only the gateway endpoints it needs.
• Use SSM Session Manager, not a bastion. No inbound port, no key management, full audit trail — administer private instances without ever opening port 22.
• Tag NAT gateways and EIPs so cost allocation can attribute the bill to a team/environment; NAT cost is invisible until it’s tagged.
• For dual-stack, remember the second default route (::/0 → eigw). IPv6 egress is a separate gateway.
• Prefer eliminating egress over cheapening it. The cheapest NAT byte is the one that never leaves — private mirrors, endpoints, and in-region designs beat any NAT tuning.
• For many VPCs, centralize egress behind a transit gateway and a shared NAT/inspection VPC rather than a NAT per spoke — the economics flip once you have more than a handful of VPCs.

Security notes

Egress is a security control, not just plumbing. A NAT gateway gives you outbound-only by construction — inbound stays blocked because the NAT only tracks flows your side started — but it does not filter what you reach, and it has no security group. Treat these as the security posture for private egress:

The subtlety worth stating plainly: a NAT gateway is not an egress firewall. It stops the internet from reaching in, but it happily lets a compromised instance reach out to anywhere. If you need to control where private instances can go — the real defense against exfiltration and malware call-home — you need AWS Network Firewall, a NAT instance with a restrictive security group and iptables rules, or a proxy, typically centralized behind a transit gateway. The deep treatment is in AWS Transit Gateway Hub-and-Spoke, Hands-On.

Cost & sizing

You don’t size a NAT gateway (it auto-scales), so “sizing” here means choosing the topology and killing the per-GB bill. The levers, ranked by impact:

Rough figures to anchor budgeting (us-east-1; verify current pricing): a single NAT gateway is ~$32.85/month in hourly charges plus ~$3.60/month for its public IPv4, before a byte of data — so a 3-AZ HA setup starts around ~$109/month fixed. On top of that, data processing at ~$0.045/GB means 1 TB/month adds ~$45 and 10 TB adds ~$450. The free-tier gives you nothing on NAT gateways (they’re never free), which is exactly why the endpoints matter: an S3 gateway endpoint is free forever and often the difference between a $150 and a $500 monthly bill. For an INR view at ~₹86/$, a 3-AZ NAT baseline is roughly ₹9,400/month fixed plus data — real money for a small shop, and the reason a dev environment often runs a single t4g.nano NAT instance (~₹260/month) instead.

Interview & exam questions

Q1. Why does a private-subnet instance need a NAT gateway to run yum update, and why not just give it a public IP? The instance must initiate outbound connections to package mirrors, but you don’t want it reachable from the internet. A NAT gateway provides exactly that asymmetry — outbound allowed, inbound blocked. A public IP plus an IGW route would make it internet-reachable, defeating the purpose of a private subnet. (SAA-C03, ANS-C01)

Q2. Where does a NAT gateway live, and what two routes make it work? It lives in a public subnet. That public subnet’s route table has 0.0.0.0/0 → IGW (so the NAT reaches the internet), and each private subnet’s route table has 0.0.0.0/0 → NAT (so instances reach the NAT). Two tables, two default routes pointing “opposite” ways. (SAA-C03)

Q3. You have one NAT gateway serving three AZs. What are the two problems? Availability and cost. It’s a single point of failure — if its AZ fails, all three AZs lose egress. And instances in the other two AZs incur cross-AZ data-transfer charges (~$0.01/GB each way) on top of NAT processing. Fix: one NAT per AZ, each private subnet routing to its local NAT. (SAA-C03, ANS-C01)

Q4. Your NAT gateway bill is unexpectedly high. First thing to check? Whether S3/DynamoDB or other in-region AWS traffic is flowing through the NAT’s per-GB meter. Attach free S3/DynamoDB gateway endpoints so that traffic bypasses the NAT entirely — usually the single biggest saving. (SOA-C02)

Q5. Under load, new connections to one API start failing but the API is healthy. What’s happening? Source-port exhaustion: a NAT gateway supports ~55,000 simultaneous connections to each unique destination, and you’re exceeding it for that one destination. Confirm with the ErrorPortAllocation CloudWatch metric (non-zero). Fix: spread across destinations, add EIPs, or add NAT gateways. (ANS-C01)

Q6. A NAT instance forwards no traffic even though routing looks right. What did you forget? To disable the source/destination check on the instance (modify-instance-attribute --no-source-dest-check). By default EC2 drops packets not addressed to itself, which kills all forwarded traffic. (SAA-C03)

Q7. How do you give an IPv6-only private instance outbound internet? An egress-only internet gateway with a ::/0 → eigw route. There is no NAT for IPv6 because IPv6 addresses are globally routable; the EIGW provides the stateful outbound-only behavior instead. It’s free. (ANS-C01)

Q8. NAT gateway vs NAT instance — when do you choose the instance? Choose the instance for cost-sensitive dev/low-scale egress (no per-GB charge, ~$3/month) or when you need a security group on the egress path (the gateway has none). Choose the gateway for anything production — managed, auto-scaling, AZ-redundant. (SAA-C03)

Q9. Can you attach a security group to a NAT gateway to filter outbound? No. A NAT gateway has no security group. Control egress with subnet NACLs, AWS Network Firewall, or a NAT instance (which does take a security group). (SOA-C02)

Q10. How do you administer a private instance with no bastion and no open port 22? SSM Session Manager: give the instance an IAM role with AmazonSSMManagedInstanceCore and either NAT egress or the three SSM interface endpoints (ssm, ssmmessages, ec2messages). The agent dials out; you aws ssm start-session. No inbound rule needed. (SOA-C02)

Q11. A gateway endpoint vs an interface endpoint — cost and coverage difference? Gateway endpoints (S3, DynamoDB only) are free and route-table based. Interface endpoints (most other AWS APIs) are PrivateLink ENIs costing ~$0.01/hr per AZ plus ~$0.01/GB. Use gateway endpoints always; interface endpoints when the AWS API traffic they remove from the NAT justifies their cost or you need a fully private path. (ANS-C01)

Q12. After deleting a NAT gateway you’re still billed a few dollars. Why? The Elastic IP is still allocated, and AWS bills unattached (and attached) public IPv4 at ~$0.005/hr. Release it with release-address. (SOA-C02)

Quick check

1. What single route-table entry makes a subnet “private” rather than “public”?
2. In which subnet type does a NAT gateway itself live, and why?
3. Name the two cost components of a NAT gateway besides the Elastic IP charge.
4. What CloudWatch metric warns you of NAT source-port exhaustion?
5. Which free change removes S3 traffic from the NAT bill, and how is it expressed in the route table?

Answers

1. The absence of a 0.0.0.0/0 → IGW route. “Private” means no direct internet route; egress instead goes 0.0.0.0/0 → NAT.
2. A public subnet — one with 0.0.0.0/0 → IGW — because the NAT reaches the internet through the IGW. It serves private subnets but must itself be public.
3. The hourly NAT-gateway charge (~$0.045/hr) and the per-GB data-processing charge (~$0.045/GB). (The public-IPv4/EIP charge is the third, separate one.)
4. ErrorPortAllocation — any non-zero value means the NAT couldn’t allocate a source port to a destination (exhaustion past ~55,000 connections to that destination).
5. An S3 gateway endpoint. It adds a prefix-list route — pl-…(S3) → vpce-… — to the private route table, so S3 traffic takes the free endpoint over the AWS backbone instead of the NAT.

Glossary

Next steps

• Build the underlying network first if it’s not second nature: AWS VPC from Scratch: Subnets, Route Tables and Internet Gateway.
• Eliminate egress entirely for AWS services with AWS PrivateLink and VPC Endpoints: Interface vs Gateway — the natural companion to cutting the NAT bill.
• Centralize NAT and egress inspection across many VPCs in AWS Transit Gateway Hub-and-Spoke, Hands-On.
• Revisit the firewall and subnet fundamentals that decide reachability in AWS VPC, Subnets and Security Groups Explained.