The AWS VPC Connectivity Troubleshooting Playbook: Why Can't A Reach B?

Quick take: When “A can’t reach B” in a VPC, do not stare at the security group — walk the packet in order and stop at the first layer that could drop it. A packet from A crosses, in this exact sequence: the route table (is there a route to B at all?), the subnet’s stateless NACL, the ENI’s stateful security group, and finally lands on a target that must actually be listening; if B lives in another VPC or on-prem it also crosses a Transit Gateway, a peering connection, or a VPN/DX, and every hop is filtered again by DNS and MTU. Almost every AWS-network drop is silent — the client just times out — so a timeout tells you nothing about which layer failed. The whole skill is (1) knowing the ordered path, (2) reading the two signals that aren’t silent (RST = “reached the host, app refused”; ICMP unreachable = “no route”), and (3) getting a deterministic answer from Reachability Analyzer and VPC Flow Logs instead of guessing.

A platform team pages you at 2 a.m.: the new order-service instance in ap-south-1 cannot reach the payments API in a peered VPC. The security group “looks right.” The instance is healthy. curl from a bastion in the same subnet works; curl from the order service hangs for 30 seconds and dies with Connection timed out. Three engineers open three consoles and all three open the security group — the single most over-inspected object in AWS networking — and find nothing wrong, because nothing is wrong there. The packet never got far enough to be judged by a security group: the peered VPC’s route table had a route back to the wrong CIDR after a subnet was re-CIDR’d, so the reply was black-holed one layer earlier. Two hours gone on the wrong layer.

That incident is the whole reason this playbook exists. “Traffic won’t flow” is not one problem; it is a dozen different problems that all present as the same symptom — a silent timeout — at a dozen different layers. If you debug them in a random order, or worse always start at the security group, you will burn hours on healthy layers while the broken one sits three hops upstream. The senior move is mechanical: there is exactly one packet path, it has a fixed order, and you check the layers in that order, using tools that give a deterministic answer rather than a hunch. This article is that path. We define every layer a packet crosses — route table, public-IP/EIP, subnet/AZ, security group, NACL, DNS, MTU, Transit Gateway, peering, endpoint, and on-prem VPN/DX — enumerate exactly how each one fails, then hand you a master symptom→layer→confirm→fix table, a Reachability Analyzer explanation-code reference, and a fast decision tree. Every check shows both the aws CLI command and the Terraform that would have prevented it. This is the map you keep open when the pager goes off, and it maps directly to the networking domains of SAA-C03, SOA-C02, and the advanced networking specialty ANS-C01.

If you want the ground-up build first — CIDRs, subnets, route tables, gateways — read AWS VPC From Scratch: Subnets, Route Tables and IGW. This playbook assumes that foundation and goes straight to why the packet stopped.

What problem this solves

In a VPC, a single connection request is judged independently by as many as eleven different controls, evaluated in a fixed order, and any one of them can drop the packet without logging a thing on the instance. The application sees Connection timed out. That is the entire diagnostic problem: the symptom is identical no matter which layer failed, so the symptom carries almost no information. Without a systematic method you are reduced to changing one thing at a time and re-testing — the slowest possible debugging loop, and the one most likely to “fix” the wrong layer (widening a security group that was never the problem) and leave a real hole behind.

What breaks without this discipline is a recognisable pattern of wasted incidents. An engineer spends an afternoon on a security group when the subnet’s route table has no route to the destination. A team “works from AZ-a but not AZ-b” and blames the app, when the AZ-b subnet is simply associated with a route table that has no NAT route. A connection is fine for small requests and dies on large ones, and nobody suspects MTU because the concept never comes up until it bites. A Transit Gateway attachment is associated to a route table but the route was never propagated, so half the mesh is dark. Someone peers two VPCs, adds routes on one side only, and cannot understand why traffic goes out but never comes back. Each of these is a five-minute fix once you know which layer — and a lost day when you don’t.

Who hits this: everyone past a single instance in a single subnet. It bites hardest at the boundaries — where a packet leaves one blast radius and enters another: subnet→subnet across a NACL, VPC→VPC across peering or a TGW, cloud→on-prem across a VPN, and app→AWS-service across a VPC endpoint. Those boundaries are where ownership splits (the platform team owns the TGW and NACLs; the app team owns the security groups and the instance) so no single person can see the whole path, and the packet dies in the seam between two teams. The cure is a shared, ordered mental model of the path — which is exactly what a playbook is.

Here is the whole field on one page: every layer a packet crosses, in order, what it does, its classic failure, and the signal it produces.

The rest of this article is that table, expanded — one section per layer with exactly how it fails and how to confirm it — followed by the tools that turn a silent timeout into a named component.

Learning objectives

By the end of this article you can:

• Recite the ordered packet path a connection crosses and, given a symptom, name the first layer that could produce it — instead of defaulting to the security group.
• Diagnose a route-table failure — missing route, wrong association, more-specific route, blackhole — and read describe-route-tables correctly.
• Distinguish a silent drop (route/SG/NACL — timeout) from a loud refusal (RST — the host answered) from no route (ICMP unreachable) at a glance.
• Run VPC Reachability Analyzer and interpret its explanation codes (NO_ROUTE_TO_DESTINATION, ENI_SG_RULES_MISMATCH, SUBNET_ACL_RESTRICTION, MORE_SPECIFIC_ROUTE, TGW_*, PCX_*) to get a deterministic blocking component.
• Read a VPC Flow Logs REJECT record and pair inbound/outbound rows to separate a security-group block from a NACL block.
• Debug TGW (association vs propagation vs appliance mode), peering (non-transitivity, cross-region SG-ref), endpoints (prefix list, policy, private DNS), and on-prem VPN/DX (tunnel, BGP, MSS).
• Recognise the three nastiest failure classes — asymmetric routing through an appliance, “works in one AZ not another,” and intermittent = MTU or one dead tunnel — from their fingerprints.
• Reproduce a route-table break in a lab, localise it with Reachability Analyzer and Flow Logs, and fix it in both aws CLI and Terraform.

Prerequisites & where this fits

You should be comfortable with VPC fundamentals — CIDR blocks, subnets, route tables, internet and NAT gateways — and able to launch an EC2 instance and run the AWS CLI with credentials. You do not need to be a network specialist; the goal here is to make you the person in the room who can localise any VPC drop in minutes.

This playbook is the hub that the focused deep-dives spoke into. Each layer below has a dedicated article that goes deeper on building that component; this one is about diagnosing the whole path when it breaks.

Core concepts

One packet, one ordered path

The mental model is a pipeline, not a cloud. A request from source A to destination B is inspected by a fixed sequence of controls, and the packet dies at the first one that says no. If you internalise the order, diagnosis becomes “walk the list until a layer could be responsible, confirm it, move on.” Here is the canonical path for an intra-VPC TCP connection, both directions:

Two facts on this table do most of the work in real incidents. First, the security group is stateful (hop 8 is automatic) but the NACL is stateless (hop 9 must be allowed explicitly), which is why the single most common block is a missing outbound ephemeral-range rule on a custom NACL. Second, the return path is a path too — hop 10 — so “the request left but the reply never came” is almost always a routing asymmetry, not a firewall.

The three signals: silent, loud, and unreachable

Most AWS drops are silent, but not all. The signal narrows the layer before you run a single command.

Burn this in: timeout = firewall/route (silent); RST = the host answered (not a network drop); unreachable = routing. Half of all mis-diagnoses come from treating a Connection refused as a security-group problem — it is the opposite, proof the packet crossed every firewall and reached a host that has nothing on that port.

Which layers drop silently

You cannot log your way out of a silent drop from the instance; the packet never arrives. That is precisely why Flow Logs (which record the drop at the ENI/subnet) and Reachability Analyzer (which reasons about config) exist.

Layer 1 — Route tables: always the first suspect

Before any firewall runs, the VPC router must have a route toward the destination. No route means the packet is dropped at hop 1, and this is the layer engineers skip most because it feels too basic to be wrong. It is wrong constantly — after a re-CIDR, a subnet move, a Terraform apply that replaced a NAT gateway, or a TGW attachment that lost its route.

Every subnet is associated with exactly one route table (explicitly, or the VPC main table by default). The router does longest-prefix match: the most specific matching route wins, and local (the VPC CIDR) always wins within the VPC and cannot be overridden. These failure modes cover essentially every routing incident:

The route target keyword tells you where a packet is being sent — memorise the vocabulary so a route table reads at a glance:

And the precedence rules that decide which route wins when several match:

# Which route table actually governs this subnet, and what does it say?
aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-0abc123" \
  --query 'RouteTables[].Routes[].{Dest:DestinationCidrBlock,Target:GatewayId||NatGatewayId||TransitGatewayId||VpcPeeringConnectionId,State:State}' \
  --output table

# Terraform: the route that prevents "works in AZ-a not AZ-b" — associate EVERY
# private subnet explicitly, and give each AZ its own NAT to avoid cross-AZ data cost.
resource "aws_route_table" "private" {
  for_each = toset(["a", "b"])
  vpc_id   = aws_vpc.main.id
}
resource "aws_route" "private_default" {
  for_each               = aws_route_table.private
  route_table_id         = each.value.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.per_az[each.key].id
}
resource "aws_route_table_association" "private" {
  for_each       = aws_subnet.private
  subnet_id      = each.value.id
  route_table_id = aws_route_table.private[each.value.availability_zone_suffix].id
}

Layer 2 — Public reachability: EIP, subnet, and AZ

If B is on the internet (or A is being reached from the internet), routing is necessary but not sufficient — the ENI also needs a public identity. A public subnet is defined as one whose route table points 0.0.0.0/0 at an IGW; an instance in it still cannot be reached from the internet unless it has a public IPv4 address or an Elastic IP. Auto-assign-public-IP is a subnet attribute that is off by default on custom subnets, so instances launched there are private even in a “public” subnet.

The subnet-type vocabulary decides which of these even apply:

For egress-from-private-subnet failures specifically — “the instance can’t yum update” — the full path is: private route table → NAT gateway → NAT’s public subnet route → IGW. Any missing link is a silent timeout. The dedicated walk-through is NAT Gateway: Private-Subnet Egress.

# Prove the instance has (or lacks) a public identity, in one call:
aws ec2 describe-instances --instance-ids i-0abc \
  --query 'Reservations[].Instances[].{PrivIP:PrivateIpAddress,PubIP:PublicIpAddress,Subnet:SubnetId,SrcDstCheck:SourceDestCheck}' \
  --output table

Layer 3 — The two firewalls, in path order

Both firewalls are covered exhaustively in the Security Groups vs NACLs Deep Dive; here they are just two hops on the path. The one fact that matters for the packet order: the NACL is evaluated at the subnet edge (outer gate), the security group at the ENI (inner gate), and they behave oppositely on the return trip.

Security-group failure modes, in the order you should check them:

NACL failure modes:

The ephemeral range depends on who is the client sending the reply-bound traffic — get this wrong and a NACL silently eats replies:

The universal safe answer is 1024-65535 outbound — anything narrower risks dropping some client’s replies.

Layer 4 — DNS: resolves to nothing, or to the wrong IP

A “connectivity” failure is often a name-resolution failure wearing a trench coat. If the name doesn’t resolve, or resolves to a public IP when you wanted the private one, no amount of route/SG/NACL work helps. VPC DNS hinges on two VPC attributes and the .2 Route 53 Resolver.

DNS failure modes:

# Resolve against the VPC's own resolver, not your laptop's, to prove the VPC's view:
dig +short myservice.internal @169.254.169.253
dig +short com.amazonaws.ap-south-1.s3 @169.254.169.253   # should be a private 10.x if endpoint private DNS is on

Layer 5 — MTU and fragmentation: works small, hangs large

This is the layer nobody suspects because it produces the weirdest symptom: small requests succeed, large ones hang. An SSH handshake works; scp of a big file stalls at 0%. A health check passes; a real API response times out. That fingerprint is almost always MTU — a hop in the path can’t carry the packet size, and because the DF (Don’t Fragment) bit is set (as it is on virtually all modern TCP), the oversized packet is dropped rather than fragmented, and if the ICMP “fragmentation needed” message that Path MTU Discovery relies on is blocked by a firewall, the sender never learns to shrink. The connection black-holes.

Know the MTU ceiling of every path — this table ends the argument:

MTU/fragmentation failure modes:

# The definitive PMTUD probe from the instance (Linux): find the largest payload that survives.
ping -M do -s 1472 10.0.2.50    # 1472 + 28 = 1500; if this fails but -s 1400 works, MTU < 1500 on the path
# Clamp TCP MSS on egress so large flows never exceed the path (interface-wide):
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Layer 6 — Beyond the VPC: TGW, peering, endpoints, and on-prem

When B lives outside the source VPC, the packet crosses one more fabric, each with its own routing plane and its own failure modes.

Transit Gateway

A TGW is a regional router with its own route tables, separate from VPC route tables. The trap is that an attachment needs two independent things: an association (which TGW route table governs traffic from this attachment) and a propagation (which attachments’ CIDRs get advertised into a TGW route table). Association without propagation = the attachment can send but its CIDR is invisible to others. Full walk-through: Transit Gateway Hub-and-Spoke.

VPC peering

Peering is a 1:1, non-transitive link. If A peers B and B peers C, A cannot reach C through B — you need a direct A–C peering or a TGW. Every peering also needs routes on both sides and has hard limits worth memorising. Deep-dive: VPC Peering: Setup, Routing, Limits.

VPC endpoints

Endpoints give a private path to AWS services. Gateway endpoints (S3, DynamoDB) inject a prefix-list route into associated route tables; interface endpoints put an ENI (with its own SG) in your subnet and optionally hijack the service DNS name. Deep-dive: PrivateLink & VPC Endpoints.

On-prem via VPN / Direct Connect

Hybrid adds the most moving parts: two IPsec tunnels (for a VPN), BGP sessions, and the customer gateway. The dedicated build is Site-to-Site VPN Hybrid.

The diagnostic toolkit: from silent timeout to named component

You do not guess your way down the layer list — you use tools that give a deterministic answer. There are four, and knowing which to reach for is half the skill.

Reachability Analyzer and its explanation codes

Reachability Analyzer is the first tool to reach for because it reasons about configuration and returns a verdict — Reachable or Not reachable — plus, when blocked, an explanation code naming the exact component and reason. It needs no live traffic, so you can run it before the app is even deployed. These are the codes you will actually see, grouped by layer (verbatim from the AWS reference):

# Create and run a path analysis A → B on port 443, then read the verdict + explanation code.
PID=$(aws ec2 create-network-insights-path \
  --source i-0source --destination i-0dest \
  --protocol tcp --destination-port 443 \
  --query 'NetworkInsightsPath.NetworkInsightsPathId' --output text)

AID=$(aws ec2 start-network-insights-analysis \
  --network-insights-path-id "$PID" \
  --query 'NetworkInsightsAnalysis.NetworkInsightsAnalysisId' --output text)

aws ec2 describe-network-insights-analyses --network-insights-analysis-ids "$AID" \
  --query 'NetworkInsightsAnalyses[0].{Reachable:NetworkPathFound,Why:Explanations[].ExplanationCode}'
# Reachable=false with Why=["NO_ROUTE_TO_DESTINATION"] tells you the layer without touching a firewall.

VPC Flow Logs: proving the drop and naming the direction

When you need to see a real packet’s fate, Flow Logs record every accepted/rejected flow at the ENI or subnet. The trick to separating a security group from a NACL is to look at the direction pairing: a security-group block shows a REJECT with no matching return; a NACL ephemeral block shows an inbound ACCEPT paired with an outbound REJECT. The full field-by-field method is in VPC Flow Logs for Network Troubleshooting.

# Hunt REJECTs to a target's port in a Flow Logs CloudWatch group:
aws logs filter-log-events --log-group-name /vpc/flowlogs \
  --filter-pattern '[version,account,eni,src,dst,srcport,dstport="443",proto,pkt,byte,start,end,action="REJECT",status]' \
  --query 'events[].message' --output text

Host-level probes and describe-* commands

From the instance itself you get ground truth — the OS’s view of DNS, routes, and whether anything is listening. Two cheat-sheets you keep in muscle memory:

Architecture at a glance

The diagram below is the playbook rendered as the actual packet path: a request from source A first needs a route (badge 1), crosses the subnet’s stateless NACL (badge 2), then the ENI’s stateful security group (badge 3), and reaches a target that must be listening and answerable (badge 4). If B lives in another VPC or on-prem, the packet also traverses a Transit Gateway, peering, or VPN/DX (badge 5), where DNS and MTU also bite. Because every one of these drops is silent, you localise the real one at the diagnose stage (badge 6) with Reachability Analyzer’s explanation code and a Flow Logs REJECT. Read it left→right and, crucially, top-down through the numbers — that is the order you check the layers.

The single most valuable habit the diagram encodes: check in packet order and stop at the first layer that could be responsible. Nine times out of ten the answer is upstream of the security group you were about to open.

Real-world scenario

Company: Nimbus Retail, a mid-size e-commerce platform running a hub-and-spoke network in ap-south-1: a shared-services VPC (10.0.0.0/16), an order-service spoke (10.1.0.0/16), and a payments spoke (10.2.0.0/16), all attached to a Transit Gateway, with a Site-to-Site VPN from the shared VPC to an on-prem fraud-scoring system (192.168.0.0/16).

The incident. After a routine Terraform apply that “just added a subnet” to the order spoke, checkout latency spiked and ~40% of orders failed fraud scoring. The on-call engineer’s first move — the wrong first move — was to open the payments security group. It was untouched. An hour in, they widened it to 0.0.0.0/0 “to rule it out.” Nothing improved (and now there was a security hole to clean up later). The symptom was maddening: orders from some instances scored fine; others timed out. It looked intermittent, which sent everyone hunting a flaky app.

The systematic pass. A senior engineer ran the playbook in order. Reachability Analyzer from a failing order instance to the on-prem fraud endpoint returned Not reachable with NO_ROUTE_TO_DESTINATION. That immediately killed the security-group theory — the packet never reached a firewall. describe-route-tables on the new subnet showed it had been created without an explicit route-table association, so it fell back to the VPC main route table, which had no route for 192.168.0.0/16 → tgw-…. The “some instances work” pattern was simply which subnet an instance landed in: old subnets had the correct table; new-subnet instances used the main table and were dark to on-prem. One propagation was also missing on the TGW side, so even fixing the association only half-worked until they enabled propagation of the VPN attachment’s routes.

The fix and the lesson. Two changes: associate the new subnet with the order spoke’s private route table (which carries 192.168.0.0/16 → tgw), and enable route propagation for the VPN attachment in the TGW route table. Reachability Analyzer re-run: Reachable. Total fix time once they stopped guessing: 12 minutes. The postmortem action items were pure playbook discipline — never start at the security group; make Reachability Analyzer the first command; and in Terraform, always create aws_route_table_association for every subnet so nothing silently inherits the main table. Nimbus added a CI check that fails any plan creating an aws_subnet without a matching association. The security group they had opened to the world was reverted the same day — a reminder that “ruling out” a layer by weakening it creates a second incident.

Advantages and disadvantages

The “advantage” here is the method — systematic, packet-order, tool-first diagnosis — versus the ad-hoc alternative most teams default to.

The disadvantages are real and worth naming. Reachability Analyzer analyses configuration, not live packets — it will not catch an application bug, a host firewall (iptables), an OS that isn’t listening, or a transient tunnel flap, and it does not support IPv6 or every component type. Flow Logs prove real drops but cost money at scale and have a capture delay. Host tools give ground truth but require access to the instance (often the hardest thing to arrange mid-incident). The mature setup uses all four: RA to localise config, Flow Logs to confirm live behaviour, host tools for DNS/listener/MTU, and describe-* to read the exact rules — and never leans on only one.

Hands-on lab

You will deliberately break connectivity at the route-table layer, watch the silent timeout, localise it with Reachability Analyzer and Flow Logs, then fix it — in both aws CLI and Terraform — and tear it down. Free-tier friendly; the only meterable items are called out. Region: ap-south-1.

1. Build a single-instance internet-reachability target

# One VPC, one public subnet, an IGW, and a client instance with a public IP.
VPC=$(aws ec2 create-vpc --cidr-block 10.20.0.0/16 --query Vpc.VpcId --output text)
aws ec2 modify-vpc-attribute --vpc-id $VPC --enable-dns-hostnames
SUBNET=$(aws ec2 create-subnet --vpc-id $VPC --cidr-block 10.20.1.0/24 \
  --availability-zone ap-south-1a --query Subnet.SubnetId --output text)
aws ec2 modify-subnet-attribute --subnet-id $SUBNET --map-public-ip-on-launch
IGW=$(aws ec2 create-internet-gateway --query InternetGateway.InternetGatewayId --output text)
aws ec2 attach-internet-gateway --internet-gateway-id $IGW --vpc-id $VPC

# A purpose-built route table (never rely on the main table), associated + default route to the IGW.
RTB=$(aws ec2 create-route-table --vpc-id $VPC --query RouteTable.RouteTableId --output text)
aws ec2 associate-route-table --route-table-id $RTB --subnet-id $SUBNET
aws ec2 create-route --route-table-id $RTB --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW

Launch a t3.micro client (CLIENT_ID) into that subnet — any AMI works; an SSM-enabled one is easiest so you need no bastion — then confirm the baseline from inside the instance:

# From the CLIENT instance (SSM Session Manager or SSH):
curl -s --max-time 5 https://checkip.amazonaws.com   # expected: your instance's public IP, e.g. 13.234.x.x

Expected: your public IP prints. The instance reaches the internet, so the security-group egress and the NACL are correct — any break you now introduce at the route layer is unambiguous.

Why the internet and not a second in-VPC server? Inside one VPC the local route (10.20.0.0/16) covers every subnet and cannot be removed or overridden — so you literally cannot break same-VPC routing at the route-table layer (that is the local-supremacy rule from Layer 1). Routing only matters, and only breaks, where a packet needs a non-local route: the internet, a peer, a TGW, or on-prem. So we break the one route that governs all of those — the default route.

2. Break it at the route table

# Delete the default route — the single most common route-table outage ("the instance lost the internet").
aws ec2 delete-route --route-table-id $RTB --destination-cidr-block 0.0.0.0/0

# From the CLIENT instance again:
curl -s --max-time 5 https://checkip.amazonaws.com   # expected NOW: hangs 5s, exits non-zero (silent timeout)

Note what you don’t see: no error explaining why, no refused, just a timeout. This is the moment most engineers open the security group. You will not — the SG never changed.

3. Localise with Reachability Analyzer

# Analyse the path from the instance to the internet gateway on 443.
PID=$(aws ec2 create-network-insights-path --source $CLIENT_ID --destination $IGW \
  --protocol tcp --destination-port 443 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
AID=$(aws ec2 start-network-insights-analysis --network-insights-path-id $PID \
  --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text)
sleep 20
aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $AID \
  --query 'NetworkInsightsAnalyses[0].{Reachable:NetworkPathFound,Why:Explanations[].ExplanationCode}'

Expected output:

{ "Reachable": false, "Why": [ "NO_ROUTE_TO_DESTINATION" ] }

There it is — the tool names the route layer, not the firewall, in one call and about ₹8 (US $0.10). You localised the break without touching a single rule.

4. Confirm with Flow Logs (optional, live proof)

Enable Flow Logs on the client ENI to a CloudWatch group, re-run the curl, and read the client ENI’s records:

aws logs filter-log-events --log-group-name /vpc/lab-flow \
  --filter-pattern '[v,acct,eni,src="10.20.1.*",dst,sp,dp="443",pr,pk,by,st,en,action,logst]' \
  --query 'events[].message' --output text

You will see the outbound SYN accepted at the ENI (the security group allowed egress) but no return record — the packet left the interface and was then dropped by the VPC router for lack of a route. That “it left, nothing came back” shape is the routing fingerprint; contrast it with a crisp REJECT, which would name a firewall instead. ⚠️ Flow Logs ingestion to CloudWatch is billable.

5. Fix it — CLI then Terraform

# Restore the default route; the fix is one line.
aws ec2 create-route --route-table-id $RTB --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
# Re-run Reachability Analyzer, then re-test:
aws ec2 start-network-insights-analysis --network-insights-path-id $PID --query NetworkInsightsAnalysis.State
# From the client:  curl -s --max-time 5 https://checkip.amazonaws.com  → your public IP again

The Terraform that makes this declarative and un-break-able — an explicit table, an explicit default route, and an explicit association so nothing silently inherits the main table:

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.lab.id
}
resource "aws_route" "default" {                     # the route whose absence caused the outage
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.lab.id
}
resource "aws_route_table_association" "public" {    # never rely on the main table
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

Re-run Reachability Analyzer a final time; expect Reachable: true.

6. Teardown (do this — some resources meter)

aws ec2 delete-network-insights-analysis --network-insights-analysis-id $AID
aws ec2 delete-network-insights-path --network-insights-path-id $PID
aws ec2 terminate-instances --instance-ids $CLIENT_ID
aws ec2 wait instance-terminated --instance-ids $CLIENT_ID
aws logs delete-log-group --log-group-name /vpc/lab-flow          # stop Flow Logs billing
aws ec2 detach-internet-gateway --internet-gateway-id $IGW --vpc-id $VPC
aws ec2 delete-internet-gateway --internet-gateway-id $IGW
aws ec2 delete-subnet --subnet-id $SUBNET
aws ec2 delete-route-table --route-table-id $RTB
aws ec2 delete-vpc --vpc-id $VPC

⚠️ Costs to remember: Flow Logs ingestion and Reachability Analyzer (~₹8 per analysis) are the only meterable items here — you deliberately did not create a NAT gateway. EC2 t3.micro is free-tier eligible; the VPC, subnet, route table, IGW, SG and NACL are free.

Common mistakes & troubleshooting

This is the section you keep open during an incident. Start at the top of the master table, and for any symptom, check the layer it names before the ones below it.

The master symptom → layer → confirm → fix table

Fast decision table

The three nastiest, in prose

Asymmetric routing through an appliance. You insert a firewall/IDS appliance (or a GWLB) and traffic mysteriously drops on the return leg. The forward packet goes A→appliance→B, but the reply from B takes a different path back (a different AZ’s appliance ENI, or straight via local), and the stateful appliance never saw the forward flow so it drops the reply. The fingerprint: SYN arrives, SYN-ACK leaves, connection never completes; it often works when A and B happen to land in the same AZ and fails across AZs. On a Transit Gateway the cure is appliance mode (which pins both directions to the same appliance ENI); RA flags it as TGW_ATTACH_VPC_AZ_RESTRICTION. Off the TGW, the cure is symmetric routing — force the return through the same appliance with more-specific routes, because the VPC local route will otherwise steal in-VPC replies straight back.

“Works from one AZ, not another.” Nothing about the app differs, yet instances in AZ-a succeed and AZ-b times out. This is almost always per-AZ routing or NAT: the AZ-b subnet is associated with a route table that lacks the NAT/TGW route the AZ-a subnet has, or there is a single NAT gateway in AZ-a and AZ-b’s route points at a NAT that was deleted (blackhole), or an interface endpoint/appliance exists only in AZ-a’s subnet. Confirm by diffing the two subnets’ associated route tables and the AZ placement of NATs/endpoints. The permanent fix is architectural: one route table and one NAT (and one endpoint ENI) per AZ, every subnet explicitly associated — never rely on the main table.

Intermittent = MTU or one dead tunnel. Two very different causes share the “sometimes works” signature. If failures correlate with payload size (small OK, large stalls), it is MTU: a DF-set jumbo packet meeting a 1500/8500 hop with PMTUD’s ICMP blocked. If failures correlate with which flow/hash rather than size (roughly half of connections fail, retries sometimes succeed), it is one of the two VPN tunnels down while ECMP keeps hashing new flows onto the dead one — or an asymmetric multi-path. Separate them fast: ping -M do -s 1472 proves MTU; describe-vpn-connections showing Tunnel1: UP, Tunnel2: DOWN proves the tunnel. Treating an MTU black hole as “flaky network” (or a dead tunnel as “app instability”) is the classic multi-hour rabbit hole.

Best practices

• Check in packet order, stop at the first plausible layer. Route → public-IP → subnet/AZ → SG → NACL → DNS → MTU → fabric. Do not start at the security group by reflex.
• Make Reachability Analyzer your first command, not your last. It is deterministic, non-destructive, works before deploy, and names the component.
• Never “rule out” a layer by weakening it. Widening an SG to 0.0.0.0/0 to test is how a debugging session becomes a security incident.
• Associate every subnet with an explicit route table in IaC. The main-table fallback causes silent “works here, not there” outages; enforce it with a CI policy check.
• One route table, one NAT, one endpoint ENI per AZ. It removes an entire class of “one AZ works” incidents and avoids cross-AZ data charges.
• On custom NACLs, always add outbound tcp 1024-65535. The stateless return-traffic omission is the single most common firewall block.
• Prefer security-group references over CIDRs within a region; they survive IP changes. Fall back to CIDRs across regions where SG-ref isn’t supported.
• Turn on VPC Flow Logs (at least at the VPC level) before you need them — you cannot retroactively capture a drop.
• Clamp TCP MSS on VPN/DX paths and standardise instance MTU per path; document which paths are 1500 vs 8500 vs 9001.
• Verify both association and propagation on every TGW attachment; they are two switches, and one without the other is a silent half-outage.
• Add routes on both sides of every peering and remember it is non-transitive — a hub needs a TGW, not chained peerings.
• Keep a describe-* runbook and test access paths (SSM Session Manager) so you can reach an instance mid-incident without SSH/bastion dependencies.

Security notes

Troubleshooting connectivity and hardening it pull in opposite directions, and the seam is where mistakes happen. First, do not debug by widening: the 0.0.0.0/0 “temporary” rule is the most common way a connectivity ticket turns into a data-exfiltration path — use Reachability Analyzer (which needs no rule changes) instead, and if you must widen, scope to the exact source CIDR and revert in the same session. Second, treat Reachability Analyzer and Network Access Analyzer as security tools too: Network Access Analyzer specifically audits unintended reachability — run it to prove that a database subnet is not reachable from the internet, not just that your app is reachable. Third, least-privilege the diagnostic access itself: the IAM permissions to run ec2:StartNetworkInsightsAnalysis, read Flow Logs, and describe-* should be a scoped read/analyse role, not admin. Fourth, VPC endpoint and bucket policies are a legitimate “deny” — an AccessDenied through an endpoint is often the control working; confirm intent before “fixing” it, and use aws:SourceVpce conditions to keep S3 access private-path-only. Finally, encrypt the transport you are debugging (TLS end-to-end, IPsec on the VPN) so that the packet captures and Flow Logs you generate during an incident never expose payload data — Flow Logs record metadata, but the habit of grabbing tcpdump mid-incident can.

Cost & sizing

The diagnostic tooling is cheap relative to an hour of engineer time, but Flow Logs and NAT data can surprise you at scale. Rough figures (USD list; INR ≈ ×83):

Sizing guidance: enable Flow Logs at the VPC level with a sampled or aggregated format for always-on visibility, and switch to full capture on a specific ENI only during an incident to control ingest cost. Send high-volume Flow Logs to S3 + Athena rather than CloudWatch Logs; the query experience is nearly as good and the storage is far cheaper. For NAT and TGW, the bill is dominated by data processing, so a connectivity “fix” that accidentally routes bulk traffic through a NAT or across a TGW when a VPC endpoint would do can quietly add real money — another reason to know exactly which path your packet takes.

Interview & exam questions

Q1. A packet from A times out reaching B in the same VPC. In what order do you check the layers, and why? Route table first (no route = dropped before any firewall), then security group ingress, then NACL, then whether the target is listening. You go in packet order because each layer can produce the identical silent timeout, and the earliest one that could be responsible must be cleared first. (ANS-C01/SAA-C03)

Q2. What is the difference between a Connection timed out and a Connection refused, diagnostically? Timed out = a stateful firewall or routing black hole dropped the packet silently; the packet may never have reached the host. Refused = an instant TCP RST, meaning the packet did reach the host but nothing is listening on that port (or a host firewall sent the RST). Refused is proof the network path works. (SOA-C02)

Q3. Why does a custom NACL so often break return traffic? NACLs are stateless, so the reply — which leaves from the server toward the client’s high ephemeral port — must be explicitly allowed by an outbound rule. If the NACL only allows outbound 443, the reply on ports 1024–65535 hits the implicit * deny and is dropped; the client just times out. Fix: outbound tcp 1024-65535 to the client CIDR. (SAA-C03)

Q4. Reachability Analyzer returns NO_ROUTE_TO_DESTINATION. What does that eliminate, and what do you check? It eliminates the security group and NACL as the first cause — the packet has no route, so it dies before any firewall. Check the source subnet’s associated route table for a matching route to the destination CIDR, and confirm the subnet isn’t silently using the main table. (ANS-C01)

Q5. Explain association vs propagation on a Transit Gateway. Association binds an attachment to one TGW route table that governs traffic leaving that attachment. Propagation advertises an attachment’s CIDRs into a TGW route table so others can route to it. You need both: association without propagation means the attachment can send but is invisible as a destination. (ANS-C01)

Q6. Why is VPC peering “non-transitive,” and what breaks because of it? Each peering is a 1:1 link that only routes between the two VPCs’ CIDRs; it does not forward to a third VPC. So a hub-and-spoke built from chained peerings fails spoke-to-spoke. Use a Transit Gateway for any-to-any. (SAA-C03)

Q7. A transfer works for small files and stalls for large ones across a VPN. Diagnose. Path MTU: the VPN caps at 1500 with IPsec overhead, so DF-set large packets are dropped, and if the ICMP fragmentation-needed message is blocked, PMTUD never shrinks the sender. Confirm with ping -M do -s 1472; fix by clamping TCP MSS (~1379) or lowering MTU and allowing ICMP type 3 code 4. (ANS-C01)

Q8. “Works from AZ-a, not AZ-b.” Top hypothesis? The AZ-b subnet is associated with a route table missing a route AZ-a’s has (commonly the NAT or TGW route), or a NAT/endpoint/appliance exists only in AZ-a. Diff the two subnets’ associated route tables and the AZ placement of NAT/endpoints. (SOA-C02)

Q9. What does an interface-endpoint name resolving to a public IP tell you? Private DNS on the endpoint (or enableDnsHostnames on the VPC) is off, so the service name resolves to the public regional endpoint instead of the private ENI. Enable private DNS and DNS hostnames. (ANS-C01)

Q10. How do you use Flow Logs to distinguish a security-group block from a NACL block? Pair the directional records: a security-group drop shows a REJECT with no established return; a NACL ephemeral omission shows an inbound ACCEPT paired with an outbound REJECT to the client’s high port. The flow-direction and action fields together localise the layer. (SOA-C02)

Q11. Why can’t you force in-VPC traffic through an appliance using only the main route table? The local route for the VPC CIDR always wins inside the VPC and cannot be overridden by a route to an ENI for an address within that CIDR. You need gateway/ingress routing or a more-specific route outside the local scope (or place the appliance on the north-south path). (ANS-C01)

Q12. When is AccessDenied (not a timeout) the correct behaviour? When a VPC-endpoint policy, S3 bucket policy aws:SourceVpce condition, or IAM policy is intentionally denying the action — that is an authorization control working, not a connectivity failure. Confirm intent before changing it. (SCS-C02)

Quick check

1. A connection returns an instant Connection refused. Is this a security-group problem? Why or why not?
2. In what order does a packet cross the route table, security group, and NACL on the way in?
3. You get MORE_SPECIFIC_ROUTE from Reachability Analyzer. What is happening?
4. What single outbound NACL rule prevents the most common firewall block, and why?
5. Failures correlate with payload size, not with which flow. What layer, and what one command confirms it?

Answers

1. No. Connection refused is an instant RST from the host, which means the packet crossed every firewall and reached a listening host that had nothing on that port — the network path works. Check ss -ltnp and the app, not the security group.
2. NACL (subnet edge) → security group (ENI) → instance. The stateless NACL is the outer gate, the stateful SG is the inner gate; the route table decides whether the packet gets to the subnet at all, before either firewall.
3. A route more specific than the one you expected is matching the destination and diverting the packet elsewhere (longest-prefix match). Read the route table and fix the prefixes or add the expected intermediate component.
4. Outbound tcp 1024-65535 to the client CIDR. NACLs are stateless, so the reply leaves on the client’s high ephemeral port and must be explicitly allowed, or it hits the implicit * deny and the client times out.
5. MTU / fragmentation. ping -M do -s 1472 <dest> — if the 1472 (→1500) probe fails but -s 1400 succeeds, a hop on the path can’t carry 1500 bytes and PMTUD is being black-holed.

Glossary

Next steps

• Master the two firewalls at depth — the stateful/stateless packet path and the ephemeral-port trap — in Security Groups vs NACLs Deep Dive.
• Learn to read a REJECT field by field and build the log pipeline in VPC Flow Logs for Network Troubleshooting.
• Build the hub that most of these failures live in with Transit Gateway Hub-and-Spoke, and understand its limits via VPC Peering: Setup, Routing, Limits.
• Get private-path access to AWS services right in PrivateLink & VPC Endpoints, and private egress in NAT Gateway: Private-Subnet Egress.
• Extend the playbook to hybrid with Site-to-Site VPN Hybrid, where tunnels, BGP, and MSS join the layer list.