EC2 SSH 'Connection Timed Out' & 'Permission Denied': The Complete Troubleshooting Guide

Quick take: EC2 SSH fails in exactly three ways, and they are not interchangeable. Connection timed out means your packet never got an answer — a network/path problem (security group, NACL, route/IGW, no public IP, OS firewall). Connection refused means the host answered but nothing is listening on the port — sshd is down, on another port, or still booting. Permission denied (publickey) means you reached sshd and it rejected your auth — wrong key, wrong username for the AMI, or bad file permissions. Read the exact error string first; it tells you which third of this guide you are in. Guessing wastes the incident; the string is the diagnosis.

It is 09:40, the deploy pipeline is red, and you cannot get onto the box to see why. ssh -i deploy.pem ec2-user@52.14.x.x just… hangs, then prints ssh: connect to host 52.14.x.x port 22: Connection timed out. Your colleague swears it worked yesterday. Nothing changed — except your home ISP handed you a new IP overnight, the My IP rule in the security group still pins last night’s address, and every SYN packet you send is now silently discarded at the ENI. Ninety seconds of reading the right signal would have told you that. Instead the team is about to widen the security group to 0.0.0.0/0, reboot the instance, and open a support case — three wrong moves for one five-second fix.

This is the most common EC2 support problem in the world, and it is maddening precisely because the same action (ssh) produces three different failures with three different root-cause families and three different fixes. The discipline that separates a two-minute recovery from a two-hour one is triage: name the failure class from the exact error string before you touch anything. A timeout and a refusal look identical in your terminal history a day later, but they are as different as “the road is closed” versus “the shop is shut.” This guide is built around that split. We trace one SSH connection through every hop — client → internet → IGW → route table → NACL → security group → ENI → sshd → authentication — map each of the three failure classes onto the exact hop where it bites, then spend the bulk of the article on a structured playbook: decode tables, a 20-row symptom → category → root cause → confirm → fix matrix, an ordered “checks in order” runbook, and the escape hatches for when SSH is dead entirely — EC2 Serial Console, EC2 Instance Connect, and SSM Session Manager, which needs no port 22 and no public IP at all.

By the end you will stop guessing. When the terminal says timed out you will look at the path; when it says refused you will look at the daemon; when it says Permission denied (publickey) you will look at the key and the username — and you will confirm each with an exact command before you change anything. Every operation shows both the aws CLI and Terraform, and the whole thing maps to the compute and networking domains of SAA-C03 and SOA-C02, where “why can’t I SSH in” is a near-guaranteed question.

What problem this solves

SSH to EC2 hides a surprising amount of machinery behind a one-line command, and when any layer of it misbehaves the feedback you get is deliberately vague. A firewall that drops your packet cannot, by design, send back “I dropped your packet” — that would help attackers map your network — so it sends nothing, and your client waits until the TCP handshake times out. The information you need is real and knowable, but it is scattered across the security group, the NACL, the route table, the instance’s public-IP assignment, the OS firewall, sshd’s config, the key’s file permissions, and the AMI’s default username. Not knowing which layer maps to which symptom is what turns a trivial fix into an afternoon.

What breaks without this triage discipline is a predictable, wasteful pattern. An engineer sees a timeout and restarts the instance (which sometimes “fixes” it by chance because a stop-start re-runs user-data or clears an OS-firewall state, teaching the wrong lesson). Another opens the security group to 0.0.0.0/0:22 — turning a firewall into an open door — because the timeout “went away.” A third burns an hour re-reading sshd config for a Permission denied that was simply the wrong username, root@ instead of ec2-user@. A fourth stares at Connection refused and blames the security group, when a refusal proves the security group is fine — the packet reached a host that answered. Each of these is a concept you learn once and never re-pay.

Who hits this: everyone who runs EC2. It bites hardest on first-time launchers (wrong AMI username, forgotten key, chmod on the .pem), on teams behind corporate networks (egress port 22 blocked at the office firewall), on anyone with a dynamic home IP (the My IP rule goes stale nightly), on private-subnet workloads (no path from the internet at all — the entire reason SSM Session Manager exists), and on modern clients hitting old servers (OpenSSH 8.8+ disabled the legacy ssh-rsa SHA-1 signature by default, and a valid old key suddenly gets Permission denied). The model is closed and small: three error classes, one connection path, a handful of hops. Learn the path and the return-traffic rules and the fog clears.

Here is the whole field in one frame — the three classes, what each proves, and where to look first:

Learning objectives

By the end of this article you can:

• Triage in one glance: map any of the three SSH errors (timed out, refused, Permission denied) to a specific hop in the connection path and name the most likely root cause for each.
• Diagnose a Connection timed out as a security-group miss, a stale My IP, a missing route to the IGW, a missing public IP/EIP, a private-subnet reachability gap, a custom-NACL block, or an in-guest firewall — and confirm which with an exact command.
• Diagnose a Connection refused as sshd not running, sshd on a non-standard port, an instance still booting, a fail2ban REJECT, or a full disk — and confirm each.
• Diagnose Permission denied (publickey) as the wrong AMI username, the wrong private key, a key not in authorized_keys, wrong permissions on the key or ~/.ssh (StrictModes), a full disk, SELinux context, or the ssh-rsa/OpenSSH 8.8 signature deprecation.
• Read ssh -vvv output and localise the failure to the network, the daemon, or the auth exchange.
• Use VPC Reachability Analyzer to get a deterministic “this component blocks it” answer for port 22.
• Recover a host with no working SSH using the four escape hatches — Serial Console, EC2 Instance Connect, SSM Session Manager, and lost-key recovery (user-data re-inject / detach-root-volume / AWSSupport-ResetAccess).
• Reproduce a timeout and a permission-denied in a free-tier lab, fix both, and tear it down — in aws CLI and Terraform.

Prerequisites & where this fits

You should be able to launch an EC2 instance, know what a key pair, security group, subnet, route table and internet gateway (IGW) are, and run the AWS CLI with credentials (aws sts get-caller-identity returns your account). You need an SSH client — OpenSSH on macOS/Linux/WSL, or a recent Windows 10/11 which ships ssh.exe. Basic Linux process and permission concepts (systemctl, chmod, file ownership) help for the server-side fixes.

This sits at the intersection of Compute and Networking troubleshooting. It assumes the firewall mechanics covered in depth in Security Groups vs NACLs: A Deep Dive and Troubleshooting Guide — this article reuses the stateful vs stateless return-traffic rule but focuses it on port 22. It is the compute-side companion to the broader AWS VPC Connectivity Troubleshooting Playbook, and the natural next read after you first Launch Your First EC2 Instance and SSH In — that walkthrough builds the working case; this one fixes it when it breaks.

A quick ownership map so you page the right person during an incident:

Core concepts

Five mental models make every later diagnosis obvious.

The error string is the triage. ssh reports the first layer that failed and stops. A timeout means the TCP three-way handshake never completed — your SYN went out and no SYN-ACK came back, because something on the path dropped it silently (firewalls drop, they do not reply). A refusal means a host did answer, with a TCP RST — the machine is up and reachable, but no process is listening on that port (or something actively rejected the SYN). A Permission denied means the TCP connection succeeded, the SSH protocol negotiated, and then authentication failed. These three sit at three different depths of the same connection, and each rules the earlier layers in: a refusal proves your security group, route, and public IP are all correct, because the packet reached a listening host to be refused.

Firewalls are silent; applications are loud. A security-group or NACL drop produces a timeout — no ICMP, no RST, nothing. An in-guest iptables -j DROP also times out. But an in-guest iptables -j REJECT, a stopped sshd, or fail2ban’s default action can produce a refusal. So “timeout vs refused” is often “packet dropped before the OS vs packet reached the OS.” This single distinction eliminates half the search space instantly.

The path is asymmetric and the NACL is stateless. Inbound, your packet crosses NACL-in → SG-in → ENI. The reply retraces SG-out → NACL-out. The security group is stateful — allow the inbound SSH and the reply is automatic. The NACL is stateless — it must explicitly allow the outbound reply, which leaves from the instance’s ephemeral port (1024–65535) back to your client. A custom NACL that allows inbound 22 but forgets the outbound ephemeral range drops every reply, and you see a timeout even though “the rule for 22 is right there.” (This is the classic covered in the SG-vs-NACL deep dive; on port 22 it is just as common.)

Reachability is not just firewalls — it is having a path at all. Even with perfect firewall rules, a packet from the internet reaches your instance only if the instance has a public IPv4 address or Elastic IP, sits in a subnet whose route table sends 0.0.0.0/0 to an internet gateway, and is actually running. A private-subnet instance has no such path by design — you must reach it through a bastion, a VPN, an EC2 Instance Connect Endpoint, or SSM Session Manager. And a public IP that is auto-assigned (not an EIP) changes on every stop-start, so yesterday’s DNS name or IP silently points at nothing.

Authentication is a contract with three terms: the key, the user, and the permissions. sshd accepts you only if (1) you present the private key whose public half is in that user’s ~/.ssh/authorized_keys, (2) you log in as the correct OS user for the AMI (Amazon Linux ec2-user, Ubuntu ubuntu, Debian admin, …), and (3) the key file on your side and ~/.ssh on the server side have tight enough permissions that sshd’s StrictModes trusts them. Miss any one and you get the same Permission denied (publickey) — which is why the message alone never tells you which term failed, and why ssh -vvv is mandatory.

The vocabulary in one table

The triage: timed out vs refused vs denied

Everything starts here. Run the connection verbosely — ssh -vvv -i key.pem user@host — and read the last thing it prints before it fails. That single line places you in one of three columns. Do not proceed to any fix until you have named the column.

Two reading notes that save the most time. First, macOS prints Operation timed out where Linux prints Connection timed out — same meaning. Second, a Connection reset by peer or kex_exchange_identification: Connection closed by remote host is a fourth, subtler signal: the TCP connection opened but sshd hung up during the banner/key exchange — almost always fail2ban/sshguard, MaxStartups throttling, TCP wrappers, or an sshd mid-restart. Treat it as “reached the daemon, daemon rejected the session” — closer to the refused/denied side than the timeout side.

Class 1 — Connection timed out (network / path)

A timeout means your packet died on the way in. Work the hops outside-in, cheapest check first. The order matters: 80% of real timeouts are the security group or a stale source IP, so start there.

Because so many timeouts are the inbound rule, here is that rule field by field — the exact shape of a correct SSH ingress and where each field goes wrong:

And the reason the source keeps going stale — where your public IP actually comes from and how stable it is:

The one-line proofs

# Your current public IP (the value the SG rule must contain)
curl -s https://checkip.amazonaws.com

# Does the instance even have a public address, and is it running?
aws ec2 describe-instances --instance-ids i-0abc123 \
  --query "Reservations[].Instances[].{State:State.Name,Pub:PublicIpAddress,Subnet:SubnetId,SGs:SecurityGroups[].GroupId}" \
  --output table

# Does the subnet route 0.0.0.0/0 to an IGW?
aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=subnet-0abc \
  --query "RouteTables[].Routes[?DestinationCidrBlock=='0.0.0.0/0']"

# Is there an inbound rule for tcp 22, and from which CIDR?
aws ec2 describe-security-groups --group-ids sg-0abc \
  --query "SecurityGroups[].IpPermissions[?FromPort==\`22\`]"

The fastest deterministic answer is Reachability Analyzer — it evaluates SG + NACL + route + IGW together and names the blocking component:

PATH_ID=$(aws ec2 create-network-insights-path \
  --source igw-0abc --destination i-0abc123 \
  --protocol tcp --destination-port 22 \
  --query NetworkInsightsPath.NetworkInsightsPathId --output text)

ANALYSIS=$(aws ec2 start-network-insights-analysis \
  --network-insights-path-id "$PATH_ID" \
  --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text)

aws ec2 describe-network-insights-analyses \
  --network-insights-analysis-ids "$ANALYSIS" \
  --query "NetworkInsightsAnalyses[].{Reachable:NetworkPathFound,Blocker:Explanations[0].ExplanationCode}"

NetworkPathFound: false with an ExplanationCode like ENI_SG_RULES_MISMATCH or NO_ROUTE_TO_DESTINATION points straight at the layer. (Each analysis costs a small per-run fee — see Cost & sizing.)

Class 2 — Connection refused (reached the host, port closed)

A refusal is good news for the network: the packet reached a live host that answered. The problem is now on or inside the instance. Because SSH is dead you will usually confirm these via Serial Console or SSM Session Manager, not SSH.

Most of R1–R7 come down to sshd’s own config, so keep this directive reference at hand. Read the effective config on the box with sshd -T (it resolves includes and defaults), never the raw file:

If the refusal is a brute-force defence biting your own IP (R4), know which tool is running and how to clear yourself:

The tell that separates R1–R2 from a genuine boot problem is the status checks. A refusal with 2/2 checks passing means the OS is healthy but the daemon or a firewall is the issue (R1, R2, R4, R5). A refusal with 1/2 (instance check failing) means the OS itself is unhealthy — kernel panic, corrupt fstab, misconfigured network — and you go to the Serial Console to read the boot messages:

aws ec2 describe-instance-status --instance-ids i-0abc123 \
  --query "InstanceStatuses[].{System:SystemStatus.Status,Instance:InstanceStatus.Status}"
# {"System":"ok","Instance":"impaired"}  → OS-level problem, open Serial Console

Read the two checks together — the pair localises the fault to AWS, the OS, or just “still booting”:

Class 3 — Permission denied (publickey) (reached sshd, auth failed)

You are through the network and the daemon is up. The three-term contract failed. This class has the widest fan-out and the message never says which term — ssh -vvv does.

The AMI username you must use (P1 — the single most common)

root@ almost never works; each AMI ships a default sudo-capable user. Using the wrong one produces Permission denied (publickey) with the correct key, which sends people down the wrong path for hours.

If you do not know the AMI, ssh -vvv shows Authentications that can continue: publickey, you offer the key, and the server declines — try each documented user, or read the AMI’s marketplace/vendor page. A generic probe: many AMIs print the right user in the SSH banner error, and cloud-init logs it in /var/log/cloud-init.log (readable via Serial Console).

The permission contract (P4–P6) in exact numbers

sshd’s StrictModes yes (the default) refuses a key if the surrounding permissions are too loose — even when the key itself is correct. These are the exact modes it demands:

One more denial family hides in the key itself — its type and format. A .ppk is not an OpenSSH key, and an old ssh-rsa SHA-1 key is refused by modern daemons:

Reading ssh -vvv — the single most useful skill

ssh -vvv prints every step. You do not read all of it; you read the last few lines before failure and match them to the class. This table is the decoder:

The most valuable single distinction: Offering public key … [decline] puts you in P1/P2 (wrong key or wrong user — the auth was attempted and refused), whereas Server accepts key then denied puts you in P5/P6 (the key is right, so it must be permissions/StrictModes on the server). That one fork saves the most time in the whole guide.

These are the client flags you actually reach for during triage — each one isolates or works around a specific class:

A Host key verification failed is not an auth failure — it is your known_hosts cache disagreeing with the host key, common after a re-launch reuses an IP:

The escape hatches — when SSH is dead

When you cannot fix SSH over SSH (locked-out key, sshd down, no port 22), AWS gives you four out-of-band ways onto the box. Pre-wiring the last one (SSM) before you need it is the single highest-leverage reliability move in this article.

SSM Session Manager — the one to pre-install

Session Manager reaches the OS through the SSM Agent making an outbound HTTPS (443) connection to the Systems Manager service. There is no inbound port 22, no public IP, no key — which is exactly why it keeps working when SSH is dead. The price of admission is three things, wired at launch:

# 1) Instance profile with the managed policy
aws iam create-role --role-name ssm-ec2 \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name ssm-ec2 \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam create-instance-profile --instance-profile-name ssm-ec2
aws iam add-role-to-instance-profile --instance-profile-name ssm-ec2 --role-name ssm-ec2

# 2) Attach it to a running instance (or set at launch)
aws ec2 associate-iam-instance-profile \
  --instance-id i-0abc123 --iam-instance-profile Name=ssm-ec2

# 3) Confirm the agent registered, then open a shell — no SSH needed
aws ssm describe-instance-information \
  --query "InstanceInformationList[?InstanceId=='i-0abc123'].PingStatus"
aws ssm start-session --target i-0abc123

In a private subnet with no NAT, the agent reaches the service through three interface VPC endpoints — miss one and Session Manager silently never registers:

EC2 Instance Connect and the Serial Console

EC2 Instance Connect pushes a temporary public key (valid ~60 seconds) into the instance’s authorized_keys via the agent, so you can log in without holding a long-lived key — but it still rides SSH, so the network must work (port 22 open, from your IP or the EC2_INSTANCE_CONNECT service prefix list). Use it for “I lost my key but the box is otherwise fine”:

aws ec2-instance-connect send-ssh-public-key \
  --instance-id i-0abc123 --instance-os-user ec2-user \
  --ssh-public-key file://~/.ssh/temp_key.pub
ssh -i ~/.ssh/temp_key ec2-user@<public-ip>   # within ~60 s

EC2 Instance Connect has a handful of prerequisites; when a browser “Connect” button greys out or the CLI push fails, it is almost always one of these:

Serial Console is the true out-of-band path — it attaches to the instance’s serial port (ttyS0) like a crash cart, works even when the network stack or sshd is dead, and needs no port 22. Its two gates are an account-level opt-in and a password set on the OS user (the serial console needs password auth for the interactive login):

aws ec2 enable-serial-console-access          # account-wide opt-in (once)
aws ec2 get-serial-console-access-status
# then: Console → EC2 → instance → Connect → EC2 Serial Console

Recovering a lost key (P3)

If no key works and you cannot use EIC, re-inject a key you control. Three documented methods, in order of how disruptive they are:

Because user-data normally runs only on the first boot, force it to run again by prepending the cloud-config that sets scripts-user to always:

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript
#!/bin/bash
echo "ssh-ed25519 AAAA...newpubkey you@laptop" >> /home/ec2-user/.ssh/authorized_keys
chown ec2-user:ec2-user /home/ec2-user/.ssh/authorized_keys
chmod 600 /home/ec2-user/.ssh/authorized_keys

Set that as the stopped instance’s user-data (aws ec2 modify-instance-attribute --instance-id i-x --user-data …), start it, and your key is appended on boot. Prefer AWSSupport-ResetAccess if SSM is already wired — it is the least error-prone.

Each escape hatch is gated by a specific IAM action — grant these deliberately (they all ultimately grant a shell) and log them in CloudTrail:

Architecture at a glance

The diagram traces one SSH connection left to right and pins each failure class onto the exact hop where it bites. Read it as the map you keep open during an incident: your client and private key on the left; the internet path through the IGW and route table into the public subnet; the two firewalls (stateless NACL, stateful security group); the EC2 ENI and the sshd daemon; and finally the authentication step where your key meets authorized_keys for the right OS user. The six numbered badges are the six ways it breaks — badges 1–3 are the timeout family (no path, SG block, NACL/OS-firewall drop), badge 4 is the refused family (port closed), and badges 5–6 are the denied family (wrong key/perms, wrong username) — and the legend narrates each as symptom · confirm · fix.

Real-world scenario

Northwind Freight runs a logistics API on a fleet of t3.medium instances behind an ALB, plus a single t3.small “ops box” the on-call team SSHes into to run migration scripts. For eight months nobody thought about SSH — it just worked. Then three incidents in one quarter taught the team the whole taxonomy.

Incident 1 — the Monday timeout. Every Monday at 09:00, on-call reported “can’t SSH to the ops box, connection times out.” It always cleared by lunch. The security group allowed 22 from a single /32 — the office’s static IP. But the on-call engineer worked from home on Mondays, on a residential connection whose IP rotated over the weekend. Classic T2: a stale source. curl https://checkip.amazonaws.com from the laptop showed 49.207.x.x; the SG rule said 203.0.113.5/32 (the office). The “fix” people had been using — wait until they drove to the office — was accidental. The real fix was to stop using SSH from arbitrary IPs at all: they attached an instance profile with AmazonSSMManagedInstanceCore, and on-call switched to aws ssm start-session. The inbound 22 rule was deleted entirely. Zero timeouts since, because there is no inbound path to get wrong.

Incident 2 — the refused after patching. A nightly patch job on the ops box upgraded OpenSSH and, on one reboot, sshd failed to come back because an old sshd_config line (Protocol 2 — long removed) now made sshd -t fail. SSH gave Connection refused, and because the SG was already deleted (Incident 1’s fix), the team briefly panicked that SSM had “broken networking.” But describe-instance-status showed 2/2 — the OS was healthy, only the daemon was down. They opened Serial Console, ran sshd -t, saw Unsupported option Protocol, deleted the line, systemctl restart sshd, done. Ten minutes, no instance replacement. The lesson they wrote in the runbook: refused + 2/2 checks = the daemon, not the network; go to Serial Console, not to the VPC.

Incident 3 — the denied on the new AMI. The team baked a new golden image on Ubuntu (the old one was Amazon Linux). The first person to SSH in used muscle memory — ssh -i ops.pem ec2-user@… — and got Permission denied (publickey). They spent forty minutes re-checking the key, the SG, and authorized_keys before someone noticed the AMI change. P1: Ubuntu’s user is ubuntu, not ec2-user. ssh -i ops.pem ubuntu@… connected instantly. They added a one-liner to the launch template’s tags — ssh-user: ubuntu — so the next person would not repeat it, and standardised on ssh -vvv in the runbook so “offered key, server declined” would be read as wrong user or key, not broken network, next time.

The through-line: each incident was a different class, and every wasted minute came from acting before naming the class. After they adopted “read the error string, then the status checks, then ssh -vvv” as a fixed order — and pre-wired SSM so the worst case always had a way in — mean-time-to-recovery for access problems dropped from ~40 minutes to under 5.

Advantages and disadvantages

The strategic choice hiding inside every “can’t SSH” incident is whether you should be exposing SSH at all. Direct SSH is simple and universal; SSM Session Manager removes the entire failure surface this article is about — at the cost of a dependency on the agent and IAM.

When direct SSH matters: quick labs, non-EC2 targets, environments where you cannot install an agent, and the break-glass Serial Console path (which is SSH-like but out-of-band). When SSM wins: any production or regulated workload — deleting the inbound 22 rule eliminates T1–T14 as a class, and session logging answers “who ran what” for free. Most mature shops run SSM as the default and keep Serial Console as break-glass, reserving direct SSH for narrow cases behind a bastion or EIC endpoint.

Hands-on lab

Free-tier friendly (t2.micro/t3.micro, ~15 minutes). You will (1) launch a reachable instance, (2) reproduce a timeout by removing the SG:22 rule and confirm it is the SG via Reachability Analyzer and ssh -vvv, (3) fix it, (4) reproduce a permission-denied with the wrong username and fix it, then (5) tear everything down. ⚠️ Reachability Analyzer analyses and Elastic IPs on stopped instances incur small charges — the teardown removes them.

Step 0 — variables

export AWS_REGION=ap-south-1
export AZ=ap-south-1a
KEY=ssh-lab-key

Step 1 — a key pair, VPC path, and instance (CLI)

# Key pair (private key saved locally, locked down)
aws ec2 create-key-pair --key-name $KEY \
  --query KeyMaterial --output text > ${KEY}.pem
chmod 400 ${KEY}.pem

# Use the default VPC + a public subnet in the AZ
VPC=$(aws ec2 describe-vpcs --filters Name=isDefault,Values=true \
  --query "Vpcs[0].VpcId" --output text)
SUBNET=$(aws ec2 describe-subnets \
  --filters Name=vpc-id,Values=$VPC Name=availability-zone,Values=$AZ \
  --query "Subnets[0].SubnetId" --output text)

# Security group allowing 22 from *your* current IP only
MYIP=$(curl -s https://checkip.amazonaws.com)
SG=$(aws ec2 create-security-group --group-name ssh-lab-sg \
  --description "SSH lab" --vpc-id $VPC --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol tcp --port 22 --cidr ${MYIP}/32

# Latest Amazon Linux 2023 AMI, launched into the public subnet with a public IP
AMI=$(aws ssm get-parameters \
  --names /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \
  --query "Parameters[0].Value" --output text)
IID=$(aws ec2 run-instances --image-id $AMI --instance-type t3.micro \
  --key-name $KEY --security-group-ids $SG --subnet-id $SUBNET \
  --associate-public-ip-address \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ssh-lab}]' \
  --query "Instances[0].InstanceId" --output text)
aws ec2 wait instance-running --instance-ids $IID
PUB=$(aws ec2 describe-instances --instance-ids $IID \
  --query "Reservations[0].Instances[0].PublicIpAddress" --output text)
echo "Instance $IID at $PUB"

Verify the happy path (wait ~40 s for boot). Amazon Linux 2023’s user is ec2-user:

ssh -o StrictHostKeyChecking=accept-new -i ${KEY}.pem ec2-user@$PUB 'hostname; echo OK'
# Expected: the instance hostname, then OK

Step 1 (Terraform equivalent)

provider "aws" { region = "ap-south-1" }

data "aws_vpc" "default" { default = true }
data "aws_subnets" "default" {
  filter { name = "vpc-id"  values = [data.aws_vpc.default.id] }
  filter { name = "availability-zone" values = ["ap-south-1a"] }
}
data "aws_ssm_parameter" "al2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
}
# NOTE: pin to your own /32; http data source shown for lab convenience only
data "http" "myip" { url = "https://checkip.amazonaws.com" }

resource "aws_key_pair" "lab" {
  key_name   = "ssh-lab-key"
  public_key = file("~/.ssh/ssh-lab-key.pub") # ssh-keygen -y -f ssh-lab-key.pem > ...pub
}

resource "aws_security_group" "ssh" {
  name   = "ssh-lab-sg"
  vpc_id = data.aws_vpc.default.id
  ingress {
    description = "SSH from me"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["${chomp(data.http.myip.response_body)}/32"]
  }
  egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] }
}

resource "aws_instance" "lab" {
  ami                         = data.aws_ssm_parameter.al2023.value
  instance_type               = "t3.micro"
  subnet_id                   = data.aws_subnets.default.ids[0]
  vpc_security_group_ids      = [aws_security_group.ssh.id]
  key_name                    = aws_key_pair.lab.key_name
  associate_public_ip_address = true
  tags = { Name = "ssh-lab" }
}

output "public_ip" { value = aws_instance.lab.public_ip }

Step 2 — reproduce the timeout (remove the SG:22 rule)

aws ec2 revoke-security-group-ingress --group-id $SG \
  --protocol tcp --port 22 --cidr ${MYIP}/32

ssh -vvv -o ConnectTimeout=10 -i ${KEY}.pem ec2-user@$PUB 'echo hi'
# Expected -vvv tail:
#   debug1: Connecting to <PUB> [<PUB>] port 22.
#   (long pause)
#   ssh: connect to host <PUB> port 22: Connection timed out

The hang-then-timeout, not a fast refusal, tells you a firewall silently dropped the SYN — the network class, not the daemon.

Step 3 — confirm it is the SG with Reachability Analyzer

IGW=$(aws ec2 describe-internet-gateways \
  --filters Name=attachment.vpc-id,Values=$VPC \
  --query "InternetGateways[0].InternetGatewayId" --output text)
PID=$(aws ec2 create-network-insights-path --source $IGW --destination $IID \
  --protocol tcp --destination-port 22 \
  --query NetworkInsightsPath.NetworkInsightsPathId --output text)
AID=$(aws ec2 start-network-insights-analysis --network-insights-path-id $PID \
  --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text)
sleep 8
aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $AID \
  --query "NetworkInsightsAnalyses[].{Reachable:NetworkPathFound,Code:Explanations[0].ExplanationCode}"
# Expected: Reachable=false, Code contains an SG mismatch explanation

Step 4 — fix the timeout, then break auth instead

# Fix: re-add the rule for your current IP
MYIP=$(curl -s https://checkip.amazonaws.com)
aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol tcp --port 22 --cidr ${MYIP}/32
ssh -o StrictHostKeyChecking=accept-new -i ${KEY}.pem ec2-user@$PUB 'echo network-fixed'
# Expected: network-fixed

# Now reproduce Permission denied — wrong username (this AMI is ec2-user)
ssh -vvv -i ${KEY}.pem ubuntu@$PUB 'echo hi'
# Expected -vvv tail:
#   debug1: Authentications that can continue: publickey
#   debug1: Offering public key: ssh-lab-key.pem ...
#   debug1: Authentications that can continue: publickey
#   ubuntu@<PUB>: Permission denied (publickey).

# Fix: correct username
ssh -i ${KEY}.pem ec2-user@$PUB 'echo auth-fixed'
# Expected: auth-fixed

You have now produced and fixed both headline failures and seen how -vvv distinguishes them: a silent hang (network) versus Offering public key → decline (auth).

Step 5 — teardown (⚠️ stops charges)

aws ec2 terminate-instances --instance-ids $IID
aws ec2 wait instance-terminated --instance-ids $IID
aws ec2 delete-security-group --group-id $SG
aws ec2 delete-key-pair --key-name $KEY
aws ec2 delete-network-insights-analysis --network-insights-analysis-id $AID
aws ec2 delete-network-insights-path --network-insights-path-id $PID
rm -f ${KEY}.pem
# Terraform: terraform destroy

Common mistakes & troubleshooting

This is the reference you keep open at 09:40. First the master playbook — 20 rows spanning all three classes, each with the exact confirm command and fix. Then the decode and the ordered runbook.

The 30-second decode

Checks in order (the runbook)

Do not skip ahead — each step rules out a layer cheaply before the next:

Best practices

• Prefer SSM Session Manager over inbound SSH for anything production. Deleting the inbound 22 rule eliminates the entire timeout class and gives you CloudTrail-audited, key-less, IAM-controlled access.
• Pre-wire the escape hatches before you need them: attach the SSM instance profile at launch, opt the account into Serial Console, and set an OS password on your break-glass user. During an incident is the wrong time to discover none of them are enabled.
• Never open 22 to 0.0.0.0/0. Scope it to your /32, your office CIDR, or a referenced bastion SG. If you catch yourself widening the SG to “fix” a timeout, stop — it is almost always a stale source IP.
• Use an Elastic IP for anything you SSH to by IP/DNS, so a stop-start does not silently move the address out from under you.
• Standardise on ssh -vvv in the runbook. The verbose tail is the fastest triage signal and it is free.
• Record the AMI’s SSH user in a launch-template tag (ssh-user: ubuntu) so nobody burns 40 minutes on ec2-user@ vs ubuntu@.
• chmod 400 your .pem the moment you download it, and keep keys out of shared/synced folders where a sync tool can loosen the mode.
• Prefer ed25519 keys over RSA to sidestep the OpenSSH-8.8 ssh-rsa SHA-1 deprecation entirely.
• Rotate away from long-lived keys: use EC2 Instance Connect (60-second keys) or SSM for humans, and keep the launch key as break-glass only.
• Test reachability with Reachability Analyzer in CI for critical paths, so a bad NACL or route change is caught before it pages someone.
• Log Session Manager sessions to S3/CloudWatch for audit, and put SSM VPC endpoints in private subnets so agents reach the service without a NAT gateway.

Security notes

The SSH front door is the most attacked port on the internet, so every fix here has a security dimension. Least privilege on the network: scope inbound 22 to the narrowest source you can — a /32, a bastion SG reference, or the EC2_INSTANCE_CONNECT prefix list — never 0.0.0.0/0. The moment you can, delete the inbound rule and move to SSM Session Manager, which needs no inbound access and authenticates with IAM, so access is governed by policies, MFA and SCPs rather than possession of a .pem. Least privilege on identity: the IAM actions that back these tools — ec2-instance-connect:SendSSHPublicKey, ssm:StartSession, ec2:EnableSerialConsoleAccess — are powerful (they grant shell), so gate them behind conditions (tags, source IP) and log them in CloudTrail. Key hygiene: private keys are bearer tokens — chmod 400, never commit them (a leaked .pem in git is a breach), rotate them, and prefer short-lived EIC keys for humans. Server-side: keep PasswordAuthentication no and PermitRootLogin no, run fail2ban to throttle brute force (and know it can be your lockout — R4), and keep sshd patched (the same 8.8 change that broke your old key also closed real weaknesses). Audit trail: direct SSH logs only to the box; SSM logs who opened a session, when, and optionally every command, to S3/CloudWatch — which is what an auditor actually asks for.

Cost & sizing

SSH access itself is free — you pay for the surrounding pieces, all modest.

Right-sizing guidance: for a pure ops/bastion box, t3.micro or t4g.micro (Graviton, cheaper) is plenty. If you go all-in on SSM in private subnets, the three interface endpoints (ssm, ssmmessages, ec2messages) cost more than a single NAT gateway only at very low scale — above a few instances the endpoints usually win and are more secure. Remember the 2024 change: every public IPv4 address is billed hourly, so an idle stopped instance still holding an EIP quietly costs money — the lab teardown removes it.

Interview & exam questions

Q1. A user reports Connection timed out on SSH. What are the first three things you check, and why in that order? Current source IP vs the SG rule (most common, stale My IP), then that the instance is running with a public IP, then the route to the IGW. You order them by probability and cost-to-check. A timeout is a network failure, so you never look at sshd or the key yet. (SAA-C03, SOA-C02)

Q2. Connection refused vs Connection timed out — what does each tell you about the security group? A refusal proves the security group (and NACL, route, public IP) is correct — the packet reached a host that answered with RST, so the problem is the daemon/port inside. A timeout means the packet was dropped before reaching a listening host, so the SG/NACL/route/path is the prime suspect. (SOA-C02)

Q3. Why does Permission denied (publickey) happen with the correct key, and how do you confirm the cause? The username is wrong for the AMI (e.g., ec2-user on Ubuntu, which wants ubuntu), the key isn’t in that user’s authorized_keys, or server-side permissions fail StrictModes. ssh -vvv: Offering public key → decline points to wrong key/user; Server accepts key then denied points to permissions. (DVA-C02, SAA-C03)

Q4. You must SSH to an instance in a private subnet with no public IP. Options? SSM Session Manager (outbound 443, no inbound, no public IP — the preferred answer), an EC2 Instance Connect Endpoint, or a bastion host in a public subnet. SSM also gives IAM auth and full session audit. (SAA-C03, SCS-C02)

Q5. sshd is down and you cannot SSH in. How do you get a shell to fix it? EC2 Serial Console (out-of-band, needs account opt-in + an OS password, works even with the network stack down) or SSM Session Manager if the agent is still running. Both bypass port 22. (SOA-C02)

Q6. A custom NACL allows inbound 22 but SSH still times out. What’s the classic cause? NACLs are stateless, so the reply must be explicitly allowed outbound on the ephemeral range (1024–65535) back to the client. Without it, the SYN-ACK is dropped and the client times out despite the inbound rule. (ANS-C01, SOA-C02)

Q7. What changed in OpenSSH 8.8 that breaks old keys, and how do you fix it? The ssh-rsa (RSA-with-SHA-1) signature algorithm was disabled by default. A valid RSA key can then get Permission denied / no mutual signature algorithm. Fix: -o PubkeyAcceptedAlgorithms=+ssh-rsa as a stopgap, or regenerate an ed25519 key. (DVA-C02)

Q8. How do you recover an instance whose only key is lost? AWSSupport-ResetAccess (SSM automation, if the agent/profile exist), re-inject a key via user-data forced to re-run (cloud_final_modules: [scripts-user, always]), or stop the instance, detach the root EBS volume, attach it to a rescue instance, edit authorized_keys, and reattach. (SOA-C02)

Q9. Your key works but you get WARNING: UNPROTECTED PRIVATE KEY FILE! and then denied. Why? The client refuses to use a private key with group/world-readable permissions, so no key is offered and auth fails. chmod 400 key.pem. (DVA-C02)

Q10. How do you get a single, deterministic answer to “what is blocking port 22”? VPC Reachability Analyzer: create a network-insights path from the IGW (or your ENI) to the instance on TCP 22, run the analysis, and read NetworkPathFound plus the ExplanationCode, which names the SG, NACL or route that blocks it. (ANS-C01, SOA-C02)

Q11. Too many authentication failures before you even type anything — why? Your ssh-agent is offering multiple keys and sshd cuts you off after MaxAuthTries. Force a single identity: ssh -o IdentitiesOnly=yes -i thecorrect.pem. (DVA-C02)

Q12. After re-launching an instance that reused an IP, SSH fails with Host key verification failed. Is this an AWS problem? No — the new instance has a different host key than the one cached in your known_hosts for that IP, which (correctly) trips a MITM warning. ssh-keygen -R <host> and reconnect. (SOA-C02)

Quick check

1. Your terminal prints Connection refused. Is the security group a likely cause? Why or why not?
2. Which AMI uses ubuntu, which uses ec2-user, and which uses admin as the SSH user?
3. A custom NACL allows inbound 22 but SSH times out. What single rule is probably missing?
4. Name the one access method that needs neither port 22 nor a public IP, and its two prerequisites.
5. In ssh -vvv, what distinguishes “wrong key/user” from “right key, bad server-side permissions”?

Answers

1. No. A refusal means the packet reached a host that answered with RST — so the SG (and NACL/route/public IP) let it through. The problem is the closed port / dead daemon inside, not the firewall.
2. Ubuntu → ubuntu; Amazon Linux/RHEL → ec2-user; Debian → admin. Using the wrong one yields Permission denied (publickey) with a correct key.
3. The outbound ephemeral-range rule (tcp 1024–65535 to the client’s CIDR). NACLs are stateless, so the reply must be explicitly allowed or the client times out.
4. SSM Session Manager. Prerequisites: the SSM Agent running, and an instance profile with AmazonSSMManagedInstanceCore (plus outbound 443 to the SSM endpoints).
5. Offering public key → server declines = wrong key or wrong user (auth attempted, rejected). Server accepts key then denied = the key is right, so it is server-side permissions / StrictModes.

Glossary

• Connection timed out — TCP handshake never completed; the SYN was silently dropped on the path (firewall, route, no public IP). A network failure.
• Connection refused — a reachable host answered with a TCP RST; nothing is listening on the port (or it was actively rejected). A daemon/port failure.
• Permission denied (publickey) — the TCP connection and SSH negotiation succeeded but authentication failed (wrong key/user/permissions). An auth failure.
• Security group — stateful, allow-only firewall attached to the ENI; the #1 place SSH timeouts originate.
• NACL — stateless, ordered allow+deny firewall on the subnet; needs an explicit outbound ephemeral rule for return traffic.
• Ephemeral ports — the high port range (1024–65535) a reply leaves from; must be allowed outbound on a custom NACL.
• StrictModes — sshd’s default refusal to use a key when ~/.ssh, authorized_keys or the home dir have loose permissions.
• EC2 Serial Console — out-of-band access to the instance’s serial port; works when the network/sshd is dead; needs account opt-in and an OS password.
• EC2 Instance Connect — pushes a 60-second temporary key so you can SSH without a long-lived key; still requires network reachability on 22.
• SSM Session Manager — shell access via the SSM Agent’s outbound 443; no inbound port 22, no public IP, IAM-authenticated and auditable.
• Reachability Analyzer — a VPC tool that evaluates SG + NACL + route + IGW and names the component blocking a path (e.g., to port 22).
• Status checks — the System (2/2) and instance reachability probes; 1/2 signals an OS-level problem to inspect via Serial Console.
• ssh-rsa deprecation — OpenSSH 8.8+ disabled the RSA-SHA-1 signature by default; old RSA keys may then be denied.
• AWSSupport-ResetAccess — an SSM automation document that resets/creates a key or password to recover a locked-out instance.

Next steps

• Master the firewall layer this article touches on port 22 in the full Security Groups vs NACLs: A Deep Dive and Troubleshooting Guide.
• Widen the lens from one port to the whole VPC with the AWS VPC Connectivity Troubleshooting Playbook.
• If you have not built the working case yet, do the Launch Your First EC2 Instance and SSH In walkthrough, then break and fix it with this guide.
• Trade inbound SSH for auditable, key-less access by standardising on SSM Session Manager across your fleet.
• Add a Reachability Analyzer check to CI for critical SSH/RDP paths so a bad NACL or route change is caught before it pages you.