Linux Networking: IP Addressing, ip/nmcli, Routing, DNS & ss/ping Troubleshooting

Every Linux engineer eventually stares at a box that “can’t reach the internet” and has to decide, fast, which of half a dozen things is actually broken: the interface, the IP, the route, DNS, the remote service, or a firewall silently eating packets. The people who fix it in ninety seconds are not smarter — they just walk a fixed ladder, one layer at a time, with a small set of modern tools. This lesson gives you that ladder and those tools.

By the end you will be fluent in the ip suite (which has replaced ifconfig/route/netstat), able to configure a persistent static address with nmcli, able to read a routing table and a dig answer without guessing, and able to decode ss -tulpn column by column. Most importantly, you will own a repeatable troubleshooting sequence that turns “the network is down” — a sentence that means nothing — into “the default route is missing,” which means everything.

This lesson assumes you have used a terminal and understand files and processes. It does not assume you remember subnetting. We start there.

---

Why this matters

Networking is where beginners feel the most helpless, because the failure is invisible: nothing is on fire, there is no error in a log you know to read, a command just hangs. The instinct is to reboot, or to blame “the network” — a word that covers seven different subsystems. That instinct wastes hours and teaches you nothing.

The cure is a mental model of the path a packet takes and a fixed order to test each step. When you type curl https://api.example.com, your machine must (1) turn the name into an IP address using DNS, (2) decide which route and gateway that IP goes through, (3) hand the packet to an interface that owns a usable address, (4) get it across the local wire to the next hop, and (5) have the far end actually answer on that port. Each of those five steps has exactly one or two commands that prove it works or shows you where it broke. Learn the path, learn the five checks, and you never guess again.

The second reason this matters: the tooling changed, and a lot of the internet’s advice is a decade stale. ifconfig, route, netstat, and arp are the deprecated net-tools package — they still exist on some systems, are missing on others, and do not even show you the full picture on a modern box (they can miss secondary addresses and modern socket state). The replacement is iproute2 (ip and ss) plus NetworkManager (nmcli) for persistence. This lesson teaches the modern tools first and gives you a translation table so you can still read the old commands when you meet them in someone else’s runbook.

Get this right and “the network is down” becomes a diagnosis, not a shrug.

---

TCP/IP fast, for people who will use it

You do not need a semester of networking theory to operate Linux. You need a working command of six ideas: the layers, the IP address, the subnet mask expressed as CIDR, the gateway, ports, and TCP vs UDP. Here they are, pitched at using them.

The layers you actually touch

Networking is built in layers so each one can ignore the details of the others. The formal seven-layer OSI model is worth knowing by name because interviewers and error messages reference “layer 3” or “layer 7”, but on Linux you work with a simpler four-layer stack. The single most useful habit is to map a symptom to a layer, because the tool you reach for depends on it.

When a name won’t resolve, that is layer 7 (DNS). When you have an IP but can’t reach another subnet, that is layer 3 (routing). When the link light is off, that is layer 1/2. Naming the layer is half the fix.

The IP address and the subnet mask (CIDR)

An IPv4 address is 32 bits, written as four decimal octets, 192.168.1.50. Each octet is 8 bits (0–255). An address alone is meaningless without knowing which part identifies the network and which part identifies the host — that split is the job of the subnet mask.

Modern Linux writes the mask as CIDR (Classless Inter-Domain Routing): a slash and a number, 192.168.1.50/24. The number is how many leading bits are the network. /24 means the first 24 bits (192.168.1) are the network and the last 8 bits (.50) are the host. Every host on the same subnet shares the network bits and can talk directly (layer 2); anything else must go through a gateway (layer 3).

The mask determines how many hosts fit on the subnet. Usable hosts = 2^(32 − prefix) − 2, because the all-zeros address is the network address and the all-ones address is the broadcast — neither is assignable to a host. This is the table you will consult for the rest of your career:

Two special cases people trip on: a /31 is legal for point-to-point links (RFC 3021) and gives you both addresses as usable, and a /32 is a single host — you will see /32 all the time in routing (ip route get output, host routes, VPN endpoints).

Certain ranges are private (RFC 1918) — never routed on the public internet — plus a few special ranges you must recognise on sight:

Burn 169.254.x.x into memory: if ip addr shows an interface with a 169.254 address, DHCP did not answer and the host gave itself a link-local address. That is a diagnosis, not an address you configured.

Ports and TCP vs UDP

An IP address gets a packet to a host; a port (a 16-bit number, 0–65535) gets it to the right program on that host. A web server listens on port 80/443, SSH on 22, DNS on 53. A connection is uniquely identified by the four-tuple: source IP, source port, destination IP, destination port. That is why one server can hold thousands of simultaneous connections on port 443 — each client uses a different source port.

The two transport protocols behave completely differently, and knowing which one a service uses tells you how to troubleshoot it:

The practical consequence: a TCP port either completes a handshake (open), gets a RST (refused — nothing listening), or times out (a firewall dropped it). A UDP port gives you no such feedback — silence could mean “open and didn’t reply” or “dropped.” That is why UDP services are harder to test, and why nc -zv on a UDP port is nearly useless.

Well-known ports you should recognise instantly (the full map lives in /etc/services, which we cover under sockets):

We will use the diagram of the whole outbound path once we have the ip commands to make it concrete, in the next section.

---

Interfaces and addresses: the modern ip suite

Everything about the local network state — interfaces, addresses, routes, and the ARP/neighbour cache — is read and written with a single command family: ip, from the iproute2 package. It has a consistent grammar: ip OBJECT COMMAND, where the object is link (layer-2 interface), addr (layer-3 address), route (routing table), or neigh (the ARP cache).

Reading the current state

Three commands show you almost everything. Learn the -br (brief) and -c (colour) flags — they turn a wall of text into a scannable table.

# Layer 2: interfaces and their up/down state
ip link show
# Brief, one line per interface:
ip -br link

lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0             UP             52:54:00:a1:b2:c3 <BROADCAST,MULTICAST,UP,LOWER_UP>

# Layer 3: addresses assigned to each interface
ip addr show          # full form
ip -br addr           # brief; 'ip a' is the common shorthand

lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             192.168.1.50/24 fe80::5054:ff:fea1:b2c3/64

The <BROADCAST,MULTICAST,UP,LOWER_UP> flags are the ones that matter for troubleshooting. UP means you administratively enabled the interface; LOWER_UP means the physical layer has carrier (cable plugged in, link negotiated). Both must be present for traffic to flow:

The neighbour (ARP) cache maps IP addresses on the local subnet to MAC addresses — this is layer 2’s phone book:

# Show the ARP / neighbour table (replaces 'arp -n')
ip neigh show

192.168.1.1 dev eth0 lladdr 00:1a:2b:3c:4d:5e REACHABLE
192.168.1.20 dev eth0 lladdr 00:1a:2b:3c:4d:60 STALE

The state on the right tells you the health of that mapping: REACHABLE (freshly confirmed), STALE (known but needs re-checking), FAILED (ARP got no answer — the host is gone or wrong), INCOMPLETE (mid-lookup), PERMANENT (static entry). A gateway stuck at FAILED is a strong signal your layer-2 path to the router is broken.

Here is the reference for the commands you will use daily:

Changing addresses and links at runtime

You can bring a link up or down and add or remove addresses instantly with ip. These changes are runtime-only — they live in the kernel until the next reboot or until NetworkManager overwrites them. That is fine for a quick test and dangerous for a permanent change; we make it persistent with nmcli in the next section.

# Bring the interface down and back up
sudo ip link set dev eth0 down
sudo ip link set dev eth0 up

# Add a second/temporary address, then verify
sudo ip addr add 192.168.1.51/24 dev eth0
ip -br addr show eth0

eth0             UP             192.168.1.50/24 192.168.1.51/24 fe80::5054:ff:fea1:b2c3/64

# Remove the temporary address again
sudo ip addr del 192.168.1.51/24 dev eth0

⚠️ If you are connected over SSH, never ip link set eth0 down or delete the address you are logged in through — you will cut your own session and, with no persistent config to restore it on reboot, may lock yourself out. Test destructive network changes from console/KVM access, or wrap them so they auto-revert.

The outbound path these commands sit on is worth seeing as one picture. When you run curl https://example.com, the request flows left to right: the app resolves the name to an IP via DNS, the kernel picks a route (usually the default gateway) for that IP, the packet leaves through an interface that owns an address, and NetworkManager is the thing that persisted that address, route, and DNS so they survived the last reboot.

Keep that path in your head; the troubleshooting ladder later in this lesson walks it rung by rung.

The deprecated net-tools → modern iproute2 translation

You will meet ifconfig, route, netstat, and arp in old blog posts, runbooks, and minimal container images. They come from the net-tools package, which is unmaintained and not installed by default on modern distros. Learn to translate them on sight — and prefer the right column in anything you write:

The rule of thumb: anything you used to do with ifconfig/route/arp, do with ip; anything you used to do with netstat, do with ss. They are faster, show more (multiple addresses, namespaces, socket internals), and are present on every modern system.

---

Making it stick: NetworkManager, nmcli, netplan, systemd-networkd

Runtime ip changes vanish on reboot. To make configuration persistent you write it into whatever your distro uses to manage the network at boot. On the great majority of desktops and RHEL/Fedora/Rocky servers that is NetworkManager, driven from the command line by nmcli. Ubuntu Server uses netplan (which usually renders down to either NetworkManager or systemd-networkd), and minimal/cloud images often use systemd-networkd directly. Know all three; live in nmcli.

The management systems, per distro

Do not mix two of these on one interface — if NetworkManager and a stray /etc/network/interfaces entry both claim eth0, you get a flapping, unpredictable interface. Pick the one your distro ships and stay in it.

nmcli: the commands that matter

nmcli has two objects you use constantly: device (the physical/virtual interface) and connection (a saved profile of settings). A device is the hardware; a connection is a named configuration you can attach to it. One device can have several connection profiles (home, office, static-lab) and you activate one at a time.

# What devices exist and are they connected?
nmcli device status

DEVICE  TYPE      STATE                   CONNECTION
eth0    ethernet  connected               Wired connection 1
lo      loopback  connected (externally)  lo

# What connection profiles exist? (--active shows only live ones)
nmcli connection show
nmcli connection show --active

# Deep detail on one device or connection
nmcli device show eth0
nmcli connection show "Wired connection 1"

Here is the working reference:

Setting a static IP + gateway + DNS

This is the single most common real task: give a server a fixed address. One nmcli connection add does it all — address, gateway, DNS, and “manual” (static) method — and it persists immediately:

# Create a static profile named 'office' on eth0
sudo nmcli connection add type ethernet con-name office ifname eth0 \
  ipv4.method manual \
  ipv4.addresses 192.168.1.50/24 \
  ipv4.gateway 192.168.1.1 \
  ipv4.dns "1.1.1.1 8.8.8.8"

# Activate it
sudo nmcli connection up office

To modify an existing profile instead of creating one, use connection modify. Note the +/- prefixes to append or remove a value from a list (like DNS) without retyping the whole list:

# Change the address on an existing profile
sudo nmcli connection modify office ipv4.addresses 192.168.1.60/24

# Add a third DNS server without replacing the others
sudo nmcli connection modify office +ipv4.dns 9.9.9.9

# Re-apply the changes (needed after modify)
sudo nmcli connection up office

The key properties you set on a connection:

DHCP vs static: which and when

DHCP (Dynamic Host Configuration Protocol) means a server hands your host its address, gateway, and DNS on boot — zero config, but the address can change. Static means you pin all of it yourself — stable, but you own the correctness.

Rule: workstations and autoscaling instances get DHCP; servers that other things connect to get static (or a DHCP reservation, which is a static lease pinned to a MAC on the DHCP server).

The Ubuntu and minimal-image cases

On Ubuntu Server, the file you edit is netplan YAML in /etc/netplan/. Netplan is a front-end that renders to NetworkManager or systemd-networkd (the renderer: line). The same static config looks like this:

# /etc/netplan/01-eth0.yaml   (YAML — indentation matters, use spaces)
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: false
      addresses: [192.168.1.50/24]
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]

# Validate + apply with an automatic rollback safety net
sudo netplan try      # applies, then reverts in 120s unless you confirm
sudo netplan apply    # applies immediately

Note the modern routes: block — the old gateway4: key is deprecated and warns on newer netplan. Always prefer netplan try over netplan apply on a remote box: it auto-reverts if you don’t confirm, so a bad config won’t strand you.

On minimal/cloud images that use systemd-networkd directly, you drop a .network file:

# /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
Address=192.168.1.50/24
Gateway=192.168.1.1
DNS=1.1.1.1

sudo networkctl reload         # re-read config
networkctl status eth0         # verify

systemd-networkd is part of systemd; if you want the mental model of how systemd-networkd.service and systemd-resolved.service are managed as units, see the lesson on systemd units, services, targets & journald.

---

Routing: how a packet finds its way out

Once your host has an IP, the routing table decides where every outbound packet goes. The rule the kernel follows is simple and absolute: for a destination IP, pick the matching route with the longest prefix (most specific), and if nothing else matches, use the default route (0.0.0.0/0). Read the table with ip route:

ip route show      # or just: ip r

default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.50 metric 100
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.50 metric 100

Two routes here, and reading them is a core skill:

• default via 192.168.1.1 dev eth0 — anything not matched by a more specific route goes via the gateway 192.168.1.1, out interface eth0. This is the line people check first; if it’s missing, you can reach your own LAN but nothing beyond it.
• 192.168.1.0/24 dev eth0 … scope link — the directly-connected subnet. scope link means “these hosts are on the same wire, no gateway needed.” The kernel created this automatically (proto kernel) when you assigned the address.

Decode every field:

Two of those fields tell you why a route exists. The proto records who created it: kernel (added automatically the moment you assigned an address to a directly-connected subnet), dhcp (learned from the DHCP client), static (you or a config file put it there), ra (an IPv6 router advertisement), or boot (set at boot from an unrecorded source). The scope records how far the destination is: global for anything reachable through a gateway, link for hosts on the directly-connected wire (no gateway needed), and host for the machine itself (the lo routes). So a line reading proto kernel scope link is just “the route the kernel built for my own subnet,” and proto dhcp on the default route means “my gateway was handed to me by DHCP, not configured by me” — a useful clue when a route you didn’t set keeps reappearing.

Adding routes and the “which route wins” question

Add a static route to reach a subnet that lives behind a different router than your default gateway:

# Reach 10.0.0.0/24 via the router at 192.168.1.254 (runtime only)
sudo ip route add 10.0.0.0/24 via 192.168.1.254 dev eth0

# Add/replace the default route
sudo ip route add default via 192.168.1.1
sudo ip route replace default via 192.168.1.1   # replace if one exists

To make a static route persistent, add it to the connection profile rather than running ip route by hand: nmcli connection modify office +ipv4.routes "10.0.0.0/24 192.168.1.254".

The most useful routing command for troubleshooting is ip route get, which asks the kernel to simulate its decision for one destination and show you exactly what it will do:

ip route get 8.8.8.8

8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.50 uid 1000
    cache

That one line answers “which gateway, which interface, and which source address will this host use to reach 8.8.8.8?” — no guessing. If it says via your gateway and the right dev, routing is fine and you look elsewhere.

A word on policy routing

The single routing table above is the common case, but Linux supports policy routing: multiple routing tables plus rules (ip rule) that choose a table based on the packet’s source address, mark, or interface — not just its destination. You reach for it when a host has two uplinks and traffic that arrives on interface B must leave on interface B. The building blocks are ip rule (the selector) and ip route add … table 100 (a numbered table listed in /etc/iproute2/rt_tables):

# Traffic from 10.0.5.0/24 uses table 100, which has its own default gateway
sudo ip route add default via 10.0.5.1 dev eth1 table 100
sudo ip rule add from 10.0.5.0/24 table 100
ip rule show      # list the rule set, in priority order

You will not need this on day one, but recognise the symptom it solves — “replies leave the wrong interface on a dual-homed box” — so you know what to search for.

---

DNS resolution: from name to address

Humans use names; the network uses addresses. DNS (Domain Name System) is the translation, and on Linux the order in which your host tries to resolve a name is configured, layered, and a frequent source of “works on that box, not this one” bugs. Four files/services are in play.

The files and their order

Resolution starts at /etc/nsswitch.conf, whose hosts: line lists sources in order. A representative modern line:

# /etc/nsswitch.conf
hosts:  files resolve [!UNAVAIL=return] dns myhostname

Read left to right: try files (/etc/hosts) first; then resolve (systemd-resolved); the [!UNAVAIL=return] means “if resolved answered at all — even NXDOMAIN — stop here and return”; only if resolved is unavailable fall through to dns (classic /etc/resolv.conf lookup); myhostname resolves the local hostname. The bracketed action syntax is worth knowing:

The classic /etc/hosts and /etc/resolv.conf still exist and still work:

# /etc/hosts  — name wins over DNS; great for pinning or blocking a name
127.0.0.1   localhost
192.168.1.50 web01 web01.corp.example.com

# /etc/resolv.conf  — on a systemd-resolved box this is a symlink:
# /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search corp.example.com

That nameserver 127.0.0.53 surprises people: it is not your router — it is the systemd-resolved stub resolver running locally, which forwards to the real upstream servers it learned from DHCP/nmcli. To see the real servers, ask resolved:

resolvectl status          # shows real upstream DNS per interface
resolvectl query example.com   # resolve through resolved, with source info
resolvectl statistics      # cache hits/misses
sudo resolvectl flush-caches   # clear the DNS cache (fixes stale results)

⚠️ Do not hand-edit /etc/resolv.conf when it is a symlink to systemd-resolved or NetworkManager — your changes are silently overwritten on the next network event. Set DNS through nmcli (ipv4.dns) or resolved instead.

Looking things up: dig, host, nslookup

To query DNS yourself you have three classic tools plus the systemd one. dig is the professional’s choice — verbose, precise, and scriptable with +short (e.g. dig example.com A); reach for it whenever you are debugging. host (e.g. host example.com) is the quick, human-readable one-liner. nslookup (e.g. nslookup example.com) is the old cross-platform standby, familiar to Windows admins and still handy in its interactive mode. And resolvectl query example.com is the one that uses the system resolver path — it respects /etc/nsswitch.conf, /etc/hosts, and the systemd-resolved cache, so its answer matches what your applications actually see.

A crucial distinction: dig talks straight to a DNS server and ignores /etc/hosts and nsswitch, while getent hosts <name> and resolvectl query use the system path (hosts file, cache, the works). When dig finds a name but your app can’t, the difference is nsswitch or /etc/hosts — that comparison is a diagnostic in itself.

Reading a dig answer is a skill. Here is a full response, annotated:

dig example.com A

; <<>> DiG 9.18.18 <<>> example.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4321
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;example.com.            IN  A

;; ANSWER SECTION:
example.com.    3512    IN  A   93.184.216.34

;; Query time: 24 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Jul 09 10:14:02 UTC 2026

The parts that matter:

• status: NOERROR — the query succeeded. NXDOMAIN means “no such name”; SERVFAIL means the server broke or DNSSEC failed.
• ANSWER SECTION — the payload. Read it as: example.com. (the name) has a TTL of 3512 seconds (how long you may cache it), class IN (internet), type A (IPv4 address), value 93.184.216.34. (The exact address for any given name changes over time — read the structure, not the number.)
• SERVER: 127.0.0.53#53 — which resolver answered: here, the local systemd-resolved stub.

Common query forms you will actually type:

dig +short example.com          # just the address(es) — perfect for scripts
dig example.com MX              # mail servers
dig AAAA example.com            # IPv6 address
dig -x 93.184.216.34            # reverse lookup (IP → name, PTR record)
dig @1.1.1.1 example.com        # ask a SPECIFIC server, bypassing local config
dig +trace example.com          # walk the delegation from the root down

That @1.1.1.1 trick is gold for troubleshooting: if dig @1.1.1.1 example.com works but dig example.com (which uses your configured server) fails, your local DNS config or server is the problem, not DNS itself. The record types you’ll meet most:

---

Ports and sockets: ss and what’s listening

When you need to know what is listening on this machine, who a process is talking to, or why a port connection is refused, the tool is ss (socket statistics) — the modern replacement for netstat. The one incantation to memorise is ss -tulpn, and you should be able to decode every letter and every column.

ss -tulpn, letter by letter

sudo ss -tulpn

Netid State  Recv-Q Send-Q Local Address:Port  Peer Address:Port Process
udp   UNCONN 0      0      127.0.0.53%lo:53     0.0.0.0:*         users:(("systemd-resolve",pid=712,fd=13))
udp   UNCONN 0      0      0.0.0.0:68           0.0.0.0:*         users:(("NetworkManager",pid=690,fd=22))
tcp   LISTEN 0      128    0.0.0.0:22           0.0.0.0:*         users:(("sshd",pid=980,fd=3))
tcp   LISTEN 0      511    0.0.0.0:80           0.0.0.0:*         users:(("nginx",pid=1120,fd=6))
tcp   LISTEN 0      128    [::]:22              [::]:*            users:(("sshd",pid=980,fd=4))

Each flag does one thing:

Now decode the columns — the two that confuse everyone are Recv-Q/Send-Q, whose meaning depends on the state:

A subtle but vital read: Local Address tells you the exposure. 0.0.0.0:80 means nginx accepts connections from any network — reachable from outside. 127.0.0.1:5432 means Postgres listens on localhost only — no remote client can reach it, which is either correct hardening or the reason your app “can’t connect to the database.” That single distinction resolves a huge fraction of “connection refused” tickets.

Established connections, states, and /etc/services

Drop -l (or use -a) to see live connections, and add a state filter to zero in:

# Every established TCP connection
ss -tn state established

# Everything talking to port 443
ss -tn dport = :443

# Summary of socket counts by state
ss -s

TCP connections move through a state machine; recognising the states saves you from false alarms:

CLOSE-WAIT piling up is the classic fingerprint of an application that leaks sockets (forgets to close()); TIME-WAIT in the thousands points to a client hammering short-lived connections. Neither is a network fault — ss tells you it’s the app.

The mapping between port numbers and service names lives in /etc/services. That is what lets ss (without -n) print ssh instead of 22, and what getent services reads:

grep -E '^\w+\s+(22|80|443|53)/' /etc/services

ssh             22/tcp
domain          53/tcp
domain          53/udp
http            80/tcp
https           443/tcp

/etc/services is purely a label file — editing it does not change what any service listens on; it only changes how tools display the number. Use -n with ss and you bypass it entirely.

---

The connectivity troubleshooting ladder

This is the payoff. When something “can’t connect,” do not guess. Walk this ladder from the bottom up; each rung isolates one layer, and the first rung that fails is your answer. Every step maps to a command and a plain question.

Steps 4 and 5 together are the sharpest single test: ping 8.8.8.8 succeeds but ping google.com fails ⇒ the network is fine and DNS is broken. That one observation eliminates half the possibilities in one command.

Running the ladder

# Rung 1 — address present? (watch for 169.254 or an empty result)
ip -br addr show eth0

# Rung 2 — is there a 'default via …' line?
ip route

# Rung 3 — the gateway (get its IP from 'ip route')
ping -c3 192.168.1.1

# Rung 4 — the internet by IP, so DNS is not involved
ping -c3 8.8.8.8

# Rung 5 — resolution; @1.1.1.1 tests DNS independently of your config
dig +short example.com
dig +short @1.1.1.1 example.com

# Rung 6 — the real service and port, verbosely
curl -v https://example.com
nc -zv example.com 443

# Rung 7 — where it dies, hop by hop
mtr -rwc10 example.com

ping is the layer-3 workhorse; its output tells you reachability and latency and loss:

ping -c3 8.8.8.8

64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=11.4 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=10.9 ms
3 packets transmitted, 3 received, 0% packet loss, time 2003ms

0% packet loss = solid; intermittent loss points at a flaky link or an overloaded hop. Note that some hosts and firewalls block ICMP, so a failed ping to a server isn’t proof it’s down — but a failed ping to your own gateway almost always is a real problem.

The deeper tools: traceroute, mtr, tcpdump, nc, curl -v

When the ladder points at “somewhere in the path” or “the service,” these four tools tell you exactly where and why:

traceroute defaults to UDP probes on Linux; if they get filtered midway you’ll see * * *. Switch to -T (TCP SYN to port 80/443) or -I (ICMP) when a firewall blocks the default. mtr is usually better — it’s traceroute and ping combined, running continuously, so you can see which hop introduces loss:

mtr -rwc10 example.com     # -r report mode, -w wide, -c10 ten cycles

tcpdump shows you the packets themselves — the ground truth when higher-level tools disagree. A few surgical examples:

sudo tcpdump -ni eth0 port 53             # watch DNS queries/replies leave
sudo tcpdump -ni any host 8.8.8.8         # anything to/from 8.8.8.8, any iface
sudo tcpdump -ni eth0 'tcp port 80'       # HTTP packets
sudo tcpdump -ni eth0 -c5 icmp            # capture 5 ping packets and stop

⚠️ tcpdump can capture sensitive data in the clear (credentials, tokens on plain HTTP). Capture the narrowest filter you need, prefer writing to a file (-w cap.pcap) for offline analysis, and delete captures when done.

nc (netcat, often ncat on RHEL) is the fast port tester and a mini-server. -z scans without sending data, -v is verbose, -l listens:

nc -zv example.com 443        # is TCP 443 open? → 'succeeded' or 'refused'/timeout
nc -zv example.com 22
# On a second host, a throwaway listener to test connectivity end to end:
nc -l 9000                    # host A listens; host B: nc -zv hostA 9000

curl -v is the best single test of an actual web service because it narrates every step — DNS, TCP connect, TLS handshake, request, response:

curl -v https://example.com 2>&1 | head -20

*   Trying 93.184.216.34:443...
* Connected to example.com (93.184.216.34) port 443
* TLS handshake, TLSv1.3 ...
* Server certificate: CN=example.com
> GET / HTTP/2
> Host: example.com
< HTTP/2 200
< content-type: text/html

Read the leading symbols: * is curl’s own commentary (connection, TLS), > is what curl sent, < is what the server replied. If it stalls at Trying … it’s a network/firewall problem; if the TLS line errors it’s a certificate problem; if you get a < response line the transport is fine and any error is application-level.

Firewall awareness: the silent drop

One failure mode deserves special mention because it looks like a network fault but isn’t. When nc -zv host 443 or curl hangs and eventually times out with no “connection refused,” that silence is the signature of a firewall DROP — the packet was thrown away with no reply. Contrast the two outcomes:

• Connection refused (a TCP RST came back) ⇒ you reached the host, but nothing is listening on that port (or the firewall is set to reject). The service is down.
• Connection timed out (no reply at all) ⇒ a firewall dropped the packet — firewalld, nftables/iptables on the host, or a cloud security group / NSG in front of it. The service may be perfectly healthy behind the wall.

On the local host, firewalld and nftables are the usual suspects; check with sudo nft list ruleset or sudo firewall-cmd --list-all. The mechanics of reading and writing those rules are a lesson of their own — see Linux firewalls: firewalld, nftables & iptables. The point here is diagnostic: timeout = dropped (suspect a firewall); refused = reached but nothing listening (suspect the service). That single fork saves hours.

---

Hands-on lab

Do this on a throwaway VM, WSL, or a container (docker run -it --rm --cap-add=NET_ADMIN debian:stable bash, then apt-get update && apt-get install -y iproute2 iputils-ping dnsutils netcat-openbsd curl network-manager traceroute mtr-tiny). ⚠️ Do not run the interface/route-changing steps on a remote box you only reach over SSH — you can cut yourself off. Every step tells you what to expect and what just happened.

Step 1 — Inventory the network state.

ip -br -c addr        # interfaces + addresses, coloured
ip route              # the routing table
ip neigh              # the ARP cache

You should see at least lo and one real interface with an address, a default via … route, and one or two neighbours. What just happened: you read all three layers of local network state with three commands — the muscle memory you’ll use forever.

Step 2 — Prove the route decision.

ip route get 1.1.1.1

Expect one line naming the via gateway, the dev, and the src address. What just happened: you asked the kernel to show its actual routing decision for a destination — no guessing which gateway is used.

Step 3 — Add and remove a temporary address.

sudo ip addr add 10.99.99.2/24 dev eth0
ip -br addr show eth0          # now shows 10.99.99.2/24 as a second address
sudo ip addr del 10.99.99.2/24 dev eth0

What just happened: you added a runtime address and removed it. Because it was never written to a profile, a reboot would have erased it — this is the runtime-vs-persistent distinction in action.

Step 4 — Walk the DNS path.

cat /etc/resolv.conf                    # who do we ask? (maybe 127.0.0.53)
getent hosts example.com                # resolve via the SYSTEM path (hosts+nsswitch)
dig +short example.com                  # resolve via DNS directly
dig +short @1.1.1.1 example.com         # resolve via a specific public server

Compare the three results — they should agree. What just happened: you saw the difference between the system resolver path (getent, honours /etc/hosts) and a direct DNS query (dig), which is exactly the comparison that isolates DNS bugs.

Step 5 — Pin a name with /etc/hosts, then prove precedence.

echo '203.0.113.10 lab.test' | sudo tee -a /etc/hosts
getent hosts lab.test           # returns 203.0.113.10 — from the file, no DNS
dig +short lab.test             # empty — dig ignores /etc/hosts
sudo sed -i '/lab.test/d' /etc/hosts   # clean up

What just happened: you proved /etc/hosts beats DNS for the system resolver, and that dig bypasses it — the mechanism behind “getent finds it but dig doesn’t.”

Step 6 — See what’s listening.

sudo ss -tulpn

Note each Local Address:Port and its process. Spot which are 0.0.0.0 (exposed) vs 127.0.0.1 (localhost-only). What just happened: you inventoried every open port and its owner — the first thing to check when a service is “unreachable.”

Step 7 — Test a port two ways.

nc -zv example.com 443     # expect: succeeded
nc -zv example.com 444     # expect: timeout (dropped) or refused

What just happened: you saw the two failure signatures first-hand — an open port succeeds, a filtered port times out — the fork that tells firewall from service-down.

Step 8 — Run the full ladder against a real target.

ip -br addr && ip route
ping -c2 $(ip route | awk '/default/{print $3; exit}')   # ping the gateway
ping -c2 8.8.8.8
dig +short example.com
curl -sI https://example.com | head -1                   # expect: HTTP/2 200
mtr -rwc5 example.com

What just happened: you executed the entire troubleshooting ladder end to end. When a real outage hits, you run these same seven commands and the first failure is your diagnosis.

---

Common mistakes and troubleshooting

The table below is the “no connectivity” decision matrix — match your symptom, check the cause, apply the fix.

Three gotchas cause more lost hours than all the rest combined:

Runtime changes that vanish on reboot. You fix an outage with ip addr add / ip route add, everything works, you close the ticket — and the next reboot brings it all back. ip writes to the running kernel, not to disk. Any change you want to keep must go into nmcli (or netplan/networkd). Treat ip addr add/ip route add as diagnosis, and the equivalent nmcli as the fix.

Editing /etc/resolv.conf when it’s a symlink. On systemd-resolved and NetworkManager systems, /etc/resolv.conf is a symlink that is regenerated automatically. You edit it, DNS works, you feel clever — then the next DHCP renewal or nmcli con up overwrites your change and DNS “randomly” breaks. Check with ls -l /etc/resolv.conf; if it’s a symlink, set DNS through nmcli ipv4.dns or resolved, never by editing the file.

Confusing “refused” with “timed out.” These are opposite diagnoses and people conflate them into “it doesn’t work.” Refused (RST) means you reached the host and nothing is listening — a service problem. Timed out (silence) means a firewall ate the packet. Reading which one you got points you at the right subsystem immediately; ignoring the distinction sends you debugging the wrong machine.

---

Cheat-sheet

---

Interview and exam questions

Q: What replaced ifconfig, route, and netstat, and why should you prefer the replacements? A: The iproute2 suite — ip (for ifconfig/route/arp) and ss (for netstat). They come from the actively maintained iproute2 package, are installed by default on modern distros, show information the old net-tools misses (multiple addresses per interface, network namespaces, deep socket state), and are faster. net-tools is deprecated and often not even installed.

Q: A host shows 169.254.14.7 on eth0. What does that tell you? A: It’s an APIPA / link-local address (169.254.0.0/16) that the host self-assigned because DHCP got no answer. The DHCP server is unreachable — check the cable/switch, the DHCP service, or the VLAN. It is never an address you’d configure deliberately.

Q: ping 8.8.8.8 works but ping google.com fails. Diagnosis? A: The network path is fine (you reached a public IP), so the problem is DNS resolution. Check /etc/resolv.conf, resolvectl status, and try dig @1.1.1.1 google.com to test DNS independently of your configured server.

Q: In ss -tulpn, what do the flags mean and what does Local Address 0.0.0.0:80 vs 127.0.0.1:80 tell you? A: -t TCP, -u UDP, -l listening, -p process, -n numeric. 0.0.0.0:80 listens on all interfaces (reachable remotely); 127.0.0.1:80 listens on localhost only (no remote client can connect). The latter is a common reason a service is “unreachable from outside.”

Q: Configure a static IP 192.168.10.20/24, gateway 192.168.10.1, DNS 1.1.1.1 on eth0, persistently. (RHCSA-style) A: nmcli con add type ethernet con-name static-eth0 ifname eth0 ipv4.method manual ipv4.addresses 192.168.10.20/24 ipv4.gateway 192.168.10.1 ipv4.dns 1.1.1.1 then nmcli con up static-eth0. Verify with ip addr, ip route, and resolvectl status.

Q: What’s the difference between a connection “refused” and “timed out”? A: Refused = a TCP RST returned; you reached the host but nothing is listening on that port (service down, or a reject firewall rule). Timed out = no reply at all; a firewall silently dropped the packet (host firewall or cloud security group). Refused points at the service; timeout points at a firewall.

Q: Read this route line: default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.5 metric 100. A: It’s the default route (matches any destination not matched more specifically). Traffic goes to gateway 10.0.0.1 out interface eth0; the route was learned from DHCP; packets get source address 10.0.0.5; metric 100 is the tie-breaker (lower wins if two default routes exist).

Q: How does the kernel choose among multiple matching routes? A: Longest prefix match — the most specific route (largest CIDR prefix) wins. If two routes are equally specific, the lower metric wins. ip route get <dest> shows the actual decision for any destination.

Q: What is /etc/nsswitch.conf and how does it relate to /etc/hosts and DNS? A: It defines the order of name-resolution sources. The hosts: line (e.g. files resolve dns) means the resolver checks /etc/hosts (files) first, then systemd-resolved (resolve), then classic DNS (dns). It’s why a name in /etc/hosts overrides DNS — and why dig (which talks straight to DNS) can disagree with what your application resolves.

Q: Why might editing /etc/resolv.conf “not stick”? A: On systemd-resolved / NetworkManager systems it’s a symlink to an auto-generated file (/run/systemd/resolve/stub-resolv.conf) and is regenerated on network events, overwriting manual edits. Set DNS via nmcli ipv4.dns or resolved instead. Check with ls -l /etc/resolv.conf.

Q: You add an address with ip addr add and it’s gone after reboot. Why, and what’s the fix? A: ip changes the running kernel state only — nothing is written to disk. Make it persistent by putting it in a NetworkManager profile (nmcli con add/mod), a netplan YAML, or a systemd-networkd .network file. Use ip addr add for testing, the config tool for the real fix.

Q: Name the layer for each symptom: (a) name won’t resolve, (b) can’t reach another subnet, © link light off, (d) connection refused on port 5432. A: (a) Application / DNS (L7), (b) Network / routing (L3), © Physical/Data-link (L1/L2), (d) Transport (L4) — the port is reachable but no process is bound, or a reject rule fired.

---

Key takeaways

• Use the modern tools: ip (addresses, links, routes, neighbours) and ss (sockets) have replaced ifconfig/route/netstat/arp. Learn ip -br -c addr, ip route, ip route get, and ss -tulpn cold.
• CIDR is the language of subnets: /24 = 254 usable hosts; usable = 2^(32−prefix) − 2. 169.254.x.x means DHCP failed; 0.0.0.0/0 is the default route; 127.0.0.53 is systemd-resolved, not your router.
• ip is runtime, nmcli is persistent. Diagnose with ip addr add/ip route add; make it survive reboot with nmcli (or netplan on Ubuntu, systemd-networkd on minimal images). Never fix a network permanently with a raw ip command.
• DNS is its own layer with its own order: /etc/nsswitch.conf sets the source order, /etc/hosts beats DNS, /etc/resolv.conf names the servers (often the 127.0.0.53 stub). dig @1.1.1.1 name tests DNS independently of your config.
• ss -tulpn tells you exposure: 0.0.0.0 = reachable from anywhere, 127.0.0.1 = localhost only. CLOSE-WAIT pileups are an app bug, not a network fault.
• Walk the ladder, never guess: ip a → ip r → ping gateway → ping 8.8.8.8 → resolve a name → curl the port → traceroute. The first rung that fails is the diagnosis; ping 8.8.8.8 OK but names failing = DNS.
• Refused ≠ timed out: RST/refused means the host is reachable but nothing is listening (service problem); silence/timeout means a firewall dropped the packet. That fork tells you which machine and subsystem to debug — and firewalls are covered in their own lesson.