Modern Linux Networking: Bonding, VLANs, and Firewalls with nftables and firewalld

A Linux server’s network is not one thing you configure — it is a stack of layers that each fail in their own way, and the reason “networking works in the lab and breaks in production” almost always comes down to two layers disagreeing. At the bottom the kernel does the real work: it aggregates NICs into a bond, tags frames with an 802.1Q VLAN id, forwards frames across a bridge, and runs every packet through the netfilter hooks where your firewall lives. Above that sits a configuration manager — NetworkManager, systemd-networkd, or Netplan — whose only job is to render kernel state and make it survive a reboot. And underneath whatever firewall front end you run — firewalld, or raw nft rules — is a single kernel subsystem, nftables, that on every current distro has replaced the old iptables machinery. Understanding where each layer starts and stops is the difference between a host you can debug in ninety seconds and one you rebuild in a maintenance window.

This article builds a production host from the wire up and then defends it. Two NICs bond for redundancy, a trunk port carries tagged VLANs, a bridge hangs off one VLAN for virtual machines, source-based policy routing sends storage replies out their own gateway, jumbo frames get sized correctly end to end, and the firewall gets built twice — once with firewalld (zones, services, rich rules, runtime-versus-permanent discipline) for the everyday server, and once in raw nftables (tables, chains, hooks, priorities, sets, maps, NAT) for the edge box where you need full control. Every layer comes with the exact nmcli, ip, nft, and firewall-cmd to build it, the way to prove it works with ip/ss/tcpdump, and — because you will return here mid-incident — the settings, modes, and failure modes laid out as scannable tables.

By the end you will stop guessing which layer is lying. When a bond flaps, a VLAN carries no traffic, an NFS mount hangs for thirty seconds after a reboot, or a “correct” firewall rule silently drops nothing, you will know whether the fault is in the renderer, the kernel bond state, the VLAN tag, the routing table, the conntrack state, or the base-chain priority — and you will confirm which with one command instead of ten. That localisation, layer by layer, bottom to top, is the whole method.

What problem this solves

Server networking on Linux is deceptively easy to get mostly working and genuinely hard to get correct. A single interface with DHCP “just works,” so teams reach production before they discover that the abstractions leak in exactly the places that matter for uptime: redundancy, segmentation, multi-homing, and a firewall that actually filters what you think it filters.

What breaks without this knowledge is depressingly predictable. Two configuration managers fight over the same NIC and your carefully built config vanishes on reboot. A bond comes up in 802.3ad against a switch nobody put in a port-channel, so it half-negotiates into a flapping link that passes ping and drops throughput. A VLAN sub-interface exists but the switch port was never trunked, so frames leave tagged and the switch drops them. A dual-homed box replies to storage traffic out its default gateway, the return path is asymmetric, and a strict reverse-path filter or a stateful firewall silently discards the reply — an outage with no error message. A NAT rule sits in a filter-hooked chain and does nothing. A jumbo-frame MTU is set on the server but not the switch, and large transfers hang while pings succeed. Each of these is invisible until you check the specific layer directly.

Who hits this: anyone running bare-metal or VM hypervisors, storage nodes, database servers, or any host that needs more than one flat NIC on one flat network. It bites hardest on virtualization hosts (bond + trunk + bridge + VM segmentation), storage backends (a second gateway and jumbo frames), edge and gateway boxes (raw nftables with NAT and policy routing), and any RHEL-to-Ubuntu mixed fleet where the config-manager syntax diverges and a runbook written for one distro quietly corrupts the other. The fix is never “restart networking and hope” — it is “find the layer that disagrees with reality and make it tell the truth.”

To frame the whole field before the deep dive, here is every layer this article covers, the kernel object it maps to, the tool that owns it, and the single command that reveals its truth:

Learning objectives

By the end of this article you can:

• Choose the correct configuration renderer for a host — NetworkManager, systemd-networkd, or Netplan — explain what each renders to, and ensure exactly one owns each interface so nothing vanishes on reboot.
• Build an interface bond in any of the seven kernel modes, pick the right mode for what the switch actually supports, and tune LACP (802.3ad) with lacp_rate and xmit_hash_policy for real flow distribution.
• Create tagged 802.1Q VLAN sub-interfaces on top of a bond, reason about tagged versus untagged (native) VLANs, and hang a Linux bridge off a VLAN for virtual-machine or container networking.
• Solve dual-gateway asymmetric routing with source-based policy routing — named routing tables, ip rule selectors, and the rp_filter reconciliation that makes it work.
• Size MTU and jumbo frames correctly end to end and prove the path with ping -M do, avoiding the black-hole failures of a partial MTU change.
• Build a correct firewall in firewalld using zones, services, rich rules, and the runtime-versus-permanent workflow that stops you locking yourself out — and know that firewalld now writes nftables underneath.
• Drop to raw nftables and reason about tables, chains, base-chain hooks and priorities, sets and maps, stateful conntrack rules, and NAT — and explain precisely how it differs from legacy iptables.
• Drive the diagnostic toolkit fluently — ip, ss, tcpdump, nft monitor, ethtool — to localise any failure to exactly one layer.

Prerequisites & where this fits

You should be comfortable on a Linux shell as root or with sudo, know the difference between a physical NIC and a logical interface, and read basic CIDR (10.10.10.0/24) and TCP/UDP port notation. Familiarity with the OSI layers helps: bonding and VLAN tagging are Layer 2, addressing and routing are Layer 3, and the firewall filters across both. You do not need prior nftables or firewalld experience — this article builds both from first principles — but you should know that a default route picks the next hop for anything not matched by a more specific route, and that a stateful firewall tracks connection state rather than filtering each packet in isolation.

This sits in the Linux server administration track, at the layer everything else assumes works. It is upstream of almost every other server topic: Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers tunes many of the same net.* sysctls this article sets; Mastering systemd: Units, Timers, Resource Control, and Service Hardening governs the systemd-networkd and nftables.service units that persist this configuration. For the firewall specifically, Designing Stateful Linux Firewalls with native nftables Rulesets and NAT is the deep companion to this article’s nftables section — this piece integrates the firewall into the whole host stack, that one goes deeper on the ruleset design itself. When packets still misbehave after everything looks right, When Logs Aren’t Enough: Packet Capture, Traffic Mirroring, and Deep Network Troubleshooting is where the tcpdump skills scale up.

Scope for the concrete examples throughout: current RHEL-family (RHEL 9/10, Rocky, Alma) using NetworkManager and firewalld, and current Ubuntu Server (22.04/24.04 LTS) using Netplan over systemd-networkd. Both kernels use the nftables backend. Where syntax diverges the article shows both. The reference host has two NICs, enp1s0 and enp2s0, bonded into bond0, carrying VLAN 10 (server network, 10.10.10.0/24) and VLAN 20 (storage network, 10.10.20.0/24), with a bridge br-vm on VLAN 20 for virtual machines.

Core concepts

Five mental models make every later decision obvious.

The renderer is not the network — it just writes it. NetworkManager, systemd-networkd, and Netplan do not carry a single packet. Their entire job is to translate a declarative intent (“bond these two NICs, put VLAN 10 on top, give it this address”) into kernel state and re-apply it at boot. The kernel — the bonding, 8021q, and bridge drivers — does the actual work. This separation is why “my config disappeared after reboot” is nearly always a renderer problem, not a kernel one: the kernel did exactly what it was told; the wrong tool told it, or two tools told it contradictory things. Pick one renderer per host and disable the others.

A bond is a policy for spreading or protecting traffic across NICs, and the switch has a vote. Bonding aggregates physical NICs into one logical interface for redundancy, throughput, or both. But the mode is not a free choice — it is a contract with the switch. active-backup needs nothing from the network and survives a link or switch failure. 802.3ad (LACP) requires the switch ports to be configured as a link-aggregation group; if they are not, the bond half-negotiates and you get a flapping, half-working link. The single most important bonding decision is matching the mode to what the switch is actually doing, not to what you wish it did.

A VLAN tag is metadata the switch trusts, and the port must be told to trust it. An 802.1Q VLAN inserts a 4-byte tag into the Ethernet frame carrying a VLAN id (1–4094). A Linux VLAN sub-interface (bond0.10) sends and receives tagged frames for that id. This only works if the switch port is a trunk that expects tags for those VLANs; on an access port the switch strips and imposes a single VLAN and drops tagged frames. The “native VLAN” (untagged traffic on a trunk) is a classic mismatch: if the server tags everything but the switch’s native VLAN differs, one VLAN silently fails. The tag is trust, and both ends must agree.

Routing is chosen by destination unless you teach it to look at the source. The kernel’s main routing table (the FIB) picks a next hop by destination prefix, longest match wins, ties broken by metric. That is fine for one uplink. With two gateways it breaks: a reply to storage traffic goes out the default route regardless of which interface the request arrived on, creating an asymmetric path that a stateful firewall or a strict reverse-path filter drops. Policy routing adds a second routing table plus ip rule selectors that pick a table by source address (or interface, fwmark, etc.), so a packet sourced from the storage IP uses the storage gateway. This is the canonical cure for asymmetric-routing drops, and it is invisible until you look.

One kernel subsystem filters everything, and where you hook in decides what you can do. Netfilter exposes a fixed set of hooks — prerouting, input, forward, output, postrouting — that every packet traverses in order. nftables is the modern framework that registers base chains at those hooks with a numeric priority that orders your chain relative to the kernel’s own (NAT, conntrack, mangle). A filter-hooked chain filters; a NAT rule only takes effect in a chain hooked at prerouting/postrouting with a NAT priority. Connection tracking (nf_conntrack) records the state of each flow so a stateful rule can accept “established, related” traffic cheaply. Getting the hook, the priority, and the conntrack ordering right is where firewall correctness actually lives — everything else is syntax.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

Picking your stack: NetworkManager vs. systemd-networkd vs. Netplan

These three are not interchangeable, and mixing them is the single most common cause of “my config disappeared after reboot.” The distinction that matters most is that Netplan is a front end, not a backend. On Ubuntu Server it renders to systemd-networkd; on Ubuntu Desktop, to NetworkManager. Those two daemons are the real backends that program the kernel. Decide which one owns your interfaces and commit.

The commands to make one renderer authoritative and stand the others down:

# RHEL family: NetworkManager owns everything
systemctl enable --now NetworkManager
systemctl disable --now systemd-networkd 2>/dev/null || true

# Minimal Ubuntu/Debian server choosing networkd directly (without Netplan)
systemctl enable --now systemd-networkd systemd-resolved

# Ubuntu with Netplan: the renderer is chosen IN the YAML, not by enabling a service
# renderer: networkd   (server default)  or  renderer: NetworkManager

A few operational truths that save reboots. Do not hand-edit rendered files: netplan apply regenerates /run/systemd/network/ and clobbers manual changes; NetworkManager keyfiles are the source of truth on RHEL and the daemon rewrites them. Netplan warns and (on newer releases) refuses to apply world-readable YAML, so chmod 600 /etc/netplan/*.yaml. And netplan try applies a config with an automatic rollback timer — the safe way to test a change over SSH without locking yourself out.

The decision, reduced to a table you can apply in ten seconds:

The failure modes when this goes wrong, and how each announces itself:

Interface bonding and choosing the right mode

Bonding aggregates NICs for redundancy or throughput. The kernel bonding driver supports seven modes; the choice is a function of what your switch supports, not preference. Here is the complete mode reference — every mode, its number, what the switch must do, and the honest trade-off:

The rule of thumb that keeps you out of trouble: if you are unsure what the switch is doing, use active-backup. It needs nothing from the network and survives a single link or switch failure. Use 802.3ad only when the switch ports are explicitly in a LACP port-channel; a mismatch there is miserable to debug because the link sort of works. For switch-agnostic throughput without touching the switch, balance-tlb/balance-alb (modes 5/6) are the pragmatic middle ground.

The bond options you will actually set, what they do, and their defaults:

xmit_hash_policy — why one flow never exceeds one link

A crucial expectation to set: LACP and static-hash bonding do not make a single TCP flow faster. The xmit_hash_policy computes a hash from packet headers and maps it to exactly one member, so all packets of one flow take one link (this is deliberate — it prevents reordering). Aggregate throughput scales across many flows, not within one. The policy just controls how flows spread:

For a hypervisor or storage node talking to many peers, layer3+4 gives the best real-world spread. For a host that mostly talks to one gateway, no hash policy helps a single flow — that is what a faster NIC is for.

NetworkManager (nmcli)

# Create an active-backup bond. miimon=100 polls link state every 100ms.
nmcli connection add type bond con-name bond0 ifname bond0 \
  bond.options "mode=active-backup,miimon=100,primary=enp1s0,primary_reselect=better"

# Enslave both NICs to the bond
nmcli connection add type ethernet con-name bond0-p1 ifname enp1s0 master bond0
nmcli connection add type ethernet con-name bond0-p2 ifname enp2s0 master bond0

nmcli connection up bond0

For LACP against a switch with a configured port-channel, swap the options string and match lacp_rate to the switch:

nmcli connection modify bond0 bond.options \
  "mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer3+4"
nmcli connection up bond0

Netplan (Ubuntu)

# /etc/netplan/01-bond.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0: {}
    enp2s0: {}
  bonds:
    bond0:
      interfaces: [enp1s0, enp2s0]
      parameters:
        mode: active-backup
        mii-monitor-interval: 100
        primary: enp1s0
        # For LACP instead:
        # mode: 802.3ad
        # lacp-rate: fast
        # transmit-hash-policy: layer3+4

Apply and confirm the bond formed before layering anything on top — /proc/net/bonding/bond0 is the authoritative view of mode, active member, and per-link status:

chmod 600 /etc/netplan/*.yaml   # Netplan warns/refuses on world-readable files
netplan apply
cat /proc/net/bonding/bond0     # mode, active slave, MII status, LACP partner (if 802.3ad)

What to look for in /proc/net/bonding/bond0, and what each field tells you:

Tagged VLAN sub-interfaces and bridges for virtualization hosts

The switch port carrying bond0 is a trunk. We split its traffic into VLAN sub-interfaces, then put a bridge on the VLAN that VMs attach to. Order matters and is a one-way street: the VLAN rides on the bond, and the bridge rides on the VLAN — never the reverse.

Tagged versus untagged is the concept people get wrong. A quick reference:

The classic silent failure is a native-VLAN mismatch: the server sends untagged frames on bond0 expecting native VLAN 1, but the switch trunk’s native VLAN is 99 — that traffic vanishes with no error. Keep the native VLAN off your data path and tag everything you care about.

NetworkManager

# VLAN 10 on top of the bond -> server network, gets an IP directly
nmcli connection add type vlan con-name vlan10 ifname bond0.10 \
  dev bond0 id 10 \
  ipv4.method manual ipv4.addresses 10.10.10.5/24 ipv4.gateway 10.10.10.1 \
  ipv4.dns "10.10.10.53"

# VLAN 20 -> a bridge for VMs (no IP on the VLAN itself)
nmcli connection add type vlan con-name vlan20 ifname bond0.20 dev bond0 id 20
nmcli connection add type bridge con-name br-vm ifname br-vm \
  bridge.stp no ipv4.method disabled ipv6.method disabled
nmcli connection modify vlan20 master br-vm
nmcli connection up vlan20 && nmcli connection up br-vm

Three decisions worth stating explicitly. The bridge gets no IP because the host does not live on the VM network — it merely switches frames for the VMs. STP is off because a single-uplink bridge has no loop to prevent; leave STP on only when multiple bridges could form a loop. And VLAN 10 gets the host’s real address and default gateway because that is the server’s own management/data network.

Netplan

# /etc/netplan/02-vlans.yaml
network:
  version: 2
  renderer: networkd
  vlans:
    bond0.10:
      id: 10
      link: bond0
      addresses: [10.10.10.5/24]
      nameservers:
        addresses: [10.10.10.53]
      routes:
        - to: default
          via: 10.10.10.1
    bond0.20:
      id: 20
      link: bond0
  bridges:
    br-vm:
      interfaces: [bond0.20]
      parameters:
        stp: false

The bridge-netfilter gotcha

Bridging only forwards frames between bridge ports if the kernel is set up to do so, and there is a subtle firewall interaction. The br_netfilter module, when loaded, routes bridged frames through the host’s IP firewall hooks via the bridge-nf-call-iptables/bridge-nf-call-ip6tables sysctls. On a hypervisor bridge that should not be filtered by the host firewall, that silently drops VM traffic against host forward rules the VM operator never sees.

# For a VM bridge that host rules should NOT touch:
cat >/etc/sysctl.d/98-bridge.conf <<'EOF'
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
EOF
# The sysctls only exist once br_netfilter is loaded:
modprobe br_netfilter
sysctl --system

Libvirt/KVM generally manage this for you; the trap is a hand-built bridge for containers where you did not, and VM traffic disappears into host forward rules. The bridge parameters worth knowing:

Static addressing, multiple gateways, and source-based policy routing

The hard part of multi-homing is two gateways. The main routing table picks a route by destination only, so a host with two uplinks replies out the wrong interface, and stateful firewalls (or RPF checks) drop the resulting asymmetric flow. The fix is policy routing: a second routing table plus a rule that selects it by source address.

Concretely: VLAN 10 (10.10.10.5, gw 10.10.10.1) is the primary network, and VLAN 20 (10.10.20.5, gw 10.10.20.1) is a storage path whose replies must go back out its own gateway. Without policy routing, a storage client on a different subnet sends a request that arrives on VLAN 20, the host replies, and the reply follows the default route out VLAN 10 — asymmetric, and dropped.

First understand the routing-table machinery. The kernel has multiple numbered routing tables and a rule database (RPDB) that decides, per packet, which table to consult:

The ip rule list (RPDB) is walked top-down by priority; the first matching rule’s table is consulted:

Name a routing table, then populate it and add the selecting rule. With NetworkManager you persist all of this on the connection itself:

# 1. Name table 100 "storage" (one-time, on the file)
echo "100 storage" >> /etc/iproute2/rt_tables

# 2. Default route for the storage table goes out VLAN 20's gateway
nmcli connection modify vlan20 \
  +ipv4.routes "0.0.0.0/0 10.10.20.1 table=100"

# 3. Rule: traffic SOURCED from 10.10.20.5 uses table 100
nmcli connection modify vlan20 \
  +ipv4.routing-rules "priority 100 from 10.10.20.5 table 100"

nmcli connection up vlan20

The equivalent raw ip commands make the mechanism explicit and are the right way to test before persisting (they vanish on reboot, which is a feature while experimenting):

ip route add default via 10.10.20.1 dev bond0.20 table storage
ip rule add from 10.10.20.5 lookup storage priority 100
ip rule show                        # confirm the rule sits above the main-table lookup
ip route show table storage         # confirm the storage default route is present
ip route get 8.8.8.8 from 10.10.20.5   # THE test: which gateway/iface for a storage-sourced packet?

In Netplan the same intent is routing-policy:

    bond0.20:
      id: 20
      link: bond0
      addresses: [10.10.20.5/24]
      routes:
        - to: default
          via: 10.10.20.1
          table: 100
      routing-policy:
        - from: 10.10.20.5
          table: 100
          priority: 100

rp_filter — the setting that fights policy routing

Reverse-path filtering validates that a packet’s source address is reachable via the interface it arrived on. In strict mode it will drop exactly the asymmetric replies that policy routing exists to enable, so the two must be reconciled:

# Check current mode (all + per-interface; the effective value is the max of the two)
sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.bond0/20.rp_filter

# Multi-homed pragmatic default: loose mode
cat >/etc/sysctl.d/99-rpfilter.conf <<'EOF'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
EOF
sysctl --system

The effective rp_filter for an interface is the maximum of conf.all and conf.<iface> — a subtlety that traps people who set the per-interface value but leave all at strict. If you have proper policy routing everywhere, mode 2 (loose) is the right global default for a multi-homed host; if you cannot guarantee symmetric routing, loose mode is the safe compromise that still catches obviously spoofed traffic.

ip route get <dst> from <src> is the single fastest sanity check for the whole subsystem — it tells you exactly which table, gateway, and interface the kernel will use for a given source. If it picks the wrong gateway, your ip rule priority or table is wrong, full stop.

MTU and jumbo frames end to end

The default Ethernet MTU is 1500 bytes of L3 payload. Jumbo frames raise it (commonly to 9000) to cut per-packet overhead on storage and backup traffic — fewer, larger frames mean less CPU and higher throughput. The catch: MTU must match at every hop, or you get a black hole. A frame larger than a link’s MTU is either fragmented (a performance tax) or, if the “don’t fragment” bit is set (as it is for most modern TCP with Path MTU Discovery), silently dropped — and the failure looks bizarre: small pings work, SSH connects, but large transfers hang.

The stack of MTU values you must keep consistent:

Set it in the renderer so it persists, not just with ip link (which is lost on reboot):

# Netplan — jumbo on the storage path
    bond0.20:
      id: 20
      link: bond0
      mtu: 9000
      addresses: [10.10.20.5/24]

# NetworkManager — set MTU on the VLAN connection
nmcli connection modify vlan20 ethernet.mtu 9000
nmcli connection up vlan20

Prove the path actually carries jumbo frames without fragmentation — the -M do sets “don’t fragment,” so a failure means a hop in the middle has a smaller MTU:

# Payload 8972 + 8 (ICMP) + 20 (IP) = 9000. If this succeeds unfragmented, the path is clean.
ping -M do -s 8972 -c 3 10.10.20.6
# "Frag needed and DF set (mtu = 1500)" => a hop is stuck at 1500 — usually the switch VLAN

The MTU failure table, because these are genuinely confusing in the field:

firewalld: zones, services, and rich rules for everyday hosts

For a normal server, drop to raw nftables only when you have a reason. firewalld gives you zones, named services, runtime/permanent separation, and on modern distros it writes nftables underneath anyway (the nftables backend replaced the iptables backend by default in RHEL 8+). You get the ergonomics of a zone model with the modern kernel backend.

The model: every interface or source belongs to a zone; each zone has a default target (typically drop or reject unsolicited input) and an allow-list of services and ports. firewalld ships predefined zones with increasing trust:

Assign interfaces deliberately, then open only what the role needs:

# Put the server VLAN in the internal zone, storage in the work zone
nmcli connection modify vlan10 connection.zone internal
nmcli connection modify vlan20 connection.zone work
nmcli connection up vlan10 && nmcli connection up vlan20

# Open services on the internal zone (permanent), then reload
firewall-cmd --permanent --zone=internal --add-service=ssh
firewall-cmd --permanent --zone=internal --add-service=https
firewall-cmd --permanent --zone=internal --add-port=9100/tcp   # node_exporter
firewall-cmd --reload

firewall-cmd --zone=internal --list-all   # the effective, human-readable config

Runtime versus permanent — the discipline that prevents lockout

This is the firewalld habit that separates operators from victims. Every command is either runtime (live now, gone on reload/reboot) or permanent (--permanent, written to /etc/firewalld/, applied on the next reload):

The lockout-proof workflow for any risky change (e.g. tightening SSH): apply it runtime only, confirm from a second SSH session that you are still in, then promote it with --runtime-to-permanent. If you locked yourself out, a reboot (or the panic timer) reverts the un-promoted runtime change:

# 1. Apply runtime only (NO --permanent)
firewall-cmd --zone=internal --add-source=10.0.0.0/8
# 2. Verify from a SECOND session that you still have access
# 3. Only then make it stick:
firewall-cmd --runtime-to-permanent

Rich rules — when a service is not enough

Rich rules express what plain services cannot: source-restricted access, logging, rate limits, and per-rule accept/reject/drop. The anatomy is rule family → source/destination → service/port → log → audit → action:

# Restrict node_exporter to the server subnet only, and log the acceptance
firewall-cmd --permanent --zone=internal --add-rich-rule='
  rule family="ipv4" source address="10.10.10.0/24"
  port port="9100" protocol="tcp" accept'

# Rate-limited, logged SSH (5 new connections/minute)
firewall-cmd --permanent --zone=internal --add-rich-rule='
  rule service name="ssh" log prefix="SSH " level="info" limit value="5/m" accept'

# Explicitly drop a known-bad source with no reply
firewall-cmd --permanent --zone=drop --add-source=203.0.113.66

firewall-cmd --reload

The firewall-cmd verbs you will reach for, mapped to what they do:

Dropping to raw nftables: tables, chains, hooks, sets, and NAT

When you need full control — an edge box, complex NAT, rules firewalld cannot express cleanly, or a ruleset you version in Git — write nftables directly. Disable firewalld first so the two do not fight over the ruleset (both target the same kernel subsystem, and two managers is the same class of bug as two renderers):

systemctl disable --now firewalld
systemctl enable --now nftables

nftables’ advantages over legacy iptables are structural, not cosmetic. One unified syntax handles IPv4 and IPv6 via the inet family; rules load atomically (all-or-nothing, no half-applied ruleset under load); and it has first-class sets and maps so you stop generating thousands of near-identical rules. The conceptual map:

Hooks and priorities — the part people skip and regret

A base chain is bound to a netfilter hook and given a numeric priority. The hook decides where in the packet’s journey the chain runs; the priority orders your chain relative to the kernel’s own operations (conntrack, NAT, mangle). Get these wrong and a rule that looks perfect does nothing. Every packet traverses hooks in this order, and each chain type has a canonical priority:

The two rules that flow from this: filtering happens in input/forward/output at priority filter (0), and NAT only works in a chain hooked at prerouting (DNAT) or postrouting (SNAT/masquerade) at the NAT priority. A masquerade rule in a filter-hooked chain is a silent no-op.

A complete stateful firewall with NAT

A production /etc/nftables.conf for a host that both protects itself and NATs VM traffic out its uplink:

#!/usr/sbin/nft -f
flush ruleset

table inet filter {
    # A named set of admin networks — the fleet-templating unit
    set admin_nets {
        type ipv4_addr
        flags interval
        elements = { 10.10.10.0/24, 192.0.2.10 }
    }

    chain input {
        type filter hook input priority filter; policy drop;

        ct state established,related accept        # cheap fast-path for known flows
        ct state invalid drop                       # out-of-window / malformed
        iif "lo" accept                             # loopback is always fine
        ip protocol icmp accept                     # allow IPv4 ping/PMTUD
        ip6 nexthdr ipv6-icmp accept                # ICMPv6 is mandatory for IPv6 to work

        # SSH only from admin networks, rate-limited to blunt brute force
        tcp dport 22 ip saddr @admin_nets ct state new \
            limit rate 10/minute accept
        tcp dport { 80, 443 } accept                # public web
        udp dport 9100 drop                         # example explicit drop
    }

    chain forward {
        type filter hook forward priority filter; policy drop;
        ct state established,related accept
        iifname "br-vm" accept                      # let VMs reach out
    }

    chain output {
        type filter hook output priority filter; policy accept;
    }
}

# Masquerade VM traffic leaving the uplink (SNAT to the uplink IP)
table inet nat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        oifname "bond0.10" ip saddr 10.10.20.0/24 masquerade
    }
}

Load and verify atomically — -c checks syntax and changes nothing, then -f applies all-or-nothing:

nft -c -f /etc/nftables.conf   # dry-run: parse and validate, apply nothing
nft -f /etc/nftables.conf      # atomic apply
nft list ruleset               # the live, effective ruleset

NAT and forwarding also require IP forwarding turned on — a sysctl entirely separate from any firewall rule; miss it and packets are filtered fine but never routed:

cat >/etc/sysctl.d/99-forward.conf <<'EOF'
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
sysctl --system

Sets and maps — why the ruleset stays small

Sets are named groups (addresses, ports, interfaces) you match against with @name; maps are keyed lookups that return a value (a verdict, an address, an nftables object). They replace thousands of generated rules with one lookup. The types you will use:

A verdict map replaces a stack of if rules with one lookup:

# Route a verdict per source address in a single rule
table inet filter {
    map src_policy {
        type ipv4_addr : verdict
        elements = { 10.10.10.0/24 : accept, 203.0.113.66 : drop }
    }
    chain input {
        type filter hook input priority filter; policy drop;
        ct state established,related accept
        ip saddr vmap @src_policy
    }
}

Persistence and fleet templating

Persistence differs by front end and is a frequent cause of “it worked until the reboot.” With firewalld, --permanent writes XML to /etc/firewalld/. With raw nftables, nftables.service loads /etc/nftables.conf at boot — so your file is the persistence mechanism, and if the unit is not enabled your carefully built ruleset evaporates. Confirm it:

systemctl is-enabled nftables    # must be "enabled"

For fleets, split the ruleset so the chain logic is identical everywhere and only the data varies. Include a directory and render the per-host sets from Ansible/Jinja:

# /etc/nftables.conf
include "/etc/nftables.d/*.nft"

This gives you the same separation firewalld achieves with zones, but versioned in Git: one reviewed chain definition, per-environment admin_nets.nft sets, and no snowflake hosts.

Connection tracking: where correctness (and the subtle bugs) live

Connection tracking is what makes a firewall stateful, and it is where the non-obvious failures hide. Four rules:

Order the conntrack accept first. ct state established,related accept must be the first stateful rule in every base chain. Put it last and every packet of an established flow walks the entire chain (wasteful); worse, an early drop can break return traffic for connections the host itself initiated.

ct state invalid drop early — but understand it. It catches out-of-window and malformed packets. On asymmetric paths (see policy routing above) legitimate packets can be marked invalid because the firewall never saw the SYN — which is another reason to fix routing before blaming the firewall.

conntrack has a finite table. A busy box, or a synchronized reconnect storm, can exhaust nf_conntrack_max; the kernel logs nf_conntrack: table full, dropping packet and new connections fail. Size it for the workload’s peak, not idle:

# Current usage vs ceiling — watch this during a controlled load test, not at idle
conntrack -C                                        # count of tracked connections now
sysctl net.netfilter.nf_conntrack_max               # the ceiling
cat /proc/sys/net/netfilter/nf_conntrack_buckets    # hash buckets (sizing affects lookup cost)

# Raise for a busy box and persist
cat >/etc/sysctl.d/99-conntrack.conf <<'EOF'
net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_buckets = 131072
EOF
sysctl --system

Bridge filtering surprises. As with the bridge section, br_netfilter can route bridged frames through conntrack and the host firewall — either account for it in forward rules or disable bridge-nf-call-*.

The conntrack sysctls worth knowing:

Architecture at a glance

Picture the host as a stack you read from the bottom up, because that is the order in which each layer must be correct before the next can work. At the very bottom sit two physical NICs, enp1s0 and enp2s0, each a cable into (ideally) a different switch in an MLAG pair. The bonding driver aggregates them into bond0 — in active-backup if the switch is plain, or 802.3ad with layer3+4 hashing if the ports form an LACP port-channel. bond0 presents one logical interface upward and hides the failover entirely; /proc/net/bonding/bond0 is where you confirm which member is live and whether LACP actually negotiated.

Riding on top of bond0 are two 802.1Q VLAN sub-interfaces. bond0.10 carries VLAN 10, the server network, and takes the host’s real address 10.10.10.5/24 with the default gateway 10.10.10.1. bond0.20 carries VLAN 20, the storage network; it has no IP of its own because a Linux bridge, br-vm, sits on top of it and hands frames to the virtual machines, which live on 10.10.20.0/24. The switch port under bond0 is a trunk that must allow both VLAN ids, or the tagged frames are dropped with no error.

At Layer 3 the interesting flow is the storage path. A reply sourced from 10.10.20.5 must not follow the main table’s default route (which points out VLAN 10) — that would be asymmetric and get dropped. A policy-routing rule, from 10.10.20.5 lookup storage, diverts those packets to a second routing table whose default route exits 10.10.20.1, keeping the path symmetric so conntrack and rp_filter (set to loose, mode 2) are satisfied. ip route get 8.8.8.8 from 10.10.20.5 is the one command that proves this hop is correct.

Wrapping everything is the firewall, running in the netfilter hooks the packets already traverse. On an ordinary server, firewalld owns it: bond0.10 is bound to the internal zone and bond0.20 to the work zone, each with a small allow-list and a default target that drops unsolicited input. On an edge or gateway box, raw nftables takes over: an inet filter table with input/forward/output base chains at priority filter, an inet nat table masquerading VM traffic out bond0.10 at postrouting/srcnat, and a ct state established,related accept as the first rule in every chain. Whichever front end you run, one subsystem — nftables — is doing the filtering, and nft list ruleset shows the truth. The whole design is a bottom-up chain of dependencies: a fault at the bond invalidates the VLAN, a fault at the VLAN invalidates the address, and so on — which is exactly why the Verify and troubleshooting sections move bottom to top.

Real-world scenario

Meridian Payments ran a fleet of twelve KVM hypervisors hosting a payment-processing platform. Each host had two 25 GbE NICs bonded in 802.3ad (LACP) against an MLAG switch pair, VLAN 10 for the hosts and VLAN 20 for an NFS storage backend that lived behind its own gateway (10.10.20.1). Jumbo frames (MTU 9000) were enabled on the storage VLAN for throughput. The platform team was five engineers; the hosts ran RHEL 9 with NetworkManager and firewalld. Everything ran clean for months.

The incident began after a routine hypervisor reboot during a patch window. NFS mounts on the rebooted host hung for about thirty seconds before recovering, and dmesg filled with nf_conntrack: table full, dropping packet. On-call’s first instinct was to blame NFS or the storage array — both were fine and serving every other hypervisor. The second instinct, scaling up, made no sense for a bare-metal host. Forty minutes in, storage latency alerts were firing and the on-call bridge was full.

The breakthrough came from checking the layers bottom-up instead of guessing. cat /proc/net/bonding/bond0 was clean — LACP negotiated, both members up. ip -d link show bond0.20 showed the VLAN correct. But ip route get 10.10.20.30 from 10.10.20.5 revealed the problem: replies sourced from the storage IP were routing out VLAN 10’s gateway, not VLAN 20’s. The host had a single default route in the main table; nobody had configured policy routing. Under normal steady state this was masked because most storage flows were long-lived and already in conntrack. But on reboot, every VM re-established its NFS connection at once, and each new storage reply took the asymmetric path out VLAN 10. Two things then compounded: the strict rp_filter (mode 1, the RHEL default) plus the stateful firewall marked those asymmetric replies invalid and dropped them, forcing retries; and the synchronized reconnect storm blew past the default nf_conntrack_max of 65536, so new connections failed outright. The thirty-second hang was NFS retrying through dropped packets until conntrack drained enough to let connections through.

The fix was three-part and precise. First, pin storage replies to their own gateway with policy routing so the path was always symmetric and conntrack always saw a clean flow:

echo "100 storage" >> /etc/iproute2/rt_tables
nmcli connection modify vlan20 \
  +ipv4.routes "0.0.0.0/0 10.10.20.1 table=100" \
  +ipv4.routing-rules "priority 100 from 10.10.20.5 table 100"
nmcli connection up vlan20

Second, reconcile rp_filter to loose mode so it stopped fighting the (now-symmetric anyway) storage path:

echo "net.ipv4.conf.all.rp_filter = 2" > /etc/sysctl.d/99-rpfilter.conf
sysctl --system

Third, size conntrack for the real boot-storm peak, not idle:

cat >/etc/sysctl.d/99-conntrack.conf <<'EOF'
net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_buckets = 131072
EOF
sysctl --system

The subtle part, and the lesson that went on the runbook: the conntrack table was never actually “too small” for steady state — it overflowed only during the synchronized reconnect, and the asymmetric routing was invisible until a controlled drain-and-reboot exposed it. Watching conntrack -C against nf_conntrack_max during a deliberate host reboot, not at idle, was what made the headroom obvious. All three changes went into Ansible so every hypervisor got identical policy-routing rules, rp_filter mode, and conntrack sizing. The next patch window rebooted all twelve hosts with zero NFS hangs and an empty dmesg.

The incident as a timeline, because the order of diagnosis is the lesson:

Advantages and disadvantages

The Linux “renderer over kernel drivers, one nftables backend under any front end” model is enormously capable and equally easy to misconfigure. Weigh it honestly:

The model is right for any host that needs redundancy, segmentation, multi-homing, or precise egress control — which is nearly every production server beyond a single flat NIC. It bites hardest where defaults are trusted blindly (strict rp_filter on a dual-homed box), where two tools own one resource, and where a change is made on the wire (ip, nft runtime) but never persisted in the renderer or nftables.conf. Every disadvantage is manageable — but only if you know it exists, verify the layer directly, and put the config in version control.

Hands-on lab

Build the whole stack on a single test host or VM with two NICs — bond, VLANs, bridge, policy routing, then both firewalls — and prove each layer before moving up. Uses only the base OS tooling; no cost. Run as root. Adjust interface names to your host (ip link to list them). If you only have one NIC, you can still do the VLAN/bridge/firewall steps by skipping the bond and putting VLANs on the single NIC.

Step 1 — Confirm your renderer and NICs.

# RHEL: expect NetworkManager active, systemd-networkd inactive
systemctl is-active NetworkManager
ip -br link show                     # note your two NIC names (e.g. enp1s0, enp2s0)

Expected: active for NetworkManager, and two NICs in state UP or DOWN (they need not be cabled for the lab).

Step 2 — Build the bond (active-backup, the safe mode).

nmcli connection add type bond con-name bond0 ifname bond0 \
  bond.options "mode=active-backup,miimon=100,primary=enp1s0"
nmcli connection add type ethernet con-name bond0-p1 ifname enp1s0 master bond0
nmcli connection add type ethernet con-name bond0-p2 ifname enp2s0 master bond0
nmcli connection up bond0
cat /proc/net/bonding/bond0

Expected: Bonding Mode: fault-tolerance (active-backup), MII Status: up, and both slaves listed. This file is your L1 proof.

Step 3 — Add two tagged VLANs and a bridge.

nmcli connection add type vlan con-name vlan10 ifname bond0.10 dev bond0 id 10 \
  ipv4.method manual ipv4.addresses 10.10.10.5/24
nmcli connection add type vlan con-name vlan20 ifname bond0.20 dev bond0 id 20
nmcli connection add type bridge con-name br-vm ifname br-vm \
  bridge.stp no ipv4.method disabled ipv6.method disabled
nmcli connection modify vlan20 master br-vm
nmcli connection up vlan10 && nmcli connection up vlan20 && nmcli connection up br-vm

ip -d link show bond0.10             # -d shows "vlan protocol 802.1Q id 10"
ip link show master br-vm            # bond0.20 should be enslaved to br-vm

Expected: bond0.10@bond0 with vlan ... id 10, and bond0.20 shown as a port of br-vm. That is your L2 proof.

Step 4 — Add source-based policy routing for a second gateway.

nmcli connection modify vlan20 ipv4.method manual ipv4.addresses 10.10.20.5/24
echo "100 storage" >> /etc/iproute2/rt_tables
nmcli connection modify vlan20 \
  +ipv4.routes "0.0.0.0/0 10.10.20.1 table=100" \
  +ipv4.routing-rules "priority 100 from 10.10.20.5 table 100"
nmcli connection up vlan20

ip rule show                                  # your "from 10.10.20.5 lookup storage" rule
ip route show table storage                   # the storage default route
ip route get 1.1.1.1 from 10.10.20.5          # should resolve via 10.10.20.1 / bond0.20

Expected: ip route get reports the storage gateway/interface for the storage-sourced packet, and the main-table route for anything else. That is your L3 proof.

Step 5 — Firewall it with firewalld (zones + a rich rule), safely.

nmcli connection modify vlan10 connection.zone internal
nmcli connection up vlan10
# Apply SSH at RUNTIME first, verify from a second session, then persist:
firewall-cmd --zone=internal --add-service=ssh
firewall-cmd --zone=internal --add-rich-rule='
  rule family="ipv4" source address="10.10.10.0/24" port port="9100" protocol="tcp" accept'
firewall-cmd --zone=internal --list-all
# Once you have confirmed access from another session:
firewall-cmd --runtime-to-permanent

Expected: --list-all shows the internal zone with ssh and your rich rule. Because these are firewalld commands, they are being rendered to nftables — confirm with nft list ruleset | grep -A3 'chain filter_IN_internal' (the chain names are firewalld-generated).

Step 6 — Swap to raw nftables (on a throwaway host only).

Do this only where losing SSH is acceptable, because you are replacing the firewall wholesale.

# Stand down firewalld, write a minimal correct ruleset
systemctl disable --now firewalld
cat >/etc/nftables.conf <<'EOF'
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
    chain input {
        type filter hook input priority filter; policy drop;
        ct state established,related accept
        iif "lo" accept
        ip protocol icmp accept
        tcp dport 22 accept
    }
    chain forward { type filter hook forward priority filter; policy drop; }
    chain output { type filter hook output priority filter; policy accept; }
}
EOF
nft -c -f /etc/nftables.conf         # validate first — changes nothing
nft -f /etc/nftables.conf            # atomic apply
systemctl enable --now nftables      # so it survives reboot — the persistence gate
nft list ruleset

Expected: nft list ruleset shows your table with the three chains and the conntrack accept first. systemctl is-enabled nftables returns enabled.

Step 7 — Validate the whole stack bottom-up.

cat /proc/net/bonding/bond0          # L1: bond mode + members
ip -d link show bond0.10             # L2: VLAN tag present
ip addr show; ip rule show           # L3: address + policy rule
ss -tlnp                             # listening sockets + owning process
nft list ruleset                     # firewall: effective ruleset with counters

Step 8 — Teardown. Remove everything cleanly so the host returns to a single-NIC baseline:

# nftables
systemctl disable --now nftables; nft flush ruleset
# NetworkManager objects (order: bridge/vlan first, then bond)
nmcli connection delete br-vm vlan20 vlan10 bond0-p1 bond0-p2 bond0
# routing table name + sysctls
sed -i '/100 storage/d' /etc/iproute2/rt_tables
rm -f /etc/sysctl.d/99-conntrack.conf /etc/sysctl.d/99-rpfilter.conf
# re-enable firewalld if this is a normal host
systemctl enable --now firewalld

Common mistakes & troubleshooting

The recurring failure modes are predictable, and each is invisible until you check the specific layer. This is the playbook — symptom, the real root cause, the exact command that confirms it, and the fix:

Two workflow habits prevent most of the above. When you change routing or firewall rules over SSH, apply them runtime-only first (firewalld: no --permanent; nftables: nft add interactively), confirm from a second session, and only then persist — and use netplan try on Ubuntu, which auto-rolls-back. And when anything is wrong, verify bottom-up: bond (/proc/net/bonding), then VLAN (ip -d link), then address/route (ip route get), then firewall (nft list ruleset). A fault at layer N makes every layer above it look broken.

Best practices

• One renderer per host, others disabled. NetworkManager or networkd/Netplan owns every interface. Confirm with systemctl is-enabled and treat “config vanished on reboot” as a two-renderers bug until proven otherwise.
• active-backup unless the switch is explicitly LACP. It needs nothing from the network and survives a switch failure. Only use 802.3ad when the switch ports are in a configured port-channel, and verify LACP negotiated (non-zero partner MAC).
• Always set miimon=100. A bond with no link monitoring cannot fail over. This is the single most-forgotten bond option.
• Build the stack in dependency order and never invert it. VLAN on bond, bridge on VLAN. A bridge under a VLAN, or a VLAN under a bridge, is a mistake you will pay for.
• Give VM bridges no host IP and set STP deliberately. The host does not live on the VM network. STP on only when multiple bridges could loop.
• Reconcile rp_filter with your topology. Loose (2) for any multi-homed host; strict (1) only for genuinely single-homed hosts. Remember the effective value is the max of all and per-interface.
• Add policy routing for every second gateway. Any host with two uplinks needs a source-based ip rule + table, or asymmetric routing will bite. ip route get <dst> from <src> is the proof.
• Match MTU end to end or leave it at 1500. Jumbo frames must match on host, bond, VLAN, bridge, and switch. A partial change is a black hole; prove it with ping -M do.
• firewalld for servers, raw nftables for edge/gateway — never both. Two firewall managers is chaos. On modern distros firewalld writes nftables anyway, so you rarely lose capability by staying with it.
• Change firewall rules runtime-first, then promote. Apply, verify from a second session, then --runtime-to-permanent (firewalld) or write nftables.conf (raw). This is how you avoid locking yourself out.
• ct state established,related accept is the first rule in every chain. Then ct state invalid drop. Ordering here is correctness, not style.
• Enable the persistence service and template for the fleet. nftables.service enabled so /etc/nftables.conf survives reboot; render per-host sets from Ansible; version the whole thing in Git.

Security notes

Networking is the layer attackers cross, so treat it as a security control, not just plumbing. Default-deny at the firewall: every base chain in nftables should have policy drop (or a firewalld zone target that drops/rejects unsolicited input), and you allow-list only what a role needs. An accidental policy accept on the input chain is a wide-open host that still “passes” every functional test.

Segment with VLANs and enforce with the firewall, not just the switch. VLAN separation is a convenience the switch enforces at Layer 2, but a host with interfaces in multiple VLANs can bridge or route between them — intentionally or through a misconfiguration. Keep the storage/management VLANs off any path that reaches the internet, and use firewalld zones or nftables forward rules to control what crosses between them. A management interface (internal/home zone or an admin-only nftables set) should never be reachable from a tenant or DMZ network.

Rate-limit and restrict management access. SSH should be allowed only from an admin set (ip saddr @admin_nets) and rate-limited (limit rate 10/minute) to blunt brute-force; the same applies to any exposed management port. Log accepted admin connections (a firewalld rich-rule log or an nftables log prefix) so you have an audit trail — and feed those logs somewhere durable, as covered in Building a Linux Audit Trail with auditd and eBPF Runtime Visibility.

NAT and forwarding are privilege grants — scope them. Turning on ip_forward makes the host a router; a masquerade rule lets an entire subnet egress under the host’s IP. Scope the SNAT rule to exactly the source subnet (ip saddr 10.10.20.0/24) and the correct out-interface (oifname "bond0.10"), never a blanket masquerade, or you have built an open relay for any traffic that reaches the box. Pair the network firewall with host-level confinement of the services behind it — Running SELinux in Enforcing Mode: Troubleshooting and Writing Custom Policy and Authoring AppArmor Profiles: Confining Services on Ubuntu and Debian are the mandatory-access-control layer beneath the packet filter.

Anti-spoofing and invalid-state hygiene. rp_filter (loose at minimum) and ct state invalid drop are cheap protections against spoofed and malformed packets — keep them on, understanding that on asymmetric paths you fix the routing rather than disabling the protection. Drop ct state invalid before it can confuse stateful logic, and consider dropping traffic to/from bogon and unused ranges at the edge with an nftables set.

Cost & sizing

Linux networking itself is free — no licenses, no per-feature SKUs — so the cost conversation is about hardware, CPU, and the operational cost of getting it wrong. What actually drives resource use and spend:

The dominant cost in this entire domain is not hardware or software — it is the engineer-hours and downtime from misconfiguration, which is precisely why config-as-code and layer-by-layer verification pay for themselves the first time a fleet-wide change lands cleanly. Two NICs bonded is the non-negotiable redundancy floor; size nf_conntrack_max from the peak connection count (measure under a reconnect storm, not at idle) since the RAM is trivial next to table-full drops; and keep firewall rulesets flat and set-driven, because rule evaluation is per-packet CPU at line rate.

Interview & exam questions

Q1. What is the difference between NetworkManager, systemd-networkd, and Netplan, and why can’t you mix them? NetworkManager and systemd-networkd are backends that program the kernel; Netplan is a front end that generates config for one of them (networkd by default on Ubuntu Server). You can’t run two backends on the same interface because they overwrite each other’s state, which is the classic “config disappears on reboot.” Maps to LFCS/RHCSA networking objectives.

Q2. When would you choose active-backup over 802.3ad for a bond? active-backup (mode 1) needs nothing from the switch and survives a link or switch failure — use it whenever the switch isn’t explicitly configured for LACP, or when the two NICs go to different switches. 802.3ad (mode 4) needs a matching LACP port-channel; a mismatch produces a flapping, half-working link. When unsure, active-backup.

Q3. Does an LACP bond make a single TCP connection faster? No. xmit_hash_policy maps each flow to exactly one member to avoid reordering, so a single flow is capped at one link’s speed. Aggregation increases throughput across many flows. layer3+4 hashing spreads flows better than the default layer2 when talking to many hosts, but it never speeds one flow.

Q4. Explain source-based policy routing and when you need it. The main routing table selects by destination only, so a dual-homed host replies out its default gateway regardless of which interface a request arrived on — an asymmetric path that stateful firewalls and strict rp_filter drop. Policy routing adds a second table plus an ip rule that selects it by source address, keeping the path symmetric. It’s the canonical fix for multi-gateway asymmetric-routing drops.

Q5. What does rp_filter do and how does it interact with policy routing? Reverse-path filtering validates that a packet’s source is reachable via the interface it arrived on. Strict mode (1) drops asymmetric traffic — exactly what policy routing enables — so on a multi-homed host you use loose mode (2). The effective value is the maximum of conf.all and the per-interface setting.

Q6. How is nftables different from iptables? nftables uses one framework with a single inet family for IPv4/IPv6, atomic ruleset loading, and native sets/maps — versus iptables’ fixed tables, separate ip/ip6 rulesets, per-rule commits, and external ipset. You declare your own tables and base chains (choosing the hook and priority) rather than using predefined ones. It’s the default backend on all current distros.

Q7. Why might a correct-looking nftables NAT rule do nothing? NAT only takes effect in a base chain hooked at prerouting (DNAT) or postrouting (SNAT/masquerade) with the NAT priority. A masquerade rule placed in a filter-hooked chain is a silent no-op. NAT also requires ip_forward=1, which is separate from any rule.

Q8. What is the runtime-versus-permanent model in firewalld and why does it matter? Runtime changes are live immediately but lost on reload/reboot; --permanent writes to /etc/firewalld/ and applies on --reload. The safe workflow is runtime-first, verify from a second session, then --runtime-to-permanent — so a bad rule that locks you out is reverted by a reboot rather than persisted.

Q9. Why must ct state established,related accept be the first rule in a chain? It’s a cheap fast-path that accepts packets belonging to known flows before they walk the whole chain. Placing it late wastes evaluation on every established packet; worse, an earlier drop can break return traffic. ct state invalid drop follows it to discard out-of-window packets.

Q10. A dual-homed host serving NFS on a second gateway hangs for 30 seconds after every reboot. What do you check? Bottom-up: /proc/net/bonding (bond healthy?), ip -d link (VLAN correct?), then ip route get <storage-dst> from <storage-src> — if replies route out the primary gateway you have asymmetric routing needing policy routing; check dmesg for nf_conntrack: table full (reconnect storm exhausting conntrack) and rp_filter mode. The fix is policy routing + loose rp_filter + a larger nf_conntrack_max.

Q11. What breaks if you enable jumbo frames on the host but not the switch? A path-MTU black hole: small packets (pings, SSH handshake) succeed, but large frames with the don’t-fragment bit set are silently dropped, so bulk transfers hang. ping -M do -s 8972 <peer> returns “Frag needed and DF set.” Fix: raise MTU on every hop including the switch VLAN, or leave everything at 1500.

Q12. How do sets and maps improve an nftables ruleset? Sets are named groups (addresses/ports) matched with @name; maps are keyed lookups returning a verdict or value. They collapse thousands of near-identical rules into one lookup, keep evaluation fast at line rate, and become the fleet-templating unit — identical chain logic everywhere, per-host data in a rendered set file.

Quick check

1. On Ubuntu Server, your nmcli changes have no effect. What is almost certainly happening, and where should you make the change instead?
2. Your bond is 802.3ad but /proc/net/bonding/bond0 shows an all-zero partner MAC. What does that mean and what are your two options?
3. A host has interfaces in VLAN 10 (default gateway) and VLAN 20 (storage). Replies to storage clients never arrive. Name the mechanism you need and the one command that proves the fix.
4. You add a masquerade rule to your nftables input chain and NAT still doesn’t happen. Why?
5. Which firewalld workflow prevents you from locking yourself out when tightening SSH, and what are its three steps?

Answers

1. Netplan renders to systemd-networkd, not NetworkManager, on Ubuntu Server. Check grep renderer /etc/netplan/*.yaml; make the change in the Netplan YAML and run netplan apply (or netplan try for an auto-rollback test). Editing NM or the generated /run/systemd/network/ files is futile.
2. The switch is not speaking LACP — the ports aren’t in a configured port-channel — so the bond only half-negotiates and the link flaps. Options: (a) configure the switch port-channel correctly, or (b) switch the bond to active-backup, which needs nothing from the switch.
3. Source-based policy routing — a second routing table with the storage gateway’s default route, plus ip rule add from 10.10.20.5 lookup storage. The proof is ip route get <storage-client> from 10.10.20.5, which must resolve via the storage gateway/interface. Also set rp_filter to loose.
4. NAT only works in a chain hooked at prerouting/postrouting with a NAT priority. An input-hooked chain is for filtering; a masquerade rule there is a silent no-op. Put it in a postrouting chain with priority srcnat, and ensure ip_forward=1.
5. Apply the change runtime-only (no --permanent), verify from a second SSH session that you still have access, then firewall-cmd --runtime-to-permanent. If you locked yourself out, a reboot reverts the un-promoted runtime change.

Glossary

• Renderer — a tool (NetworkManager, systemd-networkd, Netplan) that translates declarative network intent into kernel state and persists it across reboots.
• Bond — a logical interface (bond0) aggregating multiple physical NICs for redundancy and/or throughput, driven by the kernel bonding module.
• Bond mode — the policy governing how a bond uses its members; the seven modes range from active-backup (failover only) to 802.3ad (LACP dynamic aggregation).
• LACP (802.3ad) — Link Aggregation Control Protocol; bond mode 4, which dynamically negotiates a link-aggregation group with a switch configured for it.
• xmit_hash_policy — the hash (layer2, layer2+3, layer3+4) that maps each flow to one bond member; controls flow distribution but never speeds a single flow.
• VLAN (802.1Q) — a Layer-2 segmentation using a 4-byte tag carrying a VLAN id (1–4094); a Linux sub-interface (bond0.10) sends/receives tagged frames.
• Trunk / access port — a switch trunk carries multiple tagged VLANs (plus one untagged native VLAN); an access port carries a single untagged VLAN and drops tags.
• Bridge — an in-kernel software Layer-2 switch (br-vm) that forwards frames between its ports, used to connect VMs/containers to a network.
• Policy routing — route selection by criteria beyond destination (source address, interface, fwmark) using additional routing tables and ip rule selectors.
• rp_filter — reverse-path filtering; validates a packet’s source is reachable via its ingress interface. Strict (1) drops asymmetric traffic; loose (2) is the multi-homed compromise.
• MTU / jumbo frames — the maximum L3 payload per frame (default 1500; jumbo commonly 9000); must match at every hop or large transfers black-hole.
• netfilter hook — one of the fixed points (prerouting, input, forward, output, postrouting) every packet traverses, where firewall base chains attach.
• Base-chain priority — the numeric ordering (filter = 0, srcnat = 100, etc.) that places an nftables chain relative to the kernel’s own hook operations.
• Set / map — an nftables named group (set) or keyed lookup (map/verdict map) that replaces many rules with one lookup and serves as the fleet-templating unit.
• conntrack — the kernel connection-tracking table (nf_conntrack) recording each flow’s state, enabling cheap stateful established,related accepts; finite and exhaustible.
• Zone — a firewalld grouping of interfaces/sources with a default target and an allow-list of services/ports, ordered by trust from drop to trusted.
• Rich rule — a firewalld rule expressing what a plain service cannot: source restriction, logging, rate limiting, and explicit accept/reject/drop.

Next steps

• Deepen the firewall itself with Designing Stateful Linux Firewalls with native nftables Rulesets and NAT — the companion that goes further on ruleset structure, NAT patterns, and zone modelling in raw nftables.
• Tune the kernel side of the network with Methodical Linux Performance Tuning: tuned, sysctl, and I/O Schedulers, which covers the net.core and net.ipv4 sysctls that govern buffers, queues, and throughput.
• Manage the units that persist all of this — systemd-networkd, nftables.service — with Mastering systemd: Units, Timers, Resource Control, and Service Hardening.
• When packets still misbehave after every layer looks right, scale up your capture skills with When Logs Aren’t Enough: Packet Capture, Traffic Mirroring, and Deep Network Troubleshooting.
• Confine the services behind the firewall at the OS level with Running SELinux in Enforcing Mode: Troubleshooting and Writing Custom Policy or Authoring AppArmor Profiles: Confining Services on Ubuntu and Debian.