Security groups and network ACLs gate traffic by IP and port. They have nothing to say about which domain an exfiltrating process is calling out to over 443. Egress control at layer 7 is where AWS Network Firewall earns its keep: a managed, horizontally scaled Suricata engine you drop into the data path to allowlist destinations by TLS SNI, run IDS/IPS signatures against your outbound traffic, and log every flow. The engine is the easy part. The hard part — the part that turns a “deployed” firewall into one that actually inspects traffic — is the route-table choreography and the rule-evaluation ordering. Get the routing wrong and traffic sails past untouched, or worse, gets blackholed by asymmetry that only appears in a multi-AZ production account and never in your single-AZ staging test.
This is the field guide for building that firewall correctly and engineering the rules to match. Network Firewall is not a router and not a NAT device — it is a bump in the wire, a Gateway Load Balancer-style VPC endpoint you place in each Availability Zone and steer traffic to with route tables that you own end to end. Nothing is implicit. We will build the centralized inspection VPC behind a Transit Gateway model that scales to hundreds of accounts, walk the per-AZ subnet and route-table layout that guarantees symmetric return paths, and then author the Suricata rules — domain-list allowlists, raw tls.sni signatures, DNS-layer alerts, and managed threat-signature groups — in the correct evaluation order so your explicit drop is the floor and not dead code. Every mechanism gets the real aws CLI, the JSON rule-group definition, and the Terraform, plus the limits (30,000 stateful capacity units, per-rule-group immutable capacity) and the failure modes a senior engineer only learns after an outage bridge.
By the end you will stop treating Network Firewall as a checkbox and start treating it as a data-plane appliance with a rule engine that has ordering semantics, capacity budgets, and logging you must design. You will know why aws:forward_to_sfe on the stateless default is load-bearing, why STRICT_ORDER inverts Suricata’s native precedence in exactly the way an allowlist firewall needs, why appliance mode on the inspection-VPC Transit Gateway attachment is not optional for stateful inspection, and how to read an EVE-JSON alert record to find the one legitimate destination you forgot to allowlist before your users find it for you.
What problem this solves
Every compromised workload, every misconfigured SDK, every piece of commodity malware eventually tries to talk to something on the internet — a command-and-control server, a data-exfiltration endpoint, a cryptominer pool, a typosquatted package registry. The classic AWS controls do not see any of this at the layer that matters. Security groups and NACLs operate on the 5-tuple: source IP, destination IP, protocol, source port, destination port. They can allow 0.0.0.0/0:443 or deny it, but they cannot distinguish s3.amazonaws.com from evil-exfil.example. Both are “some IP on port 443.” An attacker who has code execution inside your VPC and an allowed egress path to 443 has, effectively, an open channel — and 443 is almost always allowed because everything legitimate needs it too.
What breaks without egress inspection is not availability — it is containment. A single compromised Lambda, container, or EC2 instance becomes a data pump, and your first signal is a bill anomaly, a threat-intel notification, or a breach disclosure. Compliance frameworks have caught up to this: PCI DSS requires that outbound traffic be restricted to only what is necessary, and auditors increasingly want destination-based allowlisting, not just port rules. “We allow 443 to anywhere” no longer passes. The teams that hit this hardest are platform and landing-zone teams running many accounts, fintech and healthcare workloads under regulatory egress requirements, and anyone who has been through an incident where the post-mortem line was “we had no idea it was talking to that host, and no log that it ever did.”
Who also hits the implementation pain — separate from the security need — is anyone who deploys Network Firewall and finds that traffic either bypasses it entirely (because the route tables don’t force it through) or breaks intermittently in production (because a multi-AZ Transit Gateway load-balances the forward and return halves of a flow across different firewall endpoints and the stateful engine drops the return as out-of-state). The firewall being “deployed” and the firewall actually inspecting symmetric traffic are two very different states, and the gap between them is pure routing.
To frame the whole field before the deep dive, here is what each control layer can and cannot do, and where Network Firewall sits:
| Control | Operates at | Can allow/deny by | Cannot see | Stateful? | Where it runs |
|---|---|---|---|---|---|
| Security group | L3/L4 | IP, port, protocol, referenced SG | Domain, SNI, payload, DNS query | Yes (connection tracking) | ENI (instance/service) |
| Network ACL | L3/L4 | IP, port, protocol (subnet-wide) | Domain, SNI, payload, state | No (evaluate each packet) | Subnet boundary |
| Route table / blackhole | L3 | Destination CIDR only | Everything above L3 | No | Subnet / TGW |
| DNS Firewall (Route 53 Resolver) | DNS | Queried domain name | Anything not going through the resolver, raw-IP egress | Per-query | VPC resolver |
| AWS Network Firewall (stateless) | L3/L4 | 5-tuple, TCP flags | Domain, SNI, payload | No | Firewall endpoint / AZ |
| AWS Network Firewall (stateful/Suricata) | L3–L7 | SNI, HTTP Host, DNS query, signatures, flow | TLS payload (encrypted) | Yes (flow tracking) | Firewall endpoint / AZ |
The stateful row is the one that changes the game: it sees reassembled flows, tracks connection state, and understands TLS, DNS, and HTTP well enough to allowlist by the hostname a workload requested — without decrypting anything, because the SNI travels in the clear during the TLS handshake.
Learning objectives
By the end of this article you can:
- Choose between distributed and centralized Network Firewall deployment and justify the choice from account count, cost, and operational surface — and lay out the inspection VPC’s per-AZ subnets and route tables so traffic is actually forced through the firewall endpoint.
- Explain why a multi-AZ Transit Gateway breaks stateful inspection by default, and fix it with appliance mode plus symmetric per-AZ routing so both directions of every flow hit the same firewall endpoint.
- Distinguish the stateless and stateful engines, set the stateless default action to
aws:forward_to_sfeso Suricata actually runs, and reason about the fixed packet path through both engines. - Author both domain-list rule groups and raw Suricata-compatible rules —
tls.sniallow/deny pairs,dns.queryalerts, flow and signature rules — with$HOME_NET/$EXTERNAL_NETvariables scoped to your spoke CIDRs. - Choose between
STRICT_ORDERandDEFAULT_ACTION_ORDERrule evaluation, understand why strict order is mandatory for a default-deny allowlist, and set the policy default actions (aws:drop_established,aws:alert_established) that form your safety net. - Plan capacity: size each rule group’s immutable capacity with headroom and track the cumulative total against the 30,000-unit ceiling so a future rule group is not rejected at apply time.
- Import AWS-managed threat-signature rule groups, roll them out safely with
DROP_TO_ALERToverrides, tune out false positives by SID, and then enforce. - Configure alert and flow logging to S3 and CloudWatch, read the Suricata EVE-JSON schema, and write the day-2 “top blocked SNI” query that keeps the allowlist honest.
- Deploy the whole topology as code with Terraform and validate it end to end from a workload subnet.
Prerequisites & where this fits
You should be comfortable with core AWS networking: VPCs, subnets, route tables, Internet Gateways, NAT Gateways, and how a default route (0.0.0.0/0) selects a next hop. You need a working mental model of Transit Gateway — attachments, TGW route tables, and route propagation/association — because centralized inspection is built on it. Familiarity with the TLS handshake (specifically that the ClientHello carries the SNI in plaintext) and with basic Suricata/Snort rule syntax helps, though we build up the rule syntax from scratch. You should be able to run the aws CLI with credentials that can create firewall, VPC, and TGW resources, and read JSON output.
This sits in the cloud network security track, one layer below application delivery and one layer above raw connectivity. It assumes the hub-and-spoke fundamentals from Hub-and-Spoke vs Virtual WAN: Choosing an Enterprise Cloud Network Topology — the inspection VPC is the security hub in that model. It is the AWS-native counterpart to the Azure-centric Centralized Internet Egress: FQDN Filtering, Explicit Proxy, and TLS Inspection, and it complements the appliance-insertion pattern in Deploying HA Third-Party NVAs in Azure: The Load Balancer Sandwich Pattern — Network Firewall is AWS’s managed answer to that same “insert an inspection appliance in the path” problem, without you running the NVAs. For the account-foundation context, AWS Control Tower Guardrails: Building a Secure Multi-Account Foundation is where a centralized inspection VPC usually gets mandated and shared. The logging half pairs with Network Flow Logs to Insight: Building a Traffic Analytics and Detection Pipeline.
A quick map of who owns what during design and incidents, so you pull in the right team fast:
| Layer | What lives here | Who usually owns it | Failure classes it causes |
|---|---|---|---|
| Spoke VPC route tables | Default route to TGW; no local IGW/NAT | App / account team | Egress bypass (a stray IGW), broken connectivity |
| Transit Gateway | Attachments, TGW route tables, appliance mode | Network / platform team | Asymmetric flow drops (appliance mode off) |
| Inspection VPC subnets/routes | Firewall/TGW/NAT subnets per AZ, endpoint steering | Network security team | Traffic not forced through firewall; cross-AZ leaks |
| Firewall policy & rule groups | Stateless/stateful rules, order, capacity | Network security team | Dead-code rules, capacity apply failures, false-positive floods |
| Managed rule groups | AWS-maintained threat signatures | AWS (curated) + you (override) | Blocking legit traffic if enforced day one |
| Logging pipeline | Flow/alert logs to S3/CloudWatch | SecOps / platform | No audit trail; silent drops nobody sees |
Core concepts
Six mental models make every later decision obvious.
The firewall is a bump in the wire, not a router. Network Firewall does not have a routing table and does not make forwarding decisions. It creates a VPC endpoint (Gateway Load Balancer style) in each subnet you designate as a firewall subnet, one per AZ. Traffic reaches that endpoint only because you wrote route-table entries pointing at it. After inspection, the endpoint hands the packet back into the VPC and normal VPC routing takes over. If no route sends traffic to the endpoint, the firewall inspects nothing — it is deployed and idle. This is the single most important thing to internalize: routing is your job, inspection is the firewall’s job, and a firewall with wrong routes is a very expensive no-op.
Two engines run in a fixed order: stateless, then stateful. A firewall policy chains a stateless engine (fast, per-packet, 5-tuple, no connection awareness) and a stateful engine (Suricata — reassembled flows, connection state, protocol awareness). Every packet hits the stateless engine first. The stateless engine can aws:pass (accept, skip Suricata), aws:drop (reject, skip Suricata), or aws:forward_to_sfe (forward to the stateful engine). For an inspection design you want the stateless engine to do almost nothing and forward everything to Suricata, because all the interesting L7 logic — SNI allowlisting, IDS signatures, DNS filtering — lives in the stateful engine. If a packet is aws:pass-ed or aws:drop-ped statelessly, Suricata never sees it.
Stateful inspection demands flow symmetry. Suricata tracks connection state. To do that correctly it must see both directions of a flow — the SYN and the SYN-ACK, the ClientHello and the ServerHello — on the same engine instance. In a single-AZ deployment this is automatic. In a multi-AZ Transit Gateway topology it is emphatically not: by default the TGW hashes forward and return packets independently and can send a flow’s outbound half through the AZ-a firewall endpoint and its return half through the AZ-b endpoint. The AZ-b engine never saw the handshake, treats the return packets as out-of-state, and drops them under the default stream-exception policy. The connection hangs. The fix — appliance mode plus per-AZ symmetric routing — exists solely to restore this symmetry.
SNI filtering reads, it does not decrypt. When a client opens a TLS connection it sends a ClientHello that includes the Server Name Indication — the hostname it wants to reach — in plaintext, before any encryption is negotiated. Suricata reads that field with the tls.sni keyword. This is why you can allowlist .amazonaws.com over 443 without terminating TLS, holding a private key, or running a man-in-the-middle proxy. The cost is honesty about the limit: you see the requested hostname, not the payload, and evasions exist (Encrypted Client Hello / ECH, domain fronting, raw-IP connections). SNI allowlisting raises the bar enormously and catches the overwhelming majority of misconfigured and commodity-malware egress, but it is one control, not the whole story.
Rule order has two modes, and the default is backwards for allowlisting. The stateful engine can evaluate rules in action order (Suricata’s native precedence: pass beats drop beats reject beats alert, regardless of where rules sit) or in strict order (rule groups by integer priority, rules top-to-bottom, first match wins). For a default-deny allowlist firewall, action order is exactly wrong — a stray pass anywhere would override your drop. Strict order lets you place explicit passes above an explicit drop floor in a deterministic sequence, which is the only sane way to build “allow this list, deny everything else.”
Capacity is reserved up front and cannot be resized. Every rule group reserves a fixed capacity (a unit count) at creation and that number is immutable — to grow a group you replace it. A firewall policy caps total stateful capacity at 30,000 units across all its stateful rule groups (with a separate stateless budget). Suricata rules cost one unit each; domain-list entries cost capacity proportional to the list size times the number of TargetTypes. Nobody plans for this until an apply fails, so you size each group with headroom and track the running total.
The vocabulary in one table
Pin down every moving part before the deep sections. The glossary at the end repeats these for lookup; this table is the mental model side by side:
| Concept | One-line definition | Where it lives | Why it matters |
|---|---|---|---|
| Firewall | The Network Firewall resource; ties a policy to VPC subnets | Inspection VPC | The thing you associate a policy with |
| Firewall endpoint | Per-AZ VPC endpoint traffic is routed through | Firewall subnet per AZ | The next hop your routes point at |
| Firewall policy | The container for stateless + stateful rule groups and defaults | Region | Reusable across firewalls |
| Stateless rule group | Per-packet 5-tuple rules | Referenced by policy | Fast pre-filter; usually just forwards to SFE |
| Stateful rule group | Suricata rules or domain lists | Referenced by policy | Where L7 allowlisting and IDS live |
aws:forward_to_sfe |
Stateless action: send to the stateful engine | Stateless default / rule | Off = Suricata never runs |
STRICT_ORDER |
First-match, priority-ordered stateful evaluation | Policy StatefulEngineOptions |
Required for default-deny allowlisting |
$HOME_NET |
Rule variable = your internal/spoke CIDRs | Rule group scope | Defines “inside” for rule direction |
| Domain-list rule group | Managed allow/deny by SNI + HTTP Host | Stateful rule group | Bulk allowlist without raw Suricata |
| Managed rule group | AWS-maintained threat signatures | Referenced by ARN | Curated IDS/IPS, auto-updated |
| Capacity | Immutable unit reservation per rule group | Rule group | Ceiling 30,000 stateful per policy |
| Appliance mode | TGW option pinning a flow to one AZ | Inspection-VPC TGW attachment | Off = asymmetric drops behind multi-AZ TGW |
| Stream exception policy | What to do with packets outside a tracked stream | Policy | DROP is the secure default |
Deployment models: distributed vs centralized inspection VPC
You can run Network Firewall two ways, and the choice dictates the entire routing design and the bill. Get this decision right before you write a single rule.
| Model | Where firewall endpoints live | Inspects | Scales to | Cost shape |
|---|---|---|---|---|
| Distributed | One firewall per spoke VPC, one endpoint per AZ in that VPC | That VPC’s own traffic only | A handful of VPCs | N firewalls (endpoint-hours × AZs × N) |
| Centralized | One firewall in a dedicated inspection VPC behind a Transit Gateway | All spoke egress and (optionally) east-west | Tens to hundreds of VPCs | One firewall, shared; TGW data-processing added |
Distributed is simpler to reason about per VPC — the routes stay inside one VPC, there is no TGW, and a mistake is blast-radius-limited to that VPC. But every new VPC that needs inspection means a new firewall, a new set of per-AZ endpoints (each billed hourly), and a new set of rule groups to keep in sync. Operationally it does not scale, and financially it multiplies the fixed endpoint cost by your VPC count. It is the right model only when you have a small, fixed number of VPCs, or when a specific VPC has a compliance boundary that forbids its traffic transiting a shared inspection VPC.
Centralized puts a single firewall in a dedicated inspection VPC and routes every spoke’s traffic through it via a Transit Gateway. Spokes have no internet path of their own — no IGW, no NAT — so there is no legal egress that bypasses the firewall. The TGW pulls spoke traffic into the inspection VPC, the firewall endpoints scrub it, and a single egress path (NAT Gateway plus Internet Gateway) sits behind the firewall. You maintain one policy, one rule set, one log pipeline, and you add a new spoke by attaching it to the TGW and pointing its default route at the TGW. For any platform running more than a few VPCs, this is the model.
The trade-offs enumerated:
| Dimension | Distributed | Centralized |
|---|---|---|
| Firewall endpoints billed | Per VPC × per AZ | Once × per AZ (shared) |
| Rule-set consistency | Manual sync across firewalls | Single source of truth |
| Blast radius of a bad rule | One VPC | Every spoke at once |
| Transit Gateway required | No | Yes (and appliance mode) |
| East-west inspection | Not without extra design | Possible via inspection routing |
| New-VPC onboarding | Deploy a full firewall stack | Attach to TGW, add a route |
| TGW data-processing cost | None | Per-GB through the TGW |
| Single point of failure | No (isolated) | Mitigated by multi-AZ endpoints |
| Best for | ≤ a few VPCs, hard isolation | Landing zones, many accounts |
The inspection VPC in the centralized model holds, per AZ, three subnets, each with a distinct role. This layout is the skeleton everything else hangs on:
| Subnet (per AZ) | Contains | Its route table sends 0.0.0.0/0 to |
Its route table sends spoke CIDRs to |
|---|---|---|---|
| Firewall subnet | The firewall endpoint for that AZ | NAT Gateway in that AZ (post-inspection egress) | Transit Gateway (return to spokes) |
| TGW-attachment subnet | The TGW ENIs for that AZ | Firewall endpoint in that AZ (force ingress into firewall) | — (traffic arrives from TGW) |
| NAT subnet (public) | The NAT Gateway + route to IGW | Internet Gateway | Firewall endpoint in that AZ (return symmetry) |
The firewall endpoint is the hinge every route table bends around. Read the table as a loop: spoke traffic enters the inspection VPC via the TGW attachment, the TGW-attachment subnet’s route forces it into the firewall endpoint, the firewall inspects and returns it, the firewall subnet’s default route sends the now-clean traffic to the NAT Gateway, the NAT subnet’s default route sends it to the IGW and out. Return traffic reverses: it comes back through the IGW to the NAT Gateway, the NAT subnet’s route sends it back through the firewall endpoint (this is the symmetry-critical entry), and from the firewall subnet the spoke-CIDR route hands it to the TGW to reach the origin spoke.
The endpoint-and-route dance: appliance mode and symmetric return paths
This is where most implementations break, and where the failure is a heisenbug that passes every test and fails in production. Read this section twice.
Stateful inspection requires that both directions of a flow traverse the same firewall endpoint. In a multi-AZ TGW topology, the default TGW behavior hashes the forward and return packets independently. A flow’s SYN can enter via the us-east-1a endpoint and its SYN-ACK return can enter via us-east-1b. The stateful Suricata instance in 1b never saw the SYN, cannot associate the return packet with a tracked stream, and drops it as out-of-state under the policy’s stream-exception setting (DROP by default). The connection hangs, then times out. In a three-AZ account, roughly one connection in three exhibits this — the “one in three” is literally the three-AZ hash.
The fix is appliance mode on the inspection VPC’s TGW attachment. Appliance mode makes the TGW pin all packets of a flow (same 5-tuple) to the same AZ for the life of the flow, in both directions, restoring symmetry. It is a single attachment option, and it is not optional — a stateful firewall behind a multi-AZ TGW without appliance mode is a latent outage.
# Enable appliance mode on the inspection VPC's TGW attachment.
# Without this, asymmetric routing silently breaks stateful rules under multi-AZ.
aws ec2 modify-transit-gateway-vpc-attachment \
--transit-gateway-attachment-id tgw-attach-0abc123def456 \
--options ApplianceModeSupport=enable
Appliance mode is necessary but not sufficient. You also must ensure each AZ’s inspection-VPC subnets reference only that AZ’s firewall endpoint. If AZ-a’s TGW-attachment subnet routes to AZ-b’s firewall endpoint, you have manufactured cross-AZ asymmetry by hand and defeated the guarantee. The rule is: one firewall endpoint per AZ, referenced only by that AZ’s subnets.
Here is the complete route choreography in the inspection VPC, per AZ, as a reference table (the CLI to create each follows):
| Route table (per AZ) | Destination | Target | Purpose |
|---|---|---|---|
| TGW-attachment subnet | 0.0.0.0/0 |
Firewall endpoint (this AZ) | Force all spoke ingress into the firewall |
| Firewall subnet | 0.0.0.0/0 |
NAT Gateway (this AZ) | Post-inspection egress toward the internet |
| Firewall subnet | Spoke summary (e.g. 10.0.0.0/8) |
Transit Gateway | Return clean traffic to the origin spoke |
| NAT subnet | 0.0.0.0/0 |
Internet Gateway | Egress to the internet |
| NAT subnet | Spoke summary (e.g. 10.0.0.0/8) |
Firewall endpoint (this AZ) | Return path re-enters the firewall (symmetry) |
The CLI to build the two most important, most-often-missed entries:
# TGW-attachment subnet: send everything to THIS AZ's firewall endpoint.
aws ec2 create-route \
--route-table-id rtb-tgw-az-a \
--destination-cidr-block 0.0.0.0/0 \
--vpc-endpoint-id vpce-0fw1111aaaa # firewall endpoint, AZ a
# NAT subnet: the return path to spokes goes BACK through the firewall (symmetry).
# This is the entry people forget, and its absence is asymmetric drops.
aws ec2 create-route \
--route-table-id rtb-nat-az-a \
--destination-cidr-block 10.0.0.0/8 \
--vpc-endpoint-id vpce-0fw1111aaaa
On the spoke side, the spoke VPC route tables point 0.0.0.0/0 at the TGW; the TGW route table associated with spokes points its default route at the inspection VPC attachment. The spokes have no IGW and no NAT of their own — that is the entire point. There is no legal egress path that bypasses the firewall. If you find a spoke with a working direct egress, you have a routing leak, and it is a finding, not a feature.
The spoke and TGW side, tabulated:
| Route table | Destination | Target | Note |
|---|---|---|---|
| Spoke VPC (each subnet) | 0.0.0.0/0 |
Transit Gateway | No local IGW/NAT exists |
| TGW route table (spoke association) | 0.0.0.0/0 |
Inspection VPC attachment | Pulls egress into inspection |
| TGW route table (spoke association) | Other spoke CIDRs | Inspection VPC attachment | Forces east-west through firewall too (optional) |
| TGW route table (inspection association) | Spoke CIDRs | Each spoke attachment | Return routing to origin spoke |
A note on the two-route-table pattern at the TGW: keep spokes and the inspection VPC in separate TGW route tables. Spokes associate to a route table whose default sends everything to inspection; the inspection VPC associates to a route table that knows the individual spoke CIDRs for return. This clean separation is what makes “all spoke traffic goes through inspection, and inspected traffic returns to the right spoke” hold without route conflicts.
The failure modes this section exists to prevent:
| Symptom | Root cause | How to confirm | Fix |
|---|---|---|---|
| ~1/N of connections hang (N = AZ count) | Appliance mode off; forward/return split across AZs | describe-transit-gateway-vpc-attachments → ApplianceModeSupport ≠ enable |
Enable appliance mode |
| Intermittent hangs even with appliance mode on | An AZ’s subnet routes to another AZ’s endpoint | Audit each route table’s vpc-endpoint-id vs AZ |
One endpoint per AZ, referenced only locally |
| Traffic reaches internet uninspected | TGW-attachment subnet has no route to the endpoint | Check rtb-tgw-az-* for the 0.0.0.0/0 → endpoint route |
Add the force-into-firewall route |
| Return traffic never reaches spokes | Firewall subnet lacks a spoke-CIDR route to TGW | Check firewall subnet route table for 10.0.0.0/8 → TGW |
Add the return-to-TGW route |
| A spoke has working direct egress (bypass) | Spoke still has an IGW/NAT and a 0.0.0.0/0 to it |
describe-route-tables on the spoke |
Remove the local internet route; default to TGW |
Stateless vs stateful engines and rule evaluation order
A Network Firewall policy chains two engines, and traffic flows through them in a fixed order. Understanding this order is the difference between a rule that fires and one that is dead code.
packet ──▶ [ stateless engine ] ──(aws:forward_to_sfe)──▶ [ stateful engine ] ──▶ verdict
│
├──(aws:pass)──▶ accept (Suricata never sees it)
└──(aws:drop)──▶ reject (Suricata never sees it)
- Stateless engine — evaluates first, per packet, on the 5-tuple plus TCP flags and a few L3/L4 attributes. Actions are
aws:pass,aws:drop, oraws:forward_to_sfe. It is fast and cheap but has no connection awareness and no L7 visibility. Rules match on source/destination CIDR, port ranges, protocol number, and optionally TCP flags; there is no notion of “SNI” or “domain” here. - Stateful engine — this is Suricata. It sees reassembled flows, tracks connection state, and understands protocols (TLS, DNS, HTTP, and more). This is where SNI allowlisting, DNS filtering, and IDS signatures live. It only ever sees traffic the stateless engine forwarded with
aws:forward_to_sfe.
The action semantics, enumerated:
| Engine | Action | Meaning | Continues to next engine? |
|---|---|---|---|
| Stateless | aws:pass |
Accept the packet | No — skips stateful entirely |
| Stateless | aws:drop |
Drop the packet | No — skips stateful entirely |
| Stateless | aws:forward_to_sfe |
Hand to the stateful engine | Yes — Suricata evaluates |
| Stateful | pass (Suricata) |
Accept the flow’s packets | Verdict |
| Stateful | drop (Suricata) |
Silently drop | Verdict |
| Stateful | reject (Suricata) |
Drop and send TCP RST/ICMP unreachable | Verdict (fails fast, not silent) |
| Stateful | alert (Suricata) |
Log, take no blocking action | Continues evaluating |
The critical knob for an inspection design is the stateless default action — the action applied to packets no stateless rule matched. For inspection you want the stateless engine to do almost nothing except hand everything to Suricata:
{
"StatelessDefaultActions": ["aws:forward_to_sfe"],
"StatelessFragmentDefaultActions": ["aws:forward_to_sfe"],
"StatefulEngineOptions": {
"RuleOrder": "STRICT_ORDER",
"StreamExceptionPolicy": "DROP"
}
}
Three settings here are load-bearing:
| Setting | Value here | Why | Alternatives and their trade-off |
|---|---|---|---|
StatelessDefaultActions |
["aws:forward_to_sfe"] |
So every unmatched packet reaches Suricata | ["aws:pass"] bypasses inspection; ["aws:drop"] blocks everything |
StatelessFragmentDefaultActions |
["aws:forward_to_sfe"] |
Fragments must also reach Suricata for reassembly | Forgetting this drops or passes fragments unevaluated |
RuleOrder |
STRICT_ORDER |
First-match, deterministic; required for default-deny | DEFAULT_ACTION_ORDER = Suricata precedence (pass wins) |
StreamExceptionPolicy |
DROP |
Secure default for packets outside a tracked stream | CONTINUE (evaluate anyway), REJECT (RST) — safety vs fewer broken flows |
StreamExceptionPolicy decides what happens to packets the engine cannot associate with a tracked stream — mid-stream packets after a flush, out-of-window data, or (crucially) the asymmetric-return packets described earlier. DROP is the secure default and the correct choice for an allowlist firewall. Note the interaction: if you have not fixed appliance mode, DROP is exactly what turns your asymmetric routing into hung connections. CONTINUE would mask the routing bug by evaluating the orphaned packets anyway — do not use it to paper over a routing problem; fix the routing.
Strict order vs default action order
By default the stateful engine uses action-order evaluation (DEFAULT_ACTION_ORDER), which applies Suricata’s native precedence: pass > drop > reject > alert. That means a pass rule anywhere in your rule set wins over a drop, regardless of where you placed it. For an allowlist firewall that is exactly backwards — you want your explicit drop of non-allowlisted SNI to be the floor, with pass rules above it in a deterministic sequence. That is what STRICT_ORDER gives you: rule groups evaluate in the order of their integer Priority (lower first), rules within a group evaluate top-to-bottom, and the first match wins.
The two modes contrasted:
| Aspect | DEFAULT_ACTION_ORDER |
STRICT_ORDER |
|---|---|---|
| Evaluation | Suricata native precedence | Priority + top-to-bottom, first match wins |
| Which action wins | pass always beats drop |
Whichever matches first |
| Default action configurable | No (implicit) | Yes — you set StatefulDefaultActions |
| Good for | Pure IDS/alerting, additive detection | Default-deny allowlisting, explicit control |
| Risk | A stray pass silently opens a hole |
A misordered rule silently shadows another |
| Rule-group priority respected | No | Yes |
With strict order you also set an explicit default action for the policy — the verdict for flows no rule matched. To get default-deny egress, add the drop action to the stateful default:
{
"StatefulEngineOptions": { "RuleOrder": "STRICT_ORDER" },
"StatefulDefaultActions": [
"aws:drop_established",
"aws:alert_established"
]
}
aws:drop_established drops packets on flows the engine has established but that no rule explicitly passed — your safety net. aws:alert_established logs those same flows so the default-drop is visible in the alert log, not silent. The full set of stateful default actions:
| Default action | Effect | Use when |
|---|---|---|
aws:drop_strict |
Drop all packets not matching a pass rule (strict) | Hard default-deny, including non-established |
aws:drop_established |
Drop packets on established flows with no pass | Default-deny that still lets handshakes reach rules |
aws:alert_strict |
Alert on all non-matching packets | Visibility without blocking (audit mode) |
aws:alert_established |
Alert on established non-matching flows | Log the default-drop so it is not silent |
A common rollout is to start with aws:alert_established only (no drop) to observe what the default would block, confirm the allowlist is complete from the alert log, then add aws:drop_established to enforce. That single change — alert-then-drop — is how you avoid severing legitimate traffic on cutover day.
Authoring Suricata rules: SNI allowlists, domain lists, and signatures
There are two ways to express stateful rules: domain-list rule groups (a managed abstraction) and Suricata-compatible rule strings (raw signatures). Use domain lists for the bulk allowlist and Suricata strings for protocol-aware logic and IDS.
Domain-list rule groups
A domain-list rule group is the easiest way to allowlist (or denylist) by hostname across both TLS SNI and HTTP Host. You provide a target list and a type:
{
"RulesSource": {
"RulesSourceList": {
"GeneratedRulesType": "ALLOWLIST",
"TargetTypes": ["TLS_SNI", "HTTP_HOST"],
"Targets": [
".amazonaws.com",
".pypi.org",
".pythonhosted.org",
"registry.npmjs.org",
".ghcr.io",
".docker.io",
".ubuntu.com"
]
}
}
}
The parameters, enumerated:
| Field | Values | Meaning |
|---|---|---|
GeneratedRulesType |
ALLOWLIST | DENYLIST |
Allow only these (deny rest) vs deny these (allow rest) |
TargetTypes |
TLS_SNI, HTTP_HOST |
Which protocol field(s) to match on |
Targets |
list of domains | The hostnames to match |
A leading dot (.amazonaws.com) matches the domain and all subdomains; without the dot the match is exact (registry.npmjs.org matches only that host, not foo.registry.npmjs.org). TLS_SNI inspects the SNI in the ClientHello — no decryption, since SNI is sent in the clear. HTTP_HOST matches the plaintext Host header of an unencrypted HTTP/1.x request. Matching behavior, tabulated:
| Target written as | Matches | Does not match |
|---|---|---|
.amazonaws.com |
s3.amazonaws.com, sqs.us-east-1.amazonaws.com, amazonaws.com |
evilamazonaws.com |
registry.npmjs.org (no dot) |
exactly registry.npmjs.org |
www.registry.npmjs.org |
.example.com |
a.example.com, a.b.example.com |
example.com.evil.net |
Domain lists are the right tool for a large, mostly-static allowlist maintained by a platform team, because a security engineer can read and PR-review the list without knowing Suricata syntax. Their limit is expressiveness: a domain list cannot express “allow this SNI only from this source CIDR,” cannot alert without blocking, and cannot do DNS-layer or signature matching. For those, drop to raw Suricata.
An important behavioral caveat: an ALLOWLIST domain-list group implicitly denies everything else of the matched protocol. Under STRICT_ORDER, be deliberate about where this group sits relative to your other rules and the policy default — the managed abstraction still participates in the ordering.
Raw Suricata rules: the TLS SNI allow/deny pair
For finer control, author raw Suricata. To allow specific SNIs and drop everything else over TLS, ordering matters (see the strict-order section). A protocol-aware allow-then-deny pair:
# Allow TLS to approved AWS endpoints (suffix match on the SNI).
pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; \
dotprefix; content:".amazonaws.com"; endswith; \
flow:to_server; msg:"ALLOW AWS endpoints"; sid:1000001; rev:1;)
# Allow the npm registry (exact host match).
pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; \
content:"registry.npmjs.org"; nocase; \
flow:to_server; msg:"ALLOW npm registry"; sid:1000002; rev:1;)
# Floor: drop any other TLS handshake by SNI (default-deny for 443).
drop tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; \
flow:to_server; msg:"DENY non-allowlisted TLS SNI"; sid:1000099; rev:1;)
Anatomy of a Suricata rule, field by field, because every part carries meaning:
| Element | Example | Meaning |
|---|---|---|
| Action | pass / drop / reject / alert |
What to do on match |
| Protocol | tls, dns, http, tcp, ip |
App-layer protocol the rule applies to |
| Source | $HOME_NET any |
Source address var + port (any) |
| Direction | -> |
Forward only (<> = bidirectional) |
| Destination | $EXTERNAL_NET any |
Destination address var + port |
tls.sni |
sticky buffer | Match subsequent content against the SNI |
dotprefix |
transform | Prepend a dot so suffix matches align on labels |
content:"…" |
".amazonaws.com" |
The bytes to match in the buffer |
endswith |
modifier | Anchor the match to the end of the buffer |
nocase |
modifier | Case-insensitive match |
flow:to_server |
flow keyword | Only client→server direction (avoids double-eval) |
msg:"…" |
metadata | Human label that appears in the alert log |
sid:… |
1000099 |
Unique signature ID (yours: use ≥ 1000000) |
rev:… |
1 |
Revision; bump when you edit the rule |
The tls.sni sticky-buffer with dotprefix + endswith is the canonical way to do suffix (subdomain) matching on a domain. dotprefix prepends a . to the buffer so that content:".amazonaws.com"; endswith matches s3.amazonaws.com but not notamazonaws.com — the dot forces a label boundary. Without dotprefix, endswith:"amazonaws.com" would also match evilamazonaws.com, which is a real allowlist-bypass class.
The keyword reference you will reach for most when writing egress rules:
| Keyword | Buffer / matches | Typical use |
|---|---|---|
tls.sni |
SNI from the ClientHello | Allowlist/denylist HTTPS by hostname |
tls.cert_subject |
Certificate subject (ServerHello) | Match on cert CN/O (seen post-handshake) |
tls.version |
Negotiated TLS version | Alert on legacy TLS 1.0/1.1 |
dns.query |
DNS question name | Filter/alert on queried domains |
http.host |
HTTP Host header |
Allowlist cleartext HTTP by host |
http.uri |
Request URI | Detect known-bad paths |
http.user_agent |
User-Agent header |
Detect tooling/malware UAs |
flow |
Flow state/direction | established, to_server, stateless |
dsize |
Payload size | Anomaly/beacon detection |
DNS-layer alerting
You can match DNS queries directly, which is useful for catching name resolution to suspicious TLDs or known-bad domains even before a TLS connection is attempted:
# Alert on DNS queries to the .onion pseudo-TLD (Tor).
alert dns $HOME_NET any -> any any (dns.query; \
content:".onion"; nocase; endswith; \
msg:"DNS query for .onion TLD"; sid:1000201; rev:1;)
# Alert on DNS queries to dynamic-DNS providers often used for C2.
alert dns $HOME_NET any -> any any (dns.query; \
content:".duckdns.org"; nocase; endswith; \
msg:"DNS query to dynamic DNS provider"; sid:1000202; rev:1;)
Note the -> any any rather than -> $EXTERNAL_NET any: DNS often goes to the VPC’s .2 resolver (an internal address), so scoping the destination to external would miss it. DNS-layer rules complement, but do not replace, SNI filtering — a workload can connect to a raw IP with no DNS lookup, and DNS filtering alone does not stop that.
An honest boundary on SNI filtering
SNI filtering is not TLS interception. You see the requested hostname, not the payload, and a determined adversary has evasions. The threats and what SNI filtering does about each:
| Evasion | How it works | Does SNI filtering stop it? | Mitigation |
|---|---|---|---|
| Plain misconfig / commodity malware | Calls a hostname you did not allowlist | Yes — the common case | Allowlist + default-drop |
| Domain fronting | SNI = allowed CDN host; real host in encrypted Host |
Partially — blocks unless the CDN is allowlisted | Restrict which CDN hostnames are allowed |
| ECH / Encrypted Client Hello | SNI is encrypted in the ClientHello | No — SNI is hidden | Block ECH-capable destinations; rely on IP/DNS reputation |
| Raw-IP egress (no SNI, no DNS) | Connects to an IP directly | No at L7 | Combine with stateless CIDR rules / IP reputation lists |
| DNS-over-HTTPS to exfiltrate | Tunnels data in DoH to an allowed resolver | Only if the DoH host is not allowlisted | Denylist public DoH endpoints; force internal resolver |
Treat SNI allowlisting as one strong control in a layered egress posture — pair it with DNS Firewall, managed threat-signature groups (IP/domain reputation), and flow-log analytics. It raises the bar enormously and catches the overwhelming majority of real-world egress; it is not a hermetic seal.
HOME_NET and rule variables
Suricata rules reference address sets by variable — $HOME_NET (your internal networks) and $EXTERNAL_NET (everything else) — and you must define these on the rule group or the rules match nothing useful. In Network Firewall, rule variables live in the rule group’s RuleVariables and are referenced by the Suricata rules within that group.
{
"RuleVariables": {
"IPSets": {
"HOME_NET": {
"Definition": ["10.0.0.0/8", "172.16.0.0/12"]
}
}
}
}
Point $HOME_NET at your spoke CIDR summary — the address space of the workloads whose egress you are inspecting. $EXTERNAL_NET defaults to “not $HOME_NET” (!$HOME_NET) if you do not override it, which is usually what you want for egress rules written as $HOME_NET any -> $EXTERNAL_NET any. The variables you will define and why:
| Variable | Typical definition | Why it matters |
|---|---|---|
HOME_NET |
Your spoke CIDR summary (10.0.0.0/8) |
Defines “inside”; rules keyed on it distinguish egress from ingress |
EXTERNAL_NET |
!$HOME_NET (default) or explicit |
Defines “outside”; the destination side of egress rules |
Custom IP set (e.g. HTTP_SERVERS) |
Specific internal server CIDRs | Scope rules to a tier without listing CIDRs inline |
Custom port set (e.g. HTTP_PORTS) |
80,8080 |
Reuse a port list across rules |
Two gotchas that bite here. First, $HOME_NET is per rule group, so if you have several stateful rule groups they each need it defined (or defined consistently) — a rule group where $HOME_NET is empty or wrong will silently fail to match. Second, get the direction right: a rule written $HOME_NET any -> $EXTERNAL_NET any matches egress (inside to outside); reversing it matches ingress, which is a different (and for pure egress inspection, usually empty) traffic set. Combined with flow:to_server, the direction keyword ensures you evaluate the client→server half of the flow once, cleanly.
Capacity planning and rule-group sizing
Capacity is the constraint nobody plans for until provisioning fails. Every rule group reserves a fixed capacity (a unit count) at creation, and you cannot change it later — you size it up front and replace the group to grow it. A firewall policy has a hard ceiling of 30,000 capacity units across all its stateful rule groups (with a separate budget for stateless). How each rule type consumes capacity:
| Rule type | Capacity cost | Notes |
|---|---|---|
| Suricata stateful rule (5-tuple) | 1 unit each | Simple, predictable |
| Domain-list entry | ~1 unit × number of TargetTypes |
TLS_SNI + HTTP_HOST doubles it |
| Stateless rule | Product of the ranges it matches | Port/CIDR ranges multiply — can be large |
| Stateful rule group (reserved) | The --capacity you set at create |
Immutable; size with headroom |
Size each group with headroom for rev: bumps and new entries, and track the cumulative total against 30,000 so a future rule group is not rejected at apply time. Reserve capacity generously — the cost of over-reserving is only that you cannot exceed the 30,000 total, not a per-unit charge:
# Reserve capacity generously; you cannot resize a rule group after creation.
aws network-firewall create-rule-group \
--rule-group-name egress-tls-allowlist \
--type STATEFUL \
--capacity 200 \
--rule-group file://allowlist.json
A worked budget for a realistic policy, so you can see the 30,000 ceiling is generous but not infinite:
| Rule group | Contents | Reserved capacity | Running total |
|---|---|---|---|
egress-tls-allowlist |
~80 Suricata SNI passes + drop floor | 200 | 200 |
egress-domain-allowlist |
~300 domains × 2 target types | 700 | 900 |
dns-suspicious-alerts |
~40 DNS alert rules | 100 | 1,000 |
aws-managed-botnet (managed) |
AWS-maintained | (counts against managed budget) | 1,000 |
custom-ids-signatures |
~250 signatures | 400 | 1,400 |
| Headroom to ceiling | — | — | 28,600 free |
The discipline: pick a capacity per group that is 2–3× current need, document the running total in your IaC, and add a check (or just a comment) so a new group’s reservation is a conscious decision, not a surprise at terraform apply.
Importing and tuning managed threat-signature groups
AWS publishes managed rule groups maintained by its threat-intelligence team, plus curated domain/IP reputation lists. They drop into a policy by ARN and update automatically — you get fresh signatures without editing anything. The families you will see:
| Managed rule group (family) | Detects | Typical posture |
|---|---|---|
ThreatSignaturesBotnet |
Botnet C2 traffic | Alert → then drop |
ThreatSignaturesMalware |
Known malware network behavior | Alert → then drop |
ThreatSignaturesMalwareCoinmining |
Cryptomining pool traffic | Drop (usually safe early) |
ThreatSignaturesScanners |
Scanning/recon | Alert (noisy; tune) |
ThreatSignaturesDoS |
DoS tool traffic | Alert → drop |
ThreatSignaturesWebAttacks |
Web-attack tooling | Alert → drop |
AbusedLegitDomains (reputation) |
Legit domains abused for malware hosting | Alert first (false-positive risk) |
MalwareDomains / IP reputation lists |
Known-bad domains/IPs | Drop |
List them so you have the exact ARNs to reference:
aws network-firewall list-rule-groups \
--scope MANAGED \
--managed-type AWS_MANAGED_THREAT_SIGNATURES \
--query 'RuleGroups[].Name'
The failure mode with managed IDS signatures is false-positive flooding: enable a broad signature set in drop posture on day one and you will sever legitimate traffic and page yourself at 2 a.m. The disciplined rollout has three phases:
| Phase | Posture | What you do | Exit criterion |
|---|---|---|---|
| 1. Observe | DROP_TO_ALERT override |
Add the group; it alerts but does not block | One week of clean-ish alert data |
| 2. Tune | DROP_TO_ALERT override |
Find SIDs firing on known-good; add targeted passes above the group (strict order) | Alert stream free of legit-traffic hits |
| 3. Enforce | Remove override | Group now drops | Confirmed low false-positive rate |
Phase 1 adds the managed group in alert-only posture using an override so you can flip its action to DROP_TO_ALERT without editing the (AWS-owned) group:
{
"ResourceArn": "arn:aws:network-firewall:us-east-1:aws-managed:stateful-rulegroup/ThreatSignaturesBotnet",
"Priority": 200,
"Override": { "Action": "DROP_TO_ALERT" }
}
Phase 2 runs for a week, then you pull the alert logs, identify the SIDs firing on known-good traffic, and build a targeted pass rule above the managed group (strict order) to whitelist that exact flow — never a blanket disable. Phase 3 removes the DROP_TO_ALERT override so the group enforces.
The suppression rule you must internalize: never bulk-suppress by disabling a managed group. Suppress the specific SID for the specific source, and document why. A blanket disable of ThreatSignaturesMalware because one signature was noisy is how a real intrusion walks straight through. If you truly must silence a single signature, do it narrowly and with a comment recording the ticket and the date.
Alert and flow logging to S3/CloudWatch and the log schema
A firewall that drops silently is a firewall you cannot operate. Network Firewall emits two log types, configured independently, and you should enable both:
| Log type | Records | Volume | Destination choice | Primary use |
|---|---|---|---|---|
| Flow | Connection-level (5-tuple, bytes, packets) for every tracked flow | High | S3 (cheap, queryable via Athena) | Audit trail; who talked to whom |
| Alert | Traffic matching a rule with alert/drop — includes SID and L7 fields (SNI) |
Lower | CloudWatch Logs (fast query) or S3 | Tuning, incident response, “what got blocked” |
| TLS (optional) | TLS-inspection events (only if you configure TLS inspection) | Varies | S3/CloudWatch | Only relevant with TLS termination configured |
A logging configuration sending flow logs to S3 and alert logs to CloudWatch:
{
"LoggingConfiguration": {
"LogDestinationConfigs": [
{
"LogType": "FLOW",
"LogDestinationType": "S3",
"LogDestination": { "bucketName": "acme-nfw-logs", "prefix": "flow" }
},
{
"LogType": "ALERT",
"LogDestinationType": "CloudWatchLogs",
"LogDestination": { "logGroup": "/anfw/alert" }
}
]
}
}
The destination options, tabulated, because the choice affects cost and query latency:
| Destination | Log types | Cost shape | Query with | Best for |
|---|---|---|---|---|
| S3 | Flow, Alert, TLS | Cheapest storage; per-GB | Athena / Glue | High-volume flow logs; long retention |
| CloudWatch Logs | Flow, Alert, TLS | Ingestion + storage per-GB | Logs Insights | Fast interactive alert queries during incidents |
| Kinesis Data Firehose | Flow, Alert, TLS | Firehose + downstream | Downstream (OpenSearch, etc.) | Streaming to a SIEM |
A decoded alert record — the event object is Suricata’s EVE JSON:
{
"firewall_name": "acme-egress-inspection",
"availability_zone": "us-east-1a",
"event_timestamp": "1717842000",
"event": {
"src_ip": "10.20.4.55",
"src_port": 51520,
"dest_ip": "151.101.0.223",
"dest_port": 443,
"proto": "TCP",
"app_proto": "tls",
"tls": { "sni": "pypi.org", "version": "TLS 1.3" },
"alert": {
"action": "blocked",
"signature": "DENY non-allowlisted TLS SNI",
"signature_id": 1000099,
"rev": 1
}
}
}
The fields that matter operationally:
| Field | Meaning | Why you care |
|---|---|---|
event.alert.action |
allowed vs blocked |
Did the rule pass or drop this flow? |
event.alert.signature_id |
The SID that matched | The rule to tune (suppress/whitelist) against |
event.alert.signature |
The msg: text |
Human-readable which rule fired |
event.tls.sni |
Requested hostname | What the workload was actually trying to reach |
event.src_ip |
Source workload | Which instance/pod originated it |
event.app_proto |
tls / dns / http |
Which protocol layer matched |
The single most useful day-2 artifact is a CloudWatch Logs Insights query that surfaces your top blocked destinations — it tells you which legitimate destinations you forgot to allowlist before users open a ticket:
fields event.tls.sni as sni, event.src_ip as src
| filter event.alert.action = "blocked"
| stats count(*) as hits by sni, src
| sort hits desc
| limit 25
The other queries you will keep saved:
| Question | Filter / stats |
|---|---|
| Top blocked SNIs | filter action="blocked" | stats count() by sni |
| Which source is hitting the wall most | filter action="blocked" | stats count() by src_ip |
| Which SID fires most (candidate to tune) | stats count() by signature_id |
| DNS queries to flagged TLDs | filter app_proto="dns" | stats count() by query |
| Blocked over time (spot a deploy that broke) | filter action="blocked" | stats count() by bin(5m) |
Architecture at a glance
Picture the traffic as a loop that always passes through inspection, never around it. A workload in a spoke VPC — which has no Internet Gateway and no NAT Gateway of its own — makes an outbound HTTPS call. Its subnet route table sends 0.0.0.0/0 to the Transit Gateway. The TGW route table associated with spokes sends that default straight to the inspection VPC attachment. Because the inspection-VPC attachment has appliance mode enabled, the TGW pins this entire flow to one AZ for its lifetime.
Inside the inspection VPC, the packet lands in that AZ’s TGW-attachment subnet, whose route table forces 0.0.0.0/0 into that AZ’s firewall endpoint. The endpoint hands the traffic to the firewall, where the stateless engine forwards it (aws:forward_to_sfe) to the stateful Suricata engine. Suricata reads the TLS SNI from the ClientHello and evaluates the rule set in strict order: pass rules for allowlisted domains, then the explicit drop floor for everything else, with the policy default aws:drop_established as the last line. If the SNI is on the allowlist, the flow passes; the firewall subnet route table sends the now-inspected traffic to that AZ’s NAT Gateway, whose public NAT subnet routes it to the Internet Gateway and out to the destination.
The return path reverses through the same AZ (appliance mode guarantees it): back through the IGW to the NAT Gateway, then — and this is the symmetry-critical hop — the NAT subnet’s route sends the spoke-bound return traffic back into the firewall endpoint, so the same Suricata instance that saw the ClientHello also sees the ServerHello and keeps the flow’s state consistent. From the firewall subnet, the spoke-CIDR route hands the return to the TGW, which delivers it to the origin spoke. Throughout, flow logs stream to S3 (the audit trail) and alert logs stream to CloudWatch Logs (the tuning and incident surface), so every allowed and blocked flow is visible. The whole design has exactly one property that matters above all others: there is no route, anywhere, that lets spoke traffic reach the internet without traversing the firewall endpoint — and appliance mode plus one-endpoint-per-AZ routing guarantees both halves of every flow meet the same engine.
Real-world scenario
Meridian Pay, a fintech platform team, migrated forty workload accounts behind a centralized inspection VPC to satisfy a PCI DSS requirement that all egress be allowlisted by destination. The design was textbook: a shared inspection VPC in a network account, a Transit Gateway pulling every spoke’s default route into it, a firewall policy in STRICT_ORDER with a TLS SNI allowlist of about 120 domains and an explicit drop floor, aws:drop_established as the default, flow logs to S3 and alert logs to CloudWatch. The team is six engineers; the shared firewall runs endpoints in three AZs at roughly ₹0.032 per endpoint-hour each plus per-GB processing, landing around ₹95,000/month all-in including TGW data processing.
The rollout passed every test in staging and then broke intermittently in production. Roughly one connection in three to *.amazonaws.com would hang — specifically the SSM and S3 gateway paths that their deployment tooling hammered — while curl tests from the same host usually passed on the first or second try. A classic heisenbug: reproducible in aggregate, unreproducible on demand. The on-call engineer’s first instinct was a rule problem, so they widened the allowlist to .amazonaws.com with no dot anchoring — no change. Then they suspected the NAT Gateway and pre-emptively raised its capacity — no change. Two hours in, the incident bridge was full and deployments to production were failing at random.
The constraint was multi-AZ, and staging had hidden it. Staging ran a single AZ; production ran three. Their TGW attachment for the inspection VPC did not have appliance mode enabled, so the TGW was load-balancing forward and return packets across AZs independently. A flow’s ClientHello would enter the AZ-a firewall endpoint, the Suricata engine in AZ-a would establish state and pass the SNI, but the return traffic hashed to the AZ-c endpoint — whose engine had never seen the handshake and dropped the packets as out-of-stream under the default DROP stream-exception policy. The “one in three” was literally the three-AZ hash. curl sometimes “worked” because a retry occasionally landed forward and return on the same AZ by chance.
The breakthrough came from reading the alert log, not the rules. The blocked events showed event.alert.action = "blocked" with the default-drop signature on the return packets of flows whose forward packets had been allowed — an impossible state unless the two halves hit different engines. That is the fingerprint of asymmetry. The fix was one API call plus a routing audit:
aws ec2 modify-transit-gateway-vpc-attachment \
--transit-gateway-attachment-id tgw-attach-0fin999 \
--options ApplianceModeSupport=enable
After enabling appliance mode, they re-audited every inspection-VPC route table to guarantee each AZ’s TGW-attachment and NAT subnets pointed only at that AZ’s firewall endpoint — the second half of the symmetry guarantee, which they found had one copy-paste error (AZ-b’s NAT subnet routed to AZ-a’s endpoint). Flow drops went to zero. Deployment success returned to 100%. The lesson they wrote into their landing-zone Terraform module: appliance mode is not optional for stateful inspection behind a multi-AZ TGW, and it must be asserted by the IaC, not clicked once and forgotten. They added an AWS Config rule to alarm if any inspection-VPC TGW attachment ever reports ApplianceModeSupport != enable, and a synthetic canary in every AZ that opens a connection to an allowlisted host every minute and pages if the success rate across AZs diverges. The total remediation was under thirty minutes of change once the cause was understood; finding the cause had taken two hours because staging never exercised the multi-AZ path.
The incident as a timeline, because the order of moves is the lesson:
| Time | Symptom | Action taken | Effect | What it should have been |
|---|---|---|---|---|
| 09:10 | ~1/3 of *.amazonaws.com calls hang |
(alert fires) | — | Ask: forward allowed but return blocked? |
| 09:20 | Still hanging | Widen SNI allowlist | No change | Not a rule problem — reads passed |
| 09:35 | Still hanging | Raise NAT Gateway capacity | No change | Not a NAT problem |
| 10:30 | Root-cause hunt | Read the alert log carefully | Return packets blocked on flows whose forward was allowed | This was the breakthrough |
| 10:45 | Cause identified | Check ApplianceModeSupport (was disabled) |
Confirmed asymmetry | — |
| 11:00 | Mitigated | Enable appliance mode + audit routes | Drops → 0; found one bad route | Correct fix |
| +1 day | Prevented | Config rule + per-AZ canary added | Regression-proofed | IaC asserts it, not a human |
Advantages and disadvantages
The managed-Suricata-endpoint model both solves the egress problem elegantly and imposes real constraints. Weigh it honestly:
| Advantages | Disadvantages |
|---|---|
| Managed, horizontally scaled Suricata — no NVA fleet to patch or size | You still own all the routing; the firewall inspects nothing without correct routes |
| SNI/domain allowlisting at L7 without decrypting TLS or holding keys | SNI is not payload — ECH, domain fronting, and raw-IP egress can evade it |
| One centralized policy scales to hundreds of accounts via TGW | A bad rule or a routing error affects every spoke at once |
| Managed threat-signature groups update automatically from AWS threat intel | Enforcing them day one floods false positives and severs legit traffic |
| Strict order gives deterministic default-deny allowlisting | Misordered rules silently shadow each other; the model is unforgiving |
| Flow + alert logging captures every allowed and blocked flow | Log volume (especially flow) is real cost; you must design retention |
| Native AWS resource — Terraform/CloudFormation, IAM, tagging, Config | Per-endpoint-hour × AZ × data-processing cost adds up; not cheap |
| Appliance mode makes multi-AZ stateful inspection possible | Forget appliance mode and multi-AZ silently breaks — the top failure |
| Capacity model is predictable once understood | Capacity is immutable per group and capped at 30,000 — plan up front |
The model is right for platform teams running many accounts under an egress-control mandate, where a shared, IaC-managed, auto-updating firewall beats running NVAs. It bites hardest on teams who deploy it without internalizing the routing and appliance-mode requirements (intermittent multi-AZ breakage), teams who enforce managed signatures without a tuning phase (false-positive outages), and cost-sensitive workloads that underestimate endpoint-hour and flow-log costs. Every disadvantage is manageable — but only if you know it exists before it becomes an incident, which is the point of this article.
Hands-on lab
Build a minimal Network Firewall in one VPC (distributed model, single AZ) so the routing is simple, author a TLS SNI allowlist with a default-drop floor, and prove that an allowlisted host passes while a non-allowlisted one is dropped and logged. This deliberately avoids the TGW so you can focus on the firewall and rule mechanics; the centralized routing is covered above and in the Terraform section. Run in a scratch account or an isolated VPC — the firewall endpoint bills hourly, so tear it down after.
Step 1 — Variables and a VPC with a firewall subnet, a protected (workload) subnet, and a public NAT/IGW subnet in one AZ.
REGION=us-east-1
VPC_CIDR=10.99.0.0/16
# Assume you have (or create) these; capture their IDs:
VPC_ID=vpc-0labdemo
AZ=us-east-1a
FW_SUBNET=subnet-0fw # firewall endpoint subnet
WL_SUBNET=subnet-0wl # workload subnet (has an instance via SSM)
PUB_SUBNET=subnet-0pub # public subnet with NAT GW + route to IGW
NATGW_ID=nat-0demo
IGW_ID=igw-0demo
Step 2 — Create a stateful allowlist rule group (SNI allow + drop floor) in strict order. Write the rule group file:
cat > allowlist.json <<'JSON'
{
"RuleVariables": { "IPSets": { "HOME_NET": { "Definition": ["10.99.0.0/16"] } } },
"RulesSource": {
"RulesString": "pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; dotprefix; content:\".amazonaws.com\"; endswith; flow:to_server; msg:\"ALLOW AWS\"; sid:1000001; rev:1;)\npass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; content:\"pypi.org\"; nocase; flow:to_server; msg:\"ALLOW pypi\"; sid:1000002; rev:1;)\ndrop tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; flow:to_server; msg:\"DENY non-allowlisted SNI\"; sid:1000099; rev:1;)"
}
}
JSON
aws network-firewall create-rule-group \
--rule-group-name lab-egress-allowlist \
--type STATEFUL \
--capacity 100 \
--rule-group file://allowlist.json \
--region $REGION
Expected: JSON with a RuleGroupResponse and RuleGroupArn. Capture the ARN as RG_ARN.
Step 3 — Create a firewall policy that forwards everything to Suricata, in strict order, with a default drop+alert.
cat > policy.json <<JSON
{
"StatelessDefaultActions": ["aws:forward_to_sfe"],
"StatelessFragmentDefaultActions": ["aws:forward_to_sfe"],
"StatefulEngineOptions": { "RuleOrder": "STRICT_ORDER", "StreamExceptionPolicy": "DROP" },
"StatefulDefaultActions": ["aws:drop_established", "aws:alert_established"],
"StatefulRuleGroupReferences": [ { "ResourceArn": "$RG_ARN", "Priority": 100 } ]
}
JSON
aws network-firewall create-firewall-policy \
--firewall-policy-name lab-egress-policy \
--firewall-policy file://policy.json \
--region $REGION
Expected: a FirewallPolicyResponse with a FirewallPolicyArn.
Step 4 — Create the firewall in the firewall subnet.
aws network-firewall create-firewall \
--firewall-name lab-egress-fw \
--firewall-policy-arn "$(aws network-firewall describe-firewall-policy \
--firewall-policy-name lab-egress-policy --query 'FirewallPolicyResponse.FirewallPolicyArn' --output text)" \
--vpc-id $VPC_ID \
--subnet-mappings SubnetId=$FW_SUBNET \
--region $REGION
Wait for it to become ready and grab the endpoint ID:
aws network-firewall describe-firewall --firewall-name lab-egress-fw \
--query 'FirewallStatus.SyncStates' --region $REGION
# When ready, capture the endpoint:
FW_ENDPOINT=$(aws network-firewall describe-firewall --firewall-name lab-egress-fw \
--query 'FirewallStatus.SyncStates.*.Attachment.EndpointId' --output text --region $REGION)
echo "Firewall endpoint: $FW_ENDPOINT"
Step 5 — Route the workload subnet through the firewall, and the firewall subnet out via NAT. The workload’s default route goes to the firewall endpoint; the firewall subnet’s default route goes to the NAT Gateway; the public subnet already routes to the IGW; and the return from the public subnet goes back through the firewall endpoint.
# Workload subnet -> firewall endpoint
aws ec2 create-route --route-table-id $(aws ec2 describe-route-tables \
--filters Name=association.subnet-id,Values=$WL_SUBNET --query 'RouteTables[0].RouteTableId' --output text) \
--destination-cidr-block 0.0.0.0/0 --vpc-endpoint-id $FW_ENDPOINT
# Firewall subnet -> NAT Gateway
aws ec2 create-route --route-table-id $(aws ec2 describe-route-tables \
--filters Name=association.subnet-id,Values=$FW_SUBNET --query 'RouteTables[0].RouteTableId' --output text) \
--destination-cidr-block 0.0.0.0/0 --nat-gateway-id $NATGW_ID
# Public/NAT subnet return path for the workload CIDR -> firewall endpoint (symmetry)
aws ec2 create-route --route-table-id $(aws ec2 describe-route-tables \
--filters Name=association.subnet-id,Values=$PUB_SUBNET --query 'RouteTables[0].RouteTableId' --output text) \
--destination-cidr-block 10.99.0.0/16 --vpc-endpoint-id $FW_ENDPOINT
Step 6 — Enable alert logging to CloudWatch.
aws logs create-log-group --log-group-name /anfw/lab-alert --region $REGION
cat > logging.json <<'JSON'
{ "LoggingConfiguration": { "LogDestinationConfigs": [
{ "LogType": "ALERT", "LogDestinationType": "CloudWatchLogs", "LogDestination": { "logGroup": "/anfw/lab-alert" } }
] } }
JSON
aws network-firewall update-logging-configuration \
--firewall-name lab-egress-fw --logging-configuration file://logging.json --region $REGION
Step 7 — Test from the workload instance (via SSM Session Manager — there is no SSH path if the design is right).
# Allowlisted host succeeds:
curl -sS -o /dev/null -w '%{http_code}\n' https://pypi.org
# Expect: 200
# Non-allowlisted host is dropped (connection hangs, then times out):
curl -sS --max-time 8 https://example.com ; echo "exit=$?"
# Expect: exit=28 (operation timed out) — the drop is silent at L4
Step 8 — Confirm the firewall saw and blocked it.
aws logs filter-log-events --log-group-name /anfw/lab-alert \
--filter-pattern '{ $.event.tls.sni = "example.com" }' \
--query 'events[].message' --region $REGION
# Expect: an alert record with action "blocked" and signature_id 1000099
Validation checklist. You created a stateful allowlist with an explicit drop floor, a policy that forwards to Suricata in strict order with a drop-and-alert default, a firewall in a subnet, routes that force workload traffic through the endpoint and return symmetrically, and alert logging. pypi.org (allowlisted) returned 200; example.com (not allowlisted) timed out and appeared in the alert log as blocked by SID 1000099. What each step proved:
| Step | What you did | What it proves |
|---|---|---|
| 2 | Allowlist rule group with drop floor | Suricata SNI allow/deny authored correctly |
| 3 | Policy: forward-to-SFE + strict order + drop default | The stateless→stateful path and default-deny |
| 5 | Route workload → endpoint → NAT; return → endpoint | Routing forces and returns traffic through the firewall |
| 7 | curl allowlisted vs not | The allow passes, the deny drops (silent timeout) |
| 8 | Filter the alert log | The drop is logged with SNI and SID — auditable |
Teardown (endpoints bill hourly — do this).
aws network-firewall delete-firewall --firewall-name lab-egress-fw --region $REGION
aws network-firewall delete-firewall-policy --firewall-policy-name lab-egress-policy --region $REGION
aws network-firewall delete-rule-group --rule-group-name lab-egress-allowlist --type STATEFUL --region $REGION
aws logs delete-log-group --log-group-name /anfw/lab-alert --region $REGION
# Remove the routes you added if the subnets/VPC persist.
Cost note. A single firewall endpoint is roughly ₹2–3 per hour plus per-GB processing; an hour of this lab is well under ₹50. Deleting the firewall stops the endpoint charge immediately.
The Terraform for the centralized model
The lab used the CLI in one VPC. Production is the centralized model as code. Here is the firewall, policy, and rule group in Terraform — the routing and TGW resources are extensive, so this focuses on the firewall-specific resources you will reuse across a landing zone, with the appliance-mode assertion called out.
# --- Rule variables + stateful allowlist rule group (strict-order friendly) ---
resource "aws_networkfirewall_rule_group" "egress_allowlist" {
name = "egress-tls-allowlist"
type = "STATEFUL"
capacity = 200
rule_group {
rule_variables {
ip_sets {
key = "HOME_NET"
ip_set { definition = ["10.0.0.0/8", "172.16.0.0/12"] }
}
}
rules_source {
rules_string = <<-EOT
pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; dotprefix; content:".amazonaws.com"; endswith; flow:to_server; msg:"ALLOW AWS"; sid:1000001; rev:1;)
pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; content:"registry.npmjs.org"; nocase; flow:to_server; msg:"ALLOW npm"; sid:1000002; rev:1;)
pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; dotprefix; content:".pypi.org"; endswith; flow:to_server; msg:"ALLOW pypi"; sid:1000003; rev:1;)
drop tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; flow:to_server; msg:"DENY non-allowlisted SNI"; sid:1000099; rev:1;)
EOT
}
}
tags = { Purpose = "egress-inspection" }
}
# --- Firewall policy: forward everything to Suricata, strict order, default drop+alert ---
resource "aws_networkfirewall_firewall_policy" "egress" {
name = "egress-inspection-policy"
firewall_policy {
stateless_default_actions = ["aws:forward_to_sfe"]
stateless_fragment_default_actions = ["aws:forward_to_sfe"]
stateful_engine_options {
rule_order = "STRICT_ORDER"
stream_exception_policy = "DROP"
}
stateful_default_actions = ["aws:drop_established", "aws:alert_established"]
stateful_rule_group_reference {
resource_arn = aws_networkfirewall_rule_group.egress_allowlist.arn
priority = 100
}
# A managed threat-signature group, added in DROP_TO_ALERT for the tuning phase.
stateful_rule_group_reference {
resource_arn = "arn:aws:network-firewall:us-east-1:aws-managed:stateful-rulegroup/ThreatSignaturesBotnet"
priority = 200
override { action = "DROP_TO_ALERT" }
}
}
}
# --- The firewall, spanning every AZ's firewall subnet ---
resource "aws_networkfirewall_firewall" "egress" {
name = "acme-egress-inspection"
firewall_policy_arn = aws_networkfirewall_firewall_policy.egress.arn
vpc_id = var.inspection_vpc_id
dynamic "subnet_mapping" {
for_each = var.firewall_subnet_ids # one per AZ
content { subnet_id = subnet_mapping.value }
}
tags = { Purpose = "egress-inspection" }
}
# --- Logging: flow to S3, alert to CloudWatch ---
resource "aws_networkfirewall_logging_configuration" "egress" {
firewall_arn = aws_networkfirewall_firewall.egress.arn
logging_configuration {
log_destination_config {
log_type = "FLOW"
log_destination_type = "S3"
log_destination = { bucketName = var.log_bucket, prefix = "flow" }
}
log_destination_config {
log_type = "ALERT"
log_destination_type = "CloudWatchLogs"
log_destination = { logGroup = "/anfw/alert" }
}
}
}
# --- Appliance mode on the inspection VPC's TGW attachment: NOT OPTIONAL for multi-AZ stateful ---
resource "aws_ec2_transit_gateway_vpc_attachment" "inspection" {
transit_gateway_id = var.tgw_id
vpc_id = var.inspection_vpc_id
subnet_ids = var.tgw_attachment_subnet_ids # one per AZ
appliance_mode_support = "enable" # <-- the line that prevents the multi-AZ heisenbug
tags = { Name = "inspection-vpc-attachment" }
}
The one line to never omit is appliance_mode_support = "enable" on the inspection VPC’s TGW attachment. Assert it in code, and add an AWS Config rule to catch any drift. The route resources (workload subnet → endpoint, firewall subnet → NAT, NAT subnet → endpoint for return) follow the tables in the routing section; because the firewall endpoint IDs are only known after the firewall is created, you reference them from aws_networkfirewall_firewall.egress.firewall_status[0].sync_states per AZ when writing the aws_route resources.
Common mistakes & troubleshooting
The failures that actually happen, as a scannable table first, then the detail on the ones that bite hardest.
| # | Symptom | Root cause | Confirm (exact command / check) | Fix |
|---|---|---|---|---|
| 1 | ~1/N of connections hang (N = AZ count), heisenbug | Appliance mode off; forward/return split across AZs | aws ec2 describe-transit-gateway-vpc-attachments --query '...Options.ApplianceModeSupport' ≠ enable |
Enable appliance mode |
| 2 | Intermittent hangs even with appliance mode on | An AZ’s subnet routes to another AZ’s endpoint | Audit each route table’s vpc-endpoint-id against its AZ |
One endpoint per AZ, referenced only locally |
| 3 | Non-allowlisted host is refused fast, not timed out; Suricata rules never fire | Stateless default is aws:drop/aws:pass, not forward_to_sfe |
Check StatelessDefaultActions in the policy |
Set ["aws:forward_to_sfe"] |
| 4 | A pass rule you added is ignored; a broader drop wins (or vice versa) |
Rule order is DEFAULT_ACTION_ORDER, precedence not position |
Check StatefulEngineOptions.RuleOrder |
Set STRICT_ORDER and order rules deliberately |
| 5 | Allowlist bypassed: evilamazonaws.com was allowed |
Suffix match without dotprefix (no label boundary) |
Inspect the rule for dotprefix; ... endswith |
Add dotprefix before content; anchor endswith |
| 6 | Rule group won’t apply: capacity exceeded | Cumulative stateful capacity > 30,000 or group over budget | Sum reserved capacities; check the create error | Reduce/replace a group; capacity is immutable |
| 7 | Everything is blocked after cutover, including known-good | Default action is aws:drop_* with an incomplete allowlist |
Read the alert log for top blocked SNIs | Start with alert-only default; complete the allowlist |
| 8 | Managed signatures severed legit traffic on day one | Enforced (drop) without a tuning phase | Alert log shows managed SIDs blocking good flows | Use DROP_TO_ALERT override, tune, then enforce |
| 9 | Traffic reaches the internet uninspected | TGW-attachment subnet has no 0.0.0.0/0 → endpoint route |
Check rtb-tgw-az-* routes |
Add the force-into-firewall route |
| 10 | A spoke has working direct egress (bypass) | Spoke still has an IGW/NAT + 0.0.0.0/0 to it |
describe-route-tables on the spoke |
Remove the local internet route; default to TGW |
| 11 | $HOME_NET-keyed rules match nothing |
HOME_NET empty/wrong in that rule group |
Inspect RuleVariables.IPSets.HOME_NET |
Define HOME_NET per rule group with spoke CIDRs |
| 12 | DNS filtering misses queries | Rule scoped -> $EXTERNAL_NET but DNS goes to the .2 resolver |
Check the rule’s destination | Use -> any any for dns.query rules |
| 13 | Silent drops nobody can explain | Logging not enabled, or only flow (not alert) | describe-logging-configuration |
Enable ALERT logs; save the top-blocked-SNI query |
| 14 | Fragmented traffic behaves oddly | StatelessFragmentDefaultActions not set to forward |
Check the fragment default | Set fragments to aws:forward_to_sfe |
1 & 2 — Asymmetric routing behind a multi-AZ TGW (the top failure). Symptom: roughly one connection in N hangs where N is your AZ count; individual curl tests pass often enough to mislead; deployments that make many connections fail at random. Confirm: aws ec2 describe-transit-gateway-vpc-attachments --transit-gateway-attachment-ids <id> --query 'TransitGatewayVpcAttachments[0].Options.ApplianceModeSupport' — if it is not "enable", that is your bug. Then read the alert log for the tell-tale: a return packet blocked by the default-drop on a flow whose forward packet was allowed. Fix: modify-transit-gateway-vpc-attachment ... --options ApplianceModeSupport=enable, then audit every inspection-VPC route table so each AZ’s subnets reference only that AZ’s firewall endpoint.
3 — Fast refusal instead of a timeout means Suricata never ran. Symptom: a non-allowlisted host is refused immediately (connection refused / RST) rather than hanging and timing out. Root cause: the stateless default action is aws:drop (or aws:pass), so the stateless engine is deciding the verdict and never forwarding to Suricata — your entire SNI rule set is dead code. Confirm: check StatelessDefaultActions in the policy; it must be ["aws:forward_to_sfe"]. Fix: set it. The signature of “silent timeout” vs “fast refusal” is your quickest tell for whether the stateful engine is even in the path.
4 & 5 — Rule order and suffix-match bypass. Symptom (4): a pass you added does not take effect, or a drop you expect to win is overridden. Root cause: the policy is in DEFAULT_ACTION_ORDER, where Suricata precedence (pass beats drop) applies regardless of your intended sequence. Fix: switch to STRICT_ORDER and order rule groups by priority and rules top-to-bottom, with the drop floor last. Symptom (5): an allowlist for .amazonaws.com also allowed evilamazonaws.com. Root cause: endswith without dotprefix matches the raw suffix, ignoring label boundaries. Fix: always pair dotprefix with the content and endswith for suffix domain matches.
6 — Capacity is immutable and capped. Symptom: a rule group or policy update fails with a capacity error. Root cause: either a single group’s reserved capacity is too small for its rules, or the cumulative stateful capacity across all referenced groups exceeds 30,000. Confirm: sum the reserved capacities; read the exact create/update error. Fix: because per-group capacity cannot be changed, you must replace the group with a larger reservation (create new, swap the policy reference, delete old), or consolidate/trim groups to fit under 30,000.
7 & 8 — Cutover and managed-signature false positives. Symptom (7): after enabling the drop default, legitimate traffic is blocked wholesale. Root cause: the allowlist is incomplete and the default is aws:drop_established. Fix: deploy with aws:alert_established only first, read the alert log’s top blocked SNIs, complete the allowlist, then add the drop. Symptom (8): a managed threat-signature group severed good traffic on day one. Fix: add managed groups with a DROP_TO_ALERT override, run a week, tune out the specific noisy SIDs with targeted passes, then remove the override to enforce.
Best practices
- Centralize behind a Transit Gateway for any non-trivial estate. One inspection VPC, one policy, one log pipeline. Spokes get no IGW/NAT — the firewall is the only egress path, by construction.
- Assert appliance mode in IaC and guard it with Config.
appliance_mode_support = "enable"on the inspection VPC’s TGW attachment is mandatory for multi-AZ stateful inspection. A human-clicked setting drifts; a Config rule that alarms on!= enabledoes not. - One firewall endpoint per AZ, referenced only by that AZ’s subnets. Never cross-AZ steer. It defeats appliance mode’s symmetry and creates a latent failure that surfaces on the next AZ event.
- Set the stateless default to
aws:forward_to_sfe(and the fragment default too). Otherwise Suricata never sees the traffic and your L7 rules are dead code. This is the most common “my rules don’t fire” cause. - Use
STRICT_ORDERwith an explicitaws:drop_establisheddefault for allowlisting. Action order lets a straypassopen a hole; strict order makes your drop floor authoritative and your ordering deliberate. - Anchor suffix domain matches with
dotprefix+endswith. It is the difference between allowing.amazonaws.comand accidentally allowingevilamazonaws.com. - Cut over in alert-only mode, then enforce. Deploy the default and managed groups as
alert/DROP_TO_ALERT, mine the alert log for missed allowlist entries and noisy SIDs, then flip to drop. Never enforce a fresh allowlist blind. - Never bulk-disable a managed group to silence one signature. Suppress the specific SID for the specific source, with a comment recording the ticket and date. A blanket disable is how a real intrusion walks through.
- Reserve rule-group capacity with headroom and track the running total against 30,000. Capacity is immutable per group; size 2–3× current need and document the cumulative budget in code.
- Enable both flow (S3) and alert (CloudWatch) logging, and save the top-blocked-SNI query. A silent firewall is unoperable. The “top blocked SNI by source” query is your day-2 workhorse for keeping the allowlist honest.
- Layer SNI filtering with DNS Firewall, reputation groups, and flow analytics. SNI is one control; ECH, domain fronting, and raw-IP egress need complementary layers.
- Add per-AZ synthetic canaries. A minute-by-minute connection to an allowlisted host from each AZ catches routing/appliance-mode regressions before users do.
Security notes
- Least privilege on the firewall control plane. Restrict
network-firewall:*write actions (create/update/delete rule groups, policies, firewalls, and logging config) to the network-security role only. A widenetwork-firewall:UpdateFirewallPolicygrant lets someone remove your drop floor; treat rule-group and policy edits as high-privilege, PR-reviewed changes. - Protect the log destinations. The flow/alert logs contain destination hostnames and internal source IPs — a reconnaissance goldmine. Lock the S3 bucket (block public access, bucket policy scoped to the firewall’s log-delivery principal, SSE-KMS) and the CloudWatch log group (KMS-encrypted, restricted read).
- Default-deny is the security posture; make it real.
aws:drop_established(oraws:drop_strict) as the policy default, with an explicit allowlist above it, is the control auditors want to see. An allowlist without a drop default is theatre. - Do not let spokes keep an escape hatch. Any spoke with its own IGW/NAT and a
0.0.0.0/0to it bypasses the firewall entirely. Enforce “no local internet route in spokes” with an SCP or Config rule, not just documentation. - Treat SNI allowlisting as necessary-not-sufficient. Combine it with Route 53 DNS Firewall (block resolution to bad domains), managed IP/domain reputation groups, and stateless CIDR rules for raw-IP egress classes. Assume a determined adversary can attempt ECH or fronting.
- Log integrity for forensics. Ship logs to an account the workload teams cannot modify (a central logging account), with object-lock / immutability where compliance requires it, so a compromised workload cannot erase evidence of its own egress.
- Rotate and review the allowlist as code. Every allowlisted domain is an attack surface; review the list in PRs, and prefer exact hosts over broad
.tldwildcards where practical.
Cost & sizing
The bill has three drivers, and endpoint-hours dominate for small estates while data processing dominates at scale:
- Firewall endpoint-hours — you pay per firewall endpoint per hour, and you run one endpoint per AZ. Three AZs means three endpoints billed continuously whether or not traffic flows. This is the fixed floor and the reason the centralized model wins: you pay it once for the whole estate, not per spoke VPC.
- Data processing — a per-GB charge on traffic the firewall inspects. At scale this overtakes endpoint-hours. It is the variable driver, and it is why you do not route traffic through the firewall that does not need inspection (e.g. keep S3/DynamoDB access on gateway VPC endpoints, which bypass the firewall path entirely and cut both data-processing and NAT cost).
- Transit Gateway data processing — the centralized model adds a per-GB TGW charge on top, since traffic transits the TGW to reach the inspection VPC.
- Logging — flow logs are high-volume; per-GB ingestion/storage in CloudWatch is pricier than S3. Send flow logs to S3 (cheap, Athena-queryable) and only the lower-volume alert logs to CloudWatch for fast interactive queries.
The cost drivers, what each buys, and how to control it:
| Cost driver | You pay for | Rough shape | Control it by |
|---|---|---|---|
| Firewall endpoint-hours | Per endpoint × per AZ, continuous | Fixed monthly floor | Centralize (pay once); right-size AZ count |
| Firewall data processing | Per-GB inspected | Variable, scales with egress | Keep S3/DDB on gateway endpoints (bypass firewall) |
| TGW data processing | Per-GB through the TGW | Variable | Same — do not send non-inspected traffic via TGW |
| NAT Gateway | Hourly + per-GB | Variable | Bypass NAT for AWS-service traffic via VPC endpoints |
| Flow logs | Per-GB ingest/storage | High volume | Send to S3, not CloudWatch; lifecycle to Glacier |
| Alert logs | Per-GB ingest/storage | Lower volume | CloudWatch for interactive; retention policy |
Rough sizing for a mid-size platform: three firewall endpoints run on the order of ₹18,000–25,000/month in endpoint-hours alone (region-dependent), plus data-processing that can add ₹10,000–40,000+/month depending on egress volume, plus TGW and NAT charges, plus logging. Meridian Pay’s ~₹95,000/month all-in for forty accounts is a realistic figure — and it is shared across those accounts, which is the economic argument for centralization: the per-account cost of egress inspection is a fraction of what forty distributed firewalls would cost. The single biggest cost lever is not sizing the firewall — it is not routing traffic through it that does not need inspection, chiefly by keeping AWS-service traffic (S3, DynamoDB) on gateway VPC endpoints that bypass the whole path.
Interview & exam questions
1. Why can’t a security group do what Network Firewall does for egress? A security group operates at L3/L4 on the 5-tuple — it can allow or deny 0.0.0.0/0:443 but cannot distinguish which domain a TLS connection is destined for, because that is application-layer information. Network Firewall’s stateful engine (Suricata) reads the TLS SNI from the ClientHello and can allowlist by hostname without decrypting. SGs gate by IP/port; Network Firewall gates by domain, DNS query, and signature.
2. Explain the packet path through a Network Firewall policy. Every packet hits the stateless engine first (per-packet, 5-tuple). It can aws:pass (accept, skip stateful), aws:drop (reject, skip stateful), or aws:forward_to_sfe (send to the stateful engine). Only forwarded traffic reaches the stateful Suricata engine, which tracks flow state and does L7 (SNI/DNS/HTTP) matching and IDS. For inspection you set the stateless default to aws:forward_to_sfe so everything reaches Suricata.
3. What is appliance mode and why is it required for stateful inspection behind a multi-AZ TGW? Stateful inspection needs both directions of a flow to hit the same firewall endpoint so the engine sees the whole connection. A multi-AZ Transit Gateway, by default, hashes forward and return packets independently and can split them across AZ endpoints — the return-side engine never saw the handshake and drops the packets as out-of-state. Appliance mode pins all packets of a flow to one AZ for its lifetime, restoring symmetry. It is not optional for multi-AZ stateful firewalls.
4. Distributed vs centralized deployment — when do you pick each? Distributed puts a firewall per VPC (per-AZ endpoints in each), simple and blast-radius-limited but multiplying fixed endpoint cost and rule-sync effort with every VPC — good for a small, fixed number of VPCs or a hard isolation boundary. Centralized puts one firewall in a dedicated inspection VPC behind a TGW, inspecting all spokes with one policy and one log pipeline — the model for landing zones and many accounts. Centralization trades a shared blast radius and TGW data-processing cost for far lower per-account cost and single-source-of-truth rules.
5. Why does an allowlist firewall need STRICT_ORDER rather than the default? The default DEFAULT_ACTION_ORDER uses Suricata’s native precedence where pass beats drop regardless of rule position — so a stray or overly broad pass silently overrides your default-deny. STRICT_ORDER evaluates rule groups by priority and rules top-to-bottom with first-match-wins, letting you place explicit passes above an explicit drop floor deterministically. Only strict order makes “allow this list, deny everything else” reliable.
6. How do you match a domain and all its subdomains in a Suricata SNI rule without a bypass? Use the tls.sni sticky buffer with dotprefix (prepends a dot so matches align on label boundaries) plus content:".example.com"; endswith. The dotprefix is essential: endswith:"example.com" alone would also match evilexample.com; with dotprefix it matches only example.com and *.example.com.
7. What is the stateful capacity limit and why does it constrain design? A firewall policy caps total stateful capacity at 30,000 units across all its stateful rule groups, and each rule group’s capacity is reserved at creation and is immutable. Suricata rules cost 1 unit; domain-list entries cost ~1 per target type. You size each group with headroom, cannot resize it later (you replace it), and must track the cumulative total so a new group is not rejected at apply time.
8. A non-allowlisted host is refused instantly instead of timing out, and your SNI rules aren’t firing. What’s wrong? The stateless default action is aws:drop (or aws:pass) rather than aws:forward_to_sfe, so the stateless engine is deciding the verdict and Suricata never runs — your entire stateful rule set is dead code. A silent timeout means the stateful engine dropped it; a fast refusal means the stateless engine did. Fix the stateless default to ["aws:forward_to_sfe"].
9. How do you safely roll out AWS-managed threat-signature groups? Add them to the policy with a DROP_TO_ALERT override so they alert but do not block. Run for about a week, mine the alert log for SIDs firing on known-good traffic, and add targeted pass rules above the group (strict order) to whitelist those exact flows — never disable the whole group. Once the alert stream is clean, remove the override so the group enforces.
10. Does SNI filtering give you full egress control? What are its limits? No — it reads the requested hostname from the plaintext ClientHello but sees nothing of the payload and can be evaded by Encrypted Client Hello (SNI encrypted), domain fronting (allowed SNI, different real host), and raw-IP egress (no SNI or DNS). It stops the overwhelming majority of misconfigured and commodity-malware egress, so it is a strong primary control, but it must be layered with DNS Firewall, IP/domain reputation groups, and flow analytics.
11. Which two log types does Network Firewall emit and where would you send each? Flow logs (connection-level 5-tuple/bytes/packets for every tracked flow — high volume, the audit trail) and alert logs (records for traffic matching an alert/drop rule, with the SID and L7 fields like SNI — lower volume). Send flow logs to S3 (cheap, Athena-queryable, long retention) and alert logs to CloudWatch Logs for fast interactive queries during incidents and tuning.
12. What is the single biggest lever on Network Firewall cost? Not sizing the firewall — it is not routing traffic through it that does not need inspection. Keep AWS-service traffic (S3, DynamoDB) on gateway VPC endpoints that bypass the firewall and NAT path entirely, cutting both firewall data-processing and NAT/TGW charges. Endpoint-hours are a fixed floor; data processing is the variable driver you control by what you route.
These map to AWS Certified Advanced Networking – Specialty (ANS-C01) — network security, inspection architectures, Transit Gateway routing, appliance mode — and to AWS Certified Security – Specialty (SCS-C02) — infrastructure protection, egress control, logging and monitoring. A compact cert-mapping for revision:
| Question theme | Primary cert | Objective area |
|---|---|---|
| Inspection architecture, distributed vs centralized | ANS-C01 | Design & implement network security |
| Appliance mode, TGW routing, symmetry | ANS-C01 | Hybrid/transit connectivity; troubleshooting |
| SNI/domain allowlisting, Suricata rules | SCS-C02 | Infrastructure protection |
| Rule order, default actions, capacity | ANS-C01 / SCS-C02 | Network security controls |
| Managed threat signatures, tuning | SCS-C02 | Threat detection & response |
| Flow/alert logging, EVE JSON | SCS-C02 | Logging & monitoring |
Quick check
- A workload behind a three-AZ inspection VPC has ~1/3 of its connections hang, but a single
curlusually succeeds. What is the most likely cause and the one setting you check first? - Your
tls.sniallow rules never take effect and a blocked host is refused instantly rather than timing out. Which policy setting is wrong? - You wrote
pass tls ... (tls.sni; content:".amazonaws.com"; endswith; ...)andevilamazonaws.comgot through. What is missing and why? - Why can’t you enlarge a rule group’s capacity in place, and what is the policy-wide stateful ceiling?
- You are cutting over a fresh allowlist to default-deny. What posture do you deploy first, and what log do you read before flipping to drop?
Answers
- Appliance mode is off on the inspection VPC’s TGW attachment, so the TGW splits forward and return packets across AZ endpoints and the return-side Suricata engine drops out-of-state packets — the “1/3” is the three-AZ hash. First check:
aws ec2 describe-transit-gateway-vpc-attachments ... --query '...Options.ApplianceModeSupport'and set it toenable, then audit that each AZ’s subnets reference only that AZ’s endpoint. - The stateless default action is
aws:drop(oraws:pass) instead of["aws:forward_to_sfe"], so the stateless engine decides the verdict and Suricata never runs. A fast refusal means the stateless engine dropped it; a silent timeout would mean the stateful engine did. SetStatelessDefaultActionsto["aws:forward_to_sfe"]. dotprefixis missing. Without it,endswith:".amazonaws.com"matches the raw suffix andevilamazonaws.comends with.amazonaws.comonly if the dot happens to align — the safe form istls.sni; dotprefix; content:".amazonaws.com"; endswith, which forces a label boundary so onlyamazonaws.comand its subdomains match.- Rule-group capacity is reserved at creation and is immutable — to grow a group you create a new one, repoint the policy reference, and delete the old. The policy-wide stateful capacity ceiling is 30,000 units across all referenced stateful rule groups.
- Deploy with the default action as
aws:alert_established(alert-only, no drop) and managed groups asDROP_TO_ALERT. Read the alert log for the top blocked SNIs (the destinations you forgot to allowlist) and noisy managed SIDs, complete the allowlist and add targeted passes, then flip the default toaws:drop_establishedand remove the overrides.
Glossary
- AWS Network Firewall — a managed, horizontally scaled stateful (Suricata) and stateless network firewall you insert into the VPC data path via per-AZ endpoints.
- Firewall endpoint — the per-AZ VPC endpoint (Gateway Load Balancer style) that traffic is routed to for inspection; the next hop your route tables point at.
- Inspection VPC — a dedicated VPC hosting the firewall (and NAT/IGW) through which all spoke egress is routed in the centralized model.
- Distributed vs centralized — one firewall per VPC vs one shared firewall in an inspection VPC behind a Transit Gateway.
- Stateless engine — the per-packet 5-tuple pre-filter; actions
aws:pass,aws:drop,aws:forward_to_sfe. - Stateful engine (SFE) — the Suricata engine that tracks flow state and does L7 (SNI/DNS/HTTP) matching and IDS; only sees traffic forwarded to it.
aws:forward_to_sfe— the stateless action that hands a packet to the stateful engine; the stateless default for an inspection design.- Suricata — the open-source IDS/IPS engine whose rule syntax Network Firewall’s stateful rules use.
tls.sni— the Suricata keyword that matches the Server Name Indication (requested hostname) from the plaintext TLS ClientHello.dotprefix— a Suricata transform that prepends a dot so suffix matches align on domain-label boundaries (preventsevilexample.combypass).- Domain-list rule group — a managed abstraction that allowlists/denylists by
TLS_SNIand/orHTTP_HOSTwithout raw Suricata. $HOME_NET/$EXTERNAL_NET— rule variables defining internal (your spoke CIDRs) and external address sets; keyed into rule direction.STRICT_ORDER— stateful evaluation by rule-group priority and top-to-bottom, first-match-wins; required for default-deny allowlisting.DEFAULT_ACTION_ORDER— Suricata’s native precedence (pass>drop>reject>alert) regardless of rule position.aws:drop_established/aws:alert_established— stateful default actions that drop/alert on established flows no rule passed.- Capacity — the immutable unit reservation per rule group; policy-wide stateful ceiling is 30,000 units.
- Managed threat-signature rule group — an AWS-maintained, auto-updated Suricata rule group (botnet, malware, scanners, reputation lists).
DROP_TO_ALERToverride — a policy-level override that flips a rule group’s action from drop to alert without editing the group (used for tuning).- Appliance mode — a TGW attachment option that pins a flow to one AZ for its lifetime, guaranteeing symmetric routing for stateful inspection.
- Stream exception policy — what the engine does with packets it can’t associate with a tracked stream (
DROP,CONTINUE,REJECT);DROPis the secure default. - EVE JSON — Suricata’s structured event format; the schema of the
eventobject in Network Firewall alert logs.
Next steps
You can now insert Network Firewall as centralized egress inspection, engineer Suricata SNI/domain rules in the correct order, and route traffic so both directions actually reach the engine. Build outward:
- Next: Centralized Internet Egress: FQDN Filtering, Explicit Proxy, and TLS Inspection — the broader egress-control patterns (explicit proxy, full TLS inspection) that go beyond SNI-only filtering.
- Related: Hub-and-Spoke vs Virtual WAN: Choosing an Enterprise Cloud Network Topology — the topology decision the inspection VPC lives inside.
- Related: Deploying HA Third-Party NVAs in Azure: The Load Balancer Sandwich Pattern — how you insert appliances when you run the NVA yourself instead of a managed firewall.
- Related: Network Flow Logs to Insight: Building a Traffic Analytics and Detection Pipeline — turn the firewall’s flow logs into detection and analytics.
- Related: AWS Control Tower Guardrails: Building a Secure Multi-Account Foundation — the account foundation where a shared inspection VPC gets mandated.
- Related: AWS VPC, Subnets and Security Groups Explained — the L3/L4 controls Network Firewall layers on top of.
- Related: Zero Trust Architecture Blueprint: Identity, Network, and Data Pillars — where egress inspection fits in a zero-trust network posture.