Deploying HA Third-Party NVAs in Azure: The Load Balancer Sandwich Pattern

Running a third-party firewall, IDS, or proxy as a Network Virtual Appliance (NVA) in Azure is straightforward until you need it to survive a single-instance failure without dropping live sessions. This article walks through the load balancer sandwich pattern end to end: why naive NVA HA breaks, how to wrap an NVA pair between an external and an internal Standard Load Balancer, and how to keep stateful inspection happy with symmetric flows.

Why NVA HA Is Hard

A stateless router can be made HA trivially. A firewall cannot, because it tracks connection state. Three independent problems collide:

1. Stateful flow pinning. The forward and return packets of a TCP session must traverse the same NVA instance. If the SYN goes through NVA-A and the SYN/ACK comes back through NVA-B, NVA-B sees a packet with no matching session table entry and silently drops it. This is the asymmetric routing problem, and it is the single most common reason “working” NVA HA setups mysteriously lose connections.

2. Route convergence on failure. The classic pattern points a User Defined Route (UDR) 0.0.0.0/0 next hop at NVA-A’s NIC. When NVA-A dies, something has to rewrite that UDR to NVA-B. Doing that with a custom script (an Azure Function watching health and calling the ARM API) is slow, racy, and brittle — convergence takes tens of seconds and the script itself becomes a failure domain.

3. Single-NIC SNAT confusion. If both NVAs SNAT to their own NIC IPs, return traffic naturally comes back to the originating instance. But the moment you want active-active throughput, or you have non-SNAT (routed/transparent) flows, symmetry is no longer free.

The load balancer sandwich solves all three at the data plane. Failover is handled by health probes in milliseconds-to-seconds, not by an ARM control-plane script. Symmetry is enforced by the load balancer’s 5-tuple (or floating-IP) hashing.

The Load Balancer Sandwich

The pattern places the NVA pair between two Standard Load Balancers:

                 Internet / on-prem
                        |
            [ External Standard LB ]   <- public or peered ingress
                  |          |
              [ NVA-A ]   [ NVA-B ]     <- multi-NIC, IP forwarding ON
                  |          |
            [ Internal Standard LB ]    <- HA Ports frontend
                        |
              UDR 0.0.0.0/0 -> ILB frontend IP
                        |
                   Spoke subnets

• External LB handles ingress from the internet or from peered/on-prem networks toward services behind the firewall (or egress return). It distributes across the NVA “untrusted” NICs.
• Internal LB (ILB) is the workhorse for east-west and egress traffic. Spoke UDRs point their default route at the ILB frontend IP, never at an individual NVA NIC. The ILB uses HA Ports so it load-balances all protocols and all ports in one rule.
• Both LBs must be Standard SKU. Basic LB does not support HA Ports and has no notion of the floating-IP / source-IP-affinity controls you need here.

Why two LBs? Because traffic flows in both directions through the firewall. East-west and egress traffic hits the ILB. Ingress to published services and the return path of egress flows hits the external LB. Each direction needs a load-balanced, health-probed entry point so that the failure of one NVA removes it from both rotations.

The critical insight: with HA Ports plus the same load distribution mode on both LBs, Azure hashes the 5-tuple consistently, so a given flow lands on the same NVA in both directions. That is what gives you flow symmetry without per-instance SNAT.

Step 1 — Deploy the NVA Instances with Multiple NICs and IP Forwarding

Each NVA needs at least two NICs: an untrusted/external NIC and a trusted/internal NIC. (Many vendor images add a third NIC for management.) Two things are non-negotiable:

• Azure IP forwarding must be enabled on every data-plane NIC. This is the Azure fabric setting that lets a NIC receive/send packets whose source/destination IP is not the NIC’s own IP. Without it, the fabric drops forwarded packets.
• IP forwarding must also be enabled inside the guest OS (the firewall product normally does this itself).

Create the NICs with forwarding on:

RG=rg-hub-nva
LOC=eastus
VNET=vnet-hub

# Untrusted (external) NICs
az network nic create -g $RG -n nic-nva-a-untrust \
  --vnet-name $VNET --subnet snet-untrust \
  --ip-forwarding true --accelerated-networking true

az network nic create -g $RG -n nic-nva-b-untrust \
  --vnet-name $VNET --subnet snet-untrust \
  --ip-forwarding true --accelerated-networking true

# Trusted (internal) NICs
az network nic create -g $RG -n nic-nva-a-trust \
  --vnet-name $VNET --subnet snet-trust \
  --ip-forwarding true --accelerated-networking true

az network nic create -g $RG -n nic-nva-b-trust \
  --vnet-name $VNET --subnet snet-trust \
  --ip-forwarding true --accelerated-networking true

Deploy the two appliance VMs. Use a vendor BYOL/PAYG image from the Marketplace (you must accept its terms once with az vm image terms accept). Place both instances in an availability set or, preferably, across availability zones so a single zone or rack failure cannot take out both:

# Example: zonal placement, one VM per zone.
# --image is a placeholder; use your vendor's URN from the Marketplace.
az vm create -g $RG -n nva-a \
  --image <publisher>:<offer>:<sku>:<version> \
  --zone 1 \
  --nics nic-nva-a-untrust nic-nva-a-trust \
  --size Standard_D4s_v5 --admin-username azureuser \
  --generate-ssh-keys

az vm create -g $RG -n nva-b \
  --image <publisher>:<offer>:<sku>:<version> \
  --zone 2 \
  --nics nic-nva-b-untrust nic-nva-b-trust \
  --size Standard_D4s_v5 --admin-username azureuser \
  --generate-ssh-keys

Note on accelerated networking: most modern firewall images support it and you want it for throughput, but confirm against your vendor’s compatibility matrix. Some appliance versions require it off.

Step 2 — Configure the Internal Standard Load Balancer with HA Ports

The ILB is where the magic happens. Create a Standard internal LB, a backend pool containing both trusted NICs, a health probe, and a single HA Ports rule.

ILB=ilb-nva-trust

# 1. Standard internal LB with a frontend in the trusted subnet
az network lb create -g $RG -n $ILB --sku Standard \
  --vnet-name $VNET --subnet snet-trust \
  --frontend-ip-name fe-trust \
  --backend-pool-name bep-nva-trust \
  --private-ip-address 10.0.1.10

# 2. Health probe — probe the firewall's data-plane health endpoint.
#    Use TCP to a port the firewall actually serves, or HTTP to a
#    vendor health URL. Do NOT probe a port that is up while the
#    dataplane is down.
az network lb probe create -g $RG --lb-name $ILB \
  -n probe-nva -p tcp --port 22 \
  --interval 5 --threshold 2

# 3. HA Ports load-balancing rule (protocol All, port 0).
#    floating-ip (Direct Server Return) is REQUIRED for HA Ports.
az network lb rule create -g $RG --lb-name $ILB \
  -n haports \
  --protocol All --frontend-port 0 --backend-port 0 \
  --frontend-ip-name fe-trust \
  --backend-pool-name bep-nva-trust \
  --probe-name probe-nva \
  --floating-ip true \
  --idle-timeout 30

Then add both trusted NIC IP configurations to the backend pool:

az network nic ip-config address-pool add -g $RG \
  --nic-name nic-nva-a-trust --ip-config-name ipconfig1 \
  --lb-name $ILB --address-pool bep-nva-trust

az network nic ip-config address-pool add -g $RG \
  --nic-name nic-nva-b-trust --ip-config-name ipconfig1 \
  --lb-name $ILB --address-pool bep-nva-trust

A few details that bite people:

• --protocol All --frontend-port 0 --backend-port 0 is the HA Ports rule. It load-balances every flow regardless of port/protocol. You cannot combine an HA Ports rule with other LB rules on the same frontend.
• Floating IP (DSR) must be true for HA Ports. With floating IP, the original destination IP is preserved to the backend; the NVA sees the real frontend IP, which is exactly what a transparent firewall needs.
• The probe must reflect real data-plane health. Probing SSH (port 22) is a placeholder; in production probe a port or HTTP path the firewall only serves when its forwarding plane is healthy. A probe that stays green during a dataplane hang gives you a black-hole, not a failover.
• Configure the external LB symmetrically: Standard SKU, backend pool of the two untrusted NICs, same probe semantics. For pure transparent designs it also uses HA Ports + floating IP; for published-service ingress you may use specific port rules instead.

Step 3 — Point UDRs at the ILB Frontend, Not a Single NVA

This is the change that eliminates the failover script. In every spoke (and in the hub subnets that need to egress through the firewall), the default route’s next hop is the ILB frontend IP, with next-hop type VirtualAppliance.

RT=rt-spoke1

az network route-table create -g $RG -n $RT

# Default route -> ILB frontend IP (10.0.1.10), NOT a NVA NIC.
az network route-table route create -g $RG --route-table-name $RT \
  -n default-to-firewall \
  --address-prefix 0.0.0.0/0 \
  --next-hop-type VirtualAppliance \
  --next-hop-ip-address 10.0.1.10

# Associate with the spoke subnet
az network vnet subnet update -g rg-spoke1 \
  --vnet-name vnet-spoke1 -n snet-app \
  --route-table $RT

Because the next hop is a load balancer frontend, Azure forwards each new flow to a healthy backend NVA. If an NVA fails its probe, the ILB stops sending it flows — the UDR never changes. Convergence is governed by your probe interval and threshold (e.g. 5s interval, 2 failures => ~10s), not by a control-plane rewrite.

Equivalent Terraform for the route, if you manage routing as code:

resource "azurerm_route" "default_to_fw" {
  name                   = "default-to-firewall"
  resource_group_name    = azurerm_resource_group.hub.name
  route_table_name       = azurerm_route_table.spoke1.name
  address_prefix         = "0.0.0.0/0"
  next_hop_type          = "VirtualAppliance"
  next_hop_in_ip_address = azurerm_lb.ilb_trust.frontend_ip_configuration[0].private_ip_address
}

Do not forget the hub’s own subnets. The GatewaySubnet, for instance, may need UDRs steering on-prem-bound or spoke-bound return traffic back through the firewall so that the return path is also load-balanced and symmetric. Mismatched forward/return steering is the classic source of one-way connectivity.

Step 4 — Ensure Flow Symmetry So Stateful Inspection Survives

Symmetry is the whole game. With the sandwich, you get it as follows:

• The ILB and external LB both default to the 5-tuple hash (source IP, source port, dest IP, dest port, protocol). For a given flow, the tuple is stable, so both LBs deterministically select the same backend index — meaning forward and return packets hit the same NVA.
• Keep the distribution mode identical on both LBs. If you switch one to source-IP affinity (2-tuple/3-tuple) and leave the other on 5-tuple, you reintroduce asymmetry. Either use 5-tuple on both, or 2-tuple on both — never mixed.
• For protocols where you must guarantee a client always lands on the same instance (some IPsec/UDP cases), set both rules to source-IP affinity:

az network lb rule update -g $RG --lb-name $ILB -n haports \
  --load-distribution SourceIP   # or SourceIPProtocol

There are two architectural ways to guarantee return symmetry, and you should pick one deliberately:

If you run no-SNAT (to preserve client IPs through the firewall), the return-path UDRs and the matching LB distribution mode are load-bearing. If you run SNAT, symmetry is largely self-healing but you give up source-IP visibility at the firewall and on the destination workload.

Active-Active vs Active-Passive Trade-offs

Active-active maximizes utilization but, by default, sessions pinned to a failed NVA are dropped and must reconnect — the load balancer reroutes new flows to the survivor, it cannot migrate an in-flight session table. Some vendors offer session-state synchronization between cluster members; if session survival (not just connectivity survival) matters, enable it per the vendor’s clustering guide and size for the sync overhead.

SNAT and Port Exhaustion

When the NVA SNATs many east-west or egress flows behind a small set of IPs, you can hit SNAT port exhaustion under high connection rates. Mitigations:

• Give the NVA a pool of outbound IPs (multiple frontend/outbound IPs) so SNAT ports are spread across more addresses.
• Tune the firewall’s source-port reuse / timeout behavior.
• Prefer no-SNAT for internal east-west where the destination can route back natively, reserving SNAT for internet egress.
• Watch idle timeouts: the LB idle timeout and the firewall’s session timeout should be aligned (the LB rule above uses 30 minutes via --idle-timeout 30) so the LB does not silently reap a connection the firewall still considers live.

Enterprise scenario

A payments platform team ran a Palo Alto VM-Series pair in the hub as the no-SNAT egress and inspection chokepoint for ~40 spokes. They needed original client IPs preserved for PCI logging, so SNAT was off. The sandwich was textbook: Standard ILB, HA Ports, floating IP, UDRs at the ILB frontend. It passed every functional test and held for months.

Then they enabled forced tunneling so spoke-to-on-prem traffic would also be inspected. Connectivity to on-prem subnets started failing intermittently — roughly half of all new flows. Classic asymmetry. The forward path (spoke -> ILB -> NVA-A -> ExpressRoute) was load-balanced; the return path was not. On-prem-bound return traffic landed on the GatewaySubnet, which had no route table, so it took the system default straight back to the spoke, bypassing the firewall entirely. With no-SNAT, the firewall never saw the return, half the sessions hashed to NVA-B on the way back, and stateful inspection dropped them.

The fix was a GatewaySubnet UDR steering RFC1918 return traffic back through the same ILB frontend, restoring symmetry:

az network route-table route create -g rg-hub-nva \
  --route-table-name rt-gateway -n return-via-fw \
  --address-prefix 10.0.0.0/8 \
  --next-hop-type VirtualAppliance \
  --next-hop-ip-address 10.0.1.10

The lesson: the load balancer sandwich only guarantees symmetry for traffic it actually sees in both directions. Any subnet originating return traffic — GatewaySubnet above all — needs an explicit UDR, or no-SNAT flows die the moment the topology grows a new path.

Verify

Confirm the data plane behaves before you trust it.

1. NICs have IP forwarding on:

az network nic show -g $RG -n nic-nva-a-trust \
  --query "enableIpForwarding"   # expect: true

2. ILB rule is HA Ports with floating IP:

az network lb rule show -g $RG --lb-name ilb-nva-trust -n haports \
  --query "{proto:protocol, fePort:frontendPort, bePort:backendPort, floatingIp:enableFloatingIp}"
# expect: proto=All, fePort=0, bePort=0, floatingIp=true

3. Both NVAs are healthy in the backend pool:

az network lb show -g $RG -n ilb-nva-trust \
  --query "backendAddressPools[].loadBalancerBackendAddresses[].name"

Then check probe health in Azure Monitor metrics (Health Probe Status / DipAvailability) for the ILB — both backends should report healthy.

4. UDR effective routes from a spoke NIC point at the ILB frontend:

az network nic show-effective-route-table \
  -g rg-spoke1 -n <spoke-vm-nic> -o table
# 0.0.0.0/0 should show next hop = VirtualAppliance, IP = 10.0.1.10

5. Symmetry / failover smoke test:

• From a spoke VM, start a long-lived connection (e.g. ssh to an internet host or a sustained download) and confirm it flows.
• On the firewall consoles, confirm the same flow appears in only one NVA’s session table (proves symmetry, not asymmetric duplication).
• Stop the firewall service (or shut down) on the NVA carrying that flow. Confirm: existing flow resets (expected in active-active without session sync), but a new connection succeeds within your probe convergence window and lands on the survivor.

Pre-Production Checklist

• Both load balancers are Standard SKU (not Basic).
• Azure IP forwarding enabled on every data-plane NIC, and inside the guest OS.
• NVAs deployed across availability zones (or at least an availability set).
• ILB rule uses HA Ports (protocol All, ports 0/0) with floating IP = true.
• Health probe targets a port/path that reflects real data-plane health, not just OS liveness.
• Probe interval/threshold tuned for an acceptable convergence time and documented.
• Spoke and hub UDRs point 0.0.0.0/0 next hop at the ILB frontend IP, type VirtualAppliance.
• Return-path UDRs (GatewaySubnet, hub subnets) steer traffic back through the firewall.
• Load distribution mode is identical on the internal and external LBs.
• SNAT vs no-SNAT decision made explicitly; client-IP preservation requirement confirmed.
• LB idle timeout aligned with firewall session timeout.
• SNAT outbound IP pool sized against expected concurrent-connection count.
• Failover tested: new-flow recovery time measured; session-reset behavior accepted or session sync enabled.

Pitfalls

• Pointing the UDR at a single NVA NIC “just to test.” It works, you ship it, and now you have no HA at all — the LB is bypassed. The next hop must be the LB frontend.
• Green probe, dead data plane. Probing SSH/management while the forwarding engine is hung creates a black hole. Probe what the firewall serves only when it can actually forward.
• Mismatched LB distribution modes between the two LBs — instant asymmetric routing and dropped return packets that look like random connectivity loss.
• Forgetting the return path. Forward traffic steered through the firewall but return traffic taking the system default route is the textbook one-way-connectivity bug.
• Expecting session survival in active-active. Without vendor session sync, failover resets in-flight connections. That is usually acceptable — but decide it on purpose, do not discover it during an incident.

Next Steps

Once the sandwich is stable, codify all of it (NICs, LBs, probes, route tables, UDR associations) in Terraform or Bicep so the symmetry-critical settings cannot drift by hand. Add alerting on the ILB DipAvailability metric and on per-NVA throughput so you see a degraded node before customers do, and run the failover smoke test on a schedule rather than once at go-live.