ExpressRoute Deep Dive: Private Peering, Route Filters, and VPN Failover

At 02:00 a backhoe three states away cuts a fiber bundle into a colocation meet-me room, and a “redundant” ExpressRoute circuit that had run flawlessly for two years goes dark on one link. If the design was right, nobody notices: the second link inside the same circuit carries every packet and the on-call engineer sleeps. If the design was subtly wrong — a second BGP session that quietly went down weeks ago, a VPN backup sized at half the circuit’s throughput, an AS-path policy that never actually made the backup less preferred — the incident that follows is the one people write post-mortems about. ExpressRoute gives you a private, predictable Layer-3 path into Azure that never touches the public internet, but the resiliency story only holds together if BGP is configured deliberately, every failure domain is understood, and the backup path is tested for capacity and not merely reachability.

This is the design deep-dive for that resilience. We will build the model from the primitives up — circuit, peering location, peering, and the two MSEE connections that every circuit carries whether you asked for them or not — then work through the BGP that makes traffic move the way you intend: the MSEE ASN 12076, your peer ASN, MED and AS-path prepend for the Azure-to-on-prem direction, and weight and local-preference for the on-prem-to-Azure direction. From there we design the whole resiliency ladder: a single resilient circuit, active-active vs active-passive patterns with a Site-to-Site VPN as backup, ExpressRoute Global Reach for datacenter-to-datacenter over the Microsoft backbone, and maximum resiliency with dual circuits at two different peering locations. We enable FastPath to take the gateway out of the data path, control advertised prefixes with route filters (Microsoft peering) and BGP policy (private peering), and — the part most teams skip — we test failover, measure BGP convergence, and stand up Connection Monitor so the next fiber cut is a graph, not a surprise.

Every design decision here comes with the exact az CLI, the Terraform, and a vendor-neutral Cisco IOS-style router snippet, because the failover behaviour you want lives half in Azure and half on your edge. And because this is a reference you will reopen the night a link drops, the trade-offs, the BGP attributes, the SKU limits, the resiliency tiers, and the failure modes are all laid out as scannable tables — read the prose once, keep the tables open when it matters. By the end you will be able to design an ExpressRoute topology to a stated availability target, defend every routing knob to an architecture review, and prove your failover works with a number instead of a hope.

What problem this solves

A single ExpressRoute circuit, correctly provisioned, already carries two independent BGP sessions to two Microsoft Enterprise Edge routers (MSEEs) — Microsoft engineers this redundancy into every circuit and you cannot turn it off. Teams read “redundant by default” and stop thinking, which is exactly where the trouble starts. That built-in redundancy protects against the failure of one MSEE or one link inside the meet-me room. It does not protect against: the peering location itself losing power, your provider’s equipment failing on both links, a fiber cut on your side of the cross-connect, a fat-fingered BGP change on your edge, or your provider’s backbone having a bad day. A design that treats “the circuit is redundant” as “we are covered” has one failure domain — the facility — sitting under everything, invisible until it fails.

What breaks without a deliberate resiliency design: a fiber cut at the peering location takes the whole circuit down (both links share that facility), and if there is no second circuit at a different location and no VPN backup, hybrid connectivity is simply gone until the provider restores service — which for a physical cut can be hours. Worse are the silent failures: one of the two BGP sessions drops and nobody alerts on it, so the circuit runs non-redundant for weeks; then the surviving MSEE goes into planned maintenance and everything fails at once, in a window Microsoft told you about and you didn’t connect to your own single-session state. Or the backup exists but was only ever tested with a ping — so when it takes over during a real cut, the VPN gateway’s aggregate throughput is a fraction of the circuit’s and batch jobs that ran in forty minutes are now projected at six hours.

Who hits this: every enterprise running production workloads over ExpressRoute — banks, payments platforms, healthcare, anyone whose hybrid path is load-bearing. It bites hardest on teams that treat ExpressRoute as “just a faster VPN” and never learn the BGP, teams whose backup VPN gateway SKU was chosen for cost without a throughput calculation, and teams that pass their quarterly failover drill because the drill only checks that traffic moves, never that it moves fast enough. The fix is never “buy a bigger circuit” in isolation — it is to enumerate every failure domain, put an independent path under each, make BGP prefer the paths in the order you intend, and test the whole thing under production-representative load.

To frame the entire field before the deep dive, here is every failure domain an ExpressRoute design must reason about, what it takes down, and the mitigation that removes it:

Learning objectives

By the end of this article you can:

• Distinguish a circuit, a peering location, a peering, and the two MSEE connections precisely, and explain why every circuit is redundant at the link level but not at the facility level.
• Stand up private peering with both BGP sessions to MSEE ASN 12076, authenticate with MD5, advertise summarized prefixes, and confirm both sessions are Established from the Azure side.
• Control routing deterministically in both directions: weight and local-preference on-prem for the forward path, AS-path prepend and MED for the return path, and more-specific routes where longest-prefix-match must win.
• Design and defend active-active vs active-passive ExpressRoute+VPN patterns so ExpressRoute is always preferred and the VPN carries traffic only on failure — and size the VPN so failover survives real load.
• Connect two datacenters to each other over the Microsoft backbone with ExpressRoute Global Reach, and reason about when Global Reach beats an on-prem WAN.
• Architect maximum resiliency: dual circuits at two different peering locations, understand ExpressRoute Metro, and place zone-redundant gateways correctly.
• Enable FastPath to bypass the gateway data path, apply route filters to Microsoft peering, stay under the route limits (4,000 Standard / 10,000 Premium), and know each feature’s prerequisites and gotchas.
• Test failover by shutting BGP sessions in a window, measure convergence with and without BFD, and stand up Connection Monitor and metric alerts on BgpAvailability/ArpAvailability so degradation is visible before it is an outage.

Prerequisites & where this fits

You should already understand the ExpressRoute basics this deep-dive builds on: that ExpressRoute is a private Layer-3 connection into Azure through a connectivity provider, that a circuit has a SKU family (billing: MeteredData vs UnlimitedData) and a SKU tier (Standard vs Premium), and that private peering carries traffic to your VNets while Microsoft peering carries traffic to Microsoft public endpoints. If any of that is fuzzy, read ExpressRoute Fundamentals: Circuits, Private vs Microsoft Peering and Connectivity Models first — this article assumes it. You should be comfortable with BGP fundamentals (eBGP sessions, the attribute decision order, AS-path, local-preference, MED, weight), reading az JSON output, and enough of a router CLI to recognise a router bgp stanza.

This sits at the top of the hybrid connectivity track — it is the design and resiliency layer above the fundamentals. It pairs tightly with Production Site-to-Site VPN to Azure: Active-Active Gateways with BGP, because the VPN backup we design here is that article’s gateway pressed into a failover role. It connects upward to Azure Multi-Region Active-Active Architecture: Designing for Zero-Downtime (ExpressRoute resiliency is one pillar of a multi-region story) and to Hub-and-Spoke vs Virtual WAN: Choosing an Enterprise Cloud Network Topology (where the circuit terminates in your topology). For verification it leans on Diagnosing Connectivity with Network Watcher: Connection Monitor, Connection Troubleshoot and Next Hop, whose Connection Monitor is the tool we use to prove reachability end-to-end.

A quick map of who owns what during an ExpressRoute incident, so you page the right team fast rather than staring at the wrong console:

Core concepts

Six mental models make every later design decision obvious.

A circuit is a logical resource, not a cable — and it is always dual-homed. The circuit is the Azure object representing your connectivity through a provider. It has a service key (s-key), a bandwidth (50 Mbps to 10 Gbps, or up to 100 Gbps on ExpressRoute Direct), a SKU family, a SKU tier, and a billing model. When you create it, Azure provisions two connections to two different MSEEs in the peering location — a primary and a secondary — and this is not optional and not a toggle. Every peering you configure on the circuit therefore comes up as two eBGP sessions, one to each MSEE. Your job is to terminate both and keep both Established. “The circuit is redundant” means link-redundant within the facility; it says nothing about the facility, your provider, or your edge.

The peering location is a physical facility, distinct from the Azure region. A peering location (e.g. “Silicon Valley”, “London2”, “Chennai”) is the colocation/meet-me facility where Microsoft’s edge and your provider interconnect. It is not an Azure region — a circuit in the “Washington DC” peering location can reach VNets in many regions (all regions in the geopolitical area on Standard; any region worldwide on Premium). The facility is your deepest single failure domain on one circuit: both the primary and secondary connections land in the same meet-me room. True path diversity means a second circuit at a different peering location.

BGP is the whole game, and it has two independent directions. ExpressRoute runs eBGP between your peer ASN and Microsoft’s MSEE ASN 12076. Traffic direction matters because different attributes govern each way. On-prem→Azure (the forward path) is decided by your routers using weight (Cisco-proprietary, evaluated first, local to one router) and local-preference (standard, evaluated early, propagated across your AS) — you make the ExpressRoute-learned routes more attractive than VPN-learned ones. Azure→on-prem (the return path) is decided by Azure/Microsoft choosing among the routes you advertise, influenced by AS-path length (prepend extra ASNs on the path you want less preferred) and MED (Multi-Exit Discriminator, a hint to a neighbouring AS about which entry point to prefer). Getting failover right means setting the correct knob for each direction — a design that only tunes one direction fails asymmetrically.

More-specific routes and longest-prefix-match are the tie-breaker. BGP’s first decision, before any attribute, is longest-prefix-match: a /24 always wins over a /16 for an address inside it, regardless of local-pref, AS-path, or weight. This is a tool and a trap. Advertising a more-specific prefix over your preferred path forces traffic onto it deterministically; advertising the same specific prefixes over both paths (correct for failover) lets your attribute policy choose while withdrawal handles the failure. Advertising a summary over one path and specifics over another accidentally pins traffic to the specifics forever — a classic self-inflicted “why is my backup carrying production” bug.

The gateway is a throughput point unless you bypass it. To reach VNets, a circuit connects to an ExpressRoute gateway — a VNet gateway of an ErGw* SKU (not a VPN gateway; a VNet can host both). The gateway sits in the data path and caps throughput at its SKU’s ceiling (from ~1 Gbps on ErGw1Az to ~10 Gbps on ErGw3Az, higher on scalable gateways). FastPath changes this: it programs the MSEE to send data-plane traffic directly to VMs in the VNet, bypassing the gateway; only control-plane (BGP) still flows through it. FastPath is how you get consistent low latency at circuit line-rate instead of gateway-limited throughput — but it has SKU prerequisites and it bypasses any UDR-forced NVA in that path, which you must decide consciously.

A backup path you tested only for reachability is not tested. ExpressRoute has no meaningful availability if a single facility outage takes you fully offline, so the supported backup is a Site-to-Site VPN over the internet, coexisting with the ExpressRoute gateway. The design invariant is ExpressRoute always preferred; VPN carries traffic only when ExpressRoute is down. The failure most teams don’t rehearse is capacity: a VPN gateway’s aggregate throughput and a single IPsec tunnel’s cap are both far below a multi-gigabit circuit, and a single TCP flow rides a single tunnel. Drive production-representative load across the backup in a window — ER preference keeps it idle in steady state, so the drill costs an iperf run, not an outage.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

Circuits, peerings and the two-link contract

Everything starts with the circuit, and the first design decision — before any BGP — is SKU tier and family, because it silently caps two things you care about: reachability scope and the route limit.

The SKU family is your billing model. MeteredData bills a flat monthly circuit fee plus per-GB egress; UnlimitedData bills a higher flat fee with no per-GB charge. The break-even is purely a function of your monthly egress: high-volume egress favours UnlimitedData, spiky or modest egress favours MeteredData. You can move MeteredData → UnlimitedData in place, but not back — Azure blocks the downgrade — so start metered unless you already know you are a heavy egress shop.

The SKU tier controls reach and route limits. Standard reaches VNets within the same geopolitical region and accepts up to 4,000 IPv4 routes on private peering. Premium reaches VNets in any geopolitical region worldwide (global reach of the circuit, not to be confused with the Global Reach feature), raises the private-peering route limit to 10,000 IPv4 routes, and raises the number of VNets a circuit can connect to. Premium is an add-on charge on top of the circuit.

The SKU decision as a matrix — pick the row your requirements land in:

And the family (billing) decision:

The circuit bandwidth is a separate axis — 50/100/200/500 Mbps, 1/2/5/10 Gbps on standard ExpressRoute, and up to 100 Gbps on ExpressRoute Direct (where you own the port pair into Microsoft’s edge). Bandwidth can be increased in place (subject to provider capacity) but not decreased; size with headroom because you cannot walk it back. The bandwidth is the aggregate across both links — it is not per-link — and it is the number your VPN backup must be sized against.

Provisioning states — the sequence that saves money

A circuit has two independent state machines you must both watch. The circuit provisioning state (circuitProvisioningState) is Azure’s view: Enabled once the resource exists. The service-provider provisioning state (serviceProviderProvisioningState) is the provider’s handoff: it starts NotProvisioned, and only your provider moves it to Provisioned after they map your physical cross-connect to the s-key. Do not configure peering until the provider flips to Provisioned — the BGP won’t come up and you’ll chase ghosts.

The states, what each means, and what you do in it:

A billing gotcha worth internalising: billing starts when the circuit reaches Provisioned, not when you send your first packet. If the provider provisions well ahead of your readiness, you are paying for an idle circuit. Sequence the provider order to land close to when you can actually configure and test peering.

Create the circuit:

az network express-route create \
  --name erc-prod-svc \
  --resource-group rg-connectivity \
  --location eastus2 \
  --provider "Equinix" \
  --peering-location "Silicon Valley" \
  --bandwidth 1000 \
  --sku-family MeteredData \
  --sku-tier Standard

Retrieve the service key and current states:

az network express-route show \
  --name erc-prod-svc \
  --resource-group rg-connectivity \
  --query "{key:serviceKey, provider:serviceProviderProvisioningState, circuit:circuitProvisioningState, bandwidth:serviceProviderProperties.bandwidthInMbps}" \
  -o table

Poll the provider state until it flips (script this against your change window):

az network express-route show \
  --name erc-prod-svc \
  --resource-group rg-connectivity \
  --query serviceProviderProvisioningState -o tsv

The same circuit in Terraform, so the SKU and billing choice is reviewed and never drifts:

resource "azurerm_express_route_circuit" "prod" {
  name                  = "erc-prod-svc"
  resource_group_name   = azurerm_resource_group.conn.name
  location              = "eastus2"
  service_provider_name = "Equinix"
  peering_location      = "Silicon Valley"
  bandwidth_in_mbps     = 1000

  sku {
    tier   = "Standard"      # or "Premium" for cross-geo reach / 10k routes
    family = "MeteredData"   # or "UnlimitedData" — one-way change only
  }

  tags = { workload = "hybrid-core", tier = "prod" }
}

Private peering and the redundant BGP pair

Private peering is the peering your VNets ride, and standing it up correctly means bringing up both BGP sessions and keeping them up. It needs two point-to-point subnets — either two /30s (one primary, one secondary) or a single /29 that Azure splits — for the link addressing. Azure takes the first usable address in each subnet; your edge router takes the second. You supply your peer ASN (2-byte or 4-byte; private ASNs in the 64512–65534 range are fine on your side, as are the 4-byte private ranges), the VLAN ID (the C-tag / inner dot1Q that must match what your provider configures on the cross-connect — Microsoft peering, if used, takes a different VLAN), and in production an MD5 shared key to authenticate the sessions.

The parameters that must line up, and who owns each:

Create the peering:

az network express-route peering create \
  --circuit-name erc-prod-svc \
  --resource-group rg-connectivity \
  --peering-type AzurePrivatePeering \
  --peer-asn 65010 \
  --primary-peer-subnet 192.168.10.0/30 \
  --secondary-peer-subnet 192.168.10.4/30 \
  --vlan-id 100 \
  --shared-key "<md5-secret>"

The same in Terraform:

resource "azurerm_express_route_circuit_peering" "private" {
  peering_type                  = "AzurePrivatePeering"
  express_route_circuit_name    = azurerm_express_route_circuit.prod.name
  resource_group_name           = azurerm_resource_group.conn.name
  peer_asn                      = 65010
  primary_peer_address_prefix   = "192.168.10.0/30"
  secondary_peer_address_prefix = "192.168.10.4/30"
  vlan_id                       = 100
  shared_key                    = var.er_md5_secret   # keep out of state where possible
}

On your edge you bring up two eBGP sessions to MSEE ASN 12076, one per link. The example is Cisco IOS-style; translate to your platform. Note the MD5 password and the summarized network statement (more on summarization in the route-limit section):

! ----- Primary link (to MSEE #1) -----
interface GigabitEthernet0/1.100
 encapsulation dot1Q 100
 ip address 192.168.10.2 255.255.255.252
!
! ----- Secondary link (to MSEE #2) -----
interface GigabitEthernet0/2.100
 encapsulation dot1Q 100
 ip address 192.168.10.6 255.255.255.252
!
router bgp 65010
 bgp log-neighbor-changes
 address-family ipv4
  ! advertise a SUMMARY, not the whole table
  network 10.50.0.0 mask 255.255.0.0
  ! Primary
  neighbor 192.168.10.1 remote-as 12076
  neighbor 192.168.10.1 password <md5-secret>
  neighbor 192.168.10.1 activate
  ! Secondary
  neighbor 192.168.10.5 remote-as 12076
  neighbor 192.168.10.5 password <md5-secret>
  neighbor 192.168.10.5 activate
 exit-address-family

Both sessions must reach Established. Confirm ARP and BGP health from the Azure side, per link — the --path argument is how you check each MSEE independently:

# ARP resolution on the primary link (proves L2/L3 up)
az network express-route list-arp-tables \
  --resource-group rg-connectivity \
  --name erc-prod-svc \
  --path primary --peering-name AzurePrivatePeering -o table

# Routes learned/advertised on the primary link (proves BGP up)
az network express-route list-route-tables \
  --resource-group rg-connectivity \
  --name erc-prod-svc \
  --path primary --peering-name AzurePrivatePeering -o table

# BGP session summary (state, prefixes) on the primary
az network express-route list-route-tables-summary \
  --resource-group rg-connectivity \
  --name erc-prod-svc \
  --path primary --peering-name AzurePrivatePeering -o table

Run each with --path secondary as well. If only one side is up, you have a redundant circuit running non-redundant — treat it as a P1, because the next MSEE maintenance event will take you fully down. The per-link checks and what a healthy result looks like:

Connecting the circuit to a VNet gateway

BGP being up gets routes to the MSEE, but to actually move packets to VNets the circuit must connect to an ExpressRoute gateway via a connection. The gateway is a VNet gateway of an ErGw* SKU — critically, not a VPN gateway, though the same VNet can host both a VPN gateway and an ER gateway in the GatewaySubnet. Choose a zone-redundant SKU (ErGw1Az/ErGw2Az/ErGw3Az) in a region with Availability Zones so the gateway itself survives a zone failure.

The ER gateway SKUs and what they buy:

Create the connection, and note the routing weight — this is the Azure-side ExpressRoute connection weight that will make ExpressRoute preferred over the VPN connection outbound from Azure:

az network vpn-connection create \
  --name cn-er-hub \
  --resource-group rg-connectivity \
  --vnet-gateway1 ergw-hub \
  --express-route-circuit2 erc-prod-svc \
  --routing-weight 20000

The same, plus a zone-redundant gateway, in Terraform:

resource "azurerm_virtual_network_gateway" "er" {
  name                = "ergw-hub"
  resource_group_name = azurerm_resource_group.conn.name
  location            = "eastus2"
  type                = "ExpressRoute"
  sku                 = "ErGw3AZ"          # zone-redundant, FastPath-capable

  ip_configuration {
    name                          = "ipcfg"
    subnet_id                     = azurerm_subnet.gateway.id  # must be "GatewaySubnet"
    public_ip_address_id          = azurerm_public_ip.ergw.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_virtual_network_gateway_connection" "er" {
  name                       = "cn-er-hub"
  resource_group_name        = azurerm_resource_group.conn.name
  location                   = "eastus2"
  type                       = "ExpressRoute"
  virtual_network_gateway_id = azurerm_virtual_network_gateway.er.id
  express_route_circuit_id   = azurerm_express_route_circuit.prod.id
  routing_weight             = 20000       # high → prefer ER over VPN outbound
}

BGP routing control: making traffic go where you intend

This is the heart of a resilient design, and the single idea to hold onto is that routing has two directions and each is controlled by a different party with different attributes. Get one direction right and the other wrong, and you get asymmetric routing — traffic leaves on ExpressRoute and returns on the VPN, or vice versa — which stateful firewalls and NAT devices hate and which produces “it works but drops half the connections” bugs that are miserable to diagnose.

The BGP decision process, in order, with who controls each step for the ExpressRoute case:

Two facts fall out of the table. Weight and local-preference are evaluated first (steps 1–2), before AS-path — so on your side they are the strongest, most deterministic levers, and they govern the forward path (on-prem→Azure). AS-path and MED are evaluated later (steps 4, 6) and are what Azure sees when choosing among the routes you advertise — so they govern the return path (Azure→on-prem). A correct failover design tunes both.

Forward path (on-prem → Azure): weight and local-preference

Azure advertises your VNet prefixes over both the ExpressRoute private peering and the VPN BGP session (when both are up). Your routers therefore learn the same VNet prefixes twice and must prefer the ExpressRoute copy. Do that with local-preference — deterministic, propagated across your AS, evaluated before AS-path so it isn’t undone by anything Azure does upstream:

! Tag VPN-learned routes with LOWER local-preference so ER wins.
ip prefix-list AZURE-VNETS permit 10.0.0.0/8 le 24
!
route-map FROM_AZURE_VPN permit 10
 set local-preference 80          ! lower than default 100
!
route-map FROM_AZURE_ER permit 10
 set local-preference 200         ! higher than default — ER strongly preferred
!
router bgp 65010
 address-family ipv4
  ! ExpressRoute neighbours (both MSEE links)
  neighbor 192.168.10.1 route-map FROM_AZURE_ER in
  neighbor 192.168.10.5 route-map FROM_AZURE_ER in
  ! VPN tunnel BGP neighbour
  neighbor 10.200.0.4  route-map FROM_AZURE_VPN in
 exit-address-family

Weight is the even-stronger, router-local lever — useful when you want a specific router to prefer a specific link regardless of what the rest of your AS thinks (e.g. to pin the primary MSEE link over the secondary on a given router). Because weight is not propagated, use local-preference for AS-wide policy and weight only for per-device tuning:

! On one router, prefer the primary MSEE link over the secondary for outbound.
router bgp 65010
 address-family ipv4
  neighbor 192.168.10.1 weight 200   ! primary link preferred on THIS router
  neighbor 192.168.10.5 weight 100   ! secondary link (default-ish)
 exit-address-family

The forward-path levers compared:

Return path (Azure → on-prem): AS-path prepend and MED

Azure must choose which path to send return traffic on. It sees the routes you advertise over ExpressRoute and over the VPN. To make the VPN path less attractive to Azure, make the routes you advertise over the VPN look worse — longer AS-path. AS-path prepending repeats your ASN on the advertisement so the path length grows; Azure prefers the shorter ExpressRoute path (step 4 in the decision process):

! Prepend our ASN on routes advertised OVER THE VPN so Azure prefers ER.
route-map TO_AZURE_VPN permit 10
 set as-path prepend 65010 65010 65010   ! three extra hops → longer path
!
router bgp 65010
 address-family ipv4
  neighbor 10.200.0.4 route-map TO_AZURE_VPN out
 exit-address-family

On the Azure VPN gateway you can also set AS-path prepend and a weight on the connection when you cannot touch on-prem policy — Azure exposes routingWeight on the connection and BGP peering config on the gateway. MED is the finer tool: it hints a neighbouring AS which of your multiple entry points to prefer (lower MED wins). Use MED when you advertise the same prefix over two ExpressRoute circuits at different locations and want Azure to prefer one entry point for return traffic without changing AS-path.

The return-path levers compared:

The more-specific-route trap, made explicit

The most common self-inflicted failover bug: advertise a summary (10.50.0.0/16) over ExpressRoute and specifics (10.50.1.0/24, 10.50.2.0/24…) over the VPN. Because longest-prefix-match wins before any attribute, Azure sends traffic for 10.50.1.5 to the VPN’s /24 even while ExpressRoute is up and preferred by every attribute you set. Your expensive circuit sits idle carrying nothing for those prefixes, and your “backup” carries production. The rule is simple: advertise the same prefixes over both paths — same summaries, or same specifics — and let attribute policy plus withdrawal do the selection.

The prefix-advertisement patterns and their outcomes:

Never advertise a 0.0.0.0/0 default over the VPN as a lazy backup. It “works” in that it only wins when ExpressRoute withdraws its specifics — but a default route also risks pulling internet-bound traffic into the tunnel and interacts badly with forced-tunneling policy. Advertise the same specific VNet prefixes over both paths; let longest-prefix-match plus your attribute policy select.

Active-active vs active-passive: designing the VPN backup

With BGP direction understood, the resiliency pattern is a choice between two shapes. Active-passive keeps ExpressRoute carrying all traffic and the VPN idle-but-ready, taking over only on ExpressRoute failure. Active-active (in the ExpressRoute sense) is really two variants: two ExpressRoute circuits both carrying traffic, or an ExpressRoute-plus-VPN where you deliberately use both — though for ER+VPN the near-universal choice is passive VPN, because the whole point of ExpressRoute is its predictable performance and you don’t want production riding an internet tunnel in steady state.

The patterns, ordered by resilience and cost:

The VPN-as-backup coexistence design

Microsoft supports ExpressRoute and VPN gateway coexistence in the same VNet. The design invariant is ExpressRoute always preferred, VPN carries traffic only when ExpressRoute is down, achieved with the two-direction BGP policy above plus the connection routing weight on the Azure side:

# Azure-side preference: ER connection weight HIGH, VPN connection weight LOW.
# Higher routing weight wins when Azure chooses between gateways.
az network vpn-connection update --name cn-er-hub \
  --resource-group rg-connectivity --set routingWeight=20000

az network vpn-connection update --name cn-vpn-backup \
  --resource-group rg-connectivity --set routingWeight=100

The complete backup design checklist, and the direction/party each item controls:

Sizing the backup so failover survives real load

This is the item teams skip and regret. A VPN gateway’s aggregate throughput and a single IPsec tunnel’s cap are both far below a multi-gigabit circuit, and — the subtle killer — a single TCP flow rides a single tunnel, so one chatty replication stream does not spread across tunnels even in active-active. If your 1 Gbps circuit runs at 700 Mbps and your backup is a VpnGw1 (~650 Mbps aggregate), the backup is structurally undersized before you even hit the single-flow problem.

The VPN gateway SKUs against what a circuit might demand — read this the day you choose the backup SKU:

Two design moves make the backup a first-class path: choose an AZ (zone-redundant) SKU sized to the circuit’s sustained throughput (not its provisioned max), and enable active-active so there are two gateway instances, two public IPs, and two tunnels to on-prem, letting flows hash across tunnels:

# Right-size and enable active-active on the backup VPN gateway
az network vnet-gateway update \
  --name vpngw-backup --resource-group rg-connectivity \
  --sku VpnGw3AZ --set activeActive=true

The single most important validation is to drive production-representative load across the backup in a window — ER preference keeps it idle in steady state, so an iperf3 run costs nothing but proves capacity. Reachability failover (a ping succeeds) is not the same as capacity failover (the batch job still finishes in time). A backup tested only for reachability is not tested.

ExpressRoute Global Reach: datacenter-to-datacenter over the backbone

Global Reach is a distinct ExpressRoute feature that connects two of your on-prem sites to each other over the Microsoft backbone, using their ExpressRoute circuits’ private peering as transit. Without it, ExpressRoute private peering connects each site to Azure but not the sites to each other — traffic between two branches would have to hairpin through a VNet or ride your own WAN. Global Reach links the two circuits’ private-peering routing domains so branch A can reach branch B directly across Microsoft’s global network.

When Global Reach earns its place versus alternatives:

Prerequisites and constraints worth knowing before you design around it:

• Both circuits must have private peering configured and be in supported peering locations (most, but confirm the current availability list for your locations).
• The on-prem prefixes you connect must not overlap — Global Reach is Layer-3 routing between two address spaces; overlapping RFC1918 breaks it.
• Global Reach is an add-on charge (an enablement fee plus egress on the Global Reach traffic), billed on the circuits.
• It connects the private-peering domains; it does not extend Microsoft peering or give internet transit.

Enable Global Reach by adding a connection between the two circuits’ private peerings (you need the second circuit’s peering resource ID and an unused /29 for the Global Reach link addressing):

# Link circuit A's private peering to circuit B over the Microsoft backbone
PEER_B_ID=$(az network express-route peering show \
  --circuit-name erc-site-b --resource-group rg-connectivity \
  --name AzurePrivatePeering --query id -o tsv)

az network express-route peering connection create \
  --circuit-name erc-site-a \
  --resource-group rg-connectivity \
  --peering-name AzurePrivatePeering \
  --name gr-a-to-b \
  --peer-circuit "$PEER_B_ID" \
  --address-prefix 10.255.255.0/29

A design note: Global Reach connects your sites — it is not a substitute for the Azure-to-on-prem resiliency this article is mostly about. It is complementary: use ExpressRoute+VPN for Azure hybrid resilience, and Global Reach when you also want your branches talking to each other over the same private backbone.

Maximum resiliency: dual circuits at different peering locations

The single most impactful resiliency upgrade is to remove the peering-location failure domain by provisioning a second circuit at a different peering location. One circuit’s two links share a facility; two circuits at two facilities survive the loss of an entire meet-me room, a regional fiber event, or a provider’s problem at one site. Microsoft’s own maximum resiliency guidance is exactly this: two circuits, two peering locations, connected to the same or geo-redundant gateways.

The resiliency ladder, from a single link to full diversity, with the failure each rung survives:

Provision the second circuit at a different location, exactly as the first:

az network express-route create \
  --name erc-prod-svc-2 \
  --resource-group rg-connectivity \
  --location eastus2 \
  --provider "Megaport" \
  --peering-location "Washington DC" \
  --bandwidth 1000 \
  --sku-family MeteredData \
  --sku-tier Standard

Then connect both circuits to your gateway (or to geo-redundant gateways) and advertise the same prefixes over both so BGP can use either. To make one circuit primary and the other standby (rather than active-active), tune the connection routing weight and on-prem local-preference/AS-path so the primary circuit is preferred and the second takes over only on failure. To run active-active (both carrying traffic, load-shared), advertise equal-cost paths and let BGP multipath.

The active-active vs active-standby dual-circuit choice:

ExpressRoute Metro — two locations in one metro

ExpressRoute Metro is a newer configuration that provisions a single circuit whose two links terminate in two different peering locations within the same metropolitan area — giving you facility diversity inside one circuit and one bill, in metros where it is offered. It is the sweet spot when you want location diversity but a full second circuit at a distant location is more than the workload needs. Availability is metro-specific; confirm your metro supports it. Where Metro is available and diversity-within-a-metro suffices, it is a lower-cost path to removing the single-facility risk than two full circuits.

Metro vs two full circuits:

FastPath: taking the gateway out of the data path

The ExpressRoute gateway sits in the data path by default and is a throughput and latency bottleneck — it caps at the gateway SKU’s bandwidth and adds a hop with variable queuing. FastPath programs the MSEE to send data-plane traffic directly to VMs in the VNet, bypassing the gateway; control-plane (BGP, route exchange) still flows through the gateway, but the packets skip it. For latency-sensitive workloads (trading, real-time media, chatty database replication) FastPath is the difference between “ExpressRoute is fast” and “ExpressRoute is fast and consistent”, because you remove the gateway’s variable queuing from every flow and get circuit line-rate instead of gateway-limited throughput.

FastPath requirements and the scenarios it does/doesn’t cover:

Requirements worth internalising:

• FastPath requires a higher-capacity gateway — ErGw3Az is the standard choice; the older Standard/HighPerformance SKUs do not qualify, and scalable gateways support it.
• Base FastPath works to VMs in the same VNet as the gateway. Advanced scenarios — traffic landing on a Private Endpoint/Private Link, or reaching spokes across VNet peering — require additional enablement and typically a Premium circuit; confirm the current support matrix for your regions.
• FastPath does not honour a UDR that would force traffic through an NVA in the bypassed path. If your security model requires all north-south traffic to traverse a firewall, FastPath to those subnets defeats it. Decide consciously — you are trading a security inspection hop for latency.

Enable FastPath (gateway bypass) on the connection:

az network vpn-connection update \
  --name cn-er-hub \
  --resource-group rg-connectivity \
  --set expressRouteGatewayBypass=true

In Terraform, set it on the connection:

resource "azurerm_virtual_network_gateway_connection" "er" {
  # ...as above...
  express_route_gateway_bypass = true   # FastPath: data-plane bypasses the gateway
}

The FastPath trade-off, made explicit:

Route control: filters and the route limit

For private peering, you control what Azure learns by what your routers advertise (the network/redistribution statements), and what your routers accept with inbound route maps. Azure advertises your VNet address spaces plus routes from connected VNets. Keep the advertised set tight: every prefix counts against the route limit, and a sloppy redistribution can blackhole traffic.

Route filters are a separate mechanism that applies to Microsoft peering only. Microsoft peering would otherwise advertise the full set of Microsoft public prefixes (all regions, all services). A route filter is an allow-list of BGP community values that scopes advertisement down to the services and regions you actually consume. Without an attached route filter, a Microsoft peering advertises nothing — the filter is what activates prefix advertisement.

Create a route filter and a community allow-rule, then attach it:

az network route-filter create \
  --name rf-m365 \
  --resource-group rg-connectivity \
  --location eastus2

# Allow only specific service/region communities (values from Microsoft's list)
az network route-filter rule create \
  --resource-group rg-connectivity \
  --filter-name rf-m365 \
  --name allow-services \
  --access Allow \
  --communities 12076:5010 12076:5020 \
  --route-filter-rule-type Community

# Attach the filter to the Microsoft peering (this activates advertisement)
az network express-route peering update \
  --circuit-name erc-prod-svc \
  --resource-group rg-connectivity \
  --name MicrosoftPeering \
  --route-filter rf-m365

Get the community values from Microsoft’s published service/region community list rather than guessing — 12076:5010 etc. above illustrate the pattern; the real per-service, per-region values are documented and change as services are added.

Where each route-control mechanism applies:

The route limit — a hard cliff, not a slope

Private peering accepts a maximum of 4,000 IPv4 routes on Standard and 10,000 on Premium (and separate, smaller limits for IPv6). Blow past it and the BGP session drops — it does not gracefully degrade, it drops — taking that whole peering down. The usual cause is redistributing your entire on-prem routing table into BGP instead of summarized supernets. Always advertise summaries (one 10.50.0.0/16 rather than 256 /24s) and watch the count.

The route limits and the discipline that keeps you under them:

Advertise summaries on the edge, and confirm the count from Azure:

! Advertise a single summary instead of many specifics
router bgp 65010
 address-family ipv4
  aggregate-address 10.50.0.0 255.255.0.0 summary-only
 exit-address-family

# Watch the advertised/received prefix count per link — stay well under the limit
az network express-route list-route-tables-summary \
  --resource-group rg-connectivity --name erc-prod-svc \
  --path primary --peering-name AzurePrivatePeering \
  --query "value[].{neighbor:neighbor, asn:asn, up:upDown, prefixes:statePfxRcd}" -o table

Failover testing and BGP convergence

A resiliency design you have not tested is a hypothesis. Testing ExpressRoute failover means deliberately breaking a path in a change window and measuring how fast — and how completely — traffic moves to the backup, then restoring. The two numbers that matter are convergence time (how long until the backup carries traffic) and capacity (whether the backup carries it fast enough).

Convergence is BGP-driven and its speed depends on how the failure is detected. With default BGP timers, a session isn’t declared dead until the hold timer expires — commonly on the order of tens of seconds up to ~180 seconds. BFD (Bidirectional Forwarding Detection) detects link failure in sub-second time and tears the BGP session immediately, collapsing convergence to a second or two. If you need fast failover, enable BFD on the ExpressRoute BGP sessions (Microsoft supports BFD on private peering).

Convergence expectations by detection method and failure type:

Enable BFD on the edge (Cisco-style) to shrink the worst case:

! Enable BFD on the ExpressRoute BGP neighbours for sub-second detection
interface GigabitEthernet0/1.100
 bfd interval 300 min_rx 300 multiplier 3
!
router bgp 65010
 address-family ipv4
  neighbor 192.168.10.1 fall-over bfd
  neighbor 192.168.10.5 fall-over bfd
 exit-address-family

The failover test procedure

Test by administratively shutting the ExpressRoute BGP sessions on your edge (do this in a window, with a rollback ready), then confirming traffic moved to the VPN and measuring the gap:

! On the on-prem edge router — shut BOTH ER sessions to simulate a circuit loss
router bgp 65010
 neighbor 192.168.10.1 shutdown
 neighbor 192.168.10.5 shutdown

While the sessions are down, confirm the effective routes on a test NIC now point at the VPN, and measure the outage window with a continuous ping:

# Effective routes on a probe NIC — next hop should now be the VPN gateway
az network nic show-effective-route-table \
  --name nic-probe01 --resource-group rg-workload -o table
# Look for your on-prem prefixes with Next Hop Type = VirtualNetworkGateway (VPN)
# In steady state (ER up) they should read ExpressRouteGateway

Then drive load, not just a ping, across the backup — this is the capacity test that reachability alone hides:

# From an on-prem host, push representative load over the (now-active) VPN path
iperf3 -c 10.50.4.10 -t 60 -P 8    # 8 parallel streams to spread across tunnels
# Compare achieved throughput to your batch/replication requirement

Restore by removing the shutdowns and confirm ExpressRoute reclaims the traffic:

router bgp 65010
 no neighbor 192.168.10.1 shutdown
 no neighbor 192.168.10.5 shutdown

The test matrix — what to break, what to observe, what “pass” means:

Monitoring: Connection Monitor and the metrics that predict outages

Alerting on aggregate circuit state is how the silent-single-session outage happens. Alert instead on the signals that predict an outage: per-session BGP availability, ARP availability, and throughput against your provisioned bandwidth — and add Connection Monitor for real end-to-end reachability, not just control-plane state.

The ExpressRoute metrics worth alerting on, and what each catches:

Wire an alert on per-session BGP availability — the single alert that prevents the most common ExpressRoute outage:

# Alert if a BGP session on the circuit goes unavailable (per-peering)
CIRCUIT_ID=$(az network express-route show -g rg-connectivity -n erc-prod-svc --query id -o tsv)

az monitor metrics alert create \
  --name alrt-er-bgp-down \
  --resource-group rg-connectivity \
  --scopes "$CIRCUIT_ID" \
  --condition "avg BgpAvailability < 100" \
  --window-size 5m --evaluation-frequency 1m \
  --severity 1 \
  --description "ExpressRoute private peering BGP availability dropped below 100"

# And a saturation alert on egress
az monitor metrics alert create \
  --name alrt-er-egress-saturation \
  --resource-group rg-connectivity \
  --scopes "$CIRCUIT_ID" \
  --condition "avg BitsOutPerSecond > 800000000" \
  --window-size 5m --evaluation-frequency 1m \
  --severity 2 \
  --description "ExpressRoute egress above 80% of 1 Gbps"

Connection Monitor — end-to-end reachability, not just BGP

Control-plane health (BGP up) does not prove the data plane works end-to-end. Connection Monitor (in Network Watcher) runs continuous synthetic probes from an on-prem host (with the monitoring agent) to an Azure VM over the circuit, alerting on real reachability, latency, and loss, and showing the hop-by-hop path so you can see where a degradation bites. Stand it up so your alert is “on-prem can’t reach the app over ExpressRoute”, not merely “a BGP metric dipped”.

# Create a Connection Monitor (v2) test from on-prem to an Azure VM over ExpressRoute
az network watcher connection-monitor create \
  --name cm-er-onprem-to-azure \
  --resource-group rg-connectivity \
  --location eastus2 \
  --endpoint-source-name onprem-host \
  --endpoint-source-resource-id "<arc-or-agent-machine-id>" \
  --endpoint-dest-name azure-app-vm \
  --endpoint-dest-resource-id "<azure-vm-id>" \
  --test-config-name tcp-443 \
  --protocol Tcp --tcp-port 443 \
  --frequency 30

What each monitoring layer proves, and why you want all of them:

Architecture at a glance

Picture the whole resilient hybrid path as three concentric rings of redundancy, read from your datacenter outward to Azure. The innermost ring is a single ExpressRoute circuit: your two edge routers each hold an eBGP session — MD5-authenticated, ideally BFD-enabled — to a different Microsoft Enterprise Edge router (MSEE, ASN 12076) across a dual-tagged (VLAN 100) cross-connect in the peering location. Those two sessions are the circuit’s built-in redundancy; they carry your summarized VNet prefixes (10.50.0.0/16 advertised, VNet space learned) and terminate, on the Azure side, at a zone-redundant ExpressRoute gateway (ErGw3Az) in the hub VNet’s GatewaySubnet. With FastPath enabled on the connection, data-plane packets skip that gateway and land directly on VMs at circuit line-rate; only BGP still flows through it.

The middle ring removes the facility as a single point of failure: a second circuit at a different peering location (or an ExpressRoute Metro circuit spanning two locations in one metro), advertising the same prefixes, connected to the same or a geo-redundant gateway. BGP prefers the primary circuit via connection routing weight on the Azure side and local-preference on-prem; if the whole primary facility drops, the second circuit’s sessions carry every packet after a few seconds of convergence. The outermost ring is the last-resort path: a zone-redundant, active-active VPN gateway (sized to the circuit’s sustained throughput) holding IPsec tunnels to on-prem over the internet, coexisting in the same hub. Its connection weight is low and its advertised routes are AS-path-prepended, so it sits idle while any ExpressRoute path lives — and takes over only if every circuit is gone.

Follow a packet from an on-prem client to an Azure VM: it hits your edge router, which prefers the ExpressRoute-learned VNet route (high local-pref) over the VPN-learned one (low local-pref), forwards over the primary MSEE link, and — with FastPath — reaches the VM directly. The return packet leaves Azure, where the ER connection’s high routing weight and the VPN’s prepended AS-path make Azure choose the same ExpressRoute gateway, so the flow is symmetric in both directions — the property stateful firewalls depend on. Every ring is watched: per-session BgpAvailability/ArpAvailability alerts catch a silent link, throughput alerts catch saturation, and Connection Monitor proves the whole app-to-app path is alive, not just that BGP is up. The design’s one job is that a failure in any ring is invisible in the ring outside it — a fiber cut in the inner ring is caught by the second link; a facility loss in the middle ring is caught by the second circuit; a total ExpressRoute outage in the outer ring is caught by the VPN.

Real-world scenario

Meridian Payments runs a card-settlement platform on Azure: a hub-and-spoke topology in Central India and South India, with core batch settlement running against an on-prem mainframe over ExpressRoute. Connectivity was a single 1 Gbps circuit at the Chennai peering location (MeteredData, Standard) plus a Site-to-Site VPN backup, with connection weights set so ExpressRoute was preferred and a quarterly failover drill that always passed. The platform team was six engineers; ExpressRoute plus the VPN gateway cost about ₹2.4 lakh/month. The circuit typically ran at 700 Mbps during the nightly settlement window; batch settlement completed in 40 minutes on ExpressRoute.

The incident began at 02:00 on a settlement night when a contractor severed a fiber bundle into the Chennai meet-me room. Both of the circuit’s links went down together — they shared the facility — and ExpressRoute withdrew. The VPN backup took over exactly as the drill had shown: az network nic show-effective-route-table on the settlement VNet flipped to VirtualNetworkGateway, pings from the mainframe subnet succeeded, and the on-call engineer confirmed “failover worked” within four minutes. Then throughput collapsed. The 40-minute batch was projected at 6+ hours over the tunnel; the settlement window would blow past the banking cut-off and trigger regulatory penalties. The team had validated routing failover for eight straight quarters and never once validated capacity failover.

Root cause was two-fold. First, the VPN gateway was a VpnGw1 (~650 Mbps aggregate) backing a circuit that regularly ran at 700 Mbps — the backup was structurally undersized before any other factor. Second, a single IPsec tunnel caps around 1.25 Gbps and, more importantly, the chatty mainframe replication was a single TCP flow that rides a single tunnel, so even active-active wouldn’t have spread it. The quarterly drill had only ever run a ping and a small file copy — both trivially within VpnGw1’s capacity — so the structural undersizing was invisible until a real cut drove real settlement load across it. And there was no second circuit at a different location: the entire design had the Chennai facility as an unremoved single failure domain.

The fix landed in three waves. That night, they invoked the manual DR runbook to run settlement from the secondary region’s mainframe replica, absorbing the cut-off risk with a regulator notification. Within the week, they rebuilt the backup as a first-class path: moved to VpnGw3AZ (~1.25 Gbps, zone-redundant) in active-active mode with two public IPs and two tunnels, and split the monolithic replication into parallel streams so it spreads across tunnels. Within the quarter, they provisioned a second 1 Gbps circuit at the Mumbai peering location (a different provider, for provider diversity too), advertised the same prefixes over both, and set connection weights and on-prem local-preference so Chennai was primary and Mumbai took over on failure — removing the single-facility domain entirely. They also enabled BFD on all ER sessions (cutting worst-case convergence from ~180 s to sub-second) and rewrote the drill to include an iperf3 capacity test at settlement-representative load.

The next drill ran the full settlement batch over the backup Mumbai circuit during a window — 41 minutes, within one minute of the primary. The lesson that went onto the wall: “A backup you have only tested for reachability is not tested. Drive production load across it — and if the whole facility can die, put a second facility under it.” The incident as a timeline, because the order of realisations is the lesson:

Advantages and disadvantages

ExpressRoute’s private, predictable path is exactly why it is worth this design effort — and exactly why its failure modes are subtle. Weigh the model honestly:

The model is right for enterprises whose hybrid path is load-bearing and who need predictable performance and private connectivity — banks, healthcare, payments, large SaaS. It is over-engineered for a small workload where a well-designed active-active VPN would meet the SLA at a fraction of the cost. The disadvantages are all manageable: enumerate failure domains and put diversity under each, tune BGP in both directions and verify symmetry, size the backup for capacity not just reachability, and alert on per-session leading indicators. Every one of these is a decision you make on purpose — which is the entire point of designing rather than accepting defaults.

Hands-on lab

You cannot provision a real ExpressRoute circuit without a provider and a physical cross-connect, so this lab builds and validates the pieces you can stand up free-of-provider: it configures the peering resource against a circuit (create the circuit only if you have a provider; otherwise use an existing test circuit), stands up the coexisting VPN backup gateway, wires the connection weights and BGP preference that make ExpressRoute preferred, sets up the metric alert and Connection Monitor, and runs the effective-route-table and route-summary validation you’d use in a real failover test. Where a step needs a real provisioned circuit, we mark it and give the exact command; the VPN-side and monitoring steps run fully. Run in Cloud Shell (Bash); delete at the end.

Step 1 — Variables and resource group.

RG=rg-er-lab
LOC=eastus2
az group create -n $RG -l $LOC -o table

Step 2 — (If you have a provider) create the circuit, else skip to Step 3.

# Only run if you have a connectivity provider ready to provision.
az network express-route create \
  --name erc-lab --resource-group $RG --location $LOC \
  --provider "Equinix" --peering-location "Silicon Valley" \
  --bandwidth 200 --sku-family MeteredData --sku-tier Standard -o table

# Retrieve the s-key to hand to the provider, and watch the provider state
az network express-route show -n erc-lab -g $RG \
  --query "{key:serviceKey, provider:serviceProviderProvisioningState}" -o table

Expected (before the provider acts): serviceProviderProvisioningState = NotProvisioned. Do not configure peering until it reads Provisioned.

Step 3 — Build the hub VNet with a GatewaySubnet (hosts both ER and VPN gateways).

az network vnet create -n vnet-hub -g $RG -l $LOC \
  --address-prefix 10.90.0.0/16 --subnet-name GatewaySubnet \
  --subnet-prefix 10.90.255.0/27 -o table
az network public-ip create -n pip-vpngw -g $RG --sku Standard --allocation-method Static -o table

Expected: a VNet with a GatewaySubnet (the exact name matters) and a standard public IP.

Step 4 — Create the coexisting VPN backup gateway (active-active, zone-redundant, BGP). This is a real, chargeable resource and takes ~30–45 minutes to provision.

az network public-ip create -n pip-vpngw2 -g $RG --sku Standard --allocation-method Static -o table
az network vnet-gateway create \
  -n vpngw-backup -g $RG -l $LOC \
  --vnet vnet-hub --gateway-type Vpn --vpn-type RouteBased \
  --sku VpnGw2AZ --public-ip-addresses pip-vpngw pip-vpngw2 \
  --asn 65515 --no-wait
# Poll until provisioningState = Succeeded
az network vnet-gateway show -n vpngw-backup -g $RG --query provisioningState -o tsv

Expected eventually: Succeeded, with active-active (two public IPs) and BGP ASN 65515.

Step 5 — (Real circuit) configure private peering and connect it with a HIGH routing weight.

# Requires the circuit at 'Provisioned'. Configure the redundant BGP pair:
az network express-route peering create \
  --circuit-name erc-lab -g $RG \
  --peering-type AzurePrivatePeering --peer-asn 65010 \
  --primary-peer-subnet 192.168.10.0/30 \
  --secondary-peer-subnet 192.168.10.4/30 \
  --vlan-id 100 --shared-key "labSecret123!"

# Create an ER gateway (ErGw1Az for the lab) and connect with high weight
az network vnet-gateway create -n ergw-lab -g $RG -l $LOC \
  --vnet vnet-hub --gateway-type ExpressRoute --sku ErGw1AZ \
  --public-ip-addresses $(az network public-ip create -n pip-ergw -g $RG --sku Standard --allocation-method Static --query publicIp.id -o tsv) --no-wait

az network vpn-connection create --name cn-er -g $RG \
  --vnet-gateway1 ergw-lab --express-route-circuit2 erc-lab --routing-weight 20000

Step 6 — Verify the redundant BGP pair (both links Established).

# Run for BOTH paths; both must be Established with your prefixes present
for P in primary secondary; do
  echo "=== $P ==="
  az network express-route list-route-tables-summary \
    -g $RG --name erc-lab --path $P --peering-name AzurePrivatePeering -o table
done

Expected (real circuit): both primary and secondary show state Established. If only one is up, that is the silent-non-redundant condition this whole article is about — treat it as a P1.

Step 7 — Wire the per-session BGP-availability alert (the single most valuable alert).

CIRCUIT_ID=$(az network express-route show -n erc-lab -g $RG --query id -o tsv 2>/dev/null)
if [ -n "$CIRCUIT_ID" ]; then
  az monitor metrics alert create \
    --name alrt-er-bgp-down -g $RG --scopes "$CIRCUIT_ID" \
    --condition "avg BgpAvailability < 100" \
    --window-size 5m --evaluation-frequency 1m --severity 1 \
    --description "ER BGP availability dropped below 100"
fi

Expected: an alert rule created and enabled (skipped cleanly if you have no circuit).

Step 8 — Validate the failover mechanism with effective routes. On a probe NIC in a spoke, the effective route table shows the next hop for on-prem prefixes; in steady state it reads ExpressRouteGateway, and after you shut the ER BGP sessions (in a window) it should read VirtualNetworkGateway.

# (Create a tiny probe VM in a spoke first if you want a live NIC to inspect.)
az network nic show-effective-route-table --name <nic-name> -g $RG -o table
# Next Hop Type for on-prem prefixes: ExpressRouteGateway (ER up) → VirtualNetworkGateway (ER down)

Validation checklist. You stood up a hub with a GatewaySubnet, an active-active zone-redundant VPN backup gateway with BGP, and (with a provider) a private-peering circuit connected at a high routing weight so ExpressRoute is preferred; you confirmed both BGP links, wired the per-session availability alert, and learned the effective-route-table check that proves failover. The lab steps mapped to what each proves:

Cleanup (VPN and ER gateways bill by the hour — delete promptly).

az group delete -n $RG --yes --no-wait

Cost note. A VpnGw2AZ gateway is the main cost here — a few tens of rupees per hour — so an hour of this lab is well under ₹200; deleting the resource group stops everything. A real ExpressRoute circuit has a monthly floor regardless of usage, which is exactly why this lab keeps the circuit optional.

Common mistakes & troubleshooting

The failover-and-BGP playbook — the part you bookmark. First as a scannable table you can read while a link is down, then the entries that bite hardest with full reasoning.

The expanded form, for the entries that cause the most damage:

1. The “redundant” circuit that wasn’t. Root cause: One of the two BGP sessions dropped (a router reload, a cross-connect flap, a config change) and nobody alerted on it, so the circuit ran non-redundant until the surviving MSEE went into planned maintenance and everything failed at once — in a window Microsoft announced. Confirm: az network express-route list-route-tables-summary --path primary and --path secondary — one shows a state other than Established, or a prefix count of zero. Fix: Alert on per-session BgpAvailability < 100 (not aggregate circuit state); restore the down session; add BFD so future flaps recover fast. This is the single most common ExpressRoute outage there is.

2. The backup that collapsed under load. Root cause: The VPN backup was sized for cost, not throughput — a VpnGw1 (~650 Mbps) behind a 1 Gbps circuit running at 700 Mbps — and a single TCP flow rides a single tunnel, so even active-active wouldn’t spread a monolithic replication stream. Confirm: az network vnet-gateway show --query sku gives the SKU; compare its aggregate throughput to the circuit’s peak BitsOutPerSecond. Fix: Move to a VpnGwN AZ SKU sized to sustained throughput, enable active-active, and split monolithic flows so they hash across tunnels. Test with iperf3 at production load, not a ping.

3 & 4. Azure returns over VPN / on-prem sends over VPN while ER is up. Root cause: The two-direction preference is half-configured. If Azure returns traffic over the VPN, the ER connection routing weight isn’t higher than the VPN’s. If on-prem sends over the VPN, your edge route-maps aren’t preferring ER (local-pref/weight/AS-path). Confirm: az network vpn-connection show --query routingWeight on both connections; show the inbound route-maps on your ER vs VPN neighbours. Fix: ER connection weight high (20000), VPN low (100); ER-learned routes local-pref 200 on-prem, VPN-learned 80; AS-path-prepend the VPN advertisements. Tune both directions or you get asymmetry (#8).

5. The backup carrying production. Root cause: You advertised a summary over ExpressRoute and specifics over the VPN. Longest-prefix-match wins before every attribute, so Azure sends the specifics’ traffic down the VPN even while ER is preferred by every knob you set. Confirm: list-route-tables --path primary shows the summary on ER; the VPN advertises specifics. Fix: Advertise the same prefixes over both paths — same summaries or same specifics. Never mix granularity across paths.

6. The whole peering dropping at once. Root cause: You exceeded the route limit (4,000 Standard / 10,000 Premium) — usually by redistributing the entire on-prem table — and the BGP session dropped (it does not degrade; it drops). Confirm: list-route-tables-summary prefix count sitting near the limit; the session flapping. Fix: Advertise summaries (aggregate-address ... summary-only); if you legitimately need >4,000 routes, move to Premium (10,000). Watch the count with an alert before it hits the cliff.

7. Slow failover. Root cause: Default BGP timers mean a stuck-but-up session isn’t declared dead until the hold timer expires (up to ~180 s), so failover is minutes, not seconds. Confirm: Measure the ping gap when you shut the ER sessions; check for fall-over bfd in the config. Fix: Enable BFD on the ER BGP neighbours for sub-second detection; Microsoft supports BFD on private peering.

8. Asymmetric routing. Root cause: Traffic leaves on one path and returns on another (out ER, back VPN, or across two circuits), and a stateful firewall or NAT in the path drops the half it never saw the SYN for. Confirm: Trace both directions during a failover test; compare Azure effective routes to the on-prem chosen path. Fix: Make the preference consistent in both directions (weight + local-pref forward, routing weight + AS-path return); for dual circuits use MED to keep return traffic symmetric.

Best practices

• Alert on per-session BGP/ARP availability, never aggregate circuit state. The silent single-session outage is the most common ExpressRoute failure; BgpAvailability < 100 per peering catches it weeks before an MSEE maintenance turns it into a total outage.
• Terminate and monitor both MSEE sessions. A circuit is redundant only if both links are Established. Check --path primary and --path secondary; treat one-down as a P1.
• Size the VPN backup for capacity, not reachability. Match the gateway SKU to the circuit’s sustained throughput, run active-active, and validate with iperf3 at production load — a ping-only drill is not a test.
• Tune BGP in both directions and verify symmetry. Weight/local-pref for the forward path, connection routing weight + AS-path prepend for the return path. Asymmetric routing breaks stateful firewalls.
• Advertise the same prefixes over every path. Same summaries or same specifics; never a summary over ER and specifics over VPN, or LPM will pin production to the backup.
• Advertise summaries and stay well under the route limit. One /16, not 256 /24s. The limit (4,000 Standard / 10,000 Premium) is a cliff — breach it and the whole session drops.
• Enable BFD on the ER sessions for sub-second failure detection where your RTO is tight — it collapses worst-case convergence from ~180 s to seconds.
• Remove the facility as a single point of failure for mission-critical workloads: a second circuit at a different peering location (or ExpressRoute Metro), advertising the same prefixes, with a clear primary/standby or active-active preference.
• Use zone-redundant gateways (ErGw*Az, VpnGw*AZ) so the gateway itself survives a zone failure — the gateway is otherwise a single point in the path.
• Enable FastPath consciously where line-rate/low latency matters, and only on subnets that don’t require UDR-forced NVA inspection — FastPath bypasses that inspection.
• Stand up Connection Monitor end-to-end from on-prem to an Azure app so your alert is “users can’t reach the app over ER”, not merely “a BGP metric dipped”.
• Codify everything in IaC — circuit, peerings, gateways, both connections with their weights, route filters — so the preference policy that protects your failover behaviour can never drift by hand.

Security notes

• Authenticate BGP with MD5 on every session. Set the --shared-key on the peering and the matching password on your edge neighbours so an attacker who lands on the transit VLAN cannot inject routes. Rotate it as a managed secret, not a wiki entry.
• ExpressRoute is private, not trusted-by-default. Private peering carries no encryption on the wire (it is a private path, not an encrypted one). For data that must be encrypted in transit regardless, run IPsec/MACsec over ExpressRoute or terminate TLS end-to-end — the private path is isolation, not confidentiality.
• Prefer Private Link over Microsoft peering for PaaS. Azure Private Link with Private DNS for PaaS over private peering avoids public IPs, route filters, and the broader attack surface of Microsoft peering — simpler and more secure for most PaaS access.
• Scope Microsoft peering route filters tightly. If you do use Microsoft peering, allow-list only the exact service and region communities you consume; advertising the full public prefix set is unnecessary exposure.
• Understand FastPath’s inspection trade-off. FastPath bypasses the gateway and any UDR that would force traffic through a firewall/NVA. If a subnet’s traffic must be inspected for compliance, do not FastPath it — or move inspection to a point FastPath doesn’t bypass.
• Least-privilege the network RBAC. Managing circuits, peerings, and route filters is high-impact; grant Network Contributor narrowly and keep circuit/peering changes behind PR review in IaC so a bad route policy is caught before it ships.
• Protect the on-prem edge. The routers holding your ER sessions advertise your VNet reachability; harden them, control who can change BGP policy, and log route-map changes — a fat-fingered set local-preference is a self-inflicted outage.

The security controls that also improve resilience — they pull the same direction here:

Cost & sizing

The bill drivers and how they interact with the resiliency design:

• The circuit itself dominates and has a monthly floor regardless of traffic — you pay for the circuit’s bandwidth tier whether it carries a byte or not. MeteredData adds per-GB egress on top; UnlimitedData folds egress into a higher flat fee. Model your monthly egress against the break-even before choosing family — and remember MeteredData → UnlimitedData is a one-way change.
• Premium is an add-on on top of the circuit, buying cross-geo reach and the 10,000-route limit. Pay for it only if you need cross-geo VNet reach or more than 4,000 routes — otherwise Standard is the honest floor.
• Dual circuits for maximum resiliency roughly double the circuit spend — the price of removing the facility failure domain. ExpressRoute Metro gives location diversity on one circuit’s bill where it is offered, a materially cheaper path to facility diversity than two full circuits.
• Gateways bill by the hour — both the ER gateway and the VPN backup gateway run continuously. A right-sized backup (VpnGw3AZ for a ~1 Gbps circuit) is cheap insurance versus the revenue/penalty cost of a failover that collapses; do not undersize to save a few thousand rupees a month.
• Global Reach adds an enablement fee plus egress on the DC-to-DC traffic — compare it to your existing WAN spend for site-to-site, where it often wins by decoupling you from a WAN provider.

A rough monthly picture for a mid-size production hybrid setup in India, and what each line buys:

The sizing rule in prose: start with one Standard MeteredData circuit sized with bandwidth headroom (you can grow it, not shrink it) plus a capacity-matched active-active VPN backup; add Premium only for cross-geo reach or >4,000 routes; and for mission-critical workloads add a second circuit at a different location (or Metro) to remove the facility failure domain. The most expensive thing you can do is not size the backup — a failover that survives routing but collapses under load costs far more in one incident than the SKU upgrade costs in a year.

Interview & exam questions

1. A circuit is described as “redundant by default.” What does that actually protect against, and what does it not? Every circuit is provisioned with two connections to two MSEEs (primary + secondary), so it survives the loss of one MSEE or one link within the peering location. It does not protect against the peering location itself failing (both links share that facility), your provider’s backbone, your edge router, or a BGP misconfiguration. True facility diversity requires a second circuit at a different peering location (or ExpressRoute Metro).

2. You need ExpressRoute preferred and a VPN backup that only carries traffic on failure. Name the knob for each direction. Two independent levers. Azure → on-prem (return path): set the ExpressRoute connection routing weight higher than the VPN connection’s (higher wins). On-prem → Azure (forward path): on your edge, give ExpressRoute-learned routes higher local-preference (and/or weight) than VPN-learned routes, and AS-path-prepend the VPN advertisements so Azure also sees the VPN path as longer. Tune both or you get asymmetric routing.

3. Why does advertising a summary over ExpressRoute and specifics over the VPN break your failover? BGP’s first decision is longest-prefix-match, evaluated before any attribute. A more-specific /24 over the VPN beats a /16 summary over ExpressRoute for any address in that /24, so Azure sends that traffic down the VPN even while ExpressRoute is up and preferred by every attribute you set. The fix is to advertise the same prefixes over both paths.

4. What is the private-peering route limit, what happens if you exceed it, and how do you avoid it? 4,000 IPv4 routes on Standard, 10,000 on Premium. Exceeding it drops the BGP session — it does not degrade, it drops — taking the whole peering down. The usual cause is redistributing the entire on-prem table; the fix is to advertise summarized supernets (one 10.50.0.0/16 instead of 256 /24s) and, if you legitimately need more, move to Premium.

5. Your failover takes about three minutes and misses the RTO. What’s happening and how do you fix it? With default BGP timers, a session that is stuck-but-up isn’t declared dead until the hold timer expires — up to ~180 seconds. Enable BFD (Bidirectional Forwarding Detection) on the ExpressRoute BGP neighbours; it detects link failure in sub-second time and tears the session immediately, collapsing convergence to a second or two. Microsoft supports BFD on private peering.

6. What is FastPath, what does it require, and what does it bypass? FastPath programs the MSEE to send data-plane traffic directly to VMs, bypassing the ExpressRoute gateway (control-plane/BGP still flows through the gateway). It requires a high-capacity gateway (ErGw3Az/scalable), and advanced scenarios (Private Endpoint, spokes over peering) need enablement and typically Premium. Critically, it bypasses any UDR that forces traffic through an NVA/firewall — so do not FastPath subnets whose traffic must be inspected.

7. What is ExpressRoute Global Reach and when do you use it? Global Reach connects two of your on-prem sites to each other over the Microsoft backbone using their circuits’ private peering as transit — the sites can reach each other without your own WAN or a VNet hairpin. Use it to decouple site-to-site connectivity from a WAN provider or to connect branches across far regions privately. The on-prem prefixes must not overlap, and it is an add-on charge.

8. What is a route filter, which peering does it apply to, and what happens without one? A route filter is an allow-list of BGP community values that scopes which Microsoft public prefixes are advertised. It applies to Microsoft peering only (private peering is controlled by what your routers advertise/accept). Without an attached route filter, a Microsoft peering advertises nothing — the filter is what activates advertisement. Scope it to only the service/region communities you consume.

9. Difference between the two ExpressRoute SKU families, and can you switch? The family is the billing model: MeteredData (lower flat fee + per-GB egress) vs UnlimitedData (higher flat fee, no per-GB egress). Ingress is always free. You can move MeteredData → UnlimitedData in place, but not back — Azure blocks the downgrade — so start metered unless you already know you are a heavy-egress shop.

10. How do you test ExpressRoute failover properly, and what do most teams get wrong? In a window, administratively shut both ExpressRoute BGP sessions on your edge, confirm the effective route table flips to the VPN (VirtualNetworkGateway), measure the convergence gap with a continuous ping, and then drive production-representative load (iperf3) across the backup — before restoring. Most teams only test reachability (a ping succeeds) and never capacity, so an undersized backup passes every drill and collapses under real load.

11. Why alert on per-session BGP availability instead of aggregate circuit state? Because the most common ExpressRoute outage is a silent single-session failure: one of the two BGP sessions drops, aggregate circuit state still looks fine, the circuit runs non-redundant for weeks, and then the surviving MSEE goes into planned maintenance and everything fails at once. BgpAvailability < 100 per peering catches the single-session condition immediately.

12. Which BGP attribute governs which direction, and in what order are they evaluated? In evaluation order: weight (Cisco-proprietary, per-router, first) and local-preference (across your AS, second) are set by you and govern the forward path (on-prem→Azure); AS-path length (fourth) and MED (sixth) are what Azure sees among the routes you advertise and govern the return path (Azure→on-prem). Longest-prefix-match precedes all of them. Getting failover right means setting the correct knob for each direction.

These map to AZ-700 (Network Engineer Associate) — design and implement hybrid connectivity, ExpressRoute, VPN, BGP — and to AZ-305 (Solutions Architect) — design network connectivity and resilience. The BGP and failover specifics also touch AZ-104 for gateway configuration. A compact cert-mapping for revision:

Quick check

1. Your circuit is “redundant by default,” yet a Microsoft-announced MSEE maintenance took it fully down. What single condition made that possible, and what alert would have caught it?
2. ExpressRoute is up, but Azure keeps returning traffic to on-prem over the VPN. Name the one Azure-side setting you check first, and what it should be relative to the VPN.
3. True or false: advertising a /16 summary over ExpressRoute and the constituent /24s over the VPN is a good way to keep the VPN idle.
4. Your failover takes ~180 seconds and misses the RTO. What feature reduces this to sub-second, and where do you enable it?
5. You enabled FastPath and now a subnet’s traffic no longer passes through your firewall. Why, and what are your two options?

Answers

1. Exactly one of the two BGP sessions was silently down (a router flap or change nobody alerted on), so the circuit ran non-redundant; when the surviving MSEE went into maintenance, there was no second session to carry traffic. The catch: alert on per-session BgpAvailability < 100 (checking --path primary and --path secondary), not on aggregate circuit state.
2. The ExpressRoute connection routing weight — it must be higher than the VPN connection’s routing weight (higher wins when Azure chooses between gateways for the return path). Set ER to e.g. 20000 and VPN to 100. (Also verify on-prem local-pref for the forward direction.)
3. False. Longest-prefix-match is evaluated before every attribute, so Azure sends the /24 traffic down the VPN even while ER is up and preferred — the VPN carries production, not idle. Advertise the same prefixes over both paths.
4. BFD (Bidirectional Forwarding Detection) — it detects link failure in sub-second time and tears the BGP session immediately instead of waiting for the hold timer. Enable it on the ExpressRoute BGP neighbours on your edge routers (Microsoft supports BFD on private peering).
5. FastPath bypasses the gateway and any UDR that forces traffic through an NVA/firewall, so the inspection hop is skipped. Options: (a) don’t FastPath the subnets that require inspection, accepting the gateway in the path for them; or (b) move inspection to a point FastPath doesn’t bypass (e.g. on-prem, or a design that doesn’t rely on a UDR-forced NVA in the Azure path).

Glossary

• Circuit — the logical Azure resource representing connectivity through a provider; has an s-key, bandwidth, SKU family/tier, and is always dual-homed to two MSEEs.
• Service key (s-key) — the GUID your provider uses to map your physical cross-connect to the circuit; billing starts when the circuit reaches Provisioned.
• Peering location — the physical meet-me facility where Microsoft’s edge and your provider interconnect; distinct from an Azure region and the deepest single failure domain on one circuit.
• Private peering — the BGP routing domain carrying traffic to your VNets (RFC1918 and beyond); controlled by what your routers advertise/accept.
• Microsoft peering — the BGP routing domain to Microsoft public endpoints (M365, public PaaS) over your public IPs; advertises nothing until a route filter is attached.
• MSEE (Microsoft Enterprise Edge) — Microsoft’s edge routers, ASN 12076; every circuit peers with two of them (primary + secondary).
• Connection (primary/secondary) — the two links from your edge to the two MSEEs; both must be Established for the circuit to be truly redundant.
• ExpressRoute gateway — a VNet gateway of an ErGw* SKU (not a VPN gateway) that connects the circuit to a VNet; a data-path throughput point unless bypassed by FastPath.
• Connection routing weight — an Azure-side preference on a connection; higher wins when Azure chooses among gateways (used to prefer ER over VPN outbound from Azure).
• Weight — a Cisco-proprietary, router-local BGP attribute evaluated first; used for per-device path preference on your edge.
• Local-preference — a standard BGP attribute propagated across your AS, evaluated early; used AS-wide to prefer ExpressRoute over VPN on the forward path.
• AS-path prepend — repeating your ASN on an advertisement to make the path longer and less preferred; used to worsen the VPN path so Azure prefers ExpressRoute on the return path.
• MED (Multi-Exit Discriminator) — a hint to a neighbouring AS about which entry point to prefer (lower wins); used to keep return traffic symmetric across dual circuits.
• Longest-prefix-match (LPM) — BGP’s first decision: a more-specific prefix always wins over a less-specific one, before any attribute; the source of the “backup carrying production” trap.
• FastPath — sends data-plane traffic directly from the MSEE to VMs, bypassing the ExpressRoute gateway (and any UDR-forced NVA); needs a high-capacity gateway.
• Route filter — an allow-list of BGP community values scoping which Microsoft public prefixes a Microsoft peering advertises.
• Route limit — the maximum IPv4 routes a private peering accepts (4,000 Standard / 10,000 Premium); exceeding it drops the BGP session.
• Global Reach — an ExpressRoute feature connecting two of your on-prem sites to each other over the Microsoft backbone via their circuits’ private peering.
• ExpressRoute Metro — a configuration where a single circuit’s two links terminate in two peering locations within one metro, giving facility diversity on one bill.
• BFD (Bidirectional Forwarding Detection) — sub-second link-failure detection that tears a BGP session immediately, collapsing failover convergence from up to ~180 s to seconds.
• Connection Monitor — a Network Watcher capability running continuous end-to-end synthetic probes (reachability, latency, loss, path) to prove the data plane works, not just BGP.
• SKU family — the billing model: MeteredData (per-GB egress) vs UnlimitedData (flat, no per-GB); switchable metered→unlimited only.
• SKU tier — Standard (same-geo reach, 4,000 routes) vs Premium (any-geo reach, 10,000 routes, more VNets).

Next steps

You can now design an ExpressRoute topology to a stated availability target and defend every routing knob. Build outward:

• Prerequisite refresher: ExpressRoute Fundamentals: Circuits, Private vs Microsoft Peering and Connectivity Models — the circuit/peering/model basics this article assumes.
• Next: Production Site-to-Site VPN to Azure: Active-Active Gateways with BGP — build the capacity-shaped VPN backup this design depends on, properly.
• Related: Diagnosing Connectivity with Network Watcher: Connection Monitor, Connection Troubleshoot and Next Hop — go deep on the Connection Monitor that proves your failover end-to-end.
• Related: Hub-and-Spoke vs Virtual WAN: Choosing an Enterprise Cloud Network Topology — where the circuit terminates in your enterprise topology.
• Related: Azure Multi-Region Active-Active Architecture: Designing for Zero-Downtime — ExpressRoute resiliency as one pillar of a multi-region design.
• Related: Azure Private Link with Private DNS for PaaS — move PaaS access to Private Link over private peering instead of Microsoft peering: simpler routing, no public IPs, no route filters.