HA for core routers: Securely configure VRRP, HSRP and GLBP

When the default gateway becomes unstable, the entire subnet feels it immediately: sessions drop, ARP entries jump, and troubleshooting becomes a continuous task. For HA for core routers, VRRP (Virtual Router Redundancy Protocol), HSRP (Hot Standby Router Protocol) and GLBP (Gateway Load Balancing Protocol) are the usual tools. This article explains in practical terms how to design, configure and operate these protocols so that Split‑Brain scenarios are avoided — situations in which multiple routers believe they are active at the same time.

HA for Core Routers: Role of VRRP, HSRP and GLBP

VRRP, HSRP and GLBP provide a virtual default gateway (VIP). One router answers for that address and forwards traffic, another remains in standby. Important note: FHRP (First Hop Redundancy Protocol) only covers the first hop; routing redundancy in the backbone, stable L2 paths and clean control‑plane policies are necessary prerequisites.

When to use which protocol?

• VRRP is standardized (RFC) and is often the first choice in multi‑vendor environments.
• HSRP is proprietary to Cisco and offers tight integration with Cisco tracking/monitoring.
• GLBP distributes gateway load via multiple virtual MACs, but increases complexity (ARP behavior, security integration).

What Split‑Brain really entails and how it manifests

Split‑Brain in FHRP is usually not a pure protocol anomaly, but the result of misdirected signaling, L2 partitions or unsuitable timings. Operational symptoms:

• ARP/MAC flapping: virtual MACs jump between ports.
• Asymmetric routing: packets go out but return via a different path.
• Intermittent packet loss or long connection setup times after failover.

Typical causes

• Control packets are blocked by ACLs, storm control or multicast limits.
• L2 partitions or inconsistent trunk/VLAN configurations.
• Preemption without delay during routing convergence.
• Tracking that only checks link up/down but does not verify actual forwarding (blackholing).

Topology prerequisites: foundation for stable failover

Before you fine‑tune timers and priorities, secure the foundation:

• FHRP participants must be in the same VLAN/subnet (SVI/L3‑Interface).
• Consistent trunks/port‑channels and VLAN allowed lists between core and access.
• STP root plan: deliberately set root bridges and edge ports (PortFast/Edge for end devices).
• Separate management/keepalive path as a secondary liveness channel, where possible.

Configuration principles: priority, preemption, timers, tracking

Stability is created by clear rules. Three core elements:

1. Clear priorities

Avoid ties. Define a preferred master per VLAN and give the backup a clearly lower priority. That reduces race conditions.

2. Preemption with delay

Preemption (the better router takes over upon return) is useful, but only with delay. Otherwise flaps can occur on restart while routing/port‑channels have not converged. A preempt delay gives the system time to stabilize IGP adjacencies, BGP and LACP.

3. Tracking for real reachability

Interface tracking is the baseline; extend it with route tracking (default route/IGP neighbor) and IP SLA (active measurement) against a stable next hop. Important: tracking targets must not themselves depend on the FHRP being checked — otherwise a feedback loop and a possible split‑brain trigger will occur.

Practical configuration examples (guidance)

Commands are vendor-dependent; here are structured examples in Cisco syntax showing how preempt delay, priority and tracking are combined.

VRRP – Preempt‑Delay and Tracking (example)

interface Vlan10
 ip address 10.10.10.2 255.255.255.0
 vrrp 10 ip 10.10.10.1
 vrrp 10 priority 120
 vrrp 10 preempt delay minimum 60
 vrrp 10 track interface Port-Channel1 decrement 40
 vrrp 10 track route 0.0.0.0/0 decrement 30

HSRP – Active/Standby with IP SLA tracking (example)

ip sla 10
 icmp-echo 8.8.8.8 source-ip 10.10.10.2
 frequency 10
ip sla schedule 10 life forever start-time now
track 10 ip sla 10 reachability

interface Vlan10
 ip address 10.10.10.3 255.255.255.0
 standby 10 ip 10.10.10.1
 standby 10 priority 110
 standby 10 preempt delay minimum 60
 standby 10 track 10 decrement 30

GLBP – only when gateway load distribution is required

interface Vlan10
 ip address 10.10.10.4 255.255.255.0
 glbp 10 ip 10.10.10.1
 glbp 10 priority 120
 glbp 10 preempt delay minimum 60
 glbp 10 load-balancing round-robin
 glbp 10 track Port-Channel1 decrement 30

IP SLA, BFD and failover acceleration

FHRP timers alone are often not the right lever for fast, reliable failover. Two complementary mechanisms are very effective in practice:

IP SLA (active reachability test)

IP SLA performs active measurements (ICMP/TCP/UDP) against a stable target host. Use IP SLA as a track object for FHRP so that actual upstream reachability, not a link-down, determines state. Pay attention to target selection: use a next hop in the carrier network or a cloud peer that is not reached via the gateway under test.

BFD (Bidirectional Forwarding Detection)

BFD is a very fast liveness mechanism that runs between routing peers or over tunnels. BFD detects forwarding failures in milliseconds and can resolve IGP adjacencies faster; consequently, FHRP tracking should include route/IGP state rather than only interface status.

! Beispiel für einfache BFD Template und Interface‑Aktivierung
bfd-template single-hop BFD_FAST
 interval 50 min_rx 50 multiplier 3
!
interface GigabitEthernet0/0
 ip address 192.0.2.2 255.255.255.252
 bfd interval 50 min_rx 50 multiplier 3

FHRP over VPNs, MPLS or remote sites

FHRP over WAN/provider networks is a common source of failures. FHRP is intended for LAN-segment redundancy; across L3 networks it is easy to create split‑brain situations because control‑plane packets and data paths can diverge.

Best practices for multi-site

• Apply FHRP only locally; for site redundancy use routing (BGP/OSPF) and anycast.
• If FHRP is built over a shared L2 (e.g. L2‑MPLS), ensure that control packets use the same path as client traffic.
• For IPsec/GRE tunnels: avoid that IP‑SLA targets or tracking targets themselves traverse the tracked gateway.

Diagnostic practice: order of checks during failures

A fixed verification sequence prevents wasted time. Objective: first determine whether FHRP is the cause or a symptom. Complementing the basic checks in the base version, here are extended checks and packet-capture examples.

Step 1 – Check FHRP status

show vrrp brief
show standby brief
show glbp brief
show logging | include VRRP|HSRP|GLBP|STATE|TRACK

Step 2 – MAC/ARP view

show mac address-table vlan 10 | include 0000.5e00
show mac address-table move update
show ip arp | include 10.10.10.1

Step 3 – Check control traffic and ACLs

show interface Vlan10 counters
show ip igmp snooping groups vlan 10
show access-lists | include 112|HSRP|GLBP

Ensure that FHRP control packets are not discarded by ACLs, QoS, or storm control. VRRP uses IP protocol 112; tcpdump can help to observe control packets.

# Beispiel: VRRP‑Pakete mit tcpdump auffangen
tcpdump -nni eth0 'ip[9] == 112' -vv

Step 4 – Check routing/forwarding

show ip route 0.0.0.0/0
show ip ospf neighbor
show bgp summary
ping <upstream-next-hop> source <SVI-IP>

A master can report an Up state but have upstream blackholing — this indicates that tracking is insufficient.

Step 5 – Client perspective

On clients, check the ARP entry, traceroute (first hop), and session persistence. Firewalls and uRPF can drop connections in cases of asymmetric routing.

Test cases for controlled changes

Perform failover tests in controlled steps and document behavior and metrics. Recommended test cases:

1. Simulated uplink failure on the master (link-down on the uplink).
2. Interface shutdown on the master (checks tracking response).
3. Preemption reinsertion: reboot the master and observe whether preempt delay prevents flapping.
4. IP SLA target unreachable: verify whether the track object triggers the FHRP.
5. MAC flap stress test: repeated up/down on access ports and observation of the flap rate.
6. VPN failover: carrier scenario with BGP failover and checking for asymmetric paths.

Operational Runbook – Short checklist

• Is the FHRP VLAN consistent across all participating switches?
• Who is the intended master per VLAN (priority documentation)?
• Are IP SLA targets independent and reachable?
• Are control-plane packets (e.g. VRRP) allowed in ACLs?
• Are there alerts for MAC flaps, FHRP state changes, and IP SLA losses?

Specific security aspects

Security features such as Dynamic ARP Inspection (DAI), IP Source Guard, or RA Guard can disrupt failover if trust ports are not configured correctly. Ensure that access switches allow the GARP/NA/Gratuitous‑ARP of the gateways and that ACLs do not filter control traffic.

Conclusion and recommendations

HA for core routers is more a matter of system design than mere configuration: clear L2 paths, prioritized master rules, preemption with delay, tracking based on actual reachability, and meaningful monitoring prevent most split‑brain cases. Test controlled failover scenarios, document dependencies (VLAN, trunk, tracking targets), and keep simple fallback options available. In multi‑site or VPN scenarios prefer local FHRP instances and rely on routing/anycast for site redundancy. This makes gateway redundancy reliable and predictable in operation.

Additional checks and notes

If you encounter persistent issues, a sequential isolation is recommended: first fully validate L2, then FHRP control, then routing and finally application-layer symptoms. Structured documentation of all failover tests and the accepted baselines minimizes „War‑Room“ deployments and accelerates Incident‑Resolution.

Operation, monitoring and emergency procedures for HA for Core‑Router

In addition to configuration, daily operation is critical: monitoring, data backup, change control and clearly defined emergency procedures significantly reduce downtime. This section provides practical guidance on which metrics you should collect, how to initiate rapid countermeasures and how automation increases reliability.

Which metrics really matter

• FHRP‑state changes per minute: sudden increases indicate flapping or preempt issues.
• MAC‑table flaps and number of distinct ports per virtual MAC: early indicator of L2 partitions.
• IP SLA reachability losses and BFD session resets: indicate real forwarding failures.
• Control‑plane drops (ACL/QoS/CP‑policing): when a router is CPU‑bound, signaling packets are lost.

Monitoring and alert rules (recommended)

• Alert when FHRP‑state changes > 3 in 5 minutes.
• Warning for MAC‑flap rate > X per minute (value depends on environment).
• Critical when IP SLA‑loss > 5% over 1 minute or BFD‑down.

Quick emergency measures (runbook excerpt)

1. Isolate: map affected VLAN trunks and, if necessary, set ports temporarily to „errdisable“ to stop flapping.
2. Stabilize: temporarily disable preemption or increase preempt delay until upstream adjacencies are established.
3. ARP refresh: allow gratuitous ARP on edge switches and, if necessary, clear clients‘ ARP caches.
4. Fallback: set master priority manually (with documented reversion) if automatic failover is not operating reliably.

Examples of quick operator commands:

# Configuration backup from the router (via SCP/SSH)
scp admin@10.0.0.1:/running-config ./backup/10.0.0.1.cfg

# Filter syslog for FHRP events
ssh syslogserver 'grep -E "VRRP|HSRP|GLBP|STATE|TRACK" /var/log/syslog | tail -n 200'

# Clear client ARP on Linux
sudo ip neigh flush all

Automation and change management

Keep FHRP configurations in a Git repository, validate changes via CI (linting, syntactic validation of templates) and run scheduled tests in a lab emulator (GNS3、EVE‑NG). Versioned backups allow rapid rollback after faulty changes.

Long-term measures

Conduct regular chaos tests (controlled link‑downs, IP‑SLA target unreachability) and document metrics as well as acceptable baselines. This lets you detect gradual degradation before users are affected. Documentation, automated backups and a clear emergency path make HA for Core‑Router truly resilient in operation.

For this topic, Configuring VRRP and Configuring HSRP are also important. The article places these aspects in a clear context and shows what matters in day‑to‑day operations.