Step-by-step: Set up a highly available NGINX load balancer with Keepalived and HAProxy

A highly available NGINX load balancer eliminates the single point of failure at the network edge by having two Linux nodes provide a virtual IP (VIP) via VRRP and run NGINX and HAProxy locally for TLS, routing and health checks. In this practical how-to I describe not only configurations but also operational implications, common failure causes, measurement and verification commands, and a practical rollback strategy.

Highly available NGINX load balancer: architecture and division of responsibilities

In short: Keepalived (VRRP) provides a VIP that clients address. The active node has the VIP bound locally and responds to ARP. NGINX handles TLS termination, header policy and redirects; HAProxy runs locally and performs precise health checks and backend routing. This separation enables clear responsibility areas: TLS and security policy in NGINX, state and performance control in HAProxy.

Prerequisites, network and design decisions

Key assumptions: both load balancers are in the same Layer-2 domain (VLAN), the VIP is within the same subnet, and VRRP traffic (IP protocol 112) is allowed between the hosts. If applications store session state locally, plan for sticky sessions or a central session store (e.g., Redis) — otherwise failovers will lead to lost session information.

Example parameters

• LB1: 10.10.10.11/24, LB2: 10.10.10.12/24, VIP: 10.10.10.10/24
• Interface: eth0, Backends: 10.10.20.21:8080, 10.10.20.22:8080
• DNS: an A record pointing to the VIP; TTL is secondary, VIP failover is transparent to clients

Installation: packages, services, order

Install NGINX, HAProxy and Keepalived on both nodes. Enable and start the services after configuration checks.

sudo apt-get update
sudo apt-get install -y nginx haproxy keepalived curl iproute2 iputils-arping

sudo systemctl enable --now nginx
sudo systemctl enable --now haproxy
sudo systemctl enable --now keepalived

Kernel and ARP tuning: why this is important

If Linux responds incorrectly to ARP requests, traffic blackholing or split-brain can occur. The following sysctl settings reduce unwanted ARP reply behavior; apply these on both nodes.

sudo tee /etc/sysctl.d/99-lb-ha.conf >/dev/null <<'EOF'
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
EOF

sudo sysctl --system

Explanation: arp_ignore controls whether the system answers ARP requests for IPs not configured locally; arp_announce influences which source IP is used in ARP requests. Incorrect settings allow the backup to answer ARP for the VIP and cause split-brain.

HAProxy: local backend layer, health checks and metrics

HAProxy provides robust health checks (HTTP, TCP, SSL), weight and rise/fall settings. Use a local bind (127.0.0.1) so NGINX can forward internally. Statistics can be used for monitoring or exported via a Prometheus exporter.

# /etc/haproxy/haproxy.cfg
global
  log /dev/log local0
  tune.maxaccept 1000
  maxconn 50000
  daemon

defaults
  mode http
  option httplog
  timeout connect 5s
  timeout client 60s
  timeout server 60s

frontend fe_local
  bind 127.0.0.1:9000
  default_backend be_app

backend be_app
  balance roundrobin
  option httpchk GET /healthz
  http-check expect status 200
  default-server inter 2s fall 3 rise 2
  server app1 10.10.20.21:8080 check
  server app2 10.10.20.22:8080 check

listen stats
  bind 127.0.0.1:8404
  stats enable
  stats uri /stats

Verify HAProxy before restarting:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

NGINX: TLS, proxy paths and timeouts

NGINX performs TLS termination. Automate certificates (e.g. certbot/ACME) or use your internal PKI. Pay attention to proxy_read_timeout and buffer settings so that long backend responses do not block the connection.

# /etc/nginx/sites-available/lb.conf
server {
  listen 80;
  server_name _;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  server_name _;
  ssl_certificate     /etc/ssl/certs/lb.pem;
  ssl_certificate_key /etc/ssl/private/lb.key;
  client_max_body_size 50m;
  proxy_read_timeout 120s;
  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://127.0.0.1:9000;
  }
}

Keepalived: VRRP configuration, track script and timers

Keepalived determines which node holds the VIP. Track scripts reduce the priority in case of service failures so that a healthy backup node can take over. The advert_int and priority values control switchover time and priority order.

Track script with exit codes

sudo tee /usr/local/sbin/chk_proxy.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Prüft Dienste und lokales HAProxy Health-Endpoint
systemctl is-active --quiet nginx || exit 1
systemctl is-active --quiet haproxy || exit 1
curl -fsS --max-time 1 http://127.0.0.1:9000/healthz >/dev/null || exit 1
exit 0
EOF
sudo chmod 0755 /usr/local/sbin/chk_proxy.sh

Keepalived configuration (Master/Backup)

# /etc/keepalived/keepalived.conf (Beispiel MASTER)
vrrp_script chk_proxy {
  script "/usr/local/sbin/chk_proxy.sh"
  interval 2
  timeout 2
  fall 2
  rise 2
  weight -30
}

vrrp_instance VI_10 {
  state MASTER
  interface eth0
  virtual_router_id 10
  priority 110
  advert_int 1
  authentication { auth_type PASS; auth_pass 7f3c9d2a }
  virtual_ipaddress { 10.10.10.10/24 }
  track_script { chk_proxy }
}

# Backup hat priority 100 und state BACKUP

Note: Authentication in Keepalived (auth_pass) is a simple mechanism; in untrusted networks you should use additional network segmentation, because VRRP itself does not provide strong cryptography.

Common causes: ARP, switch features, firewalls

Common causes of unexpected behavior:

• Switch features like Dynamic ARP Inspection (DAI) block Gratuitous ARP after a failover — in such networks switch configuration is necessary.
• Host firewall or cloud security groups block IP protocol 112 (VRRP) — check this with tcpdump.
• Conntrack (Stateful NAT/Firewall): With NAT translation a failover can interrupt existing connections because the NAT table on the new node is missing.

Diagnostics with tcpdump, arping and logs

Useful checks for analysis:

# VRRP traffic beobachten (Protocol 112)
sudo tcpdump -n -i eth0 proto 112

# ARP prüfen
sudo tcpdump -n -i eth0 arp

# GARP senden (nach Failover)
sudo arping -U -I eth0 -c 5 10.10.10.10

# Keepalived logs
sudo journalctl -u keepalived -f

If tcpdump shows no VRRP packets, a firewall or an intervening switch is often the cause. If the master’s GARP does not get through, clients retain the old MAC mapping in ARP tables and need to relearn.

Conntrack and NAT effects during failover

In setups with NAT or stateful firewalls the conntrack table is host-specific. After a VIP failover, established conntrack entries are missing on the new node, which leads to dropped sessions. Possible measures:

• Run Keepalived with conntrack sync (e.g. conntrackd) — increases complexity.
• Tolerate connection renewal: accept short reconnects, let clients reconnect.
• Consider an active/active architecture if session continuity is critical.

Security aspects and certificate management

TLS certificates must be identical on both LBs. Automate deployments with Certbot (ACME) or your internal PKI and validate private keys with file permissions. Store certificates versioned in a secure artifact repo or secret store.

# Beispiel Certbot (Let's Encrypt) - nicht für private PKI
sudo apt-get install -y certbot
sudo certbot certonly --standalone -d lb.example.com
# Verteilen Sie das resultierende PEM sicher auf beide Nodes (scp/Ansible)

Rollout and patch process — concrete runbook

A typical, tested procedure for patching or configuration updates minimizes risk:

1. Plan: communicate the maintenance window, temporarily suppress monitoring alerts.
2. Move traffic to the backup: either stop Keepalived on the master or lower its priority.
3. Validation: verify on the backup that the VIP has been taken over and latency/errors are within normal bounds.
4. Patch the master, test (check NGINX/HAProxy locally), and switch it back into the pool.
5. Patch the second node.
6. Post-checks: failover test, health checks, log analysis.

# Beispiel: Traffic auf Backup bringen
# Auf Master:
sudo systemctl stop keepalived
# Auf Backup prüfen:
ip -br addr show dev eth0
sudo curl -I --resolve lb.example.com:443:10.10.10.10 https://lb.example.com/
# Nach Patch:
sudo systemctl start keepalived

Monitoring, metrics and alerts

Important metrics you should monitor:

• Keepalived VRRP transitions (frequent switchovers indicate instability)
• VIP binding time and GARP events
• NGINX 4xx/5xx rates
• HAProxy backend status, response time and queueing
• System metrics: file descriptors, netstat for open sockets, loadavg

Export HAProxy stats via an exporter for Prometheus or use the internal stats endpoint for alerting rules.

Advanced design alternatives

If you need higher scale or global availability, consider other patterns:

• Public cloud: managed load balancer avoids ARP/VRRP issues.
• Anycast/BGP: for global distribution and lower latency; requires network routing expertise.
• Active/active: both LBs carry traffic; requires session synchronization or stateless applications.

Checklist before production deployment

• Is VRRP (IP 112) allowed between LBs and through firewall/switches in the network?
• Are sysctl‑ARP settings applied and loaded on both nodes?
• Does the Keepalived track script work and are exit codes valid?
• Do HAProxy health checks return the expected responses (200/OK)?
• Are TLS certificates installed and synchronized on both nodes?
• Have switch features (DAI, Port Security) been checked and adjusted if necessary?
• Is monitoring and alerting for VRRP transitions and backend health configured?
• Rollback plan tested (manual ip addr add, Keepalived start/stop)?

Typical pitfalls and quick mitigations

Split brain: Check VRRP packets, auth tokens, virtual_router_id and priorities. If switches have DAI enabled, configure appropriate DHCP/ARP bindings or whitelist the LB MACs.

Slow failover times: advert_int too large, ARP caching on clients/switches, or GARP suppressed. Reduce advert_int and test GARP transmission behavior, but be aware that very short intervals generate more VRRP traffic.

Conclusion and operational relevance

A highly available NGINX load balancer with Keepalived and HAProxy is a pragmatic solution to reduce outage risk at the network edge. Critical are not only correct configuration files but also network and switch settings, ARP behavior, robust health checks and a tested rollout process. Invest time in monitoring, documented runbooks and regular failover tests — that ensures predictable operational readiness.

Further check commands (Quick Reference)

# Who has the VIP?
ip -br addr show dev eth0 | grep 10.10.10.10 || true

# Observe VRRP traffic live
sudo tcpdump -n -i eth0 proto 112

# Check if Keepalived track script is running and returns an exit code
/usr/local/sbin/chk_proxy.sh && echo OK || echo FAIL

# Send GARP (Master)
sudo arping -U -I eth0 -c 5 10.10.10.10

# HAProxy stats (local)
curl -sS http://127.0.0.1:8404/stats

Operational reliability: configuration management, testing and rapid recovery

After technical commissioning, change management determines long-term stability. Store all Keepalived, NGINX and HAProxy configurations in a Git repo, version templates and produce idempotent artifacts from automated CI pipelines. This way rollbacks via commit are traceable and configuration drifts can be avoided.

Practical additions:

• Canary rollout: Deploy config changes to the backup node first, verify health checks and logs, then to the master.
• Rapid recovery: Documented steps for manually setting the VIP, emergency SSH access and restoring Git configurations minimize downtime.
• Advanced fail detection: Use BFD (Bidirectional Forwarding Detection) in addition to VRRP for faster detection of link failures, especially under critical latency requirements.
• Network segmentation: Carry VRRP signaling over a dedicated HA‑VLAN/VRF to reduce attack surface — VRRP does not provide strong cryptography.
• Monitoring integration: Push VRRP states, config hashes and Config‑Change events into your central alerting/ticketing so configuration changes are immediately visible.

These operational building blocks reduce human error, speed up recovery and make LB operation reproducible and auditable — crucial for process-oriented enterprise solutions with high availability requirements.

For this topic, Haproxy high availability and load balancing at Layer 4 and Layer 7 are also important. The article places these aspects in context and shows what matters in day-to-day operations.