Diagnosing Asymmetric Routing: Tools, Diagnostic Procedure and Countermeasures

Asymmetric Routing is not an exception in modern networks with Multi‑Homing, load balancers and container orchestration, but a plausible operational scenario. It becomes problematic whenever stateful devices (stateful Firewalls, NAT/Conntrack, load balancers with session logic) sit on the path and expect the return path. This article provides a proven, minimally invasive test sequence, explains the most important tools and shows practical countermeasures — specifically also for Kubernetes environments.

Why Asymmetric Routing disrupts operation

IP‑routing allows different forward and return paths. That is often desired (load distribution, redundancy). However, if a stateful component on one of the paths interferes, this leads to packet loss or terminated connections. Important terms briefly explained: Stateful Firewall checks session state, NAT (Network Address Translation) changes source/destination and requires consistent tables, Conntrack is the Linux‑kernel mechanism for connection tracking, rp_filter (Reverse Path Filter) drops packets whose return route is not plausible.

Early indicators: when to check for asymmetry immediately

Before making changes, check for symptoms that are particularly indicative:

• SYN from the client arrives at the server, but the SYN/ACK sent by the server does not reach the client.
• Connection drops only from certain source networks or over specific carriers.
• Different behavior between UDP and TCP (UDP appears more resilient, since stateful devices typically do not inspect it).
• Problems only at certain MTU values / with large transfers (PMTUD/ICMP affected).

Test sequence: structured, synchronized, minimally invasive

Work according to the principle: observe first, then change. Typical order:

Scope and reproduction plan

Define source, destination, protocol, port, time and affected nodes (for Kubernetes: Namespace, Service, Pod, Node). Set a time window to run captures synchronously.

Traceroute and mtr in protocol mode

ICMP routes can differ from the TCP/UDP path. Use traceroute/mtr with TCP to map the path for your service.

mtr -T -P 443 -r -c 20 
traceroute -T -p 443 

Check host routing and policy

On Linux ip route get and ip rule show which table and interface are used for replies. PBR (Policy‑Based Routing) uses rules (ip rule) and additional routing tables; VRFs fully isolate tables.

ip route get 
ip route get  from 
ip rule show
ip route show table all

Explicitly check rp_filter

Reverse Path Filtering can drop legitimate Multi‑Homing traffic. Check global and per‑interface settings.

sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf..rp_filter

For persistent changes create a file in /etc/sysctl.d/:

# /etc/sysctl.d/99-rpfilter.conf
net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.default.rp_filter=2
net.ipv4.conf.eth0.rp_filter=2

Bidirectional tcpdump captures: the gold standard

Synchronized captures on the client, affected routers/firewalls, server or Kubernetes nodes produce the evidence. Filter on the 5‑tuple (Src IP, Dst IP, Src Port, Dst Port, Proto) to obtain unique matching results.

# Client (oder Edge)
sudo tcpdump -ni any host  and tcp port 443 -w /tmp/cap-client.pcap

# Server (oder Node/POD)
sudo tcpdump -ni any host  and tcp port 443 -w /tmp/cap-server.pcap

Timing is crucial: if SYN arrives at the server and SYN/ACK leaves the server but is not visible in the client capture, that indicates a return-path problem.

Conntrack and state diagnosis

Conntrack manages connection states in the kernel. Invalid or missing entries indicate missing NAT/state handling.

sudo conntrack -L -p tcp --dport 443 | tail -n 20
sudo conntrack -S   # Statistik
cat /proc/net/stat/nf_conntrack

Interpretation: An entry with status UNREPLIED or entries that disappear quickly indicate missing return packets or aggressive timeouts.

Kubernetes: additional pitfalls and concrete checks

Kubernetes increases complexity: load balancers, kube‑proxy (iptables/ipvs/eBPF), CNI‑overlays and SNAT all affect source IP and return paths. Check service configurations and captures at the node and pod level.

ExternalTrafficPolicy and SNAT

For Services there are two operating modes for ExternalTrafficPolicy: Cluster (default, allows forwarding to any Node, can cause SNAT) and Local (preserves client IP, requires the LB to target only Nodes with local endpoints). Choose deliberately based on your firewall topology.

kubectl get svc -n   -o yaml
# Umstellen (Beispiel)
kubectl patch svc -n   -p '{"spec": {"externalTrafficPolicy": "Local"}}'

Captures in Pods and Nodes

Use a privileged debug‑pod or DaemonSet to run captures on host interfaces. That shows whether SNAT is taking place and which Node sends the return response.

# Beispiel: privilegierter Debug-Pod
kubectl run -i --tty debug --image=corfr/tcpdump --privileged --rm -- bash
# dann im Pod
tcpdump -ni any host  and tcp port  -w /tmp/pod.cap

Kube‑Proxy, IPVS and eBPF

Kube‑proxy in iptables mode creates NAT rules; IPVS works with virtual servers and can make different hashing/path decisions; eBPF proxies (e.g. Cilium) provide better observability and can optionally avoid SNAT. Check which mode you are running and what impact it has on the return-path decision.

Mitigations: when each is practical

1) Enforce symmetry via PBR

If multi‑homing is the cause, steer replies out the same interface that the incoming packet used. This is done via ip rule and separate routing tables.

# Beispiel PBR
ip rule add from 192.0.2.0/24 table 100
ip route add default via 192.0.2.1 dev eth1 table 100
ip route show table 100
ip rule show

Limitations: PBR only helps when the source address is stable and known and no upstream NAT‑proxies change the source.

2) Place stateful components consistently

Option: either consolidate stateful functions at a deterministic location (e.g. a centralized NAT/firewall cluster), or operate stateful appliances with state‑sync (operationally intensive). With active/active firewalls, state replication is possible but complex and error‑prone.

3) Adjust firewall/ACLs instead of blanket disabling rp_filter

Setting rp_filter to loose can help short‑term, but reduces spoofing protection. Better: adjust firewall rules so they allow SNAT sources and expected return paths.

4) Load balancing and stickiness

With external load balancers, session stickiness or a hashing method can determine which backend accepts the flow. This is a pragmatic workaround when symmetry cannot be fully achieved.

5) Increase observability

In the long term, flow logs (NetFlow/IPFIX), firewall session logs, eBPF tracing or centralized MRTG/Prometheus metrics are useful to quickly identify return-path nodes and detect regressions after changes.

Rollback, change management and checklists

Changes to routing and firewall are critical. Test in small steps and back up pre-change configurations.

# Save configuration
ip -details route show table all > /tmp/routes.$(date +%F_%H%M).txt
ip rule show > /tmp/iprules.$(date +%F_%H%M).txt
sudo nft list ruleset > /tmp/nft.rules.$(date +%F_%H%M).txt
kubectl get svc -A -o yaml > /tmp/all-svcs.$(date +%F_%H%M).yaml

Document rollback points and keep them automatable for restoration (Ansible/Playbooks, Git‑ops). Monitor conntrack counters, retransmits and firewall drop counters after each change.

If changes don’t help: escalatory measures

• Temporary flow mirroring or SPAN on the router to observe return-path packets.
• Reconfigure load balancer at pool level (stickiness, Only‑Local‑Nodes).
• Temporary disabling of stateful features (only with a clear risk assessment).

Practical checklist (short version)

1. Define the scope and plan captures.
2. TCP traceroute and bidirectional tcpdump captures.
3. Check ip route / ip rule / rp_filter.
4. Evaluate conntrack entries and firewall session logs.
5. Kubernetes: check ExternalTrafficPolicy, SNAT, node/pod captures.
6. Perform the smallest change possible, enable monitoring, and have a clear rollback point.

Conclusion

Asymmetric routing can be reliably diagnosed if you proceed systematically: define the scope, take bidirectional captures, check host and kernel state (routing, rp_filter, conntrack) and consider Kubernetes specifics. Operationally, three approaches make sense: symmetry via routing/PBR, consistent placement of stateful functions, or workarounds like stickiness at the load balancer. Documentation and rollback points are mandatory for every change. With good observability, diagnosis time can be reduced from hours to minutes.

Asymmetric routing: architecture, operations and observability perspectives

Beyond pure fault diagnosis, it’s worthwhile to view asymmetric routing from an architecture and operations perspective. Which components in the data path are stateful, which affect the source IP or next hop, and which measurement/rollout mechanisms do you have for rapid response? The following points provide practical guidance for decisions, risks and integrated operational procedures.

Risks from hidden state breaks

Typical operational risks arise when a flow creates state at one point (e.g. firewall/NAT) and the response returns via a different path lacking the state information. Operationally, this leads to intermittent errors that are hard to reproduce. Risks at a glance:

• Unpredictable outages during version or policy changes on firewalls/load balancers.
• conntrack overflow during load spikes that rejects new connections.
• Service mesh/sidecar interception that reroutes packets or performs SNAT.

Conntrack and kernel tuning as an operational tool

Conntrack limits and timeouts are often overlooked causes. When tables fill up, new connections are no longer tracked, which can manifest as asymmetric drops. Check and raise limits moderately and adjust timeouts to your traffic profile.

# Example: persistent conntrack settings
# /etc/sysctl.d/99-conntrack.conf
net.netfilter.nf_conntrack_max=524288
net.netfilter.nf_conntrack_tcp_timeout_established=86400
sysctl --system

After changes, verify the values and the utilization:

cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/net/stat/nf_conntrack

Observability: Metrics, synthetic tests, eBPF

Without continuous telemetry, asymmetric paths remain ghost problems. Build three levels:

1. Base: firewall/router session logs and conntrack counters (exportable to Prometheus).
2. Synthetic: periodic, targeted TCP‑SYN probes along critical paths to check return‑path behavior.
3. Deep tracing: eBPF tracing (e.g., Cilium, bpftrace) for persistent flow correlation across hosts.

# Simple SYN probe (non-invasive):
hping3 -S -p 443 --count 3 --fast 198.51.100.23
# or curl for endpoint verification
curl -sS --max-time 5 https://198.51.100.23/healthz

For Prometheus you can capture conntrack values via node_exporter or the textfile collector. Example textfile format:

# /var/lib/node_exporter/textfile_collector/conntrack.prom
# HELP node_conntrack_entries Number of conntrack entries
node_conntrack_entries 12345

Kubernetes & Service Mesh: integration tips

Service meshes and CNI overlays often alter flow (sidecar SNAT, traffic redirect). Practical measures:

• For gateway‑critical paths, consider sidecar bypass (e.g., gateway/ingress without sidecar or with hostNetwork).
• Use eBPF/CNI observability (e.g., Cilium Hubble) to correlate Node↔Pod↔Service.
• For external LB: allow ExternalTrafficPolicy=Local + LB stickiness as a compromise to enable deterministic return paths.

Change management, canary rollouts and escalation

Changes to routing/firewalls should be done in canary steps: small subnet, limited clients, automated monitoring with alert triggers. Define clear escalation thresholds (conntrack utilization, retransmit rate, firewall drop rates). Automate rollbacks via Ansible/playbooks so a failed change can be reverted within a defined timeframe.

In short: Treat asymmetric routing not only as a debugging case but as an architectural topic. With Conntrack tuning, targeted observability, eBPF tracing and a disciplined canary change process you reduce operational risk and increase response speed during real incidents.

Architecture and operational rules for prevention

Treat asymmetric paths as an architectural concern, not only a debugging case. Separate transitive network functions (routing/ECMP) from stateful services (NAT, firewalls, load balancers) and define clear paths: stateful components should be deterministically reachable or use state sync. Use BGP Communities or source‑based routing to enforce return‑path determinism in multi‑homing scenarios.

Operationally: set thresholds for conntrack utilization, retransmits and firewall drops as alerts, automate synthetic SYN probes into your monitoring and integrate routing-policy changes into a GitOps workflow with tested canary steps. Explicitly document dependencies on bespoke enterprise software components that rely on client-side IPs or session stickiness, and retain session logs for forensic analysis.

Asymmetric Routing and routing loops are also relevant to this topic. The article clearly contextualizes these aspects and shows what matters in day-to-day operations.