Analyze MTU and fragmentation issues and prevent them over GRE/IPsec

MTU and fragmentation issues are at the top of the list of unexpected performance and reachability failures for site-to-site VPNs and GRE tunnels. I mention the focus keyword MTU and fragmentation issues right at the start because many outages are directly related: reduced effective MTUs caused by additional headers (GRE, IPsec/ESP, NAT-T) lead to fragmentation or dropped packets, and when ICMP errors are blocked, Path‑MTU‑Discovery (PMTUD) fails. This article provides a hands‑on presentation of causes, diagnostic sequences with explicit commands, concrete countermeasures and a safe fallback strategy for production operation.

MTU and fragmentation issues: Why GRE and IPsec cause MTU problems

A brief technical overview: MTU (Maximum Transmission Unit) is the maximum size of a Layer‑3 packet that a link can carry without fragmentation. GRE (Generic Routing Encapsulation) is a tunneling protocol that encapsulates an inner packet with an additional GRE header; the basic GRE header is 4 bytes, additional options (Key, Sequence) extend it. IPsec/ESP (Encapsulating Security Payload) encrypts and authenticates data, which adds further headers (ESP header, initialization vector, padding, ICV/authentication tag). In addition, NAT‑Traversal (NAT‑T) adds a UDP encapsulation (UDP header 8 bytes). Together, these overheads significantly reduce the effective MTU for the inner payload.

Typical overhead components

• Outer IP header (IPv4: 20 bytes, IPv6: 40 bytes)
• UDP (optional for NAT‑T): 8 bytes
• ESP header + sequence: 8 bytes (SPI 4 + Seq 4) + IV (e.g. 16 bytes for AES) + ICV (e.g. 12–16 bytes) + padding
• GRE basic header: 4 bytes (plus optionally 4 bytes Key, 4 bytes Seq)

Because the exact size depends on the encryption scheme (e.g. AES‑GCM vs. AES‑CBC + HMAC), a blanket calculation only provides orientation. In practice, GRE over IPsec with NAT‑T can easily add 60–100 bytes of overhead — enough to force a 1500‑byte payload into multiple fragments.

Why PMTUD fails and the risks that entails

Path‑MTU‑Discovery (PMTUD) is a mechanism where a host attempts to determine the largest usable packet size along the path. Routers that see a packet that is too large and prevent fragmentation (DF‑Bit = Don’t Fragment) send back an ICMP Type 3 Code 4 (Fragmentation Needed). Two problems commonly occur in practice:

• ICMP is blocked by firewalls or filters so the Fragmentation‑Needed message never reaches the sender and the connection effectively stalls. This is especially common with IPsec tunnels because ICMP replies are not always correctly routed back to the inner address.
• Encapsulation changes source/destination addresses or ports (with NAT‑T), so ICMP messages cannot be meaningfully correlated.

The result: TCP connections stall during setup (common with TLS/HTTPS), UDP streaming drops out, or packets are fragmented, increasing CPU load and packet loss.

Detection: Which checks should be part of the initial diagnosis?

Proceed methodically and document each step. The goal is to find the point where the effective MTU is reduced or ICMP errors are being suppressed.

1) Check configured MTUs

Verify the MTU on all involved interfaces: physical, tunnel and virtual interfaces.

ip link show dev eth0
ip link show dev gre1   # Beispiel für GRE-Interface
ip -4 route get 10.0.0.1   # zeigt u.a. die MTU-Informationen

2) PMTUD test with ping

Using the DF‑flag (Don’t Fragment) you test how large an unfragmented packet can be. The correct payload size for IPv4 is typically MTU minus 28 bytes (IP + ICMP Header).

# Beispiel: Test auf 1472 Bytes Payload ergibt 1500 Gesamt (IPv4: 20 + ICMP: 8)
ping -M do -s 1472 
# langsam verkleinern, bis ping erfolgreich ist

If the ping fails but smaller sizes work, you have found the limit. If a high‑MTU ping fails despite apparent path capacity, the ICMP Fragmentation Needed message is usually missing.

3) Fragmentation capture with tcpdump

Fragmented IPv4 packets can be captured with a simple BPF filter. This lets you find already‑fragmented packets in live traffic:

tcpdump -n -i eth0 'ip[6:2] & 0x1fff != 0'
# Optional: nur Pakete zu/von einer bestimmten IP
tcpdump -n -i eth0 'host 10.0.0.1 and ip[6:2] & 0x1fff != 0'

Erklärung: Im IPv4‑Header befinden sich Flags und Fragment‑Offset in Bytes 6–7; die Maske 0x1fff prüft auf einen Offset > 0 (Nicht‑Erstfragment).

4) Check whether ICMP Fragmentation replies arrive

If PMTUD fails, run tcpdump on both sides and look for ICMP Type 3 Code 4:

tcpdump -n -i eth0 icmp and 'icmp[0]=3 and icmp[1]=4'
# oder allgemein nach ICMP für Fehlermeldungen
tcpdump -n -i eth0 icmp

If no ICMP Fragmentation Needed messages are observable, but fragmentation occurs, it is highly likely that a firewall is blocking ICMP or that the ICMP messages are not being routed back correctly.

Concrete countermeasures: what helps immediately and in the long term?

There are several proven strategies to avoid MTU and fragmentation issues. Choose a combination that fits your infrastructure and document the changes.

Option A: Reduce tunnel MTU (quick, safe)

Set the MTU on the tunnel interface so that the additional encapsulation is taken into account. This is reliable and safe, but it reduces the usable payload size.

# Beispiel Linux GRE-Interface auf 1400 setzen
ip link set dev gre1 mtu 1400
# Beispiel für IPsec-Tunnel (veth/ifname) analog

Why it works: By reducing the transmit fragment size you avoid fragmentation entirely. When it fails: If applications send very large UDP datagrams (e.g. video streams), the limit will be noticeable.

Option B: MSS clamping for TCP (recommended for web/SSH/SMB)

MSS (Maximum Segment Size) is the largest TCP payload in a segment. Many TCP stacks honor MSS during connection setup; by clamping, NATs/routers adjust the MSS in SYN packets.

# Beispiel iptables für MSS-Clamping (Linux-Router/Firewall)
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Why it works: New TCP connections negotiate smaller segments and avoid fragmentation without changing the MTU on end hosts. When it fails: UDP traffic is unaffected; TCP connections that override MSS can still cause issues.

Option C: Plan NAT‑T / UDP encapsulation deliberately

When IPsec NAT Traversal (UDP/4500) is used, an additional 8 bytes for the UDP header are added. Account for these in the tunnel MTU, or avoid UDP encapsulation when no NAT function is required.

Option D: Increase PMTUD robustness

Ensure that all firewalls allow ICMP Type 3 Code 4 (Fragmentation Needed). If that is not possible, use a combination of tunnel MTU reduction and MSS clamping.

GRE over IPsec: special considerations

GRE is often used when other protocols (e.g. multicast, IPv6 within IPv4) need to be transported over the VPN in addition to IPv4. However, when combined with IPsec, MTU risks increase:

• GRE adds an additional header (at least 4 bytes), often more with options.
• IPsec in tunnel mode adds further encapsulation; if NAT sits between the endpoints, UDP NAT‑T is added.
• Inner ICMP can be affected by IPsec/outer layers, which complicates PMTUD.

Recommendation: Calculate the effective overhead sum and reduce the tunnel MTU conservatively. Example calculation for IPv4 with NAT‑T, GRE and AES‑GCM (simplified representation):

physische MTU: 1500
- äußeres IPv4-Header: 20
- UDP (NAT-T): 8
- ESP-Overhead (IV + ICV + ESP Header): z.B. 28
- GRE-Header: 4
= effektive MTU für innere Pakete: 1500 - 60 = 1440

Set the tunnel MTU to e.g. 1400 to provide headroom for padding/variations.

Practical checklist: proceed step by step

1. Document the path and involved devices (physical MTUs, firewalls, NATs).
2. Perform PMTUD ping tests with DF set and record the maximum payload sizes.
3. Start tcpdump on both tunnel endpoints, look for fragmented packets and ICMP Type 3 Code 4.
4. If ICMP is blocked: allow ‚Fragmentation Needed‘ specifically in firewalls; if that is not possible, proceed to step 5.
5. Temporarily apply MSS clamping on the border routers and test TCP workloads.
6. Reduce the tunnel MTU on both sides conservatively (e.g. 1400) and test again.
7. Monitor CPU/MTU-related metrics; fragmented packets cause CPU load on routers/firewalls.
8. Document the change in the change log and schedule a maintenance window for a reversible rollback.

Concrete troubleshooting examples

Symptom: websites do not load, SSH does not connect over the tunnel

Procedure:

1. Check ping with DF set: if large ICMP is not possible, a PMTUD issue is likely.
2. Run tcpdump on the tunnel endpoint: fragmented packets or missing ICMP?
3. Set MSS clamping and test; if that helps, the TCP issue is resolved. If not, reduce the tunnel MTU.

Symptom: UDP streaming fails for certain streams

UDP is not affected by MSS clamping. Check the tunnel MTU, reduce it and verify whether the issue disappears. If the streaming sends very large packets, an application configuration may be required so it uses smaller UDP datagrams.

Rollback and safety strategy

Changes to MTU and firewall must be reversible and carried out within clearly agreed maintenance windows. Recommendations:

• Before changes: configuration backup and a clear test plan with measurement points (latency, packet loss, CPU).
• Make changes incrementally, first on a test segment or during off-peak hours.
• Enable automated monitoring (SNMP/Netflow/IPS alarms) to detect regressions immediately.
• Fallback: restore the previous MTU/MSS rules and perform the documented communication to affected teams.

When is an architectural change appropriate?

If you regularly experience high overheads from IPsec/GRE/NAT and many problems with UDP applications or multicast, consider whether reducing tunnel encapsulation (e.g. only IPsec to protect individual subnets instead of GRE for multicast) or using native VPN alternatives (e.g. VXLAN with IPsec, modern SD-WAN solutions) will result in less operational overhead in the long term. Such changes, however, are projects that require test phases and migration plans.

Summary and concrete course of action

MTU and fragmentation issues in VPN scenarios with GRE and IPsec are a recurring cause of otherwise unexplained connection drops and performance degradations. The pragmatic sequence in production is:

1. Identify (MTU checks, ping with DF, tcpdump on fragments and ICMP).
2. Quick measures (MSS clamping for TCP, temporary tunnel MTU reduction).
3. Permanent configuration (MTU on the tunnel, firewall rules for ICMP, a documented rolling plan).

If you follow these steps systematically, the majority of problems can be resolved without deep architectural changes. Only when fragmentation and ICMP blocking occur regularly and critical services are affected is a re-architecture advisable.

Practical snippets: key commands at a glance

Check MTU, set tunnel MTU, MSS clamping, tcpdump filters:

# Display MTU
ip link show dev eth0
ip link show dev gre1

# Set tunnel MTU
ip link set dev gre1 mtu 1400

# MSS clamping (Linux Firewall/Router)
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

# Find fragmented packets
tcpdump -n -i eth0 'ip[6:2] & 0x1fff != 0'

# Find ICMP Fragmentation Needed
tcpdump -n -i eth0 icmp and 'icmp[0]=3 and icmp[1]=4'

# PMTUD ping (IPv4)
ping -M do -s 1472 

Additional links and internal cross-references

For deeper troubleshooting, consult articles on IPsec‑troubleshooting with tcpdump/IKE logs, firewall rule review and monitoring baselines. Plan links to your internal runbooks: IKE log analysis, firewall ICMP rules, monitoring dashboards for fragmentation and CPU spike alerts.

Conclusion

MTU and fragmentation issues are avoidable if you deliberately plan the overheads of encapsulation, keep PMTUD signaling paths intact, or alternatively employ MSS clamping and a conservative tunnel MTU. Conservative testing, reproducible measurements with tcpdump and ping, and changes that can be rolled back are the basis for stable operations. GRE over IPsec provides functionality, but requires disciplined MTU planning – otherwise you pay with packet loss, increased latency and higher operational effort.

GRE tunneling is also relevant to this topic. The article places these aspects in context and shows what matters in day-to-day operations.