Slow SSH connections: Identifying causes with tcpdump, ssh -vvv and MTU checks

“Why does an SSH login take 10–30 seconds even though ping is clean?” If you are confronted with slow SSH connections, a phase-based approach helps: first determine in which phase the delay occurs, then narrow it down on the client with ssh -vvv, next inspect the transport with tcpdump, and finally test MTU/PMTUD. MTU = Maximum Transmission Unit (maximum packet size); PMTUD = Path MTU Discovery (mechanism by which the sender discovers the maximum usable packet size). In production environments typical causes are DNS/reverse-DNS, GSSAPI/Kerberos, PAM/Directory timeouts, silent packet loss on VPN/overlay paths and MTU blackholes.

Quick overview: In which phase does the delay occur?

SSH can be usefully divided into five phases: TCP connection establishment, SSH Kex/handshake (key exchange), authentication (public key, password, GSSAPI), session/PTY setup and subsequent data transfer. The root cause determines in which phase the delay is visible — measure that specific phase, not only the total latency.

• Before password/key: often DNS or GSSAPI; delays occur during name resolution or while obtaining a Kerberos ticket.
• During authentication: external AuthorizedKeysCommand, LDAP/SSSD or PAM modules can cause timeouts.
• After auth, before shell: server-side login scripts, PTY allocation or network/MTU problems.
• During data transfer: packet loss, PMTU blackholes, QoS/traffic shaping or TCP windowing issues.

Diagnostic sequence: structured procedure

A fixed diagnostic sequence saves time and reduces unintended changes. Recommended order:

1. Client: ssh -vvv and basic checks (IP instead of hostname, disable GSSAPI).
2. Server: sshd logs, sshd -T, check PAM/SSSD/AuthorizedKeysCommand.
3. Network: tcpdump on client, server (and bastion) for TCP and ICMP captures.
4. MTU/PMTUD: ip link, DF-ping, optionally MSS clamping as a workaround.
5. Kubernetes: check CNI MTU, use a DaemonSet for tests.

Client-side debugging with ssh -vvv

ssh -vvv shows the client-side execution timeline. The output contains time progress and pauses that you can use as indicators for the affected phase. If you see gaps, note timestamps — that makes correlating with tcpdump or server logs easier.

ssh -vvv user@zielhost

What to look for:

• “Resolving host…” or a long time until “Connecting to …” → DNS/network.
• Pause at “Authentications that can continue…” → GSSAPI/Kerberos or PAM/directory timeout.
• Delay after “Entering interactive session.” → server-side login scripts, PTY setup or an MTU blackhole where the first packet with larger payload does not arrive.

Quick check for GSSAPI / Kerberos

In many enterprise setups GSSAPI (single sign-on via Kerberos) is enabled. If the KDC is unreachable or DNS is misconfigured, the client will wait. Test temporarily:

ssh -vvv -o GSSAPIAuthentication=no user@zielhost

If that is significantly faster, check KDC, DNS and NTP reachability. Caution: in environments that require SSO, disabling this is only a temporary workaround.

DNS and PTR checks

sshd often performs reverse-DNS lookups (PTR) of the client host. Slow or missing PTRs cause delays. Test directly by IP:

ssh -vvv user@203.0.113.10

# Auf dem SSH-Server prüfen:
getent hosts 198.51.100.27
# oder
dig -x 198.51.100.27 +time=2 +tries=1

If PTR resolution is slow, correct the DNS/zones or set UseDNS no in /etc/ssh/sshd_config if PTR-related issues persist. Always test with sshd -t before reloading.

Server checks: sshd, PAM and AuthorizedKeysCommand

On the target host the usual investigation paths are: sshd logs, PAM/SSSD timeouts and external AuthorizedKeysCommand scripts (these fetch public keys from databases). If delays are found there, fix the root cause or add timeouts/cache.

# Logs anzeigen
sudo journalctl -u ssh -S "-30min" --no-pager
# oder
tail -n 200 /var/log/auth.log

# Effektive sshd-Konfiguration prüfen
sudo sshd -T | egrep -i "gssapi|usedns|usepam|authorizedkeyscommand|login"

If AuthorizedKeysCommand performs external API calls, check latency, timeouts and fallbacks. Caching and local fallbacks reduce impact during API outages.

Network diagnosis with tcpdump: common patterns and interpretation

tcpdump reveals retransmits, missing ACKs or ICMP error messages — precisely the indicators that point to MTU/PMTUD issues or packet loss. Captures should be taken on both ends (client and server) for comparison.

# Enger Mitschnitt: nur SSH-Verkehr
sudo tcpdump -i any -nn -s 96 -w /tmp/ssh-slow.pcap "host 198.51.100.27 and tcp port 22"
# Parallel ICMP/Meldungen
sudo tcpdump -i any -nn -s 96 "icmp or icmp6"
# TShark (CLI) für schnelle Analyse
sudo tshark -r /tmp/ssh-slow.pcap -q -z io,stat,0, "tcp.analysis.retransmission or icmp"

Typical observations:

• Repeated identical sequence numbers → retransmits → packet loss on the path.
• TCP-SYN sent, SYN-ACK received, long pause until ACK → packet loss in one direction of the path.
• ICMP Type 3 Code 4 (IPv4 „Fragmentation Needed“) or ICMPv6 „Packet Too Big“ → PMTUD info present; sender can adjust MTU.
• Absence of ICMP messages despite DF-ping failures → PMTUD blackhole (often caused by firewalls, NAT or tunnels that block ICMP).

An example tcpdump excerpt and its meaning:

12:00:01.123456 IP 10.0.0.1.54321 > 10.0.0.2.22: Flags [P.], seq 1:1449, ack 1, win 229, length 1448
12:00:01.234567 IP 10.0.0.2.22 > 10.0.0.1.54321: Flags [.], ack 1449, win 65535, length 0
12:00:10.345678 IP 10.0.0.1.54321 > 10.0.0.2.22: Flags [P.], seq 1:1449, ack 1, win 229, length 1448 (retransmission)

Here the long pause and subsequent retransmission indicate packet loss or a drop. If instead an ICMP „Fragmentation Needed“ follows, the cause is fragmentation.

MTU checks: concrete steps

MTU issues are very common in environments with encapsulation (VPN, WireGuard, VXLAN, GRE). Steps to check:

1) Check interface MTU

ip link show

Note tunnel/overlay interfaces such as wg0, tun0, vxlan0 or CNI devices (cni0, flannel.1). The effective path MTU = smallest MTU minus encapsulation overhead.

2) DF-Ping to determine path MTU

# Beispiel IPv4: 1472 payload entspricht MTU1500 (1500-28)
ping -c 3 -M do -s 1400 zielhost
ping -c 3 -M do -s 1472 zielhost

If large DF pings fail, reduce the size stepwise until replies are received — this reveals the actual path MTU.

3) MSS-Clamping as a pragmatic workaround

If ICMP is blocked on the path (PMTUD blackhole), MSS-Clamping reduces the payload size negotiated during the TCP handshake. Example with iptables:

# MSS-Clamping am Gateway (nur TCP SYN)
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

With nftables:

# nftables Beispiel
sudo nft add table inet mangle
sudo nft 'add chain inet mangle forward { type filter hook forward priority 0 ; }'
sudo nft add rule inet mangle forward tcp flags syn tcp option maxseg size set rt --clamp-mss-to-pmtu

Note: MSS-Clamping only fixes TCP; UDP-based protocols remain affected. Apply this measure narrowly at the edge of the problematic path and document it in the change log.

Kubernetes: specific testing and operational notes

Kubernetes clusters often use overlay networks (VXLAN, Geneve) or host networks with CNI devices. These add additional headers and reduce the effective MTU. In many organizations the ICMP policy is also RESTrictive, which interferes with PMTUD.

Practical: Check MTU and tcpdump in a cluster

Distribute the test via a DaemonSet or a temporary debug pod to all nodes to ensure the entire route is checked.

# Beispiel: Debug-Pod auf einem Node starten
kubectl run -it --rm ssh-debug --image=alpine --overrides='{"spec":{"containers":[{"name":"c","image":"alpine","command":["/bin/sh","-c","apk add --no-cache iproute2 tcpdump bind-tools; sleep 3600"]}],"hostNetwork":true}}' --RESTart=Never

# Im Pod:
ip link show
ping -c 3 -M do -s 1400 10.244.0.5
tcpdump -i any -n -s 96 'tcp port 22' -w /tmp/ssh-node.pcap

Alternatively, deploy a DaemonSet with tcpdump, collect the pcaps centrally and compare node-to-node. Pay attention to permissions and data protection — PCAPs can contain sensitive content.

Monitoring, automation and prevention

To prevent recurrence, a combination of monitoring and preventive measures is recommended:

• Measurement of SSH login latency: a simple synthetic check that periodically opens a connection and measures the time until the prompt. Script example below.
• Network health checks: periodically verify path MTU and ICMP availability.
• Alerting: If SSH login latency > x seconds or retransmits/ICMP errors appear in tcpdump, open a ticket.
• Documentation: Record all temporary workarounds (MSS-Clamp, MTU changes, UseDNS) in the change log and assign an owner.

Example script for synthetic SSH latency measurement (BatchMode prevents password prompt):

#!/bin/bash
# ssh-latency-check.sh
TARGET=$1
if [ -z "$TARGET" ]; then
  echo "Usage: $0 user@host"
  exit 1
fi
START=$(date +%s%3N)
ssh -o BatchMode=yes -o ConnectTimeout=10 -o PasswordAuthentication=no -q $TARGET exit
RC=$?
END=$(date +%s%3N)
DUR=$((END-START))
if [ $RC -eq 0 ]; then
  echo "ok $TARGET $DUR ms"
  exit 0
else
  echo "fail $TARGET $DUR ms (rc=$RC)"
  exit 2
fi

Rollback and security strategy

All interventions must be reversible and tested. Basic rules:

• Validate changes to /etc/ssh/sshd_config with sshd -t and test them in a second admin session before terminating existing sessions.
• Assign an expiration to firewall and iptables rules or deploy them via configuration management so an automatic revert is possible.

Practical pitfalls

Typical errors that cost time:

• One-sided tcpdump captures: asymmetric routing leads to incorrect conclusions. Always capture at multiple points.
• ICMP blocked on firewalls but not documented: PMTUD collapses without obvious errors.
• AuthorizedKeysCommand without timeout/cache: external API failure blocks login.
• Setting MSS clamp globally without documentation: later performance issues remain unexplained.

Summary / Conclusion

With a phase-based analysis, slow SSH connections become manageable: use ssh -vvv to identify the affected phase; tcpdump shows transport and PMTUD indicators; DF pings and MTU checks provide the path MTU. In Kubernetes and VPN environments MTU/PMTUD are particularly relevant because encapsulations reduce the effective packet size and ICMP messages are often blocked. Implement pragmatic, documented workarounds (MSS clamping, temporary tunnel MTU adjustment) and plan the root-cause fix in parallel (DNS-PTR, ICMP policy, KDC/NTP stabilization). A short, reproducible runbook and automated synthetic checks prevent recurrence and reduce incident effort.

Checklist and runbook (short version)

• Document reproduction (client, time, path).
• Run ssh -vvv, note the phase.
• Test IP instead of hostname; temporarily disable GSSAPI.
• Server: check logs, sshd -T, AuthorizedKeysCommand/LDAP/SSSD timeouts.
• tcpdump on both ends: SSH filter + ICMP.
• Check MTU: ip link, DF ping, MSS clamp only after analysis.
• In Kubernetes: check CNI-MTU on all nodes, use debug pods/DaemonSet.
• Document changes and ensure they are rollback-capable.

Architecture and operations perspective: why slow SSH connections become systemic

Beyond individual errors, slow SSH connections are often a symptom of architectural or operational inconsistencies. Considerations that go beyond packet captures help plan durable solutions: load balancers, NAT gateways, stateful firewalls, SD-WAN providers and cloud security groups alter TCP/SYN behavior, can eliminate ICMP or produce asymmetric routing. Document the full path (Client → Zones → Bastion → Target) and check which components actively terminate or proxy TCP.

Check commands for infrastructure effects

A few simple checks often reveal hidden influences:

# Check offload features (NIC/VM host)
ethtool -k eth0

# Kernel flags for MTU probing
sysctl net.ipv4.tcp_mtu_probing

# Conntrack load (relevant for NAT/firewalls)
sudo sysctl net.netfilter.nf_conntrack_count
sudo sysctl net.netfilter.nf_conntrack_max

Typical operational causes and risks

• NIC offloads (GRO/LRO/TSO) can change the order and size of visible packets; disable temporarily for debugging, but only briefly — performance may otherwise suffer.
• SYN proxies or TCP termination at load balancers shift handshake timing: check whether the proxy has additional timeouts.
• Conntrack exhaustion stops new connections or increases retransmits; changes to nf_conntrack_max require planning and monitoring.

Betriebsleitlinien und Integration

Plan changes as simple, roll‑backable steps: Canary‑Rollout (one gateway), measurement before/after (SYN‑Latency, tcp_retrans), and automation via configuration management. Include verification scripts and tcpdump collection in your CI/CD runbooks so that debugging is reproducible and responsibilities are clear. Be mindful of data protection for PCAPs and assign explicit owners for temporary workarounds such as MSS‑Clamping or changes to UseDNS.

In short: Do not treat slow SSH‑connections merely as isolated incidents, but as indicators of network and operations architecture. Only then can scalable, documented and roll‑backable solutions be implemented.

For this topic, Tcpdump Ssh and Mtu Check are also important. The article contextualizes these aspects and shows what matters in day‑to‑day operations.