Wireshark How-to: TCP retransmissions, the three-way handshake and locating performance hotspots

This Wireshark-Howto begins with clear prerequisites and a pragmatic checklist so you can quickly and reliably narrow down TCP retransmits, three-way handshake problems and performance hotspots in your network. The focus keyword Wireshark-Howto deliberately appears at the beginning: the goal is practical guidance for administrators, system engineers, operators and IT service providers. I briefly explain technical terms such as RTT (Round-Trip-Time: latency for the outbound and return path) or Zero Window (the receiver signals it cannot currently buffer data) and show why a finding is typically network- or host-side.

Capture strategy: where to capture and why it matters

The value of your analysis depends primarily on the capture location. A capture in the wrong place delivers misleading retransmit patterns or obscures asymmetric routing. Plan captures so you can follow the direction of message flow.

Capture locations compared

• Client‑side: Covers the client stack, local firewalls, VPN clients and Wi‑Fi retransmits.
• Server‑side: Shows whether the server receives SYNs, how it responds and whether local limits (backlog, ulimits) apply.
• Firewall/Load‑Balancer‑side: Important with NAT, conntrack or TLS inspection; you often see only one direction, so an additional capture is advisable.
• SPAN vs. TAP: SPAN ports are easy to access but can drop packets under high load; TAPs provide more complete data but require hardware and planning.

Operational risks and compliance

• Data protection: Captures often contain user data and must be minimized, anonymized or deleted promptly in accordance with data protection requirements.
• Performance: Capturing on production hosts can increase CPU/IO load. Use ring buffers or remote storage for pcap files.
• TLS: Encryption hides payload contents, not timing, flags or window state — these metadata are sufficient for TCP mechanics analysis.

Pragmatic capture examples

Use targeted filters and ring buffers to produce reliable PCAPs. Here are two field-tested examples for Linux and Windows.

# Linux: gezielt Host/Port mitschneiden, Ringbuffer nutzen
sudo tcpdump -i eth0 -s 0 -nn 
  'host 10.20.30.40 and tcp port 443' 
  -C 200 -W 10 -w /var/tmp/capture_%Y%m%d_%H%M%S.pcap

# Windows: pktmon verwenden, dann ins pcap-Format konvertieren
pktmon filter remove
pktmon filter add -p 443
pktmon start --etw -m real-time
# Tests durchführen, dann stoppen:
pktmon stop
pktmon format PktMon.etl -o capture.pcap

Always run test cases before and after changes with the same parameters so measurements are comparable.

Important Wireshark views for TCP

Wireshark offers several features that are particularly useful: display filters (for a focused view), Conversations (flows), Follow TCP Stream (reconstructs the flow) and Time-Sequence Graph (visualizes sequences, ACKs and retransmits). These views help identify patterns and determine the direction of the problem.

Understanding offloading effects

Terms such as TSO (TCP Segmentation Offload), GRO (Generic Receive Offload) and LRO (Large Receive Offload) mean that the network card takes over parts of TCP processing. Captures on the host can therefore show oversized segments or missing small segments. If findings conflict, check the capture on a TAP or temporarily disable offloading.

# Offloading (temporär) deaktivieren - Linux
sudo ethtool -K eth0 tso off gso off gro off

Analyze the Three‑Way Handshake

The Three‑Way Handshake (SYN, SYN‑ACK, ACK) confirms that a TCP connection can be established. If this step fails, data transfer typically does not occur. Pay attention to TCP options in SYN/SYN‑ACK: MSS (Maximum Segment Size), Window Scale (window scaling) and SACK (Selective Acknowledgement) provide clues about MTU, window and retransmit issues.

Verification sequence for the handshake

1. Filter the flow: 5‑tuple (Client IP, Server IP, Client Port, Server Port, Protocol).
2. Is the SYN from the client visible? Does a SYN‑ACK return from the server?
3. Is the final ACK missing? Check captures on both sides to verify the return path.

Typical causes for failed handshakes are firewall ACLs, asymmetric routing (stateful box drops replies), SYN‑Proxy/SYN‑Cookies or server limits such as full backlogs.

Classifying TCP retransmits

Retransmits are TCP’s response to suspected packet loss. Important distinctions:

• Fast Retransmission: Occurs after Duplicate ACKs and indicates real packet loss.
• RTO Retransmission: Occurs after a timeout and strongly affects throughput.
• Spurious Retransmission: Appears to exist but can be caused by reordering or capture artifacts.

Useful display filters

• tcp.analysis.retransmission
• tcp.analysis.fast_retransmission
• tcp.analysis.duplicate_ack
• tcp.analysis.out_of_order
• tcp.analysis.zero_window || tcp.analysis.zero_window_probe

Interpretation tips: Duplicate ACKs without retransmits suggest reordering (e.g. ECMP). Retransmits without Duplicate ACKs can be capture gaps or real RTOs.

Performance spots: typical causes and measurement patterns

If the handshake succeeds, various wait states remain that limit throughput. The most common causes and their signatures are:

High RTT and jitter

High RTT (Round‑Trip‑Time) or highly variable jitter indicate queueing (full buffers), link congestion, Wi‑Fi retransmits or firewall inspection. In the time‑sequence graph you see extended gaps between data segments and ACKs.

Zero window: host or application issue?

Zero Window means the receiver sets its receive window to 0 because the application buffer is full. This is often an application or I/O issue (slow processing, garbage collection or blocking writes). Only when network latencies are so high that ACKs are delayed is it primarily a network problem.

MTU / MSS issues

Path MTU issues (e.g. caused by tunnels such as GRE, VPN or PPPoE) lead to fragmentation or the drop of larger segments when the DF flag is set and ICMP messages are blocked. Check MSS in the SYN and use DF pings for validation.

# Test MTU (Linux): set DF flag, adjust payload
ping -M do -s 1472 10.20.30.40
# On failure: reduce iteratively

Firewall and NAT-specific checks

Stateful firewall devices and NAT/conntrack are common causes of apparent packet loss or asymmetric flows. Here are some practical checks and commands.

Check conntrack limits and timeouts

Conntrack (Connection Tracking) is a mechanism in firewalls that manages TCP flows in a table. If the table is full or timeouts are too short, sessions will be discarded.

# Conntrack statistics (Linux nftables/conntrack-tools)
sudo conntrack -S
# Check number of entries
sudo conntrack -L | wc -l

Reduce unnecessarily aggressive timeouts for established connections and check for port exhaustion on NAT machines (source port exhaustion when many connections originate from a single IP).

Detect asymmetric routing

If requests take one path and responses another (e.g. due to ECMP, path changes or multi-ISP), a stateful firewall will drop responses because no state exists. Two captures (inside/outside) quickly remove uncertainty.

Metrics, thresholds and monitoring

For sustainable troubleshooting you should define and monitor metrics. Important metrics:

• Retransmits per second and as a percentage of sent packets
• Duplicate ACKs per flow
• Average and 95th-percentile RTT
• Zero-window events per minute
• Conntrack table utilization

As a guideline: isolated retransmits are normal; persistent retransmit rates >1–2% of traffic indicate a serious problem. Thresholds depend on the application type (interactive applications are latency-sensitive, bulk transfers are more tolerant).

Practical troubleshooting procedures

1. Isolate the flow: Conversations → Top Talkers → apply 5-tuple filter.
2. Check the handshake: compare SYN/SYN-ACK/ACK on both ends.
3. Quantify retransmits: use tcp.analysis.* filters, determine count and direction.
4. Middlebox checks: check conntrack, NAT, firewall logs and load balancer health.
5. Host checks: CPU, I/O, socket buffers, netstat/tcpstat

# Example: TCP socket statistics (Linux)
ss -tan state established sport = :443
# Kernel TCP counters (Rx/Tx/Retransmits)
cat /proc/net/snmp | egrep 'Tcp|TcpExt' -n
# Check switchport errors (vendor-specific example)

Changes, tests and rollback

Changes to firewalls, MTU or NAT are effective but can have side effects. Follow these rules:

• Limit scope: Apply changes first to a subnet, VIP or test segment.
• Before/after: Use the same test cases, same capture location, save PCAPs.
• Prepare rollback: Export configuration before changes, define a timebox and metrics for rollback criteria.

Common pitfalls and how to avoid them

• Capture artifacts: SPAN port drop, offloading or timestamps can distort interpretation. Use a TAP if possible or disable offloading.
• Incomplete data: Capturing only one side often leads to misattribution. Two sides are the rule.
• Incorrect filters: Filters that are too broad flood the view, filters that are too narrow hide patterns. Start broad, then narrow down.
• Data protection: retain captures no longer than necessary; anonymize sensitive data flows.

Conclusion

This Wireshark-Howto provides a structured working method: start with the correct capture location, check the Three‑Way‑Handshake for basic validation, quantify retransmits and classify them by their signatures (Fast Retransmit vs. RTO). Zero Window often indicates host or application issues, while Duplicate ACKs and Fast Retransmits point to network or link faults. For firewall and NAT problems in particular, synchronized captures on both sides and Conntrack checks are critical. Plan changes with a rollback and measure before/after – this reduces operational risk and speeds resolution.

Use this guide as a basis for runbooks in your team: clearly defined verification paths, reproducible tests and close coordination between network, firewall and systems teams accelerate troubleshooting and increase the reliability of your production environment.

Wireshark-Howto: Capture automation, time synchronization and SIEM integration

As a supplement to the practical howto, it is worth integrating capture work into existing operational processes. This chapter describes architecture and operational aspects that are decisive in extended incident scenarios, for compliance requirements or for recurring performance checks. The focus keyword Wireshark-Howto helps place the guide within your documentation.

Architecture: central versus temporary capture points

• Persistent collectors: A dedicated aggregation interface (TAP or Mirror + dedicated capture host) provides PCAPs for several days or weeks, enabling long-term analysis and automatic indexing. Advantage: consistent baseline for comparison. Disadvantage: storage, data protection, and access control.
• Ephemeral captures: Short-term captures during incidents; they are flexible but less suitable for trend analysis. Combine both: permanent metadata extraction, short-term full PCAPs on alarm.

Time synchronization and correlation

Timestamps are the backbone of distributed analysis. If hosts and network collectors are not synchronized, ACKs, retransmits and firewall logs cannot be reliably correlated. Check the time source on all involved devices:

# Check NTP/chrony (Linux)
timedatectl status
# or for chrony
chronyc tracking

For very fine-grained latency analysis PTP may be required; for typical TCP issues consistent NTP synchronization is sufficient.

Automated extraction of metrics

Full PCAPs are large. Automatically extract metrics (retransmits, duplicate ACKs, RTT percentiles) with tshark and index the results into your SIEM or a time-series DB. Example: count retransmits per flow in a PCAP.

tshark -r capture.pcap -Y "tcp.analysis.retransmission" 
  -T fields -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport 
  | sort | uniq -c | sort -rn

The output can be forwarded via a log forwarder into Elasticsearch/Prometheus and tied to alerting rules (e.g. Retransmit‑Rate > X% over 5 minutes).

Containerized environments and namespaces

In container environments, interfaces are often short-lived (veths, bridge). Captures on the host do not always see the internal stack inside the namespaces. Use namespace-specific capture agents or port-mirroring CNI features to obtain complete flows.

Secure retention, redaction and compliance

• Define clear retention periods and automated deletion jobs to minimize GDPR/P11D risks.
• Reduce capture scope using filters (IP/Port) or packet-header-only mode when payload is not required.
• Encrypt PCAP archives and log access.

Operationalization / Runbook steps

1. Alert defined → trigger automatic snapshot (full PCAP).
2. Extract metadata (tshark/Zeek) and update dashboard.
3. Initial analysis: time/flow correlation, retransmit rate, zero-window events.
4. Apply change with scope, metrics and rollback plan (as already described).

These operational extensions make your Wireshark how-to reproducible and scalable: automated extraction, a clean time base and legally compliant retention reduce operational effort and accelerate troubleshooting across the team between network, firewall and system owners.

TCP retransmits and TCP handshake analysis are also important for this topic. The article contextualizes these aspects clearly and shows what matters in daily operations.