Live migration with shared storage: measuring latency, root-cause analysis and tuning

Live migration with shared storage is considered „the easy way“: the virtual machine (VM) keeps its virtual disks on a common storage while only CPU and RAM state are moved to another host. In practice, however, failures surprisingly often stem not from the virtualization function but from latency — and at points that are easily overlooked in day-to-day operations: storage response times, network jitter, overflowing queues, incorrectly set timeouts, or a multipath route that is „floating“ in the background.

This guide therefore focuses on operational reality: how to reliably measure latency during Live-Migration with Shared-Storage, systematically narrow down causes and then tune systems so that migrations work reproducibly — including prerequisites, typical pitfalls, verification steps and a clean fallback strategy.

What actually happens during Live-Migration with Shared-Storage (and where latency comes into play)

Even with shared storage, more is moved during a live migration than „just a bit of RAM.“ A typical process is iterative: memory pages are copied in advance (Pre-Copy) while the VM continues to run. Depending on the platform, „dirty pages“ (recently written pages) are copied afterwards. For a short moment the VM is paused (stun/pause), the remaining state is transferred and the VM is resumed on the target host.

Shared storage reduces the amount of block data that must be copied, but increases the requirements for consistent, low storage latency. From the moment the VM runs on the target host, its I/O requests (reads/writes) must traverse new host HBAs/NICs, new queue structures and possibly different paths to the storage. Any imprecision in path selection, MPIO/multipathing, LUN ownership, NFS mount options or switch buffering will manifest as latency spikes. Those spikes increase stun times, prolong the migration duration or cause timeouts.

Interpreting symptoms correctly: latency is not all the same

In troubleshooting it is crucial to identify which type of „slowness“ you are seeing. Three patterns are particularly common:

• Consistently high latency: e.g. 8–15 ms instead of 1–3 ms. Often caused by overload, incorrect queue depth, wrong storage tier, or links that are too slow.
• Spikes: e.g. every 30–120 seconds 50–200 ms. Often caused by retransmits, microbursts, path flaps, background jobs (snapshots, rebuilds), or cache evictions.
• Asymmetric latency: only on one host or only on one path. Typical for multipath misconfiguration, differing firmware/drivers, mixed VLAN/MTU, or incorrect LACP/MLAG parameters.

Important: a live migration is a „stress test“ for multiple layers at once. If you only look at the hypervisor, you often see only the symptom (migration takes long, VM stutters), not the cause (a queue fills up, NFS shows retransmits, iSCSI is undergoing path recovery).

Prerequisites and common pitfalls (before measuring)

Minimum technical requirements

• Synchronized time (NTP/Chrony): without consistent timestamps, correlating events between host, switch and storage is hardly possible.
• Stable paths to the shared storage: redundancy is good, but only when it is configured cleanly (Multipath/MPIO, failover policy, ALUA/Asymmetric Logical Unit Access).
• Separate migration network (when possible): otherwise migration competes with storage or client traffic.

Operational pitfalls

• Mixed MTU operation: Jumbo Frames (MTU 9000) applied only on parts of the path cause fragmentation or drops; both manifest as latency.
• Bufferbloat on switch ports/uplinks: high throughput with simultaneously increasing latency; migrations are sensitive to jitter.
• Snapshots/backups during migration: storage-side copy-on-write costs or additional reads occur.
• Overlooked queue limits: HBA/NIC queues, dm-multipath, iSCSI sessions or NFS-RPC slots can behave differently during migrations than in normal operation.
• Unclear success criteria: „Migration was slow“ is not a metric. Define target values (Max-stun, Max-duration, Max-latency-spike).

Measuring latency: Which metrics you really need

For a reliable root-cause analysis, a ping and a storage dashboard are rarely sufficient. You need measurements on three levels, ideally in parallel within the same time window:

• Guest/VM impact: stun time, application latency, timeouts, increased I/O wait time.
• Hypervisor/host: block-device latency (read/write await), queue utilization, CPU steal/ready, network retransmits.
• Storage/network: port errors, drops, retransmits, path failover events, array latency (frontend/backend), cache hit rate.

Rule of thumb: If host block latency is high while the array reports „green“, the issue is often a path or network problem. If array latency is high, the storage itself is under pressure (tiering, cache, backend disks, rebuild, snapshotting).

Linux host: measuring block latency and queue behavior

On Linux hypervisors (z. B. KVM-Hosts) sind iostat und sar robuste Startpunkte. Sie liefern „await“ (mittlere I/O-Wartezeit) und „svctm“ je nach Kernel/Tool-Version eingeschränkt; wichtiger ist die Kombination aus Latenz und Auslastung.

# Paket: sysstat
# 1) I/O-Latenz und Auslastung je Device, 1s Intervall, 60 Samples
iostat -x -d 1 60

# 2) Block-Device-Throughput und Queue in sar (falls aktiviert)
sar -d 1 60

# 3) VM-Statistiken (CPU-Wartezeiten, Runqueue) als Kontext
sar -q 1 60

Interpretation in the migration context:

• await increases significantly during migration: the storage path or array is under additional load.
• %util near 100% on individual devices: bottleneck on that device/path (e.g., a multipath path is being preferred).
• High avgqu-sz: requests pile up (queuing), often a precursor to timeouts.

Multipath/MPIO: make path state and flapping visible

Multipathing (Linux dm-multipath oder Windows MPIO) bündelt mehrere physische Wege zum Storage. Es schützt vor Ausfällen, kann aber Latenz massiv verschlechtern, wenn Pfade instabil sind oder die Policy ungünstig ist (z. B. unpassendes Load-Balancing, falsche ALUA-Prioritäten).

# Überblick über Multipath-Devices, Policies, Prioritäten und Pfade
multipath -ll

# Kernel-Log nach Storage-Resets, Path-Down/Up, Timeout/Abort prüfen
journalctl -k --since "-2h" | egrep -i "multipath|scsi|iscsi|nvme|path down|path up|abort|reset|timeout"

Wenn Sie während der Migration „Path down/up“-Ereignisse sehen oder häufige SCSI-Resets, sind das keine „kosmetischen“ Meldungen. Jede Recovery kann Latenzspikes im Sekundenbereich verursachen. Genau solche Spikes sind es, die Live-Migrationen destabilisieren.

Netzwerk: Verlust, Retransmits und Jitter erfassen

Für Shared-Storage ist das Netzwerk oft doppelt relevant: einmal für die Migration selbst (Memory-Transfer) und einmal für Storage-Protokolle wie NFS oder iSCSI. Drops und Retransmits wirken wie Latenz, weil TCP neu senden muss und Anwendungen warten.

# Interface-Statistiken: Drops, Errors, Retransmits-Hinweise
ip -s link

# TCP-Statistiken: Retransmits, RTO, Out-of-order
ss -s
netstat -s | egrep -i "retrans|timeout|failed|listen|segments"

# Paketmitschnitt zur Korrelation (nur gezielt, kurzzeitig!)
# Beispiel: iSCSI (3260) oder NFS (2049) an einem Storage-Interface
tcpdump -i ethX -nn -s 128 -w /tmp/storage-trace.pcap '(port 3260 or port 2049)'

Ein häufiger Fehler ist, „Latenz“ ausschließlich als Ping-Zeit zu messen. Ping misst ICMP und oft nur eine geringe Paketgröße. Storage-Protokolle und Migrationen reagieren aber auf Queueing, Drops unter Last, MTU-Fehler und Microbursts. Deshalb sind Interface-Counter und TCP-Statistiken so wichtig.

Windows/Hyper-V-Umfeld: Basismessung ohne Spezialtools

In Windows-Umgebungen können Sie mit Bordmitteln zumindest die Richtung bestimmen: Storage-Latenz per Performance Counter und Netzwerk-Fehler per Adapterstatistik. (Je nach Umgebung liefern Hyper-V- und SMB-Direct/RDMA-Setups zusätzliche Counter.)

# Netzadapter-Statistiken (Errors, Discards)
Get-NetAdapterStatistics | Sort-Object -Property Name | Format-Table -Auto

# Performance Counter für Datenträger-Latenz (Beispiel: alle Instanzen)
Get-Counter -Counter "LogicalDisk(*)Avg. Disk sec/Read","LogicalDisk(*)Avg. Disk sec/Write" -SampleInterval 1 -MaxSamples 30

Als grobe Einordnung: Einzelne Millisekunden sind in vielen SAN/NVMe-oF/FC-Umgebungen normal; zweistellige Millisekunden unter Last sind ein Warnsignal, insbesondere wenn sie spiken.

Ursachenanalyse: Vorgehen in klaren Schritten

The art is not to tune migrations „on suspicion“ but to test hypotheses. The following sequence has proven effective:

1. Establish reproducibility: Migrate a test VM or a non-critical VM, always the same time window, similar load.
2. Separate migration vs. storage traffic: Where possible use separate NICs/VLANs; otherwise at least measure per interface.
3. Localize host-side latency: Which device/multipath device shows high await/queue?
4. Look for path events: Path flaps, link errors, CRC errors, iSCSI reconnects, NFS retransmits.
5. Correlate storage-side events: Snapshot, rebuild, cache flush, controller failover, tiering.
6. Check migration settings: Concurrency, bandwidth limit, stun threshold, pre-copy parameters.

Check: Is the VM itself the driver (dirty page rate)?

Some VMs are „hard to migrate“ even though storage and network are fine: databases with very high write activity or memory-intensive applications generate a high dirty-page rate (memory pages change faster than they can be copied). The migration then oscillates through repeated iterations or ends with a high stun.

Practical test: Migrate a relatively „quiet“ VM and a „hot“ VM in the same time window. If only the „hot“ VM is conspicuous, the focus is more likely on workload/migration parameters than on shared storage latency.

Check: Path asymmetry and ALUA/Ownership

With SANs using ALUA there are often „optimized“ and „non-optimized“ paths. If a host predominantly uses non-optimized paths after migration, latency increases without obvious errors. You can see this in dm-multipath (priorities) or in the array front-end. The remedy is rarely „more bandwidth“, but correct ALUA detection, proper priorities and consistent driver/firmware levels across all hosts.

Check: NFS retransmits and RPC slots

With NFS, latency is often a mix of server response, network and client queue. Retransmits occur when packets are lost or responses arrive too late. A common pitfall: mount options that are too aggressive, provoking timeouts under load, or settings that are too conservative, prolonging recovery. This requires judgement, because „tuning“ otherwise only shifts symptoms.

# NFS-Client-Statistiken (Linux)
nfsstat -c

# Mount-Optionen und NFS-Version prüfen
mount | egrep -i " nfs "
cat /proc/mounts | egrep -i " nfs "

Tuning approaches that are effective in practice

The following measures are intentionally framed so you can justify and test them within change processes. Not every measure suits every environment; what matters is that each targets a concrete symptom.

1) Decouple and limit migrations (concurrency and bandwidth)

A classic: multiple concurrent live migrations produce traffic spikes that briefly overwhelm switch queues and storage frontends. That then looks like “sudden latency”. The often most effective measure is organizational/technical and simple: fewer parallel migrations and bandwidth limiting for migration traffic so storage traffic does not starve.

Why this works: you reduce microbursts and stabilize queue latency. In many environments a somewhat longer but steady migration is clearly better than a short migration with hard spikes and stun.

2) Harden network fundamentals: MTU, LACP, ECN, buffers

If you use Jumbo Frames: verify the MTU end-to-end (host NIC, vSwitch/bridge, switch ports, uplinks, storage ports). A single 1500 link in a 9000 path leads to fragmentation or drops. For storage protocols that is lethal.

# MTU am Host prüfen
ip link show | egrep -i "mtu|state"

# Pfad-MTU testen (Beispiel 8972 Payload für MTU 9000; anpassen je nach Overhead)
ping -M do -s 8972 -c 5 <ziel-ip>

If retransmits/out-of-order are high, also look at LACP/MLAG hashing and asymmetric paths. A “working” bonding can still fluctuate strongly if hashing and traffic patterns combine unfavorably.

3) Stabilize multipathing: policies, timeouts, path health

The goal is not “maximally aggressive failover” but predictable behavior. Timeouts that are too short can trigger unnecessary path switches on short spikes; timeouts that are too long make real failures painful. Also check whether a path is permanently worse (CRC errors, light levels on FC, bad transceivers, faulty cables) and thereby spoils the load balancing.

Rule of thumb: before you change parameters, stabilize the physical layer (cables/optics/ports), then adjust software policies.

4) Contain storage-side collateral load: snapshots, rebuilds, tiering

Live migrations often occur in maintenance windows — and exactly then storage tasks run: rebuilds, scrubs, snapshot consolidation, tiering moves. That is technically legitimate but operationally risky if all load spikes coincide. Schedule migration windows so storage background jobs are either finished or run throttled. If the array supports QoS (Quality of Service) or prioritization, use it for critical datastores.

5) VM-side preparation: smooth I/O, mitigate large writes

If individual VMs undermine migrations by heavy writes, simple measures sometimes help: pause batch jobs for the migration, shift database checkpointing in time, check log-flush intervals. The intent is not to “bend” applications, but to reduce dirty-page rate and I/O spikes during the critical phase.

Practical runbook: measurement, test migration, correlation

The following procedure is deliberately written as a repeatable checklist. The goal is an audit trail: you can later trace why a change helped (or did not).

1) Before the test: document the state

• Hosts: kernel/hypervisor version, driver levels (NIC/HBA), multipath policy
• Network: MTU, VLANs, LACP/MLAG, dedicated migration network yes/no
• Storage: protocol (NFS/iSCSI/FC), datastore/LUN, current background jobs

2) Open measurement window (host + network)

# Terminal A: I/O metrics
iostat -x -d 1 300

# Terminal B: Kernel events (storage/net)
journalctl -k -f

# Terminal C: TCP/interface counters (every 10s)
while true; do date; ip -s link; ss -s; sleep 10; done

3) Perform a test migration and record the time

Record start/end, VM name, source/target host, datastore/LUN and concurrently running migrations. These simple metadata elements will save hours later.

4) After the test: answer three questions

• Was latency visible on the block device? (await/queue increases)
• Were there network indicators? (drops, retransmits, out-of-order)
• Were there path/reset events? (multipath/scsi/iscsi logs)

Only when you can answer these three points clearly is tuning targeted. Everything else is „Trial and Error“.

Risks and side effects of tuning

Many performance changes have trade-offs. Three typical risks:

• Too aggressive timeouts can trigger unnecessary failovers during short spikes (more instability instead of less).
• Excessive bandwidth limiting may stabilize latency but prolong migrations so much that maintenance windows fail.
• Incorrect QoS use can starve other workloads or merely shift latency (e.g. from VM-A to VM-B).

Therefore: always apply changes one at a time, measurably, with a rollback plan.

Fallback strategy: If live migration does not stabilize within the window

A good operations strategy accepts that not every environment can live-migrate every VM at any time. Define in advance when to abort and how to proceed safely:

• Abort criteria: Stun > X ms, Migration > Y minutes, Storage latency > Z ms for N seconds, repeated path resets.
• Fallback 1: Orderly shut down the VM within the defined window, perform a cold migration, RESTart services under control.
• Fallback 2: Move the workload (e.g. suspend batch), retry migration later.
• Fallback 3: Extend the maintenance window or split into multiple smaller migrations (fewer parallel moves).

Important: an abort is not a failure if performed in a controlled way. Uncontrolled timeouts and recovery storms are the real risk.

Best practices for consistently stable migrations

• Monitoring for latency spikes: Not only averages, but percentiles (p95/p99) and maxima.
• Change discipline: Version and document network and storage changes (firmware, switch config, path policy).
• Regular „migration drills“: Don’t only migrate in emergencies. Small, planned tests keep paths, policies and runbooks honest.
• Segregation of traffic classes: Separate storage, migration, management and VM client traffic where possible.
• Know dependencies: Backup windows, snapshotting, rebuilds, ETL jobs — anything that causes latency spikes must be part of the plan.

Conclusion: Make latency measurable, then tune

In live migration with shared storage, „the hypervisor“ rarely decides success or failure on its own. What matters is whether storage and network paths deliver predictably low latency under migration load and whether your operations detect and mitigate the typical causes of spikes (queues, retransmits, path flaps, background jobs). If you consistently correlate metrics at the host, network and storage levels, the subjective „migration is sometimes sluggish“ becomes a clear finding — and from that a tuning that still holds months later.

Measuring storage latency and vMotion latency are also important for this topic. This article places these aspects into a clear context and shows what matters in everyday operations.