KVM Live Migration Without Shared Storage: Block Replication, Consistency and Downtime Minimization

KVM live migration without shared storage is a recurring challenge in heterogeneous datacenters, at edge sites and during planned hardware replacements when no central SAN or distributed storage backend like Ceph is available. The focus keyword KVM live migration without shared storage therefore appears right at the start: In this article I explain in practical terms which technical patterns for block replication exist, how to ensure filesystem and application consistency, and how to reduce downtime to seconds. The target audience is administrators, system engineers and operators who plan, test and run migrations.

Core concepts: What does „without shared storage“ mean and what options exist?

„Without shared storage“ means: source and target host do not share persistent block devices via a common storage backend. Shared storage denotes central systems such as SAN, NFS or distributed backends that present the same LUN or RBD device to both hosts. If this is missing, disk blocks must be actively synchronized between hosts.

The common patterns are:

• Block replication at the block layer (DRBD or NBD-based).
• Libvirt/QEMU-backed storage migration (copy-storage-all, block-copy, postcopy).
• Filesystem-oriented methods (LVM snapshots + rsync for guest image files).

Each pattern brings different operational, network, monitoring and rollback requirements.

KVM live migration without shared storage: architectural variants

DRBD: block-level replication

DRBD (Distributed Replicated Block Device) replicates block devices between hosts at the level below the filesystem. It can be operated synchronously (protocol C, secures every write), semi-synchronously (protocol B) or asynchronously (protocol A). Advantage: the device remains locally visible to the VM and the cutover can be very short. Disadvantage: operational effort for fencing (automatically isolating a misbehaving host), split-brain avoidance and regular resync checks.

# Grundlegende DRBD-Schritte (Beispiel)
drbdadm create-md r0
drbdadm up r0
# Initiale Promotion und Datenübernahme (Achtung: überschreibt Daten auf Ziel!)
drbdadm -- --overwrite-data-of-peer primary r0
cat /proc/drbd

Critical for operation: fencing (externally isolating a faulty host) and monitoring for resyncs are mandatory. Without fencing, network partition can cause split-brain — both sides believe they are Primary, and divergent write states must be merged manually.

DRBD config example (minimal)

resource r0 {
  protocol C; # synchrone Replikation
  on hostA {
    device /dev/drbd0;
    disk /dev/sdb1;
    address 10.0.0.1:7789;
    meta-disk internal;
  }
  on hostB {
    device /dev/drbd0;
    disk /dev/sdb1;
    address 10.0.0.2:7789;
    meta-disk internal;
  }
}

Why this is configured: Protocol C ensures a write is considered successful only after it has been committed on both sides — important for zero-RPO requirements. However, at high latencies C will affect application latency.

QEMU/libvirt storage migration (copy-storage-all, block-job, postcopy)

Libvirt can copy storage of running VMs. The default is a pre-copy approach: an initial volume copy, subsequent incremental copies of changed blocks, and a final cutover. Postcopy is a mode in which the VM is started on the target and missing blocks are fetched over the network on demand. Postcopy reduces the cutover window but is more susceptible to packet loss or source-host failure.

# Example: virsh migrate with copy-storage-all and optional postcopy
virsh migrate --live --verbose 
  --copy-storage-all 
  --persistent 
  --unsafe --postcopy 
  vmname qemu+ssh://targethost/system

# Check job status
virsh domjobinfo vmname

Practical tip: Use postcopy only in a controlled network and after load testing. Test failover scenarios (e.g. packet loss or brief interruption) before deploying it in production.

LVM snapshot + rsync (image-based)

If VM disks are stored as files (qcow2/raw) on the host, an LVM snapshot is a pragmatic way to produce a consistent image. Afterwards, synchronize with rsync to the target host. Disadvantages are longer downtime during the final sync and potential inconsistencies without quiesce.

# Example sequence: snapshot, rsync and cleanup
lvcreate -L 10G -s -n vmname-snap /dev/vg/vmname
rsync -av --progress /var/lib/libvirt/images/vmname-snap.img target:/var/lib/libvirt/images/
# After successful test remove snapshot
lvremove /dev/vg/vmname-snap

Consistency requirements: who needs to flush what and why?

Consistency means that the target image represents a state that filesystems and applications can correctly interpret. One distinguishes block consistency (all blocks are in a coherent state), filesystem consistency (metadata and journal are correct) and application consistency (e.g. complete DB transactions).

Key mechanisms to achieve this:

• QEMU Guest Agent: guest-fsfreeze for briefly freezing filesystems inside the guest.
• Application-specific hooks: WAL switch for PostgreSQL, flush commands for other DBMS.
• Filesystem snapshots (LVM/XFS/Btrfs) for atomic images.

# guest-fsfreeze example with virsh
virsh qemu-agent-command vmname '{"execute":"guest-fsfreeze-freeze"}'
# Application flush (example PostgreSQL)
psql -c "SELECT pg_switch_wal();"
# After completion
virsh qemu-agent-command vmname '{"execute":"guest-fsfreeze-thaw"}'

Concrete migration runbook (step-by-step)

A clear, concise runbook reduces errors during cutover phases. Here is a practical procedure for a pre-copy migration with guest-agent quiesce:

1. Preparation: check QEMU/libvirt versions, verify storage space, run network tests (iperf), ensure monitoring is active.
2. Start the initial volume copy (pre-copy).
3. Run multiple incremental copies; monitor until the change rate (dirty rate) is low.
4. Guest quiesce: guest-fsfreeze + application flush.
5. Final incremental sync and cutover (stop the VM on the source and start it on the target, or perform a live switch via libvirt).
6. Post-cutover checks: filesystem checks, application health, compare monitoring metrics.
7. If everything is stable: remove snapshot/backup on the target and mark the target as production.

# Beispiel: Cutover mit virsh (vereinfachte Darstellung)
# 1) Initiate migration
virsh migrate --live --copy-storage-all --persistent vmname qemu+ssh://targethost/system
# 2) Monitor progress
watch -n 2 virsh domjobinfo vmname
# 3) Nach Erfolg: prüfen
ssh targethost virsh list --all | grep vmname
# 4) Falls Abbruch: Abbruchbefehl
virsh migrate --abort vmname || echo "Abort attempted"

Document responsibilities for each step (who runs guest-fsfreeze, who starts the DB flush, who monitors logs).

Rollback‑ und Recovery‑Strategie

A rollback must be possible quickly and safely. Key principles:

• Keep a recovery point (snapshot or backup) before the final cutover.
• Define timeouts: if cutover takes longer than X minutes, abort and roll back to the source.
• Role change with DRBD: clear commands and checklists for promote/demote.

# DRBD: Promoten (auf Ziel) / Demoten (auf Quelle)
# Ziel promoten
drbdadm primary r0
# Quelle demoten (falls noch Primary)
drbdadm secondary r0
# Resync anstoßen
drbdadm connect r0
drbdadm status r0

If you must abort a libvirt migration, record whether the target has already modified parts of the disk. A common pattern is: stop the VM on the target, keep the snapshot on the target, continue running the VM on the source and perform a root-cause analysis.

Testing Postcopy mit Netzwerkausfall‑Simulation

Before using postcopy in production you should simulate network failures. Use tc (Traffic Control) to introduce latency, packet loss or connection interruptions:

# Beispiel: 2% Paketverlust und 100ms Latenz auf der Ziel-schnittstelle
tc qdisc add dev eth1 root netem delay 100ms loss 2%
# entfernen
tc qdisc del dev eth1 root netem

Test procedure: start a postcopy migration in the test network, inject network faults and observe whether the VM on the target remains stable or whether missing blocks cause errors. Log response times and recovery steps.

Monitoring, Metriken und Alerts

Practical metrics you should monitor:

• Dirty rate (Änderungsrate der VM-Disks) — beeinflusst Anzahl Pre-copy-Runden.
• Inkrementelle Bytes pro Job und verbleibende Bytes (über virsh domjobinfo).
• Netzwerkdurchsatz und Latenz auf Migrationsnetzwerk (iperf, SNMP).
• DRBD Resync-Status und eventuelle Backlogs.
• Guest-Agent Health, Prozess- und Services-Health im Gast (via monitoring agent).

Konkrete Alarme: wenn Pre-copy länger als erwartet läuft, wenn dirty-rate > X MB/s über Y Minuten bleibt, wenn DRBD Resync fällt oder Split-Brain erkannt wird.

Performance‑Tuning: Parameter, die wirklich helfen

A few practical tuning approaches:

• DRBD-Protokollwahl: Protocol C für Konsistenz, A/B für geringere Latenz — wählen nach RPO-Anforderung.
• QEMU-I/O-Optionen: cache=none, io=native reduzieren Host-Seiten-Caching-Effekte beim Kopieren.
• Bei qcow2: Prüfen, ob temporäre Konvertierung auf raw Kopierzeit reduziert — beachten Sie zusätzlichen Speicherbedarf.
• Netzwerk: Dediziertes Migrations-VLAN, QoS oder separate physische Verbindung für große Datenmengen.

Typische Stolperfallen und wie Sie sie vermeiden

• Guest Agent fehlt oder ist veraltet: Testen Sie guest-fsfreeze / thaw vor der Migration.
• Unterschätzte Änderungsrate: Messen Sie dirty-rate im Vorfeld, planen Sie mehrere Pre-copy-Zyklen ein.
• Network path overloaded: define a separate migration network or a QoS policy.
• DRBD without fencing: test Split‑Brain recovery and deploy SBD/STONITH.
• qemu/libvirt version incompatibilities: compare versions in advance and perform migration tests.

Checklist before production migration

1. Backup: current backup and validation available.
2. Compatibility check: CPU models, QEMU/libvirt versions, image formats.
3. Network check: latency, bandwidth, QoS configured.
4. Guest agent and application hooks verified.
5. Monitoring and alerting for migration metrics enabled.
6. Rollback runbook known and confirmed by team members.
7. Dry run executed in staging.

Conclusion: selection criteria and operational maturity

Choose DRBD when you need short cutover times and are prepared to operate fencing and split‑brain procedures. Choose libvirt –copy-storage-all / block-copy for one‑off migrations without additional storage infrastructure, provided network and tests are adequate. LVM snapshots plus rsync are suitable for scenarios with predictable maintenance windows and less stringent downtime requirements.

Operational maturity is decisive: test each method in a staging environment with a comparable I/O profile, document runbooks, automate pre‑checks and keep manual intervention available for cutover phases. Only then will you reduce downtime for critical workloads to a minimum while retaining control over consistency and recovery.

With systematic testing, clear verification and rollback strategies, and monitoring, you can operate KVM live migrations without shared storage safely and reliably.

KVM live migration without shared storage: operation, security and orchestration

Beyond the pure technique of block replication, operation, security and orchestration determine the success of production migrations. The following points extend the previous technical patterns with practical operational rules, integration aspects and automation approaches that have proven effective in projects with custom enterprise software and critical services.

Security and compliance aspects

Replication and migration traffic carries complete VM states. Protect this data strictly: encryption in transit (IPsec, WireGuard or TLS tunnels) and authentication of peers are mandatory, especially for asynchronous replication. For encrypted VMs (LUKS) clarify key transport: the target host must have key material or a remote‑unlock mechanism before cutover. A common mistake is to start the migration and only discover key problems when booting on the target.

• Recommendation: dedicated migration network with ACLs and VPN; enable logging for audit trails.
• For DRBD: do not expose management ports to the network; use access control and monitoring authentication.

Consistent multi‑VM or cluster migration

Distributed applications (e.g. database clusters, distributed caches) require coordinated migrations. Procedure in brief:

1. Orchestrator/runbook starts scheduled quiesce hooks on all involved VMs (application flush, Guest‑Agent).
2. Pre‑copy increments run until the dirty rate decreases.
3. Final quiesce and synchronous cutover of all nodes with defined timeouts.

Without coordination you risk split-brain at the application level or inconsistent transactions. A simple mutex token (e.g. in etcd) reduces the risk, since cutover must only occur when all nodes have acknowledged.

Automation: small orchestration example

Automated hooks reduce hands-on errors. Below is a minimal Ansible task that orchestrates guest-fsfreeze and a DB flush (as a template, adapt to your environment):

- name: Quiesce VM and switch WAL
  hosts: controlhost
  tasks:
    - name: Freeze guest filesystem
      command: virsh qemu-agent-command vmname '{"execute":"guest-fsfreeze-freeze"}'
    - name: Trigger DB WAL switch on guest
      command: ssh dbuser@guest "psql -c 'SELECT pg_switch_wal();'"
    - name: Thaw guest filesystem
      command: virsh qemu-agent-command vmname '{"execute":"guest-fsfreeze-thaw"}'

Automation must be idempotent and have clear error-handling paths: on errors, rollback triggers and notification to the on-call team.

Capacity planning and SLA calculation

Plan migrations with a simple formula: estimated duration ≈ (initial data volume / available net bandwidth) + (estimated cumulative changed data / bandwidth) + cutover time. Measure the dirty rate beforehand and use these values for realistic windows. Set alert thresholds: if pre-copy runs longer than expected or the dirty rate remains above threshold, automatic abort or escalation.

Validation after migration

After cutover, verify not only that the VM is running but validate application transactions, integrity checks (checksums, DB health) and latency/throughput metrics against baselines. Automate smoke checks and compare telemetry before permanently decommissioning the source.

These additions help operationalize KVM live migrations without shared storage: security, coordinated orchestration and measurable SLAs make migrations predictable and auditable.

Libvirt Copy-Storage-All are also important for this topic. The article places these aspects in context and shows what matters in day-to-day operations.