When a KVM‑VM suddenly no longer starts, more is often at stake than a one‑off reboot. In this article I present a systematic fault analysis and concrete repair commands so that administrators, system engineers and operators can proceed safely and with awareness of risk. The focus keyword „KVM-VM no longer boots“ is stated at the outset, because the diagnostic steps described here apply to pure libvirt/QEMU setups as well as Proxmox and other KVM hosts.
KVM-VM no longer boots: Why a structured fault analysis is important
An unplanned boot failure can range from a simple configuration error to storage corruption. Without a structured approach, data loss or unnecessary downtime may occur. In practice, rapid identification, safe read‑only modes and a clear rollback strategy are essential.
Overview: Possible causes (at a glance)
- Host services stopped (e.g. libvirtd, qemu‑kvm)
- Storage issues: LVM‑Thin full, NFS/iSCSI lost, Ceph OSD down
- Disk image damaged (qcow2/raw) or snapshot inconsistency
- Bootloader/EFI/Grub missing or initramfs corrupted
- Network/Cloud‑Init boot events that block startup
- Hardware/kernel panic in the guest — no console configured
Preparation: Work safely and minimize risks
Before executing repair commands that write changes, you should create a backup or at least a copy of the affected disk image. Many repair tools are powerful but not reversible. If possible, work on a copy.
Example: Create a copy of a qcow2 file (local):
cp -v --reflink=auto vmdisk.qcow2 vmdisk.qcow2.bakComment: reflink=auto uses CoW on filesystems like XFS or btrfs if available; otherwise a normal copy is created. For large images this is often faster and more space-efficient. If you work on network storage (NFS/SMB), check available free space beforehand.
Initial checks on the host
Start with host checks — many problems are not caused by the guest but by the host.
1) Status of the virtualization services
systemctl status libvirtd.service qemu-kvm.service --no-pagerWhy: libvirtd manages VMs via libvirt; if the service is stopped or errors appear in journalctl, VMs will not start. On Proxmox the service is called pvedaemon or pveproxy — check these as well.
2) Search host logs
journalctl -u libvirtd -n 200
journalctl -k -n 200Why: dmesg/journal provides indications of block device errors, kernel I/O errors, or missing drivers. If the host reports I/O errors, a storage problem is likely.
3) Check the storage layer
Check LVM, ZFS, Ceph, NFS or iSCSI depending on your setup. Examples:
# LVM Pools
lvs -a -o +lv_name,lv_attr,lv_size,lv_free
# ZFS Pools
zpool status
# Ceph
ceph -sWhy: LVM Thin can run out of space, ZFS pools can be degraded, Ceph may have missing OSDs. If the backend is unavailable, the VM will not start or will hang when attaching the disk.
Check the VM configuration
Errors in the VM XML (libvirt) or QCOW snapshot chains often cause boot failures.
1) View domain info and XML
virsh dominfo vmname
virsh dumpxml vmname > /tmp/vmname.xmlWhy: dumpxml shows the disk path, firmware (BIOS vs. OVMF/UEFI), console configuration and attached devices. Watch for incorrect disk paths, missing backing stores for qcow2, or incorrect driver-type settings.
2) Check device file paths
ls -lh /var/lib/libvirt/images/vm-100-disk-1.qcow2
qemu-img info /var/lib/libvirt/images/vm-100-disk-1.qcow2Why: qemu-img info provides the format (qcow2/raw), virtual size and snapshot chain. If a backing file is missing, qemu cannot start.
Enable guest console and read logs
The serial console often helps to see boot messages in the guest. The serial console is a simple text access to the kernel/GRUB output.
1) Use virsh console
virsh console vmname
# Bei Bedarf: Escape-Sequenz ~. zum BeendenWhy: If the guest has a console ttyS0 configured, you will see kernel and initramfs messages. If the console is missing in the guest, this method will not work — then use the disk-attach procedure below.
2) Identify the sequence of boot hooks
Read the messages: does it hang at GRUB, in the initramfs (e.g. busybox shell) or later in the kernel (kernel panic)? Each phase has its own indicators.
If the disk image is suspicious: diagnostics and safe checks
If the VM hangs when mounting the root partition or the initramfs throws errors, check the disk image on a rescue host.
1) Check the image integrity
qemu-img check -r all /var/lib/libvirt/images/vm-disk.qcow2Why: qemu-img check analyzes qcow2 structures. Warning: repairs should only be performed after a backup. If errors are found, first create a copy of the image.
2) Mount the image read-only (guestfish/guestmount)
# Nur lesen, mit libguestfs
guestfish --ro -a /var/lib/libvirt/images/vm-disk.qcow2 -i -- command :
# Alternativ mounten mit libguestfs guestmount
guestmount -a /var/lib/libvirt/images/vm-disk.qcow2 -i /mnt/vmroot --roWhy: guestfish / guestmount (libguestfs) allows access to partitions inside a VM image without kernel loop manipulation. This lets you inspect /etc/fstab, kernel‑cmdline or cloud‑init logs. Use –ro to avoid accidentally modifying the image.
3) Loopdev & kpartx verwenden (Alternative)
losetup -f --show /var/lib/libvirt/images/vm-disk.raw
kpartx -av /dev/loopX
mount /dev/mapper/loopXp1 /mnt/vmroot -o roWhy: If libguestfs is not available, you can convert the image (qcow2 → raw) and mount it via a loop device. Conversion takes time and space; work on copies.
Reparatur im Dateisystem oder Bootloader
If you can access the root partition, repairs are possible: fsck, regeneration of initramfs or reinstallation of GRUB.
1) Filesystem reparieren (immer auf Kopie!)
# Beispiel für ext4 (auf gemounteter oder loopdev ersetztem Device)
e2fsck -f -y /dev/mapper/loopXp1Why: e2fsck repairs filesystem errors. -f forces a check, -y answers Yes automatically — use -y only if you understand the consequences. For XFS use xfs_repair; note that XFS typically cannot be repaired in read‑only mode.
2) Initramfs neu erstellen und GRUB wiederherstellen
If the bootloader or initramfs is damaged, chroot into the image and regenerate:
# Beispielablauf nach Mount der Partitionen
mount --bind /dev /mnt/vmroot/dev
mount --bind /proc /mnt/vmroot/proc
mount --bind /sys /mnt/vmroot/sys
chroot /mnt/vmroot /bin/bash
update-initramfs -u -k all
grub-install --target=i386-pc /dev/sda
update-grub
exit
umount -l /mnt/vmroot/{dev,proc,sys}
Why: update-initramfs builds the required initramfs that loads runtime modules and drivers at kernel boot. grub-install / update-grub writes a working bootloader. For UEFI VMs pay attention to OVMF/EFI configuration and use grub-install –target=x86_64-efi on an EFI system if applicable.
Risk: chroot and grub-install modify disk contents. Therefore create a backup beforehand.
Wenn es an Firmware (UEFI / OVMF) liegt
Many modern VMs use OVMF (UEFI firmware for QEMU). If OVMF binaries are missing or incorrectly referenced, the VM will not boot.
# Prüfen ob OVMF vorhanden ist
ls -l /usr/share/OVMF /usr/share/ovmf /usr/share/qemu/OVMF* /usr/share/ovmf/*
# Beispiel: libvirt XML zeigt Why: Missing files or incorrect permissions prevent the firmware from loading. Check for package upgrades or provider changes that may have moved OVMF.
Wenn die VM nach Kernel‑Änderung nicht startet
Updates to the guest kernel or initramfs may require rollbacks. If you have snapshots, check the option to roll back; otherwise boot into a rescue environment and switch the initrd or kernel version.
# In chroot: alte Kernelpakete auflisten und ggf. wieder installieren
apt list --installed | grep Linux-image
apt install Linux-image-
update-grub
Why: Package upgrades can deliver incompatible drivers/modules. A rollback is often the fastest solution if it can be done promptly.
Wenn Netzwerk/Cloud‑Init den Boot blockiert
In Cloud-Init-based images, faulty network configuration can delay or stop the boot (e.g. systemd timeout when requesting a DHCP address). Check /etc/cloud/cloud.cfg and Netplan/ifupdown configurations inside the image.
# Check cloud-init logs (via guestmount/guestfish)
cat /var/log/cloud-init.log
cat /var/log/cloud-init-output.logWhy: Cloud-Init can cause network waiting; often adjusting to a static configuration or a more tolerant NetworkManager/Netplan setting helps.
Recovery attach: attach VM disk to a rescue VM
If repairs inside the image are necessary, attaching the disk to a working rescue VM is the safest method.
# Example libvirt: attach disk to rescue VM
virsh attach-disk rescue-vm /var/lib/libvirt/images/vm-disk.qcow2 vdb --driver qemu --subdriver qcow2 --persistent
# Alternative: in Proxmox GUI or qm set / qm importdiskWhy: This lets you use known tools in the rescue VM to repair the filesystem or read logs without starting the target VM.
If all else fails: RESTore disk snapshot and rollback plan
Keep a rollback path ready: existing snapshots, backups, or storage replicas. Predefine the minimally required steps to achieve a service RESToration (e.g. start the VM on a different host, import the disk from a replica).
Rollback checklist
- Locate a valid backup/snapshot
- Check space and I/O on the target host
- Import/RESTore the disk and test on a separate host
- Perform service tests (SSH, application sanity)
- Communication: downtime, issues, rollback time window
Typical pitfalls and how to avoid them
- Snapshot chains: too many qcow2 snapshots increase complexity — consolidate with care.
- Thin provisioning: LVM/ZFS thin can suddenly run out of space — configure alerts.
- Incompatible OVMF versions after host upgrade — check OVMF binaries.
- Missing console: without a serial console kernel panic messages are not visible — configure the console permanently.
- Typos in libvirt XML (path, driver name) — always verify with dumpxml.
Practical example: qcow2 image has backing file missing
Symptom: VM does not start, qemu reports missing backing file.
# Diagnosis
qemu-img info vm-disk.qcow2
# Output shows backing file: /var/lib/libvirt/images/base.qcow2 (missing)
# Solution: replace or recreate backing file
cp /backup/base.qcow2 /var/lib/libvirt/images/
chown libvirt-qemu:kvm /var/lib/libvirt/images/base.qcow2
# Alternatively: rebase to remove the backing reference path (on a copy!)
qemu-img rebase -u -b "" vm-disk.qcow2
Why: qcow2 can operate with a backing file; if that is missing the image is inconsistent. qemu-img rebase removes the link to the backing file — this only works if the image data is self-consistent, so take a backup first.
Monitoring and prevention notes
To avoid future incidents, implement the following:
- Alerting for storage fill (LVM/ZFS/thin/datastore)
- Regular scrubs/checks (e.g. ZFS scrub, Ceph health checks)
- Automated test RESTores during maintenance windows
- Serial console as standard in VM templates
- Documented recovery runbooks for critical VMs
Conclusion and priority list for troubleshooting
When a KVM VM no longer boots, proceed in a structured manner: host services, storage, VM configuration, console and disk integrity. Always work on copies, document every step and provide a rollback option. In most cases the fault can be resolved with a combination of read-only analysis (guestmount/guestfish), filesystem repair and, if necessary, regeneration of initramfs/GRUB.
If you want a quick checklist, start with these steps:
- Check host logs and services (libvirtd/qemu, dmesg)
- Check storage backend (LVM/ZFS/Ceph/NFS)
- Verify VM XML and disk path
- Enable the serial console and read logs
- Safely mount the image read-only and inspect contents
- On a copy: fsck, regenerate initramfs and reinstall GRUB
This post is intended as a practical runbook introduction: technically precise, with concrete commands and always focused on operational safety and recovery strategies.