Boot issues in NVMe-only setups: verify UEFI, initramfs and fstab

Boot problems in NVMe-only setups occur in production environments frequently after kernel updates, storage changes or firmware updates. This runbook is intended for administrators, system engineers and operators and describes a validated sequence of checks: visibility of the NVMe hardware, UEFI/ESP, bootloader and kernel cmdline, initramfs contents and /etc/fstab. Each action includes the relevant risks, practical commands and a rollback strategy.

Why proceed systematically? The interaction of the boot chain

The boot chain is not a single artifact but several layers that cooperate: the firmware (UEFI) reads the EFI system partition (ESP) and starts an EFI loader; the bootloader (e.g. GRUB2 or systemd-boot) loads the kernel and initramfs and passes the kernel cmdline; the initramfs (a temporary root filesystem) initializes drivers and tools (NVMe drivers, LVM, mdadm, cryptsetup) and mounts the root filesystem; only after that does /etc/fstab take over further mounts. Failures in one layer often appear as hardware faults — therefore the order of checks is important.

Classify symptoms and weigh consequences

Before making changes, classify the symptom. This saves time and prevents incorrect interventions:

• Firmware/UEFI reports “No bootable device”: focus on the ESP, NVRAM entries and partition types.
• Bootloader starts the kernel, afterwards drop to initramfs shell: focus on the kernel cmdline and initramfs contents.
• Root mounted, later systemd Emergency Mode: /etc/fstab, missing mounts or timeouts are likely.

Precautions before intervention

Minimize risks, limit downtime:

• Provide a remote console (IPMI/iDRAC/iLO/VM console) so you can view and control boot messages.
• Keep older kernel entries; do not delete anything you cannot reproduceably back up.
• Create copies of the ESP and initramfs before writing.
• Lock critical changes to maintenance windows and document each step.

Check step 1: NVMe hardware and kernel visibility

Goal: Verify whether the system and kernel see the NVMe devices. If /dev/nvme* devices are missing, no bootloader repair will help.

# Geräte und Partitionen anzeigen
lsblk -e7 -o NAME,TYPE,SIZE,MODEL,SERIAL,FSTYPE,UUID,MOUNTPOINTS

# Dateisystem-UUIDs
blkid

# Kernel-Meldungen nach NVMe/PCIe-Fehlern durchsuchen
dmesg | grep -iE 'nvme|pcie|iommu|timeout|reset' | tail -n 200

Interpretation: If lsblk shows no NVMe devices, check BIOS/UEFI settings (PCIe mode, ACS/ASPM, hotplug), firmware updates for the motherboard/NVMe controller or physical connections (backplane). Sometimes the rescue kernel lacks the appropriate NVMe driver.

Check step 2: UEFI, ESP and NVRAM entries

The ESP (EFI system partition) must be a valid partition with type EF00 (GPT) and formatted as FAT32. If the ESP is damaged or contains incorrect paths, UEFI will not find a loader.

# ESP mounten und Inhalt prüfen
mkdir -p /mnt/esp
mount -t vfat /dev/nvme0n1p1 /mnt/esp
ls -la /mnt/esp

# NVRAM-Einträge anzeigen (Rescue muss im UEFI-Modus gebootet sein)
efibootmgr -v

Pitfalls:

• When cloning an NVMe the partition table can change; NVRAM entries may then point to non-existent PARTUUIDs.
• Secure Boot can block unsigned loaders; check signature and MOK status.

Step 3: Bootloader, kernel cmdline and root=

The kernel cmdline defines which device serves as root. If root= is set incorrectly, the system drops immediately into the initramfs shell. Check the bootloader configuration, not just the currently running kernel argument.

# Aktuelle Kernel-Cmdline prüfen
cat /proc/cmdline

# GRUB-Konfiguration im gemounteten System untersuchen
grep -R "Linux .*root=" -n /mnt/sysroot/boot/grub*/grub.cfg | head -n 50

# systemd-boot Einträge lesen
find /mnt/esp/loader -maxdepth 2 -type f -name "*.conf" -exec sed -n '1,120p' {} ;

Practical tip: Use UUID= or PARTUUID= in root=; /dev/nvme0n1p2 can shift after hardware changes. PARTUUID refers to the partition table entries and remains more stable during repartitioning.

Step 4: initramfs — inspect contents, regenerate and error sources

The initramfs is a temporary root containing the drivers and utilities required to prepare the root filesystem. Two common tools generate the initramfs: update-initramfs (Debian/Ubuntu) and dracut (RHEL/Alma/Rocky). If, for example, the NVMe driver or cryptsetup is missing, the system cannot unlock or mount.

# Beispiel: initramfs-Inhalt prüfen (Debian/Ubuntu)
lsinitramfs /mnt/sysroot/boot/initrd.img-$(ls /mnt/sysroot/lib/modules | sort -V | tail -n 1) | grep -iE 'nvme|lvm|crypt|mdadm|ext4|xfs|btrfs'

# Beispiel: initramfs-Inhalt prüfen (RHEL/Rocky/Alma)
lsinitrd /mnt/sysroot/boot/initramfs-*.img | grep -iE 'nvme|lvm|crypt|mdraid|ext4|xfs|btrfs'

Regenerate the initramfs from the target system’s chroot and always back up the old file:

# In chroot (Debian/Ubuntu)
cp -a /boot/initrd.img-$(uname -r) /boot/initrd.img-$(uname -r).bak.$(date +%F)
update-initramfs -u -k all

# In chroot (RHEL/Rocky/Alma)
KVER=$(ls /lib/modules | sort -V | tail -n 1)
cp -a /boot/initramfs-${KVER}.img /boot/initramfs-${KVER}.img.bak.$(date +%F)
dracut -f /boot/initramfs-${KVER}.img ${KVER}

Common causes of errors when regenerating:

• /etc/crypttab is missing or incorrect: cryptsetup parameters are not included in the initramfs.
• LVM PVG filters in /etc/lvm/lvm.conf prevent volume groups from being detected.
• Dracut modules were disabled via custom dracut.conf; check /etc/dracut.conf.d.

Step 5: Clean chrooting and activating dependencies

Work, where possible, in the target system’s chroot. This prevents system paths from the rescue environment from distorting the repair. Bind /dev, /proc, /sys and /run.

mount /dev/nvme0n1p2 /mnt/sysroot
mount -t vfat /dev/nvme0n1p1 /mnt/sysroot/boot/efi
mount --bind /dev  /mnt/sysroot/dev
mount --bind /proc /mnt/sysroot/proc
mount --bind /sys  /mnt/sysroot/sys
mount --bind /run  /mnt/sysroot/run
chroot /mnt/sysroot /bin/bash

If LUKS/LVM/RAID are used, open/activate them beforehand:

# LUKS öffnen
cryptsetup luksOpen /dev/nvme0n1p3 cryptroot

# LVM aktivieren
pvscan
vgscan
vgchange -ay

# mdadm-RAIDs zusammenbauen
mdadm --assemble --scan

/etc/fstab check: stability rather than fragile device files

/etc/fstab controls persistent mounts. Faulty or blocking entries are a common cause for later boots ending up in emergency mode, even if root was mounted correctly.

# fstab und Geräteliste prüfen
cat /etc/fstab
lsblk -f
blkid

Recommendations:

• Use UUID= or PARTUUID= instead of /dev/nvme0n1pX, as /dev device names can shift when enumeration changes.
• For non-critical mounts set nofail temporarily so the boot does not stop on every failed mount.
• For network mounts or volumes at risk of spindown use x-systemd.automount.
• Check resume and swap entries: incorrect resume devices cause long timeouts.

# Beispiel fstab-Eintrag mit UUID und nofail
UUID=1111-2222  /data   ext4  defaults,nofail,x-systemd.device-timeout=10  0 2

Timing and initialization issues

Some NVMe controllers or backplanes require more time to initialize. The initramfs by default attempts to find devices for a fixed period. For slow initialization you can temporarily set rootdelay=30 or specific retry parameters until firmware/BIOS fixes are implemented. Permanent measures are firmware updates, BIOS settings (e.g. disable Fast Boot) or physical changes to the hardware.

Special cases: Secure Boot, signed kernels/loaders and MOK

Secure Boot verifies signatures of EFI loaders and kernels. If you use custom kernels or unsigned loaders, boot will fail without dropping to the initramfs shell. Check:

• Whether the loader in the ESP is signed.
• Whether the kernel has a valid signature (when using Shim/MOK).
• Whether MOK (Machine Owner Key) was enrolled and accepted.

If necessary you can temporarily disable Secure Boot to perform repairs — observe security policies and log the action.

Bootloader repair: GRUB vs. systemd-boot

Repair procedures differ depending on the bootloader. Examples:

# GRUB UEFI neu installieren (im chroot, ESP gemountet)
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB
update-grub

# systemd-boot installieren
bootctl --path=/boot/efi install

Important: Document efibootmgr -v before making changes and create backups of the ESP. An incorrect grub-install can overwrite NVRAM entries.

Docker hosts: why NVMe boot issues are particularly critical

Container hosts are often minimal and run with many static volumes. Boot failures affect not only individual services but container orchestration, volume availability and logging. Additional aspects:

• Docker (or containerd) uses the host root filesystem for container storage (e.g. /var/lib/docker). Problems mounting root lead to read-only or missing volumes.
• overlay2 and device-mapper are vulnerable to inconsistent filesystem states; a corrupted initramfs that mounts root late can cause container bottlenecks during startup.
• In clustered setups orchestrators synchronize state and should provide health checks to trigger failover.

Practical checks on a Docker host (after a successful chroot):

# Docker-Status prüfen
systemctl status docker
docker info | sed -n '1,120p'

# Storage-Pfade überprüfen
ls -la /var/lib/docker
df -h /var/lib/docker

Best practices before kernel/boot changes:

• Back up container volumes (snapshots, rsync, registry pushes for images).
• Test boot changes first on a replica or canaries.
• Treat boot artifacts like configuration objects: version, review and record changes.

Recovery and fallback strategy

If the cause remains unclear, act minimally and in a reversible way:

1. Try booting with an older kernel instead of rebuilding.
2. Set critical fstab entries to nofail rather than removing them.
3. Back up the ESP and initramfs, export the efibootmgr output.
4. Work stepwise: first NVMe visibility, then bootloader/kernel cmdline, then initramfs and fstab.
5. In emergencies: provision a replica from backup/imaging and plan migration of the IP/services.

Practical checklist (quick reference)

1. Determine symptom class: UEFI / initramfs / fstab.
2. Is the NVMe visible? (lsblk, dmesg)
3. Prepare chroot and activate LUKS/LVM/RAID.
4. Mount ESP, save efibootmgr -v.
5. Check kernel cmdline: root=UUID/PARTUUID.
6. Inspect initramfs contents and regenerate from chroot (backup).
7. Compare /etc/fstab with blkid, use nofail.
8. Reinstall bootloader if necessary (grub-install / bootctl).
9. Test boot, check logging, write postmortem and change record.

Conclusion

For boot problems in NVMe-only setups, successful diagnosis is a combination of methodical procedure, knowledge of the boot chain and conservative action. NVMe is rarely the sole culprit; UEFI/NVRAM, kernel cmdline, missing initramfs modules or faulty fstab entries usually interact. Work from a chroot, back up boot artifacts, regenerate initramfs with care and plan fallback measures. For Docker hosts there are additional strict requirements on volumes and orchestrator resilience — test changes first on replicas or canary hosts.

Following the verification sequence and the concrete commands described here reduces downtime and avoids unnecessary reinstallations. Documentation and change control are as important as the technical repair.

Boot problems in NVMe-only setups: operation, monitoring and automation

Beyond classic repair, it is crucial to design operations so that boot failures are detected early, reproducibly tested and safely rolled back. Three areas consistently pay off in practice: artifact backup (ESP, NVRAM, GPT), observable boot telemetry and automated validation in the CI/CD pipeline.

Artifact backup and securing metadata

Always back up the EFI partition, the NVRAM entries and the GPT table before making changes. These metadata are often the fastest way to restore a working state.

# GPT backup
sgdisk --backup=gpt-backup-$(date +%F).bin /dev/nvme0n1

# Backup NVRAM/efibootmgr
efibootmgr -v > /root/efibootmgr-$(date +%F).txt

# ESP snapshot
mount /dev/nvme0n1p1 /mnt/esp
tar -c -C /mnt/esp . | gzip -c > /root/esp-backup-$(date +%F).tar.gz

Observability: early logging and remote access

• Enable a serial console or netconsole for early kernel messages. This lets you see whether the NVMe controller is initialized.
• Configure systemd-journal to be persistent so logs are retained across boots.
• For encrypted root volumes, plan a remote unlock path (e.g. SSH-Dropbear in the initramfs or a central key escrow) and strictly control access and auditing.

CI/CD and Canary strategy for kernel/initramfs

Build initramfs and bootloader artifacts reproducibly in your pipeline and automatically check whether necessary modules (nvme, nvme_core, lvm, dm‑crypt etc.) are included. Distribute kernel and firmware updates first to canary hosts with monitoring of boot time, dmesg errors and container health (for Docker hosts).

Special element: LUKS‑Header and key management

# LUKS-Header sichern
cryptsetup luksHeaderBackup /dev/nvme0n1p3 --header-backup-file /root/luks-header-$(date +%F).bin

Ohne Header‑Backup sind LUKS‑Volumes nach unglücklichen Schreiboperationen oft nicht mehr wiederherstellbar. Dokumentieren Sie Prozesse und halten Sie Schlüssel/Backups in einem gesicherten Vault mit Zugriffskontrolle.

These operational measures significantly reduce outage risk: backing up metadata, enabling early telemetry, building reproducible artifacts and rolling out in stages are often more effective than ad‑hoc repairs in an emergency.

UEFI boot and regenerating initramfs are also important for this topic. The article places these aspects in a clear context and shows what matters in everyday operations.