Disaster Recovery Playbook for Proxmox: Rebuilding a cluster from PBS, VM images and configuration

An incident Disaster-Recovery Playbook for Proxmox is indispensable for every operations manager: this playbook provides a hands-on description of how to rebuild a Proxmox cluster after total loss or a severe outage using Proxmox Backup Server (PBS), VM images and configuration files. It is aimed at administrators, system engineers, operators and technical IT service providers and focuses on operations, interfaces, checks and rollback strategies. The focus keyword Disaster-Recovery Playbook for Proxmox is deliberately placed in the introduction because it precisely reflects the search intent and the practical value of this article.

Disaster-Recovery Playbook for Proxmox: Why a playbook and when it applies

A playbook creates clarity under stress: it defines sequence, responsibilities, timing and checks. It applies when the production cluster no longer boots, /etc/pve is lost, Corosync cannot reach quorum, or the primary storage is irreparably damaged. A central prerequisite is that you have valid PBS backups and copies of the cluster configurations (for example from an automated export routine).

Terminology (brief)

Proxmox VE is the virtualization platform. Proxmox Backup Server (PBS) is the dedicated backup appliance for snapshots/deduplicated backups. pmxcfs is Proxmox’s distributed configuration filesystem (control plane config), Corosync is the quorum/cluster communication layer. VM configurations reside in /etc/pve/qemu-server/ (KVM) and /etc/pve/lxc/ (LXC).

Preparation: prerequisites, roles and priorities

Before you start a RESTore, clarify the following points:

• RTO/RPO per VM category: define which VMs must be rebuilt immediately and which can be RESTored later.
• Available media: access to the PBS repository, archive copies and, if applicable, offsite copies.
• Roles: Incident Lead, Storage Operator, Network Operator, PBS Operator, App Owner.
• Communication: channels, status updates and change-freeze rules during recovery.

Prioritize: start with control-plane recovery, then storage, followed by critical VMs and finally less critical systems.

Initial analysis: determine the scope

Analyze the scope of the outage systematically: hardware failed, pmxcfs corrupted, network partition, or PBS lost? Check whether individual nodes are reachable and whether PBS repos are accessible via SSH/HTTPS. Document the current state and maintain an incident log.

RESTore the control plane (pmxcfs/Corosync)

The control plane manages cluster configuration. If at least one node is intact, check the status:

pvecm status
systemctl status pve-cluster
systemctl status corosync

If no node is available, build a new first node (seed node). Pay strict attention to hostname and IP plan, because Corosync and DNS/hosts require consistent name resolution. Initialize a cluster only if you have no other source for pmxcfs:

pvecm create mycluster

Note: creating a new cluster will overwrite any existing pmxcfs. If you have a secured copy of /etc/pve or an export file, prefer RESToring those data instead of creating a new cluster.

Reconstructing pmxcfs from backup

If you have a consistent backup of /etc/pve or an export of pmxcfs, RESTore those files onto the seed node. A typical procedure is a tar archive of the /etc/pve structure, which you copy to the seed node and then RESTart the pve-cluster service:

# auf Seed‑Node
rsync -av --progress /mnt/backup/etc-pve-backup/ /etc/pve/
systemctl RESTart pve-cluster
systemctl RESTart corosync

Why: /etc/pve is the control plane; its files define VM IDs, storage mappings and networking. Risks: inconsistent version states or missing storage aliases can prevent nodes from joining correctly.

Corosync, Quorum und Fencing

Quorum issues are common pitfalls. Avoid split‑brain by testing fencing (automatic isolation of a node that the cluster considers faulty). If you use a QDevice (external quorum device), ensure its reachability.

Check quorum status:

pvecm status
# oder
corosync-cmapctl | head

If a node is no longer reachable, remove it cleanly or adjust the expected votes, but document every step and have rollback commands ready.

PBS: Zugriff prüfen und Repository wiederherstellen

Ensure that PBS repositories are available. Verify reachability, certificates and user access. If PBS itself was destroyed, reinstall PBS and synchronize the datastore files (rsync or storage-replica). Copy files only if you understand how PBS uses internal data structures (chunks, indexes)—incomplete copying can corrupt repositories.

# Beispiel: rsync eines PBS‑Datastores (nur, wenn Storage‑Kopie valide ist)
rsync -av --progress /mnt/offsite/pbs-datastore/ /var/lib/proxmox-backup/datastore/

Validation: Use the PBS WebUI or CLI to list datastores and snapshots before starting RESTores.

VM‑RESTore‑Strategien und detaillierte Umsetzung

Choose between three approaches depending on the situation:

1. qmRESTore: Direct import of vzdump archives; creates VM configuration and disks.
2. qm importdisk + qm set: Import raw disk images and subsequent manual VM configuration.
3. pct RESTore: For LXC containers with bind‑mount checks.

qmRESTore – Empfehlung für getestete VMs

qmRESTore /backup/vzdump-qemu-101-2026_01_01.vma.zst 101 --storage local-lvm
# optional: sofort starten
qm start 101

Risks: qmRESTore expects matching storage IDs. If the target storage is named differently, create temporary storage aliases or use qm importdisk.

qm importdisk – flexibler bei Storage‑Mapping

qm create 101 --name RESTored-101 --memory 4096 --cores 4 --net0 virtio,bridge=vmbr0
qm importdisk 101 vm-101-disk-1.qcow2 local-lvm
qm set 101 --scsi0 local-lvm:vm-101-disk-1
qm set 101 --boot c --bootdisk scsi0

Advantage: You retain control over thin/thick pools and storage types. After the import the VM meta-configuration (cloud‑init, qemu‑agent) may be missing. Add these manually or via a template script.

Container (LXC) – pct RESTore

pct RESTore 102 /backup/vzdump-lxc-102-2026_01_01.tar.lzo --storage local
pct set 102 --net0 name=eth0,bridge=vmbr0,ip=dhcp
pct start 102

Check bind‑mounts, AppArmor profiles and privileges after the RESTore.

Datenbanken und konsistente Backups

Database VMs require carefully tuned recovery steps: either you have consistent snapshots (quiesced backups) or you perform additional DB recovery steps (WAL‑Replay, replication rebuild). For PostgreSQL, for example, check WAL segments and, if necessary, execute PITR steps. For MySQL check binlog positions and, where possible, prefer reinitializing replication over complex binlog repair.

Encryption: LUKS‑headers and key management

If disk volumes were encrypted, LUKS headers and key material are critical. Store LUKS headers separately. If headers are damaged, data access is often impossible. Document and test remote-unlock mechanisms (e.g. Network‑Unlocked LUKS or key servers).

Storage types: LVM, ZFS, Ceph – specifics

Each storage type has its own recovery rules:

• LVM: Activate volume groups and check thin pools. Use lvdisplay and vgchange -ay.
• ZFS: Import pools with zpool import -f and check for errors with zpool scrub.
• Ceph: Bring monitors (MON) up first, check ceph -s and RESTore OSDs. Ceph cluster recovery can be complex; plan sufficient time and follow Ceph best practices.

# Beispiel: LVM aktivieren
vgchange -ay
lvdisplay
# Beispiel: Ceph Status prüfen
ceph -s

Configuration conflicts: storage IDs, MACs and networks

Common problems after RESTore are storage ID mismatches and duplicate MAC addresses. Check /etc/pve/qemu-server/*.conf for outdated storage identifiers and adjust them. Example of a VM‑conf (excerpt):

# /etc/pve/qemu-server/101.conf
boot: cdn
cores: 4
memory: 4096
scsi0: local-lvm:vm-101-disk-1,size=50G
net0: virtio=AA:BB:CC:DD:EE:FF,bridge=vmbr0

If MAC addresses cause conflicts, change them before boot and document the adjustments. Use qm set to make changes instead of direct file manipulation, when possible.

Validation and Smoke‑Tests (detailed)

Validation is not optional. For each RESTored VM, perform at minimum:

1. Check the serial console/boot log for kernel errors (GUI/Web or qm monitor).
2. Network tests: ping, traceroute, DNS resolution and port checks.
3. Application smoke: HTTP/HTTPS health, DB connection tests, check background jobs.
4. Data integrity: checksums of important directories or application checks (e.g. SELECT COUNT(*) for tables with a reference value).

# Beispiel Smoke‑Test
ssh admin@vm-ip 'systemctl is-active postgresql && curl -sfS http://localhost:8080/health'
# Prüfsummevergleich
ssh admin@vm-ip 'sha256sum /var/lib/app/data/important.db | cut -d" " -f1'

Automation and scripts: robust, idempotent and audited

Automate recurring tasks, but not critical decisions. An idempotent Bash script for batch RESTore reduces errors, logs steps and allows controlled execution. Example (extended):

#!/bin/bash
# batch_RESTore_checked.sh
set -euo pipefail
MAPFILE=${1:-RESTore-map.csv}
LOG=/var/log/proxmox-recovery.log
exec &> >(tee -a "$LOG")
while IFS=, read -r backup vmid storage ; do
  echo "[INFO] RESToring $backup to VM $vmid on $storage"
  if [ ! -f "$backup" ]; then
    echo "[WARN] Backup $backup missing, skipping"
    continue
  fi
  # Quick storage check
  if ! pvesm status | grep -q "$storage" ; then
    echo "[ERROR] Storage $storage not present, aborting for $vmid"
    continue
  fi
  qmRESTore "$backup" "$vmid" --storage "$storage" || { echo "[ERROR] qmRESTore failed for $vmid"; continue; }
  echo "[OK] RESTore triggered for $vmid"
done < "$MAPFILE"

Fallback strategy: What if a RESTore fails?

Have a Plan B: if RESTores fail, you can:

• Manually reprovision VMs (installer + data import) for business‑critical services.
• Fall back to read‑only replicas or older snapshots.
• Temporarily relax network/security settings if they prevent boot (e.g., firewall rules).

It is important to define time limits and decision points: after X hours of recovery the Incident Lead switches to Plan B in order to meet the overall RTO.

Hyper‑V special section: transferable lessons

Although this playbook is focused on Proxmox, several principles are applicable to Hyper‑V. For Hyper‑V, VHD(X) exports, the state of the Hyper‑V configuration (export/import) and the Hyper‑V hosts (failover cluster) are critical. Key points are: consistent VM exports, verifying CSV/NAS/SMB mounts and network mapping (vSwitch). Test import/attach workflows in a preprod environment before relying on them in an incident.

Post‑recovery: hardening, lessons learned, documentation

After the RESTore perform a post‑mortem analysis, document changes and adjust backups, tests and responsibilities. Align RTO/RPO to realistic measurements from the drill and schedule follow‑up drills to remain regression‑free.

Practical checklist (compact)

1. Define scope & priorities.
2. Reconstruct the control plane (pmxcfs/corosync) or create a seed node.
3. Check / RESTore the PBS repository.
4. Attach storage targets (verify LVM/ZFS/Ceph).
5. RESTore VMs selectively using qmRESTore / qm importdisk / pct RESTore.
6. Perform database recovery (WAL/binlog/PITR).
7. Run smoke tests, integrity checks and application checks.
8. Document lessons learned and plan follow‑up.

Conclusion

A reliable disaster recovery playbook for Proxmox is more than a collection of commands: it is an operational document with priorities, roles, checks and fallback strategies. Test regularly, keep pmxcfs exports and PBS access up to date and automate supporting checks—but not critical recovery decisions. That way you reduce RTO and avoid costly mistakes in a real incident.

This guide is intended as a modular supplement to your operational documentation. Adapt it to your storage topology, RTO/RPO requirements and internal communication channels, and conduct regular RESTore drills.

Integrations, secrets and operational aspects

Often undeRESTimated: network and infrastructure integrations determine whether a RESTore actually becomes productive. RESTore core infrastructure first—DNS, DHCP, NTP and PKI—before bringing up application layers. If DNS is missing, VMs may boot but be undiscoverable for services; missing timestamps (NTP) can break database replication.

Store key material and certificates separately from the PBS and document Unseal/Unlock procedures (e.g. HashiCorp Vault, KMIP). External secrets stores should appear in the playbook with explicit access tests.

Practical note: Temporarily set monitoring alarms to „maintenance“ to avoid alarm floods, and re-register hosts selectively after successful smoke tests. For multi-tier applications, plan coordinated snapshot/RESTore windows so database and app remain consistent.

Audit every recovery step: logs, checksums, version states and who approves changes — this makes the playbook operationally reliable and organizationally robust.

VM image RESTores and Proxmox cluster recovery are also important for this topic. This article places these aspects into context clearly and shows what matters in day-to-day operations.