HA cluster with Proxmox and 2 nodes: correctly configure quorum, QDevice and fencing

An HA cluster with Proxmox and 2 nodes is, in practice, a special case: it can be set up quickly, but on the topics of Quorum (majority decision in the cluster) and Fencing (hard-isolating a faulty node to prevent data corruption) it is significantly more demanding than a classic 3‑node setup. The reason is simple: with exactly two votes there is no majority without a third “timekeeper”. As soon as communication or storage falters, “Split Brain” situations threaten – that is, two sides that each believe they are the valid instance.

This article shows in a practical way how to correctly solve quorum with two Proxmox nodes (QDevice), how fencing in Proxmox can be implemented and tested realistically (including watchdog) and which verification and fallback strategies help in operation. The focus is on impacts to availability, data integrity and operations – not framework details.

HA cluster with Proxmox and 2 nodes in practice

In Proxmox cluster communication is based on Corosync (cluster messaging) and the cluster configuration resides in pmxcfs (Proxmox Cluster File System, a distributed, quorum-dependent configuration filesystem). For cluster decisions Corosync uses a quorum: only when a majority is reachable is the cluster view considered “valid” and write-enabled cluster actions are permitted.

With three nodes this is robust: two out of three constitute a majority. With two nodes it is fragile without aids: if a node or only the connection fails, one vote remains — and that is not a majority. The cluster then blocks (in the best case). In the worst case administrators bypass quorum protection mechanisms and risk split brain: both sides continue running, write to the same storage, or perform conflicting HA actions.

Important for planning: “HA” is not just “VM starts on another host”. HA primarily means consistent decisions under failure. For that you need either at least three votes or an external, controlled “third vote”.

Basic principle: QDevice as a third vote for Proxmox quorum

The clean solution for two Proxmox nodes is a QDevice. This is not an additional Proxmox host, but a quorum aid: an external service (qnetd, Corosync Quorum Net Daemon) provides an extra vote over the network. The cluster nodes attach to this service as clients (qdevice). That way, in the event of a failure or a partition a majority can form again: 1 node + QDevice = 2 votes (majority of 3).

Why this works: QDevice does not make „HA decisions“, but serves as a controlled witness that indicates which partition is given the additional vote. That significantly reduces split‑brain risk, but does not replace fencing. Because even with quorum a node can still be „running“ while already unsafe (e.g. storage is hung, kernel is hung, I/O timeouts). Then you need a mechanism that ensures: Only one host may continue writing to shared resources.

Prerequisites and design check before implementation

Before you configure QDevice or fencing, clarify the boundary conditions. Many 2‑node setups do not fail because of Proxmox itself, but because of unstated assumptions about network, storage, or power delivery.

1) Network: at least two paths, defined latency, consistent MTU

For Corosync it’s not „bandwidth“ that counts, but stable latency and low packet loss. Ideally plan separate networks (or VLANs) for management/cluster (Corosync) and VM traffic. If you use a dedicated Corosync interface, ensure identical MTU and consistent switch configuration. „Jumbo frames only somewhere“ is a classic cause of sporadic partitions.

2) Storage: Shared Storage vs. Replication

In HA setups with Proxmox, two patterns are common:

• Shared Storage (e.g. iSCSI/NFS/SAN): Both hosts see the same datastore. Advantage: a VM can be started quickly on the other host. Risk: split brain can corrupt storage if both hosts write simultaneously.
• Replizierter Storage (e.g. ZFS replication): Data is synchronized/periodically replicated between hosts. Advantage: less simultaneous writing to the same block device. Disadvantage: RPO/RTO depend on replication intervals and the failover process.

For both patterns: without fencing, shared storage in a 2‑node operation is particularly risky. Even with QDevice, fencing should be provided as the „last resort“.

3) Out-of-Band-Management and Watchdog

For reliable fencing you typically need an out-of-band interface such as IPMI (BMC remote management) or a switched PDU. Alternatively, storage-based mechanisms (e.g. SBD) exist, but in Proxmox environments they are often outside the standard path. Additionally, a Watchdog should be active: a hardware or kernel timer that reboots the host if the system „hangs“ and no longer regularly „feeds“ it (keepalive). This addresses deadlocks where a host still has power but can no longer make safe decisions.

Implementation: QDevice (qnetd) for two Proxmox‑nodes

The recommendation from an operations perspective: run the QDevice service on a third system that is independent of the two Proxmox hosts (different power circuit/UPS, ideally a different rack/location segment). This can be a small VM or a mini server. What matters less is performance than reachability and clean network paths.

Step 1: Install QNetd on the QDevice host

Example for a Debian/Ubuntu system as QDevice host:

sudo apt update
sudo apt install -y corosync-qnetd
sudo systemctl enable --now corosync-qnetd
sudo systemctl status corosync-qnetd

Ensure the firewall allows the qnetd port (default TCP 5403). In RESTrictive networks this is often the first stumbling block: Corosync typically runs internally over UDP/multicast or UDP/unicast, whereas qnetd uses TCP.

Step 2: Ensure correct time and name resolution

Cluster components are sensitive to time drift. Enable NTP/Chrony on all systems. In addition, name resolution should be stable (DNS or /etc/hosts), because certificates and identities can play a role for qdevice/qnetd.

Step 3: Register QDevice in the Proxmox cluster

On a Proxmox node (typically the one that initialized the cluster) attach QDevice. Proxmox provides pvecm (Proxmox VE Cluster Manager) for this.

# Check status
pvecm status

# Add QDevice (hostname or IP of the QNetd host)
pvecm qdevice setup <QNETD_HOSTNAME_ODER_IP>

The setup process generates and distributes the necessary keys/certificates and rewrites the Corosync configuration accordingly. Afterwards check the cluster status again:

pvecm status
pvecm qdevice status

Expectation in a 2-node cluster: in addition to the two node votes you will see additional quorum information from QDevice. If QDevice is not reachable, the quorum in a failure case falls back to the problematic two-vote model.

Typical pitfalls with QDevice

• QDevice in the same Layer‑2 segment as both nodes, but on the same switch edge: if the switch fails, QDevice fails too. The ‚third witness‘ then is not independent.
• QDevice as a VM on the Proxmox cluster itself: that’s a circular dependency. If the cluster becomes unstable, so does the VM intended to stabilize quorum.
• Unclear firewall rules: qnetd requires stable TCP connectivity. Packet loss or TLS inspection on the path can lead to sporadic disconnects.
• Too aggressive Corosync timeout tuning attempts: in small networks shorter timeouts may appear ‚faster‘, but they increase false triggers for brief disturbances. Measure first, then change.

Fencing in Proxmox: Why quorum alone is not enough

Even with QDevice it holds: quorum only says who is allowed to decide. It does not say whether another node is actually stopped. This is exactly where fencing comes into play (often also called STONITH: ‚Shoot The Other Node In The Head‘). Goal: if a host must be considered failed from the cluster’s perspective, it is reliably powered off or reset before resources (VMs, storage) continue running on the other host.

In Proxmox wird HA über den integrierten HA Manager gesteuert. Der HA Manager kann Services (VMs/Container) verwalten und im Fehlerfall verschieben. Aber: Wenn beide Hosts gleichzeitig glauben, sie seien „aktiv“, kann HA ohne Fencing falsche Aktionen ausführen. Mit Shared Storage ist das besonders gefährlich, weil Datenblöcke parallel beschrieben werden können.

Realistische Fencing-Optionen im 2‑Knoten‑Betrieb

• IPMI/Redfish Power Off/Reboot: Der Klassiker im Serverbetrieb. Voraussetzung: BMC ist erreichbar, getrennt abgesichert, nicht über dasselbe Netz wie das Problemsegment.
• Schaltbare PDU / Smart Power: Funktioniert auch, wenn BMC instabil ist. Muss aber organisatorisch sicher betrieben werden (Zugriffsrechte, Logging).
• Watchdog + Selbstfencing: Wenn der Host intern erkennt, dass er unsicher ist (z. B. verliert Quorum/Cluster), kann er sich selbst neu starten. Das ist kein Ersatz für externes Fencing, aber erhöht Robustheit gegen „hängende“ Zustände.

Wichtig: Ein „sauberes Shutdown“ ist nicht Fencing. Fencing muss im Zweifel hart sein, weil genau im Fehlerfall „sauber“ oft nicht mehr zuverlässig funktioniert.

Watchdog in Proxmox aktivieren und prüfen (Best Practice)

Ein Watchdog ist ein Mechanismus, der das System zurücksetzt, wenn es nicht mehr reagiert. Unter Linux ist häufig /dev/watchdog über ein Kernelmodul verfügbar (z. B. iTCO_wdt bei Intel‑Plattformen). Proxmox kann Watchdog für HA nutzen, um in Deadlock-Situationen nicht „halb tot“ weiterzulaufen.

Schritt 1: Prüfen, ob ein Watchdog-Gerät verfügbar ist

ls -l /dev/watchdog* || true

# Kernelmeldungen zum Watchdog
dmesg | grep -i watchdog || true

# Geladene Module
lsmod | grep -i wdt || true

Wenn kein Gerät auftaucht, prüfen Sie BIOS/UEFI‑Optionen (Watchdog/Server Management) und passende Kernelmodule. In virtualisierten Umgebungen kann Watchdog über die VM‑Plattform bereitgestellt werden; auf Bare Metal ist es typischerweise Hardware/Chipsatz.

Schritt 2: Proxmox Watchdog-Konfiguration prüfen

Je nach Proxmox‑Version und Setup wird Watchdog über Systemdienste/HA-Komponenten aktiviert. Als grober Check im Betrieb:

systemctl status pve-ha-lrm pve-ha-crm || true
journalctl -u pve-ha-lrm -u pve-ha-crm --since "-2h" | tail -n 200

Interpretation: Sie suchen nicht „grüne“ Services, sondern Hinweise, dass HA‑Komponenten laufen und keine wiederkehrenden Timeouts/RESTart-Loops auftreten. Wenn HA nicht genutzt wird, ist Watchdog trotzdem als Sicherheitsnetz sinnvoll – aber dann müssen Ihre Betriebsprozesse (Monitoring/Alerting) Host‑Hänger zuverlässig erkennen.

Praxis-Checkliste: Vor dem ersten HA-Failover testen

Before productive VMs run as HA resources, perform controlled tests. The goal is not ‚it works once‘, but ‚we understand when it does not work and how to respond then‘.

1) Cluster- und Quorum-Status baseline

pvecm nodes
pvecm status

# Corosync-Health grob prüfen
systemctl status corosync
journalctl -u corosync --since "-1h" | tail -n 200

Watch for packet loss, token timeouts and recurring membership changes. A 2-node cluster must run ‚boring‘: no constant rejoins, no flaps.

2) QDevice-Status und Abhängigkeit prüfen

pvecm qdevice status

# QNetd vom Node aus erreichen
nc -vz <QNETD_HOSTNAME_ODER_IP> 5403

If the TCP connect fails intermittently, resolve that before enabling HA. Otherwise, in a failure you’ll be left without a majority vote.

3) Simulierter Link-Ausfall: Corosync-Netz trennen

Controlled disconnect the Corosync interface (not the management interface, so you can continue to administer) and observe what happens: Does one side obtain quorum? Does the other side remain cleanly blocked? This is exactly where QDevice and network design prove themselves.

Observation during the test:

watch -n 2 'pvecm status; echo; pvecm qdevice status'

If both sides still appear ‚active‘ or HA actions are unclear, that’s a warning: you need to re-plan fencing/isolation and network paths cleanly.

Typische Fehlerbilder und Troubleshooting im Betrieb

Fehlerbild A: ‚Cluster hat kein Quorum‘ nach kurzem Netzruckler

Causes: packet loss on the Corosync path, switch buffer/queue issues, MTU mismatch, QoS/policing, virtual switches dropping packets. In 2-node setups you notice this immediately because there is no third vote to stabilize the view.

Procedure:

• Check Corosync logs for token timeouts.
• Measure the network path: drops on NIC/switch ports, errors, CRC, duplex.
• If possible: configure a second Corosync network (redundancy), rather than ‚tuning‘ timeouts.

Fehlerbild B: QDevice ist erreichbar, aber Quorum verhält sich ‚unerwartet‘

Causes: QDevice not independent (shares cause of failure), name resolution/certificate issues, unstable TCP session, asymmetric routing. Check whether both nodes actually speak stably with qnetd and whether qnetd itself runs without interruptions.

Quick checks:

# Auf dem QDevice-Host
systemctl status corosync-qnetd
journalctl -u corosync-qnetd --since "-2h" | tail -n 200

Fehlerbild C: VM läuft nach Failover, aber Storage ist inkonsistent oder ‚stuck‘

This is the dangerous case: quorum/HA have ’somehow‘ reacted, but without secure fencing the old host can still issue I/O or block locks. Depending on the storage (NFS, iSCSI, Cluster‑FS, Ceph) this manifests as hangs, filesystem errors or corrupted VM disks.

Measures:

• If in doubt: power off the affected host hard (Out-of-Band) before further debugging.
• Check storage logs (target/controller side). Many causes are not on the hypervisor.
• Only release HA resources again when it is clear that no ‚double writers‘ are active.

Umsetzungsempfehlung: Minimal robustes 2‑Knoten‑HA-Design

If you must stay with two nodes for budget or space reasons, the goal is a design that reacts predictably in a failure. A practical minimal package looks like this:

• 2 Proxmox hosts with separated network paths for Corosync/management/VM traffic (at least logical via VLAN, preferably physical).
• 1 independent QDevice host (qnetd), not running on the two Proxmox hosts.
• Out-of-Band Fencing (IPMI/Redfish or PDU) defined as an emergency process, including responsibilities and access paths.
• Watchdog active and tested to mitigate hang states.
• Runbooks for partition/storage hangs: Who powers off which host when, how is resynchronization performed, how is data consistency validated?

This is less “glamorous” than feature lists, but it reduces real outages and, above all, data risks.

Fallback strategy and recovery: What to do if Quorum/Fencing reacts incorrectly?

In practice you must assume that at least once an unclear state will occur: partial partition, storage freeze, BMC unreachable, HA failover stuck. The crucial thing then is to have a conservative recovery sequence that protects data.

Conservative recovery sequence (proven in operation)

1. Stop write access: If shared storage is involved and you suspect double-writers, prioritize hard-stopping a node (OOB Power Off) before attempting to “nicely” reconfigure.
2. Stabilize cluster state: Only when it is clear which host should be the “master” restore Corosync/Quorum (network path, qnetd reachability).
3. Validate storage: Depending on the backend: filesystem checks, storage controller logs, iSCSI sessions, NFS locks. Goal: no silent inconsistencies.
4. Activate HA services in a controlled manner: Not immediately “everything on autopilot”, but stepwise, with monitoring for I/O latency, locks, kernel errors.

As an operator rule: Better to take 10 minutes longer to decide cleanly than 10 hours of data recovery. Two nodes forgive no ambiguity.

Final conclusion: 2 nodes are possible — but only with discipline

An HA cluster with Proxmox and 2 nodes can be operated reliably if you consciously compensate for the systemic weakness (no natural majority). QDevice is not a “nice-to-have” here but the foundation to sensibly achieve quorum in failure cases. Fencing and Watchdog are the safety nets that prevent data corruption and undefined states when reality hits (storage stalls, network flaps, partially failed hosts).

If you are planning today: include QDevice and Fencing from the start, test critical failures in a controlled manner and document a conservative recovery procedure. That turns a “small” 2-node cluster into a setup that in operation lives not on hope but on clear states.

For this topic Proxmox Qdevice and Proxmox Fencing are also important. The article places these aspects into context and shows what matters in everyday operation.