High Availability with Pacemaker/Corosync: Resource Groups, STONITH and Split‑Brain Prevention

Anyone who wants to operate productive services with true high availability often relies under Linux on High-Availability with Pacemaker/Corosync. Pacemaker is the resource manager; Corosync provides messaging, membership and quorum information. This guide supplements the basics with deeper operational aspects: concrete STONITH configurations, SBD usage, Corosync tuning, monitoring integration, verification steps for fencing tests, and rollback strategies — all with a focus on operation, data integrity and safe migrations of process-near software solutions.

Advanced risks: When Split‑Brain is particularly likely

Split‑Brain occurs when two partitions of the cluster independently activate resources. Setups with Single‑Writer‑Storage (e.g. traditional LVM/LUNs without a cluster filesystem), an unstable cluster network or missing/defective fencing are especially at risk. Network partitions combined with a storage timeout are a classic scenario: one partition loses Corosync communication, the other still sees the LUN as accessible — both can become Primary.

SBD (STONITH Block Device): When it makes sense and how to get started

SBD is a fencing mechanism that uses a dedicated block device as a token. The idea: only the node that can hold the token may perform write access. SBD is particularly practical when an external SAN is available or a small, quickly reachable block device (e.g. an iSCSI LUN) can be provided to the cluster.

SBD configuration: Example /etc/sbd.conf

# Minimal example /etc/sbd.conf
SBD_DEVICE=/dev/sdb
SBD_WATCHDOG=yes
SBD_PACEMAKER=yes
SBD_TIMEOUT=120
SBD_STARTMODE=dual
SBD_OPTS="-p 30"

Explanation: SBD_DEVICE is the shared device; WATCHDOG uses a hardware watchdog, PACEMAKER enables integration with Pacemaker; TIMEOUT is the wait time until fencing. Startmode=dual allows two nodes to use SBD. Always test changes outside business hours.

Install and start SBD (Debian/Ubuntu, RHEL variants similar)

# Debian/Ubuntu
apt-get update && apt-get install -y sbd
# RHEL/CentOS
yum install -y sbd

# Start and check
systemctl enable --now sbd
journalctl -u sbd --no-pager --since "-5m"

Important: SBD requires a reliable shared device. If the device disappears intermittently, SBD will cause false fencing. Test device path persistence and failover path before production start.

STONITH via BMC: IPMI / Redfish examples and pitfalls

Power fencing via BMC is widespread because it enables a true hardware reset. However, BMCs often need separate configuration and present their own security risks (default passwords, unencrypted management networks).

Example: STONITH with fence_ipmilan via pcs

# Example: create device for "node1" (use placeholders!)
pcs stonith create fence-node1 fence_ipmilan 
  ipaddr=192.0.2.120 login=ADMIN passwd='SECRET' 
  pcmk_host_list=node1 op monitor interval=60s

# Check
pcs stonith show

Note: Use secure credentials, restrict access to the management network and use role-based access on the BMC. Always test the power cycle in a controlled and documented manner.

Redfish: a more modern API

# Example: fence_redfish with token (placeholder)
pcs stonith create fence-node1-redfish fence_redfish 
  ip=192.0.2.121 user=admin password='SECRET' 
  pcmk_host_list=node1 op monitor interval=60s

Redfish provides clear APIs and improved auditing capabilities. Still, the basic rule remains: BMC access only via a dedicated management network, rotate credentials, and enforce access limits.

Corosync‑Tuning: Parameters that improve stability

Corosync communicates over a ring; delays or packet loss lead to repeated membership changes. Three useful adjustments:

• token: the time a node will wait at most for its turn. A higher token reduces split‑brain sensitivity under latency but increases failover latency.
• consensus: number of messages required for membership changes; increases resilience against transient packet loss.
• join‑timeout / rrp_mode (for multiple rings): how quickly reconnections are expected.

Changes require testing in a test network. Small token increases are often preferable to aggressive resource timeouts.

Pacemaker Resource‑Meta and failure handling

Pacemaker provides meta attributes such as migration‑threshold, failure‑timeout or resource‑stickiness. These determine how often and how quickly a resource is migrated or retried after failures.

# Beispiel: Metaparameter setzen
pcs resource meta grp_app migration-threshold=3 failure-timeout=10m 
  resource-stickiness=100

Recommendation: resource‑stickiness prevents unnecessary back‑and‑forth moves. migration‑threshold limits the number of automatic migration attempts; failure‑timeout defines the time window for counting.

Monitoring and alerting: End‑to‑end rather than only cluster status

Cluster metrics alone are not sufficient. Complement Prometheus‑exporters (e.g. crm_exporter or the native pacemaker exporter) with end‑to‑end probes that verify service functionality (HTTP health checks, DB write/read). Alerts should provide clear indications: fencing errors, repeated failovers, long‑running recovery.

Fencing tests: secure, reproducible and auditable

A fencing test must meet the following criteria: it is controllable, reproducible and documented. Procedure:

1. Put the node into maintenance mode (in test environments this can be omitted, but only with explicit approval)
2. Backup: pcs config export and secure the logs
3. Simulated node failure (e.g. network disconnect) and observe whether fencing triggers
4. Verification: Is the LUN/FS actually offline? Were power‑off/reset commands executed on the BMC?
5. Document all steps and timestamps

# Beispiel: Logs prüfen (Pacemaker, Corosync, SBD)
journalctl -u pacemaker -u corosync -u sbd --since "-30m" --no-pager
# Corosync Quorum Status
corosync-quorumtool -s

Avoid testing during peak business hours. If fencing fails, a power‑off can hard‑stop a production VM or make a storage path inaccessible.

Upgrade and migration strategy for existing clusters

Cluster upgrades are risky because changes to the messaging stack or resource agents can alter behavior. Recommended steps:

• Rollback plan and configuration backup before each step
• Rolling upgrade, if supported by the distributor: upgrade nodes one at a time and verify cluster functionality
• Mirror the test environment: same storage configuration and comparable network conditions
• After the upgrade: extended observation period, increased monitoring sensitivity

Operational checklist: before a production Go‑Live

• Documentation: architecture diagram with Failure Domains, fencing methods and maintenance windows
• End‑to‑End‑Checks: VIPs, NS/ARPs, Service‑Bind, DB‑Writes
• Fencing test: successfully reproduced and documented at least once
• Monitoring: alerts for fencing, repeated failures, high failcounts
• Backups: configuration snapshot and recovery runbook available

Practical examples: typical pitfalls and countermeasures

Pitfall: BMCs are in the same management‑VLAN as production clients

Problem: If the production network fails, the BMC is also unreachable and fencing fails. Countermeasure: migrate BMCs to a separate management network or provide redundant out‑of‑band access paths.

Pitfall: incorrect monitor intervals

Problem: overly aggressive monitors during periods of high IO latency cause flapping. Countermeasure: adjust monitor intervals to actual IO conditions and increase resource stickiness.

Conclusion: disciplined infrastructure instead of configuration magic

High‑Availability with Pacemaker/Corosync operates reliably when architecture, fencing and quorum are designed as an integrated system. STONITH is not a „nice to have“, but indispensable for Single‑Writer‑Setups and DRBD. SBD offers a robust alternative when a shared block device is available, while BMC‑Fencing provides true power isolation. The decisive factors are: test, document, integrate monitoring and keep rollback plans ready. This prevents split‑brain and makes operation predictable.

Additional resources: useful check commands

# Quick‑Checks im Betrieb
pcs status --full
pcs stonith show
corosync-cfgtool -s
corosync-quorumtool -s
# Logs zusammenführen
journalctl -u pacemaker -u corosync -u sbd --since "-2h" --no-pager

FAQs

• Is STONITH always required? In shared‑storage setups without a Multi‑Writer‑Cluster‑FS and with DRBD, STONITH is mandatory. In fully distributed storage systems with built‑in consistency it can be omitted in some architectures, but the decision requires a precise understanding of the storage semantics.
• How do I test SBD without risking production data? Use a test LUN with an identical path structure, simulate node failures and verify whether token transfer and fencing behave as expected. Document differences to the production environment.
• What if fencing fails? Enter maintenance mode immediately, analyse BMC/storage reachability and execute the recovery plan. In critical situations, controlled single‑node operation is often safer than unreliable automatic actions.

High-Availability with Pacemaker/Corosync: qdevice, resource constraints and recovery runbook

In addition to STONITH and SBD, it is worth looking at three operational areas that are often overlooked but have a major impact on stability: the use of an external quorum witness (qdevice/qnetd), clean resource constraints (Order/Colocation) and a pragmatic recovery runbook for real split‑brain cases. These aspects affect architecture, automation and the safe return to normal operation — relevant for the operation of process‑close software solutions with shared storage or VIP failover.

qdevice (Quorum Witness) vs. SBD: When to use which pattern?

qdevice (also called qnetd) provides a small, external witness service that supplies votes in tight quorum situations. Advantage: no shared block devices required, low overhead, simpler in cloud/VM setups. Disadvantage: the witness must be reachable and performant; in network partitions it only helps if it remains clearly reachable.

SBD is hardware- or storage-based and protects at the block-device level. Choose qdevice if you cannot provide shared LUNs or when using a small external witness VM in virtualized environments. Choose SBD for physical environments with a reliable SAN path and when you need token-based fencing guarantees.

Practical notes on operating qdevice

• Witness location: preferably in a third site or a separate network segment so it does not skew the decision in a two-node split.
• Resilience: run qdevice in HA (two active witness instances with a floating IP are possible) or use a cloud-hosted witness if network links are stable.
• Monitoring: record liveness and RTT to the witness; fluctuating latency can cause membership flapping.

Resource constraints: correctly model ordering and colocation

Many failover problems arise because services start in the wrong order or are placed on different nodes. Use order and colocation constraints deliberately to enforce dependencies:

# Example: ensure the filesystem starts first, then the application
pcs constraint order start fs_resource then app_resource
# Example: application must run on the same node as the mounted filesystem
pcs constraint colocation add app_resource with fs_resource INFINITY

Explanation: the order constraint prevents timing issues (e.g. the app starts before the FS is ready). The colocation constraint prevents the application from running on a node that has no access to the storage.

Multipath and device persistence

Dependencies on shared block devices require stable device names. Use persistent WWIDs, configure multipathd sensibly and secure udev rules. If path changes or timeout events make a LUN temporarily invisible, the cluster will quickly interpret that as a node failure — with potentially incorrect fencing.

Recovery runbook: safe steps in suspected split-brain

A clear, tested runbook prevents hasty actions. Example procedure before you act:

1. Notify stakeholders and declare a maintenance window. If necessary, activate a global alarm level so automatic scripts do not take additional actions.
2. Back up configuration and logs: pcs config export > /root/pcs-config-$(date +%F).xml and collect journalctl output.
3. Isolate nodes: stop Pacemaker on at least one node if you want to inspect the integrity of a storage device (systemctl stop pacemaker).
4. Determine the dataset with the highest authority (e.g. which replica last had write rights; for DRBD: Primary). Document timestamps and I/O metrics.
5. If an authoritative node is clear: replicate data (depending on the storage technology), bring the other node to a clean state and rejoin it in a controlled manner.
6. After recovery: extended monitoring phase, elevated alerts and manual verification of end-to-end checks (VIP, DB writes, application logs).

Important: Avoid automatic „force-join“ without a data consistency check. Document every step so that any forensic analysis is possible.

Automation, Configuration Control and Test Strategy

Keep cluster configurations versioned (Git) and automated via validated playbooks. Test fencing and quorum scenarios automatically in CI/CD-like labs (e.g. Vagrant/VM-based staging). Only validated, repeatable playbooks may perform changes to Pacemaker/Corosync in production.

These additions help reduce architectural and operational risks: an appropriate witness, clear constraints and a tested recovery runbook make the difference between occasional failovers and predictable, auditable operations.

Pacemaker clusters and Stonith fencing are also important for this topic. The article places these aspects in context and shows what matters in day-to-day operations.