Disaster Recovery: Runbooks, Failover Tests and Communication

Disaster recovery is not a single measure but a chain of decisions, dependencies and well‑practiced procedures. In practice, recovery attempts rarely fail because „no backup exists.“ More often, runbooks are incomplete, responsibilities unclear, failover paths have never been tested in reality, or communication creates unnecessary pressure and wrong priorities.

This article is aimed at administrators, system engineers, operators and technical IT service providers. The focus is on a practical approach: how to write runbooks so they work under stress, how to plan failover tests without endangering production, and how to set up communication and decision paths so technical teams remain able to act. Examples are oriented on typical MariaDB setups (replication, backups, point‑in‑time recovery), but deliberately also include infrastructure topics such as DNS, load balancers, identities and monitoring.

Disaster recovery in practice

A „disaster“ is operationally less a single event than a state: a service is unusable, the cause is not immediately clear, and the normal change and approval chain is too slow. Typical triggers are ransomware (including compromised admin accounts), storage failure, update errors, cloud/provider outage, network segment problems or logical data corruption (e.g., a faulty application writes incorrect data).

Runbooks often fail for the same reasons:

• Incorrect assumptions: „The monitoring host is running“, „DNS is available“, „we have internet“, „the backups are readable“.
• Overly coarse steps: „RESTore DB“ without specifications for version, recovery mode, validation, dependencies and rollback point.
• No decision points: Runbooks need clear „if‑then“ switches (e.g., data corruption vs. infrastructure failure).
• No practiced communication: Engineering is overwhelmed with status questions while decisions (e.g., accept data loss?) are not made.

The remedy is a combination of (1) well‑structured runbooks, (2) realistic tests, (3) clear role definitions and (4) communication rules that support the technical recovery rather than block it.

RTO and RPO: Define objectives before you write a runbook

Without target values every runbook will be either too conservative (too slow) or too risky (too much data loss). Two terms are central:

• RTO (Recovery Time Objective): Target time until the service is usable again. Practically: when users can work again, not when „everything is perfect“.
• RPO (Recovery Point Objective): Maximum tolerable data loss measured in time. Practically: „in the worst case we may only roll back up to 15 minutes.“

For MariaDB this often means: RTO is limited by RESTore duration, DNS/load balancer switching, application cache invalidation and schema compatibility. RPO depends on replication lag, backup intervals and Point-in-Time Recovery (PITR, recovery to a point in time using binary logs).

Important: Record RTO/RPO per service, not “for the database”. A database can be technically running while the business software remains unavailable due to missing secrets, incorrect endpoints or broken jobs.

Runbook design: structure, roles and “Stop-the-Line” rules

A runbook is an operational manual for an incident. It must be comprehensible within 2–3 minutes and still deep enough to make decisions and actions reliable. The following structure has proven effective:

1) Header: purpose, scope, prerequisites

• Purpose: “RESTart MariaDB service X after total failure of primary datacenter”.
• Scope: Which systems are included (DB, proxy/load balancer, DNS, monitoring, backup storage, secrets)?
• Prerequisites: Access (break-glass accounts), offline copies, keys/passwords, emergency network.

2) Role model: who decides, who executes

In a disaster scenario parallelism is critical. Define at minimum:

• Incident Lead: coordinates, prioritizes, shields the team from context switches.
• DB Operator: executes MariaDB steps, documents times and findings.
• Infra Operator: DNS, load balancer, storage, network, VM/container, access paths.
• Communications: status updates, stakeholders, if applicable provider tickets.

“Stop-the-Line” rules belong explicitly in the runbook: in which situations is action immediately halted (e.g. suspected active ransomware, unclear data integrity, unexplained binlog gaps)? That prevents frantic “bringing systems back up” into an environment that is still compromised.

3) Decision tree: failover or RESTore?

Runbooks should not only list steps but paths. Core question: Failover (switch to an existing standby environment) or RESTore (recover from backups). Failover is usually faster (RTO), RESTore can be cleaner (security/integrity) — particularly after ransomware or logical corruption.

Technical fundamentals for MariaDB DR: replication, backups, PITR

For MariaDB in enterprise operation three building blocks are typical:

• Replication: secondary system(s) receive changes from the primary DB. Depending on the setup, asynchronous or semi-synchronous. Risk: replication also carries “bad” changes (corruption, deletions).
• Physical/logical backups: physical (file-level, e.g. Percona XtraBackup-compatible methods) is usually faster to RESTore; logical (mysqldump) is more portable but slow with large data volumes.
• PITR (Point-in-Time Recovery): recovery to a point in time between a backup and “now” using binary logs. Requirement: binary logs are complete, time-assignable and available.

A DR runbook must state concretely which combination you use: “Failover to Replica A” is different from “RESTore last full backup + binary logs up to T-15min”. And: for MariaDB the integrity check (e.g. checking tables, application sanity checks) after RESTart is often the bottleneck, not copying the data.

Preparation: What must be documented and available before a disaster

Many disaster recoveries fail operationally because small details are missing. This checklist is deliberately pragmatic and written from an operations perspective:

Identities and Access (Break Glass)

“Break Glass” refers to emergency access accounts that are managed separately from normal admin accounts and given additional protection (e.g., MFA tokens stored offline, sealed credentials, separate password policies). Important for DR:

• Emergency SSH/RDP access into an isolated admin network
• DB admin access that does not depend on SSO/IdP (IdP = Identity Provider, centralized authentication)
• Access to the backup repository and key material (encryption, object storage keys)

Configuration and dependency inventory

At minimum: versions (MariaDB, OS, Proxy), parameters (innodb_flush_log_at_trx_commit, binlog_format), topology (Primary/Replica, GTID yes/no), DNS names, ports, firewall rules, storage classes, monitoring checks, cronjobs/batch jobs. Without this, every restart becomes an improvised rebuild.

Offline and „No-Internet“ path

Plan for the case that neither the Internet nor the cloud console is reachable. That sounds extreme, but is realistic in provider outages, BGP problems or security incidents. Practical measures:

• Offline copy of the runbooks (PDF/print) and the most important secrets (securely stored)
• Local repo/cache for packages or container images (otherwise the rebuild will fail because of downloads)
• Out-of-band access (e.g., iLO/iDRAC/IPMI or serial console access) on a separate network

Failover tests: types, risks and success criteria

Failover tests are not an “event” but a repeatable process. The decisive point is that you define success criteria that go beyond “ping works”. For MariaDB and dependent services, tests should at minimum cover: write/read access, transaction consistency, application login, critical jobs/queues, reporting, as well as monitoring/alerting.

Test types (from safe to realistic)

• Tabletop/Walkthrough: Team walks through the runbook without changing systems. Good for roles and communication, weak on technical detail.
• Simulated failover in test/staging: Technically valuable, but often different data volumes/load profiles.
• Planned failover in production: High insight gain, but must be properly prepared (maintenance window, rollback, stakeholders).
• Unplanned failover drill: A “chaos” approach with surprise elements. Only for mature teams and stable automation.

Typical pitfalls in MariaDB failover

• Replica lag: Replication delay leads to a larger RPO than expected. Often caused by I/O bottlenecks or large transactions.

Minimal failover test (practical, 60–120 minutes)

If you only have limited maintenance windows, prioritize a test that yields the most meaningful results:

1. Freeze changes: stop deployments, pause batch jobs, reduce write load.
2. Check replica status: ensure replication has caught up, document lag.
3. Switch over: point the application endpoint to the replica, set the primary to read-only or isolate it.
4. Sanity checks: login, read/write test, most important reports/jobs.
5. Rollback: switch back or keep the new primary, but make a clear decision.
6. Follow-up: create tickets for identified gaps, update the runbook, record timings.

RESTore runbook for MariaDB: sequence of steps, checks, rollback strategy

RESTore is the „hard“ path because it contains more unknowns: backup consistency, keys, performance during recovery, and above all validation. A good RESTore runbook strictly separates: RESToration (bringing data back) and recommissioning (making the service usable).

Phase 1: assess the situation and define the target state

• Incident class: infrastructure failure, data corruption, security incident (e.g. ransomware)?
• Target point: „last consistent backup“ or PITR to time T?
• Risk decision: RPO trade-off (data loss) vs. RTO (time). This decision must be made consciously and documented.

Especially with logical corruption, failing over to a replica is dangerous because the incorrect changes may have been replicated. In that case, RESTore/PITR is usually the safe path.

Phase 2: provision infrastructure (no shortcuts)

RESTores often fail because of seemingly simple things: wrong package versions, missing kernel parameters, different filesystem options, volumes that are too small, or incompatible InnoDB settings. Record the following in the runbook:

• Target form: VM, bare metal, container (and which storage class)
• Capacity: data + overhead for RESTore (often significantly more temporarily)
• Network: segment, firewall, only necessary ports open
• Time synchronization: NTP/Chrony (important for logs, binlog time references, audits)

Phase 3: Locate backup, verify, make RESTorable

Verification is mandatory. “Backup exists” is not proof of “backup is readable”. Check at minimum: existence, size/trend, checksums, encryption keys, access permissions and whether the backup was created consistently.

Example: A simple, repo-independent verification step for archive files (integrity and extractability) can look like this:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_TAR="/mnt/backup/mariadb-full-2026-07-28.tar.gz"
SHA_FILE="${BACKUP_TAR}.sha256"

# 1) Verify checksum
sha256sum -c "$SHA_FILE"

# 2) Test-read archive (without extracting)
tar -tzf "$BACKUP_TAR" > /dev/null

echo "Backup archive is readable and checksum matches."

Why this works: checksums detect silent bit errors; test-listing the tar archive detects broken compression/container structures. When it fails: if the backup is intact as a file but inconsistent in content (e.g. not a clean snapshot, missing tablespaces, incomplete binlogs). Therefore it also needs DB-proximate validation (see below).

Phase 4: Perform RESTore (physical/logical) and then validate

The concrete RESTore method depends on your backup procedure. Important universal checkpoints for the runbook are:

• Start DB in a controlled state: do not attach the application; only allow it after validation has passed.
• Read-only phase: first read/verify only, then release for write load.
• Schema/migration checks: Does the schema match the application version? Otherwise you risk follow-up errors.

A practical SQL block for quick plausibility checks (without diving deep into engine internals):

-- Connection and basic state
SELECT VERSION() AS mariadb_version;
SHOW VARIABLES LIKE 'read_only';
SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';

-- Replication/binlog relevance (if used)
SHOW VARIABLES LIKE 'log_bin';
SHOW BINARY LOGS;

-- Coarse consistency signals: table status and errors
SHOW DATABASES;
-- For selected core tables (adjust example names):
CHECK TABLE app.core_customer;
CHECK TABLE app.core_order;

Why this helps: you check whether you are on the expected version, whether the server is accidentally writable, whether binlogs are present (relevant for PITR and later replication) and whether central tables at least superficially appear consistent. Where the limit is: CHECK TABLE is no cure-all, and InnoDB can show some problems only under load. Therefore supplement with application sanity checks (login, core process, critical reports).

Phase 5: Execute PITR (Point-in-Time Recovery) cleanly

PITR means: you RESTore a full backup and then apply binlogs up to a defined point in time. For this to work, binlogs must be complete and temporally unambiguous. Typical pitfalls:

• Time zones/clock drift: time offset complicates the ’stop-date-time‘ decision.
• Binlog gaps: missing files due to retention, storage failures or incorrect offloading.
• Imprecise target time: Business says „before the incident“, but the incident time is not cleanly correlated (logs!).

In the runbook you should therefore provide: (1) a timeline from monitoring/application logs, (2) a clear definition of „last good commit“, (3) a documented sequence of commands matching your tooling. If you use mysqlbinlog, test the procedure regularly in an isolated environment, because small parameter errors (e.g. incorrect GTID interpretation) otherwise only become apparent in a real incident.

Phase 6: Fallback strategy (Rollback) for the RESTore itself

A RESTore can fail: backup is corrupt after all, wrong key, RESTore takes longer than the RTO, or validation fails. Therefore plan the fallback not as „we’ll keep trying“, but as a defined option:

• Plan B Failover: If RESTore takes too long, switch to a replica with accepted data loss.
• Plan C Minimal operation: Read-only mode for core functions (e.g. reporting) with clear communication.
• Forensics path: In case of a security incident: isolate systems, no „cleanup actions“ without approval.

Communication in a disaster: cadence, content, decision log

Communication is a technical factor in an emergency because it determines focus and throughput. A proven pattern is a fixed update cadence (e.g. every 15–30 minutes) and a consistent status format. That reduces follow-up questions and prevents contradictory statements.

Status template (brief but robust)

• What is affected? Service/scope, user impact.
• What is known? Cause as a hypothesis, not as an assertion.
• What are we doing now? Concrete step (failover in progress, RESTore running, validation).
• What is the risk? Data-loss window (RPO), integrity risk, security risk.
• When is the next update? Fixed time.

Run a decision log in parallel: who decided when which RPO/RTO trade-off to accept? This is not bureaucracy but protects the team afterwards and helps with audits.

Communication pitfalls

• Technology becomes a „live ticker“: An operator should not simultaneously perform the RESTore and handle stakeholders.
• False precision: „Back online in 12 minutes“ is usually unreliable. Better: „RESTore running, next milestone: validation in approx. 30–45 minutes, then ETA“.
• No clear „go“: Who authorizes that writes may resume? This is a defined decision point.

Checks and troubleshooting: when failover/RESTore does not proceed as planned

In real recovery attempts there are often a few recurring error classes. A good runbook therefore contains short troubleshooting blocks with „Symptom → Check → Action“.

Symptom: Application does not connect to the DB after failover

• Check: Is DNS still pointing to the old endpoint? TTL/cache? Load balancer health checks?
• Check: Firewall rules in the DR segment? Security groups? NAT?
• Action: Verify name → IP, port reachability, then check application config/secrets.

A quick network check from an app host (example) can look like this:

#!/usr/bin/env bash
set -euo pipefail

DB_FQDN="db.service.example"
DB_PORT="3306"

echo "DNS-Auflösung:";
getent hosts "$DB_FQDN" || true

echo "Port-Test:";
( echo > /dev/tcp/$DB_FQDN/$DB_PORT ) && echo "Port offen" || echo "Port blockiert/timeout"

Why this helps: It separates name resolution from port reachability. Where it fails: /dev/tcp is bash-specific and not available everywhere; in such environments you need nc/telnet or appropriate tools.

Symptom: Replica is read-only after failover or behaves inconsistently

• Check: read_only/super_read_only set? (Intentionally so in some HA setups.)
• Check: Replication configuration still active and still writing „from the back“?
• Action: Establish a clear primary role, set other nodes read-only and/or reattach them as replicas.

Symptom: RESTore takes „forever“

• Check: Storage performance (IOPS/throughput), CPU bottlenecks, compression/decryption
• Check: Target volume too small or wrong filesystem (e.g. noatime, journaling options)
• Action: Parallelize only where tooling and storage can tolerate it; otherwise activate Plan B.

Make runbook quality measurable: „Definition of Done“ for DR

Otherwise runbooks become documents nobody wants to touch. Define objective quality criteria:

• Executable in the real world: All accesses, paths and dependencies are available (including offline).
• Tested: Each critical runbook variant (Failover, RESTore, PITR) has been practiced at least once.
• Timestamps: Last test, last change, responsible role.
• Validation: Clear acceptance criteria for when „service RESTored“ applies.
• Fallback: For every major step there is a rollback option.

For MariaDB, an additional fixed cadence is recommended: small RESTore drills (e.g. monthly) with minimal data and a larger end-to-end exercise (e.g. quarterly or semi-annually) depending on change rate and risk.

Practical conclusion: Disaster recovery is a practiced operational procedure

In a disaster, teams that made decisions in advance win: RTO/RPO per service, clear failover vs. RESTore paths, secured access, and a communication scheme that protects engineering. A runbook is then not „documentation“ but a tool: it makes steps reproducible, reduces errors under stress and enables deriving concrete improvements from each test.

If you take away only one point: Plan recovery not as a „backup topic“ but as an End-to-End RESTart including DNS, identities, secrets, jobs, monitoring and approval processes. Only when failover tests and RESTore drills succeed regularly is recovery in a disaster case more than hope.

For this topic, Disaster Recovery Runbook and incident communication are also important. The article puts these aspects into context and shows what matters in day-to-day operations.