Setting up Air‑Gapped Backup Nodes in the Data Center: Network Design, Hardware and Synchronization

Air‑gapped backup nodes — i.e., backup servers or storage systems that are physically or logically isolated from the production network — are a proven measure to mitigate ransomware, lateral movement and network compromise. This technical white paper describes how to plan such nodes in the data center, which hardware and network topologies are appropriate, how secure synchronization is implemented technically and which peculiarities must be observed for MySQL databases. Target audience: administrators, system engineers, operators and technical IT service providers.

Why air‑gapped backup nodes? Risks and operational objectives

An air gap means a system has no routine network path to the production network. In backup practice there are two variants: a physical separation (no network cable, removable media) or a logical one‑way transfer (hardware data diode or strictly controlled transfer windows). The objective is to reduce the attack surface and to retain a final clean data set that malware in the production network cannot reach.

It is important to define the operational objectives clearly: Is the focus Recovery Time Objective (RTO), Recovery Point Objective (RPO), compliance or forensic integrity? Air gaps primarily help with integrity and compliance but generally worsen RTO/RPO compared with online replicas.

Prerequisites and project checklist

Before design you should verify organizational and technical prerequisites. The following minimum points belong on the checklist:

• Define recovery objectives (RTO/RPO) and retention periods.
• Which data require protection? (production databases, configuration backups, certificates)
• Budget for hardware, media rotation and, if applicable, data diode appliances.
• Operational processes: who physically connects media, who signs/verifies manifests?
• Test plan for regular RESTore exercises.

Network design: topologies for air‑gapped backup nodes

There are three practical topologies:

1) Physically isolated nodes with removable media

Backup servers are installed locally in the data center but have no permanent L2/L3 routing to the production network. Data are transferred via removable media (encrypted HDDs, LTO tapes) offline. Advantage: very high isolation. Drawback: manual processes, longer RTO.

2) Logical one‑way transfer with data diode or unidirectional appliance

A hardware data diode is a device that physically allows data to flow in only one direction (simpler: optical). Alternatively, routers/ACLs can be configured so that connections may only be initiated into a specific subnet. Advantage: automatable transfers, reduced manual effort. Limitations: higher cost, dependency on the appliance vendor and the security of their firmware.

3) Temporarily connected, tightly controlled nodes (transfer windows)

The air gap is opened only temporarily: a network cable is physically connected or firewall rules are changed for a defined window. Transfers run encrypted, followed immediately by re‑isolation and signature checks. Suitable when automation without a data diode is required. Risk: human error when re‑isolating.

Planning network segments and firewalls

Define at least three segments: production network, transfer DMZ (if present) and Air‑Gapped Zone. Use clear ACLs (Access Control Lists), VLANs (Virtual LANs) and physical separation of switch ports. A simple rule is: no inbound connections from the production network into the Air‑Gapped Zone. Test rules with netcat or tcpdump before putting into production.

Hardware: servers, storage and media selection

Hardware decisions affect availability, integrity and operating costs. Selection criteria:

• Form factor: 1U/2U servers or a dedicated tape library depending on volume.
• Storage type: HDD‑based archive (cheap, high capacity), SSD (for fast RESTore testing), LTO tape (durable, offline‑friendly).
• Redundancy: For Air‑Gapped nodes RAID is often sufficient; for tape rely on media rotation and offsite copies.
• Encryption: hardware encryption on media or host‑side encryption before transfer.

Ensure traceable media labeling and secure storage: access controls, CCTV logs and signatures.

Synchronization: procedures, tools and integrity principles

The central question is: how do data reliably and verifiably arrive on the isolated node? Common methods with pros and cons:

• rsync / rclone over a one‑way link: flexible, fine‑grained, supports checksums. Requires a network connection (or a Data Diode).
• Block‑based replication (ZFS send/receive, btrfs send): efficient for large datasets with snapshot consistency.
• Physical media rotation (scp/physical HDD, LTO tape): highly isolated but slow and manual.
• Percona XtraBackup / MySQL consistent backups: necessary for MySQL dumps and incremental backups (see MySQL section below).

Example: rsync over a controlled window

rsync is proven in practice for file synchronization. Important: use checksums and create a manifest with sha256 for each transfer batch.

# Auf der Produktionsseite: Backup vorbereiten und Manifest erstellen
rsync -aH --delete /var/lib/appdata/ /staging/backupdir/
find /staging/backupdir -type f -print0 | xargs -0 sha256sum > /staging/backupdir/manifest.sha256
gpg --detach-sign --armor /staging/backupdir/manifest.sha256

After transfer into the Air‑Gapped Zone verify the checksums and the GPG signature.

# Auf Air-Gapped-Node: Integritätsprüfung
gpg --verify manifest.sha256.asc manifest.sha256
sha256sum -c manifest.sha256

Signatures and immutable manifests

Always sign manifests with an offline key or a key whose secret portion is not located in the production network. This prevents an attacker from manipulating manifests in the transfer chain.

MySQL specifics: consistency, tools and common pitfalls

MySQL (including MariaDB) imposes special requirements: you need consistent database backups that guarantee application consistency on RESTore. Important concepts: binlog positions (binlog = write‑ahead log), GTID (Global Transaction ID) for reproducible positioning, and quiesce methods (LVM snapshot or lock strategies).

Option A: Percona XtraBackup (recommended for large InnoDB databases)

Percona XtraBackup enables incremental, non‑blocking backups of InnoDB databases. Flow: XtraBackup creates a file copy plus transaction log (redo) and provides a consistency point that must be prepared before RESTore with xtrabackup –prepare.

# Full backup with xtrabackup
xtrabackup --backup --target-dir=/backup/xtrabackup/full --datadir=/var/lib/mysql --user=xbackup --password='secret'
# Incremental backup
xtrabackup --backup --incremental --target-dir=/backup/xtrabackup/inc1 --incremental-basedir=/backup/xtrabackup/full

Common mistakes: incomplete preparation, missing binlog/GTID metadata, or forgotten permissions when copying InnoDB system files. Test RESTore procedures regularly in an isolated test environment.

Option B: mysqldump / Logical Backups

mysqldump is simple and portable, but produces large dumps and can be problematic for RTO with very large DBs. For InnoDB you should use –single-transaction, which provides consistent snapshot reads (without locks) as long as no DDL operations are running.

# Consistent mysqldump for InnoDB
mysqldump --single-transaction --routines --events --triggers --databases appdb > appdb.sql
# Query binlog position
mysql -e "SHOW MASTER STATUSG"

When using mysqldump, pay attention to binlog positions or GTIDs so that you can perform Point-in-Time Recovery (PITR) after a RESTore.

Verification steps and RESTore exercises for MySQL

• Document the RESTore procedure and test it once per quarter.
• After RESTore check: number of tables, checksums (pt-table-checksum or comparable tools), application smoke tests.
• Verify user privileges, binlog state and replication configuration after RESTore.

Specific RESTore steps for XtraBackup

A typical RESTore path after transfer into the air-gapped system:

# Preparing the backup
xtrabackup --prepare --target-dir=/backup/xtrabackup/full
# Stop the MySQL service
systemctl stop mysql
# Copy to datadir (Note: file permissions)
rsync -aH /backup/xtrabackup/full/ /var/lib/mysql/
chown -R mysql:mysql /var/lib/mysql
# Start MySQL
systemctl start mysql

If –prepare fails, check xtrabackup’s log files for missing redo logs or incremental dependencies.

Operations: Runbooks, transfer procedures and automation

Good runbooks are the heart of operations. A transfer runbook describes in detail:

1. Preconditions: available media, signature keys, responsible parties.
2. Step by step: start backup, create manifest, sign, start transfer, post-transfer checks, re-isolation.
3. Monitoring and logging: syslog/central logs, audit entry of who connected media and when.
4. Fallback: What if the manifest fails? (e.g., RESTart the transfer or use the previous media-based copy).

Automation example: controlled transfer window

Automate opening/closing firewall rules via API or configuration management (Ansible/Chef). Execute automated preflight checks before opening (integrity check, quota verification).

Integrity, signatures and auditing

Integrity must be ensured on three levels: file integrity (checksums), transfer integrity (TLS, data diode), and authenticity (GPG signatures). Additionally, you should create audit logs that record physical media movements and user actions.

# Create and sign manifest (example)
find /backupdir -type f -print0 | xargs -0 sha256sum > manifest.sha256
gpg --default-key backup-admin --detach-sign --armor manifest.sha256

Monitoring, alerting and automatic sanity checks

Observe the following metrics: last successful transfer time, number of verified files, result of the sha256 verification, available media capacity. Integrate alerts (e.g. via Prometheus Alertmanager or RZ‑Monitoring) for missing transfers or faulty manifests.

Example: simple Healthcheck‑Script for Air‑Gapped‑Node

This script combines integrity verification and file counting and returns exit codes for monitoring systems.

#!/bin/bash
MANIFEST=/var/airgap/manifest.sha256
MANIFESTSIG=/var/airgap/manifest.sha256.asc
BACKUPDIR=/var/airgap/data
# Verify signature
if ! gpg --verify "$MANIFESTSIG" "$MANIFEST" >/dev/null 2>&1; then
  echo "GPG verification failed" >&2
  exit 2
fi
# Verify checksums
if ! sha256sum -c "$MANIFEST" >/dev/null 2>&1; then
  echo "Checksum mismatch" >&2
  exit 2
fi
# Check file count
FILECOUNT=$(find "$BACKUPDIR" -type f | wc -l)
if [ "$FILECOUNT" -lt 10 ]; then
  echo "Too few files: $FILECOUNT" >&2
  exit 2
fi
echo "OK: $FILECOUNT files verified"
exit 0

Typical pitfalls and how to avoid them

• Relying on a single method: combine checksums + signatures + physical audit.
• Human error during re‑isolation: automate re‑isolation where possible or require two‑person approval.
• Missing RESTore tests: without regular tests you cannot know whether backups are actually usable.
• Key management: never keep private keys in production systems; use offline HSM or separate air‑gapped keys.
• Insufficient documentation: runbooks and responsibilities must be up to date and accessible.

Capacity planning, throughput and performance aspects

Plan capacity not only by current volume but by RESTore windows and growth. Factors that affect RTO:

• Write throughput of the source (backup job duration)
• Transfer throughput (network or tape‑streaming rate)
• RESTore I/O at the target (how quickly data can be RESTored)

Example: a Data‑Diode with 1 Gbit/s has a theoretical ~125 MB/s. After overhead, protocol and encryption you should realistically assume 80–90 MB/s under good conditions. For multiple terabytes this means many hours; document these time windows in your runbook.

Deduplication, compression and storage strategies

Deduplication and compression reduce data volume but can affect the RESTore path and compatibility. Deduped stores typically require specialized RESTore tools; in an air‑gapped context many teams prefer simple, portable formats (tar, compressed streams) for long‑term retention. If you use dedupe, be sure to test complete recoveries from deduplicated stores.

Key management, HSM and forensic requirements

For signatures and encryption, secrets should never reside in the online production network. Options:

• Offline GPG keys on an air‑gapped laptop
• HSM (Hardware Security Module) or cloud HSM with access RESTrictions
• Smartcard/token‑based two‑person approval for critical signatures

Document key rotation, retention and key access protocols. For forensic evidence preservation a verifiable chain‑of‑custody is essential: who moved and inspected which medium and when.

Practical troubleshooting checklist

If a transfer fails or the manifest verification reports errors, follow these steps systematically:

1. Check log files (rsync/xtrabackup/gpg/syslog).
2. Compare file count and total size between source and target.
3. Verify GPG key fingerprints against a trusted list.
4. If –prepare in XtraBackup fails: check whether all required redo logs are present and whether incremental backups were linked correctly.
5. If media are faulty: attempt a bitwise copy (dd) and analyze the defective sectors; catalogue the faulty media for audits.

Migration and phased rollout

A full migration to Air-Gapped Nodes is operationally intensive. Recommended stepwise approach:

1. Pilot with a small data set (configuration data, less critical databases).
2. Automate manifest creation and signing.
3. Introduce a healthcheck monitor and quarterly RESTore drills.
4. Scale to larger volumes and readjust RTO/RPO targets.

Final assessment

Air-Gapped Backup Nodes are an effective component of a comprehensive backup strategy, particularly when integrity and tamper protection are priorities. They do, however, require disciplined processes, solid key and media management, and regular RESTore tests. Technically, data diodes, controlled transfer windows and physical media rotation present different trade-offs between automation, cost and isolation — choose the variant that aligns with your RTO/RPO targets and your operations team.

Operationally, focus on verifiable manifests, offline-managed signing keys, documented runbooks and regular RESTore drills. This avoids the most common pitfalls and ensures that your Air-Gapped Backup Nodes respond reliably in an incident.

Air gap and backup network design are also important for this topic. This article contextualizes these aspects and shows what matters in day-to-day operations.