Automated Backups with BorgBackup: Deduplication, Remote Storage and RESTore Checks

In this post I explain how to set up Automated Backups with BorgBackup in a technically sound way: from repository architecture and deduplicating storage through remote transport to automated RESTore checks, prune strategies and typical troubleshooting steps. The goal is operationally reliable procedures for administrators, system engineers and operators that prioritize availability, integrity and maintainability.

Why BorgBackup? A concise architectural overview

BorgBackup (short: Borg) is a file-level backup tool that provides content-dependent deduplication, optional end-to-end encryption and efficient transfer via SSH. Deduplication means identical data segments (chunks) are stored only once in the repository; this significantly reduces storage use for repeated backups. Borg stores backups in a repository that can reside locally on a server or be addressed via SSH with borg serve. Repository topology is central to performance, locking and maintenance, so it should be planned early.

Automated Backups with BorgBackup: Architecture and operational rules

For production operation several aspects are mandatory:

• Tenant or service separation: Create a separate repository per tenant or critical service when retention policies or access control differ.
• Version compatibility: Keep client and server Borg versions synchronized; test version changes in staging environments before deploying them to production.
• SSH security: Use key-based authentication, a dedicated backup account (e.g. backup-user) and AuthorizedKeys-command RESTriction to limit SSH access to borg serve.
• Resource planning: Initial backups are CPU- and I/O-intensive; plan for increased RAM/CPU requirements on clients and potential peak loads on the repository host.

Initializing the repository, encryption strategies and key management

Borg supports encryption modes such as repokey (key stored in the repository, protected by a passphrase) and keyfile (private key stored externally). Repokey is operationally simpler, while keyfile enables stricter key management because the private key is kept separately. Key loss in encrypted repositories usually means permanent data loss — therefore plan key rotation, backup and retention processes.

# Repository lokal initialisieren (repokey)
borg init --encryption=repokey /srv/backup/repo

# Remote-Repository-Init per SSH (auf Backup-Host)
ssh backup-admin@backup.example.com "borg init --encryption=repokey /srv/backup/repo"

A secure AuthorizedKeys entry with RESTriction prevents interactive shell access and permits only Borg operations for the specified repository:

command="/usr/bin/borg serve --RESTrict-to-path /srv/backup/repo",no-agent-forwarding,no-port-forwarding,no-pty ssh-rsa AAAA... backup-client@example

Manage SSH keys and passphrases in a dedicated secrets store (e.g. HashiCorp Vault) and back up keys according to defined policies to a separate, offline-available location. Documentation and regular key rotation are operationally critical.

Practical backup workflow and robust scripting

Backups should be executed idempotently, atomically and with thorough logging. Use a wrapper script with set -euo pipefail, locking (e.g. flock), structured output for monitoring parsing and exit-code handling.

#!/usr/bin/env bash
set -euo pipefail

LOCKFILE=/var/lock/borg-backup.lock
exec 9>&1

flock -n 9 || { echo "Backup läuft bereits"; exit 2; }

export BORG_REPO=ssh://backup-user@backup.example.com:22/srv/backup/repo
export BORG_PASSPHRASE_FILE=/etc/borg/passphrase
export BORG_RSH="ssh -i /etc/borg/backup_key -o StrictHostKeyChecking=yes"

LOGFILE=/var/log/borg-backup/$(date +%F).log
mkdir -p $(dirname "$LOGFILE")

/usr/bin/borg create -v --stats --compression zstd,6 
  $BORG_REPO::"$(hostname)-$(date +%Y-%m-%d_%H:%M:%S)" 
  /etc /var/www /srv/data 
  --exclude '/var/cache' --exclude '/proc' --exclude '/sys' 2>&1 | tee -a "$LOGFILE"

exit_code=${PIPESTATUS[0]}

if [ "$exit_code" -ne 0 ]; then
  echo "Borg create failed with exit $exit_code" | tee -a "$LOGFILE"
  exit $exit_code
fi

# Prune nur nach erfolgreichem Backup
/usr/bin/borg prune -v --list $BORG_REPO --keep-daily=7 --keep-weekly=4 --keep-monthly=6 --keep-yearly=1 2>&1 | tee -a "$LOGFILE"

Run this script via systemd-timer, not via Cron, to obtain better startup behavior, automatic retry policies and native logging integration with journalctl.

Example: systemd service and timer

# /etc/systemd/system/borg-backup.service
[Unit]
Description=Borg Backup Job
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/run-borg-backup.sh
Nice=10

# /etc/systemd/system/borg-backup.timer
[Unit]
Description=Daily Borg Backup Timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Set Nice and CPU‑Affinity, if necessary, to reduce the impact of backup jobs on production services.

Deduplication: mechanism and operational consequences

Borg uses content-defined chunking: data is split into variable-sized blocks identified by a cryptographic hash. Deduplication saves space but has implications:

• Deduplication is repository-wide: savings occur only within the same repository, not across repositories.
• Chunking generates CPU and sometimes RAM load; clients with many small files benefit particularly, but chunk creation stresses resources.
• RESToring large amounts of data generates many small reads; plan I/O profiles and test recovery windows.

Remote storage options: evaluation and recommendations

SSH repository (borg serve) is the recommended option: well-tested code paths, correct lock handling and lower complexity. Alternative backends carry risks:

• NFS/SMB: can cause locking and consistency problems and promote repository corruption; mounted network filesystems are therefore not recommended.
• Object storage (e.g. S3): Borg does not support S3 natively; gateways (SFTP or filesystem bridges) are possible but increase complexity and must be evaluated for integrity, latency and performance.

Hardware recommendations for the repository

For repositories with high RESTore or write load, SSD-based backends are advantageous because they serve many small I/Os more efficiently. For pure long-term archival, cost-effective HDD arrays with appropriate RAID/erasure coding can be used, provided you account for recovery testing and sufficient bandwidth.

Retention strategies, prune planning and implications

Retention should balance recovery point objectives (RPO) against storage costs. Practically proven:

• Short-term retention: daily snapshots (e.g. 7 days)
• Mid-term: weekly and monthly snapshots
• Long-term: yearly archives for compliance

# Prune dry-run for verification
borg prune -v --list $BORG_REPO --keep-daily=7 --keep-weekly=4 --keep-monthly=6 --keep-yearly=1 --dry-run

Prune only removes archives; through deduplication chunks are physically deleted once they are no longer referenced by any remaining archive. Run regular dry-runs and document which RESTore points were lost to ensure SLA compliance.

Automated RESTore checks and validation workflows

A backup is only as good as its RESTore. Two levels of checks are practical:

1. Repository integrity: borg check --repository-only regularly; periodically borg check --verify-data in maintenance windows, since the latter is IO-intensive.
2. Functional RESTore test: extract critical artifacts (e.g. configurations, DB dumps) into a test environment and validate with checksums, DB import tests or by starting services in an isolated environment.

# RESTore validation: example for nginx configuration
set -euo pipefail
TMPDIR=$(mktemp -d /tmp/borg-RESTore-test-XXXX)
trap 'rm -rf "$TMPDIR"' EXIT

ARCHIVE=$(borg list --short $BORG_REPO | tail -n1)

borg extract $BORG_REPO::"$ARCHIVE" etc/nginx/nginx.conf --target "$TMPDIR"
sha256sum "$TMPDIR/etc/nginx/nginx.conf" | awk '{print $1}' > /tmp/RESTore-check-actual
# Compare the resulting checksum with an expected checksum

Automated RESTore checks should alert based on results (email/ChatOps/monitoring event) and run at least weekly for critical artifacts. Use test hosts or containers for isolation so validation does not alter production data.

Monitoring, parsing outputs and alerts

Collect these metrics for observability: last successful run time, duration, size of transferred data, number of deduplicated bytes, prune results and Borg exit codes. Write a robust parser that accounts for locale-dependent text variations.

# Example: very simple parsing (starting point only)
transferred_bytes=$(grep "Transferred" $LOGFILE | awk '{print $2}')
processed_files=$(grep "Number of files" $LOGFILE | awk -F: '{print $2}' | tr -d ' ')

# Send to monitoring (pseudo)
# curl -X POST http://monitoring.example.local/metrics -d "borg_transferred_bytes=$transferred_bytes"

For production environments, a dedicated exporter or connector that produces structured logs (JSON) or sends metrics directly to Prometheus/Grafana is recommended. Test the parser against different Borg versions.

Typical error cases and a quick check sequence

For backup failures follow a standardized check sequence that you should include in the incident runbook:

1. Check SSH connection:
   ShellCode kopieren

   ssh -vvv backup-user@backup.example.com

   — shows key and auth errors.

2. Check space on the repo host:
   ShellCode kopieren

   ssh backup@backup.example.com df -h /srv/backup

3. Check repo status:
   ShellCode kopieren

   borg list $BORG_REPO
   borg info $BORG_REPO::ARCHIVNAME

4. Lock issues:

borg break-lock $BORG_REPO

— only after analysis and if no Borg process is active.

borg check --repository-only $BORG_REPO

Avoid hasty repair attempts such as borg check --repair without first backing up the repository metadata; document every step.

Migration and emergency strategy

For migrations or emergencies, the following clear measures are recommended:

1. Back up repository metadata and create a filesystem snapshot of the backup host (e.g. an LVM or ZFS snapshot).
2. Perform a test RESTore on a separate host and validate critical workloads.
3. In case of repository corruption: run borg check --repository-only first, document the findings, contact the community or support, and then plan targeted repair actions.

Keep a fallback strategy ready: if a planned Borg version change fails, ensure you can roll back to the previous Borg version and continue working from a snapshot of the repository filesystem.

Performance tuning & filesystem integration

Optimizations that have proven effective in practice:

• Compression level: --compression zstd,6 is a good compromise between CPU load and size; higher levels save more space but cost more CPU.
• Files cache: Borg can use file-list caches; test BORG_FILES_CACHE for very large filesystems.
• Snapshots for consistency: for databases use storage snapshots (LVM, ZFS) or consistent dumps (e.g. pg_dump) before running Borg, since Borg is a file-level tool.

Repository maintenance: compact, upgrade and version changes

Regular maintenance helps limit the number of segment files and maintain performance. Use:

# Repository komprimieren/neu packen
borg compact $BORG_REPO

# Vor einem Versionswechsel: Backup aller Repository-Metadaten und Tests in Staging
borg upgrade --help  # prüfen, wenn Versionswechsel nötig ist

Perform repository maintenance during maintenance windows and test the effect on RESTore times and I/O load.

Best-practices checklist for operations

• Regular, automated RESTore checks (e.g. weekly for critical files)
• Separation of backup and production hosts; SSH hardening for backup users
• Document the prune policy; run prune dry-runs regularly
• Monitoring based on exit codes and structured logging
• Secure key management: backup and rotation of passphrases/keyfiles
• Staging tests for Borg version updates and repository maintenance
• Documented incident runbooks with clear verification sequences

Conclusion

BorgBackup is a proven solution for automated, deduplicating backups when repository topology, SSH security, encryption and RESTore validation are planned properly. Crucial are automated RESTore checks, a verifiable prune process and monitoring of job results. Deduplication significantly reduces storage and bandwidth needs, but requires careful resource planning and key management. With clear runbooks, regular tests and a strict monitoring workflow, you achieve a resilient backup strategy that reliably supports operations and recovery.

If you require a concrete implementation plan or an audit for your backup topology, you can derive an operationally reliable runbook from it that covers both storage and recovery requirements.

Automated Backups with BorgBackup: Geo-redundancy, Audit and File Attributes

In addition to routine scheduling you should explicitly consider three practical areas: geo-redundant storage, traceability of access, and the handling of file attributes/ACLs. These points directly affect recoverability, compliance and incident response.

Geo-redundancy and Replication Patterns

Borg does not provide built-in multi‑site replication. Established patterns are:

• Sequential pushing: write the backup sequentially to two repositories (local → Remote A → Remote B). Advantage: simple logic; disadvantage: longer total duration.
• Storage-level replication: use ZFS send/receive, block replication or object-store replication beneath the filesystem to produce atomic replicas. Advantage: consistent copies; disadvantage: higher infrastructure complexity.
• Snapshots as a transfer point: create a consistent storage snapshot (LVM/ZFS) and replicate that to the secondary site instead of copying individual repository files.

Make sure to include locks, consistency checks and bandwidth planning. Replication without integrity verification can multiply corruption.

Audit, Access Control and Traceability

RESTrict SSH access via the AuthorizedKeys command, log all Borg operations centrally (journal/syslog → SIEM) and instrument the backup account with audit rules (auditd) for file and process events. This way you detect unauthorized exports or RESTores and can reconstruct access times and responsible parties.

File Attributes, ACLs and SELinux‑contexts

Check whether your backups require extended attributes, POSIX ACLs and SELinux‑contexts (configuration files, home directories). Enable the corresponding archive options and validate on RESTore that permissions and contexts are preserved; this is critical for productive RESTarts.

Quick Emergency Runbook (Short version)

1. Ensure reachability: switch DNS/LoadBalancer to the RESTore host.
2. Quick check: run borg list / borg info on the secondary repo.
3. Integrity check: borg check –repository-only.
4. Failover RESTore: extract and validate critical configurations first.
5. Audit & documentation: log all steps, rotate keys if compromised.

These measures close the gap between ‚backup is running‘ and ‚we can RESTart productive operations in a disaster‘ and should be part of your operational runbook.

Backup pruning is also important for this topic. This article places these aspects in context and shows what matters in day-to-day operations.