Hardening Debian/Ubuntu 22.04 LTS: auditd, SSH, systemd profiles and checklist

This post explains how to harden Debian/Ubuntu 22.04 LTS, specifically using auditd for forensic audit logs, a robust SSH setup and systemd security profiles (systemd is the Linux init and service manager). The goal is a practical guide: prerequisites, common sources of error, verification procedures, concrete configuration examples and rollback strategies. The instructions target administrators, system engineers and operators who must run production servers reliably.

Hardening Debian/Ubuntu 22.04 LTS: Why hardening on LTS servers matters

Debian/Ubuntu 22.04 LTS is a widely used platform for servers. LTS stands for Long Term Support and means extended security updates; however, that alone does not protect against misconfigurations, attack vectors or compliance requirements. Hardening reduces the attack surface, improves traceability (audit) and facilitates incident response.

Important: Hardening is not a one‑time step but an operational process. Changes must be tested, documented and reversible.

Overview: Focus areas of this article

• Set up auditd and define audit policies — for traceability and compliance.
• Secure SSH — authentication, key management, protocols and session control.
• systemd security profiles (sandboxing) — service granularity, privilege reduction and exploit containment.
• Hardware operations: LVM snapshots, disk I/O, boot recovery and troubleshooting.
• Log centralization, TLS forwarding and integrity strategies for audit data.
• Verification and rollback strategies as well as a practical checklist for operations.

Auditd: Basics, benefits and typical use cases

Auditd is the Linux audit daemon (auditd). It collects security‑relevant events (e.g. file access, execve, SELinux/Auditing Hooks) and produces audit logs that are required for forensics, compliance (e.g. ISO, PCI) and intrusion detection. Auditd differs from syslog/journald: audit events are more structured and less lossy for evidentiary purposes.

Prerequisites and design decisions

Before installation check disk space, the central log path (e.g. remote syslog/audit server) and resources: auditd can generate I/O load with a large number of rules. Decide whether audit logs remain local, are forwarded to a log server (e.g. rsyslog/ELK/Graylog) or forwarded to a SIEM.

Installation and basic configuration

Install and enable auditd:

sudo apt update
sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd

By default the configuration is in /etc/audit/auditd.conf. Important parameters are LogFile (path), max_log_file (rotation in MB) and max_log_file_action (e.g. rotate or keep_logs). For production systems a conservative value for max_log_file and forwarding of the logs is recommended.

# Example: /etc/audit/auditd.conf (important lines)
log_file = /var/log/audit/audit.log
max_log_file = 200
max_log_file_action = rotate
space_left_action = SYSLOG
action_mail_acct = root

Audit rules: What belongs in the rule set?

Good audit rules focus on:

• Authentication events (sshd, sudo, su)
• Changes to /etc (configuration files)
• Binaries with SUID/SGID
• Critical files/directories (e.g. application configs, secret stores)
• execve for sensitive process start monitoring (use with care, as it is volume‑intensive)

Example of an audit.rules file:

# /etc/audit/rules.d/50-monitor.rules
# Authentication events (sshd, sudo)
-w /var/log/auth.log -p wa -k auth_changes

# /etc modifications
-w /etc/ -p wa -k etc_changes

# SUID/SGID binaries
-w /usr/bin/sudo -p x -k privileged

# Monitor sensitive config file
-w /etc/ssh/sshd_config -p wa -k sshd_config

# execve for specific users (example)
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k user_exec

Why this works: auditd installs kernel hooks that generate events even when changes are made by root. When it fails: overly broad execve rules quickly produce voluminous logs and cause performance issues. Test rules incrementally in a test environment.

Centralization and integrity

Forward audit logs to a central log server or SIEM. Ensure transport encryption (TLS) and integrity checks (e.g., signed log rotation or write-once storage). For short-term forensics, locally retained logs are useful; for long-term storage they belong in centralized, searchable stores.

Audit forwarding with rsyslog over TLS

Auditd itself has limited native remote capabilities; a proven approach is to keep the audit log locally and have rsyslog monitor the file (imfile module), then forward it to a central Rsyslog/SIEM server secured with TLS. This separates collection (auditd) from transport (rsyslog) and allows resumability in case of network issues.

# /etc/rsyslog.d/10-audit-imfile.conf (Client)
module(load="imfile")
input(type="imfile"
      File="/var/log/audit/audit.log"
      Tag="audit"
      Severity="info"
      PersistStateInterval="200")

# TLS forwarding (Auszug)
*.* action(type="omfwd" Target="logs.example.internal" Port="6514" Protocol="tcp" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="logs.example.internal")

On the central Rsyslog/SIEM side, use server certificates, client certificate verification and appropriate firewall rules. Test whether the file monitoring buffers and resumes after a network outage.

Hardening SSH: authentication, keys, protocol and operational rules

SSH remains the standard access protocol for Linux‑administration. Vulnerabilities arise from weak keys, default configurations, open root logins and missing session control. A hardened SSH mitigates credential theft and lateral access.

Preventive measures

Basic rules:

• Disable root login over SSH; use sudo with audit tracking.
• Allow only modern crypto algorithms (e.g., Ed25519, ECDSA); disable legacy ciphers.
• Limit authentication attempts and deploy Fail2ban or similar protective mechanisms.
• Use a bastion/jump‑host architecture instead of direct SSH access from the internet.

Concrete sshd_config optimizations

Edit /etc/ssh/sshd_config. Here is a recommended minimal set:

# /etc/ssh/sshd_config (Auszug)
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
AllowUsers adminuser
ClientAliveInterval 300
ClientAliveCountMax 2
PermitEmptyPasswords no
KexAlgorithms curve25519-sha256@libssh.org
HostKey /etc/ssh/ssh_host_ed25519_key
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com
LogLevel VERBOSE

Explanation: PasswordAuthentication no forces key-based login; PermitRootLogin no prevents direct root access; LogLevel VERBOSE generates usable audit events (and should be correlated with Auditd). When it fails: If all admins suddenly have no key – plan transitions and test accounts.

Key‑Management und Rotation

Organize keys centrally (e.g. via SSH CA or an identity management system). SSH Certificate Authority (SSH CA) enables short-lived certificates instead of permanent authorized_keys. Rotation is important: regularly audit existing keys and revoke them when employees change.

# Beispiel: ssh-keygen für Ed25519
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""

# Signieren eines Nutzerkeys mit einer SSH-CA (Kurzbeispiel)
ssh-keygen -s /etc/ssl/ssh/ca_key -I user-cert -n username -V +52w user_key.pub

systemd‑Sicherheitsprofile: Dienst‑Sandboxing und sichere Unit‑Konfiguration

systemd offers numerous security options at the unit level: ProtectSystem, ProtectHome, PrivateTmp, NoNewPrivileges, CapabilityBoundingSet, etc. These settings reduce a service’s privileges and limit impact in case of compromise.

Grundprinzip: Least‑Privilege für Dienste

Each service receives only the rights it actually needs. systemd‑Drop‑ins (Directory: /etc/systemd/system/.d/*.conf) are a safe way to maintain changes without modifying the package unit file.

Beispiel: systemd Drop‑In zur Härtung einer Web‑Service‑Unit

# /etc/systemd/system/myweb.service.d/hardening.conf
[Service]
PrivateTmp=true
ProtectSystem=full
ProtectHome=yes
NoNewPrivileges=true
RESTrictAddrFamilies=AF_INET AF_INET6 AF_UNIX
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
ReadOnlyPaths=/etc/myweb
# Optional: Limit CPU/Memory
MemoryMax=500M
CPUQuota=70%

Why this works: ProtectSystem=full makes /usr and /boot read-only for the service; PrivateTmp isolates temporary files; NoNewPrivileges prevents SetUID/execve after privilege escalation. When it fails: Some applications need write access to /usr or global tmp directories; test Drop‑ins incrementally.

Testing und Canary‑Deployment

Use canary hosts or rollouts with reduced load. For systemd changes the following is recommended:

1. Create the Drop‑in
2. systemctl daemon-reload
3. RESTart during a maintenance window
4. Monitor startup time, OOM events and application-specific errors

sudo systemctl daemon-reload
sudo systemctl RESTart myweb.service
journalctl -u myweb.service -f

Hardware: LVM‑Snapshots, Disk‑I/O und Boot‑Recovery

The hardware layer strongly affects hardening and operations: full partitions, high I/O latency or a faulty boot device can render hardening measures ineffective or prevent critical rollbacks. Here are some established action areas and troubleshooting steps.

Platzplanung und Monitoring

Audit logs grow. Plan dedicated partitions or LVM volumes for /var/log and /var/log/audit. Monitor free space, inode usage and I/O with simple tools:

# freien Speicher prüfen
df -h /var/log
# I/O Belastung beobachten
iostat -xz 1 10
# Top Prozesse nach I/O
iotop -oPa

Typical pitfall: logrotate disabled or faulty → full log partition → services hang.

LVM Snapshot als Rollback‑Option

Before extensive changes (systemd‑Drop‑ins, SSH mass rollout, audit rule update) create an LVM snapshot of the affected volume. Snapshots allow fast RESTore without the time required to RESTore from backups.

# Beispiel: Snapshot anlegen (root-Volume lv_root im vg0)
sudo lvcreate --size 2G --snapshot --name root_snap /dev/vg0/root

# Änderungen durchführen ...

# Snapshot zurückspielenvia lvconvert (oder Mount snapshot und rsync zurück)
sudo lvconvert --merge /dev/vg0/root_snap
# Danach Neustart erforderlich
sudo reboot

Warning: snapshots consume space; under high write load they grow quickly and can fill the volume. Invalid snapshots degrade performance.

Boot‑Recovery and Out‑of‑Band

Ensure IPMI/iLO/DRAC or the provider console is functional. Without out‑of‑band access you will have limited options in case of lockouts (e.g. SSH). Document rescue accounts and keep rescue keys separate from central automation.

NTP/chrony and timestamp integrity

Consistent time is critical for audit and forensics. Use chrony or systemd‑timesyncd, validate NTP peers and set up alerting for drift. Timestamps with discrepancies make log correlation difficult.

# Chrony Status prüfen
chronyc tracking
chronyc sources -v

# systemd-timesyncd Status
timedatectl status

Checks, monitoring and typical pitfalls

Before changes verify the baseline state: kernel version, installed packages, active services and current audit rules. Typical pitfalls include:

• Overly broad auditd‑rules → log explosion
• sshd_config too RESTrictive → admins locked out
• systemd RESTrictions that block legitimate file/socket access
• Lack of centralized logging → loss of forensic data
• Missing out‑of‑band console → difficult recovery

Essential check commands

# auditd Status und Regeln
sudo systemctl status auditd
sudo auditctl -l

# SSH Überprüfen
sshd -t
sudo systemctl reload sshd

# systemd-Unit check
systemd-analyze blame
systemctl status myweb.service
journalctl -u myweb.service -n 200

# Disk und Snapshotchecks
lvs -a
df -h /var/log
iostat -xz 1 5

Rollback strategies and emergency access

Plan a rollback option for every change. For SSH changes consider a parallel admin account with key access that is not subject to the new rules, or use an out‑of‑band console (iLO, DRAC, IPMI). For systemd changes keep the drop‑in as a file with versioning and create snapshots (LVM/LUKS) or system backups before critical changes.

# Beispiel Rollback systemd: Drop-in entfernen und daemon neu laden
sudo rm /etc/systemd/system/myweb.service.d/hardening.conf
sudo systemctl daemon-reload
sudo systemctl RESTart myweb.service

Advanced practical checklist: step by step

This checklist is intended for production rollouts. Test each step in a staging environment.

1. Backup: create current system and configuration backups, snapshot if possible.
2. Check space & I/O: ensure /var/log is on a separate volume, configure monitoring.
3. Auditd: install, create base rules, perform local tests.
4. Logs: configure in‑file monitoring and TLS forwarding to the central log system and verify TLS.
5. SSH: enable key‑based auth, disable RootLogin, set up test accounts and store an emergency key.

Security operations and maintenance

Maintenance means: periodic rule reviews, log retention checks and incident exercises. Audit rules age with service behavior; perform checks after releases and major updates. Implement automated tests for sshd_config (sshd -t checks) and systemd unit start in CI/CD for infrastructure changes. Also plan capacity reviews for log volumes and snapshot resources.

Concrete troubleshooting scenarios

auditd generates high I/O load

Cause: overly generic execve rules or monitoring of very active directories. Immediate measures: disable rules stepwise, consider write-back to a central log server, pause auditd only as an emergency measure. Long term: sampling rules, targeted process monitoring and filtering.

Admins locked out after sshd_config change

Rollback: access via out-of-band or console, or an audit step to reproduce: RESTore /etc/ssh/sshd_config from backup and systemctl RESTart sshd. Prevention: always keep a fallback key on a dedicated rescue user that is not managed by automation.

Service fails to start after systemd drop-in

Check journalctl -u for errors related to the applied RESTrictions (e.g. Permission denied on paths). Temporarily remove the drop-in, reload and RESTart to RESTore the service. Use strace or fault-specific logs if necessary.

Conclusion

Hardening Debian/Ubuntu 22.04 LTS means more than isolated configuration changes: it is a layered operational concept. auditd provides traceability, hardened SSH secures access, systemd security profiles limit risks at the service level and hardware/operational measures ensure availability. Test every change in a controlled manner, plan rollbacks and integrate hardening into your lifecycle process (Release, Monitoring, Review). This minimizes outage risks and creates a maintainable security foundation.

Recommended next steps

• Set up a test cluster and automate the verification steps.
• Integrate audit logs into your SIEM and alerting with TLS-secured forwarding.
• Document systemd drop-ins, LVM snapshot procedures and key management policies.

A structured implementation reduces operational risks and facilitates compliance.

For this topic, configuring auditd and systemd security profiles are also important. The article contextualizes these aspects and shows what matters in day-to-day operations.