SSH Hardening and Key Management: Bastion Hosts, Certificate Authorities and Key Rotation

In many environments SSH remains the primary operational access for Linux servers, virtual hosts, network gateways and also for Docker hosts. Precisely for that reason SSH hardening is not a „set-and-forget“ task, but an interplay of protocol hardening, access paths, identities (keys/certificates) and a reliable lifecycle for key material. Anyone who only adjusts individual controls (e.g. „root login off“) but tolerates key proliferation, missing rotation or uncontrolled jump paths merely shifts risk.

This article presents a practical target architecture with a Bastion-Host (jump host as the central entry point), a SSH Certificate Authority (CA for signing short-lived SSH certificates) and Key Rotation as a planned operational process. The focus is on operations, administration, auditability, troubleshooting, common pitfalls and a rollback strategy you do not have to improvise during an incident.

Why SSH hardening often fails without key management

SSH itself is cryptographically robust, but in practice failures usually occur in three areas:

• Identity chaos: long-lived personal keys without expiry, widely copied, without a central overview.
• Uncontrolled paths: direct access from arbitrary networks to production systems, often compounded by VPN exceptions or temporary firewall openings.
• Poor operational discipline: no rotation, no ability to revoke within minutes, no traceable changes to authorized_keys.

The consequences are creeping privilege escalation, difficult offboarding processes, high MTTR in incidents (because it is unclear which key still works) and gaps in compliance and audit requirements.

Target state: bastion host, SSH CA and short-lived access

A practical target state is not a „big bang“ but can be introduced incrementally:

• Bastion host: a hardened, closely monitored entry point through which SSH connections to internal segments run. The bastion host reduces the attack surface, centralizes logging and simplifies network rules.
• SSH certificates (OpenSSH Certificates): instead of distributing public keys on every target system, short-lived user certificates are signed by an SSH Certificate Authority. The target system trusts the CA and determines access rights via certificate attributes and local policies.
• Key rotation: host keys, CA keys and (where still necessary) user keys are rotated according to fixed rules. Rotation is an operational process with metrics, not just a cryptographic issue.

Important: „Zero Trust“ is often used as a buzzword; in the SSH context it concretely means weighting identity, context and validity (short lifetimes) more heavily than „who is on the network, therefore allowed“.

Hardening OpenSSH: sshd_config, crypto parameters, attack surface

The foundation remains proper hardening of the SSH daemon (sshd). The goal is: only necessary functions, secure crypto parameters, clear authentication paths.

Minimal principle for authentication and features

Many risks arise from a „compatible at any cost“ configuration. Typical measures:

• Disable password login (where organizationally feasible) in favor of keys or certificates.
• Control root login (ideally: disable; alternatively: only via certificate and tightly restricted).
• Allow port forwarding / tunneling selectively, instead of enabling it globally.
• Login grants via groups instead of individual accounts.

# Example: conservative sshd hardening (adapt to your policies)
# File: /etc/ssh/sshd_config

Protocol 2
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Allow only selectively, if you really need it
AllowTcpForwarding no
X11Forwarding no
PermitTunnel no

# Reduce exposed attack surface
MaxAuthTries 4
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2

# Control access by policy (example)
AllowGroups ssh-admins ssh-ops

# Logging: for traceability (without log flood)
LogLevel VERBOSE

Why this works: Every disabled authentication path is one less attack vector (credential stuffing, weak passwords, phishing). Reduced forwarding options prevent SSH from being used as a „universal tunnel“ for lateral movement.

When it fails: When you still have legacy automations in parallel (e.g., backups/deployments) that require password login or port forwarding. Therefore: inventory dependencies before disabling and move services to pilot hosts for testing.

Host keys: fingerprints, algorithms, replace planned Trust On First Use

Many teams live with „Trust On First Use“ (TOFU): the host key is accepted on first SSH contact and then cached. That is risky in large environments because a man-in-the-middle at first contact is hard to detect. Better: host certificates via an SSH CA or at least a managed known_hosts distribution.

If you rotate host keys or migrate from RSA to more modern defaults, plan the cascade: monitoring/automation, bastion, configuration management, developer laptops and break-glass clients.

Bastion host (Jump Host): architecture, operation and common pitfalls

A bastion host is a dedicated server in a tightly controlled segment that serves as the only SSH entry point into an internal network. It does not replace hardening of target systems, but it consolidates access controls and telemetry.

Network and firewall principles

• Inbound only from defined admin networks (e.g., VPN, Privileged Access Segment) to port 22 (or a fixed alternative port, but do not treat it as „security by obscurity“).
• Outbound from the bastion host only to the necessary target subnets/ports (usually 22). No „bastion may connect everywhere“.
• No direct reachability of production hosts from the user/office network.

Pitfall: If you operate Docker hosts, these often reside in networks that are „somehow reachable“ because build/registry access paths evolved historically. Separate management access (SSH) strictly from data paths (Registry, APIs) and avoid allowing a compromised build runner to speak SSH directly into production.

Session control and traceability

For audits it matters not only „who was allowed“, but also „who did what when“. Classic approaches:

• SSH-LogLevel VERBOSE for traceable key/certificate information.
• Central log forwarding (e.g. via journald/rsyslog) to an SIEM or log backend.
• Session Recording (keystrokes/terminal output) only when legally and organizationally cleared; technically useful primarily for highly privileged access.

Important: bastion logging does not replace the local logs on target systems. In an incident you need both perspectives (ingress and host events).

Checks: validating the bastion host in daily operations

# 1) Auf dem Bastion: eingehende SSH-Verbindungen prüfen (Linux)
ss -tnp | grep ':22 '

# 2) Journald: Authentifizierungsereignisse
journalctl -u ssh -S -2h --no-pager

# 3) Fail2ban/Rate-Limits (falls eingesetzt)
sudo fail2ban-client status sshd 2>/dev/null || true

If you already see „noise“ here (many failures from unexpected networks), that’s a signal: inbound rules and upstream controls (VPN, MFA, Conditional Access) are too permissive.

SSH Certificate Authority: distributing keys is no longer necessary

OpenSSH supports SSH certificates: a user still owns a key pair, but instead of placing the public key on every server in authorized_keys, a certificate is issued. This certificate contains, among other things, identity (Principals), validity (NotBefore/NotAfter) and optional restrictions. The server trusts the CA via a TrustedUserCAKeys entry.

Operational advantages

• Short-lived access (e.g. 8–24 hours) significantly reduces the risk of stolen keys.
• Offboarding becomes immediate: stop CA issuance, optionally keep certificates short-lived. No need to touch „authorized_keys“ on 200 hosts.
• Central policy: Principals can map roles (e.g. „ops“, „db-admin“) instead of maintaining per-host lists.
• Auditability: certificate serial numbers and Principals appear in logs.

Basic configuration on target hosts

On target systems you define trust in the user CA and map Principals to local permissions (typically via Unix groups and sudo rules). Example concept:

• The CA public key is stored as a file on the host.
• TrustedUserCAKeys points to it.
• AuthorizedPrincipalsFile or AuthorizedPrincipalsCommand specifies which principals are accepted for which account.

# File: /etc/ssh/sshd_config.d/10-ca.conf (example path, distro-dependent)

# Trust in the user CA
TrustedUserCAKeys /etc/ssh/trusted-user-ca.pub

# Define principals per target user
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

# Enforce certificates (optional, depending on your transition phase)
AuthenticationMethods publickey

# Create principals files (example)
sudo install -d -m 0755 /etc/ssh/auth_principals

echo 'ops' | sudo tee /etc/ssh/auth_principals/admin >/dev/null
echo 'breakglass' | sudo tee /etc/ssh/auth_principals/emergency >/dev/null

sudo chmod 0644 /etc/ssh/auth_principals/*

# reload sshd
sudo systemctl reload sshd

Why this works: The host no longer checks “is the public key in my file”, but “is this certificate signed by my CA and is the principal allowed for this account”. The CA thus becomes the central trust root.

When it fails: If there is time drift. Certificates are time-bound; with incorrect system time they will be “not yet valid” or “expired”. NTP/Chrony is therefore not optional. Check time stability especially after VM snapshots/RESTores.

Protect the CA keys: this is your „master key“

The User CA is highly critical. Whoever possesses the CA private key can sign arbitrary access. Minimum requirements:

• Offline or strongly isolated (ideally not on the bastion host itself).
• Signing via a controlled workflow (e.g. ticket/approval, short lifetimes, auditing).
• Backup and recovery with clear access control.

Many teams use an internal PKI or a secrets platform here. The decisive factor is not the tool, but that issuance, lifetimes and revocation/stop are operationalized cleanly.

Key rotation: Host keys, user keys, CA keys – and what commonly breaks

Key rotation is the planned replacement of keys before they are compromised. In the SSH context there are three classes you should consider separately:

• Host keys: The server’s identity toward clients. Rotation affects known_hosts, automation and monitoring.
• User keys / certificates: The identity of users/automation toward servers. With SSH certificates you primarily rotate the CA policy and rotate the user key base less often.
• CA keys: The trust root. Rotation is possible but must be planned as a migration window (trust multiple CAs in parallel).

Rotation without downtime: parallel trust and migration windows

A practical approach is „allow in parallel, then switch off“:

• Introduce a new CA and add trust on target hosts (TrustedUserCAKeys can contain multiple keys, either in a single file or across multiple files, depending on the setup).
• Clients receive new certificates from the new CA.
• After the old certificates expire and on the defined cut-off date, remove the old CA trust.

The same principle applies to host keys via managed known_hosts entries or host certificates.

Typical pitfalls during rotation

• Overlooked automations: CI runners, backup servers, monitoring checks, configuration management. These often use their own keys.
• Embedded clients: Appliances or old images with hard-coded known_hosts entries.
• ‚Once accepted‘: Admin workstations where host-key warnings were dismissed. This causes problems during an actual rotation.
• Time drift (for certificates) and DNS/IP inconsistencies (host-key mismatch).

Implementation in phases: from quick wins to a clean target architecture

If you currently have a heterogeneous environment (traditional servers, VMs, Docker hosts, possibly Kubernetes nodes), a staged plan is more realistic than a complete cutover.

Phase 1: Visibility and hygiene

• Inventory: Where is port 22 open? Which systems are directly reachable?
• Key hygiene: Which authorized_keys files exist where? Who owns them? Are there shared keys?
• Centralize logging: At minimum, send logs from the bastion and critical server classes to a log backend.

# Grober Check: Welche Accounts haben authorized_keys?
sudo find /home /root -maxdepth 2 -type f -name authorized_keys -print

# Inhalte prüfen (Vorsicht: sensible Daten)
# Tipp: nur Fingerprints der Public Keys extrahieren
sudo awk '{print $1" "$2}' /home/*/.ssh/authorized_keys 2>/dev/null | sort -u | head

Phase 2: Enforce bastion host

Technically enforce this via network rules (Security Groups/Firewall) and via SSH server policies on target hosts (only the bastion IP may reach Port 22). Organizationally, admin tools/runbooks must be adapted (ProxyJump, ProxyCommand).

# Beispiel: Client-Konfiguration ~/.ssh/config
Host bastion
  HostName bastion.example.net
  User admin

Host internal-*
  User admin
  ProxyJump bastion
  ServerAliveInterval 60

Pitfall: If you operate the bastion as a single point of failure, you are exchanging one risk for another. Plan at least redundancy (e.g. two bastions, separated failure domains) and a documented break-glass procedure.

Phase 3: Introduce SSH CA for user access

Start with a pilot segment. Use short lifetimes (e.g. one workday) and define principals so they match your roles. Build a revocation capability in parallel: if you can stop issuance, the maximum damage from a compromised certificate is limited to its lifetime.

Phase 4: Model automations cleanly (not ‚admin key for everything‘)

Automations need their own identities. Separate:

• Human admin access (interactive, short-lived, auditable)
• Machine access (non-interactive, strictly limited, ideally also via certificate logic or clearly isolated deploy keys)

If you administer Docker hosts via SSH (e.g. for emergencies), define dedicated principals/accounts for that and avoid CI systems using the same path.

Troubleshooting: When SSH certificates or bastion connections fail

In practice, you need quick diagnostic steps that work without specialist knowledge.

1) Certificate expired or “not yet valid”

Symptom: Login fails despite correct configuration. Cause is often time drift. Check:

# Auf Client und Zielhost: Zeit prüfen
date -u

# NTP/Chrony Status (Beispiel, distroabhängig)
chronyc tracking 2>/dev/null || true
timedatectl status

If the time is wrong: resolve the time issue first, then test again. Certificate authentication is unforgiving here.

2) Server trusts the wrong CA (or none)

Check whether the CA public key is correct and loaded:

sudo sshd -T | grep -i 'trustedusercakeys|authorizedprincipals'

# Datei vorhanden?
sudo ls -l /etc/ssh/trusted-user-ca.pub

Pitfall: configuration split across sshd_config.d and the ordering. Always validate with sshd -T, because that shows the effective configuration.

3) Bastion proxy drops (network/firewall)

If ProxyJump hangs or drops, it is often not an SSH issue but a routing/firewall issue. From the bastion, check the destination port:

# Auf dem Bastion-Host: Zielhost erreichbar?
TARGET=internal-host-01
nc -vz $TARGET 22

# Alternativ mit bash TCP (ohne nc)
timeout 3 bash -c "</dev/tcp/$TARGET/22" && echo OK || echo FAIL

If that fails, SSH logs on the client are of little help. The network path and Security Groups are then the correct level to investigate.

Checklist: SSH hardening in enterprise environments (operationally oriented)

• Network: Port 22 accessible only from the admin segment/bastion; no direct ‚temporary‘ exceptions without an expiration.
• Auth: Password login disabled or tightly RESTricted; clear group control (AllowGroups).
• Bastion: hardened, minimal software footprint, RESTrictive outbound, centralized logs, defined Break-Glass process.
• Certificates: SSH-CA in place, short lifetimes, Principals role-based, stable time/NTP.
• Rotation: Host-Keys/CA-Keys/User-Keys with a plan, parallel trust during the migration window, automations inventoried.
• Audit: LogLevel, centralized log correlation, traceability of admin access.

Fallback strategy: What to do if the new authentication blocks access?

Any hardening can, in case of an error, lead to a “Self-Inflicted Outage”. Therefore explicitly plan a fallback strategy:

• Out-of-Band access: console/iDRAC/IPMI/Cloud-Serial-Console – but secured and tested.
• Break-Glass Account: dedicated account with strictly controlled access, ideally only via bastion and active for a limited time.
• Staged Rollout: pilot first, then waves. Never switch the entire fleet simultaneously.
• Configuration changes atomic: roll out changes so that a reload is possible (and not a forced RESTart in the middle of a failure).

Practically proven: before any change to sshd_config, keep a second already-open session, check syntax (sshd -t) and only then reload.

# Sichere Prüfsequenz vor dem Reload
sudo sshd -t && echo "Config OK" || echo "Config ERROR"

# Erst wenn OK:
sudo systemctl reload sshd

Conclusion: SSH hardening becomes truly manageable with a CA and rotation

SSH hardening is most effective when you operationalize not just „sshd“ but the entire access path: bastion hosts reduce attack surface and consolidate telemetry; an SSH Certificate Authority makes identities controllable and short-lived; key rotation ceases to be a bothersome exception and becomes a planned process. Crucial is that you consider time stability (NTP), dependencies (automations) and a tested fallback strategy from the outset. Then SSH becomes not only more secure but also easier to administer in day-to-day operations.

SSH key management and SSH certificates are also important for this topic. This article places these aspects into a clear context and shows what matters in everyday operations.