MFA for SSH: YubiKey, PAM‑FIDO2, Jump‑Hosts and pragmatic fallback strategies for admin access

A secure remote admin access is a basic prerequisite for reliable operation. MFA for SSH, i.e. the combination of a public key and a second factor (e.g. FIDO2/YubiKey), significantly reduces the risk of compromised passwords or stolen SSH keys. In this article I explain how the technology works, which components (YubiKey‑resident keys, PAM‑FIDO2, OpenSSH, jump hosts) interact, which typical pitfalls and risks occur, and how to plan real fallback strategies for emergencies. The target audience is admins, system engineers and operations teams looking for secure, practical procedures.

What does MFA for SSH mean and why is it important?

MFA (Multi‑Factor Authentication) is an authentication principle involving at least two distinct factors: something the user possesses (e.g. a YubiKey) and something they know or have (a private SSH key or a password). For SSH this typically means publickey authentication plus FIDO2/U2F touch or a PAM‑based second stage. The concrete benefit is: even if a private SSH key has been exfiltrated, the absence of the physical token prevents access.

MFA for SSH: core elements and variants

OpenSSH with FIDO2‑resident keys

OpenSSH has supported key types such as ed25519-sk, which use FIDO2 tokens, in recent releases. Resident keys are private keys stored directly on the token. This replaces local key files and enforces Touch/PIN when using the token. The drawback is the dependency on the token lifecycle — plan replacement and recovery procedures.

# Resident FIDO2‑Key erstellen (speichert Key auf Token, verlangt Touch beim späteren Login)
ssh-keygen -t ed25519-sk -f ~/.ssh/id_ed25519_sk

PAM‑FIDO2 as the central second factor

PAM (Pluggable Authentication Modules) is the modular authentication system under Linux. A PAM‑FIDO2 module (e.g. pam_fido2 or libpam-u2f) can be integrated into /etc/pam.d/sshd to require SSH logins to use FIDO2 in addition to publickey. PAM is powerful, but a faulty configuration can block logins entirely — therefore test first in an isolated environment.

# Minimaler relevanter Ausschnitt von /etc/ssh/sshd_config
PubkeyAuthentication yes
ChallengeResponseAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive

# Beispiel‑PAM‑Snippet in /etc/pam.d/sshd (schematisch)
auth    required    pam_fido2.so debug
account required    pam_nologin.so

Important: the AuthenticationMethods parameter forces OpenSSH to require the publickey first and then trigger PAM (keyboard‑interactive), which then performs the FIDO2 prompt. Incorrect ordering or missing modules lead to lockouts.

SSH certificates as a complement

SSH certificates are signatures for public SSH keys issued by an internal CA. They simplify revocation and centrally controlled validity periods. As a complement to MFA they allow granting short‑term access without distributing files to user devices — ideal for temporary emergency access.

# CA-Schlüssel erzeugen (auf geschütztem CA‑Host)
ssh-keygen -t ed25519 -f /root/ssh_ca_key -N ""
# Benutzer-Öffentlichen Schlüssel signieren (z. B. 1 Stunde gültig)
ssh-keygen -s /root/ssh_ca_key -I emergency -n admin -V +1h user.pub

Jump‑Hosts and bastion architecture: central to MFA operation

A jump host (bastion host) is a controlled access point into a protected segment. It reduces the attack surface and enables centralized authentication, session recording and access control. Key operational principles:

• Harden the jump host: minimal software stack, strict firewall rules, restricted user accounts.
• Enforce MFA on the jump host; only connections via the bastion should be permitted to production hosts.
• Session recording and audit: add recordings (e.g. tty recording or auditd) for forensic traceability.

# ProxyJump usage (Admin workstation → bastion → target host)
ssh -J bastion.example.com admin@internal-host.example.net

Agent forwarding should generally be disabled, because it allows attackers to leverage the agent via a compromised target machine. Enable it on the workstation level only selectively for trusted sessions.

Practical PAM and token configuration: examples and checks

PAM mapping for FIDO2 differs by module. Typical is a mapping file that assigns a token identifier to a Unix UID. Examples help identify potential sources of error.

# Example: /etc/u2f_mappings (schematic)
# token_hex_serial:username
1234567890abcdef:alice
fedcba0987654321:bob

# Client-side debug tools
# Check whether token is detected
ssh -v -i ~/.ssh/id_ed25519_sk alice@bastion.example.com
# PAM debug logs (on the target host)
sudo journalctl -u sshd -f

When testing, note: PAM modules often write their own debug logs; enable debug mode only temporarily so that sensitive information does not remain permanently in log files.

Rollout plan: pilot to production

A structured, phased rollout minimizes operational interruptions:

1. Pilot: 2–5 admins, isolated test jump-host environment and complete test checklist.
2. Expansion: inclusion of the operations team, logging KPIs and emergency drills.
3. Production: organization-wide rollout, training, on/offboarding processes operationalized.

During the pilot phase you should systematically test:

• Generation and use of resident keys.
• PAM error scenarios (token defective, PIN errors) and the associated log entries.
• Backup scenarios (break-glass, temporary SSH certificates, OOB console).

Fallback strategies in detail — plans, checklists and automation

A fallback strategy prevents a lost token or a PAM problem from crippling operations. Good fallback plans are tiered and documented.

Break-Glass: process flow

A Break-Glass account is a tightly controlled emergency identity. Suggested process flow and technical measures:

• Maintain a Break-Glass account per critical service, protected by a physical key copy (in a safe) or a time-limited certificate.
• Before use: approval via a defined dual-control process (e.g. second signature in ticketing) and automatic rotation of the password after expiry.
• Every use is automatically audited and triggers an alert chain (pager/SMS/email) to incident responders.

Temporary SSH certificates via signing service

An automated signing service (internal tool with RBAC) can issue short-lived certificates when tokens are missing. Implement audit steps and short validity intervals (e.g. 15–60 minutes). A simple systemd-driven signer with auth hooks is often sufficient.

# Example: temporary certificate (1h validity)
ssh-keygen -s /root/ssh_ca_key -I emergency -n admin -V +1h user.pub
# Check validity (local):
ssh-keygen -L -f user-cert.pub

Physical OOB access and console

Out‑of‑Band (IPMI/Redfish, console servers) must not only be present but also hardened and regularly tested. OOB ensures you have access even if SSH services or authentications fail. Rules:

• Physically/logically isolate OOB networks.
• Allow access to OOB only via MFA‑protected admin VMs.
• Regularly patch firmware and remove default credentials.

Concrete troubleshooting checks

When login fails, systematic debugging quickly reveals the cause. Example sequence:

1. Verbose SSH on the workstation: ssh -vvv check whether the sk key is offered.
2. Check server logs: sudo journalctl -u sshd -b or /var/log/auth.log.
3. Temporarily test PAM modules with a separate PAM context or on a test host.
4. Check token health: Is the token damaged? PIN attempts exceeded?

# Example commands
# Debug on client
ssh -vvv -i ~/.ssh/id_ed25519_sk alice@bastion.example.com
# Server logs
sudo journalctl -u sshd -n 200
# Search for FIDO errors
sudo journalctl -u sshd | grep -i fido || sudo grep -i fido /var/log/auth.log

Token lifecycle management

Token lifecycle includes issuance, replacement, revocation and disposal. Manage tokens centrally with inventory, responsibilities and time windows:

• Issuance: Documented handover with signature and ticket reference (e.g. Zammad ticket number).
• Replacement: Standard process for damaged or non‑responsive tokens; issue a temporary certificate for reprovisioning.
• Revocation: On loss, immediately mark the token as compromised and revoke associated SSH certificates and PAM bindings.
• Disposal: Securely erase token (if possible) and physically destroy when decommissioned.

For automation you can use YubiKey Manager CLI (ykman) to query information about serial numbers and configured slots. Note that not all token models support the exact same commands.

# YubiKey Manager: show serial number
ykman info

Integration into directory services and CI/CD

In large environments admins often authenticate against LDAP/AD. PAM‑FIDO2 can run in parallel with LDAP bindings: LDAP provides account information, PAM‑FIDO2 validates the second factor. Pay attention to the order in /etc/pam.d/sshd so LDAP account checks do not block PAM‑FIDO2.

CI/CD runs and automations must not depend on physical tokens. For pipeline jobs use machine accounts with SSH certificates or host keys that are centrally managed and time‑limited. Store secrets in vaults and rotate them regularly.

Typical pitfalls and how to avoid them

• Incorrect PAM configurations: Always test with a separate admin account and keep an OOB console available.
• Incompatible token models: Not all FIDO2 tokens support resident keys; check the hardware matrix before purchase.
• USB passthrough in virtual desktops: Tokens are sometimes not reliably forwarded; test in your user environments.
• Agent Forwarding: Disable it, as it can undermine MFA protection on the workstation.

Specific notes for Zammad operations teams

Zammad instances often require differentiated access roles: developers, application maintainers, DB admins. Practical recommendations:

• Perform Zammad-specific admin tasks only via the bastion and separate DB admin sessions from application deployments.
• Record token assignments and Break‑Glass events directly in the ticketing system (e.g. Zammad) so audit trails and change context are preserved.
• For emergency recovery, retain a procedure for granting database access with minimal privileges and temporary certificates — documented in the Disaster‑Recovery‑Runbook.

{
  "ticket_type": "Break-Glass",
  "summary": "Temporärer Zugang: CA-Zertifikat ausstellen",
  "requested_by": "alice",
  "approver": "operations_lead",
  "reason": "Verlorener YubiKey",
  "issued_certificate_ttl": "60m",
  "audit_note": "Certificate issued per emergency procedure"
}

Checklist for production (short)

• Run a pilot with 2–5 admins and document lessons learned.
• Harden the jump host, enforce MFA, disable agent forwarding.
• Set up temporary SSH certificates and test the signer service.
• Define break-glass accounts, implement an approval workflow and enable auditing.
• Verify OOB access and conduct semi-annual emergency drills.

Conclusion and recommendations

MFA for SSH with YubiKey and PAM‑FIDO2 is a practical measure that makes administrative access materially more secure. Critical is a responsible rollout: a pilot group, documented onboarding/offboarding processes, tested fallback strategies and strict logging. For Zammad operations teams additionally apply strict separation of access, document token assignments in the ticketing system and rehearse recovery procedures.

Quick checklist to take away:

• Define a pilot group and build an isolated test environment.
• Test OpenSSH‑sk keys and PAM‑FIDO2 in parallel and validate logs.
• Harden jump hosts; disable agent forwarding by default.
• Implement and rehearse fallback strategies (Break‑Glass, OOB, temporary certificates).
• Integrate token inventory and change processes into ticketing (e.g. Zammad) and schedule regular emergency exercises.

If needed, a focused workshop is appropriate to consolidate technical options, organizational roles and emergency playbooks and integrate them into your operational processes.

Operational perspectives: CA protection, fail policies and bastion HA

Beyond user authentication, hardening and operational architecture determine the production readiness of MFA‑SSH. Protect your SSH CA private keys like production cryptographic material: ideally in an HSM or at least offline on a dedicated, secured host. Define a clear compromise plan: key rotation, emergency recovery and communication playbooks in the event of key loss.

Decisions on fail policies are critical: a PAM outage can either deny logins (fail‑closed) or permit access (fail‑open). Prefer fail‑closed with tested OOB fallback paths rather than silent exceptions, because the latter undermine auditability and accountability.

Achieve bastion high availability through active backup instances, centralized session recording and distributed signer services for SSH certificates. Synchronize only public metadata (CA public keys, audit logs); private keys remain strictly isolated.

Automate token lifecycle and emergency certificates via your ticketing API (e.g. Zammad) or your bespoke enterprise software to seamlessly link issuance, revocation and audit. Monitor metrics: authentication failure rate, signing rates, PAM latency and break-glass events — alerts on deviations are mandatory.

# Sicherer CA‑Rotation‑Ablauf (vereinfachtes Beispiel)
ssh-keygen -t ed25519 -f /root/ssh_ca_key_new -N ""
cp /root/ssh_ca_key_new.pub /etc/ssh/trusted_user_ca_keys/ca_new.pub
# Signieren und testen, dann swappen
ssh-keygen -s /root/ssh_ca_key_new -I test -n admin -V +5m user.pub

For this topic, PAM-FIDO2 and jump hosts are also important. The article contextualizes these aspects clearly and shows what matters in day-to-day operations.