Automate internal PKI renewal: integrate ACME, HashiCorp Vault and systemd timers

Introduction: Automating internal PKI renewal is essential for operators and administrators when many internal TLS certificates are managed — for example for internal services, IoT gateways, VPNs or client authentication. In this article I show a practical approach to couple ACME (Automated Certificate Management Environment — protocol for automated certificate issuance) with HashiCorp Vault (Vault — secret management and optional PKI signing engine) and to use systemd‑timers (systemd‑Timers — modern scheduler on Linux with journald integration), so that renewals are reliably scheduled, executed and distributed.

Why automating PKI renewal is necessary

Certificates typically expire simultaneously on hundreds of hosts. Manual replacement is error‑prone, causes outages and audit gaps. Automation reduces effort, lowers outage risk and ensures consistent security processes. Governance remains central, however: validity period, roles, audit logs and least‑privilege access concepts must be defined before going into production.

Automating internal PKI renewal: Architecture overview

The recommended separation consists of three layers:

• Signing layer: Vault as internal CA/signer. Vault’s PKI engine signs CSRs; the root key should ideally remain offline.
• Protocol/Integration layer: an ACME proxy or ACME‑compatible frontend service receives ACME requests and translates them into Vault sign calls.
• Operations layer: systemd‑timers on hosts or central runners schedule and execute renewal jobs as well as distribute and validate certificates.

This separation improves security, auditability and operational flexibility: signing, authentication and scheduling have distinct responsibilities.

Design decisions and prerequisites

Important questions before starting:

• Root vs. intermediate: Use Vault as an intermediate signed by an offline root. This minimizes root exposure.
• Validity period: Shorter lifetimes (e.g. 90 days) increase security but also raise renewal frequency and load.
• Trust distribution: Ensure clients receive the CA bundle (via CM tooling, package, MDM).
• Machine authentication: mTLS (e.g. with production machine certificates) or Vault AppRole are common patterns — both have trade‑offs regarding secret handling and lifetime.
• Observability and audit: Enable Vault audit devices and collect proxy/host logs.

Implementation patterns: direct client request vs. ACME proxy

Two common patterns:

Direct ACME client on hosts

Clients (e.g. acme.sh, certbot) request certificates themselves. Advantage: decentralized and robust; disadvantage: more configuration effort and more complex authentication to Vault.

Central ACME proxy

A central proxy validates requests, authenticates hosts (mTLS, OAuth), and talks to Vault. Advantage: central control, simplified auditing; disadvantage: additional network path and availability requirements.

Concrete setup: Vault PKI, ACME proxy, systemd‑timers

The core steps are:

1. Enable Vault PKI engine and configure an intermediate.
2. Implement an ACME proxy (e.g. with lego or a lightweight Go service component) that authenticates requests and maps them to Vault.
3. Create systemd timer units that periodically run renewal scripts, check certificates and reload services.

Vault PKI: example commands

vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
vault write pki/root/generate/internal common_name="Company Internal Root" ttl=87600h
vault write pki/intermediate/generate/internal common_name="Company Intermediate" | tee csr.json
vault write pki/root/sign-intermediate csr=@csr.json format=pem_bundle ttl=43800h > signed_intermediate.pem
vault write pki/intermediate/set-signed certificate=@signed_intermediate.pem

Vault thereby separates root and intermediate, enabling secure key management. If signatures fail: check Vault token permissions, audit logs and TTL settings.

ACME‑Proxy: Aufgaben und Mapping

The proxy should:

• Authenticate hosts (mTLS or token).
• Validate values such as CN/SAN against allowed patterns.
• Map requests to a Vault role and have them signed.
• Enforce rate limiting and produce audit records.

Example YAML mapping:

acme:
  auth_method: mTLS
  allowed_roles:
    - name: webserver
      allowed_sans:
        - '*.svc.internal'
      vault_role: webserver-role
    - name: dbserver
      allowed_sans:
        - 'db-*.internal'
      vault_role: dbserver-role
rate_limit:
  requests_per_minute: 60

systemd‑timers: reliable scheduling

systemd timers provide RandomizedDelaySec, journald integration, Persistent=true (runs jobs after system outage) and better control than cron. Example service+timer:

# /etc/systemd/system/pki-renew.service
[Unit]
Description=Run PKI renewal script
After=network-online.target

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/pki-renew.sh

# /etc/systemd/system/pki-renew.timer
[Unit]
Description=Timer for PKI renewal

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=3600
Persistent=true

[Install]
WantedBy=timers.target

The script itself should operate atomically: lockfile, temporary directory and clean exit codes. Example skeleton:

#!/bin/bash
set -euo pipefail
LOCKDIR=/var/lock/pki-renew.lock
if ! mkdir "$LOCKDIR" 2>/dev/null; then
  echo "Another run in progress" >&2
  exit 0
fi
trap 'rm -rf "$LOCKDIR"' EXIT

# Anfrage an ACME-Proxy und lokale Installation
/usr/local/bin/acme-request --cn "$(hostname -f)" --out /etc/ssl/private/host.pem
systemctl reload nginx || systemctl RESTart nginx

Practical example: Vault‑Policy and ACME‑Role

Policies limit which roles may sign which SANs/CNs. Example Vault policy (HCL):

# webserver-policy.hcl
path "pki/issue/webserver-role" {
  capabilities = ["create", "update"]
}

# Allow read of CA bundle
path "pki/cert/ca" {
  capabilities = ["read"]
}

The role itself defines SAN patterns and TTL:

vault write pki/roles/webserver-role  
  allowed_domains="svc.internal"  
  allow_subdomains=true  
  max_ttl="72h"

If the policy is too permissive, abuse is possible; if it is too RESTrictive, automation breaks. Test iteratively in staging.

Deployment and atomic installation of certificates

A common operational error is an inconsistent certificate installation where files are partially written and services RESTart with incomplete files. Use atomic patterns: write new artifacts to a temporary path, validate the file and swap via symlink. This provides a consistent filesystem view and makes rollback simple.

# atomare Installation
TMPDIR=/tmp/new-cert-$$
mkdir -m 700 "$TMPDIR"
cp cert.pem "$TMPDIR/"
cp key.pem "$TMPDIR/"
# Validieren
openssl x509 -noout -text -in "$TMPDIR/cert.pem" >/dev/null
# Atomisch tauschen
mv /etc/ssl/current /etc/ssl/old-$(date +%s) || true
ln -sfn "$TMPDIR" /etc/ssl/current
systemctl reload myservice

Auf Windows oder in containerisierten Umgebungen weichen Sie auf passende Mechanismen aus: z. B. PKCS#12‑Bundles für IIS/Windows oder Mount‑Overlays für Container.

HSM, TPM und Vault Transit: Schlüssel nie exportieren

Für besonders schützenswerte Schlüssel verwenden Sie HSMs (Hardware Security Module — dedizierte, zertifizierte Key‑Stores) oder TPM (Trusted Platform Module — motherboard‑gebundenes Root of Trust). Vaults Transit Engine erlaubt signieren/verifizieren, ohne private Keys zu exportieren. Betriebsfolge: Root in HSM, Vault konfiguriert Transit als Signer, intermediates werden erzeugt und signiert über Transit, Clients erhalten nur Zertifikate.

Container und Kubernetes: Besonderheiten

In Cloud‑ und Containerumgebungen gelten zusätzliche Einschränkungen: Dateisysteme sind ephemer, Init‑Containers können Zertifikate vor Start bereitstellen. Kubernetes‑Cluster verwenden häufig Secrets; hier sollten Sie die Secrets verschlüsseln (e.g. Sealed Secrets) und Rollouts über Deployments mit readiness‑Probes steuern. Achten Sie auf die Rechte von kubelets: diese dürfen nicht unbegrenzten Zugriff auf Signierpfade erhalten.

Monitoring‑Details und Metriken

Bauen Sie Metriken granular auf: pki_renew_requests_total, pki_renew_success_total, pki_renew_failures_total, pki_cert_age_seconds sowie exporter‑seitige Gauges für verbliebene Tage bis Ablauf. Alerts sollten nicht nur Fehlerquoten, sondern auch Anstiege der Renew‑Latenz beobachten — ein Indikator für Performance‑Probleme beim Proxy.

Chaos‑Tests, Staging‑Checks und Validierung

Testen Sie resilient: simulieren Sie Vault‑Ausfälle, Netzwerk‑Partitionen und Rate‑Limit‑Hits. Führen Sie Canaries durch, bevor Sie global umstellen: eine kleine Gruppe von Hosts erhält das neue CA‑Bundle, z. B. per Ansible‑Playbook. Nutzen Sie synthetische End‑to‑End‑Tests, die ein TLS‑Handshake vor und nach Renewal prüfen.

Operational Runbook: Schritt‑für‑Schritt bei Renewal‑Fehlern

1. Prüfen Sie Systemzeit: timedatectl status — Clock drift bricht TLS.
2. Vault Health und Audit: vault status und Audit‑Logs einsehen.
3. Proxy‑Logs: prüfen Sie 401/403 (Auth), 429 (Rate limit) und 5xx (Serverfehler).
4. Host‑Skript prüfen: Lockfiles, SELinux‑Kontext, fehlende Binaries, Berechtigungen.
5. Führen Sie manuelles Request per Staging‑ACME aus, um Pfad‑Probleme zu identifizieren.

Beispiel: robustes Renewal‑Skript mit flock und Logging

#!/usr/bin/env bash
set -euo pipefail
exec 3>&1
LOG=/var/log/pki-renew.log
flock -n /var/lock/pki-renew.lock -c "bash -c '
  echo "$(date -Iseconds) START" | tee -a $LOG
  /usr/local/bin/acme-request --cn "$(hostname -f)" --out /etc/ssl/private/host.pem || { echo "request failed" | tee -a $LOG; exit 2; }
  chown root:ssl-cert /etc/ssl/private/host.pem && chmod 640 /etc/ssl/private/host.pem
  systemctl try-reload-or-RESTart myservice || { echo "reload failed" | tee -a $LOG; exit 3; }
  echo "$(date -Iseconds) OK" | tee -a $LOG
'"

Exit‑Codes helfen Alerting‑Tools, eindeutige Ursachen zuzuweisen. Logrotation nicht vergessen.

Typische Stolperfallen und Vorsichtsmaßnahmen

• Uhrzeit/Timezone: TLS ist zeitabhängig; abweichende Systemzeiten führen zu Validierungsfehlern.
• File permissions/SELinux: Services cannot read certificates, even though they exist.
• Cataloging of responsibilities: Who is allowed to perform short-term manual signing? Clear roles prevent unauthorized actions.
• Rate‑Limits: Test against a staging proxy so production is not suddenly blocked.

Conclusion

Automating internal PKI‑renewal reduces operational effort and outage risk, but requires a clean architecture, RBAC policies, observability and defined emergency paths. The combination of HashiCorp Vault as signer, an ACME‑proxy as protocol bridge and systemd‑timers as scheduler is proven in practice: it separates responsibilities, simplifies audit and scales if you account for rate limits and staggering. Test in staging, run chaos checks and document runbooks — this keeps your renewal system controllable and resilient.

Operational reliability, recovery and scaling — practical additions

In production operation of an automated PKI‑renewal stack, success is determined not only by configurations but also by operational processes and failure scenarios. Below you will find concrete guidance on backup/recovery, high availability, rollover processes and protection against abuse — all with a focus on admins and IT decision-makers.

Vault‑backup, unseal and auto‑unseal strategies

Back up not only database backups but, above all, the unseal materials: Shamir‑shares, HSM‑keys or Cloud‑KMS configurations. When using Shamir you must define recovery shops and roles (who has access to how many shares). Auto‑unseal via Cloud‑KMS or HSM reduces manual effort on restarts, but creates new dependency risks: restrict KMS access rights tightly and maintain access logs (audit) for unseal events.

HA, replication and regional outage

Vault in HA mode (Integrated Storage or a consistent backend such as Consul) allows read/write replication. Decide whether you need active‑active or active‑passive replication. For the ACME‑proxy, horizontal scaling with sticky sessions or central persistence for rate‑limit counters is recommended so that a failover does not generate duplicate requests. Test cross‑DC failover: take a region offline, validate signing latencies and whether auto‑unseal engages.

Key/CA‑rollover and compatibility

A planned rollover of the intermediate CA is a normal but critical operation. Perform the rollover in stages: prepare a new intermediate, have it signed by the root CA, distribute new CA bundles in parallel and set extended overlap periods so clients accept both chains. Document rollback scenarios: how to revert to the old intermediate if clients are incompatible.

Revocation, CRL and OCSP design

Decide early whether to deploy CRLs, OCSP or both. CRLs are simple to implement but scale poorly; OCSP is online-capable but requires low-latency, highly available responder infrastructure. For internal networks, a lightweight OCSP responder in front of the Vault‑PKI can serve; ensure clients receive correctly configured CRL/OCSP URLs and test offline scenarios when responders are not reachable.

Emergency procedures and least privilege

Define clearly: who is allowed to manually sign/unsign in an emergency? Establish a process with change approval, short-term auditing and temporary RBAC escalations. Automate audit export for rapid forensics. Protect signing endpoints with additional authentication (e.g., mTLS + Vault AppRole) and limit signing rates per role.

Scaling the ACME proxy and monitoring checks

Scale proxy instances behind a load balancer; maintain a central Redis/DB for rate limits and request state. Measure latency and error rates separately: high signing latency indicates backend bottlenecks (Vault CPU/HSM). Alerts should report, in addition to errors, increased renewal density — an early indicator of system or planning failures.

In summary: invest time in recovery procedures, clear rollback plans, rollover tests and monitoring baselines. Only by doing so will your PKI automation remain not only convenient but also operationally reliable, auditable and scalable.

For this topic, systemd timers and certificate management are also important. This article places these aspects in clear context and shows what matters in daily operations.