Operating a private PKI: deploying OpenSSL & CFSSL, CRL/OCSP and HSM integration

In this article I explain how to operate a Private PKI — that is, an internal Public Key Infrastructure (PKI) that issues and manages X.509 certificates. A PKI is central to TLS, code signing, client authentication and VPN access. I cover practical aspects: choosing OpenSSL vs. CFSSL, CA architecture, CRL (Certificate Revocation List) and OCSP (Online Certificate Status Protocol) for revocation checking, and HSM integration via PKCS#11. The target audience is administrators, system engineers and operators: therefore the focus is on operation, risks, verification steps, troubleshooting and fallback strategies.

Why operate a private PKI?

A private PKI gives control over issuance rules, lifetimes and key material. Unlike public CAs it is intended for internal certificates that are not in the browser root store. This enables short lifetimes, automated rollouts and strict policy enforcement. However, risks are greater: root key compromise, incorrect revocation configuration or lack of automation can endanger availability and security.

Fundamental architectural decisions

Before deployment you should clarify architectural questions. Critical are the CA hierarchy, degree of automation, key storage and the revocation strategy.

CA topology: Root, Sub‑CA, Issuing

Best practice is a multi‑tier hierarchy: an offline Root‑CA (Root‑Key, highest trust anchor) and one or more online Sub‑CAs (Issuing CAs) that sign active certificates. This separation reduces risk: a compromised Issuing CA is more controllable; the Root‑Key remains secured offline for longer periods.

Software choice: OpenSSL vs. CFSSL (vs. Vault)

OpenSSL is a flexibly adaptable toolkit; suitable for simple, low‑volume PKIs or admin‑controlled workflows. CFSSL (CloudFlare SSL) is a specialized PKI server with REST API, JSON configuration and built‑in automation. HashiCorp Vault additionally offers a KMS‑like model with dynamic certificates and policies. Choose based on requirements:

• OpenSSL: full control, scripts, but more in‑house development for APIs and automation.
• CFSSL: API‑ready, simpler automation paths, well suited for internal DevOps/CI.
• Vault: strong policy and secret‑management functionality; sensible if Vault is already in use.

Prerequisites and security fundamentals

Before deployment you need clear policies: key sizes (2048/3072/4096 RSA or ECDSA P‑256/P‑384), lifetimes, renewal processes, audit logging and roles (e.g., CA officer, auditor). HSM (Hardware Security Module) enables protection of private keys against extraction; PKCS#11 is the standard API for that. Define backup and recovery plans for key material and CA databases.

Policies: Certificate Policy (CP) and Certification Practice Statement (CPS)

CP/CPS are documented rules that govern issuance, use and revocation. Even when internal, these documents are essential for operational clarity and as a reference during audits.

Step‑by‑step: OpenSSL Root and Sub‑CA (short example)

This example shows the minimum: generate the Root CA offline, Sub‑CA for signing operations. Note: OpenSSL is configured via a config file (purpose: extensions, paths). Failures frequently occur due to incorrect permissions or forgotten serial/index files.

Generate Root CA (offline):

# Root private key (offline, auf HSM empfohlen) und self-signed cert (10 Jahre)
openssl genpkey -algorithm RSA -out root.key.pem -pkeyopt rsa_keygen_bits:4096
openssl req -x509 -new -nodes -key root.key.pem -sha256 -days 3650 -out root.cert.pem -subj "/C=DE/O=ACME/OU=PKI Root/CN=ACME Root CA"

Create Sub‑CA and have it signed by the Root (Sub‑CA online):

openssl genpkey -algorithm RSA -out subca.key.pem -pkeyopt rsa_keygen_bits:4096
openssl req -new -key subca.key.pem -out subca.csr.pem -subj "/C=DE/O=ACME/OU=PKI SubCA/CN=ACME SubCA"
openssl x509 -req -in subca.csr.pem -CA root.cert.pem -CAkey root.key.pem -CAcreateserial -out subca.cert.pem -days 1825 -sha256 -extensions v3_ca -extfile /etc/ssl/openssl.cnf

Typical pitfall: copying root keys to the online system compromises the root. A better approach is a signature workflow: sign the CSR offline and transfer only the Sub‑CA certificate online.

HSM‑Integration: Why and how (PKCS#11)

HSMs protect private keys physically and prevent easy export. PKCS#11 is a platform-independent API that allows software to address HSMs. SoftHSM is a software-based PKCS#11 implementation for testing; Cloud‑HSMs (AWS, Azure, Google) provide managed HSMs with their own integration steps.

SoftHSM as a test run

SoftHSM is useful for testing; it does not replace a production FIPS‑certified HSM. To create a key in SoftHSM:

# Initialisierung (Beispiel mit SoftHSM2)
softhsm2-util --init-token --slot 0 --label "test-token" --pin 1234 --so-pin 5678
# Import eines RSA-Schlüssels (PKCS#12) in SoftHSM
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so -l --pin 1234 --import mykey.p12 --type private

In production environments, configure your CA software (OpenSSL, CFSSL, Vault) so that private keys remain in the HSM via PKCS#11 and signing operations are performed in the HSM. OpenSSL requires, for example, a PKCS#11 engine or the p11tool/engine_pkcs11 configuration.

# Beispiel: OpenSSL mit engine_pkcs11 (vereinfacht)
openssl engine dynamic -pre SO_PATH:/usr/lib/engines/engine_pkcs11.so -pre ID:pkcs11 -pre LIST_ADD:1 -pre LOAD

Common errors: incorrect slot/token IDs, PIN timeouts, ACLs on the HSM. Automate testing of loops such as signing and signature workflows before going live.

Revocation: CRL and OCSP in operation

Revocation is critical: CRL (Certificate Revocation List) is a list of revoked serial numbers; clients download it periodically. OCSP enables online queries of the status of individual certificates. Both have advantages and disadvantages:

• CRL: simple, scalable via CDN/HTTP, but the list can become large and clients must download regular updates.
• OCSP: real-time status, low data transfer per request, but it requires a reliable OCSP responder (high availability) and signed responses (OCSP responder certificate or OCSP stapling for TLS).

For internal PKIs a combination is often sensible: the Issuing CA publishes CRLs periodically (e.g. every 12 hours) and operates an OCSP responder for low latency and real online checks.

Generate CRL with OpenSSL

# CRL erstellen (angenommen index.txt und serial vorhanden)
openssl ca -config openssl.cnf -gencrl -out crl.pem
# CRL in DER für HTTP-Distribution konvertieren
openssl crl -in crl.pem -outform DER -out crl.der

Important: Web servers, CDNs or file servers must serve CRLs with consistent cache headers. Verify that clients correctly follow the CRL Distribution Point (CDP) URL from the certificate.

OCSP‑Responder (example with OpenSSL)

# OCSP responder starten (vereinfacht, für Tests)
openssl ocsp -index index.txt -port 2560 -rsigner ocsp.cert.pem -rkey ocsp.key.pem -CA root.cert.pem -text

For production use specialized OCSP responders (e.g. from CFSSL, EJBCA or commercial appliances) and ensure high availability (load balancers, Anycast). OCSP‑Stapling (TLS‑Registration des OCSP‑Status) reduces client‑side queries.

CFSSL: REST‑API und Automatisierung

CFSSL provides APIs for signing requests and CRL/OCSP management. Typical are JSON policies and straightforward deployment in Docker/Kubernetes. CFSSL is suitable if you want to integrate automated certificate issuance into CI/CD or provisioning pipelines.

{
  "signing":{
    "default":{
      "expiry":"8760h"
    },
    "profiles":{
      "server":{
        "expiry":"720h",
        "usages":["signing","key encipherment","server auth"]
      }
    }
  }
}

CFSSL is simpler to operate for REST‑clients than OpenSSL script jobs; pay attention to API authentication (mTLS, Token) and rate limits, otherwise a misused account can generate large numbers of certificates.

Operations, monitoring and audit

Essential are audit logs (who requested/approved a certificate), monitoring (CA‑service health, OCSP response times, CRL publication status), backups of the CA databases (index.txt, serial), and regular restore tests. Ensure alerting on failures: failed CRL publications, OCSP responder downtime or HSM communication errors.

Audit and log integrity

Log integrity means: logs must be verifiable. Sign audit logs or store them append‑only in an external log service. Without verifiable logs, reconstruction during incident response is problematic.

Troubleshooting common pitfalls

Here are the most common causes and diagnostic sequences:

1. Client incorrect date/time: Check NTP; expired certificates or invalid NotBefore/NotAfter fields lead to TLS errors.
2. CRL/OCSP not reachable: Test the CDP URL in the certificate and the OCSP URL; check HTTP(S) reachability and firewall.
3. HSM PIN/Lifecycle: PIN lockout or block due to failed authentication; check token status and HSM logs.
4. Missing chain: server provides only the end‑entity certificate but not the sub‑CA; check the TLS chain configuration on web servers or provisioners.
5. Format inconsistencies: PEM vs DER; many tools expect explicit formats.

Example verification steps

# Prüfen des Zertifikatspfads und CRL/OCSP URLs
openssl x509 -in service.cert.pem -text -noout | sed -n '/X509v3 CRL/D, /Authority Information Access/ p'

# OCSP-Abfrage eines spezifischen Zertifikats (Testumgebung)
openssl ocsp -issuer subca.cert.pem -cert service.cert.pem -url http://ocsp.example.local:2560 -text -resp_text

Automation and lifecycle management

Automation reduces human errors in renewal/revoke. For internal environments there are two typical approaches: ACME‑compatible internal CA (e.g. cfssl+acme‑bridge or Boulder‑like solutions) or API‑based automation via CFSSL/Vault. ACME is a protocol that allows clients to order and renew certificates automatically; client authentication is important here (HTTP‑01 internal, DNS‑01 or mTLS).

Important aspects of the lifecycle:

• Automatic notification before expiration (e.g. 30/7/1 days).
• Zero‑touch renewal for servers and load balancers via hooks/agents.
• Automatic testing after renewal: verify the chain and OCSP‑stapling.

# Beispiel: Ansible Task (vereinfachte Darstellung) um CRL zu verteilen
- name: Upload CRL to webserver
  copy:
    src: /var/pki/crl/crl.der
    dest: /srv/www/ssl/crl/crl.der
    owner: root
    mode: '0644'
  notify: RESTart nginx

- name: RESTart nginx
  service:
    name: nginx
    state: RESTarted

Client‑Trust‑Verteilung und Bereitstellung

It’s important that all relevant clients and systems trust the internal root/intermediates. Typical distribution channels:

• Windows: GPO distributes the root‑cert into Trusted Root Certification Authorities.
• Linux/Servers: central CA‑bundle in /etc/pki/ca‑trust/source/anchors and run update‑ca‑trust.
• Mobile/Endpoints: MDM (Mobile Device Management) or manual installation for managed devices.

Test the rollout in stages and verify whether clients establish TLS connections correctly after distribution and perform OCSP/CRL checks.

Skalierung, Performance und Hochverfügbarkeit

Scaling concerns primarily OCSP and CRL distribution. OCSP‑responders must provide low latency; common measures:

• OCSP caching (responder and load balancer) and Anycast for geographic distribution.
• CDN for CRL delivery with appropriate Cache‑Control headers and TTLs.
• Monitoring of response times and error rates; automatic failover of the responder.

# Beispiel: Prometheus Alert (vereinfachtes Beispiel)
- alert: OCSPResponderDown
  expr: probe_success{job="ocsp_probe"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "OCSP Responder nicht erreichbar"
    description: "OCSP Responder {{ $labels.instance }} antwortet nicht."

Disaster Recovery und Incident‑Runbook

Prepare a clear runbook for incidents. Important steps in case of root/CA compromise or HSM failure:

1. Isolate affected systems and preserve evidence (logs). Document timestamps.
2. Revoke compromised certificates and publish an immediate CRL update and OCSP status „revoked“.
3. If the root is compromised: plan for cross‑signing or rebuilding the PKI with a parallel rollout of new root/sub‑CAs; inform affected teams.
4. Test recovery from backup, including HSM import or replacement‑HSM procedures.

Regular DR drills (at least annually) are mandatory: test recovery steps in writing and perform an end‑to‑end validation.

Praktische Prüf‑Commands & Beispiele

Some useful checks that you should regularly integrate into your runbooks:

# Zertifikatspfadprüfung
openssl verify -CAfile chain.pem service.cert.pem

# CRL herunterladen und prüfen
curl -sS -o crl.der http://crl.example.local/crl.der
openssl crl -in crl.der -inform DER -text -noout

# OCSP Testabfrage einer Produktions-URL
openssl ocsp -issuer subca.cert.pem -cert service.cert.pem -url http://ocsp.example.local:2560 -header "HOST" "ocsp.example.local" -resp_text

# HSM Slot/Token-Status (pkcs11-tool)
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so -L
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so -T

Conclusion

Running a private PKI requires more than issuing certificates: clear policies, secure key storage (ideally HSM), automation for issuing/renewing/revoking, monitoring and tested recovery procedures. OpenSSL offers maximum control; CFSSL facilitates API-based automation. CRL and OCSP complement each other in robustness and response time. Test HSM workflows, verify CRL/OCSP reachability automatically, and maintain a written runbook for rotation and incident response. A well-documented PKI minimizes operational risk and keeps your internal TLS dependencies running reliably.

Further resources and internal links

For deeper integrations, review tool documentation: CFSSL, OpenSSL Engine PKCS#11, SoftHSM and the HSM vendor specifications. Also plan an audit of PKI policies and regular RESTore drills, similar to backup-recovery tests in other critical systems.

OpenSSL PKI and HSM integration are also important for this topic. The article contextualizes these aspects and highlights what matters in day-to-day operations.