TLS handshakes failing: reliably resolve certificate chains, SNI, and automatic renewal with ACME

TLS is often ‚invisible‘ in operation – until the moment when TLS handshakes fail and suddenly business software, portals, APIs or integrations can no longer establish connections. The symptoms often look similar („Handshake failure“, „unknown ca“, „certificate verify failed“), but the causes are not: an incomplete certificate chain, an SNI problem (Server Name Indication, i.e., selecting the appropriate certificate based on the hostname), an expired or incorrectly renewed certificate via ACME (Automated Certificate Management Environment, e.g. Let’s Encrypt), or a proxy that replaces a certificate ‚in transit‘.

This article is structured as a runbook for administrators, system engineers, operators and technical IT service providers: with a clear sequence of checks, concrete checks, typical pitfalls, and implementation and rollback strategy. The goal is not only „back to green“, but a stable, verifiable state — including automation and monitoring so the problem does not recur in four weeks.

What actually happens during the TLS handshake (and where it typically breaks)

A TLS handshake is the negotiation of encryption and identity between client and server. Simplified: the client connects and, as is common for HTTPS, indicates the desired hostname via SNI; the server provides a certificate (and ideally the intermediate certificates); they negotiate protocol version and cipher suite; the client validates the Chain of Trust up to a root CA in the trust store.

Failure points in practice:

• Incomplete certificate chain: Server delivers only the leaf certificate; the intermediate is missing. Some clients cannot „fetch“ it (depending on platform/policy/offline).
• SNI does not take effect: Server delivers a default certificate that does not match the host (CN/SAN mismatch). Common with multiple vHosts on one IP or with load balancers.
• ACME renewal/deployment faulty: The certificate was renewed, but the service still uses the old file / the old binding / the old keystore.
• Protocol/cipher policy incompatible: Legacy clients do not support TLS 1.3, or the server blocks TLS 1.2; ALPN (Application-Layer Protocol Negotiation) negotiates HTTP/2 incorrectly.
• mTLS (Mutual TLS) / client certificates: Server expects a client certificate; the client supplies none or the wrong CA.
• Time/CRL/OCSP: Incorrect system time, OCSP/CRL unreachable, or „Must-Staple“ configurations.

Initial assessment: determine the symptom class before you „tinker with the certificate“

Before you swap certificates: classify the error. That saves time and prevents side effects, for example when a proxy causes the problem and you change the backend.

Check 1: Does it affect all clients or only certain ones?

• All clients affected: more likely an expired certificate, wrong certificate deployed, SNI/default certificate, incorrect endpoint (DNS/load balancer).
• Only certain platforms: often chain issues (intermediate), trust store, old Java or Windows policies, TLS 1.3/1.2 interoperability.
• Only internal clients: proxy/inspection (TLS interception), internal CA distribution, split DNS.

Check 2: Where does TLS actually terminate? (Find the termination point)

In modern setups, TLS often does not terminate at the actual application server but at the reverse proxy (Nginx/Apache), at the load balancer, at a WAF or at an API gateway. „TLS termination“ means: the TLS connection is decrypted there, and behind it HTTP runs or TLS is established again (Re-Encryption).

Consequence for troubleshooting: you must test the termination point, not „the server“. If a load balancer terminates, the certificate on the backend is irrelevant for the external client.

Verification steps with OpenSSL and systematic evidence preservation

The quickest reliable information usually comes from openssl s_client. Important: always test with the expected hostname (SNI), otherwise you may see the default certificate.

Check SNI and certificate chain in a single step

# set SNI explicitly, show certificate chain, request OCSP information
openssl s_client 
  -connect example.com:443 
  -servername example.com 
  -showcerts 
  -status 
  </dev/null

What to look for in the output:

• subject / SAN: does the hostname match the Subject Alternative Names (SAN)? CN alone is no longer sufficient for modern clients.
• issuer: who signed it? Expected CA/Intermediate?
• Verify return code: ‚0 (ok)‘ is good; codes like ‚unable to get local issuer certificate‘ indicate chain issues.
• Certificate chain: are Leaf + Intermediate(s) included? The root typically does not need to be sent.
• OCSP response: if stapling is used, a response should be present; on errors you often see timeouts/’no response‘.

Test ALPN and protocol version (TLS 1.2 vs TLS 1.3)

# force TLS 1.2
openssl s_client -connect example.com:443 -servername example.com -tls1_2 </dev/null

# force TLS 1.3
openssl s_client -connect example.com:443 -servername example.com -tls1_3 </dev/null

# simulate ALPN offer (e.g. HTTP/2 and HTTP/1.1)
openssl s_client -connect example.com:443 -servername example.com -alpn "h2,http/1.1" </dev/null

Why this helps: Some errors only appear on specific negotiation paths, e.g. when a proxy announces HTTP/2 (h2) but maps it incorrectly internally, or when older appliances do not properly support TLS 1.3.

Common Cause 1: Incomplete certificate chain (intermediate missing)

A certificate chain consists of the leaf certificate (for your host), one or more intermediate certificates and a root CA that resides in the client’s trust store. If the server does not send the intermediate certificates, some clients cannot complete the chain — particularly in RESTrictive environments without access to AIA URLs or with proxy blocking.

Typical symptoms

• Browser A works, Browser B or a Java-/Windows-client fails.
• Error messages: “unable to get local issuer certificate”, “unknown ca”, “self signed certificate in certificate chain” (misleading, often a chain problem).
• Only certain middleboxes/MTLS clients are affected.

How to check the chain properly

Besides s_client, a second view helps: extract the certificates and check whether the chain validates up to a known root.

# Zertifikate aus der s_client-Ausgabe in Dateien schreiben (manuell oder via Skript)
# Danach: Prüfen, ob Leaf gegen eine angegebene Chain verifizierbar ist
openssl verify -CAfile root-and-intermediate.pem leaf.pem

In practice the solution is usually trivial but critical: the TLS endpoint must be configured with a fullchain (leaf + intermediate), not just the leaf certificate. For ACME clients the file is often named fullchain.pem (not cert.pem).

Operational pitfalls

• Wrong file bound: Nginx/Apache points to cert.pem instead of fullchain.pem.
• Load balancer ‚drops‘ the intermediate: depending on the product the chain must be imported separately or is not correctly preserved on upload.
• Keystore formats: Java (JKS/PKCS12) and Windows (Zertifikatsspeicher) often expect import including the chain; otherwise the leaf is „孤“.

Common cause 2: SNI misconfiguration and default certificates

SNI (Server Name Indication) is a TLS extension that allows the client to indicate the hostname during connection setup. The server can then select the appropriate certificate when multiple domains share the same IP/port pair.

Typical symptoms

• Access by IP does not work or returns the wrong certificate (expected), but access by DNS also returns the “wrong” certificate.
• One host on the same IP works, another does not.
• Only older clients fail (without SNI support), e.g. very old embedded systems.

Check whether the correct certificate is actually being served

# Without SNI: server will deliver the default certificate
openssl s_client -connect example.com:443 -showcerts </dev/null

# With SNI: the correct vhost certificate is expected
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

If a different certificate appears when connecting “without SNI”, that is normal. If the wrong certificate still appears “with SNI”, the cause is at the termination point: vHost mapping, listener assignment, incorrect binding or an upstream device that has already terminated the TLS handshake.

Risk: legacy clients without SNI

If you still have clients in the field without SNI, you need a strategy: a dedicated IP/port for that domain, or a separate legacy endpoint. In B2B environments this occurs, for example, with old appliances, print/scan gateways or embedded gateways in production.

Common cause 3: automatic renewal with ACME – certificate is renewed but not active

ACME automates issuance and renewal. The actual operational risk is often not the renewal itself but the deployment: the certificate on disk is renewed, but the service still uses the old certificate because a reload/RESTart is missing or the wrong path is bound.

Understanding ACME validation: HTTP-01, DNS-01, TLS-ALPN-01

• HTTP-01: ACME server fetches a token file via HTTP (port 80). Fails with redirect policies, WAF rules, or missing inbound allowance.
• DNS-01: Token as DNS TXT record. Good for wildcard certificates and when port 80 is not possible, but dependent on DNS automation/propagation.
• TLS-ALPN-01: Validation over port 443 and ALPN. Practical when port 80 is closed, but can collide with certain proxies/terminators.

Check whether the new certificate is actually active

Compare the NotAfter date (expiration) and, ideally, the fingerprint of the live certificate with the expected artifact.

# Live certificate fetch and show expiration date
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null 
  | openssl x509 -noout -dates -subject -issuer -fingerprint -sha256

If the live certificate is old even though the ACME files are new, it is a deployment/reload problem.

Typical pitfalls with ACME in daily operation

• Missing reload: the service reads certificates only on start. Solution: scheduled reload after successful renewal.
• Multiple instances: certificate is renewed on Node A, but traffic goes via Node B. Solution: central storage, configuration management, or distribute via hooks.
• Container/immutable deployments: the certificate resides on the host, the container cannot see it (missing volume) or an ingress controller manages it separately.
• File permissions/SELinux/AppArmor: renewal writes a new file, the service can no longer read it afterwards.
• Clock skew: time deviations lead to „not yet valid“ or fragile OCSP checks.

Robust ACME automation: hooks, reload and idempotence

Regardless of the ACME client (certbot, acme.sh, win-acme etc.), a pattern has proven effective: after a successful renewal a deploy-hook that (1) consistently deploys files/bundles, (2) sets permissions, (3) performs a controlled reload and (4) starts a smoke test.

# Example: certbot with deploy-hook (Linux)
# The hook only runs if it was actually renewed.
certbot renew 
  --deploy-hook "/usr/local/sbin/tls-deploy-and-reload.sh"

The hook itself should be „idempotent“ (runnable multiple times without side effects) and return clear exit codes so monitoring/jobs can reliably alert.

mTLS and client certificates: when the server rejects the client

With mTLS (mutual TLS) not only does the server authenticate to the client, but the client also authenticates to the server with a client certificate. This is common in internal integrations, B2B partner connections, or admin endpoints.

Typical symptoms

• Handshake aborts with ‚handshake failure‘ or ‚bad certificate‘.
• Server logs show: ’no required SSL certificate was sent‘ or ‚unknown ca‘ (referring to the client CA).

Diagnostic path

• Does the endpoint actually require mTLS (policy/location/listener)?
• Is the CA that issues client certificates configured as trusted on the server?
• Is EKU/Key Usage (Extended Key Usage: ‚Client Authentication‘) in the client certificate correct?
• Is the SNI/host correctly mapped to the mTLS policy (not accidentally to a ‚public‘ listener)?

Protocol and cipher policy: when security hardening unexpectedly locks out clients

Many TLS failures occur after hardening measures: disabling TLS 1.0/1.1 (usually correct), RESTrictive cipher lists, forcing TLS 1.3, or strict curve selection. This is not an argument against hardening — but for a controlled migration with measurement points.

Practical rule: measure first, then change

• Which clients are connected (browsers, Java, .NET, appliances, partners)?
• Which protocols are actually used (TLS 1.2/1.3)?
• Are there compliance requirements that already mandate TLS 1.2+?

In troubleshooting situations, a temporary relaxation can help narrow down the cause. Important: only within a change window, documented, and with a clear rollback.

Diagnostic checklist: to the most likely cause in 20 minutes

If you are under time pressure, use this sequence. It minimizes „guesswork“ and quickly provides actionable evidence.

1. Determine the termination point: DNS → Load Balancer → WAF → Reverse Proxy → App. Test there.
2. Pull the live certificate (with SNI): SAN, Issuer, NotAfter, Fingerprint.
3. Check the certificate chain: is the fullchain delivered? Check the verify code.
4. SNI A/B test: compare with and without -servername.
5. Check TLS versions: force tls1_2 / tls1_3, test ALPN.
6. Clarify mTLS: does the server expect a client certificate?
7. Check ACME status: renewal logs, last successful renewal, did the hook/reload run?

Implementation: proper operational measures to prevent recurrence

TLS issues are rarely „one-off“. Without operational measures they will recur: on the next intermediate change, the next certificate rollover, the next load balancer update, or when a node in the cluster is rebuilt.

1) Monitoring for expiration and for an actual handshake

Monitoring only the expiry date is insufficient. You want at least two signals:

• Certificate expiry: NotAfter within a threshold (e.g. 14/7/3 days).
• Real handshake from outside: SNI correct, chain valid, TLS version OK. That covers SNI/chain/deployment issues.

2) Change and rollback plan for certificates

A pragmatic rollback means: you can revert to the previous certificate/binding within a few minutes. For that you need:

• Versioned storage of certificate artifacts (at least the previous version), with access control.
• Documented paths/bindings per service (e.g. Nginx configuration, IIS binding, load balancer listener).
• A defined reload/RESTart process and a smoke test (handshake + HTTP status + if applicable an API call).

3) Run ACME cleanly in multi-node setups

In HA environments distribution decides. Typical robust patterns:

• Central certificate management at the termination point: ACME runs directly on the load balancer/reverse proxy, not “somewhere” in the backend.
• Pull instead of push: Nodes fetch the certificate periodically from a secured store (secrets management) and reload in a controlled manner.
• Hooks with a health gate: Only commit a deployment when the new handshake on the target listener is successful.

Fallback strategy during an incident: stabilize, then improve

When production interfaces fail, order matters:

• Stabilize: serve a correct certificate with a valid chain; if necessary, temporarily revert to a known, functioning policy (allow TLS 1.2, clean cipher baseline).
• Verify: test from multiple networks/clients (internal/external), document SNI tests, record fingerprints.
• Fix the root cause: ACME deployment, incorrect fullchain, incorrect SNI mapping, cluster distribution.
• Harden again: tighten the security policy again, but with measurement points and rollback.

It is important not to treat the “fix” as only addressing the symptom. If, for example, renewal succeeds on node A but node B continues to serve the old certificate, this is not a certificate issue but a distribution and operations problem.

Conclusion: TLS handshakes fail — but the causes are manageable with a clear runbook

When TLS handshakes fail, the greatest risk is not the cryptography but operational complexity: multiple termination points, SNI mapping, certificate chains, automation and reload mechanics. With a fixed verification order (termination point → live certificate with SNI → chain → protocols/ALPN → ACME deployment) you reach the root cause quickly. Sustainable results require operational measures: handshake monitoring, versioned certificate artifacts, controlled hooks and a practiced rollback path.

For this topic, SNI misconfiguration and ACME automatic renewal are also important. The article places these aspects clearly and shows what matters in day-to-day operations.