Automated TLS certificate management in Kubernetes with cert-manager and ACME workflows: operationally reliable from initial installation to rollback

TLS in everyday Kubernetes operations is not a “install a certificate once” topic, but an ongoing operational process: certificates expire, domains change, Ingress controllers are migrated, DNS is delegated, and at the worst moment an ACME challenge fails. This is exactly where Automated TLS certificate management in Kubernetes pays off: you reduce manual interventions, avoid expiry surprises and gain reproducible procedures for issuance, renewal and rotation.

This article explains in practical terms how cert-manager (a Kubernetes controller for certificate lifecycle) interacts with ACME (Automated Certificate Management Environment, the protocol behind Let’s Encrypt & Co.). The focus is on operations and administration: prerequisites, architecture, security decisions, common pitfalls, validation steps, troubleshooting and a fallback strategy that actually helps during an incident.

Automated TLS certificate management in Kubernetes in practice

In traditional setups TLS is often handled by a single reverse proxy or load balancer. In Kubernetes, however, several moving parts come together: Services, Ingress resources, Ingress controllers (e.g. NGINX, HAProxy, Traefik), possibly a cloud load balancer, plus DNS. cert-manager runs in the cluster and creates/renews certificates as Kubernetes objects, typically as Secrets (Kubernetes objects for storing sensitive data such as private keys).

That changes the questions: Who is allowed to read Secrets? How is rotation achieved without downtime? What happens during an Ingress migration? And how do you ensure ACME challenges work in multi-cluster or strictly segmented environments?

Architecture overview: cert-manager, Issuer/ClusterIssuer, Certificate and Challenges

cert-manager extends Kubernetes with Custom Resources (CRDs). For operations, four resource classes are central:

• Issuer / ClusterIssuer: Defines the issuing instance. Issuer is namespace-scoped, ClusterIssuer is usable cluster-wide.
• Certificate: Describes which certificate you want (DNS names, validity/rotation, target Secret).
• Order: Internal ACME order process; managed by cert-manager.
• Challenge: The proof to the CA that you control the domain (HTTP-01 or DNS-01 are the usual variants).

ACME itself is “only” the standardized process: client (cert-manager) requests a certificate, CA requires domain validation, client fulfills the Challenge, CA signs the certificate, cert-manager writes it into the Secret, Ingress controller uses the Secret for TLS termination.

Prerequisites and upfront decisions (that save effort later)

1) Challenge type: HTTP-01 vs. DNS-01

HTTP-01 validates via an HTTP endpoint under /.well-known/acme-challenge/…. Advantage: no DNS API required, straightforward and fast to understand. Disadvantage: you need externally reachable HTTP routing to the Ingress IP and must not have any hard redirect/auth rules that block the path.

DNS-01 validates via a TXT record in DNS. Advantage: works without public HTTP reachability (e.g. TLS-only setups, internal ingress, wildcard certificates). Disadvantage: you need DNS automation (API access) and must manage TTL/propagation and permissions carefully.

Practical rule: For wildcard certificates DNS-01 is effectively mandatory. For simple public services HTTP-01 can be sufficient, as long as ingress and DNS are stable.

2) Where does TLS terminate? Ingress, service mesh or external load balancer

Many teams terminate TLS at the ingress controller. Alternatives are a cloud load balancer (TLS terminated before the cluster) or a service mesh (mTLS internally, TLS externally). It is important that cert-manager is integrated where the private key is required. If TLS terminates at the cloud load balancer, cert-manager only helps directly if you synchronize certificates to that load balancer (this depends on cloud/provider and controller and must be solved separately). For the common Kubernetes pattern of “TLS at the Ingress”, cert-manager is particularly suitable.

3) Security and compliance points (secrets, keys, access)

cert-manager generates private keys and stores them in Kubernetes Secrets. That is convenient but relevant for security: anyone with read permissions on Secrets can exfiltrate keys. Therefore review RBAC (Role Based Access Control) and your namespace strategy. In regulated environments it is also relevant whether keys should be generated and protected in a KMS/HSM; cert-manager can, depending on the environment, work with external Issuers/integrations, but that is a separate design topic.

Installation of cert-manager: a clean operational starting point

cert-manager is typically installed via Helm. For operations teams the priority is less “how fast” and more “how reproducible”: versioned, with clear namespaces, and with monitoring in mind.

Helm installation (example)

helm repo add jetstack https://charts.jetstack.io
helm repo update

kubectl create namespace cert-manager

helm install cert-manager jetstack/cert-manager 
  --namespace cert-manager 
  --version v1.16.0 
  --set crds.enabled=true

Practical notes:

• CRDs (Custom Resource Definitions) are cluster-wide definitions. In many change processes CRD changes must be explicitly approved.
• Pin the version and plan upgrades like other cluster components: staging, then production, with a rollback option.
• Check PodSecurity/Admission policies: cert-manager requires specific permissions and runs as a controller.

Basic checks after installation

kubectl -n cert-manager get pods
kubectl -n cert-manager get deploy
kubectl get crd | grep cert-manager

If pods crash: read Events first, then logs. In Kubernetes Events are often the quickest indicator of RBAC, webhook issues, or imagePull problems.

kubectl -n cert-manager get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n cert-manager logs deploy/cert-manager --tail=200
kubectl -n cert-manager logs deploy/cert-manager-webhook --tail=200

ACME setup: ClusterIssuer for Let’s Encrypt (staging and production)

For reliable operation, enable staging first. Let’s Encrypt has rate limits; staging reduces risk during testing and repeated failures. Switching to production then becomes a controlled step.

ClusterIssuer (Staging) with HTTP-01 via Ingress

This example uses HTTP-01 and assumes your Ingress controller is reachable via an IngressClass (e.g. „nginx“). The IngressClass maps which controller processes an Ingress resource.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: admin@example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx

ClusterIssuer (Production) – identical, but with a different ACME endpoint

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: admin@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx

Important: The Account-Key (privateKeySecretRef) is not the certificate, but the key for your ACME account with the CA. You should treat it like an important credential and not delete it lightly.

Issuing a certificate: Certificate resource and Ingress integration

There are two common patterns: (1) Ingress annotations that cause cert-manager to create a Certificate itself, or (2) you define a Certificate explicitly. For operational transparency, explicit is often better: you clearly see which DNS names, which issuer reference, and which secret are affected.

Certificate (example)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: portal-tls
  namespace: production
spec:
  secretName: portal-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - portal.example.com

Ingress that uses the secret

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: portal
  namespace: production
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - portal.example.com
    secretName: portal-tls-secret
  rules:
  - host: portal.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: portal-svc
            port:
              number: 80

Why does this work? cert-manager writes tls.crt and tls.key into the Secret. The Ingress controller loads the Secret and terminates TLS. On renewal the Secret is updated, and the controller reloads (depending on the controller with a slight delay). For stable rotation it is crucial that the controller reliably detects Secret updates.

Check whether everything is truly „green“ (and not just „somehow created“)

Status on Certificate/Order/Challenge

kubectl -n production get certificate portal-tls -o wide
kubectl -n production describe certificate portal-tls

kubectl -n production get order
kubectl -n production get challenge
kubectl -n production describe challenge -l cert-manager.io/certificate-name=portal-tls

Practical tip: If certificates are not issued, describe is often more informative than logs. cert-manager writes precise Condition messages, e.g. „Waiting for HTTP-01 challenge propagation“ or „DNS record not found“.

Validate secret content and expiration date

You don’t only want to see that a Secret exists, but that it contains a valid certificate and the hostname matches.

kubectl -n production get secret portal-tls-secret
kubectl -n production get secret portal-tls-secret -o jsonpath='{.data.tls.crt}' | base64 -d > /tmp/portal.crt

openssl x509 -in /tmp/portal.crt -noout -subject -issuer -dates -ext subjectAltName

If you are not allowed to run OpenSSL on admin workstations in production networks, build this check into controlled admin tooling (jump host) or into an internal runbook job image.

Typical pitfalls in practice (and how to detect them early)

HTTP-01 fails due to redirects, authentication, or incorrect IngressClass

Common cause: a global redirect from HTTP to HTTPS or an authentication layer that also applies to the ACME path. The ACME server must be able to reach the token via HTTP without login, without WAF blocking, and without „HTTPS only.“ Some setups solve this by explicitly excluding the challenge path or by configuring the solver to create a separate Ingress.

A second classic: your Ingress resource is not processed by the expected controller (wrong ingressClassName), or you have multiple controllers in the cluster. In that case the challenge Ingress ends up in the wrong place.

DNS-01 fails due to missing API permissions or propagation

With DNS-01 the most frequent cause is not cert-manager itself but the DNS automation: the DNS provider account used is not permitted to create TXT records, or it can only write in one zone while the domain resides in another zone. Add TTL/propagation issues: the ACME server queries distributed resolvers; if your TXT record is not visible everywhere, the challenge stays pending or times out.

Operationally, a standardized „propagation check“ against public resolvers is useful here, rather than checking only your own DNS.

# Beispiel: TXT-Record für _acme-challenge prüfen
# (Record-Name und Token entnehmen Sie aus der Challenge-Resource)

nslookup -type=TXT _acme-challenge.portal.example.com 1.1.1.1
nslookup -type=TXT _acme-challenge.portal.example.com 8.8.8.8

Rate Limits und „Testen in Produktion“

If you retry requests multiple times after failures, you can hit rate limits with public CAs. That will not present as a „technical error“ but as „too many requests.“ Therefore: use a staging issuer for tests, and only use production once Ingress/DNS are stable. This is especially important in migration projects (e.g., changing the Ingress controller).

Zeit, DNS und Netzwerkpfade: unscheinbare Ursachen

ACME is time-sensitive. If cluster nodes or key components have significantly drifting clocks (NTP/Chrony issues), TLS validations and time windows can fail. Equally important is egress firewalling. cert-manager must be able to reach the ACME endpoints, and for DNS-01 the DNS API must be reachable. That sounds trivial, but in segmented networks it is a common showstopper.

Best Practices für stabilen Betrieb: Monitoring, Alerting, Rotation

Überwachung: nicht erst reagieren, wenn das Zertifikat abgelaufen ist

Set up at least three levels of checks:

• Kubernetes-Status: Certificate-Conditions (Ready/Not Ready) and events.
• Ablaufzeit: metrics/checks for „days until expiry“ (e.g., via Prometheus exporter or external TLS checks).
• End-to-End: check from the outside what is actually served (SNI, chain, expiry date).

cert-manager provides Prometheus metrics if you run monitoring. Crucially, alerts should represent not only „has expired“ but „renewal failed.“ That gives you buffer time for root-cause analysis.

Rotation und Downtime vermeiden: worauf Ingress-Controller reagieren

On renewal, the Secret is updated. Ingress controllers differ in how quickly they pick up Secret updates. In robust setups you should test this deliberately: trigger a renewal (e.g., in staging), observe whether new certificates become active without reload issues, and whether existing connections remain stable.

If you have very strict requirements (e.g., many long-lived TLS connections), it is also worth looking at session tickets/resumption and the controller’s reload behavior. That is not a cert-manager topic, but it directly affects the perception that „certificate changes cause disruption.“

Namespace- und Issuer-Strategie

A ClusterIssuer is convenient, but it expands the scope: every namespace can potentially request certificates if RBAC is not properly RESTricted. For managed service providers or multi-tenant clusters, it is often worth using issuers per tenant/namespace and deliberately controlling access to Issuer/Certificate CRDs.

Troubleshooting-Runbook: strukturierte Vorgehensweise bei Fehlschlägen

When „certificate is not issued“ appears in tickets, a clear procedure helps. This sequence has proven effective:

1. Clarify scope: Does it affect a single Certificate, a namespace, or the entire cluster?
2. Read status: Certificate/Order/Challenge conditions and events.
3. Check network paths: Can cert-manager reach the ACME servers? Can ACME reach your challenge endpoint (HTTP-01), or is the TXT publicly visible (DNS-01)?
4. Check Ingress/DNS responsibility: IngressClass, hostname, LoadBalancer IP, DNS A record, if applicable CDN/WAF rules.
5. Targeted logs: cert-manager controller and, if applicable, ingress controller — not „everything at once“.

Commands for quick diagnosis

# 1) cert-manager Health
kubectl -n cert-manager get pods
kubectl -n cert-manager get events --sort-by=.lastTimestamp | tail -n 30

# 2) Certificate Status
kubectl -n production describe certificate portal-tls

# 3) Challenge Details
kubectl -n production get challenge -o wide
kubectl -n production describe challenge <challenge-name>

# 4) Ingress Check
kubectl -n production get ingress portal -o yaml
kubectl -n production describe ingress portal

# 5) DNS Check
nslookup portal.example.com 1.1.1.1

# 6) External TLS from a client's perspective
echo | openssl s_client -connect portal.example.com:443 -servername portal.example.com 2>/dev/null | openssl x509 -noout -subject -issuer -dates

If openssl s_client shows a different certificate than the one in the Secret, this is usually caused by a preceding layer (CDN, cloud LB), an incorrect host/SNI, or an ingress controller that did not pick up the Secret.

Fallback strategy: What to do if automation fails or rotation goes wrong?

A fallback strategy must be fast, low-risk and auditable. In practice, three scenarios are relevant:

Scenario A: Renewal fails, but the current certificate is still valid

• Priority: fix the root cause (DNS/ingress/reachability) without hasty reinstallation.
• Monitoring: set alert thresholds to give you days of lead time.
• Action: analyze the challenge/order, adjust solver config if necessary, then request renewal.

Scenario B: Certificate has expired or is about to expire, automation is blocked

You need a „Break Glass“ path here. Two common options are:

• Temporary manual certificate (e.g., from an internal CA or provided on short notice) inserted as a Secret to ensure availability.
• Switch to alternative validation (from HTTP-01 to DNS-01), if that can be implemented quickly and a DNS API is available.

Important: document the temporary state and set a reminder to return to the normal ACME workflow.

Scenario C: Incorrect certificate rolled out (hostname/chain does not match)

This is rare but critical: incorrect DNS names, wrong SecretName in ingress, or a shared secret being overwritten. The fallback here is to revert to the last known good Secret. Kubernetes does not version Secrets historically. Therefore backups or GitOps/cluster-backup concepts are important: either you have the Secret (encrypted) in a backup, or you can issue a correct certificate on short notice.

Operationally useful: keep a short checklist per critical endpoint: „Which Secret? Which hosts? Which IngressClass? Which external IP? Which DNS response?“ — so you don’t have to reconstruct the system model during an incident.

Hardening and clean administration: RBAC, secret access, change control

An underrated point is who is allowed to request certificates. In Kubernetes, a team can, by creating a Certificate object in combination with a permissive ClusterIssuer, suddenly request certificates for additional domains if DNS/ingress routing allows it. Limit this with:

• RBAC on cert-manager CRDs (who is allowed to create Certificate/Issuer?).
• Namespace isolation: separate tenants and minimize privileges.
• DNS governance: manage domain delegation and tightly control DNS API access.
• Change control: treat changes to Issuer/Solver configurations like infrastructure changes.

If you already have established processes for secrets and automation, these patterns combine well with Infrastructure-as-Code and GitOps: reproducible manifests, reviews and clear rollback paths. In related areas (e.g., secret handling and idempotence) similar principles apply.

Conclusion: cert-manager and ACME are an operational process, not a „plugin“

cert-manager in combination with ACME is a highly practical way to automate TLS certificate management in Kubernetes: issuance, renewal and rotation become a controlled lifecycle. Crucial is that you take the dependencies seriously: ingress class and routing (HTTP-01), DNS API and propagation (DNS-01), RBAC/secret protection, and monitoring with sufficient lead time.

If you use staging consistently, define runbooks for troubleshooting and fallback, and tightly control access to Issuers/Secrets, automation will not become a black box but a reliable building block in Kubernetes operations — even for teams that do not work with PKI topics daily.

Acme Workflow and Let’s Encrypt In Kubernetes are also important for this topic. The post situates these aspects clearly and shows what matters in everyday operations.