Secure Boot for Custom Kernel Modules: Signing, MOK Workflow, and Distribution Strategy

Secure Boot for custom kernel modules is a core security and operational concern for operators of modern Linux and Kubernetes infrastructures. This guide provides practical instructions: which components play a role, how to sign modules securely, how the MOK workflow (Machine Owner Key) is operationalized, which distribution strategies make sense for fleets, how to integrate DKMS, and which specific precautions Kubernetes clusters require. The goal is reproducible operation, clear verification steps and a reliable rollback strategy.

Why are custom modules blocked by Secure Boot?

UEFI Secure Boot validates the integrity of bootloaders and kernels through a chain of certificates. shim is a boot stub signed by the distribution team that is allowed to load the kernel and at the same time provides mechanisms (e.g. MOK) for operators to enroll their own certificates. Once the kernel has enabled lockdown mode, it verifies subsequently loaded kernel modules (.ko) for valid signatures. If a trusted signature or the appropriate certificate is missing, loading is refused and drivers, storage plugins or network functions can fail.

Terminology: PK, KEK, db and MOK

PK stands for Platform Key (firmware master key), KEK are Key Exchange Keys (for signature management) and db is the firmware database of trusted certificates. MOK (Machine Owner Key) is a mechanism in shim that allows operators to enroll their own certificates without changing the firmware PK/KEK/db. In the kernel accepted certificates are managed in keyrings — if they are missing there, verification fails.

Preparatory system checks

Before you adjust processes or pipelines, determine the status per system. Check Secure Boot, lockdown and tool status.

# Basischecks
sudo mokutil --sb-state 2>/dev/null || echo "mokutil fehlt oder Secure Boot nicht aktiv"
sudo mokutil --list-enrolled 2>/dev/null || echo "Keine enrolled MOKs oder mokutil nicht vorhanden"
cat /sys/kernel/security/lockdown 2>/dev/null || echo "Lockdown-Status nicht verfügbar"
dmesg | egrep -i "module verification failed|Required key not available|lockdown" | tail -n 50

If dmesg shows messages like „module verification failed“, that is a clear indicator of signature or key problems.

Signing: concept, key protection and procedures

A kernel module is signed with a PKCS#7 signature that the kernel verifies when loading. Formally you need two artifacts: a private signing key (for creating the signature) and the accompanying certificate (public), which is registered as trusted on target systems. The most important operational principle: the private key must not be exposed to many servers.

Key generation (secure example)

mkdir -p /root/module-signing && cd /root/module-signing
# Privaten RSA-Schlüssel + Self-Signed-Zertifikat erzeugen
openssl req -new -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes 
  -subj "/CN=Kernel Module Signing (MOK)/" 
  -keyout MOK.priv -out MOK.pem
# Konvertieren ins DER-Format für mokutil/import
openssl x509 -in MOK.pem -outform DER -out MOK.der
chmod 600 MOK.priv
chmod 644 MOK.pem MOK.der

Note: the -nodes flag removes the passphrase; this allows automated signing but increases the protection requirements for the key’s storage location (HSM, secure signing server or CI/CD safe).

Operationalizing the MOK workflow

The classic flow: You import the public certificate (DER format) with mokutil –import. That creates a pending enrollment request that must be confirmed in the graphical/firmware-based shim interface on the next boot. This is where rollouts commonly fail: confirming requires console/display or serial access.

# Zertifikat importieren (erzeugt Enrollment-Request)
sudo mokutil --import /root/module-signing/MOK.der
# Nach Reboot: enrolled keys auflisten
sudo mokutil --list-enrolled

Operations options for large fleets:

• Pre-baked Images: Image bereits mit MOKs versehen (für VM- und Cloud-Umgebungen, wenn die Cloud-Provider das erlaubt).
• Remote-Konsole/Serial-Automation: Nutzen Sie iKVM oder Consoleredirection-APIs für automatisierte Confirmation.
• Firmware-Management: Bei physischen Hosts kann der OEM/Hosting-Provider Schlüssel zentral in der Firmware DB verwalten.

There is no universal, non-interactive standard method for MOK enrollment across arbitrary firmware implementations; therefore this requires precise inventorying and process design.

Signing modules and integrating into CI/CD

Signing must become a standard step in your build/release pipeline. Use either a dedicated signing container/host in CI or a signing service with secured secret storage (HSM, Vault). Important principles:

• Signature as the last build step before packaging.
• Never store signing keys directly in target packages or on production systems.
• Verify modules after signing with modinfo (Signer, sig_key, sig_hash).

Example: GitLab-CI job for signing

stages:
  - build
  - sign

build_module:
  stage: build
  script:
    - make -C src
    - cp src/mydriver.ko artifacts/
  artifacts:
    paths:
      - artifacts/

sign_module:
  stage: sign
  dependencies:
    - build_module
  image: ubuntu:22.04
  variables:
    SIGN_KEY_PATH: /buildsecrets/MOK.priv
    SIGN_CERT_PATH: /buildsecrets/MOK.pem
  script:
    - apt-get update && apt-get install -y openssl Linux-headers-$(uname -r)
    - /usr/src/Linux-headers-$(uname -r)/scripts/sign-file sha256 "$SIGN_KEY_PATH" "$SIGN_CERT_PATH" artifacts/mydriver.ko
  artifacts:
    paths:
      - artifacts/mydriver.ko

In CI store the private key in a protected secret store (GitLab CI/CD Variables with masking or HashiCorp Vault). The runner performs signing in a controlled environment.

DKMS integration: hooks and automation

DKMS rebuilds modules on kernel updates. Without hooks DKMS often produces unsigned modules that cannot be loaded after a kernel upgrade. Add DKMS scripts that sign after each build.

Example: DKMS post-install hook

# /usr/src//2.0/dkms.conf
# In dkms.conf
POST_BUILD="/usr/src//2.0/dkms-sign.sh"

# /usr/src//2.0/dkms-sign.sh
#!/bin/bash
set -euo pipefail
MODULE_PATH="$1/$2"
SIGN_KEY="/etc/secure-signing/MOK.priv"
SIGN_CERT="/etc/secure-signing/MOK.pem"

if [ -f "$MODULE_PATH" ]; then
  /usr/src/Linux-headers-$(uname -r)/scripts/sign-file sha256 "$SIGN_KEY" "$SIGN_CERT" "$MODULE_PATH"
  echo "Signed $MODULE_PATH"
else
  echo "Module $MODULE_PATH not found"
fi

Store the signing keys on a dedicated signing host or in a protected path with restricted access. On target hosts distribute only the public certificate for the enrollment phase.

Distribution strategy for fleets

A robust target state includes:

• Centralized signing in CI/CD or a dedicated signing service.
• Distribute the public certificate (MOK.der) in a controlled manner via image, package or configuration management (e.g., Ansible/AWX), but enrollment still requires reboot + console.
• Packaged delivery (DEB/RPM) instead of single-file copies; post-install scripts run depmod and initramfs updates.
• Canary rollouts: start with a small number of hosts for validation, then roll out broadly.

Avoid distributing the private key or wholesale disabling Secure Boot.

Kubernetes perspective: Worker operations and best practices

In Kubernetes, worker nodes are direct operational points: missing modules for CNI (network), CSI (storage), or hardware drivers cause pods not to start or lead to storage errors. Therefore you should clearly differentiate infrastructure and cluster strategies.

Strategies for cluster operators

• Immutable node images: build node images (AMI/VM templates) in advance with already-signed modules — eliminating DKMS builds on production nodes.
• Rolling upgrade with a canary pool: test new images first on exclusive test nodes.
• Node batching: never reboot all workers at once; observe PodDisruptionBudgets.
• Preflight gates: automated checks before reboot to verify signed modules exist for the target kernel.

Example: reboot process for a node

# Drain Node
kubectl drain node-01 --ignore-daemonsets --delete-emptydir-data --timeout=10m
# Reboot and Healthcheck
ssh root@node-01 'reboot'
# Nach Reboot: Node wieder in Betrieb nehmen
kubectl uncordon node-01
kubectl get nodes --selector=kubernetes.io/hostname=node-01 -o wide

DaemonSet for preflight module checks

You can deploy a privileged DaemonSet that runs modinfo on each node and reports signature fields (Warning: security implications due to privileges).

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: module-signature-check
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: module-signature-check
  template:
    metadata:
      labels:
        name: module-signature-check
    spec:
      hostPID: true
      hostNetwork: true
      containers:
      - name: checker
        image: busybox
        securityContext:
          privileged: true
        command: ["/bin/sh", "-c"]
        args:
          - for m in /lib/modules/$(uname -r)/**/*.ko; do if [ -f "$m" ]; then modinfo "$m" | egrep -i "signer|sig_key|sig_hash|vermagic" || echo "$m: no sig info"; fi; done; sleep 3600
      tolerations:
      - operator: "Exists"
      RESTartPolicy: Always

The checker provides indications whether critical modules are signed; however, it must not replace the primary trust anchor.

Key rotation, revocation and contingency planning

Key rotation is an organizational task: plan a transition period in which old and new keys are accepted in parallel.

1. Generate a new key pair and publish the new certificate (MOK) for the enrollment phase.
2. Enroll the new certificate on all hosts (canary → batch).
3. Gradually sign new modules with the new key and distribute them.
4. After an observation period, remove the old key.

For revocation (e.g., compromised private key) you must evaluate firmware- or kernel-level blocking mechanisms (dbx or local policy); document break-glass processes and the communication protocol.

Monitoring, Tests und CI-Preflight-Checks

Automated tests are crucial: CI should verify before release that a module, after signing, reports a signature component with modinfo. In production monitor dmesg entries and node readiness metrics. Example check in CI:

# CI Preflight
modinfo artifacts/mydriver.ko | egrep -i "signer|sig_key|sig_hash" || (echo "ERROR: Modul nicht signiert" && exit 1)
# Simulierter modprobe (falls sicherheitsseitig erlaubt)
sudo modprobe -v artifacts/mydriver.ko || true
sudo dmesg | tail -n 50 | egrep -i "module verification failed|Required key not available" && exit 1 || echo "Preflight OK"

Rückfallstrategie (Break-Glass) — Schrittweise

If a rollout causes failures:

• Determine dmesg and modprobe errors via the console.
• If possible, boot an earlier kernel from the boot menu.
• If enrollment is faulty: perform MOK enrollment manually via the console (with physical access) or use prearranged remote console procedures.
• As a last resort and only temporarily: disable Secure Boot (firmware) to rescue critical systems — and then analyze the cause forensically.

Every break-glass incident must be documented, assessed and the process subsequently changed to prevent recurrence.

Kurze Betriebs-Checkliste vor großem Rollout

• Inventory: hosts with Secure Boot, firmware variants and console options.
• Signing model: private key secured, public certificate distributed.
• CI/CD: signing automated; Preflight-Checks implemented.
• DKMS: use hooks or built images.
• Kubernetes: node image strategy, canary rollout, drain/uncordon scripts.
• Monitoring: dmesg patterns, node readiness, alerting.
• Rollback: boot backups/kernel, console procedures, administrative contacts.

Fazit

Secure Boot protects the boot chain and increases security in the data center, but it is operationally useful only if signing processes, key management and enrollment are cleanly organized. Based on a central signing model, automated signing in CI/DKMS hooks, packaged delivery and careful canary rollouts, Secure Boot can be operated reliably in heterogeneous Linux and Kubernetes environments. Plan enrollment paths, test preflight checks and define clear break-glass procedures — that way Secure Boot remains enabled and your own kernel modules stay available and maintainable across kernel updates.

Secure Boot für eigene Kernel-Module: Betriebsrisiken und Architektur‑Alternativen

Besides signing and MOK enrollment you should consider the operational architecture and potential single-point risks. Three areas are decisive: key custody, distribution and enrollment path, and observability. Proven architecture patterns minimize attack surface and failure risk.

Recommended architecture model:

• Central signing service with HSM or Vault: the private key remains in a secured service, CI/CD receives only short-lived authorized tokens. This avoids distributing private keys to target hosts.
• Signature metadata in packages: augment DEB/RPM with signer info and checksum provenance so deployment tools can make preflight decisions.
• Enrollment‑strategies separated by host classes: Cloud‑VMs, bare‑metal and bare‑metal with restricted console require different enrollment workflows (image with MOK in advance, iKVM/serial for enrollments, provider firmware changes for critical hosts).

Operational precautions:

• Inventory: firmware implementation, console available?, supported capabilities (shim, mokutil).
• Canary pool with telemetry: validate module loads, node readiness and dmesg errors before a large rollout.
• Monitoring: centralize journald/dmesg patterns for „module verification failed“ and configure alerting with SLOs.

Emergency path: define clear, tested break‑glass procedures (fallback kernel, remote console plan, temporary disabling of Secure Boot only as a last resort) and document responsibilities. This links security requirements with a robust operational model for individual enterprise and cluster environments.

Kernel module signing and Machine Owner Key (Mok) are also important for this topic. The article places these aspects into clear context and shows what matters in day‑to‑day operations.