Securing CI/CD Runners: Credential Isolation, Secure Workspace Cleanup, and Protection of the Build Chain

Securing CI/CD runners is not a security hack but a sustainable operational task: Runners pull source code, build artifacts and often have access to credentials, caches and the internal network. For administrators and operators it is critical to measurably reduce attack surface and have clear audit and rollback paths. This guide provides concrete measures, verification steps and troubleshooting for credential isolation, robust workspace cleanup and protection of the build chain.

Why runners are a critical attack vector

Runners are execution environments for CI/CD jobs (builds, tests, packaging). A compromised runner can:

• Exfiltrate secrets (tokens, SSH keys, cloud credentials).
• Tamper with artifacts or sign them maliciously.
• Perform cache poisoning and thereby influence subsequent builds.
• Enable pivoting into the internal network.

Important distinction: trusted-builds (e.g. protected branches or releases) versus untrusted-builds (forks, external PRs, public contributions). For untrusted-builds the operational assumption must be: the job is potentially malicious.

Fundamental principles: least privilege, isolation, auditability

Harden along three measurable objectives:

• Isolation objective: No job may see another job’s state.
• Credential objective: Jobs receive only the minimally required, preferably short-lived credentials.
• Integrity objective: The provenance and immutability of dependencies and artifacts are traceable.

These objectives can be operationalized, tested and audited.

Credential isolation: implementation, rationale and common pitfalls

Credential isolation means: no static all-purpose tokens. Split identities by pipeline phase (build, package, release, deploy) and by context. Use short-lived tokens (TTL in minutes/hours) via OIDC, STS (Security Token Service) or HashiCorp Vault leases. Short-lived credentials limit the blast radius in case of compromise and simplify revocation.

Technical options and their use

OIDC (OpenID Connect) is a protocol for issuing short-lived tokens; it connects CI systems to cloud IAM or Vault without persistent secrets. KMS/HSM (Key Management Service / Hardware Security Module) stores private keys outside the runners. Vault provides dynamic secrets (e.g., database credentials on a lease basis). Each option has operational requirements: OIDC requires reliable token claims, Vault requires high availability and access policies.

Example: Vault policy for package push

# Vault policy (HCL) - erlaubt Token zum Schreiben in ein internes Artefakt-Repo
path "secret/data/ci/artifacts/*" {
  capabilities = ["create", "update", "read"]
}

Why this works: Vault issues time-limited tokens and the policy restricts paths. When it fails: if runners persist the Vault token (e.g. in cache) or policies are defined too broadly.

Policy-restricted signing via a signing service

Private signing keys must not reside on ordinary runners. A signing service (an internal service that signs via KMS/HSM) accepts artifact hashes, verifies policies (e.g. that the build originates from a protected branch) and then signs. The signing operation requires separate audit logs and strict authentication.

Securing CI/CD runners: infrastructure-level isolation

Choose isolation according to risk and cost-effectiveness:

• Host-Runner: Fast, but only for fully-trusted jobs.
• Container-Runner: Good performance, but only with rootless / strict mount policies.
• Ephemeral VM/Instance-Runner: Highest isolation; a fresh VM or snapshot per job. Costs higher, security highest.

Ephemeral runners are particularly recommended for untrusted builds: after completion the instance is destroyed so persistence is not possible.

Example: Kubernetes runner as an ephemeral job

apiVersion: batch/v1
kind: Job
metadata:
  name: runner-job-{{ .RunID }}
spec:
  template:
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
      containers:
      - name: builder
        image: registry.internal/runner-image:stable
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
      RESTartPolicy: Never
  backoffLimit: 0

Why this helps: Kubernetes allows namespaces, NetworkPolicies and pod security contexts for RESTriction. Caution: faulty RBAC or volume settings can undermine isolation.

Secure workspace cleanup: robust and tamper-resistant

A faulty cleanup enables persistence. Problems arise from open handles, mounted filesystems, symlink tricks and RESTrictive ACLs. A robust cleanup implements multiple layers of protection: dedicated workspaces, ownership normalization, –one-file-system when deleting and verification runs with audit logging.

Linux: Extended cleanup including handle check

#!/usr/bin/env bash
set -euo pipefail
JOB_DIR="$1"
RUNNER_UID=1001
RUNNER_GID=1001

# 1) Prozesse beenden, die in JOB_DIR arbeiten
fuser -k -TERM -m "$JOB_DIR" || true
sleep 1
fuser -k -KILL -m "$JOB_DIR" || true

# 2) Ownership und Rechte normalisieren
chown -R "${RUNNER_UID}:${RUNNER_GID}" "$JOB_DIR" 2>/dev/null || true
chmod -R u+rwX,go-rwx "$JOB_DIR" 2>/dev/null || true

# 3) Prüfen auf Mountpoints im Jobdir
mountpoints=$(findmnt -n -o TARGET --target "$JOB_DIR" || true)
if [ -n "$mountpoints" ]; then
  echo "Found mounts: $mountpoints" >&2
  # Option: detach mounts safely or warn and abort
fi

# 4) Löschen sicher durchführen
rm -rf --one-file-system "$JOB_DIR"

Why fuser is necessary: open file handles prevent deletion; processes must be terminated cleanly. The script can fail if a job has started system processes that the script is not allowed to terminate — therefore logging and escalation levels are important.

Windows: Handles, reboot plan and antivirus interaction

On Windows locks by services and AV are common. Complement the cleanup with handle checks (Sysinternals Handle/Process Explorer), retries and a planned reboot path if deletion is not possible. Mere retries do not always suffice for persistent locks — document a rollback path.

Build chain security: package sources, caches and signing

Protect the build chain by isolating package sources, proxies with allow-lists, controlled caches and immutable artifact policies.

Internal mirrors and egress RESTriction

In addition to allow-lists for package endpoints, runners should have only defined egress: VCS, mirror, artifact repo, KMS/signing service. Egress-ACLs reduce opportunities for exfiltration / connections to command-and-control infrastructure. Test changes first in monitor mode (logging only) before blocking.

Artifact repo hardening

Set write policies (non-overwrite), retention policies and require-signed-artifact flags when your repo supports them. Audit-Logs must show who published an artifact and when. Example rule: „Releases may only be published after being signed by the signing service“.

Cache-Strategien gegen Poisoning

Separate caches by trust zones and projects. Avoid shared writable caches for untrusted jobs. Use TTL invalidation and recorded cache hashes to detect poisoning.

Netzwerk- und Host-Härtung: konkrete Maßnahmen

Treat Runner like critical infrastructure components:

• dedicated network segments or VLANs
• defined Egress-ACLs
• host firewall and patch process
• monitoring and centralized logs

Beispiel: einfache iptables-Regel für Egress (VM-Runner)

# Erlaube nur DNS, HTTP(S) zu mirror.example und signing.example
iptables -A OUTPUT -m owner --uid-owner runner -p udp --dport 53 -j ACCEPT
iptables -A OUTPUT -m owner --uid-owner runner -p tcp -d mirror.example --dport 443 -j ACCEPT
iptables -A OUTPUT -m owner --uid-owner runner -p tcp -d signing.example --dport 443 -j ACCEPT
iptables -A OUTPUT -m owner --uid-owner runner -j DROP

Why this helps: Even if a job attempts to establish malicious communication, egress remains limited. Note: DNS-over-HTTPS and other bypass methods can circumvent this—test and monitor.

Testing, Audit und Validierung

Validate measures through automated tests and audits:

• Regular audit jobs that attempt to perform disallowed actions from a Runner (only in an isolated test network!).
• Post-job audits: check workspace remnants, open handles, active processes.
• Log analysis: look for token exfiltration or unusual egress connections.

Beispiel-Testskript: Post-Cleanup-Validation

#!/usr/bin/env bash
# Prüft, ob ein Job-Verzeichnis nach Cleanup noch existiert
JOB_DIR="$1"
if [ -e "$JOB_DIR" ]; then
  echo "CLEANUP FAILED: $JOB_DIR still exists" >&2
  ls -la "$JOB_DIR" >&2
  exit 2
fi
# Prüfe auf aktive Prozesse des Runner-User
if pgrep -u runner >/dev/null; then
  echo "ACTIVE PROCESSES FOUND" >&2
  ps -u runner -o pid,cmd
  exit 3
fi
exit 0

Typische Stolperfallen und Troubleshooting

1) Geheimnisse versehentlich ins Build-Log

Cause: debug prints or missing masking. Measures: enable CI log masking, do not output sensitive variables as plain text. Automatically scan logs for patterns (API keys, bearer tokens) with a scanner job.

2) Privileged Container missbraucht

Cause: privileged containers or passing through the Docker socket. Measure: rootless builds, no host socket mounts, use Buildkit/remote daemon. Test: job attempts to start new containers on the host (only in the test environment!).

3) Signierschlüssel auf Runner

Cause: practicality over security — teams store keys locally. Measure: centralize signing services, keep keys only in an HSM/KMS. Fallback: immediate rotation and invalidation of all potentially compromised keys.

Rollback- und Notfallstrategie

Introduce changes in stages:

1. Monitor-Mode: logging only, no blocking.
2. Stepwise closure of egress/policy areas.
3. Run old and new Runner pools in parallel; switch via feature flag/route.
4. Break-glass tokens for emergencies with short-lived validity and audit.

Important: document the rollback steps and test them regularly in an isolated test run.

Practical checklist: implementation in 90–180 minutes

• Inventory active tokens and verify permissions for each pipeline phase.
• Migrate untrusted jobs to separate runner pools or ephemeral VMs.
• Extend cleanup scripts: process-kill, chown, chmod, –one-file-system, Post‑Validation.
• Audit the signing process and plan key movement into HSM/KMS.
• Create egress ACLs in monitor mode and observe traffic.
• Audit job: attempt to perform unauthorized actions from runners (isolated).

Conclusion

Securing CI/CD runners means reconciling practicality with measurable security. Prioritize credential separation by pipeline phase (using short-lived tokens), guaranteed workspace cleanup (or ephemeral runners) and signing via a signing service or HSM/KMS. Complement this with egress RESTrictions, monitoring and regular audit tests. With phased rollouts, monitor-mode phases and tested rollback paths, security remains manageable and operationally robust.

FAQ

Securing CI/CD runners: operation, monitoring and integration

Technical measures are only as good as their operation. Plan monitoring, metrics and integrations already when introducing Runner hardening so that security remains repeatable and measurable.

Key metrics and alerts:

• Cleanup-failure rate: proportion of jobs where Post-Job‑Validation failed. Define a threshold and trigger an alert before cleanup deficits lead to persistence.
• Vault-lease errors and OIDC token failures: indicate problems with short-lived credentials or invalid claims.
• Unusual egress connections per Runner pool: sudden increases suggest exfiltration or evasion attempts.
• Signature requests per hour and signing failures: deviations can indicate compromised pipelines or policy errors.

Integration notes:

• IAM/IdP: Bind CI as an OAuth/OIDC client to existing identity providers (e.g., AD, Okta). This keeps auditing and user lifecycle centrally manageable.
• Secrets backends: Use Vault/KMS dynamically; automate lease renewal and rotation. Document emergency rotations and test the key revocation path.
• Artifact repository: Augment signing policies with attestations (build metadata, SBOM). This makes provenance verifiable and simplifies incident analysis.

Runbook and fallback practice:

• Create a concise runbook: steps to isolate a Runner pool, key rotation, a disable switch for the signing service and rollback to warm-standby runners.
• Conduct regular chaos tests in a test zone (cleanup fail, egress block) and evaluate operational processes against concrete SLAs.

Conclusion: Operations and integration are not afterthoughts. Good metrics, automated rotation and coordinated runbooks make CI/CD security robust and manageable in daily operations.

Supply-chain security and protecting the build chain are also important for this topic. The article places these aspects into context and shows what matters in day-to-day operations.