Securing Kubernetes: Pod Security Admission, NetworkPolicies, OPA Gatekeeper and Runtime Scanning

Securing Kubernetes starts with a clear prioritization: network separation, admission control for deployments, and continuous anomaly detection. In this article I explain in practical terms how to combine Pod Security Admission (a Kubernetes Admission Controller for enforcing pod security profiles), NetworkPolicies (network segmentation inside the cluster), OPA Gatekeeper (policy engine for admission control) and runtime scanning (near-runtime detection of attacks and misbehavior). The intended audience is administrators, system engineers and technical service providers responsible for operation, migration and hardening.

Securing Kubernetes: Core components and security strategy

A structured approach reduces risk and operational overhead. Start with a simple, incremental protection model:

• Hardening the pod runtime (Pod Security Admission) – rules for permissions, host access, capabilities.
• Network isolation (NetworkPolicies) – minimal access surface between services.
• Admission control with policies (OPA Gatekeeper) – enforceable corporate rules before persistence.
• Runtime detection and forensics – agents like Falco or eBPF tools capture anomalous behavior.

These layers complement each other: PSA prevents risky pods, NetworkPolicies limit lateral access, Gatekeeper enforces organizational constraints, and runtime scanners detect active attacks or misconfigurations.

Threat model and operational assumptions

Before technical measures, it must be clear which risks you are addressing. Typical scenarios:

• Faulty container images with root privileges.
• Lateral movement between pods due to permissive cluster network rules.
• Misconfigurations in deployments (hostNetwork, hostPath, privileged).
• Exploit against runtime processes that executes a shell inside containers.

Prerequisites for the following measures: you have cluster-admin access, your CNI supports NetworkPolicies (e.g. Calico, Cilium, Weave), and you have access to CI/CD or image registry scanning.

Pod Security Admission: What it is, how it works, common pitfalls

Pod Security Admission (PSA) is a built-in Admission Controller since Kubernetes 1.22+ that enforces pod security profiles via namespace labels. Profiles are predefined policies such as „privileged“, „baseline“ and „RESTricted“. PSA acts on creation/modification of PodSpec: it denies or warns on violations.

Why PSA is effective

PSA intervenes early in the lifecycle of a resource (admission), i.e. before a pod is persisted to etcd. It is simple to configure and requires no external components. PSA limits risky fields such as hostPath, privileged containers, certain capabilities or container user root.

Prerequisites and risks

PSA is a must if you want to enforce consistent minimal privileges. Risks: incorrectly configured profiles can break deployments. Therefore you should apply enforcement incrementally (audit → warn → enforce) and have a rollback strategy.

Checklist before activation

• Identify critical namespaces and starter namespaces.
• Test profiles in a staging cluster or in „warn“ mode.
• Document exception rules for special workloads (e.g. node agents).

Example: Namespace Label for „RESTricted“ (enforce)

A label on the namespace sets the profile. „enforce“ blocks violations.

kubectl label ns finance pod-security.kubernetes.io/enforce=RESTricted

You can observe beforehand with „audit“:

kubectl label ns staging pod-security.kubernetes.io/audit=baseline

Typical failure scenarios

Common pitfalls are configurations for DaemonSets/NodeAgents that require hostPath or CAP_SYS_ADMIN. Pragmatism helps: place systems that require exceptions into separate Namespaces, document the reasons, and thereby consolidate the attack surface.

NetworkPolicies: segmentation in Kubernetes

NetworkPolicies govern which Pod may reach which Pod. Important: NetworkPolicies only work if the CNI supports them. Without NetworkPolicy, intra‑cluster traffic is allowed by default (Allow all).

Principles

Adopt a „deny by default“ posture per Namespace: allow only necessary connections (e.g. Frontend→Backend, Monitoring→Exporters). This reduces lateral movement in case of compromise.

Prerequisites

Check your CNI:

kubectl get pods -n kube-system -o wide

Look for Calico, Cilium, or Weave pod names. If your CNI does not support NetworkPolicy, you must plan a CNI change or use supplementary network devices.

Example: default-deny and targeted allow

Default‑deny for incoming traffic in the Namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: app
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Allow only from frontend Pods to the backend service:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: app
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 8080

Verification and debugging

Test connectivity with a temporary Pod:

kubectl run --rm -it --image=busybox nettest -- /bin/sh
# inside the Pod
# telnet backend 8080 or nc -zv backend 8080

If you expect connectivity and it is absent: check NetworkPolicy resources, labels, and CNI logs. Common mistakes: incorrectly set labels, namespaces not taken into account, or policies that only govern Ingress but need to allow Egress.

OPA Gatekeeper: centrally manage admission policies

Open Policy Agent (OPA) is a generic policy engine; Gatekeeper is a Kubernetes controller that integrates OPA as an admission controller. With Gatekeeper you write company‑specific rules (Constraints) that are checked before resources are persisted.

Why Gatekeeper complements PSA and NetworkPolicies

PSA provides predefined pod security profiles; Gatekeeper allows flexible, organization‑specific rules (e.g. „Only internal registries allowed“, „No hostPath for non-ops teams“). Gatekeeper can also run audits on existing resources.

Installation (brief)

Gatekeeper can be installed via Helm or manifests. Example with Helm:

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm repo update
helm install gatekeeper/gatekeeper --name-template gatekeeper --namespace gatekeeper --create-namespace

Example: ConstraintTemplate and Constraint (not a complete Rego tutorial)

ConstraintTemplate defines the schema; Constraint activates the rule with parameters. Example: disallow privileged: true in PodSpec.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8snoPrivileged
spec:
  crd:
    spec:
      names:
        kind: K8sNoPrivileged
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8snoPrivileged
      violation[{
        "msg": msg,
        "details": {"container": container_name}
      }] {
        input.review.object.spec.containers[_] as c
        c.securityContext.privileged == true
        container_name := c.name
        msg := sprintf("privileged container %v is not allowed", [container_name])
      }

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoPrivileged
metadata:
  name: disallow-privileged
spec:
  match:
    kinds:
    - apiGroups: ["" ]
      kinds: ["Pod"]

When Gatekeeper detects a violation, the admission is rejected and an explanatory error is returned to the API user.

Common operational pitfalls

Overly RESTrictive Rego policies can break CI/CD pipelines. Recommended approach: test policies locally in a staging cluster, use Gatekeeper Audit Mode (searches for existing objects) and roll out policies gradually.

Runtime‑Scanning: detection of near‑runtime threats

Runtime scanning refers to two areas: static image scans before deployment and runtime agents that monitor events on the host. Both are important: image scans (Trivy, Clair) find known CVEs; runtime agents (Falco, eBPF tools) detect anomalous behavior such as shell exec, unusual network connections, or write access to sensitive paths.

Why runtime detection is necessary

Even vetted images can contain vulnerabilities or be exploited via misconfigurations. Runtime agents provide real‑time detection and context‑rich alerts for incident response.

Example: image scan with Trivy

trivy image --severity HIGH,CRITICAL --ignore-unfixed myregistry.local/myapp:1.2.3

Result: Use the output to block builds or to create tickets in the ticketing system.

Example: Falco for runtime alerts

Falco monitors syscalls and can be deployed in Kubernetes as a DaemonSet. Example rule (simplified excerpt) that alerts on execve in containers:

- rule: Detect Shell in Container
  desc: Shell executed in a container
  condition: evt.type = execve and container.id != host
  output: "Shell or suspicious exec in container (user=%user.name container=%container.name cmd=%proc.cmdline)"
  priority: WARNING

Installation via Helm:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco --namespace falco --create-namespace

Integration into monitoring and incident response

Send Falco alerts to SIEM/Alertmanager/Slack. Define playbooks for typical alerts (e.g. „exec in production container“): stop the container task, isolate the Pod (change NetworkPolicy), secure snapshots and forensic data.

Concrete implementation sequence and test plan

Recommended order for rollout in production:

1. Baseline‑analysis: inventory of all namespaces, workloads, CNI type.
2. Enable Pod Security Admission initially in audit mode. Collect violations.
3. Integrate image scanning into CI (Trivy) and mark failing builds.
4. Roll out NetworkPolicies gradually from least privilege (start with non-critical namespaces).
5. Develop Gatekeeper policies and run them in audit mode.
6. Deploy Falco or a comparable runtime agent as a DaemonSet, integrate alerts into the SIEM.
7. Stabilize enforcement (PSA enforce, NetworkPolicies active deny, Gatekeeper enforce) and monitoring.

Typical checks

Perform these checks before enabling enforcement:

• List pods with risky settings (privileged, hostPath):

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.containers[].securityContext.privileged==true) | {ns:.metadata.namespace, pod:.metadata.name}'

• Verify that NetworkPolicies allow the desired connections: temporary test pods for connectivity checks.
• Gatekeeper audit: Review violations CRs that Gatekeeper writes.

Rollback and emergency strategy

Security systems can impact operations. Plan the following fallback paths:

• PSA: Removing the namespace label disables enforcement for the namespace.
• NetworkPolicies: A default-deny can be removed with kubectl delete; keep a temporary „allow all“ policy ready.
• Gatekeeper: Use kubectl patch/scale to briefly disable Gatekeeper if strictly necessary (but only as a last resort).
• Runtime agent: Temporarily disable alerts if they produce mass false positives, but retain the logs.

Emergency runbook (summary)

1. In case of massive blockages, check recent audit logs (PSA/Gatekeeper) and identify the policy.
2. If a critical job is stopped: remove the namespace label or temporarily disable individual constraint objects.
3. Communication: Inform affected teams, deploy a hotfix, and document the cause.
4. Post-mortem: Adjust the policy, expand tests, and perform change control.

Operational knowledge, troubleshooting and best practices

Pragmatic operational tips:

• Versioning: Manage policies (Gatekeeper ConstraintTemplates, NetworkPolicy YAMLs) like code in Git and validate them via CI.
• Testing: Test policies automatically against reference workloads (canary namespace).
• Observability: Centralize and correlate audit logs, CNI logs, Falco alerts, and kube-audit.
• Least privilege: Start with more RESTrictive defaults for newer namespaces; treat long-running services as legacy with documented exceptions.
• Documentation: Emergency paths, who removes labels or disables Gatekeeper, and change approval are operational requirements.

Performance and scaling aspects

Security has costs. PSA itself introduces little latency because it performs lightweight checks. Gatekeeper, however, executes Rego evaluations on every admission; under a high deploy rate this can cause latency or CPU load in the API path. Therefore plan resource requests/limits for Gatekeeper and monitor API server latency (kube-apiserver metric: apiserver_request_duration_seconds).

NetworkPolicies are enforced by the CNI: Calico can use iptables/ipsets or eBPF; Cilium operates natively with eBPF. eBPF-based enforcement mechanisms generally provide lower latency and better scalability, but they require kernel support and a recent distribution. Validate CPU/network performance before and after rollout using load tests on a test cluster.

Handling of false positives and optimizing rules

False positives are an operational risk. Procedure:

1. Run the policy in audit mode and collect violations.
2. Analyze the context (Pod, user, image) and then decide: adjust the rule or document an explicit exception.
3. Test rule changes in staging, then promote them incrementally.

Examples for analyzing Falco alerts: determine the triggering syscall pattern, inspect the process chain with kubectl logs and exec, and extend the Falco rule with exception criteria instead of disabling rules globally.

Host and hardware operation: kernel, eBPF and resources

For runtime protection the host is important: eBPF tools require kernel support (4.9+ with eBPF backports or newer distributions). Check kernel versions and modules. On hosts with a high containerd density pay attention to I/O limits, since forensic snapshots can become resource-intensive.

Practical check: verify kernel version and validate eBPF support:

uname -r
# eBPF: prüfen ob bpftool vorhanden
bpftool version || echo "bpftool not installed"

If eBPF is missing, plan kernel updates or alternative agents such as Falco in syscall mode. Document the required host packages and perform rolling kernel upgrades with a live-kernel strategy to avoid downtime.

Audit logging and forensics: what should be collected

Collect the following artifacts for rapid response and post-mortem:

• kube-apiserver audit logs (Audit-Policy configured).
• CNI logs and Netflow traces for lateral movement.
• Falco/Zabbix/Prometheus alerts and full events with timestamps.
• Container logs and a snapshot of the affected filesystem.

Snapshot example: create a tar archive for forensics in a controlled manner (only if you account for storage and I/O):

kubectl exec -n ns pod -- tar -czf /tmp/fs-snapshot.tar.gz -C / --exclude=/proc --exclude=/sys --exclude=/dev
kubectl cp ns/pod:/tmp/fs-snapshot.tar.gz ./fs-snapshot.tar.gz

Store snapshots securely, sign metadata, and maintain correlations with timestamps.

Compatibility and integrations

Key integration points you should plan for:

• CI/CD: image scanning (Trivy) as a build gate, Gatekeeper as a pre-apply gate.
• Registries: enforce signed images and allowed registry lists via Gatekeeper.
• Monitoring/SIEM: Falco → Alertmanager/SIEM, CNI logs → Elastic/Kafka.
• Secrets management: avoid direct secrets in manifests; integrate ExternalSecrets/Vault.

Conclusion: pragmatism over perfection

Securing Kubernetes is a multi-layer process: Pod Security Admission, NetworkPolicies, OPA Gatekeeper and runtime scanning complement one another. A phased rollout with audit phases, automated tests and clear rollback rules is essential. Prioritize measures by attack surface and operational effort: small investments in PSA and NetworkPolicies often deliver immediately tangible risk reduction; Gatekeeper and runtime scanners improve governance and detection over the long term.

Operationalization is critical: versioned policies, playbooks for incidents, and monitored test zones prevent security measures from impairing operations. Plan performance tests and host checks (kernel, eBPF) before using eBPF-dependent features in production.

Further checks (short checklist to take away)

• Kubectl: Check for privileged containers, hostPath, hostNetwork.
• CNI: Confirm support for NetworkPolicy.
• CI: Integrate an image scanner into the pipeline.
• Gatekeeper: Run policies in audit mode, review violations.
• Runtime: Deploy Falco or an eBPF agent and route alerts to the SIEM.

Runtime scanning is also important for this topic. The article clearly contextualizes these aspects and shows what matters in day-to-day operations.