Analyzing High CPU Load Caused by Kernel Threads and irq/softirq: Atop & perf Practical Handbook

High CPU load caused by kernel threads and irq/softirq is a classic operational issue: monitoring shows rising load averages and sluggish services, yet top or htop reveal no obvious process culprit. In such cases work is executed in the kernel (kernel threads) or as interrupt/softirq processing. This runbook describes how to use atop to analyze the timeline and perf together with eBPF to trace kernel paths, classify typical causes, derive measures, and plan safe fallback strategies.

Terms briefly explained

A short alignment of terms prevents misinterpretation:

• Load Average: Measures the average number of threads that are running or waiting for CPU/I/O. A high load does not automatically mean full CPU utilization; it can also indicate I/O wait or blocking.
• CPU shares: user (application processes), system (kernel work), irq (hardware interrupts), softirq (software interrupts, i.e. deferred processing), iowait (waiting for storage I/O) – this breakdown is important for root-cause analysis.
• Kernel threads: Processes in kernel context such as kworker/* or ksoftirqd/*. They perform kernel work and appear in process listings, but are not „user CPU“.
• IRQ / SoftIRQ: IRQs are hardware-driven interrupts; SoftIRQs are their processing in software context, commonly used by network and I/O stacks (e.g. NAPI for networking).

Prerequisites and security framework

Analyses often require root privileges and can themselves generate load. Plan for:

• Coordination with Security/Change Management, especially when using perf or eBPF.
• Short measurement windows (e.g. 10–60 seconds) and passive log capture before making changes.
• Rollback and documentation path for sysctl changes, systemd units or driver flags.

Quick initial diagnosis (5 minutes)

This sequence quickly determines whether IRQ/SoftIRQ, I/O or VMM stealing are the cause.

uptime
mpstat -P ALL 1 5
top -H -b -n 1 | head -n 80

Interpretation: If mpstat shows elevated values in irq/soft and top lists threads such as ksoftirqd/N or kworker, kernel/interrupt processing is likely. Steal indicates hypervisor resource contention.

Atop as a timeline tool

atop records historical metrics more persistently than top and maps CPU breakdown over time. Ideally log files already exist; otherwise observe live.

# Live monitoring
atop 1

# Read historical file (example)
atop -r /var/log/atop/atop_20260728 -b 10:00 -e 10:30

What to look for: temporal correlation between network or storage spikes and rising softirq/irq time, stable versus sporadic patterns, and which kernel threads are most active.

Focus: perf for kernel paths

perf reveals at function and symbol level where CPU time is spent in the kernel – for example in the network stack (napi, skb), block I/O (nvme, block), or Netfilter/conntrack. Results are limited in containers or when debug symbols are missing.

Preparation and safe use

# Check for RESTrictions
sysctl kernel.perf_event_paranoid
sysctl kernel.kptr_RESTrict
perf --version || true

If kernel.perf_event_paranoid > 1 is set, perf may be RESTricted. Changes should be coordinated with Security. Start without callgraphs (-g) to keep overhead low.

Short profile and reporting

# Kurzüberblick (interaktiv, geringerer Overhead)
sudo perf top

# Konservatives Recording: 30s, Sampling-Frequenz 99Hz
sudo perf record -a -F 99 -- sleep 30
sudo perf report --stdio

Review the results for hotspots: names like napi_poll, __netif_receive_skb_core, xfrm_output or nf_conntrack_* indicate network/Netfilter; block_rq_issue, nvme_submit_io indicate storage.

In-depth perf analysis and artifacts

If the initial recording yields indications, extend selectively. Callgraphs (-g) provide context but increase sampling overhead; use them judiciously.

# Callgraph nur wenn nötig, kurze Dauer
sudo perf record -a -F 249 -g -- sleep 15
sudo perf script > perf.raw
# Optional: FlameGraph-Erzeugung (auf Admin-Workstation)
# git clone https://github.com/brendangregg/FlameGraph.git
# ./FlameGraph/stackcollapse-perf.pl perf.raw > out.folded
# ./FlameGraph/flamegraph.pl out.folded > perf.svg

Why this works: sampling profilers collect stack traces of the currently running execution; aggregated stacks reveal dominant paths. When it fails: with very short load or sampling configurations that swallow system-call signatures (e.g. missing symbols).

eBPF and tracepoints for targeted questions

eBPF (Extended Berkeley Packet Filter) enables low-latency tracing in the kernel. For production systems, bpftrace is a good choice for short hypothesis tests; bpftool helps with metrics. As always: coordinate with security and keep runtimes short.

# Beispiel: Zähle Aufrufe von ksoftirqd-Handlern (bpftrace)
sudo bpftrace -e 'tracepoint:irq:softirq_entry { @[comm] = count(); }' -c 'sleep 10'

# Alternativ: Trace network napi poll duration
sudo bpftrace -e 'kprobe:napi_poll { @[comm] = hist(nsecs); }' -c 'sleep 10'

Why use it: eBPF measures at fine granularity without massive overhead, ideal to check whether, for example, napi_poll runs for a long time. When it fails: older kernels without BPF support or RESTrictions imposed by distribution policies.

Common causes, investigative paths and concrete commands

1) Network: PPS, offload, CNI/overlay

Many small packets (high packets-per-second, PPS) drive SoftIRQs. Check for drops, errors, offloads (TSO/GSO/GRO) and CNI encapsulation.

ip -s link show dev eth0
ethtool -k eth0
ethtool -S eth0 | sed -n '1,200p'
# RPS (Receive Packet Steering) prüfen und setzen
cat /proc/sys/net/core/rps_sock_flow_entries
# Beispiel: RPS für rx-queues setzen (Queue anpassen)
echo 32768 | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpus

Note: RPS/RFS attempts to distribute processing across multiple CPUs. If set incorrectly, it can violate cache locality and increase latency.

2) irqbalance and CPU affinity

irqbalance distributes interrupts. With isolated CPUs or NUMA this can be suboptimal. Check smp_affinity per IRQ.

sudo systemctl status irqbalance --no-pager
# Beispiel: IRQ-Affinity anzeigen
awk 'NR>1{print $1}' /proc/interrupts | head -n 5 | sed 's/://g' | while read irq; do
  echo "IRQ $irq:"; cat /proc/irq/$irq/smp_affinity_list 2>/dev/null || true
done

If an IRQ is dominated by a single CPU, targeted pinning can provide relief. Risk: incorrect pinning can worsen throughput or latency — always measure.

3) Storage: Completion storms, driver errors

Many short I/Os, timeouts or driver warnings lead to kworker activity.

iostat -x 1 5
nvme list 2>/dev/null || true
dmesg -T | tail -n 200

Driver errors in dmesg or Kernel OOPS are critical indicators; these often require a driver update or kernel rollback.

Kubernetes: Node vs Pod distinction and checks

Common causes in Kubernetes: pod with high PPS, kube-proxy NAT/conntrack, CNI overlay or CSI driver behavior. Important steps:

kubectl get nodes -o wide
kubectl top nodes
kubectl top pods -A --sort-by=cpu | head -n 30
# On the node: check conntrack counter
sudo sysctl net.netfilter.nf_conntrack_count
# CNI: check whether overlay (vxlan/geneve) is running
ip link show | grep vxlan -A 2 || true

Tip: A single pod can cause high Node softIRQ without showing much user CPU itself. Use DaemonSet diagnostic runners for node-level profiling, not just kubectl top.

Metrics, dashboards and retention

To make diagnosis reproducible, collect these metrics persistently (Prometheus/Grafana or similar): per-CPU irq/softirq, NET_RX/NET_TX counters, PPS, rx_errors/drops, iowait, kworker/ksoftirqd thread counts, conntrack size. Ensure sufficient retention (e.g., 7–30 days) to detect regressions after kernel updates.

Practical checklist: minimal diagnostic path

1. Capture: uptime, uname -a, dmesg, /proc/interrupts, /proc/softirqs.
2. Quick checks: mpstat, top -H, ip -s link, ethtool -S, iostat.
3. Atop: identify temporal correlation.
4. perf (conservative): short capture, initially without -g.
5. eBPF for hypothesis testing: short bpftrace scripts.
6. Apply changes incrementally: offloads, RPS, irqbalance, pinning — measure after each.
7. Plan and document rollback.

When simple measures don’t help

There are cases where simple tuning measures fail and deeper interventions are necessary:

• Kernel or driver bug: dmesg or perf show kernel paths that can only be solved by patching or rolling back.
• Hardware offload incompatibility in virtualization: disable offloads and test.
• Architecture-imposed limits: e.g. massive NAT/conntrack load requires architectural changes (LoadBalancer instead of NodePort/SNAT).

Rollback and communication strategy

Communication is critical: when making changes inform SRE/stakeholders, schedule maintenance windows if necessary and record pre/post metrics. Best practices:

• Document changes individually with timestamps (system-level changelog).
• Automated measurement jobs before/after (mpstat, /proc/interrupts, atop intervals).
• In the Kubernetes context: cordon/drain a test node, then apply change and measure with load simulation.

Conclusion

High kernel or irq/softirq load is often hard to pinpoint because the work happens in the kernel rather than in user processes. A methodical approach with a fast coarse analysis (mpstat/top/atop), targeted profiling (perf) and focused tracing (eBPF) yields testable hypotheses. Measure, change, rollback: that is the sequence. In Kubernetes, the distinction between pod- and node-causer is particularly important; the wrong measure can otherwise affect the whole cluster.

This runbook provides the tools and diagnostic paths with which administrators, system engineers and operators can perform grounded analysis, test safely and roll back changes responsibly.

High CPU load from kernel threads and irq/softirq — operational and architectural measures

Beyond the immediate analysis, sustainable operational and architectural measures are decisive to prevent the problem from recurring. Do not make decisions based on a single profile: develop permanent detection, test, and rollback paths that integrate into change processes and automation.

Continuous detection and alerting

A short-term perf snapshot only helps once. Configure alerts that detect SoftIRQ changes per CPU and PPS increases and alert on regressions in kernel or CNI components. Example of a simple Prometheus rule:

- alert: HighSoftirqPerCpu
  expr: increase(node_softirq_total[5m]) / count(node_cpu_seconds_total{mode="system"}) > 1000
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Erhöhter SoftIRQ-Anteil auf {{ $labels.instance }}"

Important: calibrate rules to your baselines to avoid false positives during seasonal load.

Canary changes and rollout strategy

Changes to sysctl, IRQ pinning, or offloads should be tested as canaries on a small number of nodes. Practical workflow:

• Cordon/drain a test node.
• Change using a version-controlled sysctl drop-in.
• Automated measurements (5–15 minutes) against the baseline.
• Rollback on degradation.

Example sysctl drop-in:

# /etc/sysctl.d/99-softirq-tuning.conf
net.core.default_qdisc = fq
net.core.rps_sock_flow_entries = 32768
net.ipv4.conf.all.rp_filter = 1

Automation and runbook integration

Integrate check scripts into your automation (Ansible, Salt, Terraform for cloud instances) and version sysctl/irqbalance configs in Git. A runbook trigger should automatically run a short profile after kernel updates and write a health report into your ticketing flow. This avoids surprises after patches.

Architectural measures: shift load, don’t just patch

Some SoftIRQ issues cannot be solved by tuning; architectural steps are more effective here:

• Reduce PPS at the node level by using L4/L7 load balancers instead of SNAT/NodePort to avoid conntrack pressure.
• Batching and keep-alive in custom enterprise software reduce packet counts; check socket options (TCP_CORK, sendmmsg) under high PPS load.
• Coordinate offloading strategies with the cloud or NIC vendor: HW offload can shift load but also reveal driver incompatibilities.

Risks, vendor coordination, and audit artifacts

Collect findings in a standardized way (perf raw, flamegraphs, atop snapshots, tcpdump‑pcap with timestamps) before opening vendor tickets. Without reproducible artifacts, diagnosis is prolonged. For kernel/driver issues, plan staged kernel rollouts and keep boot entries (GRUB) for quick revert.

In short: invest in monitoring hygiene, versioned configurations, canary rollouts, and architectural review. This makes the response to high kernel or irq/softirq load predictable, measurable, and reversible — in line with existing change and security processes for your digital enterprise solutions.

Atop analysis and Perf CPU profiling are also important for this topic. The article contextualizes these aspects clearly and shows what matters in everyday operations.