Energy optimization in the data center: CPU frequency governors, C-states and their latency impact

Energy optimization in the data center is no longer solely a cost issue. It affects thermal stability, hardware lifetime, rack density and, not least, the predictability of performance. In practice, an inconspicuous combination of CPU frequency control (CPU-Freq-Governors) and CPU sleep states (C-States, i.e. idle states) often determines whether systems run “sated” and predictably or whether sudden latency spikes occur that are hard to reproduce.

This article classifies the mechanisms clearly and translates them into operational decisions: What can be safely saved? Where does the risk of jitter (variations in response times) increase? And how should you proceed specifically in Kubernetes environments where many workloads share hosts, but individual applications still have strict latency requirements?

Energy optimization in the data center in practice

On modern server CPUs there are two central control levers:

• Frequency/voltage scaling via P-States (performance states): the CPU clocks higher or lower, often coupled with supply voltage. A CPU-Freq-Governor is the policy that decides how aggressively the CPU is scaled up or down.
• Idle sleep states via C-States: when a core has no work, it can enter progressively deeper sleep states. The deeper the C-State, the more energy it saves, but typically the longer the wake-up (exit latency) takes.

The operational misconception: “more saving = always slower.” In reality it is more nuanced. Many workloads are on average not CPU-bound but benefit from high efficiency as long as tail latencies (e.g. 99th percentile) do not run out of control. It becomes critical especially when load is bursty (short, frequent peaks), or when individual threads/requests must be served very quickly and the transition out of a deep C-State or from a low frequency to “full clock” falls exactly into that time window.

Additional amplifiers typical in data centers are: NUMA (Non-Uniform Memory Access, i.e. different memory access times depending on CPU socket), shared interrupts, Hyper-Threading/SMT (two threads sharing a core’s resources) and virtualization/containerization with changing CPU allocation.

CPU-Freq-Governors, P-States and drivers: what admins really need to know

Under Linux the CPU frequency subsystem has evolved historically. Crucial is which driver is active, because the driver defines the real control options and the observability. Common variants:

• intel_pstate: Intel-specific driver, often the default. It uses hardware/CPU-internal DVFS (Dynamic Voltage and Frequency Scaling) and offers policies like “performance”/“powersave”. The names can be misleading: “powersave” here often means “hardware-managed, efficient” and is not necessarily “slow”.
• amd_pstate: AMD equivalent on newer platforms, varying in maturity depending on kernel/BIOS.
• acpi-cpufreq: generic ACPI-based driver, provides classic governors like “ondemand”, “conservative” etc.

The governor policy determines how quickly the system reacts to load. In many data center setups reaction times (ramp-up) are more important than maximum frequency. That means: a policy that scales up quickly can stabilize tail latencies without remaining at maximum clock permanently.

Quick assessment on Linux hosts

For a reliable baseline you should first capture driver, governor and frequency limits. Use cpupower (package name varies by distribution):

# Überblick: Treiber, Governor, aktuelle Frequenz, Grenzen
cpupower frequency-info

# Kurzstatus pro CPU (je nach Distribution verfügbar)
cpupower frequency-info -p

Additionally, a look into sysfs is worthwhile (for automation/CMDB):

# Treiber und Governor pro CPU (CPU0 als Beispiel)
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Grenzen (kHz)
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq

Typical pitfall: BIOS settings can override or restrict Linux (e.g. „Energy Efficient Turbo“, „C-State Control“, „Package C-State Limit“, „Power Regulator“). If you tweak settings at the OS level but the BIOS/management controller enforces hard limits, you’ll see seemingly “correct” policies but no effect.

Understanding C-states: Core vs. Package and exit latency

C-states are idle states. A core can move from C1 (very shallow) down to deeper states. In addition there are package C-states, where the entire CPU socket (the „package“) sleeps deeper when enough cores are idle. These package states are often the source of hard-to-explain latency spikes because waking involves more components (uncore, cache hierarchy, interconnect).

The operational question is not „Which C-states are enabled?“, but: how often and for how long the CPU resides in which C-states, and whether exit latencies correlate with your SLOs (Service Level Objectives)?

Measuring C-state residency

On Linux turbostat (from the linux-tools) provides very useful indications. It shows, among other things, percentage residency in C-states as well as frequency/turbo behavior. Example:

# 10 Messzyklen à 1 Sekunde, kompakt
turbostat --quiet --Summary --interval 1 --num_iterations 10

If you see high residency in deep package states during periods when latency spikes occur, that’s a strong indicator. Conversely: if the CPU rarely goes deeper than C1/C2 anyway (e.g. due to background load, interrupts, monitoring agents), aggressive C-state tuning often yields little — and the cause lies elsewhere (scheduler, IRQ storms, storage, network).

Typical latency symptoms that match governors/C-states

Not every latency is „power“-related. The following patterns, however, often match frequency and idle topics:

• Millisecond-range spikes with otherwise stable average latency, often in bursty request patterns (API gateways, ingress, auth, cache-miss paths).
• Regular jitter clusters occurring at one-second intervals that correlate with idle/wakeup cycles.
• Only certain Nodes are affected (different BIOS profile, different microcode/kernel version, different power limits).
• Problem disappears under sustained load (because the CPU is no longer deeply idle) and occurs in „quiet“ phases.

Important: these symptoms can also be caused by IRQ balancing, remote NUMA accesses, cgroup CPU quotas, CPU steal (in virtualization) or garbage-collector pauses. Therefore, a measurement-and-exclusion strategy is central.

Practical workflow: From hypothesis to a robust decision

A viable process in the data center requires reproducibility, telemetry and rollback. A proven procedure:

1. Classify the workload: latency-sensitive (e.g. near-real-time services), throughput-oriented (batch), mixed (APIs with peaks).
2. Define SLOs and measurement points: p95/p99 latency, queue lengths, error budget, additionally energy/temperature (if available).
3. Measure the baseline: turbostat, cpupower, system load, IRQ rates, scheduler runqueue, Kubernetes metrics.
4. Make minimally invasive changes and only on selected Nodes (Canary): e.g. change the governor or limit C-states.
5. Compare against the baseline: same load profiles, same time windows, same measurement methodology.
6. Prepare rollback: restore OS/boot parameters, document BIOS profile, adjust automation playbooks.

Implementation on Linux: set governor, define limits, establish persistence

For many server workloads a pragmatic starting point is: Governor „performance“ on latency-critical Nodes, and an efficient hardware-policy mode (depending on the driver) on general Nodes. The decisive factor is Node differentiation, not a global dogma.

Temporarily set the governor (test)

# Performance-Policy für alle CPUs (temporär bis Reboot/Policy-Reset)
cpupower frequency-set -g performance

# Alternativ: powersave (bei intel_pstate oft "hardware-managed")
cpupower frequency-set -g powersave

If cpupower fails, it is often due to missing privileges (sudo), missing modules/tools or because the driver does not offer classic governors (e.g. depending on the intel_pstate mode). Then check what cpupower frequency-info reports as „available governors“.

Persistent configuration with tuned (RHEL-/CentOS-/Rocky-/Alma family)

tuned is a daemon that applies profiles for power/performance. For production operation this is often more stable than individual scripts, because it considers boot/service ordering.

# Status und aktives Profil
tuned-adm active

# Beispiel: auf Performance schalten (Host-weit)
tuned-adm profile performance

Pitfall: tuned can also modify IRQ balancing, disk settings or network parameters. For targeted CPU tuning a custom profile that sets only the necessary knobs is recommended, instead of blindly adopting a large default profile.

Limiting C-States: useful, risky, sometimes necessary

Limiting C-States is a coarse but effective lever when exit latencies dominate tail latency. Technically this is often achieved via kernel boot parameters or BIOS limits. That reduces potential power savings but can stabilize latency.

Kernel boot parameters (examples) and rollback capability

Which parameters make sense depends on platform and kernel. Commonly used knobs are:

• intel_idle.max_cstate: limits the maximum C-states in the intel_idle driver.
• processor.max_cstate: generic limit parameter (effects vary depending on driver).
• idle=poll: prevents deep idle states (very power-hungry, useful mainly for diagnostics).

Apply such parameters first on a canary node and document the exact before/after state. Example for GRUB (distributions differ; check your standards):

# 1) Aktuelle Kernel-Cmdline prüfen
cat /proc/cmdline

# 2) GRUB Default anpassen (Beispiel)
sudo sed -i 's/GRUB_CMDLINE_Linux="/GRUB_CMDLINE_Linux="intel_idle.max_cstate=1 processor.max_cstate=1 /' /etc/default/grub

# 3) GRUB-Konfig regenerieren (Pfad je nach Distribution)
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# 4) Reboot planen und nach dem Boot wieder prüfen
sudo reboot

Rollback is identical: remove the parameters, regenerate GRUB, reboot. Schedule a maintenance window for this, as it is a boot change.

Important for the decision: If you limit C-states, also monitor temperature, fan curves and power limits. In dense environments this can have secondary effects (thermal throttling) that ultimately cost performance again — just in a different way.

Kubernetes-specific: Node tuning without side effects

In Kubernetes it becomes more demanding because workloads are distributed dynamically. Energy optimization and latency stability here are not a „host fine-tuning“ exercise, but a question of node roles, scheduling and isolation.

Basic principle: separate node pools for different SLOs

If you mix latency-sensitive workloads (e.g. ingress, service-mesh data plane, auth, message-broker frontends) and batch/CI/ETL on the same nodes, every power policy becomes a compromise. Better:

• Node pool „low-latency“: conservative power-saving mechanisms (higher performance policy, limited C-states), stricter CPU isolation.
• Node pool „balanced“: efficient policies, but with monitoring of tail latency.
• Node-Pool „batch“: maximum energy efficiency, with higher tolerance for jitter.

You implement this separation via Labels/Taints/Tolerations (Label = criterion, Taint = „only with permission“). That prevents a single deployment from dictating your global CPU policies.

CPU Manager, cgroups and „Guaranteed“ Pods

The Kubelet CPU Manager can assign cores exclusively to Pods (policy „static“). That reduces scheduler jitter because a Pod does not constantly migrate between CPUs. The prerequisite is typically Guaranteed Pods (Requests = Limits) for CPU. Cgroups (Control Groups, resource control in the kernel) enforce the limits; with cgroup v2 some details are different, but the principle remains.

For an initial check: which cgroup version do your Nodes use?

stat -fc %T /sys/fs/cgroup

And whether the CPU Manager is active at all (Kubelet configuration is cluster dependent). On many systems you will find the configuration, for example, under /var/lib/kubelet/config.yaml:

sudo grep -E "cpuManagerPolicy|cpuManagerReconcilePeriod" -n /var/lib/kubelet/config.yaml || true

Typical pitfall: CPU Manager „static“ achieves little if CPU frequency is being aggressively saved at the same time or deep package C-states remain active. Exclusive cores reduce migration, but not the wakeup latency from deep idle. For Low-Latency nodes these topics should be considered together.

IRQ affinity and „Noisy Neighbors“

Interrupts (IRQs) are hardware events that consume CPU time outside of normal processes. If many NIC-/Storage-IRQs still land on an „exclusive“ core, the isolation is practically worthless. Therefore check IRQ load and distribution:

# IRQ statistics (snapshot)
cat /proc/interrupts | head -n 30

# Softirqs (network/block processing in the kernel)
cat /proc/softirqs | head -n 30

In Kubernetes a common trigger is: node-local DNS, CNI (Container Network Interface) and storage plugins generate background traffic that is processed on the „wrong“ CPUs. Depending on the platform, irqbalance configuration, CPU sets and a clear separation of „system cores“ and „workload cores“ (e.g. via isolcpus/nohz_full as advanced options) help here. However, these steps are far-reaching and should only be performed with a proper test and rollback plan.

Checklist: Check before tuning (so you don’t optimize the wrong thing)

• BIOS/UEFI: power profiles, C-State limits, Turbo, SMT, deterministic performance options. Document the state per node model.
• Microcode and kernel: differing versions measurably change power and idle behavior. Standardize them before you compare.
• Virtualization: in VMs CPU steal/host policy is often more dominant than guest tuning. This article primarily concerns Bare Metal or the host OS.
• Monitoring: measure p95/p99 and not just averages; correlate with CPU idle residency, frequency and IRQ rates.
• Load profile: tuning under synthetic sustained load can miss real burst behavior.

Troubleshooting: When new problems appear after tuning

Symptom: Higher temperatures but no improved latency

Then the cause is often not C-State-Exit, but queuing elsewhere (network, storage, lock contention) or thermal throttling. Check whether the CPU scales its clock under load as expected and whether power limits are taking effect. On many servers BMC/Redfish or vendor tools provide the necessary values; at the OS level turbostat is useful as an indicator.

Symptom: Higher throughput but worse tail latency

This often indicates „too aggressive“ boost/turbo phases that run into thermal limits, or increased competition from background tasks (e.g., because everything runs faster but more happens in parallel). In Kubernetes: check pod density per node, CPU throttling (cgroup) and whether QoS classes are set correctly.

Symptom: Only individual pods are affected

Then CPU policies are usually only one factor. Common causes are CPU quota/throttling (limits), unfavorable CPU affinities, GC pauses (on JVM/.NET), or „noisy neighbor“ effects from shared caches/SMT. For latency-critical pods you should set Requests/Limits consistently and, if necessary, move to dedicated nodes.

Rollback strategy: How to safely return to the original state

A rollback strategy is not a formality, but the difference between a controlled experiment and a late-night escalation. Plan at least these layers:

• OS-Policy Rollback: Reset governor (cpupower/tuned), reload services, documented desired state in configuration management.
• Boot-Parameter Rollback: Revert GRUB changes, keep previous kernel cmdline available, if necessary boot via the boot menu.
• Node-Pool Rollback in Kubernetes: Remove canary nodes from the pool (cordon/drain), shift workloads back to „balanced“, then make further changes.

For Kubernetes operations (Canary/Drain) as a standard toolbox:

# Cordon node to prevent new pods
kubectl cordon <node-name>

# Evacuate pods in a controlled manner (Note: PDBs and stateful workloads)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Re-enable node after rollback
kubectl uncordon <node-name>

Pitfall: drain can fail because of PodDisruptionBudgets (PDB) or local data (emptyDir). That is not a „Kubernetes problem“ but part of operational reality — and precisely why you should treat power/latency tuning as a controlled node-pool process.

Best practices: Aligning energy efficiency with stable latency

• Segment rather than tweak globally: Different policies per node pool are almost always better than a cluster-wide compromise.
• Measure residency, not just watts: C-state and frequency metrics often explain latency spikes better than power consumption alone.
• Standardize drivers/BIOS first: Without a reproducible baseline, comparisons are worthless.
• IRQ and CPU isolation for low latency: Exclusive cores, system cores, IRQ affinity — but only with proper change management.
• Rollback as a runbook: Tuning without a rollback is not production-ready in a data center.

Conclusion: Energy optimization is an operational design, not a switch

Energy optimization in the data center is sustainable when you treat CPU frequency governors, P-States and C-States as part of an operational design: node roles, Kubernetes scheduling, measurement methodology and fallback strategy belong together. For general workloads, efficient hardware scaling is often non-critical. For latency-critical services, by contrast, you should deliberately apply more conservative policies, limit deep idle states and consistently enforce isolation (CPU/IRQ) – not everywhere, but where it actually protects SLOs.

If you want to address the topic systematically in your environment, next plan a canary pool, define measurement criteria (p95/p99 + Residency) and document BIOS/kernel parameters per platform. That turns “perceived faster” into a robust, operationally safe decision.