Memory leak diagnosis on servers with perf, pmap and Valgrind

Memory-leak diagnosis starts with observation: slowly rising RAM usage, recurring OOM-killer logs or unexplained swap activity are clear warning signs. In this article I present a practical procedure for memory-leak diagnosis on Linux servers. The methods combine simple inspections (pmap, /proc/*/smaps), sampling and uprobe analyses with perf as well as deep heap inspections with Valgrind. The goal is to provide administrators and system engineers with a reliable diagnostic path: prerequisites, typical pitfalls, concrete commands and safe fallback strategies.

When is it really a memory leak?

Before you start tooling: not every increasing RAM pattern is a leak. Typical sources that can appear like leaks are growing caches, memory-mapped files, fragmentation or short-lived load spikes. A memory leak, in the narrow sense, is unwanted, non-released memory held by a process or the kernel that persistently grows over time.

Diagnostic indicators:

• Long-term monotonic increase of a process’s RSS (Resident Set Size) over hours/days.
• OOM entries in the kernel log associated with the same process.
• High Private-Dirty or Private-Clean values in /proc/<pid>/smaps that cannot be explained by legitimate caching strategies.

Initial observation: metrics and monitoring

Before any debugging run the problem must be measurable. Use monitoring tools (e.g. Prometheus, Zabbix, Datadog) or simple commands to establish trends. Capture baseline metrics: RSS, VSZ, swap usage, page-ins/outs and number of processes/threads.

# Einmalig RSS und VSZ eines Prozesses prüfen (PID bekannt vorausgesetzt)

ps -o pid,user,vsz,rss,cmd -p 1234

Interpretation: VSZ is the virtual address space (incl. mmaps), RSS is actually resident in RAM. A rising RSS is a stronger indication of a real leak.

# Systemweite Tendenzen: freier Speicher, Swap, Pageins/outs

vmstat 5 12

If page-outs and swap usage grow in parallel with the RSS increase, there is immediate operational relevance (performance degradation). Document values as a reference before making changes.

Process focus: pmap, smaps and PSS

pmap shows a process’s memory map. It is useful to find large mmaps (e.g. big files, shared memory). Complementarily, /proc/<pid>/smaps provides detailed metrics like Pss (Proportional Set Size), Private_Dirty and Shared_Clean. PSS allocates the size of shared pages fairly among involved processes and is therefore more useful for accounting.

# Detaillierte Zuordnung mit pmap

pmap -x 1234

# Smaps liefert pro-mapping Angaben (PSS, Private_Dirty, Referenced ...)

grep -A5 "^Rss: |^Pss: |^Private_Dirty:" /proc/1234/smaps | sed -n '1,200p'

Practice: look for mmaps with unusually large Private_Dirty or steadily growing PSS values. Tools like smem consolidate PSS per process and are helpful for reports.

Sampling with perf: quickly validate hypotheses

perf is a kernel-based performance tool. In diagnosis it helps find hot paths and caller chains. For memory leaks we use two patterns:

• Sampling of CPU stacks to identify code paths that cause memory allocations or cache growth.
• Uprobes (user-space probes) on allocators such as malloc, realloc or mmap to observe actual allocation calls.

Important: For perf you need appropriate privileges (root or adjustment of /proc/sys/kernel/perf_event_paranoid) and ideally debug symbols so that stack traces are meaningful. Uprobes only work if the library/executable to be instrumented is stable and the paths are known.

# Register uprobe on malloc (glibc as an example) - root only

perf probe -x /lib/x86_64-Linux-gnu/libc.so.6 malloc

# Sampling malloc calls for PID 1234 for 30 seconds

perf record -e probe:libc:malloc -p 1234 -g -- sleep 30

# Analysis

perf report -i perf.data --stdio

Why this works: Uprobes allow capturing function calls in userland; combined with callchains (stack traces) you can see which application code frequently triggers allocations. When it fails: statically linked binaries, JIT-compiled processes, or stripped binaries provide little in the way of meaningful symbols.

Alternative perf traces

You can also use tracepoints for sys_enter_mmap or kernel events like kmem:kmalloc (for kernel leaks):

# Observe mmap syscalls

perf record -e syscalls:sys_enter_mmap -p 1234 -g -- sleep 30

This data shows whether the process generates many mmaps — typical for growing memory-mapped caches or library-side behavior.

In-depth heap analysis with Valgrind

Valgrind (Memcheck) is a dynamic analysis tool that detects memory leaks and invalid reads/writes. It does not replace production monitoring: Valgrind significantly increases runtime and memory usage and belongs in a test or staging environment that reproduces production behavior.

Important options:

# Classic leak analysis

valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --log-file=valgrind.log ./your_binary --args

# Massif for heap profiling (heap snapshot over time)

valgrind --tool=massif --pages-as-heap=yes --massif-out-file=massif.out ./your_binary --args
# Visualization on the admin desktop
ms_print massif.out

Why Valgrind: Memcheck finds real retention leaks (alloc without free) and shows precise call stacks when the binary and libraries are built with debug symbols. Limitations: GC-based languages (e.g., managed runtimes) or complex asynchronous servers behave differently under Valgrind; browsers or large services are often impractically large.

Typical pitfalls and how to avoid them

• Production Valgrind runs: avoid. Use a staging environment or capture via core dumps and offline analysis.
• Fragmentation vs. leak: Sometimes virtual and resident memory grow due to fragmentation. Check /proc/<pid>/smaps and tools like the malloc-stats of your malloc implementation.
• Shared Memory and mmaps: Large mmaps can appear as RSS, but are not necessarily leaks—check the backing files.
• Threads and TLS: Thread-specific caches (Thread-Local Storage) increase persistent memory, especially with a high thread count.

When Valgrind is not practical: Alternatives

For production-near heap analyses without Valgrind, consider:

• jemalloc with built-in profiling (jeprof)
• gperftools (tcmalloc) heap profiler
• heaptrack (sampling-based userland analysis)
• ASan/LSan (Address/LeakSanitizer) during the build/test process — better for CI than for production

These tools are generally less invasive than Valgrind and can run in a (customized) staging or canary environment.

Pragmatic diagnostic workflow — checklist

1. Observe: export trend data from monitoring, document RSS/RAM/swap behavior.
2. Per-process focus: ps, top, pmap -x and /proc/<pid>/smaps analyze. Goal: determine process ID and affected mappings.
3. Sampling with perf: set uprobes on malloc/realloc/free or syscalls like mmap; evaluate call chains.
4. Reproduce in staging: run Valgrind (Memcheck) and Massif, generate and analyze heap dumps.
5. Develop fix: object release, cache limits, pooling or swapping allocator/libs.
6. Deploy with canary: define versioning and rollback plan, set monitoring alerts.
7. Postmortem: document root cause, indicators, fix and automated monitoring alerting.

Safe operational and rollback strategies

Diagnoses can be risky: perf with uprobes is minimally invasive, Valgrind disruptive. Never perform invasive tests directly in Production. When rolling out a fix, keep a fast rollback ready (Package-Repository, Service-Manager-Skript, Kubernetes-Revision). Define alert levels: e.g. warning on a 20% increase within 1 hour, critical alarm on OOM entries.

Kernel-side: slab and kernel leaks

Sometimes leaks are kernel-side (slab allocator, network buffers). Check:

# Slab-Statistiken

slabtop -s c

# Kernel dmesg auf OOM oder allocation failures prüfen

dmesg --ctime | tail -n 200

Kernel leaks are significantly harder and usually require kernel profiling or live-patching; as an admin, document findings and escalate to kernel developers or vendor support.

Practical examples — short workflows

1) Quick test for increasing mmaps (Production, low invasiveness):

# Beobachtung: welche Prozesse verursachen viele mmaps in kurzer Zeit
perf record -e syscalls:sys_enter_mmap -a -g -- sleep 30
perf script | head -n 200

2) Uprobe on malloc in Staging (map which callchains trigger many allocations):

perf probe -x /lib/x86_64-Linux-gnu/libc.so.6 malloc
perf record -e probe:libc:malloc -p 1234 -g -- sleep 60
perf report -i perf.data --stdio

3) Valgrind-Memcheck in Testumgebung:

valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log ./service_binary --config /etc/service/conf

Memory leak diagnosis: advanced strategies for complex environments

In real-world hosting environments, services rarely run in isolation. Containers, systemd units, sidecars or shared libraries alter the situation and require adapted workflows.

Containers and cgroups (practical notes)

In container environments, cgroups limit memory usage. The process may have a leak but will be killed earlier due to the cgroup limit. Check cgroup statistics:

# cgroup v1 Beispiel (Pfad anpassen)
cat /sys/fs/cgroup/memory/docker//memory.usage_in_bytes
cat /sys/fs/cgroup/memory/docker//memory.max_usage_in_bytes
# cgroup v2 Beispiel
cat /sys/fs/cgroup//memory.current
cat /sys/fs/cgroup//memory.max

Tips: In Kubernetes check events (kubectl describe pod) and container logs; set conservative limits so a leak remains reproducible but does not immediately destroy the environment.

Managed-Runtimes: Java, Go, Node.js

Valgrind is often unsuitable for managed runtimes. Use runtime-native profilers:

• Java: jmap/jcmd/jstack, Heap Dumps analysiert mit Eclipse MAT (Memory Analyzer Tool).
• Go: runtime/pprof, pprof-UI, heap-profiles über net/http/pprof.
• Node.js: heapdump, –inspect, v8 heap-profiling.

# Beispiel: Go-Profil aktivieren (Servercode muss pprof importieren)
# Zugriff lokal: go tool pprof http://localhost:6060/debug/pprof/heap
# Beispiel: Java Heap-Dump erzeugen
jmap -dump:live,format=b,file=heap.hprof 
# Analyse lokal mit Eclipse MAT
mat heap.hprof

These tools provide heap snapshots, reference graphs and dominator trees that show which objects retain memory.

Debug-Builds, Sanitizer und Build-Flags

For repeatable analysis, developers should provide debug builds and sanitizers. Important compiler flags:

# Beispiel GCC/Clang Debug/Sanitizer-Flags
CFLAGS="-g -O0 -fsanitize=address,leak -fno-omit-frame-pointer"
# Für ASan/LSan in Tests verwenden; nicht für Produktion

Why: Debug symbols make stack traces meaningful. ASan/LSan find Use-After-Free and leaks quickly during test runs; limitations are performance overhead and other runtime changes.

Automation, CI and annual checks

Memory leaks are often latent and manifest under long-term load. Plan automated checks:

• Nightly regression in CI with sanitizers and reproducible workloads.
• Periodic Massif runs in staging with known load, artifact-based reports in the artifact repo.
• Alert playbooks that automatically create a snapshot (pmap/smaps) and start a short perf sampling when RSS rises.

# Beispiel-Skript: bei Alarm automatisch pmap+smaps anlegen
#!/bin/bash
PID=$1
OUTDIR=/var/tmp/heap-dumps/$(date +%F_%T)
mkdir -p "$OUTDIR"
pmap -x "$PID" > "$OUTDIR/pmap.txt"
cp /proc/$PID/smaps "$OUTDIR/smaps"
# optional: perf kurz laufen lassen
perf record -e probe:libc:malloc -p $PID -g -- sleep 10
mv perf.data "$OUTDIR/"

Automated artifacts help developers recreate the reproducible scenario more quickly.

Measurable metrics and Prometheus examples

Define clear metrics in Prometheus for alerting rules:

# PromQL-Beispiel: plötzlicher RSS-Anstieg über 1 Stunde
increase(process_resident_memory_bytes{job="myservice"}[1h]) > 200000000
# oder: relative Änderung
(rate(process_resident_memory_bytes{job="myservice"}[15m]) > 1048576)

Add application-instrumentation: runtime heap-size metrics, count of open caches, cache hit/miss rates — this makes it easier to distinguish leaks from legitimate cache growth.

Rollback plan and emergency runbook

A clean rollback is mandatory. Example steps for systemd services:

# Example rollback (systemd unit)
# 1. Stop new pods / services
systemctl stop myservice
# 2. Deploy old version (from repo or package)
apt-get install --allow-downgrades myservice=1.2.3-1
# 3. Start and monitor
systemctl start myservice
journalctl -u myservice -f --lines=200

In Kubernetes use rollback mechanisms (kubectl rollout undo) and ensure that alerts are disabled during the rollback or placed into a monitoring maintenance window.

When to escalate — vendor or kernel support

Escalate when:

• you see clear kernel indicators (slab growth, allocation failures).
• process backtraces point to library code you do not maintain (e.g., glibc, a third-party library).
• you observe reproducible memory leaks in managed services and no viable workaround exists.

Prepare these artifacts for support cases: dmesg excerpt, slabtop snapshot, pmap/smaps, perf captures, Valgrind logs or heap dumps, and exact reproduction steps.

Summary and conclusion

The memory-leak diagnosis can be performed systematically and stepwise: detect via monitoring, use pmap/smaps to narrow the scope, employ perf (incl. Uprobes) for quick hypothesis tests, and use Valgrind for deep heap analysis in an appropriate environment. Additional tools include runtime-native profilers (jemalloc, heaptrack), sanitizers in CI, and automated artifact generation on alerts. Important prerequisites are administrative privileges for perf, debug symbols to better interpret call chains, and isolated staging environments for invasive analysis tools. Common root causes are caches, mmaps, or fragmentation — these should always be checked as possible causes before resorting to invasive debugging measures. In closing: plan every step with a rollback strategy, document artifacts and alerts, and coordinate closely with development teams so fixes are reproducible and can be deployed safely to production.

Further guidance: Consider using allocator-specific profilers (jemalloc, tcmalloc) for recurring issues; these are often more production-friendly than Valgrind. Keep debug builds and reproduction scripts ready so developers can promptly reproduce what you observed during analysis.