Container Storage Troubleshooting: OverlayFS, device-mapper and loopback pitfalls

Containers rarely run slowly or unstable because of “Docker itself” – very often the underlying storage setup is the actual trigger. This Container storage troubleshooting focuses on three classic operational problem areas: OverlayFS (usually used as overlay2), device-mapper (devicemapper/thin provisioning) and the particularly insidious loopback configurations. The goal is a runbook you can use even under time pressure: classify symptoms, narrow down causes, verify with reliable checks and implement changes so that a rollback path is available.

Important: Many storage problems initially appear as “app errors” (timeouts, 500s, build aborts), but are in truth I/O latency, metadata bottlenecks or a faulty driver/filesystem combination. Those who separate these cleanly save hours in incident calls.

How container storage works internally (brief overview for troubleshooting)

Container engines like Docker store images and container writable layers in a Storage-Backend. The Storage-Driver decides how “layers” (read-only image layers) and the writable container layer are implemented. With Linux overlay2 (OverlayFS) is the standard today; older setups sometimes use devicemapper. Additionally there are Volumes (persistent data, usually as a host path or via volume plugins) that are independent of image layering.

For troubleshooting the distinction is central:

• Image/Container layers (Storage-Driver): builds, pull/push, start times, “copy-on-write”.
• Volumes/Bind-mounts: databases, uploads, queue data, log directories – often the performance hotspot.
• Filesystem & Blockdevice: XFS/ext4, LVM, RAID, SAN, virtio, NVMe – this is where latency spikes and metadata limits occur.

Symptoms: How to detect storage problems early

Typical warning signals in container operation are remarkably repeatable. Pay particular attention to these patterns:

• “no space left on device”, even though df -h still shows space (often inodes, thin pool full, or overlay metadata).
• Extremely slow builds with many small files (metadata operations, copy-up costs with OverlayFS).
• Containers start delayed or hang during I/O-intensive init steps.
• High iowait on the host, accompanied by application timeouts.
• Unclear errors in the Docker daemon log around mounts, “failed to mount overlay”, “Invalid argument”.
• Sudden performance degradation after a kernel/distribution upgrade or filesystem migration (e.g. XFS options, d_type/ftype).

Initial assessment: Which storage configuration is actually running?

Before you debug details, establish a baseline: storage driver, root dir, filesystem, mount options and whether volumes or layers are affected.

Check Docker info and paths

docker info --format 'Driver={{.Driver}}
DockerRootDir={{.DockerRootDir}}
BackingFilesystem={{.BackingFilesystem}}
SupportsDType={{.DriverStatus}}'

docker info | sed -n '/Storage Driver/,$p' | sed -n '1,80p'

Why this helps: The storage driver narrows the class of problems. DockerRootDir shows where the data lives (default: /var/lib/docker). The filesystem underneath determines whether OverlayFS operates correctly.

Filesystem and mount options at DockerRootDir

DOCKER_DIR=$(docker info --format '{{.DockerRootDir}}')

df -Th "$DOCKER_DIR"
findmnt -no SOURCE,FSTYPE,OPTIONS -T "$DOCKER_DIR"

# Inodes: important for "no space left" despite free space
df -ih "$DOCKER_DIR"

Interpretation: For OverlayFS, XFS or ext4 are common. XFS must support d_type (historically visible for Docker as ftype=1). Inode exhaustion is frequently the real bottleneck when many small files and layers exist.

OverlayFS/overlay2: common causes, checks, fixes

OverlayFS is a kernel mechanism that „stacks“ two directory layers: a lowerdir chain (image layers, read-only) and an upperdir (container writable layer). When a container modifies a file from a lower layer, a copy-up happens: the file is copied into the upper layer and then modified there. Functionally this is robust, but it has clear performance and metadata characteristics.

Pitfall 1: XFS without d_type (ftype=0) or „Invalid argument“

A classic: Docker starts, but mount errors occur during certain operations or container starts fail. Background: OverlayFS requires d_type (directory entry type) to reliably identify file types. With XFS this is a formatting option; with ext4 it is typically not an issue in practice.

# Nur sinnvoll, wenn das Backing-FS XFS ist:
# Ausgabe: ftype=1 ist gut; ftype=0 ist problematisch.
XFS_DEV=$(findmnt -no SOURCE -T "$(docker info --format '{{.DockerRootDir}}')")

# xfs_info erwartet das Blockdevice oder den Mountpoint
xfs_info "$(findmnt -no TARGET -T "$(docker info --format '{{.DockerRootDir}}')")" | tr ' ' 'n' | grep -E '^ftype=' || true

Fix reality: ftype is not switchable. If XFS was formatted with ftype=0, the only option is migration to a correctly formatted filesystem (e.g. new XFS with ftype=1 or ext4) and then reinitializing the Docker data root. Plan downtime and an image repull for this (see section „Migration and rollback strategy“).

Pitfall 2: „no space left“ due to inode or metadata constraints

Overlay2 creates many directories and metadata per layer. For CI workloads or build hosts, inode exhaustion is more common than actual capacity shortage.

DOCKER_DIR=$(docker info --format '{{.DockerRootDir}}')

df -h "$DOCKER_DIR"
df -ih "$DOCKER_DIR"

# Hotspot-Suche: wo liegen besonders viele Einträge?
# (Kann dauern, bei Bedarf außerhalb der Peak-Zeit laufen lassen.)
sudo du -x -d 2 -h "$DOCKER_DIR" 2>/dev/null | sort -h | tail -n 30

Why this fails: On some filesystems/partitions inodes are fixed. Even with 200 GB free, a exhausted inode quota can hard-block write operations. In addition, thin-provisioning or quota mechanisms at the VM/SAN level can produce similar effects.

Pitfall 3: Performance regressions caused by copy-up and „chown -R“ in images

Many admins first attribute the problem to the application. Typical, however, is an image/entrypoint that recursively sets permissions or modifies large directories at startup. On OverlayFS this triggers copy-ups and metadata updates — on many small files that is expensive.

Check whether the hotspot is in the container layer or in volumes. A quick indicator is the I/O load at startup and whether the paths are inside bind mounts. In production setups: data directories (e.g. databases) belong in volumes, not in the container writable layer.

OverlayFS checks that actually help during an incident

# Kernel- und OverlayFS-Sichtbarkeit
uname -r
lsmod | grep -i overlay || true

# Docker-Daemon-Logs (systemd)
sudo journalctl -u docker --since "-2h" | tail -n 200

# Aktive Mounts: Overlay-Mounts zeigen lowerdir/upperdir/workdir
mount | grep -E ' type overlay ' | head -n 20

Interpretation: You will often see „failed to mount overlay“ directly here, or kernel errors that point to incompatible options, too-long lowerdir chains (rare today) or corrupted metadata. If mounts increase massively, also check for container leaks (containers not removed, temporary builders filling up).

device-mapper/devicemapper: thin pool, metadata and operational pitfalls

device-mapper is a Linux kernel subsystem that virtualizes block devices. In the Docker context devicemapper is usually implemented as thin provisioning: data and metadata are managed in a thin pool. This can run stably, but is sensitive to incorrect provisioning (pool/meta full), monitoring gaps and — especially — loopback setups.

Important: loopback for devicemapper is almost always an anti-pattern

With loopback, „virtual block devices“ are stored as files on a filesystem. devicemapper is then built on top of those files. That adds latency, cache effects and failure modes. It is acceptable for tests, but in continuous operation it is a common cause of performance problems, I/O spikes and hard-to-explain „Device is busy“ errors.

Read devicemapper status from Docker

docker info | sed -n '/Storage Driver: devicemapper/,$p' | sed -n '1,120p'

Pay attention to terms such as Pool Name, Data file/Metadata file (indicator of loopback) and Deferred Removal/Deferred Deletion.

Check thin-pool usage and metadata (lvs)

When devicemapper runs on top of LVM (common), LVM thin pools are the core. Important is not only the data portion but also the metadata (mapping tables). Full metadata means: write stop, often abrupt.

# Übersicht über LVM Thin Pools und deren Auslastung
sudo lvs -a -o +devices,lv_size,data_percent,metadata_percent,segtype,lv_attr,origin,pool_lv

Interpretation: data_percent near 100% is critical. metadata_percent near 100% is often even more critical, because then even „small“ changes fail. Plan monitoring/alerts on both values.

Check directly via dmsetup (if LVM info is not sufficient)

sudo dmsetup status
sudo dmsetup ls --tree

Why this helps: dmsetup also shows states that are not visible in Docker (e.g. deferred deletion backlog). With high container churn, „deferred deletion“ can create the illusion that space is never freed.

Typical error scenarios for devicemapper

• Pool/meta full: Containers fail to start, pulls fail, „no space left“ despite the host partition appearing free.
• Loopback latency: Sporadic timeouts, high iowait, especially during parallel builds.
• Unexpected „Device is busy“: Cleanup operations block when devices are still referenced.

Detect and assess loopback pitfalls

Loopback is not only „slower“ — it also changes diagnostics. You then have two layers where space can become constrained: the filesystem holding the loopback files, and the thin pool above them. Additionally, fragmentation and filesystem journaling can increase latency.

Is it really loopback?

docker info | sed -n '/Storage Driver/,$p' | sed -n '1,160p'

# Loop-Devices auf dem Host sichtbar?
losetup -a || true
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS | sed -n '1,120p'

Interpretation: If you see many /dev/loop* entries and Docker is using devicemapper, the probability is high that loopback is involved. With overlay2, loopback is usually not relevant (unless you run Docker on a loopback-based underlying layer, e.g. nested lab setups).

Decision: immediate measure vs. sustainable correction

In an incident the goal is often stabilization: relieve pressure on the system before you migrate. Sustainable correction is almost always: remove loopback, standardize the storage driver and place Docker data on an appropriate block device.

Runbook: sequence of steps for reproducible container storage troubleshooting

The following verification sequence is designed so you can quickly decide whether you are dealing with cleanup, capacity or a structural migration.

1) Clarify scope: Layer-Storage or Volume-Storage?

# Welche Container schreiben besonders viel? (Grobindikator über Log/IO nicht direkt sichtbar)
# Hilft, Kandidaten zu identifizieren.
docker ps --format 'table {{.Names}}t{{.Image}}t{{.Status}}'

# Volumes und Mounts eines auffälligen Containers anzeigen
C=<container_name_or_id>
docker inspect "$C" --format '{{json .Mounts}}' | jq .

Why this helps: If a problem is primarily in Volumes (e.g. NFS, CIFS/SMB, iSCSI), storage-driver tuning does little. Conversely, „slow image pull“ or „build hangs“ are often layer-storage issues.

2) Host capacity, inodes, top consumers

DOCKER_DIR=$(docker info --format '{{.DockerRootDir}}')

df -Th "$DOCKER_DIR"
df -ih "$DOCKER_DIR"

# Größte Docker-Verzeichnisse
sudo du -x -d 1 -h "$DOCKER_DIR" 2>/dev/null | sort -h

3) Docker objects: How much is actually occupied?

docker system df
docker system df -v | sed -n '1,200p'

Interpretation: High „Reclaimable“ shares indicate cleanup potential. Caution: „Reclaimable“ is not the same as „safe to delete.“ Deletion can remove build caches and increase pull load.

4) Safe cleanup (with clear boundaries)

If you need to free space short-term, start conservatively. Plan for running containers not to be removed, but for unused resources to be.

# Unbenutzte Images, Container (stopped), Netzwerke, Build-Cache entfernen
# Vorsicht in CI-Umgebungen: Build-Cache-Verlust kann Builds verlangsamen.
docker system prune

# Aggressiver: auch unbenutzte Images entfernen
docker system prune -a

# Volumes nur entfernen, wenn Sie absolut sicher sind
docker volume ls
# docker volume prune

Why this works: Many „full“ hosts are simply the result of missing lifecycle rules. When it fails: If the bottleneck is not capacity but inodes, thin-pool metadata or I/O latency. Or if space is not freed immediately due to deferred deletion (devicemapper).

5) Make I/O latency visible (host perspective)

For storage problems, not only %util matters — latency is the primary concern. Depending on the distribution, different tools are available; iostat is often readily available.

# Pakete ggf. installieren: sysstat
# sudo apt-get install -y sysstat  |  sudo yum install -y sysstat

iostat -x 1 10

# Grobe Prozesssicht, um I/O-Wait zu erkennen
vmstat 1 10

Interpretation: High await or svctm (depending on version) and consistently high utilization indicate block device bottlenecks. In that case, cleanup measures are often only treating symptoms.

Migration: Change storage driver or relocate DockerRootDir (with fallback strategy)

When the diagnosis is clear (e.g. devicemapper loopback in permanent use, XFS without ftype, persistently too few IOPS), a structural change is cleaner than „more space.“ For operators it’s important: changing the storage driver usually means existing local images/layers in DockerRootDir must be rebuilt. Persistent data must reside in Volumes anyway; everything else is replaceable.

Preparation: What needs to be backed up?

• Compose-/Stack-Definitionen, systemd-Units, Environment-Dateien: damit Sie Container reproduzierbar starten.
• Registry-Zugriffe (Credentials, Mirrors): damit ein Repull funktioniert.

Downtime plan (generic, without orchestrator)

# 1) Laufende Container geordnet stoppen
docker ps -q | xargs -r docker stop

# 2) Docker-Dienst stoppen
sudo systemctl stop docker

# 3) Datenverzeichnis sichern (Rollback-Pfad)
DOCKER_DIR=$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || echo /var/lib/docker)

# Hinweis: Bei großen Verzeichnissen dauert das. Alternativ: Snapshot auf Storage-Ebene.
sudo tar -C "$(dirname "$DOCKER_DIR")" -cpf /root/docker-rootdir-backup.tar "$(basename "$DOCKER_DIR")"

# 4) Neues Filesystem/Blockdevice mounten und als DockerRootDir konfigurieren
# (Konkrete Mount-/mkfs-Schritte sind umgebungsspezifisch.)

# 5) Docker mit neuer Konfiguration starten
sudo systemctl start docker

# 6) Images neu ziehen und Workloads starten
# docker compose up -d  (falls genutzt)

Why a tar backup as a fallback path? Not because you want to „always“ restore it, but because it gives you an option if unexpected dependencies reside in the old root (e.g., local images without a registry, forgotten data in the writable layer). In larger environments a storage snapshot (LVM, SAN, VM snapshot used with caution) is often more practical.

Set DockerRootDir via daemon.json

Many teams use a dedicated partition for Docker to clearly isolate space, inodes and IOPS. This is configured in /etc/docker/daemon.json.

{
  "data-root": "/var/lib/docker",
  "storage-driver": "overlay2"
}

Important: Do not change storage-driver „just like that“ on a host with production local layers without planning for a rebuild. And: after a relocation, check SELinux/AppArmor contexts and mount options, otherwise subsequent errors can occur (e.g., Permission Denied on label-based systems).

Best practices for stable operation (so you don’t end up back in an incident)

1) Capacity is not just GB: inodes, metadata, IOPS

Plan monitoring across at least three axes: used storage, inode utilization, and I/O latency. Build hosts with many layers in particular need inode reserves.

2) Place and separate volumes deliberately

Data-intensive workloads (databases, artifact repositories, logs) should not reside on the container writable layer. Use volumes or bind mounts on filesystems designed for that purpose. That reduces copy-up and makes performance more predictable.

3) Define a cleanup strategy instead of „manual prune“

Regular maintenance is better than ad-hoc deletions during an incident. Define rules for how long build caches, dangling images and old containers are allowed to remain on the host. In CI environments, dedicated runner hosts and a controlled cache approach belong to the operational routine.

4) Treat kernel/filesystem changes like deployments

A kernel update can change OverlayFS behavior; a filesystem migration can affect d_type/options. Test such changes on a staging host with representative builds and I/O workloads before rolling them out widely.

Conclusion: Storage problems are rarely „mystical“, but often multi-layered

OverlayFS, device-mapper and loopback may appear as internal details in daily operations — in an incident they decide stability, performance and the speed of your recovery. Clean Container-Storage-Troubleshooting begins with a clear baseline (Driver, Filesystem, DockerRootDir), separates layer issues from volume issues and consistently checks Inodes, thin-pool metadata and I/O latency. If loopback or an unsuitable filesystem is the cause, a planned migration is usually more cost-effective than continuous ‚cleanup‘ under pressure.

If you want to standardize this topic in your environment, create a short runbook with the check sequence, thresholds and a tested fallback path – then the next Storage-Incident becomes a manageable maintenance event.

For this topic, Overlayfs Docker and Device-Mapper Docker are also important. This article contextualizes these aspects and shows what matters in day-to-day operations.