Analyzing I/O drops after storage failover with iostat, blktrace and dmsetup

An ideal storage failover is a brief switchover moment: path gone, alternative path takes over, applications continue to run. In practice, however, after the failover one often sees exactly the opposite: I/O drops after storage failover, significantly higher latencies, “stuttering” jobs, timeouts in databases or virtual machines that feel sluggish. The insidious thing: the actual fault is over, but performance remains degraded – sometimes for minutes, sometimes until the next reboot.

This article presents a field-proven analysis chain for Linux systems with block storage (FC, iSCSI, SAS, NVMe-oF) and Device-Mapper stacks (Multipath, LVM, dm-crypt). The focus is on three tools that complement each other well: iostat for the quick status snapshot, blktrace for block-layer tracing (i.e. tracking individual I/O requests in the kernel) and dmsetup for visibility into the Device Mapper (the layer that provides, for example, Multipath and LVM devices). Added are typical causes, pitfalls, verification steps, tuning options and a fallback strategy that is operable in production.

I/O drops after storage failover: Why failover often “survives” but performance doesn’t

During a failover, rarely only “a cable” changes. Typical side effects that throttle I/O after the switchover:

• Path degradation instead of true redundancy: Multipath may be using only one active path or attached to a suboptimal controller (ALUA problem). ALUA („Asymmetric Logical Unit Access“) describes that in dual-controller arrays not every path is equally good.
• Queue/timeout effects: While a path is gone, I/Os back up. After the switchover the backlog has to be processed. At the same time, timeouts and retries can massively increase latency.
• Write cache behavior: Arrays switch caches, re-synchronize mirrors or change cache policies. An array can run „safely“ but in a more conservative cache mode.
• Device Mapper layer is “alive” but misconfigured: Multipath/DM can be stuck in an unfavorable policy (e.g. wrong path_selector, min_io, rr_min_io_rq).
• Application and filesystem consequences: Journaling, log replay, database recovery, VM storage timeouts – these can generate load spikes that falsely look like “storage is slow”.

The most important operational rule: After a failover, “system back online” is not the same as “system back to normal”. You need a measurement chain that cleanly separates: device/array, paths/multipath, kernel block layer, filesystem and workload.

Prerequisites and safety framework for the analysis

Before you trace deeper, clarify two points: (1) Are you allowed to install tools on the affected host and start kernel traces? (2) Is the system in a state where additional diagnostics will not make things worse? blktrace generates overhead, especially at very high I/O rates. Therefore work as briefly and as targeted as possible and – if possible – first in a maintenance window or on a comparable node.

The following steps typically require packages: sysstat (iostat) as well as blktrace/btt. On many distributions these packages are available in the standard repositories. Also check whether your storage is attached via multipath (Device Mapper) or, for example, used directly as /dev/sdX – this affects where you measure.

Quick status snapshot with iostat: What is actually broken?

iostat quickly gives indications whether you have a latency issue, a saturation issue, or a CPU/scheduler issue. For failover scenarios it is particularly important to distinguish between service time, queue and throughput. In iostat outputs you typically encounter:

• r/s, w/s, rkB/s, wkB/s: IOPS and throughput
• await: average wait time per I/O (including queue + service)
• svctm (depending on version): service time (not always reliable in modern kernels)
• %util: approximation for utilization (interpret with caution on NVMe/multiple queues)

Capture a baseline: especially on the correct devices

A common pitfall: you observe /dev/sdX, but the application uses /dev/mapper/mpathX or an LVM LV on top. Measure both the DM devices and the underlying block devices. Start with an output that shows devices, extended stats and short intervals:

# 1-Sekunden-Intervalle, 30 Samples, inklusive Device-Stats
iostat -dxm 1 30

Interpretation for failover:

• await rises significantly, %util remains moderate: often retries/timeouts, path problems, or queueing in an upstream layer (DM, HBA, network).
• %util near 100% and throughput low: saturation on small I/Os or a path that has become congested (e.g. only a single controller active).
• Only a single device shows outliers: more likely path/device-specific (e.g. a LUN set), not generic.

Typical iostat signatures after failover

In practice you often see after a storage failover a combination of (a) significantly higher await and (b) fluctuating IOPS, while the CPU is quiet. That frequently points to non-deterministic latencies caused by retries, path flip-flops, or an array performing internal re-synchronization. If you see kernel messages about SCSI/iSCSI in the same period (see next section), that is a strong indicator that the latency originates from the I/O stack below the filesystem.

Capture context: logs and kernel state around the failover

Before you go deeper with blktrace, capture the context. Especially during failover, timestamps are critical to correlate I/O spikes with path events.

# Kernel- und Systemlogs im relevanten Zeitraum (Beispiel: letzte 2 Stunden)
journalctl -k --since "2 hours ago" --no-pager
journalctl --since "2 hours ago" --no-pager | tail -n 300

Watch for indications such as: link down, target reset, abort task, timed out, recovered error. With iSCSI you will often find messages about session reconnects; with FC more often HBA- or SCSI-layer resets. These messages frequently explain why await increases even though the LUN is „present“.

Examining the Device Mapper with dmsetup: What is Multipath really doing?

If you use Multipath, dmsetup is a precise tool to understand the Device Mapper perspective. Device Mapper is a kernel layer that composes virtual block devices from other block devices — Multipath, LV M and dm-crypt are typical consumers.

Start with an overview to see names and dependencies:

# Übersicht über DM-Devices und Abhängigkeiten
lsblk -o NAME,KNAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS

dmsetup ls --tree

The important question is: which DM device is the bottleneck? An LV (LVM) can appear healthy while Multipath beneath it is currently using only a single path or constantly switching.

dmsetup status and table: policy, paths, error states

With status and table you see how the kernel currently operates the device:

# Beispiel: Status und Tabelle eines Multipath-Devices
dmsetup status /dev/mapper/mpathX
dmsetup table  /dev/mapper/mpathX

What to look for:

• path groups: how many path groups exist, which one is active?
• failed/faulty paths: Are there paths marked as faulty that are nevertheless flapping?
• Policy: round-robin, queue-length, service-time — depending on the array one policy may be more appropriate than another.

If you also use multipath-tools, multipath -ll often provides the most readable view including ALUA status. That is not dmsetup, but in practice the quickest complement:

multipath -ll

Common failure patterns in Multipath after failover

• Only a single active path remaining: throughput halved or worse, plus higher latency under peaks.
• ALUA misinterpreted: host uses “non-optimized” paths (suboptimal controller), which degrades latency and throughput.
• Queueing on path-down: in certain configurations I/Os are queued when all paths are briefly lost. That protects against errors, but upon return can produce a long backlog of outstanding operations.
• Flush/failback: overly aggressive failback can cause the host to continually switch back while the array is not yet stable.

Bridging symptom and cause: using blktrace correctly

If iostat and dmsetup show that there is an I/O problem, blktrace often shows where the time is lost. blktrace hooks into the block layer and records events (Queue, Dispatch, Completion). This lets you distinguish: is I/O queuing in the kernel, or is it „outside“ (storage/transport)?

Important: trace the correct device. With Device Mapper it can make sense to trace both the DM device and the underlying physical device. In many environments it is sufficient to target the DM multipath device, because queueing becomes visible there.

Short, targeted trace during the problem

Start with a short trace (e.g. 30–60 seconds) to keep overhead low:

# 60-second trace on a device (example: dm device or nvme0n1)
# -d: device, -w: duration, -o: output directory
mkdir -p /var/tmp/blktrace-iofailover
blktrace -d /dev/mapper/mpathX -w 60 -o /var/tmp/blktrace-iofailover/trace

The raw data is hard to read. Then use btt (Block Trace Times) to analyze waits and distributions:

# Analysis of the trace files
btt -i /var/tmp/blktrace-iofailover/trace.* -o /var/tmp/blktrace-iofailover/report

What you typically want to extract:

• Queue-to-Dispatch: Time requests wait in the kernel before being sent to the device. High values indicate kernel/DM queueing or scheduler effects.
• Dispatch-to-Complete: Time „in transit“ to the storage and back. High values point to storage/transport latency, path problems, or retries.

When blktrace fails or misleads

There are limits: under extremely high I/O load the trace itself can become too large or distort timing. Also, blktrace only shows what happens in the block layer — not the precise cause in the SAN/iSCSI network or the array. If Dispatch-to-Complete explodes, you need to proceed with HBA/NIC and array tools (e.g. switch ports, iSCSI RTT, FC counters, array front-end ports).

Investigation sequence as a runbook: From „symptom“ to „root cause“

In hosting operations a fixed sequence helps ensure you don’t miss anything under pressure. The following sequence is structured so that you start with a low level of intervention and only then move to traces.

Step 1: Narrow down the affected layer

• Is only one host affected or multiple? Multiple hosts point to array/transport, a single host is more likely HBA/NIC/multipath configuration.
• Is only one LUN set affected or all LUNs? A single set can indicate a tier/pool issue or an incorrect LUN mapping.
• Is the problem read- or write-heavy? Write latencies after failover can increase more due to cache changes or rebuilds.

Step 2: iostat on DM and physical level

# Parallel-Ansatz: iostat laufen lassen und Zeitpunkt notieren
iostat -dxm 1

Record the conspicuous devices (dm-*, mpath*, sdX, nvme*), await spikes and %util behavior. This will later serve as the anchor for log correlation and traces.

Schritt 3: Pfadstatus und DM-Stack prüfen

dmsetup ls --tree
multipath -ll 2>/dev/null || true
dmsetup status /dev/mapper/mpathX

If you see here that only one path is active or paths appear as „faulty“, you usually already have the main focus: stabilize the transport, check ALUA/failback, adjust path-checkers/timeouts.

Schritt 4: Kernel-Logs auf Resets/Timeouts

journalctl -k --since "30 min ago" --no-pager | egrep -i "scsi|iscsi|nvme|timeout|reset|abort|multipath|blk_update_request"

Resets and timeouts often directly explain latency spikes. It’s important whether errors persist (recurring) or were only present in the failover window.

Schritt 5: blktrace kurz und gezielt

mkdir -p /var/tmp/blktrace-iofailover
blktrace -d /dev/mapper/mpathX -w 30 -o /var/tmp/blktrace-iofailover/trace
btt -i /var/tmp/blktrace-iofailover/trace.* -o /var/tmp/blktrace-iofailover/report

If the majority of time is in dispatch-to-complete, escalate toward Storage/Network/SAN. If it’s in queue-to-dispatch, look more closely at DM queueing, the scheduler, queue parameters and the interaction with the workload.

Typische Ursachen nach Failover – und was Sie konkret prüfen

1) ALUA und „falscher Controller“: Optimized vs. Non-Optimized

With dual-controller arrays a path can „work“ but not be the preferred one. Then you observe latency plus reduced throughput without clear errors. Check with multipath -ll whether paths are marked as active/optimized. If not: check host ALUA configuration, array path assignments (Target-Ports), and, where applicable, failback settings. Risk: too aggressive „failback immediate“ can lead to a ping-pong after failover.

2) Timeout- und Retry-Kaskaden im SCSI/iSCSI-Stack

Failover often means short periods without responses, then responses return. If timeouts and retries are too long, every I/O request gets „carried along.“ You see this as high await values, often with burst characteristics. Check kernel logs and iSCSI session stability (for iSCSI also network latencies, drops, MTU/Path-MTU). Countermeasures depend heavily on the stack: timeouts must match array failover timings without causing the application to run into errors.

3) Multipath Queueing: Schutz vor Errors, aber teuer in der Nachwirkung

Multipath can buffer I/Os with „queue_if_no_path“ or similar when all paths are gone. That prevents I/O errors in applications, but upon return can create a very long queue. iostat then shows high await values for a long time, even though the path is back. Here the operational decision is decisive: is it acceptable to have short errors (and application retry), or is queueing mandatory because otherwise files/VMs could become corrupt? There is no universal answer — but you should make a deliberate decision and document it.

4) Scheduler- und Queue-Parameter nach Device-Wechsel

After a failover the visible device can change (e.g. a different HBA path) or parameters may behave differently. With NVMe and modern SCSI devices the classic I/O scheduler is less dominant, but queue settings, nr_requests and device-queue limits can still be limiting. Check whether block-queue parameters differ between “normal” and “degraded”. Pitfall: udev rules or tuning profiles are applied only at boot, not on the path event.

5) Filesystem and application: “catch-up” after I/O pause

If writes were briefly blocked during the failover, applications will catch up afterwards: journals are processed, caches are filled, database logs are flushed. That looks like “slow storage”, but is often simply a load spike. Differentiate using blktrace (queue vs. device latency) and application metrics (e.g. DB checkpoint times). Also check whether the host is experiencing memory pressure or CPU steal (virtualization) — these can indirectly throttle I/O.

Practical pitfalls

• Tracing the wrong device: Tracing /dev/sdX while /dev/mapper/mpathX is used (or vice versa). Result: apparently “nothing notable”.
• Measurements without time correlation: Without exact timestamps (failover event, log messages, iostat intervals) cause and effect are quickly confused.
• Overinterpreting single outliers: Short peaks after a failover are normal. What matters are persistent degradation or recurring peaks.
• Making changes under load: Changing multipath policy or timeout tuning in the middle of an incident can improve or worsen the situation. Plan a rollback option.

Implementation: stabilize first, then optimize

When you have narrowed down the cause, prioritize in this order:

1. Restore stability: Paths stable, no flapping, no recurring resets/timeouts.
2. Use correct paths: ALUA/controller mapping correct, appropriate failback strategy.
3. Decide on queueing consciously: Queueing protects against errors but can extend the RTO (Recovery Time Objective).
4. Performance tuning: Only when stable, work on queue depth, scheduler and policies.

Document the decisions made in the runbook: Which multipath.conf parameters are set? Which failover times are realistic on the storage side? Which applications tolerate short I/O errors and which do not?

Rollback strategy: revert changes safely

Especially for multipath and timeout changes you need a clean rollback strategy. Practically this means:

• Version configurations: multipath.conf, udev rules, sysctl/kernel parameters in Git or a configuration management system.
• Define the rollback step: How do you return to the last known state? Who is authorized to approve it?
• Schedule a maintenance window: Some changes require a service restart or only take effect after a re-scan. Plan this before making on-the-fly changes.
• Follow-up checks: iostat baseline, path status, logs for new errors, and a short blktrace spot check if the symptoms were previously clear.

Best practices for operations: so the next failover is not a performance incident

• Test and measure failovers: Not just “works/doesn’t work”, but capture latency/IOPS before, during and after failover.
• Path-level monitoring: Not just LUN reachability, but the number of active paths, ALUA state, recurring resets.
• Runbook with clear measurement points: iostat patterns, dmsetup/multipath outputs, blktrace window, log filters.
• Align application-side timeouts: database, VM and filesystem timeouts must match the storage failover reality.

Conclusion

I/O drops after storage failover are rarely „just bad luck“; they are usually the result of the interaction between path state, device-mapper behavior, timeouts/retries and the load that accumulates after the switchover. With a disciplined diagnostic chain of iostat (symptom and scope), dmsetup (stack and paths) and blktrace (where time is lost) you obtain robust conclusions quickly, instead of guessing in the fog.

If you document the results as a Runbook and evaluate failover tests not only functionally but also with regard to performance, you reduce the risk that a successful failover nevertheless becomes an operational incident.

For this topic, storage-failover troubleshooting and iostat analysis are also important. The article places these aspects in context and shows what matters in daily operations.