Repairing a Ceph OSD Failure: Data Rebalancing, Backfilling Issues, and the OSD Replace Process

Repairing a Ceph OSD failure is a routine task in distributed object storage environments, but it carries risks: incorrect tuning can degrade production I/O, and removing OSDs prematurely can lead to unsized/lost PGs. This article explains step by step how to safely detect an OSD failure, address backfill/rebalance issues, responsibly adjust recovery parameters and perform an OSD replace both manually with ceph-volume and orchestrated with cephadm. The target audience is administrators, system engineers and storage operators.

Why focusing on rebalance and backfill matters

OSD stands for Object Storage Daemon; it represents a disk and the associated daemon that manages objects in Placement Groups (PGs). If an OSD fails, Ceph attempts to restore missing replicas via backfill and rebalancing. Backfill (copying data from existing replicas) and rebalance (redistributing object placements) generate significant network and disk I/O. Without targeted management, this can increase latency or substantially slow recovery — and thus prolong the time the cluster is not in an ideal state.

Initial steps: quick assessment

Start with a „first-look“ check to determine scope and urgency. That helps you decide whether you need parameter tuning, temporary flags or an immediate OSD replace.

# Gesamtstatus, betroffene PGs, Recovery-Zeit
ceph -s
# Topologie und welcher Host betroffen ist
ceph osd tree
# Kapazitätsverteilung und freie Kapazität
ceph osd df

Important: Record the number of degraded/recovering PGs, recover throughput and whether MON messages mark existing OSDs as out/down. These values determine the urgency.

Repairing Ceph OSD failure: priorities and decision tree

When deciding how to act, distinguish three priorities: 1) preserve application data access, 2) restore cluster health, 3) sustainably clean up/replace the faulty OSD. A quick mental diagram:

• If SMART‑errors or repeated short outages occur → replace the OSD.
• If network/host reachability is the cause → host diagnosis, possibly noout for planned maintenance.
• If backfill is extremely slow → root‑cause analysis (network, I/O, parameters) and temporary tuning with a rollback plan.

Essential checks and metrics

Monitoring is critical. Essential metrics:

• PG status: active+clean, recovering, degraded — shows the direct state.
• Recover bytes / recover throughput: how many bytes/sec are being transferred.
• OSD latencies (read/write): a sustained increase indicates hotspots or saturation.
• Network metrics: bandwidth, packet loss, MTU mismatch.
• Host level: iostat, CPU, memory, SMART for disk errors.

# Live Überwachung
ceph -w
# PG-Übersicht
ceph pg dump pgs_brief
# Beispiel iostat
iostat -x 1 5

Logs and typical error messages

OSD and MON logs reveal causes. Common messages:

• „heartbeat from osd.X timed out“ → network/host reachability; check MTU, switch logs, IPMI.
• SMART‑errors or I/O‑errors → physical disk failure.
• „slow request“ → OSD/device performance, often the cause of slow backfill.

# OSD-Logs prüfen
journalctl -u ceph-osd@.service -n 200
# Alternativ Ceph-Logs
grep -i "heartbeat" /var/log/ceph/ceph-osd.*.log

Typical causes of very slow backfill

Slow backfill is usually not just a parameter issue. Common causes:

• Network bottlenecks or packet loss (e.g., MTU mismatch, QoS, Spanning Tree events).
• Individual slow OSDs (hardware degradation, IOPS limit on RAID controller).
• Cluster hotspots: some OSDs receive significantly more data than others (CRUSH imbalance, misconfigured CRUSH rules).
• Insufficient free capacity on target OSDs: rebalancing requires free space.

Step-by-step troubleshooting for stalled backfills

1. Check network: iperf3 tests between affected hosts.

# Auf Host B (Server, Empfänger)
iperf3 -s
# Auf Host A (Client, Sender)
iperf3 -c  -t 30

2. Check and compare MTU on all OSD interfaces.

ip link show eth0
ethtool -k eth0

3. Check device health and performance (SMART, iostat).

smartctl -a /dev/sdX
iostat -x 1 10

4. Detect hotspots: evaluate OSD DF and OSD latency metrics.

ceph osd df tree
# Falls Prometheus vorhanden, prüfen: ceph_osd_op_latency_seconds

Temporary tuning: adjust responsibly

Changes to recovery parameters can speed up recovery but also impact production I/O. Save the current values beforehand and increase them gradually while monitoring.

# Werte sichern
ceph config get global osd_max_backfills > /tmp/osd_max_backfills.before || true
ceph config get global osd_recovery_max_active > /tmp/osd_recovery_max_active.before || true
# Beispielhafte, konservative Erhöhung
ceph config set global osd_max_backfills 4
ceph config set global osd_recovery_max_active 8

Why it works: osd_max_backfills controls how many backfill threads run concurrently per OSD; osd_recovery_max_active limits parallel recovery operations. Increasing them allows more parallel data movement but requires significantly more network and disk headroom. If individual OSDs remain slow or network bandwidth is constrained, increased recovery only worsens latencies.

Use flags wisely: noout, norebalance, nobackfill

Flags can help, but they are risky. noout prevents MONs from automatically removing OSDs from the CRUSH map, for example during temporary maintenance. Use:

# Vor geplanter Wartung
ceph osd set noout
# Nach Abschluss
ceph osd unset noout

Warning: noout can mask real failures. norebalance or nobackfill are similar: useful for short maintenance windows, dangerous if an OSD is actually lost.

Replacing an OSD: workflow and commands (manual with ceph-volume)

If a drive is defective or SMART errors are present, a clean replace is usually the best option. Here is a conservative, documentable procedure for manual replace with ceph-volume (LVM-based OSDs):

1. Take the OSD cleanly out of service (out) and wait until PGs are back to normal.

# Markieren und prüfen
ceph osd out osd.
ceph -s    # beobachten, bis PGs recovering/clean werden

2. Stop the OSD service on the host.

systemctl stop ceph-osd@.service
# Prüfen, dass der Prozess gestoppt ist
systemctl status ceph-osd@.service

3. Zap/remove the old device (for physical replacement).

# Zap löscht LVM/Partition-Header. ACHTUNG: unwiderruflich für das Device
ceph-volume lvm zap /dev/sdX --destroy

Explanation: ceph-volume lvm zap removes Ceph metadata from the device and prepares it for reuse. It can fail if the device is busy; check LVs with lvs.

4. Install the new drive and create the OSD.

# Beispiel: neues OSD auf /dev/sdY anlegen
ceph-volume lvm create --data /dev/sdY

After create the OSD is automatically registered with the MONs and CRUSH rules are applied. Monitor the rebalancing progress.

OSD replacement: with cephadm (orchestrated)

In cephadm-managed clusters many operators prefer the orchestrated variant because cephadm takes over lifecycle management (container image, unit deploy, logging). Typical steps:

1. Prepare: make the new device available on the host.
2. With cephadm add the OSD on the host or replace it selectively.

# Alle verfügbaren Devices erkennen (nur lesen)
ceph orch device ls
# Alle verfügbaren Devices als OSDs nutzen (vorsichtig; meist in Staging testen)
ceph orch apply osd --all-available-devices
# Alternativ: gezielt ein Device zu einem Host hinzufügen (prüfen in Ihrer Ceph-Version)
# ceph orch daemon add osd :  <-- je nach Version und Politik

Note: ceph orch apply osd --all-available-devices is powerful and should only be used in production if you understand the orchestrator policy. Test in staging.

Careful validation after replace

After each replace validate:

• Cluster health is HEALTH_OK or at least there are no longer degraded/recovering PGs.
• ceph osd df tree shows evenly distributed data and no extreme hotspots.
• Spot read accesses to critical pools/objects and checksum comparisons if available.

# Wichtige Prüfungen
ceph -s
ceph health detail
ceph osd df tree
# Beispiel: stichprobenartiger Objekt-Check
rados -p  ls | head
rados -p  get 

Rollback plan and documentation

Any change to recovery parameters or the removal of OSDs must be reversible. Create scripts that restore the original config values and keep timestamps for all actions.

# Rücksetzskript-Beispiel
prev1=$(cat /tmp/osd_max_backfills.before || echo "2")
prev2=$(cat /tmp/osd_recovery_max_active.before || echo "2")
ceph config set global osd_max_backfills "$prev1"
ceph config set global osd_recovery_max_active "$prev2"
ceph osd unset noout || true

Common pitfalls and how to avoid them

Practical mistakes that occur more frequently:

• Changes to recovery parameters without observability: always check dashboards and alerts.
• Deleting an OSD (ceph osd rm) while PGs are still recovering: causes lost/unsized PGs.
• Blindly applying ceph orch apply osd --all-available-devices on heterogeneous hosts: unintended OSD creations.
• Insufficient free capacity before replace: ensure target OSDs have sufficient space.

Pragmatic checklist for the OSD replace

1. Collect: ceph -s, ceph osd tree, ceph osd df, relevant logs.
2. Backup: export current config values (/tmp) and save monitoring screenshots.
3. Decide: Replace or repair (host vs. drive). If unsure, prefer Reweight instead of rm.
4. Execute: set noout for planned maintenance; OSD out; stop; zap; create with ceph-volume or cephadm.
5. Observe: recovery throughput, latencies and network utilization; keep rollback points ready.
6. Validate: HEALTH_OK, ceph osd df tree, sample reads.
7. Document: time, metrics, decisions, rollback action.

Conclusion

Repairing a Ceph OSD failure means: work in a structured way, use monitoring, change parameters conservatively and with documentation, and test replace workflows. Distinguish host from drive problems, prefer reweighting over immediate removal, and use orchestrated tools like cephadm only after staging tests. With clear checks, rollback strategies and automated rollback scripts you reduce the risk of data inconsistencies and shorten recovery times.

Operational notes

Ensure your runbooks are exercised regularly. Test runs in staging increase confidence for live operations. Also define which team roles are informed and act in which order (Operator, NetAdmin, Storage‑Engineer) — this accelerates escalation and reduces errors during critical interventions.

Repairing a Ceph OSD failure: architecture and operational aspects

When addressing Ceph OSD failure, immediate remediation is only one side — architecture decisions and ongoing operations largely determine how quickly and safely recovery proceeds. Critical factors are placement strategy (CRUSH), pool type (replication vs. erasure coding), network topology and capacity headroom.

CRUSH map and topology: Ensure CRUSH rules represent physical boundaries (server, chassis, rack, AZ). Without rack‑aware placement you increase the risk that many PG replicas are affected simultaneously on a rack failure, concentrating backfill load.

Pool design: Replicated pools behave differently during backfill than erasure‑coded pools. EC pools often require temporarily more IO and network bandwidth for reconstruction and are less tolerant of multiple concurrent OSD failures. Plan larger headroom reserves and longer recovery times for EC pools in your SLOs.

Capacity headroom: Never operate Ceph continuously at very high utilization. A recommended free capacity (depending on workload) is at least 10–20% so rebalancing and temporary replicas have space. If target OSDs have little free space, backfill stalls and recovery time increases.

Network architecture: Separate public and cluster traffic physically or via QoS. Cluster interconnect saturation is one of the most common causes of slow backfills. Plan monitoring metrics for packet loss, MTU errors and per‑OSD throughput and trigger alerts before rebalancing begins.

Operational integration notes: Link OSD metadata with your CMDB (host, slot, serial number) and your orchestration tools. Automated workflows for hardware replacement (BMC reboot, ticketing, asset update) reduce errors during manual interventions. Orchestrator actions (cephadm) should be captured in change pipelines and tested in staging.

Monitoring and automation: Define clear alert thresholds (e.g., number of degraded PGs, recovery throughput below expectation, OSD latency increase). Automate preflight checks before a replace and a rollback script that restores all previously saved config values. Practice the procedure regularly in a replica environment — this reduces human error in a live incident.

Quick check before intervention: Verify CRUSH topology, consider pool type, verify free capacity, check network health, confirm OSD→hardware mapping in the CMDB. These architectural and operational aspects significantly shorten recovery times and reduce the risk of data inconsistencies.

Ceph backfill and Ceph rebalancing are also important for this topic. The article contextualizes these aspects clearly and shows what matters in day-to-day operations.