Setting up Ceph with Proxmox: MON/OSD/MGR layout, placement groups and performance troubleshooting

Ceph with Proxmox is a common combination in virtualized infrastructures. In this post I provide a practical description of how to plan MON/OSD/MGR, correctly size Placement‑Groups (PGs) and systematically diagnose and resolve performance issues. The focus keyword Ceph with Proxmox is intentionally placed early: the integration affects operation, monitoring, migration and SLA compliance.

Why architecture planning is important for Ceph

Before you integrate Ceph into Proxmox, you should understand the responsibilities of the components. In short: MONs (monitors) manage cluster quorum and configuration; OSDs (Object Storage Daemon) store data and perform replication and scrubbing; MGRs (managers) provide the dashboard, API and metrics. This separation of responsibilities determines hardware, network and operational decisions — and thus performance and fault tolerance.

Basic rules for MON/OSD/MGR layout

The distribution of MONs, OSDs and MGRs affects stability and maintainability. Basic rules:

• MONs: Always an odd number (at least 3) for quorum resilience; MONs are lightweight but require stable network access.
• OSDs: Distribute OSDs across hosts and failure domains (host, rack); OSDs are I/O‑critical and constitute the system bottleneck.
• MGRs: At least two manager instances for redundancy of the dashboard and exporters; the failure of an MGR primarily affects observability, not consistency.

Recommendation for typical clusters

For small to medium Proxmox installations (6–20 nodes) a possible configuration is: 3 MONs on separate hosts, 2 MGRs distributed, and OSDs distributed so that at least three OSDs exist per failure domain. Avoid co‑locating MONs/MGRs with all OSDs of a rack to reduce the blast radius.

Ceph with Proxmox: realistic performance‑tuning measures

Performance tuning begins with measurement. Ceph bottlenecks can only be understood with metrics, I/O profiles and controlled tests. Typical areas are network, OSD device, CPU/memory on the OSDs and the PG configuration (placement groups, logical distribution groups).

Important Ceph configurations and their impact

• osd_memory_target (Ceph): Controls the RAM footprint of the OSD cache structures — too low → performance drop, too high → memory pressure. Values depend on OSD RAM and number of PGs.
• osd_recovery_max_active / osd_recovery_op_priority: Throttle parallel recovery tasks, reduce short‑term backfill load, but extend recovery time.
• osd_max_backfills: Limits parallel backfills per OSD; useful in cases of network saturation.

Recommendations (conservative)

Start with conservative defaults and adjust gradually. Example values you should test:

# Beispiel: vorsichtigere Recovery‑Limits setzen
ceph config set osd osd_recovery_max_active 2
ceph config set osd osd_max_backfills 1
# Memory‑Target nach OSD‑RAM anpassen (Beispiel 8GB RAM)
ceph config set osd osd_memory_target 5368709120 # 5GB

Why: These settings reduce concurrent I/O pressure on network and CPU, prevent short‑term latency spikes and are reversible. Test and document every change.

Placement‑Groups (PGs): concept, calculation and pg_autoscaler

Placement‑Groups aggregate objects and are the atomic unit that CRUSH distributes to OSDs. The number of PGs influences data distribution, rebalancing size and OSD memory footprint.

Calculation approach and practical example

As a rule of thumb, 50–200 PGs per OSD; a simplified formula:

target_pgs = (osd_count * pgs_per_osd) / replication_size

Example: 12 OSDs, repl_size=3, pgs_per_osd=100 → target_pgs = (12*100)/3 = 400 → choose 256 or 512 (a power of 2 is often recommended). Modern Ceph‑versions provide pg_autoscaler, which adjusts PG counts incrementally — check compatibility with your Proxmox version.

pg_autoscaler: Advantages and disadvantages

pg_autoscaler automates adjustments and reduces human error. Drawbacks: in older versions it can trigger unwanted rebalancing, so enable it only after you have verified the version and its behavior.

CRUSH‑Map, Failure Domains and Device Classes

The CRUSH map determines physical placement. Define failure domains (e.g. host, rack) and device classes (ssd/hdd) clearly so that policies (e.g. separate placement for an SSD tier) take effect. An incorrect CRUSH map leads to uneven distribution and increased data traffic during failures.

Scrubbing, Backfilling and Recovery: differences and operational rules

Scrubbing (regular integrity checks) is a scheduled activity; Deep‑Scrub inspects contents more intensively. Backfilling describes copying data after adding/removing OSDs. Recovery is the synchronized restoration after failures. These processes consume I/O and network resources differently — plan maintenance windows and set limits on parallel tasks.

Performance‑Troubleshooting: Systematic incident runbook

In a performance incident follow a structured approach: measure first, then intervene; never apply global configuration changes blindly.

Immediate actions (first 10 minutes)

1. Check cluster overview:

ceph -s
ceph osd df tree
ceph pg stat
ceph osd perf

2. Check network data and switch statistics (MTU, Errors, Drops).
3. Check OSD I/O: iostat, iotop, blktrace.
4. Observe clients (Proxmox VM‑IO via iostat inside the VMs).

Common interventions and commands

If backfill is the cause, throttle recovery/backfill:

# Temporär OSD Noout (Achtung: nur bei geplanter Wartung)
ceph osd set noout
# Drosseln
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_max_backfills 1
# Überwachen
ceph -w

If OSD CPU/memory is the bottleneck, check the PG count and osd_memory_target. Reduce PGs only incrementally and monitor OSD stability.

Concrete runbook procedure for critical latency spikes

1. Identify: ceph osd perf, ceph pg dump | grep stuck/degraded.
2. Isolate: temporarily reduce clients (QoS on Proxmox), set noout if performing maintenance.
3. Throttle: set recovery/backfill values as above.
4. Investigate root cause: switch counters, ethtool -S, iostat, dmesg for device errors.
5. Stepwise rollback: when load decreases, revert the values and monitor.

Testing and validation: fio as a tool

Validate storage profiles with fio (I/O pattern simulation). An example for a 4k random R/W mix:

fio --name=randrw --ioengine=libaio --direct=1 --rw=randrw --rwmixread=70 
    --bs=4k --numjobs=8 --size=1G --runtime=300 --group_reporting

Run such tests either isolated on a test VM or directly on OSD devices; interpret IOPS and latency in the context of expected workloads of the Proxmox VMs.

OSD Replacement: Safe procedure and rollback

When replacing an OSD document the status in advance, set the OSD out, monitor recovery until stable and create backups of the CRUSH map and MON state. Example commands:

# Remove OSD from the cluster
ceph osd out 
# Monitor
ceph -w
# Recreate after replacement (example ceph-volume)
sudo ceph-volume lvm create --data /dev/sdb

Rollback strategy: If the replace fails, you can temporarily reattach the old device and re-attach the OSD or adjust the weight-scale to prevent uncontrolled rebalancing.

Metrics, alerts and observability

MGR modules provide Prometheus metrics. Priority alerts should include: ceph_health != OK, high recovery/backfill rates, OSD latency increase, many PGs in STATE ‚active+undersized‘ or ‚degraded‘. Dashboards help to observe trend behavior and not only isolated events.

Practical operational rules and common pitfalls

• Enable MTU/Jumbo frames only after an end-to-end test; inconsistent MTUs cause fragmentation and drops.
• Avoid RAID mode on storage controllers for OSDs; HBA/pass-through reduces unexpected latencies.
• Make changes to PG counts or the CRUSH map in stages and accompany them with monitoring sprints.
• Schedule scrub and deep-scrub windows during low I/O periods.

Checklist before production deployment

1. Proxmox and Ceph version compatibility verified.
2. Public/cluster network configured, MTU checked.
3. MON/MGR count and distribution set (at least 3 MONs, 2 MGRs).
4. OSD sizing: device classes, WAL/DB devices planned.
5. PG planning: target PGs calculated and validated in a test environment.
6. Backups: MON/MGR/CRUSH map backup workflow established.
7. Monitoring: MGR modules, Prometheus exporter and alerting configured.
8. Rollback plans documented and tested (OSD replace, MON RESTore).

Conclusion: operational discipline over magical settings

Ceph with Proxmox offers high scalability and good integration capabilities into virtualized environments — provided architecture, network and PG planning are aligned. Start conservatively, measure effects and optimize iteratively. In case of performance issues a structured layer check helps (Hardware → Kernel → Network → Ceph → Client) and targeted throttling of recovery/backfill. Documentation, monitoring and tested rollback strategies reduce operational risks and make the system predictable.

Use the checklist as an operational document and add internal links to deeper how-tos (e.g. Proxmox upgrades, ZFS design) to create a complete operations manual.

Operational and integration aspects that are often overlooked

In Ceph deployments on Proxmox not only the pure cluster configuration determines success, but also how you integrate storage into operational processes, backups and change management. Below are practical aspects that often lead to later surprises in projects, along with concrete recommendations.

Backup and snapshot strategy for VMs

RBD‑Snapshots are fast, but do not replace a systematic backup. Use snapshots for short‑term consistency points (e.g., before patches), but export regular offsite backups (e.g., export the VM data or use an object backend such as RGW/S3). Be aware that snapshots themselves generate I/O load and that many snapshots can lead to performance degradation. Therefore plan a deletion interval and automation for snapshot cleanup.

Coordination with Proxmox features

Ensure that Proxmox Storage Pools (RBD) are correctly tagged and that you use admission controls for VM placement to avoid hot‑OSD hosts. Define rules that prevent too many I/O‑heavy VMs from converging on OSD‑rich hosts — this reduces hot‑spot risks.

Security and compliance layer

Enable transport encryption between Ceph services (mTLS) and consider encrypting devices (dmcrypt) for sensitive data. Pay attention to key management: keys should not only reside locally but be referenced in your PKI/external secret management (e.g., HashiCorp Vault). Document the decryption and recovery process so operations staff can navigate it in emergencies.

Upgrade and rollback playbook

A formal upgrade playbook reduces risk. Core principles: rolling upgrades, health checks after each stage, conservative timeouts and a defined rollback procedure. Test the upgrade in a replication environment and keep backups of MON and MGR state data available.

# Beispiel: Health‑Check vor Upgrade (Kurzversion)
ceph status --format=json | jq '.health'
ceph osd tree --format=json | jq '.nodes | length'

Automation and orchestration

Use orchestration tools (cephadm, Ansible) for repeatable deployments and documented configurations. Store Ceph configuration changes in a Git repository and link them to your change control system. Automated tests (smoke checks after configuration changes) prevent unintended side effects in production pools.

Multi‑site, replication and DR scenarios

If geo‑redundant requirements exist, plan for RGW‑bucket replication or Ceph‑RADOS mirroring. Such setups drastically change latency and bandwidth requirements; test replication resilience under simulated network failures and quantify the necessary bandwidth SLAs for recovery windows.

Observability: sensible thresholds and alerts

An alert is only as good as its action. Define responsible parties and automated runbooks for alerts such as rising OSD latency, increased backfill rate or PG degradation. Examples of pragmatic thresholds:

• latency p95 > X ms über Y Minuten → Ticket + Ausrollen drosselnder Recovery‑Parameter
• mehrere OSDs offline gleichzeitig → sofortige Eskalation
• PGs in degraded/stuck > 0 für Z Minuten → Wartungsmodus prüfen

Organizational recommendation

Create a small operations playbook (2–3 pages) with checklists for: daily health checks, procedure for OSD replacement, upgrade sequence and rollback steps. Combine this playbook with technical automations and regular chaos tests (controlled failures) — this ensures your teams gain familiarity with recovery procedures and that the technology is stably integrated into production operations.

For this topic, Mon Osd Mgr and Ceph Performance are also important. The article clearly contextualizes these aspects and shows what matters in daily operations.