Ceph does not perform like the datasheet: it performs like the benchmark the OSD ran at boot

The same conversation, over and over: someone swaps the drives in their cluster for faster ones and performance barely moves. Then the bottleneck hunt begins—the network, the CPU, the firmware, the kernel—and almost nobody looks at the place where it is actually decided: the operation scheduler inside every OSD, and the number that scheduler believes your drive can deliver.

We run Proxmox VE with Ceph in production across several datacenters, so we say this with the affection of people who have lived through it: in Ceph, the performance you see is not signed off by the drive vendor. It is signed off by an internal allocation that configured itself, using a measurement your OSD took the day it booted and that nobody has looked at since. The numbers that follow are not ours: they are the ones in Ceph's documentation and the ones you will find in your own cluster if you look. What is ours is the order we look at them in, and what we do about them.

The scheduler that arrived in Quincy and almost nobody configured

Inside an OSD, four classes of work compete: client operations (your VMs), recovery and backfill, scrubbing, and housekeeping. Up to Ceph Pacific that was arbitrated by a weighted priority queue (wpq). With Quincy, the default scheduler for BlueStore OSDs became mclock_scheduler, and the sharing model changed with it: instead of relative priorities, mClock gives each class a reservation (guaranteed minimum), a weight and a limit over the OSD total capacity.

The profile active by default is called balanced —it has been since 17.2.7 and 18.2.0; earlier Quincy releases shipped high_client_ops— and it does exactly what it says: it reserves 50% for client operations and 50% for recovery (plus 5% for everything else). The other two built-in profiles shift that balance: high_client_ops leaves 60/40 in the client's favour and high_recovery_ops flips it to 30/70 so a degraded cluster heals sooner.

So far, reasonable. The problem is not the split: it is what the split is calculated on.

315 IOPS: the number that governs your cluster and you did not set

To share out percentages, mClock needs to know what 100% is. That ceiling lives in osd_mclock_max_capacity_iops_hdd and osd_mclock_max_capacity_iops_ssd, and the OSD itself determines it by running a benchmark the first time it boots; the result is stored in the monitors' central config and reused on later boots. If that measurement never happened or was thrown away, the documented defaults stand:

And here is the part that catches people out. The boot measurement is not accepted blindly: if the result falls outside the range Ceph considers credible, it discards it and falls back to the default value, leaving a warning in the OSD log. At the top, osd_mclock_iops_capacity_threshold_hdd = 500 and osd_mclock_iops_capacity_threshold_ssd = 80,000. At the bottom it works the same way, and that is the half nobody mentions: osd_mclock_iops_capacity_low_threshold_hdd = 50 and osd_mclock_iops_capacity_low_threshold_ssd = 1,000. A measurement that is too good and one that is too bad end up in the same place: the previous valid capacity or, failing that, the factory value.

How to find out what your cluster believes (five minutes)

This needs no maintenance window and takes nothing offline. You just look:

# Which scheduler is active?
ceph config show-with-defaults osd.0 | grep op_queue

# What IOPS ceiling does each OSD believe?
ceph config show-with-defaults osd.0 | grep mclock_max_capacity

# Actually measure that OSD (4 KiB random, as the docs prescribe) and set the value
ceph tell osd.0 cache drop
ceph tell osd.0 bench 12288000 4096 4194304 100
ceph config set osd.0 osd_mclock_max_capacity_iops_ssd <valor>

If you would rather have the OSD measure itself again, there is osd_mclock_force_run_benchmark_on_init (default false), meant exactly for when the underlying hardware has changed: you enable it temporarily, restart the OSD so it refreshes its capacity, then remove it. Two caveats: the benchmark writes, so it is not something to fire at peak hours across every OSD at once; and the measurement is a snapshot taken by a machine that, at that moment, is not serving your VMs.

The parameters you have been tuning for years no longer do anything

This is the second surprise, and the biggest time-waster. With a built-in mClock profile active, the Ceph documentation is explicit: these options are overridden with mClock values.

With two caveats the documentation itself provides, worth stating so as not to oversell the point. First: those limits can be changed, but only if you first enable osd_mclock_override_recovery_settings (default false); if you do not and change them anyway, Ceph reverts your value to its own and logs a warning in the cluster log. It is not that your setting is ignored: it is undone, and it is on the record. Second: the values it reverts to match the historical ones —osd_max_backfills 1, osd_recovery_max_active_hdd 3, _ssd 10— so on most clusters the net effect is the same as never having touched anything. What you lose is the illusion of being in control of it.

The Proxmox wiki puts it bluntly: the old methods for controlling how much performance goes to backfill and recovery are ignored by mClock. So if someone copied a generous osd_recovery_sleep_hdd out of a 2019 guide to stop the cluster choking during a rebuild, that setting is doing precisely nothing today. And the same wiki flags a detail that breaks old recipes: up to 17.2.6 the res and lim values of the custom profile were expressed in IOPS; from 17.2.7 and 18.2.0 onwards they are a fraction of the OSD IOPS capacity, from 0.0 to 1.0. The same line copied from a forum means two different things depending on your version.

Scrubbing has no time window by default

Ceph verifies its own data, and that is a good thing: it is one of the reasons we trust it. But it pays to know when. By default, light scrubbing becomes eligible after one day (osd_scrub_min_interval) and deep scrubbing—which reads the data, not just metadata—every seven days (osd_deep_scrub_interval). Each OSD allows up to three concurrent scrubs (osd_max_scrubs). And the time window, osd_scrub_begin_hour and osd_scrub_end_hour, ships as 0 and 0: no restriction, any hour of the day.

On an all-NVMe cluster nobody notices. On a spinning-disk one with payroll closing on a Tuesday morning, everybody does. The sensible move is not to switch scrubbing off—people who do that discover the corruption on restore day—but to give it a schedule and accept that verification is part of the cost of healthy data. And here is a lever that does work: mClock has forced osd_scrub_sleep to zero, but the begin and end hours, their per-weekday equivalents and osd_max_scrubs are still yours.

The network is part of the drive too

A client write to a replicated pool does not finish when the primary disk has stored it: it finishes when every replica has. The latency the VM sees is therefore set by the slowest OSD in the group, not by the cluster average. A single sick drive—or one with a badly set capacity—drags everyone down.

Which is why the network recommendation in Ceph's own documentation is not a switch vendor's wish list. Their figures: replicating 1 TiB over a 1 Gb/s network takes three hours; 10 TiB, thirty hours; that same TiB at 10 Gb/s drops to twenty minutes. The recommendation is to provision at least 10 Gb/s, and it warns that dense or NVMe nodes can saturate 10 GE or even 25 GE; for substantial workloads it suggests 25 Gb/s, and 100 Gb/s links when nodes are dense. Those thirty hours are not a catalogue figure: they are how long your cluster stays degraded after losing a node, and all that time the balanced profile is reserving half of every OSD for recovery.

How long that rebuild takes also depends on the pool scheme: recovering a replica means copying, and recovering an erasure-coded shard means reading several chunks and reconstructing. We did that full calculation in replica 3 versus erasure coding, and the optimisations that change the arithmetic on the latest branch are covered in Fast EC ships switched off.

When the drive really is the problem

It would be dishonest to stop at "it is the scheduler". There are two cases where the hardware is exactly to blame, and Ceph's documentation points at them without diplomacy:

What we look at, in this order

The short version

Ceph shares out work over a capacity it estimated on its own, during one boot, and which may well have stayed at a textbook value. Before signing off on faster drives, it is worth five minutes asking the cluster what it thinks it has. Sometimes the answer is that the hardware is genuinely maxed out and you need to buy. And sometimes the answer is that the new hardware was going to come in through the same narrow door as the old one.

Sources (verified on 13 August 2026 against the latest and reef branches of the documentation): profiles, reservations, IOPS capacity, thresholds and overridden options — mClock Config Reference; scrub and recovery intervals — OSD Config Reference; PLP, DWPD, WAL/DB ratios and network replication times — Hardware Recommendations; replicated write acknowledgement by the primary OSD — Ceph Architecture; the wpq to mclock change in Quincy, ignored settings and res/lim units per version — Proxmox VE Wiki: Ceph mClock Tuning.