In Ceph, capacity is not set by the cluster: it is set by the fullest disk

The cluster stops accepting writes on a Tuesday afternoon. Virtual machines hang, the hypervisor does not crash and no disk has failed. Meanwhile, ceph df still shows dozens of free terabytes. Both things are true at once, and that is precisely the lesson: in Ceph, usable capacity is not the sum of the disks. It is whatever fits in the one that fills up first.

We are writing this while running Proxmox VE with Ceph storage in production, spread across several data centres, so we sell precisely what is about to be described. Worth saying up front, because the most useful part of this article — working out your capacity with one node missing — takes a spreadsheet and two commands, and requires buying nothing from anybody.

The threshold does not belong to the cluster: it belongs to each disk

Ceph has three fullness thresholds, and all three are evaluated per OSD, not against the total. The defaults shipped in the documentation are 0.85 for nearfull, 0.90 for backfillfull and 0.95 for full. Almost nobody changes them, and that is fine: the problem is not the number, it is where it gets measured.

The OSD_FULL health check says it in one sentence, and it is worth reading slowly because the subject is singular and the object is plural: "One or more OSDs have exceeded the full threshold and are preventing the cluster from servicing writes." One or more OSDs have crossed the threshold and are preventing the cluster from servicing writes. Not that this particular disk stops accepting data: that the cluster stops serving writes.

To be precise: what stops are writes to the pools that hold any placement group on that OSD. In a textbook hyperconverged cluster — a single CRUSH rule spreading across every disk on every node — that means everything you have. One disk out of twenty-four holds up the virtual machines of twelve customers. There is a curious detail too, worth flagging because anyone going to the documentation will run into it: the monitor configuration page is harsher than the health-checks page and states that Ceph "prevents you from writing to or reading from OSDs" — writes and reads. Both pages are official and they do not agree; in practice what falls over are the writes, while reads keep answering. What you will see in the Proxmox console is a handful of frozen virtual machines.

Why one disk fills up before the rest

Because CRUSH is not a free-space scheduler. It places placement groups with a deterministic function over weights, not by looking for the disk with the most room, and that is exactly its virtue: any client can work out where an object lives without asking anyone. The price is that the distribution is statistical. With few placement groups per disk the variance shows, the groups do not all weigh the same, and every time somebody swaps a 4 TB disk for an 8 TB one the balance tilts again.

That is what the balancer is for, and it ships enabled: the documentation says "the default mode is upmap" and that in that mode automatic balancing is on. It helps a great deal and should not be turned off. But two things are worth knowing. First, that upmap optimises the placement of groups, not of bytes — and that is not our reading: the documentation has a section of its own, titled "Limitation: count-based balancing vs. size-based balancing", which spells it out: "Ceph's built-in balancer optimizes only by PG shard count, not by the actual size of the data stored in each PG." If your groups are uneven, the balancing only goes halfway. Second, that it comes with a written requirement: "to use upmap, all clients must be Luminous or newer." One old client hanging off that cluster — a backup server nobody has touched, an ancient kernel mounting RBD — is enough to make the option unavailable.

Which yields a reading rule that saves you trouble: the average utilisation of a Ceph cluster is the most useless number on the dashboard. The maximum is what rules. ceph osd df shows it disk by disk, with a VAR column that is each disk's utilisation divided by the cluster average: 1.00 means sitting exactly on the mean. If the dashboard says 68% and there is an OSD at 91%, your cluster is at 91%.

The order of the thresholds is the warning nobody reads

Look at the three numbers again: 0.85, 0.90 and 0.95. The middle one, backfillfull, does not warn about anything. It switches something off. The OSD_BACKFILLFULL check says that one or more OSDs have crossed the threshold — or would cross it if the currently-mapped backfills were to finish — "which will prevent data from rebalancing to this OSD": blocking data from being rebalanced onto that disk. Note the aside, because it decides the rest of this article: Ceph does not wait for the disk to reach 90%, it does the projection and refuses beforehand.

In other words: the mechanism that repairs switches off five points before the one that serves. When the fullest disk crosses 90%, Ceph can no longer move data onto it — which is precisely the tool you fix an imbalance with. When it crosses 95%, everything stops. Between the two, on a 4 TB disk, sit around 200 GB. One medium-sized virtual machine. One night of ordinary growth.

That the order matters is proved by Ceph itself: it has a dedicated check, OSD_OUT_OF_ORDER_FULL, whose only reason to exist is to tell you that "the utilization thresholds for nearfull, backfillfull, full, and/or failsafe_full are not ascending." There is an official warning for the case where somebody leaves the thresholds out of order. Somebody has done it before you, and it hurt.

The calculation almost nobody runs: capacity with one node missing

The Ceph documentation frames the example with a 33-node cluster, one 3 TB OSD each, 99 TB in total: "with a mon osd full ratio of 0.95, if the Ceph Storage Cluster falls to 5TB of remaining capacity, the cluster will not allow Ceph clients to read and write data." And it adds the part that almost never makes it into the budget: you have to plan for several OSDs failing at once with the cluster still able to return to active+clean.

Translated to the small company running four or five nodes instead of thirty-three, the arithmetic is short, and it is ours here, not the documentation's. If your failure domain is the host and you lose one, once Ceph gives it up for dead it has to recreate on the remaining N−1 nodes every replica that lived on the one that left. The total raw data does not go down: it gets redistributed across fewer disks. For none of them to cross 95% after that rebuild, utilisation before the failure cannot exceed 0.95 × (N−1) / N.

With four nodes, that 71% is the ceiling for nobody to cross 95% after rebuilding. But rebuilding has to be possible, and there the middle threshold rules: if an OSD's projected utilisation goes past 0.90, Ceph refuses to send it data and the placement groups sit in backfill_toofull — halfway there. The number that actually gets you back to active+clean is 0.90 × (N−1) / N: with four nodes, 67%. That is the middle column, and it is the one to look at, because the criterion the documentation asks for is not "that it does not stop", it is that the cluster can recover to active+clean. The third column adds the margin of not even crossing the warning while the rebuild runs. And all of this assumes identical nodes and a perfect spread, which does not exist: take the real maximum from ceph osd df, subtract a few points, and you have the honest number.

The three-node case deserves its own paragraph, because it is the one most often sold and the one that breaks the table. With replica 3 and a host failure domain, if a node goes down there are no longer three distinct places to put three copies: Ceph cannot rebuild, so nothing fills up. It sits degraded and waiting, with min_size 2 letting you carry on working over two copies. Proxmox's warning on this is blunt and points in exactly the opposite direction from the temptation: "do not set a min_size of 1", because allowing I/O with a single replica leads to data loss and incomplete placement groups. On three nodes, then, what bites is not the node: it is the single disk. If one OSD dies while its host is still alive, its placement groups are recreated on the other OSDs of that same host, because the failure domain will not let them leave. With four disks per node that multiplies the utilisation of the remaining three by 4/3 — worse than losing a whole node out of six — and there the calculation is done with one node's disks, not with nodes. We went into it in Proxmox HA does not prevent the outage, it shortens it.

And there is a third factor that shifts the whole table: the protection scheme. With replica 3 you write three times what you store; with erasure coding you write considerably less, but you need more failure domains and you pay for it on small writes. We did that calculation in full in replica 3 versus erasure coding, and it belongs next to this one, because the raw utilisation this whole article talks about comes from there.

What gets done at three in the morning, and what it costs

With the cluster stopped and customer virtual machines frozen, the first thing any search turns up is raising the threshold:

ceph osd set-full-ratio 0.96

It works, and we are not going to pretend we have never typed it. As a manoeuvre it is legitimate: it hands writes back for exactly long enough to delete an old snapshot, move a disk elsewhere or bring up a new OSD. As a state it is a mortgage, and it is worth knowing which ceiling you are signing against: above the monitor's full ratio there is another threshold enforced by the OSD itself, osd_failsafe_full_ratio, which defaults to 0.97. Raising it that far pins one to the other and leaves nothing between the threshold that stops writes and the disk's own hard stop; and above 0.97 the cluster answers with a red OSD_OUT_OF_ORDER_FULL, the same warning from the previous section. So the manoeuvre is 0.96, with the clock running: what goes up in an emergency comes back down the next day, and that is a task with a date on the calendar, not an intention.

The orderly way out is adding capacity, and that is where the problem stops being Ceph's. One more OSD means a free bay; one more node means rack units, contracted power and cooling to shed it — none of which is solved from a console, as we covered in colocation is no longer negotiated in U, it is negotiated in kW. And even with the hardware in place, rebuilding is not free: moving terabytes between disks competes with production for the same I/O queues, which is exactly what the OSD scheduler governs and what we explained in Ceph does not perform like the datasheet. Reclaiming space with the cluster already under strain is the worst possible moment to ask it for throughput.

Four commands and one number on a spreadsheet

ceph osd df
ceph df
ceph balancer status
ceph osd test-reweight-by-utilization

1. ceph osd df. Sort by %USE and keep the maximum, not the average. The VAR column is each disk's utilisation divided by the mean, so 1.00 is the mean: OSDs above 1.15 mean a skewed spread, not a full cluster. To calibrate it, reweight-by-utilization itself goes by default after those exceeding the average by 20%.
2. ceph df. Look at MAX AVAIL per pool, not the global AVAIL, which is the misleading one. The documentation warns that the value is "a complicated function of the replication or the kind of erasure coding used, the CRUSH rule that maps storage to devices, the utilization of those devices, and the configured mon_osd_full_ratio setting": device utilisation and the full ratio are already baked into the number.
3. ceph balancer status. Check it is active and in upmap mode. If it tells you it cannot use it, go hunting for the old client blocking it before anything else.
4. ceph osd test-reweight-by-utilization. The dry run: the documentation describes it as the way to find out how many placement groups and OSDs a reweight-by-utilization would affect before running it. Look at it before touching weights by hand.

The number on the spreadsheet is a single one and it hardly ever changes: the raw utilisation percentage above which you stop surviving the loss of a node. Take it from the table above, subtract the imbalance points ceph osd df shows you, and set it as the alert threshold in your monitoring. The Proxmox manual itself uses the same logic when explaining how to remove an OSD: "make sure that all OSDs have their Used (%) value well below the nearfull_ratio of default 85%." Comfortably below. Not brushing against it.

When none of this should keep you up at night

If your cluster sits at 40%, the disks are all the same size, the balancer is in upmap and you grow by 200 GB a month, you have years ahead of you and no decision to make. Run the numbers one afternoon, set the alert and forget about it. It does not affect you either if growth is flat because what you keep is operational data with a fixed retention: there the problem is not capacity, it is how long it takes you to get it back.

It does affect you, and quite a lot, in three specific situations. When growth comes from virtual machines nobody ever deletes — snapshots taken before an update that went fine eighteen months ago, test clones, the disk of a server somebody powered off a year ago and nobody deleted. When you have grown the cluster with disks of different sizes, which is what happens when you buy every year. And when the node count is small, because that is where the (N−1)/N fraction really bites: the difference between four nodes and ten is fourteen percentage points of disk you thought you had and do not.

Sources

Thresholds and health-check wording (OSD_FULL, OSD_BACKFILLFULL, OSD_NEARFULL, OSD_OUT_OF_ORDER_FULL): official Ceph documentation, "Health checks". Default values 0.85 / 0.90 / 0.95 for mon_osd_nearfull_ratio, mon_osd_backfillfull_ratio and mon_osd_full_ratio, and the 33-node, 99 TB cluster example: "Monitor Config Reference", "Storage Capacity" section. Definition of MAX AVAIL: "Monitoring a Cluster". Balancer default mode and the client version requirement: "Balancer Module". The ceph osd df and reweight-by-utilization commands and its dry-run form: "Control Commands". Proxmox pool defaults (size 3, min_size 2), the warning about min_size 1 and the one about keeping OSDs well below 85% before removing one: Proxmox VE manual, "Deploy Hyper-Converged Ceph Cluster" chapter. Quotations are reproduced in their original language and translated in the body text. The capacity table with one node missing is our own calculation, not a figure published by Ceph: it applies 0.95 × (N−1) / N to the first column, 0.90 × (N−1) / N to the second — the one that decides whether the rebuild finishes at all — and 0.85 × (N−1) / N to the third. It assumes a host failure domain, nodes of equal capacity, a perfect spread, that the failed node stays out long enough for Ceph to rebuild its replicas, and that the protection scheme fits in the N−1 remaining domains (with 4+2 erasure coding on six nodes, for instance, losing one does not rebuild either). The percentages in the table are truncated downwards. With uneven disks or a skewed spread, the real number is worse. The opinions on raising the full ratio in the heat of the moment, and on what to watch in monitoring, come from running Proxmox VE with Ceph in production, not from the sources cited.