Growing Ceph through a single node grows nothing

The Ceph blog post is dated 11 November 2026 — the blog dates it tomorrow; we read it today — and signed by Anthony D'Atri. It opens on Thanksgiving pie and ends with an example worth a budget meeting: three nodes weighted 100, 100 and 1,138. That is 1,338 of raw disk. The pool stores 95. And 1,043 — 78% of what you paid for — is something nobody can ever use.

We run Proxmox VE with Ceph in production across several datacentres, and the conversation that comes up most often when someone's cluster fills up is always the same: "there are free bays in node three, we put the disks in there and we are done". It is the natural answer, it is the cheapest one, and in a three-node cluster with replica 3 it does absolutely nothing. That is not our opinion: it is CRUSH arithmetic, and the author has just published it with a calculator alongside so nobody has to take it on trust.

The rule that decides your capacity fits on one line

Every placement group (PG) in a pool is a pie. A replicated pool cuts it into as many identical slices as its size; an erasure-coded one cuts it into k data slices and m coding slices. CRUSH hands those slices out to the failure domains named in the pool's rule, at most one slice each, so that losing a host, a rack or a room costs every PG at most a single slice. That is the part that keeps your data safe. The part that decides your invoice is the corollary, and the author writes it plainly: "A failure domain can never hold more than 1/n of what the pool stores, however big it is."

Where n is the replicated pool's size or the erasure-coded pool's k+m. The default failure domain is the host: both the default replicated rule and the default erasure-code profile use it. And a domain's CRUSH weight is the sum of the weights of its OSDs, which by default is their capacity in TiB. So the mental shift is this: your capacity is not the sum of your disks. It is the raw that CRUSH can spread without any domain going over 1/n, scaled down by full_ratio (0.95 by default, the point where Ceph stops writes) and divided by n — or multiplied by k/(k+m) if you are running erasure coding. And when you have exactly n domains, that is simply whatever fits in the smallest one. Backfill, on top of that, stops earlier, at backfillfull_ratio (0.90).

1,338 of disk, 95 writable

These are the three layouts the author publishes, stacked one under the other. Read the "stores" column carefully: those are units of data, not raw. With replica 3, storing 95 consumes 285 of disk.

The last two rows are the same figure. A fourth node of 1,701 and a fourth node of 150 store exactly the same: 142.5. Of those 1,701, 150 do any work. The other 1,551 are there so somebody can see them on the inventory. And the comparison we find genuinely painful is this other one, which is not in the article but falls out of its own numbers: going from three nodes of 100 to four of 100 raises capacity from 95 to 126.7, 33% more; adding 1,038 of disk to one of the three leaves it at 95, exactly where it was.

The formula to take to the purchasing meeting

The author states it two equivalent ways: no failure domain can usefully hold more than 1/n of the pool's raw capacity; or, turned around, the biggest failure domain can be at most the sum of the others divided by n−1. That is a spreadsheet formula, and it survives a meeting with the rep who wants to sell you the big box.

Let us apply it to a case that is not in the article. You have five nodes of 100 and you are about to buy the sixth. With replica 3, n = 3, the others add up to 500, and the new node can go up to 250 without wasting anything. With erasure coding 4+2, n = 6, that same 500 divided by five caps it at 100: not one disk more. The same hardware, the same budget and the same room give two different answers depending on a decision taken when the pool was created, probably years ago, probably never written down anywhere.

CRUSH on its own does not reach the ceiling

All the figures above assume a perfectly balanced placement, and CRUSH alone does not get there. Its straw2 algorithm draws each domain in proportion to its weight, but when it draws one the PG already uses, it draws again. The effect is that a heavy domain gets picked less often than its weight suggests, and the light ones fill early. In the 100·100·100·150 layout, a balanced placement reaches 142.5 with nothing stranded; CRUSH alone stops at about 132, when the first small node fills.

The same dead disk, two different endings

This is where the arithmetic stops being accounting and becomes operational. An OSD of weight 10 dies in one of three nodes of 100, replica 3. Every PG already has a copy on the other two nodes, so whatever was on that OSD can only be re-protected inside its own node. That node is now 90, and since every PG has a copy on it, it sets the pace: the pool's ceiling drops from 95 to 85.5. You lose 9.5 to the dead disk and another 9.5 on each of the two healthy nodes, which can no longer be used.

And now the sentence that ought to be pinned to the wall of anyone running a three-node Ceph: backfill stops at backfillfull_ratio, so if the pool held more than 81 — 85% of its ceiling — when the disk died, recovery stops with the PGs still degraded. Not slow: unfinished. With a fourth node of 100, the same dead disk lowers the ceiling only from 126.7 to 123.5, and full re-protection just needs the pool under 117, 92% of the ceiling. If what you lose is a whole node out of three, there is nowhere to re-protect: the PGs stay degraded until the node comes back. Out of four, they re-protect on the other three as long as the pool is under 90, 71% of its ceiling.

The same dead disk, in the same cabinet, on the same Tuesday. With four nodes it is paperwork — it re-protects itself and you mention it on Monday; with three, it is a degraded cluster waiting for a replacement to arrive. The difference is decided neither by the disk nor by luck: it was decided months earlier, when someone chose three nodes instead of four because three was what quorum needed. We wrote about it in more detail in why a three-node Ceph does not heal, it holds on, and this article finally puts the capacity numbers next to it.

Erasure coding: the same arithmetic, with k+m

A 4+2 pool across six nodes of 100 stores 380. An OSD of weight 10 dies and the ceiling drops to 342; full re-protection needs the pool under 324. Across seven nodes of 100, the same pool stores 443.3, the same failure leaves it at 437, and re-protection asks for under 414. What that seventh node buys, in percentages we worked out from his figures: 17% more capacity, a 38-point drop turned into a 6-point one, and moving from having to sit at 85% of the ceiling to being able to sit at 93%. Three different things, and only one of them shows up on a purchasing spreadsheet.

And a warning the author dispatches in one sentence, which we repeat every time someone proposes m = 1 to save a parity disk: "if you choose m=1, expect data unavailability during maintenance and data loss when component failures overlap". This is not a textbook "not recommended". It means that during planned maintenance — yours, on Tuesday — the pool is unavailable, and if one failure overlaps another, the data is gone. The rewritten PG Calc marks exactly that in amber: pools with fewer than three replicas, and pools with m = 1.

The five commands we run before signing off an expansion

When someone calls us to "grow the Ceph", the first question is not which disks, it is how many failure domains there are and what each one weighs. That is the arithmetic outside the node; the one inside it — the per-OSD thresholds and why the fullest disk rules the whole cluster — is in capacity is set by the fullest disk. This is what we run before looking at a catalogue, and it can be copied as is:

With those five outputs and the formula above, the recommendation writes itself, and often it is the one that goes down worst in a budget meeting: do not buy disks yet, buy a node. It is a harder invoice to defend and a good deal more commissioning work, but it is the only answer that fixes capacity and recovery at the same time. And there is an equally honest "when not to": if your cluster already has five or six well-balanced domains and what you need is volume, then yes, disks; in that layout the formula gives you plenty of room and there is no need to spend a whole server.

A note on versions, because the article mentions it in passing and it has consequences: the PG autoscaler answers the question of how many PGs a pool should have well for many clusters running recent releases — the caveat is his, he does not sell it as universal — once mon_target_pg_per_osd is at least 200, and that is its default beginning with the Umbrella release. If your cluster goes back a while, that parameter is probably still where it was. On who has Umbrella today and who does not, we already wrote about what the Proxmox wizard does and does not ship. And if what worries you is the other side — what happens when re-replication is switched off on purpose — that is in what the noout flag actually does.

The leftovers, which is where the food is

The article closes with five lines under the heading "Leftovers". We translate them because they are worth more than the rest of this text:

• Have at least one more failure domain than the pool's size or k+m.
• Keep failure domains of about the same weight: none can usefully hold more than 1/n of the raw capacity.
• Add capacity across failure domains, not to one.
• Keep the balancer on.
• Leave room below the backfillfull ratio so recovery fits.

We add a seventh, which is not technical: measure the weight, not the catalogue. You run ceph osd df tree before the purchase order, not after the rep has put the quote on the table. It takes thirty seconds and it is the only thing separating "we expanded the storage" from "we expanded the inventory".

Sources (consulted on 2026-11-10): every placement, failure and re-protection figure — 100·100·1,138 storing 95 with 1,043 stranded (78% of raw); 100·100·100·1,701 storing 142.5 with the same 78% stranded; the 150 cap for the fourth node; 100·100·100·150 reaching 142.5 balanced and about 132 with CRUSH alone; the drop from 95 to 85.5 and the 81 threshold when an OSD of weight 10 dies in three nodes of 100; 126.7 to 123.5 and the 117 threshold with four; the 90 (71% of the ceiling) when a whole node is lost out of four; and the 4+2 pool at 380 → 342 → 324 across six nodes and 443.3 → 437 → 414 across seven —, both quoted sentences ("A failure domain can never hold more than 1/n of what the pool stores, however big it is" and "if you choose m=1, expect data unavailability during maintenance and data loss when component failures overlap"), the rule that the biggest domain ≤ the sum of the others ÷ (n−1), straw2's redraw behaviour, the balancer being on by default in upmap mode, the 0.95 and 0.90 defaults for full_ratio and backfillfull_ratio, mon_target_pg_per_osd defaulting to 200 from Umbrella onwards, PG Calc's amber for m=1 and fewer than three replicas, and the "Leftovers" list — "Slicing the Pie: CRUSH Failure Domains and Your Capacity", by Anthony D'Atri, official Ceph blog, dated 2026-11-11, which also ships an interactive calculator — the same one the documentation publishes as FD Calc — and the rewritten PG Calc. The following are our own work, derived from those same figures: the comparison between going from three to four nodes of 100 (95 → 126.7, 33% more) and fattening one of the three (95 → 95); the sixth-node example with five of 100, capped at 250 under replica 3 and at 100 under 4+2; the calculation that the 10.5 CRUSH leaves on the table without the balancer is 7% of usable capacity; the 85% and 93% headroom percentages in the six- and seven-node 4+2 cases, and the seventh node's 17% extra capacity; the 1,551 on the fourth node of 1,701, which comes from subtracting the useful cap of 150 the article publishes and not from its own stranded-capacity convention (which for that case gives 78% of raw); the formulation of capacity as distributable raw ÷ n after full_ratio; the five checking commands; and the closing "when not to". Photograph: "20-Bay Hard Drive Case 20251006 – fünf 3,5\"-HDDs", by PantheraLeo1359531, CC BY 4.0 via Wikimedia Commons, original 9,531 × 6,357, cropped and brightened by us.