Your three-node Ceph does not heal itself: it holds on

If a disk dies in a three-node cluster, Ceph fixes it by itself while you have lunch. If the whole node dies, it fixes nothing: it sits and waits for it to come back. Holding on and healing are two different things with two different bills, and the difference lies in how many distinct places Ceph has to put a copy.

Start with what ships by default. The Proxmox VE manual says, on its Ceph page, that when you create a pool "we set a default of 128 PGs, a size of 3 replicas and a min_size of 2 replicas". Three copies, and the cluster accepts writes as long as two survive. Those three copies go to distinct hosts, and that is not a convention either: it is decided by osd_crush_chooseleaf_type, which is 1 by default. The Ceph documentation spells it out when explaining how to build a single-node cluster, because there you have to lower it "from the default of 1 (meaning host or node) to 0 (meaning osd)". With three servers and three copies the arithmetic comes out exact: every host holds one copy of absolutely everything.

That exactness is the problem. A design whose arithmetic comes out exact runs out of room the moment one term goes missing.

A dead disk does fix itself

Worth starting with the good case. If one disk dies and the host is still alive, the CRUSH rule picks a host first and then descends to a disk inside it, so there is somewhere to place the copy: another OSD on the same server. The cluster rebuilds and returns to HEALTH_OK without anyone touching anything. And that is no exotic setup: the Proxmox manual itself recommends "at least three nodes and at least 12 OSDs, evenly distributed among the nodes", four disks per server at the recommended minimum.

Everything from here on is about the other case: the whole host going away, which is the case that decides whether your continuity plan is worth anything.

The clocks nobody looks at

Say a node loses power. The first thing that happens is an absence of heartbeats. OSDs ping each other every six seconds (osd_heartbeat_interval) and treat a peer as down after twenty seconds of silence (osd_heartbeat_grace), although that interval is not rigid either: mon_osd_adjust_heartbeat_grace ships enabled and Ceph stretches it for OSDs with a history of flapping. The peers report to the monitor, and two reporters are required (mon_osd_min_down_reporters), counted at whatever bucket level mon_osd_reporter_subtree_level sets, by default host. With three hosts and one down, exactly two buckets remain to report from, which is precisely what the parameter asks for: detection works, but it stops working with two hosts, and that is the real floor of this design.

Between "down" and "out" there is a second clock, and it is the one that matters, because marking an OSD out is what triggers data movement: mon_osd_down_out_interval, ten minutes by default. Ten minutes in which, deliberately, not a byte moves. It is well judged: a switch reboot should not set off a cluster-wide migration. But it is a published interval, so it belongs inside your RTO rather than turning up as a surprise on the bad day. Nor is it fixed: mon_osd_adjust_down_out_interval ships as true, so Ceph scales it according to its estimate of whether that OSD is one of the unstable ones.

There is a third clock, slower, for the node that does not quite die. If an OSD simply stops reporting to the monitor, the grace period before it is declared down is mon_osd_report_timeout, fifteen minutes. That is the path of the server that does not switch off but does not answer either, the failure that genuinely costs you a diagnosis.

After the ten minutes, still nothing

The monitor marks the dead node's OSDs out and Ceph tries to place the third copy somewhere else. With the failure domain at host and three hosts, one of them gone, that somewhere else does not exist: the two survivors already hold their copy and the rule forbids two copies of the same data on the same host. Placement groups settle at active+undersized+degraded, which the Ceph documentation describes in these words: "have the degraded or undersized flag set, which means that there are not enough instances of that PG in the cluster".

You may not even see all those OSDs marked out. Ceph carries a brake, mon_osd_min_in_ratio at 0.75, defined as "the minimum ratio of in Ceph OSD Daemons before Ceph will mark Ceph OSD Daemons out": on a cluster with many disks per server, emptying a whole host hits that limit and some of its OSDs stay down without ever going out. The conclusion is unchanged — there is still no recovery — but it explains why the ceph osd tree you look at afterwards may not match what you expected.

Meanwhile the cluster keeps serving: two copies remain and min_size is two, so your machines write as if nothing had happened. That is why almost nobody looks at it. A degraded cluster that works produces a yellow warning on a dashboard; a stopped cluster produces twenty phone calls. And the fault that produces no phone calls is the one that stays for weeks.

The price of holding on is on the same manual page

The Proxmox VE manual has a sentence almost everyone reads in another context — it sits in the section on changing the Ceph network — and which is really the operating rule for a degraded cluster: "Do not restart OSDs on multiple hosts at the same time. Chances are that for some PGs (placement groups), 2 out of the (default) 3 replicas will be down. This will result in I/O being halted until the minimum required number (min_size) of replicas is available again".

Read it again with one node already out. A downed host turns any operation on a second host into precisely the scenario that sentence forbids. Updating node 2's kernel and rebooting it, swapping one of its disks, moving a cable: any of those drops some placement groups below min_size and I/O stops dead until the second one comes back. While one node is out you are left with no maintenance window on the other two. That is the price of holding on, and it appears on no invoice.

The returning node brings the traffic

The day you hand the node back to the cluster, the thing that had not happened until then begins: rebuilding everything written while it was away. And here is another sentence from the Proxmox manual worth reading twice: "The volume of traffic, especially during recovery, will interfere with other services on the same network, especially the latency sensitive Proxmox VE corosync cluster stack can be affected, resulting in possible loss of cluster quorum".

Healing can cost you quorum. If Ceph traffic and corosync traffic share a cable, the recovery that fixes your storage is able to knock over the cluster that holds it up, and we have already written about how little latency corosync tolerates. Hence the manual asking for at least 10 Gbps exclusively for Ceph and, in high-performance installations, three physically separate networks. It is the condition for recovery not being the second incident of the day.

On how long that rebuild takes, the manual is honest and brief: "Especially with small clusters, recovery might take long", and it recommends SSDs in small setups "to reduce recovery time, minimizing the likelihood of a subsequent failure event during recovery". So the reason to buy SSDs in a three-node cluster is to shorten the window in which a second failure catches you with no safety net, more than day-to-day performance. There is another purchasing decision hidden on the same page: "a single OSD failure forces Ceph to recover more data at once" the bigger the disk. Our judgement, not the manual's: that fact fights against price per terabyte, which always pushes towards bigger disks, and it deserves a deliberate decision — just as usable capacity is set by the fullest disk and not by the spreadsheet total.

The memory you do not have on the day you need it

In a hyper-converged cluster the two survivors have to do two things at once, and both cost memory. One: start the dead node's machines, because that is what high availability is for — and it does not prevent the outage, it shortens it. Two: pay Ceph's memory surcharge while it rebuilds. The Proxmox manual says it plainly: OSDs consume more "during critical operations, such as recovery, rebalancing, or backfilling", it recommends 8 GiB per OSD, and it warns against running memory to the edge in normal operation, advising instead to "leave some headroom to cope with outages".

If you sized the RAM for the good day, on the bad day you have a memory problem on top of a storage problem. And it arrives just as high availability has finished spreading the dead node's machines across two survivors, which is fifty per cent more load on each. Memory headroom in a three-node cluster is not an architect's luxury: it is half of your recovery plan.

Count places, not nodes

What separates "holding on" from "healing" is a small number: one more failure domain than copies. With replica 3 and four hosts, when one dies three distinct places remain, Ceph has somewhere to put the third copy, it rebuilds, and when it finishes you are protected against the next failure again without anyone driving to the data centre. With three hosts that never happens when what dies is the host, however many disks you have inside it. The fourth node does not buy capacity: it buys Ceph's permission to heal itself. The same arithmetic, with different numbers, is what decides whether replica 3 or erasure coding suits you.

The drawback we have seen forgotten most often deserves its own paragraph: a failure domain is whatever CRUSH believes, not whatever the invoice says. If the fourth server hangs off the same power strip, the same switch and the same room as the other three, you have bought one more host and not one more place; against the failure that genuinely takes you down, the room, you still have exactly one. What CRUSH knows about your installation is what you have told it, and by default all it knows is what each machine is called. The other two drawbacks are easier to live with: the fourth node costs money and adds traffic and one more vote, and it does not need to be a monitor — the manual says "You won't need more than 3 monitors, as long as your cluster is small to medium-sized".

What we check before signing off a three-node cluster

• How long this cluster takes to get from degraded back to HEALTH_OK once the node returns, measured once with the data it holds today and not the data it held empty on installation day. That number goes into the plan; the vendor datasheet number does not.
• The "hands off" window, in writing: while one host is out, the second is not rebooted, does not get a disk swapped and does not get a cable touched. And alongside it the deliberate use of ceph osd set noout for planned maintenance, which is the other half of the conversation. Written and signed, not merely known. A classic study of why large internet services fail (Oppenheimer, Ganapathi and Patterson, USENIX 2003) put manual configuration changes ahead of hardware as a cause, and the degraded window is exactly when a reasonable change turns expensive.
• Power strip, switch, room: the places counted one by one. With that list in front of you, you decide whether the CRUSH rule's failure domain stays at host or has to move up a level. And while you are there, look at mon_osd_down_out_subtree_limit, which exists so an entire subtree is not emptied at once and ships set to rack: if your map has no racks, that brake never engages.
• Genuinely separating Ceph's network from corosync's. Not a different VLAN over the same pair of cables: separate. It is the difference between a slow recovery and a recovery that takes quorum down with it.
• And the backup, outside the cluster. Replica 3 is not a backup: a cluster that holds up against a dead node will not hand you back a file deleted by mistake nor survive an encryption event. That is disaster recovery, a different problem and a different bill.

What we are not saying

All of the above describes the default configuration: three hosts, replica 3, failure domain host. If your CRUSH map has another level, or somebody edited the rule, the conclusion changes. Checking takes two commands, ceph osd crush rule dump and ceph osd tree, which cost ten seconds and are worth more than anything we say.

Nor are we saying a three-node Ceph is badly designed. Three is the documented minimum — "you must use at least three (preferably) identical servers" — and it works; we run Proxmox VE with Ceph storage in production across several data centres, and we have worked with Proxmox since its 3.x branches. For plenty of companies, three well-built nodes protect more than an expensive SAN they have never actually failed over.

And we are not saying "buy a fourth node". If the budget stretches to three, the answer is not more hardware: it is a written procedure for the degraded window and a backup that lives outside. All we maintain is that "it heals itself" is not part of what you buy with three when what you lose is a whole server, and that a continuity plan assuming that healing is counting on something the default configuration does not provide.

At everyWAN we design and operate distributed storage with Ceph and the infrastructure that holds it up, on Proxmox VE and in production. We sell nobody's licences, so when we say three nodes give you no automatic recovery from a lost host, nobody is paying us to say it — or to keep quiet about it.

Sources (consulted on 25 Sep 2026): the default values of osd_heartbeat_interval (6 s), osd_heartbeat_grace (20 s), mon_osd_adjust_heartbeat_grace (true), mon_osd_min_down_reporters (2), mon_osd_reporter_subtree_level (host), mon_osd_down_out_interval (10 minutes), mon_osd_report_timeout (15 minutes), mon_osd_adjust_down_out_interval (true), mon_osd_min_in_ratio (0.75) and mon_osd_down_out_subtree_limit (rack) are in Configuring Monitor/OSD Interaction in the Ceph documentation (Tentacle branch). The description of PG_DEGRADED and of the degraded and undersized flags is in Health checks. That osd_crush_chooseleaf_type defaults to 1 is in Pool, PG and CRUSH config reference, and the sentence translating that 1 into "host or node" is in Troubleshooting PGs. In that same reference osd_pool_default_size is 3; mind osd_pool_default_min_size, whose default value is 0, meaning "no particular minimum", which Ceph evaluates as size − (size/2) and which comes to 2 with replica 3 — the explicit 2 is set by Proxmox when it creates the pool, according to its own manual. The Proxmox quotes (three servers minimum, the default pool, the 12 evenly distributed OSDs, not restarting OSDs on several hosts at once, recovery traffic and corosync quorum, 10 Gbps exclusively, 8 GiB per OSD and memory headroom, the SSD recommendation for small clusters, a larger OSD forcing more data to be recovered at once, and the monitor count) are in the Ceph chapter of the Proxmox VE manual. The reference on human error outweighing hardware in serious outages is Oppenheimer, Ganapathi and Patterson, Why Do Internet Services Fail, and What Can Be Done About It?, USENIX 2003. What this post does NOT claim: we did not reproduce a node failure in a lab for this post; the behaviour described follows from the cited documentation and from the default replication rule, which is why we keep insisting that what you can verify on your own cluster is ceph osd crush rule dump and ceph osd tree. The timers are defaults and may have been changed on your installation (ceph config get). We give no figure for how long a real recovery takes, because it depends on volume, disk and network, and any number we gave would be made up. And the rule about counting places instead of nodes, like the reading of price per terabyte against rebuild time, is our own judgement and not a recommendation from any vendor.