Corosync adds 650 milliseconds per node: the clock your upgraded cluster is still carrying

There is a number written into your servers that nobody chose and that grows every time you extend the cluster. It is not on the invoice, it does not show up on a dashboard and no inventory checks it: it is how long Proxmox takes to notice that one of its nodes has died. With factory values, that time goes up by 650 milliseconds for every node you add. Proxmox fixed it in version 9.2, but the fix only lands on clusters created from scratch. If you upgraded yours from 8 to 9 these past weeks, you are still running the old clock.

This post is the declared sequel to another one. A few weeks ago we wrote that Proxmox high availability does not prevent downtime, it shortens it, with the ceiling the documentation itself publishes: around two minutes of detection and failover, hence the 99.999% limit the vendor puts on itself in writing. That one was about what you buy when you switch HA on. This one is about what happens before those two minutes, in the part almost nobody looks at, and about a new setting your cluster has probably not received.

Before anything moves, the cluster has to agree

When a node powers off, the first thing that happens has nothing to do with virtual machines. The surviving nodes have to rebuild the list of who is still alive, and corosync handles that with two timers. The token is how long a node waits for the token before declaring it lost: 3,000 milliseconds by default. The consensus is how long they then wait to agree on the new membership; if you never set it, it is automatically calculated at 1.2 × token.

The detail that changes the outcome is that those 3,000 milliseconds are not the value in use. As soon as the configuration file has a node list with three or more members — that is, on any real cluster — corosync computes the actual timeout like this: token + (number of nodes − 2) × token_coefficient. And that coefficient is 650 milliseconds out of the box. The documentation explains why plainly: it exists so the "cluster can scale without manually changing token timeout every time new node is added". It is a reasonable decision — a bigger cluster needs more slack — with a consequence almost nobody has worked out.

The sum of the two timers is, in the words of the Proxmox documentation, "the minimum time needed to reestablish a new cluster membership after a node goes offline". In other words: the floor under your failover, everything that happens before the HA manager lifts a finger. With the formula and the factory values, the arithmetic writes itself:

The arithmetic is ours; the formula and the defaults come from the official corosync documentation. The check that the formula is the one we think it is comes from Proxmox's own manual, which publishes as an example token 4,950 and consensus 5,940 — "10.89 seconds" — which is exactly the five-node row. These are configured timers, not times measured in your machine room.

You do not have to take the table on faith. These are your two real numbers, in milliseconds, right now:

The race: 45 seconds against 60

That growing number would be a curiosity if it were not racing another one that does not grow. Proxmox isolates lost nodes through self-fencing: a node that loses quorum cannot reset its watchdog timer and reboots itself "after the watchdog has timed out (this happens after 60 seconds)". That 60 is fixed. The other one, the agreement clock, climbs with every server you buy.

Proxmox's documentation puts the race in writing. Keeping the sum of token and consensus below 45 seconds is "especially important" when HA is enabled, "to ensure that a new cluster membership is formed before the watchdog timeout of 60 seconds expires, which would trigger a node fence". And it asks for margin below those 45 "so that transient timing variations do not push the cluster past the watchdog threshold". Its escalation is explicit: lowering the coefficient is suggested above 30 seconds, recommended above 40 and strongly recommended above 45.

The 9.2 fix only reaches brand-new clusters

Proxmox has seen the problem and fixed it. The documentation says it verbatim: "Since Proxmox VE 9.2, new clusters are created with a lower token coefficient of 125 milliseconds explicitly set in /etc/pve/corosync.conf". The effect is large and measurable: with 125 instead of 650, the 29-node cluster in the last row goes from 45.21 to 14.03 seconds. Three times faster, with the 60-second watchdog at a comfortable distance.

Now read that sentence again, slowly: new clusters are created. The coefficient is written into the file when the cluster is created. If yours has been running for years and you upgraded it from the 8 branch to 9 — which plenty of people are doing right now, because Proxmox VE 8 reaches end of support this month — your corosync.conf is the one you already had: no explicit coefficient, and therefore the same old 650. The documentation confirms it from the other side, when it explains how to lower the value: you have to add the line by hand. Upgrading did not hand it to you, and the documentation mentions no warning that would tell you.

When NOT to touch anything

You could read this post as "go and change the coefficient". Do not. If you run between three and eight nodes, your sum runs from 8 seconds to a little over 15, nowhere near 45, and the factory value is not your problem. Touching corosync.conf as a fashion statement means reaching into the one file that can split your cluster into two halves that no longer talk, with the machines powered on. You change it when the arithmetic asks for it, following the documented procedure, inside an agreed window.

What is worth doing today, whatever your size, is the cheap part: run that corosync-cmapctl, write the two numbers into your cluster documentation and know where on the table you sit. It costs two minutes and tells you whether you have homework. The documentation is also clear that margin is not bought by lowering the coefficient alone: it demands a cluster network that holds up, and tightening timers on a network with irregular latency makes things worse rather than better.

The other clock still running: storage

If your storage is Ceph, there are two more timers and they run on a different scale. An OSD is declared down when it stops reporting for osd_heartbeat_grace, 20 seconds by default — with the caveat that the monitor wants several OSDs on different hosts to report it before marking it. But down does not equal out: to mark it out and start rebuilding the missing replicas, Ceph waits mon_osd_down_out_interval, and there we are talking 10 minutes.

Those ten minutes are deliberate and they are good: they stop a maintenance reboot from kicking terabytes across the network. It pays to know they exist because they mark the exact moment a drill stops being free. If the node comes back in five minutes, nothing happened; if it takes you twelve to bring it up, the cluster has already started rebalancing and you have earned an afternoon of recovery traffic. There is a third value almost nobody looks at, and it matters before you power down a whole rack: mon_osd_down_out_subtree_limit, default rack, is the smallest CRUSH map unit Ceph will not automatically mark out. A whole node, yes; a whole rack, no.

With one precondition we have covered here before and which shows up again: for that rebuild to finish, it has to fit. If the cluster is tight on space, what you will see is not a recovery but a traffic jam, and we go into it in our post on why in Ceph capacity is set by the fullest disk.

You only learn this by switching something off on purpose

The numbers above are the theoretical floor. Your company's real number comes out one way only, and it has had a name for years: "Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production". Experimenting, confidence, production. Not auditing, not documenting, not a lab.

The method they propose has four steps, and the first is the one everyone skips: define the "steady state", a measurable output saying the system is behaving normally. Then you hypothesise that this state will hold, you introduce the real-world event — "servers that crash, hard drives that malfunction, network connections that are severed" — and you try to disprove the hypothesis. Here is how we bring that down to earth for a thirty-person company:

Our last full recovery drill closed in 14 minutes. We give it as an internal figure and as proof that it can be measured, not as a contractual promise: every company's number depends on its architecture, its application and how many times it has rehearsed. Ours was not that on day one either.

What you are actually buying

Failure is inevitable: a power supply dies, a disk goes, someone trips over a cable. Whether that becomes an outage — the business stopping — is a design decision, and part of that decision is a handful of numbers already written into your servers, set as defaults by people who had never seen your cluster. Finding out what they are costs a morning; not knowing them costs whatever a day of downtime costs.

At everyWAN we run Proxmox VE with Ceph in production across several data centres, and we run timed recovery drills. That is why we look at these timers before signing a continuity figure with anyone: it is what sits behind infrastructure and cloud and compliance and continuity, rather than a high-availability box ticked in a proposal. When the clock really starts, what helps is having a team watching.

Sources (verified one by one): the defaults for token (3,000 ms) and token_coefficient (650 ms), the condition that the formula only applies with a node list of at least three members, the formula token + (nodes − 2) × token_coefficient and the automatic calculation of consensus at 1.2 × token — corosync.conf(5); the 125 ms coefficient on clusters created from Proxmox VE 9.2 onwards, the 45-second threshold against the 60-second watchdog, the 30/40/45 escalation, the 4,950 / 5,940 ms example and the verification command — Proxmox VE: Cluster Manager; watchdog self-fencing at 60 seconds — Proxmox VE: High Availability; osd_heartbeat_grace (20 s), mon_osd_down_out_interval (10 min) and mon_osd_down_out_subtree_limit (rack) — Ceph: Monitor/OSD Interaction; the definition of chaos engineering and the four steps of the experimental method — Principles of Chaos Engineering. Documentation consulted on 28 August 2026. The node-versus-token+consensus table is our own calculation from the official formula and defaults, checked against the five-node example published in Proxmox's own documentation; these are configured timers, not times measured on your cluster. The 14-minute drill is an internal everyWAN figure, not a service commitment.