The cluster picks a node by counting guests: anti-affinity in Proxmox VE 9 and the rule nobody declares

A node dies in the small hours. The rest of the cluster keeps quorum, the watchdog does its job, and after a while the machines from the dead node — the ones registered as HA resources; the rest stay switched off — are booting somewhere else. Everything worked. The question hardly anyone asks is who chose that somewhere else, and on what basis. On a freshly installed Proxmox VE the answer is written in the official documentation and is far simpler than people assume: the scheduler looks at the number of active guests on each node and picks the one with the fewest. It counts. It has no idea that 101 is the second leg of 100.

Who decides where your machine comes back up

The component is called the Cluster Resource Scheduler, CRS. On every HA manager round — roughly ten seconds — it goes through the resources and decides whether any of them needs a new placement. It has three modes and the factory default is basic: it counts active guests per node. The other two do look at CPU and memory: static-load uses the configured quotas of the active guests, and dynamic-load adds average real CPU and memory usage on top of those quotas. None of the three ships with a list of which machines are pairs.

The documentation calls the moments when the CRS kicks in scheduling points, and marks two of them as always active: resource recovery when a node fails, and changes to the rule configuration. Which means placement is not optional. Even if you have touched nothing, on the day of the incident something decides which piece of iron each thing comes back on. The only open question is whether that decision takes into account what you know about your own architecture.

What changed in 9.0 and arrived on your cluster by itself

For years the only thing you could express in Proxmox was HA groups: this machine, on these nodes, with these priorities. It was good for saying where yes, and no good at all for saying with whom not. Proxmox VE 9.0 replaced that mechanism with HA rules, which come in two distinct families: node affinity rules, which tie resources to nodes, and resource affinity rules, which tie resources to each other. And there is something worth pinning down, because it trips people up: both families accept the --affinity negative modifier, but it means different things in each. In a node rule, negative means "on any node except these", which is a handy way of excluding without listing the whole cluster. In a resource rule, negative means "these two, never on the same node". Whenever anti-affinity appears here without a qualifier, it is the second one.

The detail worth not skipping is that none of this waits for you to decide. The release notes are explicit: groups are deprecated and existing ones are automatically migrated to node affinity rules once every node in the cluster runs Proxmox VE 9. The groups' nofailback option disappears, replaced by failback, now configured per resource. If you upgraded the cluster and never went back to look, you have new rules written by the system itself and an option that has changed both place and polarity.

A preference and an order

Here is the asymmetry that, in our experience, most people discover late. The two rule families have opposite defaults.

The practical consequence is that many people have the opposite of what they think they have. The rule restricting a machine to a specific node, if you do not mark it strict, restricts nothing on the day that node is missing. And the rule keeping two machines apart wins even when the alternative is that one of them does not start at all. They are declared like this:

# keep two resources apart: never on the same node
ha-manager rules add resource-affinity keep-separate \
    --affinity negative --resources vm:200,ct:300

# pin a resource to a node... and actually mean it
ha-manager rules add node-affinity ha-rule-vm100 --resources vm:100 --nodes node1
ha-manager rules set node-affinity ha-rule-vm100 --strict 1

# list the rules, and which ones affect a given resource
ha-manager rules list
ha-manager rules config --resource vm:100

The arithmetic of a negative rule

A rule that separates n resources needs n nodes to put them on, and that is written down as a feasibility test rather than as advice: "A negative HA resource affinity rule cannot specify more HA resources than there are nodes in the cluster. Otherwise, the HA resources do not have enough nodes to be separated." A rule that fails that test is left disabled, which is the first way of losing a guarantee without noticing. And for the ones that do pass, the documentation says what happens when the maths breaks later on: if the constraint cannot be satisfied, the HA manager leaves the resource in recovery state in the case of a failover, and in error state otherwise.

Let us do the maths with the most common cluster we come across: three nodes. If you put a negative rule across three resources — say the three instances of a distributed application — in normal running they fit exactly: one per node, no slack. The day a node dies there are two places for three things that cannot share a place, and one of the three does not come back. The scheduler has not got it wrong; it has done exactly what the rule says. With two resources in the rule, on the other hand, there is a spare node and recovery is clean.

Our reading, not the documentation's: we would not call this a defect. It is a design decision you were making without knowing it. There are services where bringing two replicas up on the same iron is worse than having one down, because the second node gives a false sense of cover that evaporates at the next reboot. And there are services where running and degraded beats stopped and correct by a mile. That choice is yours, and it only gets made properly on a calm Tuesday. Never at three in the morning.

And there is a calculation that comes before this one, even if many people meet it afterwards: the high availability chapter describes rules, priorities and fencing, but it describes no mechanism that reserves capacity for failover. Nothing stops you filling all three nodes to the brim with memory. The day one is missing, either the remaining RAM is enough to start everything or it is not, and that subtraction is one you do beforehand or nobody does. It is one of the things we already put on the table when writing about when NOT to migrate from VMware to Proxmox, and it remains the most honest difference between the two platforms on this ground.

It is the same idea we set out when writing about why redundancy does not survive the procedure, only inverted: there the redundant pair was undone by a person following a manual, and here it is undone by a scheduler doing its job. And it is first cousin to the distinction between redundancy and route diversity: two duplicated things are only two if they do not share the point that can fail. In a cluster that point is the node, and stating which one is not shared is a text file.

What the rule drags along without being asked

Rules combine with each other, and that is where the scope grows on its own. Three behaviours that are in the documentation and are almost never in anyone's head when writing the first rule:

• Two positive rules sharing a resource merge into one. If 100 and 101 go together, and 101 and 102 do too, then all three go together. Nobody wrote that third relationship.
• A group bound by positive affinity inherits the constraints of any of its members. If 102 is tied to node 3 by a node rule, 100 and 101 end up tied to node 3 as well. And if one of the group has a negative affinity with another resource, all of them have it.
• Rules are put through feasibility tests before being applied, and those that fail are disabled until you resolve the conflict. Among the tests: a resource can only be part of a single node affinity rule; a negative node rule cannot carry priorities nor list every node in the cluster; a resource affinity rule needs at least two resources. Two resources that sit in both a positive and a negative rule are a straight contradiction, and both rules are switched off.

That last point strikes us as the most dangerous in practice, not for its complexity but for its silence. A disabled rule does not complain every morning. It sits in the interface, with its name on it, doing nothing. Whoever wrote it believes the cluster has a guarantee the cluster does not have, and will find out on the day of the failover. Which is why the first command in the list below is not writing a rule: it is reading the ones already there.

Since 9.2 the cluster also moves things on its own

All of the above already mattered, but it mattered mostly on the day of the incident. Proxmox VE 9.2, released on 21 May 2026, added the CRS automatic load balancer: the cluster measures its own imbalance and, if it stays over the line for enough consecutive rounds, issues migrations to correct it. It only moves HA-managed resources, migrations are executed one at a time, and the imbalance threshold ships at 30 %. It is not on by default — it requires static-load or dynamic-load mode — but the day somebody switches it on, placement stops being a still photograph of the day you built the cluster and becomes a continuous decision.

The documentation makes clear that those migrations always obey the affinity rules in force. That is the reassuring sentence of the chapter, and it is true. It is also exactly the problem: obeying the rules only protects you if the rules exist. A balancer that does not know 100 and 101 are the same thing twice will do the only thing you asked of it, which is to spread load, and spreading load may well mean putting them together. Two further details are worth noting: automatic rebalancing can be disabled resource by resource, and if a resource has positive affinity with another one that has it disabled, neither of them moves.

Five minutes in front of the cluster

This is what we look at when we sit down in front of a Proxmox VE 9 that is already in production. None of the five points needs a maintenance window.

1. Read before writing. ha-manager rules list and look for two things: rules you were not expecting — the ones 9.0 migrated by itself — and rules disabled by conflict. The second is a guarantee somebody believes they have.
2. Write down the list of pairs. On paper, before touching anything: the two domain controllers, the database and its replica, the two load balancer nodes, the pair of application servers. There are almost always fewer than you think, and they are almost never written down anywhere.
3. Do the maths before creating the rule. Resources in the negative rule against nodes able to run them, minus one, which is the one that can fail. If the subtraction does not work out, do not create the rule yet: first decide what you would rather happen.
4. Review failback per resource. It is the option that replaced the old groups' nofailback. It decides whether a machine returns to its preferred node on its own when that node reappears, which is a second window of risk right after the first one.
5. Rules first, balancer second. If you are going to enable 9.2 automatic balancing, the order matters: declare the pairs, check that no rule ended up disabled, and only then switch on dynamic mode. The other way round, you are giving it permission to move things whose relationships it does not yet know.

When not to write the rule

If your cluster is two nodes, the rule still buys you something, but it is worth knowing exactly what. It keeps the two machines apart while both nodes are alive, and that is not nothing: the scheduler spreads guests by total count, so if one of the two has more guests it can happily put them together. What it does not buy you is anything for the day of the incident, because as soon as a single node is left the rule's answer is no. On two nodes, anti-affinity protects the ordinary days and steps aside on the loud one. If what you run is a handful of machines with no real pairing — one VM per service — then yes, you need none of this, and adding rules will only give you conflicts to resolve.

And if yours is the case above — you would rather have running and degraded than stopped and correct — there is a workaround we use and that we will call a workaround, because it is not a feature designed for this: instead of a negative resource affinity, two non-strict node affinity rules, one per machine, pointing at different nodes. It separates while it can and does not block when it cannot, which is exactly the asymmetry of defaults this post opened with. It is not as clean as the negative rule and gives no hard guarantee; it simply does the other thing.

All of this is about the same thing as the rest of our work: node failure is inevitable and not up for negotiation. What is decided, and decided beforehand, is the size of the outage. A cluster with its rules written and its maths done loses a node and carries on. One without them also survives nearly always, because the scheduler spreads things reasonably well, and that "nearly always" is precisely the problem: it works right up until the day it puts your two replicas in the same place, and on that day there is nobody left to ask. If you are building or reviewing one, the hardening checklist we apply to every Proxmox VE 9.2 cluster covers the other half of the conversation.

Note on sources

Everything technical in this post comes from three primary Proxmox sources, consulted on 10 September 2026: the High Availability chapter of the official Proxmox VE documentation and the Roadmap page of the official wiki, with the changelogs for versions 9.0, 9.1 and 9.2. From the documentation: that the CRS default scheduling mode is basic and that this mode picks a node by the number of active guests; the existence of the static-load and dynamic-load modes and what each measures; that the HA manager round lasts around ten seconds; that resource recovery and rule configuration changes are always-active scheduling points; that node affinity rules default to strict 0 and what that implies; that resource affinity rules are strict by default and, in practice, only strict — the command exposes no --strict for that family — and that, if the constraint cannot be met, the resource is left in recovery state during a failover or in error state otherwise; the merging of positive rules sharing resources and the inheritance of constraints within a positive group; the feasibility tests and the fact that rules failing them are disabled; how the automatic load balancer works, its default 30 % imbalance threshold, that it only moves HA-managed resources and does so sequentially, that it requires static-load or dynamic-load mode, that its migrations always obey the affinity rules in force, and the behaviour of per-resource disabled rebalancing. The commands quoted are those of the documentation itself and of the ha-manager manual page. From the Roadmap: that HA rules were introduced in Proxmox VE 9.0, that HA groups are deprecated and automatically migrated to node affinity rules once every node runs Proxmox VE 9, that nofailback is replaced by failback, that Proxmox VE 9.2 was released on 21 May 2026 and that dynamic balancing arrived in that release. The following is OUR reading, not the documentation's: that a cluster with no declared rules has a latent problem rather than a problem; that an unsatisfiable negative rule is a design decision about whether you prefer degraded or stopped; the working order of the checklist; and the workaround of two non-strict node affinity rules as a soft substitute for anti-affinity, which we present as a workaround precisely because the documentation does not frame it that way. We give no performance figures, recovery times or percentages of affected clusters, because we have not measured them and would rather say so. That we run Proxmox VE with Ceph in production and that we are resellers of neither Proxmox nor VMware is everyWAN's own information. Test any of these commands in your own lab before running them in production. The cover photograph is "Front of server racks at NERSC", by Lawrence Berkeley National Laboratory, published on Wikimedia Commons in the public domain.