Proxmox HA does not prevent downtime: it shortens it (and sometimes causes it)

The Proxmox VE documentation says something about its own ha-manager that hardly any virtualisation brochure would dare put in writing: it has "typical error detection and failover times of about 2 minutes, so you can get no more than 99.999% availability". It is in the official manual, on the page people skip on their way to the enable button. That paragraph is the honest definition of what you are buying when you turn HA on: you are not buying "it will not go down"; you are buying "it will come back up on its own".

The distance between those two sentences is what separates a project that lands well from an awkward meeting six months later. We have been running Proxmox VE clusters in production for years, and the conversation we repeat most often starts right after the three clicks that enable it: what stops happening and what starts happening from then on. Because things do start happening.

What actually happens when a node dies

A node running twenty virtual machines dies on the spot: dead power supply, hung kernel, somebody tripping over a cable. The rest of the cluster stops seeing it and checks that it still holds a majority of votes. Then comes the wait, and here is the detail that orders everything else: in Proxmox nobody isolates the lost node from outside — it isolates itself, because its own watchdog reboots it the moment it can no longer service it. Once that window has passed, the cluster can be sure nothing is being written from the other side, and that is when it starts the services on the nodes still alive. The documentation calls that chain error detection and failover, and attaches the figure we quoted above: about two minutes.

It is worth spelling out what "move" means here. It is not a live migration. A live migration copies the memory of a running machine to another node and the guest operating system never notices; it needs the source node alive in order to copy that memory. Here the source node no longer exists. The machines are cold-started somewhere else, exactly as if you had pulled their plug. On a diagram those are two similar-looking arrows. On the clock on Monday morning they are not the same thing at all.

Five nines come out of a subtraction

The "99.999%" in the documentation sounds like marketing until you convert it into minutes. A year has 525,600 minutes, and everything else follows from there:

And there is a caveat that never makes it into the slide deck: that ceiling belongs to the hypervisor, not to your service. It measures from the moment the cluster spots the problem until the virtual machine boots. What happens inside that virtual machine from that second onwards, nobody measures.

Inside the virtual machine, the power just went out

Nobody told the guest system to shut down cleanly. So it boots the way any machine boots after a blackout: the filesystem replays its journal, the database recovers its own log and decides which transactions to commit and which to throw away, services come up in whatever order they come up in — rarely the order the application needs — and half-finished queues stay half-finished. On a file server that is seconds. On an ERP with its database, its application server and three integrations retrying, it is not seconds.

Which leads to the one measurement that actually matters and is almost never taken: how long your application needs to be operational from a cold start. Not how long the VM takes to power on: how long until a user can raise an invoice. That figure, added to the hypervisor's two minutes, is your real recovery time for a dead node. It is the same conversation we had about >the two numbers people sign without ever calculating them, applied to a scenario far smaller and far more common than a disaster.

And sometimes HA is what causes the outage

This is the part that surprises people who have just enabled it, and it is described with admirable clarity in the manual. When a node loses cluster quorum, "the LRM waits for a new quorum to form. Until there is a cluster quorum, the node cannot reset the watchdog. If there are active services on the node, or if the LRM or CRM process is not scheduled or is killed, this will trigger a reboot after the watchdog has timed out (this happens after 60 seconds)".

Read that again with your own network in mind. A healthy node, with its virtual machines running and its storage perfectly reachable, reboots itself after sixty seconds for having stopped talking to its peers, with every piece of hardware intact. And what decides whether that conversation survives is a switch, a cable or a network card.

That is why the cluster documentation is so insistent about something that looks like a luxury on a quote: "The Proxmox VE cluster stack requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably", and it recommends "a dedicated physical NIC for the cluster traffic" because the cluster network, while it produces small amounts of data, "is very sensitive to latency". It also asks you explicitly to keep that network clear of "any high traffic purposes, like network storage or live-migration". That last sentence is the one we have seen ignored most often, and it sits behind more than one reboot that made no sense at first: a live migration of a large machine saturating the very link corosync has to cross on time.

That brutal reboot is the feature, not the bug

It is hard to accept the first time it happens to you, but that automatic reboot is what makes everything else safe. It is called fencing, and its purpose is to let the cluster be certain, beyond any doubt, that the missing node is no longer writing to shared storage. Without that certainty nothing can be recovered: the documentation warns that writing from two nodes at once "can destroy all VM data".

So the deal is explicit, and it is worth signing with your eyes open: you accept losing a healthy node for a minute in exchange for never splitting a disk in two. We think that is an excellent deal. But it is a deal, not a gift, and anyone who misses that will experience their first watchdog reboot as a betrayal by the product when it is the system doing precisely what it promised.

The requirements everybody skips

The manual lists them without ceremony, in four lines, and they are the four places where, in our experience, the clusters that arrive asking for help have broken:

When we do not deploy HA

Selling high availability is easy: nobody argues with a proposal that promises you will not go down. We find it more useful to say when we advise against it, even when that means a smaller project.

If the application already knows how to cluster itself, putting HA underneath usually gets in the way: two database nodes replicating to each other do not want the hypervisor restarting one of them on its own mid-failover. Something similar happens with machines that have hardware passed straight through — a dongle, a card, an industrial I/O device: they cannot start on another node, so promising them somewhere to land is pointless. And when there are two servers and one shared array, the honest exercise is to ask what happens if the array is what fails: if the answer is "everything", HA is protecting the component that fails least.

And then there is the most common case of all, the one nobody says out loud: when five minutes of downtime a month costs the business nothing. In those cases, high availability adds complexity, and complexity is the other name for weird failures. We have advised companies to spend that money on tested backups and a timed restore plan instead, and sleep better. None of them has since told us they regret it.

What we do before we call a cluster finished

The question we start with

When somebody asks us for high availability, the first question is not how many nodes they have or what storage they use. It is this: how long does your main application take to work again after a blackout? If nobody has ever measured it, that is this week's job, and it will probably change the whole project. If the number is high, HA will barely improve it and something else needs fixing first. And if the number is low and the business can absorb a couple of minutes, then yes: two minutes of automatic failover at four in the morning is a full night's sleep, which is roughly what high availability buys you when it is done properly.

Sources: error detection and failover times of about 2 minutes, the 99.999% ceiling, the requirements (three nodes, shared storage, hardware redundancy, server components), watchdog behaviour 60 seconds after quorum loss, the purpose of fencing and maintenance mode — Proxmox VE ha-manager documentation chapter and High Availability wiki; sub-5 ms latency, dedicated NIC, the warning about network storage and live migration, redundant links and the QDevice as a third vote — cluster manager chapter. The chapter publishes its own availability table (99.9% → 8.76 hours; 99.99% → 52.56 minutes; 99.999% → 5.26 minutes a year); the conversions into minutes and seconds are direct arithmetic over 525,600 minutes a year. Cover image: photograph of an emergency stop button, CC0, via the WordPress photo directory.