Ceph to Tentacle: during the upgrade your cluster runs two versions at once

Ceph 19.2 Squid has an estimated end of life of 31 October, and Ceph 20.2 Tentacle has been in Proxmox's enterprise repository since 20 May. With those two dates on the table, this autumn's maintenance window has stopped being hypothetical: it is a point in somebody's calendar. And inside that window something happens that almost no plan accounts for, because it sounds like a detail and it decides the entire risk: for a few hours — sometimes days — the cluster is not running Squid and is not running Tentacle. It is running both at once.

We run Proxmox VE with Ceph storage in production, spread across several data centres, so this is written from the side of whoever opens the window. Let us start with what we have not done yet: we have not put Tentacle into production for any customer. The reason is at the end, and it has nothing to do with performance.

The announcement still doing the rounds no longer describes reality

The original announcement thread, from 9 January, is still the first result almost everyone finds when searching for "Proxmox Ceph Tentacle". It says two very quotable things: that Tentacle is "now available on the Proxmox Ceph test repository for installation or upgrade", in preview state, and that Ceph 19.2 Squid "will stay supported until September 2026 for the time being". Both were true when they were written. Neither describes what is in front of you in September 2026.

On 2 June, a Proxmox team member put it on the forum with no ceremony at all: "Ceph Tentacle is in the enterprise repository since May 20th." On 30 June another confirmed that 20.2.2 was in the test repository, and on 30 July a third announced that "Ceph 20.2.2 is now available in the ceph-tentacle enterprise repository." The official upgrade guide — the one you will follow step by step — no longer mentions a preview anywhere: the example it gives you is the enterprise line, with no-subscription and test as alternatives if you have no subscription. The door opened on 20 May, with no announcement of its own, and the thread everybody links to still says it is shut.

In Proxmox, a package's status is checked in the repository, with apt policy ceph-common, not in the forum thread where it was announced. An announcement is a snapshot of one day. The repository is the present.

The upstream dates complete the picture. Squid was released on 26 September 2024 and the Ceph project gives it an estimated end of life of 31 October 2026. Tentacle was released on 18 November 2025 and runs until 1 June 2027. Between today and Squid's expiry there are less than two months.

apt does not upgrade your cluster

An apt full-upgrade changes files on disk. The processes actually serving your storage are still, in memory, exactly the ones from before. That is why the official procedure devotes three separate steps to restarting three different things: systemctl restart ceph-mon.target for the monitors, systemctl restart ceph-mgr.target for the managers and systemctl restart ceph-osd.target for the OSDs. And that is where the state this article is about comes from, because those three restarts do not happen at once, nor should they. The monitors, too, go one node at a time.

The mixed state is designed, not tolerated

Ceph is built to work this way. The orchestrator documentation says so plainly when describing an upgrade: "Each daemon is restarted only after Ceph indicates that the cluster will remain available." And you have a command for checking where you are, ceph versions, which returns the daemon count per version: how many monitors on each, how many OSDs on each. For much of the window that count will be split, and that is precisely what is expected.

One detail is worth flagging before it bites you, because you will find it yourself if you read both sets of documentation: they do not agree on the order of the first two steps. The Proxmox wiki has you restart monitors first, then managers. The Ceph documentation, describing an upgrade with its own orchestrator, says the opposite: "The upgrade order starts with managers, monitors, then other daemons." They are different tools and different procedures, so neither is wrong, but if you have read up on both you will stop halfway through the window wondering which one to follow. Follow the Proxmox one: it matches the packages you have installed and it is the one their team tests.

"Restart OSDs on one node at a time"

The wiki bolds the part that matters: "Restart all OSDs. Only restart OSDs on one node at a time to avoid loss of data redundancy." It is the instruction that decides how long your window lasts, so it deserves to be understood rather than obeyed blindly.

The arithmetic is short. A pool with Proxmox's default values keeps three copies and requires two to serve: size 3, min_size 2. If your failure domain is the host, which is the norm in a hyperconverged cluster, restarting a node's OSDs momentarily leaves every placement group that had a copy there down to two. You keep serving, nobody notices, and that is what it is designed for. But for those minutes you are one failure away from a group dropping to a single copy, falling below min_size and refusing all I/O — not just writes: reads from that group too. Restarting two nodes at once saves you an afternoon and costs you the entire margin that sentence buys you.

And one clarification that prevents a false sense of safety: noout does not protect you from this. It prevents rebalancing, not degradation. With the flag set, that node's groups sit at two copies exactly as they would without it. Put the way we always put it: failure is inevitable, an outage is a design decision. During the window you are deliberately giving up a level of redundancy, and that should be decided wide awake with somebody watching, not on a Friday at seven. It also pays to have already worked out how much your cluster can take with one node missing, because if you are tight on space the window stops being an inconvenience and becomes a different kind of risk.

noout: "optional, but recommended"

The procedure says, verbatim: "Set the noout flag for the duration of the upgrade (optional, but recommended)." The command is ceph osd set noout. Here we will take a side: calling it optional is technically true and operationally misleading.

Without that flag, Ceph does what it always does with an OSD that stops answering: it waits, then takes it out of the cluster. The parameter governing the wait, mon_osd_down_out_interval, defaults to ten minutes. Taking a whole node of OSDs out of the cluster amounts to ordering every copy that lived there to be recreated across the remaining disks. An apt full-upgrade that drags on, a node slow to boot, a wait for a service to stop cleanly — and you have crossed the threshold. The unpleasant part is that once started, the operation does not cancel itself: you bring the node back, Ceph notices and undoes some of the work, but in the meantime your cluster has been shifting terabytes that did not need shifting, mid-window, competing with production.

And then there is the step genuinely forgotten: clearing it at the end. A cluster left with noout set has its self-healing switched off and nobody will remember. It works perfectly until the day a disk actually dies and nothing happens, which is the worst thing that can fail to happen.

There are two answers to "what version is my Ceph running?"

The first is the one dpkg gives you: the installed binaries. The second is what the cluster has declared, and it is changed with a separate command, right at the end: ceph osd require-osd-release tentacle. The procedure pairs it with a warning that says more than it appears to: "Before raising the minimum required OSD version, you should ensure all OSDs got upgraded successfully and report running a Ceph 20.2 version."

What you have declared is read with ceph osd dump | grep require_osd_release. And here is the warning that motivates half this article: do not confuse it with ceph mon dump | grep min_mon_release. That second command is in the wiki, yes, but in the monitor step and to verify that step; it returns min_mon_release 20 (tentacle) as soon as the monitors have restarted, and it will keep saying 20 even if you never run require-osd-release in your life. It is the command most people mistake for the finish line, because it shows exactly the number you are hoping to see.

The practical consequence: a cluster can spend months with every binary on 20.2, the dashboard green and the declaration still on 19. Nothing breaks. No dramatic alert appears. It simply keeps running in a compatibility mode nobody asked for, and anything that depends on the whole cluster being new never switches on. It is the same pattern we described with Fast EC, the Tentacle performance improvement that ships switched off and has to be enabled by hand on each pool. If you are going to open the window, have it written down in advance which commands run after everything goes green: that is the part of the procedure lost when the dashboard turns green and everyone goes to bed.

CephFS is where you pay in capacity

If you run CephFS, the procedure asks two things of you before touching the MDS daemons: disable standby_replay and reduce the number of ranks to one. The commands are ceph fs set <fs> allow_standby_replay false and ceph fs set <fs> max_mds 1. This is the point where you give up service capacity and not merely redundancy: you go from spreading the filesystem across several metadata servers to concentrating it on one, with everything that means on a Monday morning.

It is also the only place where the documentation asks you, in brackets and twice, to write something down: if you plan to restore it later, take a note of whether allow_standby_replay was enabled, and take a note of the original number of MDS daemons. That is a very unusual request in a technical manual, and it is there for a reason obvious to anyone who has done this before: in the small hours nobody remembers how many ranks there were. Without that note, the upgrade ends with a single-rank CephFS that works, performs worse, and that nobody connects to a maintenance window six weeks ago.

The prerequisites are a work order

There are three and they take thirty seconds to read: Proxmox VE 9.1 or higher, specifically the pve-manager package at version 9.1.4 or newer; Ceph on Squid, specifically 19.2.3-pve3 or newer; and a healthy cluster. That last one they write with an exclamation mark, which is uncommon in technical documentation: "The cluster must be healthy and working!"

Read together, those three prerequisites order the work. If you are on Proxmox VE 8, your next step is Proxmox and Ceph comes after it — and that branch already has a closed calendar of its own, so you probably have two windows ahead of you rather than one. The third is the one people in a hurry skip: a cluster already carrying a degraded placement group, or an OSD that has been flapping for weeks, is not a candidate. The upgrade fixes none of that; it multiplies it, at exactly the moment you have also deliberately lowered your redundancy.

Why we have not, yet

The reason is no longer the repository, because that excuse ran out on 20 May. It is cadence, and bedding-in time. Tentacle has had five releases: the initial 20.2.0 in November and four point releases — 20.2.1 on 6 April, 20.2.2 on 16 June, 20.2.3 on 5 August and 20.2.4 on 19 August. Two of them in the same month, fourteen days apart. That speaks of a release still in motion, not of a bad one.

There is also an asymmetry worth looking at before arguing about versions: the most recent enterprise-repository announcement we could verify is the one from 30 July and it covers 20.2.2, while upstream is already on 20.2.4 from 19 August. That lag is precisely what you buy with the subscription: a deliberate delay while somebody watches. Check yours with apt policy ceph-common before assuming which version the window will install. Setting Tentacle up in a lab now makes complete sense, because you rehearse a procedure before performing it, and anyone running CephFS with several ranks should rehearse it twice.

On performance, which is what will circulate: in the announcement thread, one user posted that after upgrading he went from 60,207 to 68,516 4K random IOPS (13.8% more) and from 7,488 to 10,120 64K sequential (35.1%). Credit where it is due, he published more than most: three Intel NUC14 nodes with NVMe and the rbd bench script. The objections lie elsewhere. He measured it on a replicated pool of size 2 and min_size 2, which is neither Proxmox's default nor that of almost any serious production cluster; he measured it on 17 January against 20.2.0, four point releases ago; and there is no repetition and no second source. It may be perfectly true on his machine. It is not a figure we would put in a proposal, and it is certainly not a reason to bring a window forward.

Four questions before opening the window

1. How long will the cluster be on two versions, and who is watching during that time? There is no correct answer — some clusters take an afternoon and others three days — but there has to be a written answer before you start, and a name next to it.
2. What happens if a disk dies while a node of OSDs is restarting? If the answer is "I do not know", the window is not planned yet. If the answer is "that group falls below min_size and stops serving I/O until the node comes back", at least it is an answer, and it decides what time you open.
3. Who decides to stop halfway, and what does stopping mean? Leaving the cluster on two versions for a week is a valid and supported decision. Leaving it that way without anybody knowing is not. The difference between the two is one email.
4. Is the previous MDS count and standby_replay state written down somewhere other than in somebody's head? And while we are at it, who runs require-osd-release, who checks ceph osd dump and who clears noout once everything is green?

All four are about change governance, which is how systems actually fall over: the classic work by Oppenheimer, Ganapathi and Patterson on three large internet services already pointed in 2003 to operator error — and within it, configuration — as the leading cause of user-visible outages in two of the three services, ahead of hardware. A Ceph upgrade is a manual configuration change to the place where all your data lives.

Sources

The original announcement from 9 January 2026 — with the sentences about the test repository, the preview state and Squid being supported "until September 2026" — and the user's performance report, with his hardware and script, are in that Proxmox forum thread; they are quoted with their date on purpose, because they describe the state of that day and not of today. Tentacle entering the enterprise repository (20 May 2026) is confirmed by a Proxmox team member on 2 June in this thread. Tentacle 20.2.2 moving to the test repository (30 June 2026) and to the enterprise one (30 July 2026) are messages from the Proxmox team in this other thread. Prerequisites (Proxmox VE 9.1 / pve-manager 9.1.4, Ceph 19.2.3-pve3, healthy cluster), the enterprise, no-subscription and test repository lines, daemon restart order, the noout flag described as optional but recommended, the instruction to restart OSDs one node at a time, the CephFS commands and ceph osd require-osd-release tentacle with its preceding warning: Proxmox VE wiki, "Ceph Squid to Tentacle". In that same guide, ceph mon dump | grep min_mon_release appears in the monitor step and serves to verify that step; that it does not work as a check for require-osd-release, and that ceph osd dump | grep require_osd_release is what you use instead, is our own clarification about how the MonMap and the OSDMap work. Release and estimated end-of-life dates for Squid (2024-09-26 and 2026-10-31) and Tentacle (2025-11-18 and 2027-06-01), and the dates of releases 20.2.1 to 20.2.4: the official Ceph releases index and the Tentacle release notes. The sentence about each daemon being restarted only when the cluster will remain available, and the "managers, monitors, then other daemons" order, come from the cephadm upgrade documentation, and are quoted as what they are: another deployment tool's procedure. Default value of mon_osd_down_out_interval (ten minutes): "Monitor/OSD Interaction". Proxmox pool defaults (size 3, min_size 2): Proxmox VE manual, "Deploy Hyper-Converged Ceph Cluster" chapter. Operator error as the leading cause of user-visible outages in two of the three services studied: Oppenheimer, Ganapathi and Patterson, "Why Do Internet Services Fail, and What Can Be Done About It?", USENIX 2003. Quotations are reproduced in their original language; the text explains in each case what they say. The opinions are ours: that calling noout "optional" is misleading, that the cadence of point releases says more than the version number, that the enterprise repository's lag is what you buy with the subscription, that the forum figure is no use in a proposal, and that it is not yet time to put this into customer production, come from running Proxmox VE with Ceph, not from the sources cited.