Ceph changes cipher: rebooting the virtual machine does not refresh the key

The most important sentence in the Ceph key migration Proxmox published on 9 September is not in the announcement. It is buried in the documentation, in the list of how to refresh clients, and it is six words long: "A guest reboot is not enough." Rebooting the virtual machine does not do it. You have to live-migrate it, or stop it and start it. And if you get that wrong, you do not find out that day: you find out when a ticket expires, which can be minutes or can be days.

We run Proxmox VE with Ceph storage in production, so this lands on us and on most of the clusters we maintain. On 9 September at 02:39, on the official forum, Proxmox opened the thread "Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon", and with it a procedure that is not optional and looks nothing like an apt upgrade. This article is about what it actually is, what has changed since August, where the trap sits, and in what order we do it.

Your cluster went red on its own

The origin is August. On 19 August Ceph shipped 20.2.4 and 19.2.6 together to close four CVEs, and one of them — CVE-2025-30156, an authentication bypass in cephx caused by misuse of AES-CBC — is not fixed by installing the package. The key type has to change. We covered it at the time in our analysis of those four CVEs, where we already said one of them is not closed by the package on its own. This is the second half of that.

Proxmox describes it without hedging: "Recent security findings and fixes in Ceph make it necessary to upgrade ceph and migrate authentication keys from the aes to the aes256k cipher", and adds the caveat that sets your priority: "especially if your Ceph service networks are not isolated". If your Ceph service network really is separated, you can breathe a little. If it shares a switch with everything else, you cannot.

The first thing people notice is that the cluster goes to HEALTH_ERR without anyone touching it. It is not a failure. 19.2.6 and 20.2.4 add new health checks on key ciphers — the reference table in the documentation lists six — and two of them — AUTH_INSECURE_SERVICE_KEY_TYPE and AUTH_INSECURE_SERVICE_TICKETS — carry error severity. The documentation clears it up: "This does not mean that storage access or a Ceph service has failed." Storage keeps working; what changed is what the traffic light measures.

What changed since August: now there is a safety net

In August, rotating cephx keys by hand was ten steps and a red cluster for the duration. That is what Proxmox has changed, and it deserves credit: they added a migration helper — /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys — and, more importantly, improved Ceph's key-staging mechanism so the handover is gradual: "both keys remain valid while you refresh clients, including running guests and CephFS mounts, to use the new key".

That grace period is what the new procedure really adds, and it comes with concrete requirements worth checking before you start: pve-manager 9.2.17 or newer, and for staged client-key rotation, Ceph 19.2.6-pve3, 20.2.4-pve3 or newer on every monitor. The reason: an older monitor promotes a staged key the first time anybody uses it, and that ends the grace period for the whole cluster. So the helper checks on every run that every monitor supports the feature, and refuses to stage if one does not.

You do not set the clock on this one

The Proxmox notice: if you retire the old key or restrict ciphers before every client has been refreshed, "incompatible or not-yet-refreshed clients may see I/O failures on reconnecting or when existing service tickets expire, which can be minutes or days after the change". And: "Existing IO can appear to work until a reconnect and then fail."

Translated into real life: you run the command at 23:00 on Saturday, look at the cluster, all green, every VM writing, and go to bed. On Tuesday mid-morning a machine nobody has touched loses its disk. It is not that the change failed late; it is that this change never fails at the moment you make it. You set the maintenance window and the expiry of a service ticket sets the outage. Two different clocks, and the second one is not in your calendar.

Hence the rule in the documentation: "Do not use --force to bypass a blocker." A blocker can be a session still holding the old key, but it can also be a version check that does not add up, or daemon keys for hosts that no longer exist; the thread has examples of all three. Overriding it does not speed the migration up, it moves the failure to a day when you are no longer watching.

What counts as "refreshing a client"

Before refreshing anything you need to know which kind of client each workload uses, because that decides whether the userspace library you already updated with the package is enough, or whether you depend on the machine's kernel. The documentation sums it up in three lines:

• Virtual machine with RBD disks: userspace, unless krbd is enabled.
• Container on RBD: always the kernel client. Always. No variant.
• CephFS mount: kernel, unless fuse is enabled.

To refresh an affected VM, the instruction is to live-migrate it from the web interface, or stop it and start it. And right after: "A guest reboot is not enough." It makes perfect sense once you think it through — the QEMU process on the host is what holds the connection to Ceph, and restarting the operating system inside does not touch it — but it is the exact opposite of what anyone told to "restart the clients" will instinctively do. If someone on your team does the round of reboots from inside the VMs and tells you it is done, it is not done.

Containers and other RBD clients get stopped and started. Idle CephFS mounts are refreshed by the helper itself, and it leaves busy or unresponsive ones alone: once nothing is using them, you run it again with --apply and they are retried. And there is one detail we think is the easiest to forget in a cluster with night work: backups, restores, disk imports and clones on Ceph storage can retain the key they started with. Let them finish before confirming. A long backup that kicked off at two in the morning is a client holding an old key, even if the interface does not make it look like one.

The gate that decides whether you can finish: kernel 7.0

The Ceph programs in the updated Proxmox VE packages support aes256k. Kernel clients do not all: you need a running kernel 7.0 or newer. And that sentence, which reads in the thread like one more technical requirement, is actually the one that decides whether your cluster can close this migration this month or not.

It is worth reading the requirement carefully, because it says running kernel, not Proxmox VE version. Let us do the arithmetic with the real versions: Proxmox VE 9.2, released on 21 May 2026, ships 7.0 as the new default; Proxmox VE 9.1 ran 6.17 and 9.0 ran 6.14; and Proxmox VE 8 uses 6.8 since 8.2, with 6.14 available as an option since 8.4. That said, across the whole 9 series the 7.0 kernel has been available opt-in since 2 April — apt install proxmox-kernel-7.0 and reboot — so a node on 9.1 can meet the requirement without a major version change. The check is not "which Proxmox version am I on?", it is uname -r on every node hosting kernel clients, and it has to be done node by node: in a cluster that has been upgraded in stages, different kernels coexist quite naturally.

For anyone still on Proxmox VE 8 the chain closes itself, and not pleasantly: the 8 branch stopped receiving support in August, has no kernel 7.0, and therefore cannot complete the last step of this migration no matter what else it installs. The right move there is what the documentation says: leave that user's key alone and the old cipher enabled, and mute the warning in the meantime. Do not restrict. Restricting with an incompatible client out there is precisely the scenario from the previous section.

What is already happening in the thread

Six days after the announcement, the thread is still alive and paints a fairly honest picture of how this is going. Some people have done it without incident on a three-node cluster and report that the uncomfortable part was not knowing when to proceed, because many steps do not print a clear green "this went fine". Others are asking for the procedure to be pulled out of the general documentation into its own guide, arguing that it is a one-off and risky migration, not a day-to-day task. Both strike us as reasonable complaints.

And there are already people with half-migrated clusters. Two users have posted the same message after running the rotation, Not a proper rbd authentication file:, each about a different keyring file — CEPH00.keyring in one case and ceph.keyring in the other — and one of them with a virtual machine that will not start after being shut down; he explains the cause was a copy-paste that mixed options from two different steps. Two more are stuck at the preflight check: one because a monitor of theirs lives on a machine Proxmox does not manage and the helper cannot verify its version; the other because the cluster still reports insecure keys for two managers that no longer exist.

We are not telling this to point at anybody: we are telling it because all three cases say the same thing. The risk in this change is not in the cryptography, which is the part that is already solved and tested. It is in the inventory: knowing which clients you have, which ones Proxmox does not manage, which ones keep a copy of the key in a file of their own, and which ones have not reconnected in weeks. The helper does not know that. It says so itself when it lists suspicious sessions: they are hints, not a complete inventory.

The order we follow

This is not an alternative to the official procedure — the official one is good and you should follow it — but the order in which we fit it together so the inventory part does not end up last:

1. The inventory, before any command. It is the same first point we put down in August, and what the new procedure adds is that the inventory now also decides who you can confirm and who you cannot. Which Ceph users exist, which workload uses each one, which are kernel clients, what lives outside Proxmox, and where each key has been copied. This is done in a spreadsheet, and it is the part that decides whether the migration goes well.
2. Versions and kernel. pve-manager 9.2.17 or newer, Ceph with the -pve3 suffix or later on every monitor, and a running kernel of 7.0 or newer on every node hosting kernel clients.
3. Cluster-owned keys first (--rotate-cluster-keys). This is the safe step: it does not touch storage keys or client.admin, existing tickets stay valid, and the old cipher stays enabled. It clears the two error-severity checks.
4. Patience with the warning that remains. The one about rotating service keys can take a few hours to clear and clears itself. It does not block the final restriction, so you do not need to wait for it, but it is worth knowing before you start second-guessing.
5. Stage and refresh, user by user. Live migration or stop-start; never a guest reboot. Let backups and clones finish.
6. Check with the right tool. pveceph auth status shows current and pending key ciphers; ceph auth ls does not list pending keys, so looking there shows you a cleaner cluster than you actually have.
7. Confirm only at the end, and only if the inventory is closed. If an incompatible client remains, leave its key and the old cipher, and mute the warning. A muted warning with a review date is better risk management than a restriction that takes a container down on Tuesday.
8. And do not delete the journal. /etc/pve/priv/cephx-key-migration.json records progress and contains the old secret keys: protect it, and keep it until the migration is complete. Delete it earlier and you lose what you need to resume.

It is worth knowing there is a way back and an emergency exit, because knowing it changes how you approach the risky step. A staged key can be aborted with --abort-staged-key, which restores the current key to every managed copy while both remain valid. But it is not one command, it is three: abort, refresh the clients back — including disconnected ones and external key copies — and confirm with --confirm-abort-clients-refreshed; and if your monitors are not on 19.2.6-pve4 or 20.2.4-pve4, the ones that can identify each session's key, every visible client of that user has to disconnect before confirmation. Going back costs almost as much as going forward. And if someone overshoots and restricts the ciphers while locking out the administrative key, there is the startup option mon_auth_emergency_allowed_ciphers, which replaces the allowed list on a monitor so you can recover access. It is there to recover access and nothing else: while it is in place Ceph raises AUTH_EMERGENCY_CIPHERS_SET and the helper refuses to do the final restriction.

Two migrations in the same quarter

The calendar is what turns this into a management decision and not just a sysadmin task. The packages reached the no-subscription repositories recently, after a long period of internal testing, and on 9 September Proxmox wrote that they planned to roll them out to the enterprise repositories "in the second half of next week", warning it might slip by a few days depending on QA and feedback in the thread. That next week is this week. If your clusters are on enterprise, the update that lights up the new health checks may reach you within days, and the HEALTH_ERR with it.

And it does not come alone. The same notice reminds you that Ceph 19.2 Squid has an estimated end of life of 31 October 2026, with the recommendation to move to Tentacle while Squid is still supported — and that requires Proxmox VE 9.2 or newer, so anyone on 8 has another migration ahead of them first. Where that date comes from, and why we treat it as an estimate rather than a guarantee, is in our article about the commit that moved it.

Our opinion, and it is opinion rather than data, and it is not new either — we wrote it in August, in these words: "do not put Tentacle in the same window" —: do not stack them. The key rotation and the Squid-to-Tentacle jump are two changes that touch the same thing and fail differently, and putting them in the same window saves one night and complicates next week's diagnosis. If something breaks on Thursday, you want to be able to say what changed on Saturday without picking between two candidates. We do the rotation first, let it settle for a few days with the old cipher still enabled, and only then touch versions.

In this change, the check that it went well is not that the cluster is green. It is that no client is still holding the old key. Those are two different questions, and only the second one is answered by the inventory you did before you started.

Note on sources

The announcement and the quoted sentences about the procedure, kernel 7.0, external clients and the enterprise repository schedule come from the thread "Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon" on the official Proxmox forum, posted by a team member on 9 September 2026 at 02:39 (forum time). The steps, the names of the six health checks, the version requirements, the client type table, the sentence "A guest reboot is not enough", the warning about --force, the cephx-key-migration.json file, pveceph auth status, --abort-staged-key and mon_auth_emergency_allowed_ciphers are in the "Migrate Cephx Keys from aes to aes256k" section of the Proxmox VE reference documentation (pve-docs, pveceph chapter). Attributing CVE-2025-30156 to the old method comes from that same section; the four CVEs and versions 19.2.6 and 20.2.4 come from Ceph's combined advisory of 19 August 2026. The kernel versions per Proxmox VE branch come from the Proxmox VE Kernel wiki; that 7.0 is the default in 9.2, from the official Proxmox VE 9.2 announcement of 21 May 2026; and that it is available opt-in across the 9 series, from the thread "Opt-in Linux 7.0 Kernel for Proxmox VE 9 available" of 2 April 2026, which is also where the install command comes from. The user reports and the quoted error message are public posts in the same thread, read on 15 September 2026; we have not reproduced them with names. Squid's estimated end-of-life date (31 October 2026) is the one given in the Proxmox notice itself. What is our opinion here — not stacking the two migrations, the working order, and the risk sitting in the inventory — is stated as such in the text.