Ceph v20.2.5 Tentacle came out on 2 November. In its notable changes, the security section is one line long: "CephX: hardened service ticket verification". No CVE number, no urgent mailing-list advisory, no press note. One line. And it is the second half of the fix many of us rushed through in August, with four CVEs on our backs and the cluster sitting in HEALTH_ERR while we rotated keys by hand.
That is what this post is about, and it is not a Ceph problem: patch queues are built out of identifiers. A scanner asks for CVEs. A steering committee asks for CVEs. A compliance report asks for CVEs. Anything that arrives without a number stays out of the queue, even when it changes precisely what your cluster will accept as valid.
What August closed
On 19 August Ceph published a combined advisory: 19.2.6 Squid and 20.2.4 Tentacle fixed four CVEs. The one that matters here is CVE-2025-30156: CephX was using AES in CBC mode without authentication. In plain terms: the ticket was encrypted, but nothing guaranteed it had not been tampered with on the way. An attacker holding legitimate CephX permissions could flip one bit in a service ticket and set the allow_all field to true. Unlimited permissions for one bit.
The fix was a new key type, aes256k (AES256-CTS-HMAC-SHA384-192, the one from RFC 8009), plus manual rotation of every CephX key, with ceph orch daemon rotate-key deliberately disabled in those releases. We wrote it up at the time with all ten steps and the noise they made: the package closed three CVEs and the fourth was on you.
That fix is cryptographic. It makes a tampered ticket fail to decrypt cleanly. What it does not touch is the verifier's posture: what the daemon does when decryption did go fine.
What it did not close
Two days after the advisory, on 21 August, Matthew Heler opened pull request #71231 against Ceph's main branch. His description of the problem opens with a sentence of the kind you read twice:
"The verifier took a service ticket at its word once it decrypted, so nothing checked that the fields inside it were the ones the key server put there."
Decrypting correctly and being legitimate are not the same thing, and the verifier treated them as if they were. As long as the authenticated cipher holds, the difference is theoretical. The day it does not hold —a leaked key, an implementation flaw, or simply an entity still using the old cipher because nobody rotated it— that difference is all there is.
The four things the verifier no longer swallows
The change lives in src/auth/cephx/CephxProtocol.cc, inside cephx_verify_authorizer, and it is four new families of rejection:
| Field | What it now rejects | Why |
|---|---|---|
auid |
Any value other than CEPH_AUTH_UID_DEFAULT. |
A legacy field the key server no longer varies. If it holds anything else, someone touched it. |
caps.allow_all |
Any ticket that arrives with it set. | Monitors never issue a service ticket with unrestricted capabilities. This was exactly August's bit. |
caps buffer |
Capabilities that fail to decode, or trailing bytes after the string. | A well-formed buffer holds one string and ends. Whatever comes after it has no business being there. |
| timestamps | A ticket creation time that does not line up with the session key's, with 300 seconds of slack (const utime_t slop(300, 0)); and, in the same block, reversed dates or impossible nanoseconds. |
A recycled or back-dated ticket stops sliding through the easy way. |
None of the four is spectacular on its own. Together they change something structural: a ticket stops being trustworthy by virtue of having opened cleanly and becomes a document that is checked field by field. That is boring defence in depth — the kind you only notice the day the layer in front of it fails.
The five minutes do not measure your clock
The first thing anyone reading "300 seconds of slack for clock steps" thinks is: I need to check NTP on the clients. You don't. And it is worth saying plainly, because the fastest way to lose an afternoon is to go poking at chrony over a change that says nothing about chrony.
Both timestamps being compared are stamped by the same machine: the monitor. One on the ticket, one on the session key it was issued against. The slack is there to absorb a clock step on that monitor or a scheduling delay in issuing, not drift on your clients. A client with the wrong time does not fail here.
Ceph was already watching the clock elsewhere, on a different scale: MON_CLOCK_SKEW fires when the skew between monitors exceeds mon_clock_drift_allowed, which defaults to 0.05 seconds. Fifty milliseconds there, three hundred seconds here: six thousand times more tolerant, because they are not measuring the same thing or for the same reason.
Each branch gets it on a different day
Here is the part that decides whether this is yours today, next week or in December:
| Branch | Merged | Ships in |
|---|---|---|
main |
27 Sep 2026 (#71231) | — |
| Umbrella (21.x) | 7 Oct 2026 (#72126) | v21.2.0, 28 Oct 2026 (first stable of the series) |
| Tentacle (20.x) | 12 Oct 2026 (#72127) | v20.2.5, 2 Nov 2026 |
| Squid (19.x) | 3 Nov 2026 (#72128) | v19.2.8 — estimated 7 Dec 2026 |
One detail explains more than it looks like: 19.2.7 shipped on 12 October and the Squid backport did not merge until 3 November. That is why 19.2.7 does not carry it. If that was your last Squid update, you have the cryptographic half of the fix and not the verifier half, and there is no line in its notes telling you so, because absence does not get documented.
And the kicker, with the caution it deserves: Ceph's release schedule puts 19.2.8 as the final Squid release, and that same page warns in small print that its estimates are "generated daily by a script that was written with the help of an LLM" and "not reviewed by the release team". It is an estimate, not a commitment, and we know that first-hand: in July we wrote that Squid would end on 19 September, and the date has moved twice since. That it moves is exactly why it is an estimate. What does not move is the shape of it: if your cluster is on Squid, this layer reaches you in the very package the branch signs off with, so the window in which you install it and the conversation about where you jump next are the same meeting.
If August's rotation stalled halfway, this one is more yours
August's fix was not "install it and move on": every key had to be rotated to aes256k one at a time, and in plenty of places the daemons got rotated and the clients were left for later —or the other way round. In Proxmox environments we saw it through the most repeated trap of all: rebooting the virtual machine does not refresh the key.
Ceph does tell you, in some detail. These five health checks are the ones you will run into, and they are not all there are: AUTH_INSECURE_SERVICE_KEY_TYPE (daemons on the old key), AUTH_INSECURE_CLIENT_KEY_TYPE (clients), AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE (rotating keys, which also expire on their own), AUTH_INSECURE_SERVICE_TICKETS (monitors still issuing on the old cipher) and AUTH_INSECURE_KEYS_ALLOWED (the old cipher is still on the allow list). Five places where a single cipher change can get stuck tells you how easy it is to leave half-done.
And here is the twist: for any entity still on the old cipher, August's crack is still open on its path, and the new verifier is the only thing left in between. This CVE-less line matters more the worse your August homework went. Exactly the opposite of what its missing number suggests.
The counterweight, which also needs saying: the old cipher is still allowed on purpose. aes256k support in the Linux kernel client started in 7.0 —though August's own advisory adds that it has been backported to CentOS Stream 9 and 10, so the kernel version number is not enough: you have to ask your distribution—. If you close auth_allowed_ciphers with a CephFS or RBD mount that cannot speak it hanging off the cluster, you drop it. No scanner makes that call: you do, knowing what each client mounts.
The ten-minute audit
All of this is read-only. It changes nothing, needs no maintenance window, and tells you whether this is yours:
# 1. What version each daemon ACTUALLY runs (not the repo's)ceph versions
# 2. Any authentication warning open?ceph health detail | grep -i AUTH_
# 3. Which cipher the monitors use to issue and for new keysceph --format=json mon dump | jq -r '.auth_service_cipher.name, .auth_preferred_cipher.name'
# 4. Which entities are still on the old key typeceph --format=json auth dump-keys
If steps 2 and 4 come back clean and step 1 returns 20.2.5 or 21.2.0 everywhere, you have both halves. You are done. If not, the order of the fix matters, and the last step is the one that cuts:
# Rotate each pending entity; the resulting keyring goes to ITS daemonceph auth rotate --key-type=aes256k mds.a > mds_a.keyring
# Make new keys and new tickets be born on the new cipherceph mon set auth_preferred_cipher aes256k
ceph mon set auth_service_cipher aes256k
# LAST. Shuts the door on the old cipher: drops clients that cannot speak aes256kceph mon set auth_allowed_ciphers aes256k
That last one is not run "to leave things tidy". It is run when step 4 is empty and you know which kernel every client mounts. If you do not know, that is today's job — not the command.
When this is NOT urgent for you
- ✗If you already run 20.2.5 or 21.2.0 everywhere and
ceph health detailshows noAUTH_: do not open a window for this. It is already done. - ✗If you are on Squid 19.2.7: do not rush. No CVE has been assigned to this change and there is no exploitation advisory. What deserves a meeting is not the changelog line, it is that 19.2.8 closes the series.
- ✗If your cluster lives in one room on three nodes: this is nowhere near your biggest crack. Fixing what takes you offline on a Tuesday afternoon before fixing what leaves you exposed is a defensible order.
What we take from this
In our own operation all of this boils down to something deeply unheroic: somebody reads the release notes in full, not the CVE feed. It is what separates "we have it" from "we thought we had it", and there is no trick to it: it is maintenance, the boring word for the thing that holds up everything else.
We repeat it often because it is the axis of how we design: failure is inevitable, an outage is a design decision. Here there was not even a failure. There was a line in a changelog, one day in November, and the decision of whether anyone reads it.
Sources (verified at origin): release notes for v20.2.5 Tentacle (2 Nov 2026), v21.2.0 Umbrella (28 Oct 2026) and v19.2.7 Squid (12 Oct 2026) — ceph.io; the change and its code in pull requests #71231 (main), #72126 (umbrella), #72127 (tentacle) and #72128 (squid, milestone v19.2.8); schedule and series end-of-life — ceph.io/users/releases; AUTH_INSECURE_* health checks and auth_* options — docs.ceph.com (health checks) and CephX Config Reference; mon_clock_drift_allowed — Monitor Config Reference.
Do you know which Ceph version each daemon in your cluster is running right now?
We design and run distributed storage with Ceph in production — upgrades, key rotations and release notes read in full. If you want someone to go through your cluster with this list in hand, let's talk.
Talk to everyWAN