Ceph CVE-2025-30156: rotate CephX keys off insecure `aes` cipher to `aes256k` #71

New issue

Open

opened 2026-09-16 15:53:51 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-09-16 15:53:51 +00:00

Collaborator

Summary

Ceph's cluster health has been HEALTH_ERR since 2026-09-16, triggered by six new AUTH_INSECURE_* checks that Ceph 20.2.4 (already partially rolled out via ccnet-proxmox-nodes-runbook/proxmox-update.yml node testing) added to flag every existing CephX key still using the old aes cipher (CVE-2025-30156 — an authentication-bypass vulnerability). This is not an active outage or data-risk issue — the cluster is otherwise fully healthy (273 active+clean PGs) — but it does block proxmox-update.yml's Ceph-health precheck from proceeding on further nodes, since that check correctly refuses to drain a node while HEALTH_ERR is present.

Current plan: finish rolling proxmox-baseline.yml/proxmox-update.yml across the remaining Proxmox nodes first (tracked separately), muting the specific health checks in the meantime so the update roll isn't blocked. Come back to this issue once all 7 nodes are on current versions and do the actual key rotation as its own deliberate, tested piece of work — not rushed, per Proxmox staff's own guidance below.

What this actually is

CVE-2025-30156: CephX's old aes cipher (AES-128-CBC) has no HMAC and a hardcoded IV — no authentication of the ciphertext, and identical plaintexts always produce identical ciphertexts. An attacker with a low-privilege key or network access to CephX traffic can flip bits in ciphertexts to forge credentials, including forging admin credentials.

Fixed in Ceph Tentacle 20.2.4+ / Squid 19.2.6+, which introduce a new key type aes256k (AES256-CTS-HMAC-SHA384-192, per RFC 8009: random confounders per operation, HMAC-SHA384 auth, ciphertext stealing against padding-oracle attacks) and six new health checks that fire against every existing key still on the old cipher until rotated:

AUTH_INSECURE_CLIENT_KEY_TYPE (WARN) — 9 client entities (client.admin, all bootstrap-*, client.crash, client.swarm-data-fs) on aes
AUTH_INSECURE_KEYS_ALLOWED (WARN) — mons configured to permit aes auth
AUTH_INSECURE_KEYS_CREATABLE (WARN) — mons configured to permit creating new aes keys
AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE (WARN) — mon/mds/osd/mgr rotating service keys on aes
AUTH_INSECURE_SERVICE_KEY_TYPE (ERR) — 19 service entities (every OSD, every MDS, both MGRs) on aes
AUTH_INSECURE_SERVICE_TICKETS (ERR) — mons issuing service tickets with aes

The two ERR-severity checks are what push the cluster into HEALTH_ERR (vs. just HEALTH_WARN).

How urgent, really

Proxmox staff (Thomas Lamprecht) on the equivalent forum thread: "as long as your ceph setup is not accessible from untrusted networks, the impact here is much reduced and a migration should not be rushed." Our Ceph network is internal/VLAN-isolated, not internet-facing, per docs/homelab-architecture.md — so this is real and worth fixing, but not an emergency. It should still be done deliberately given the failure modes other operators have hit (below).

How to actually fix it

Do not use raw ceph auth rotate commands manually — Proxmox ships its own migration helper specifically for this, pve-cephx-rotate-service-keys (dry-run by default; documented in pve-docs 9.2.6+):

# 1. Review what it would do first — no changes made without --apply
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-admin-key
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys
# --rotate-lockbox-keys only applies if OSDs are LUKS-encrypted (ours aren't)

# 2. Apply once the dry-run output looks right, phase by phase:
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys --apply
# ... wait, verify, then admin key, then storage keys, same pattern

It uses a staging/graceful-handover mechanism — both old and new keys stay valid while clients (running VMs, CephFS mounts) refresh. Running VMs need to be rebooted to actually pick up the new key.

Real risks other operators hit (from the Proxmox forum threads)

VMs failing to start post-migration with Not a proper rbd authentication file errors
Old keys reappearing in local keyrings after a node reboot
Confusion about how long to wait between phases before confirming — rotating service keys "clear automatically" over hours, not instantly
Explicitly not a trivial migration — Proxmox's own guidance says it "requires caution" on hyper-converged setups, which is exactly our situation (same 7 nodes run Ceph and VMs/CTs)
Client kernel 7.0+ required for RBD/CephFS — already satisfied here (nodes are on 7.0.14-x)
Squid 19.2 (the other affected branch) reaches EOL 2026-10-31 — not directly relevant to us since we're on Tentacle, but a useful external deadline marker if timeline pressure is ever needed

Suppressing the health error in the meantime

Ceph supports muting specific health checks so they don't block automation that gates on HEALTH_ERR/HEALTH_OK:

ceph health mute AUTH_INSECURE_SERVICE_KEY_TYPE 4w
ceph health mute AUTH_INSECURE_SERVICE_TICKETS 4w

(The four WARN-level checks don't need muting to unblock proxmox-update.yml's "not HEALTH_ERR" precheck, but can be muted too for a clean ceph -s if desired.) A mute is not --sticky by default, so it will automatically re-arm if the underlying condition changes; it also expires after the given TTL (set to 4 weeks here — adjust if the Proxmox fleet roll runs longer). ceph health unmute <code> reverses it early. This is a deferral, not a fix — the six checks (and the underlying CVE exposure) remain exactly as-is underneath the mute.

Next steps

Finish proxmox-baseline.yml + proxmox-update.yml across all 7 Proxmox nodes (separate tracked work)
Come back to this issue once the fleet is current
Dry-run pve-cephx-rotate-service-keys (no --apply) against the live cluster, review output carefully
Phase the actual rotation: cluster keys → admin key → storage keys, verifying health between each phase
Reboot VMs/CTs as needed to pick up new keys
Unmute the six AUTH_INSECURE_* checks once rotation is complete and confirm HEALTH_OK

Sources

## Summary Ceph's cluster health has been `HEALTH_ERR` since 2026-09-16, triggered by six new `AUTH_INSECURE_*` checks that Ceph 20.2.4 (already partially rolled out via `ccnet-proxmox-nodes-runbook/proxmox-update.yml` node testing) added to flag every existing CephX key still using the old `aes` cipher (CVE-2025-30156 — an authentication-bypass vulnerability). This is **not an active outage or data-risk issue** — the cluster is otherwise fully healthy (`273 active+clean` PGs) — but it does block `proxmox-update.yml`'s Ceph-health precheck from proceeding on further nodes, since that check correctly refuses to drain a node while `HEALTH_ERR` is present. **Current plan:** finish rolling `proxmox-baseline.yml`/`proxmox-update.yml` across the remaining Proxmox nodes first (tracked separately), muting the specific health checks in the meantime so the update roll isn't blocked. Come back to this issue once all 7 nodes are on current versions and do the actual key rotation as its own deliberate, tested piece of work — not rushed, per Proxmox staff's own guidance below. ## What this actually is **CVE-2025-30156**: CephX's old `aes` cipher (AES-128-CBC) has no HMAC and a hardcoded IV — no authentication of the ciphertext, and identical plaintexts always produce identical ciphertexts. An attacker with a low-privilege key or network access to CephX traffic can flip bits in ciphertexts to forge credentials, including forging **admin credentials**. Fixed in **Ceph Tentacle 20.2.4+ / Squid 19.2.6+**, which introduce a new key type `aes256k` (AES256-CTS-HMAC-SHA384-192, per RFC 8009: random confounders per operation, HMAC-SHA384 auth, ciphertext stealing against padding-oracle attacks) and six new health checks that fire against every existing key still on the old cipher until rotated: - `AUTH_INSECURE_CLIENT_KEY_TYPE` (WARN) — 9 client entities (`client.admin`, all `bootstrap-*`, `client.crash`, `client.swarm-data-fs`) on `aes` - `AUTH_INSECURE_KEYS_ALLOWED` (WARN) — mons configured to permit `aes` auth - `AUTH_INSECURE_KEYS_CREATABLE` (WARN) — mons configured to permit creating new `aes` keys - `AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE` (WARN) — mon/mds/osd/mgr rotating service keys on `aes` - `AUTH_INSECURE_SERVICE_KEY_TYPE` (**ERR**) — 19 service entities (every OSD, every MDS, both MGRs) on `aes` - `AUTH_INSECURE_SERVICE_TICKETS` (**ERR**) — mons issuing service tickets with `aes` The two `ERR`-severity checks are what push the cluster into `HEALTH_ERR` (vs. just `HEALTH_WARN`). ## How urgent, really Proxmox staff (Thomas Lamprecht) on the equivalent forum thread: *"as long as your ceph setup is not accessible from untrusted networks, the impact here is much reduced and a migration should not be rushed."* Our Ceph network is internal/VLAN-isolated, not internet-facing, per `docs/homelab-architecture.md` — so this is real and worth fixing, but not an emergency. It should still be done deliberately given the failure modes other operators have hit (below). ## How to actually fix it **Do not use raw `ceph auth rotate` commands manually** — Proxmox ships its own migration helper specifically for this, `pve-cephx-rotate-service-keys` (dry-run by default; documented in `pve-docs` 9.2.6+): ```bash # 1. Review what it would do first — no changes made without --apply /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-admin-key /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys # --rotate-lockbox-keys only applies if OSDs are LUKS-encrypted (ours aren't) # 2. Apply once the dry-run output looks right, phase by phase: /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys --apply # ... wait, verify, then admin key, then storage keys, same pattern ``` It uses a staging/graceful-handover mechanism — both old and new keys stay valid while clients (running VMs, CephFS mounts) refresh. **Running VMs need to be rebooted to actually pick up the new key.** ### Real risks other operators hit (from the Proxmox forum threads) - VMs failing to start post-migration with `Not a proper rbd authentication file` errors - Old keys reappearing in local keyrings after a node reboot - Confusion about how long to wait between phases before confirming — rotating service keys "clear automatically" over hours, not instantly - Explicitly **not a trivial migration** — Proxmox's own guidance says it "requires caution" on hyper-converged setups, which is exactly our situation (same 7 nodes run Ceph and VMs/CTs) - Client kernel 7.0+ required for RBD/CephFS — already satisfied here (nodes are on `7.0.14-x`) - Squid 19.2 (the other affected branch) reaches EOL 2026-10-31 — not directly relevant to us since we're on Tentacle, but a useful external deadline marker if timeline pressure is ever needed ## Suppressing the health error in the meantime Ceph supports muting specific health checks so they don't block automation that gates on `HEALTH_ERR`/`HEALTH_OK`: ```bash ceph health mute AUTH_INSECURE_SERVICE_KEY_TYPE 4w ceph health mute AUTH_INSECURE_SERVICE_TICKETS 4w ``` (The four `WARN`-level checks don't need muting to unblock `proxmox-update.yml`'s "not HEALTH_ERR" precheck, but can be muted too for a clean `ceph -s` if desired.) A mute is not `--sticky` by default, so it will automatically re-arm if the underlying condition changes; it also expires after the given TTL (set to 4 weeks here — adjust if the Proxmox fleet roll runs longer). `ceph health unmute <code>` reverses it early. This is a deferral, not a fix — the six checks (and the underlying CVE exposure) remain exactly as-is underneath the mute. ## Next steps 1. [ ] Finish `proxmox-baseline.yml` + `proxmox-update.yml` across all 7 Proxmox nodes (separate tracked work) 2. [ ] Come back to this issue once the fleet is current 3. [ ] Dry-run `pve-cephx-rotate-service-keys` (no `--apply`) against the live cluster, review output carefully 4. [ ] Phase the actual rotation: cluster keys → admin key → storage keys, verifying health between each phase 5. [ ] Reboot VMs/CTs as needed to pick up new keys 6. [ ] Unmute the six `AUTH_INSECURE_*` checks once rotation is complete and confirm `HEALTH_OK` ## Sources - [CVE-2025-30156 — Ceph Documentation](https://docs.ceph.com/en/latest/security/CVE-2025-30156/) - [Health Checks — Ceph Documentation](https://docs.ceph.com/en/latest/rados/operations/health-checks/) - [Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon — Proxmox Forum](https://forum.proxmox.com/threads/cephx-key-migration-procedure-and-ceph-19-2-squid-going-eol-soon.186276/) - [Ceph CVEs – keyring rotation strongly advised, probably required in the future — Proxmox Forum](https://forum.proxmox.com/threads/ceph-cves-%E2%80%93-keyring-rotation-strongly-advised-probably-required-in-the-future.185863/) - [CEPH mon/mds/mgr/osd "using insecure key type: aes" — Proxmox Forum](https://forum.proxmox.com/threads/ceph-mon-mds-mgr-osd-using-insecure-key-type-aes.186299/)

claude-bot added the

ceph

p:high

labels

2026-09-16 15:54:05 +00:00

claude-bot referenced this issue

2026-09-16 19:28:01 +00:00

Add ccnet-proxmox-nodes-runbook: package consistency + rolling OS update for the 7 Proxmox nodes #72