Add ccnet-proxmox-nodes-runbook: package consistency + rolling OS update for the 7 Proxmox nodes #72

Merged

copper merged 6 commits from worktree-proxmox-nodes-runbook into main

2026-09-16 19:33:10 +00:00

claude-bot commented

2026-09-16 19:28:01 +00:00

Collaborator

Summary

New Ansible coverage for the 7 Proxmox nodes (node01-node07) — previously none existed at all, package drift and OS patching were entirely manual.

proxmox-baseline.yml — idempotent package consistency (curated 12-package list, built from an apt-mark showmanual diff, not a blind full-package sync) plus a conditionally-deployed CPU throttle fix for the Dell OptiPlex 5050 nodes (node02-node05), gated on live hardware detection (ansible_facts['product_name']) with a matching self-check baked into the deployed script itself.
proxmox-update.yml — rolling OS patch/reboot, one node at a time: HA node-maintenance drain (VMs live-migrate, CTs shutdown→migrate→start automatically), Ceph noout/norebalance bracketing the whole run, quorum-headroom and Ceph-health preflight asserts, an offline-node confirmation gate, and PG-convergence/corosync-rejoin waits — all block:/rescue:-wrapped so a failure cleans up (disables that node's maintenance mode, unsets the Ceph flags) before re-raising, instead of leaving the cluster silently stuck.

Validation

Both playbooks were fully live-tested across all 7 nodes — not just dry-run/--check (which can't meaningfully validate this playbook anyway, since the polling command/shell tasks are skipped rather than simulated under --check). This surfaced and fixed several real bugs along the way, documented in full in ccnet-proxmox-nodes-runbook/README.md:

Corosync membership lists nodes by IP, not hostname — two checks were matching the wrong field
apt upgrade: safe silently keeps proxmox-kernel-* packages back forever (each PVE kernel point release is a new package name) — needed upgrade: dist
No /var/run/reboot-required on these hosts — reboot-need detected via a pveversion -v comparison instead, verified against a real divergent case
The Ceph PG-convergence gate was blocking on routine background scrubbing, not just real recovery states
A systemd ordering anti-pattern (After=multi-user.target + WantedBy=multi-user.target on the same unit) left the CPU throttle-fix service silently dormant after a real reboot on two different nodes — root-caused and fixed, then re-verified across a full HA-drain/reboot/reactivate cycle

An unrelated but consequential finding during this work: node02's Ceph packages upgrading to 20.2.4 surfaced CVE-2025-30156 (AUTH_INSECURE_* health checks) cluster-wide. That's deliberately out of scope for this PR — tracked separately in issue #71, with the two ERR-level checks muted (~4wk TTL) so this playbook's Ceph-health precheck isn't blocked in the meantime.

Test plan

proxmox-baseline.yml — dry-run + real run validated on every node (node01-07)
proxmox-update.yml — real run (drain → upgrade → reboot → reconverge → reactivate) validated on every node (node01-07)
CPU throttle fix confirmed surviving a real reboot on all 4 OptiPlex nodes after the systemd ordering fix
Full-fleet Ceph health confirmed clean (273 active+clean, quorum solid) after the complete roll

🤖 Generated with Claude Code

## Summary New Ansible coverage for the 7 Proxmox nodes (`node01`-`node07`) — previously none existed at all, package drift and OS patching were entirely manual. - **`proxmox-baseline.yml`** — idempotent package consistency (curated 12-package list, built from an `apt-mark showmanual` diff, not a blind full-package sync) plus a conditionally-deployed CPU throttle fix for the Dell OptiPlex 5050 nodes (`node02`-`node05`), gated on live hardware detection (`ansible_facts['product_name']`) with a matching self-check baked into the deployed script itself. - **`proxmox-update.yml`** — rolling OS patch/reboot, one node at a time: HA `node-maintenance` drain (VMs live-migrate, CTs shutdown→migrate→start automatically), Ceph `noout`/`norebalance` bracketing the whole run, quorum-headroom and Ceph-health preflight asserts, an offline-node confirmation gate, and PG-convergence/corosync-rejoin waits — all `block:`/`rescue:`-wrapped so a failure cleans up (disables that node's maintenance mode, unsets the Ceph flags) before re-raising, instead of leaving the cluster silently stuck. ## Validation Both playbooks were fully live-tested across **all 7 nodes** — not just dry-run/`--check` (which can't meaningfully validate this playbook anyway, since the polling `command`/`shell` tasks are skipped rather than simulated under `--check`). This surfaced and fixed several real bugs along the way, documented in full in `ccnet-proxmox-nodes-runbook/README.md`: - Corosync membership lists nodes by IP, not hostname — two checks were matching the wrong field - `apt upgrade: safe` silently keeps `proxmox-kernel-*` packages back forever (each PVE kernel point release is a new package name) — needed `upgrade: dist` - No `/var/run/reboot-required` on these hosts — reboot-need detected via a `pveversion -v` comparison instead, verified against a real divergent case - The Ceph PG-convergence gate was blocking on routine background scrubbing, not just real recovery states - A systemd ordering anti-pattern (`After=multi-user.target` + `WantedBy=multi-user.target` on the same unit) left the CPU throttle-fix service silently dormant after a real reboot on two different nodes — root-caused and fixed, then re-verified across a full HA-drain/reboot/reactivate cycle An unrelated but consequential finding during this work: node02's Ceph packages upgrading to 20.2.4 surfaced CVE-2025-30156 (`AUTH_INSECURE_*` health checks) cluster-wide. That's deliberately **out of scope for this PR** — tracked separately in [issue #71](https://code.coppercore.net/copper/ccnet-prod-devops/issues/71), with the two ERR-level checks muted (~4wk TTL) so this playbook's Ceph-health precheck isn't blocked in the meantime. ## Test plan - [x] `proxmox-baseline.yml` — dry-run + real run validated on every node (node01-07) - [x] `proxmox-update.yml` — real run (drain → upgrade → reboot → reconverge → reactivate) validated on every node (node01-07) - [x] CPU throttle fix confirmed surviving a real reboot on all 4 OptiPlex nodes after the systemd ordering fix - [x] Full-fleet Ceph health confirmed clean (`273 active+clean`, quorum solid) after the complete roll 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 5 commits

2026-09-16 19:28:01 +00:00

Inital commit and creation of proxmox nodes Ansible runbook with assistance from Claude Sonnet 5. 95134f756a

Claude:
Add proxmox-baseline.yml: package consistency + OptiPlex throttle fix

New ccnet-proxmox-nodes-runbook/ (no prior Ansible coverage existed for
the 7 Proxmox nodes). proxmox-baseline.yml enforces a curated set of
manually-installed packages across all nodes (derived from an
apt-mark showmanual diff against node01, not a blind full-package sync,
to avoid flipping dependency packages to manually-installed) and
conditionally deploys the Dell OptiPlex 5050 CPU throttle fix
(optiplex-cpu-throttle-fix/) gated on ansible_facts['product_name'],
with a matching self-check baked into the deployed script itself.
Verified live: node02-05 are genuine OptiPlex 5050, node01/06/07 are
Lenovo and correctly skip the fix. Full-fleet dry run and live run both
clean.

proxmox-update.yml (rolling patch/reboot with HA maintenance-mode
draining and Ceph noout/norebalance) is stubbed but not yet written.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188fKeMYB2cx7boV7BN5H2m

 Date:      Sat Sep 12 12:52:05 2026 -0400
 Changes by Copper:

 On branch worktree-proxmox-nodes-runbook
 Changes to be committed:
	new file:   CLAUDE.md
	new file:   README.md
	new file:   ansible.cfg
	new file:   files/cpu_unthrottle.service
	new file:   group_vars/proxmox_nodes/vars.yaml
	new file:   handlers/main.yml
	new file:   inventory.ini
	new file:   proxmox-baseline.yml
	new file:   proxmox-update.yml
	new file:   tasks/optiplex-throttle-fix.yml
	new file:   templates/cpu_unthrottle.sh.j2

Add proxmox-update.yml: rolling OS patch/reboot with HA drain + Ceph gates e064303516

Three-play design (pre-flight / per-node serial roll / post-flight) to
avoid pre_tasks/post_tasks re-running per batch under serial: 1, which
would otherwise unset Ceph noout/norebalance after every single node
instead of once at the end of the whole run. Per-node body wrapped in
block/rescue so a failure mid-roll disables that node's HA maintenance
mode and unsets the Ceph flags before re-raising, instead of leaving
the cluster silently stuck. Pre/post-flight plays target the whole
group with run_once so --limit <node> still gets the safety plays
instead of skipping them.

Design went through an advisor review that caught 4 blocking issues
before any live testing (a Ceph gate that didn't actually gate, no
cleanup on abort, --limit skipping the safety plays, a quorum check
that didn't verify headroom survives losing the node). Live testing
against node07 (2026-09-16: enable maintenance, real apt upgrade,
verify kernel-detection logic, reboot, reconverge, disable maintenance)
then caught three more real bugs no dry-run would have found:

- corosync membership lists nodes by IP, not hostname — two checks
  were matching on the wrong field entirely
- `upgrade: safe` silently keeps proxmox-kernel-* back forever, since
  each PVE kernel point release ships as a new package name; needed
  `upgrade: dist`
- the PG-convergence gate required literal "only active+clean", which
  blocks on routine background deep-scrub (harmless, unrelated to node
  recovery) for minutes at a time — fixed to gate on absence of actual
  recovery-incomplete states instead

/var/run/reboot-required doesn't exist on these PVE hosts
(update-notifier-common isn't installed); reboot need is detected by
comparing pveversion -v's running-kernel line against the
proxmox-kernel-<major.minor> meta-package line instead, verified
against a real pre/post-upgrade divergence on node07.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188fKeMYB2cx7boV7BN5H2m

fix(proxmox-nodes): correct truncated verify-task path in throttle-fix 0697705b89

tasks/optiplex-throttle-fix.yml's verify task read
/sys/devices/system/cpu/cpu0/cpufreq (a directory) instead of
.../cpuinfo_max_freq (the file) — truncated since the file's very first
commit. Several other lines in the same file were also cut off mid-word
(task names, the assert fail_msg), apparently from a paste that got cut
short when the file was originally typed in.

Root cause discovered live on node02 (2026-09-16): the file on disk was
still the original manually-installed script from before this project
existed (June 28 mtime) — proxmox-baseline.yml's throttle-fix block had
never actually completed successfully on any OptiPlex node, since the
broken verify command would fatal-fail every time it was reached. The
earlier "full fleet, 100% good" baseline run was never independently
re-verified against this specific block afterward, so this went
unnoticed until a real reboot exposed a node stuck at 800MHz.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188fKeMYB2cx7boV7BN5H2m

fix(proxmox-nodes): drop fragile After=multi-user.target from throttle unit f5e7ac7986

After=multi-user.target combined with WantedBy=multi-user.target on the
same unit is a known systemd anti-pattern — it can cause the start job
to be silently dropped during ordering-cycle resolution, with no error
anywhere: the unit stays enabled, but never actually fires at boot.

Reproduced live on both node02 and node03 (2026-09-16): each came back
from a real proxmox-update.yml reboot with cpu_unthrottle.service
enabled but inactive/dead, zero journal entries for that boot, CPU
pinned at 800MHz. Manually starting the service worked fine every time
(confirming the script/unit content itself was correct) — only the
automatic boot-time trigger was failing. Re-tested node03 through a
full HA-drain/reboot/reactivate cycle after this fix: service fired
automatically and cleanly this time (journal shows normal
Starting/Finished entries), CPU correct at 3.3GHz.

Default ordering (implied by WantedBy=multi-user.target alone) is
sufficient for a oneshot hardware-init fix like this one — no need for
an explicit After= on the same target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188fKeMYB2cx7boV7BN5H2m

docs(proxmox-nodes): write README.md and CLAUDE.md runbook docs 6845d5d6a0

Full runbook for both playbooks, written after live-validating the
entire fleet (all 7 nodes through both proxmox-baseline.yml and
proxmox-update.yml, including real reboots). Documents every real
gotcha hit during that validation: corosync membership keyed by IP not
hostname, upgrade:safe silently skipping kernel updates, the
scrub-blocking PG convergence gate, the missing reboot-required file
and the pveversion-based alternative, and the systemd
After=/WantedBy=multi-user.target ordering anti-pattern that left the
CPU throttle fix dormant after a reboot with zero error anywhere.

Same shape as ccnet-swarm-nodes-runbook/README.md and CLAUDE.md per
this repo's documented convention that every top-level directory gets
both.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188fKeMYB2cx7boV7BN5H2m

copper added 1 commit

2026-09-16 19:32:37 +00:00

Merge branch 'main' into worktree-proxmox-nodes-runbook 159710a4e2

copper merged commit 45193a87f6 into main

2026-09-16 19:33:10 +00:00

copper deleted branch worktree-proxmox-nodes-runbook

2026-09-16 19:33:10 +00:00

copper referenced this pull request from a commit

2026-09-16 19:33:10 +00:00

Merge pull request 'Add ccnet-proxmox-nodes-runbook: package consistency + rolling OS update for the 7 Proxmox nodes' (#72) from worktree-proxmox-nodes-runbook into main