Write swarm-update.yml rolling-update playbook #1
Labels
No labels
blocked
borg-backup
ceph
forgejo
in-progress
infra
netdata
nextcloud
p:high
p:low
p:medium
searxng
service-onboard
swarm-nodes
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
copper/ccnet-prod-devops#1
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Design manually validated 2026-07-05 during the swarm-nvme DB migration:
docker node update --availability drain->apt upgrade-> reboot -> wait for rejoin ->--availability active, one node at a time, Raft quorum never below 2/3. Needs writing up as an actual Ansible playbook (swarm-update.yml) followingccnet-swarm-nodes-runbook/swarm-provision.yml's layout (serial: 1, etc).See
docs/homelab-build-history.mdItem 1 for the full design (docs/next-steps.mdwas merged into that file 2026-07-08, commit9c6e167; old "see docs/next-steps.md item N" references still resolve via the kept "Item N" headers).Still open as of 2026-07-23:
ccnet-swarm-nodes-runbook/only hasswarm-provision.yml—swarm-update.ymlhas not been written. Checked live cluster state today: all 5 nodes (3 managers, 2 workers) areReady/Active, but every node has pending apt upgrades (8-10 packages) and/var/run/reboot-requiredis set fleet-wide, so this playbook has real work to do right now, not just a validated-but-idle design.swarm-update.ymldrafted 2026-07-23 (plus a newccnet-swarm-nodes-runbook/README.md, which didn't exist before). Syntax-checked clean; not yet run against real nodes.Beyond the manually-validated drain/upgrade/reboot/active sequence, it adds:
any_errors_fatal, leaving a failed node drained rather than force-reactivating itFull design writeup in
docs/homelab-build-history.mdItem 1. Plan next: test with--limit swarm-wkr-01(lowest blast radius, no quorum risk), then a full 5-node roll — will close this issue once that's done.Hit a real, nasty failure testing
--limit swarm-wkr-02: the VM's QEMU process wedged completely (QMP unresponsive, RSS pinned ~3GB over its 8GB budget, guest unreachable) and needed a hardqm stop/qm starton node04 to recover. No data loss — the node had already drained cleanly before this happened, so no Swarm workloads were stranded.Root cause, traced via the guest's own journal (
journalctl -b -1): the docker-ce package's postinst restarts the Docker daemon on apt upgrade (upstream packaging behavior). That restart landed 5 seconds before the reboot task fired. Docker/containerd's network-namespace cleanup from the drain was still settling when systemd tried to SIGTERM everything for final shutdown — the guest hung forever at "Sending SIGTERM to remaining processes," which is the exact same point every healthy reboot passes through, just normally in well under a second.Fixed:
swarm-update.ymlnow waits fordocker info --format '{{.Swarm.LocalNodeState}}'to reportactiveafter the apt task and before ever checkingreboot-required, so a daemon mid-restart from the upgrade gets a chance to settle before a reboot lands on top of it.Also fixed along the way (same test run): a separate Ansible gotcha where regex backreferences (
\1) insideuntil:get mangled by the extra template pass Ansible applies to conditionals — silently never matched, spun all 30 retries, failed even when already converged. Replaced with a split-and-compare check. Both gotchas fully written up inccnet-swarm-nodes-runbook/README.md.swarm-wkr-01andswarm-wkr-02are both now fully patched and back in service. Re-runningswarm-wkr-02end-to-end with the docker-settle fix next, thenswarm-mgr-*(first real exercise of the manager quorum-check path) before closing this out.Clean re-run of
swarm-wkr-02with the docker-settle fix — no failures, no retries needed on the docker-settle check, convergence gate passed on its normal first retry. Bothswarm-wkr-01andswarm-wkr-02are now fully validated end-to-end (patched,Ready/Active, quorum untouched).Worker path is done. Next:
swarm-mgr-*, which will be the first real exercise of the manager quorum-check path — nothing has touched a manager yet. Will close this issue once a full clean fleet roll (or at least one manager, to prove that path) is confirmed.swarm-mgr-01rolled clean end-to-end — first real exercise of the manager path, and it's the important one: quorum check correctly asserted the other two managers were Leader/Reachable before draining, apt upgrade had real packages this time, reboot completed normally (no repeat of the wkr-02 hang — the docker-settle fix held), node rejoined, reactivated, and all 20 services reconverged. Cluster confirmed healthy after: all 5 nodesReady/Active, mgr-03 still Leader, mgr-01 now on engine 29.6.2.Manager and worker paths are both now proven individually. Only remaining validation: a full unattended fleet roll (all 5 nodes, no
--limit) to confirm the wholeserial: 1sequence holds end-to-end back-to-back, including mgr-02 and mgr-03 which are still unpatched. Will close this issue after that.Full unattended fleet roll completed clean — all 5 nodes, no
--limit, zero failures (failed=0across the board in the recap: mgr-01/02/03, wkr-01/02).swarm-mgr-02andswarm-mgr-03both got real apt upgrades + reboots this time and came back clean.Final state: all 5 nodes
Ready/Active, engine29.6.2fleet-wide, Leader correctly re-elected toswarm-mgr-02during mgr-03's cycle, all 20 services back at desired replica count, and spot-checked the real public endpoints (code,nc,search.../healthz) all responding normally post-roll.swarm-update.ymlis done: written, hardened against two real bugs found in testing (an Ansibleuntil:regex-backreference gotcha, and a docker-ce postinst-restart race that once wedged a VM badly enough to need a hardqm stop/qm start), and now proven end-to-end on real production hardware — worker path, manager quorum path, and a full unattended 5-node roll. Closing.