Swarm stack updates: low-risk image bumps + Forgejo Postgres 16→18 #73

Merged
copper merged 4 commits from worktree-swarm-stack-bumps into main 2026-09-16 20:07:42 +00:00
Collaborator

Summary

Part of the 2026-09-12 infra update audit. Two batches of work on the same branch:

Low-risk image bumps (2026-09-12):

  • searxng: 2026.7.5-a6438586a → 2026.9.5-c7f3080aa
  • nextcloud: redis:7-alpine → redis:8-alpine, collabora/code pinned to 26.04.2.4.1 (was floating :latest)
  • mcp-agents: proxmoxmcp-plus 0.5.14 → 0.5.16
  • Documents a Swarmpit update_stack gotcha hit while deploying these: its generated-compose export/import silently drops entrypoint: overrides, secret uid/gid/mode, and placement constraints — broke several mcp-agents and nextcloud services until redeployed for real via docker stack deploy against the repo YAML. Documented as a repo-wide convention in root CLAUDE.md.

Forgejo Postgres major migration (2026-09-16):

  • db: postgres:16-alpine → postgres:18-alpine (dump/restore, same shape as the mcp-agents metamcp-postgres 16→17 runbook)
  • Verified: PostgreSQL 18.6 running, all 128 tables restored, row counts match baseline exactly, code.coppercore.net returns 200, real issue/PR traffic serving correctly post-cutover
  • Real gotcha hit and documented: the official postgres image changed its bind-mount convention at major 18 (docker-library/postgres#1259) — now wants the mount at /var/lib/postgresql (the parent), not .../data. Fixed in the stack file; cross-referenced in ccnet-mcp-agents/README.md's runbook since that service's own postgres (17→18, planned) will hit the same thing.
  • Safety backups (full forgejo dump zip + pg_dump SQL + baseline row counts) kept at /mnt/swarm-data/backups/forgejo-migration-2026-09-16/ on shared CephFS storage, old pg16 datadir kept as .pg16-bak for ~1 week as instant rollback.

Test plan

  • All affected swarm services confirmed N/N running, update: completed via Swarmpit
  • Forgejo: DB version, row-count diff against pre-migration baseline, HTTP 200, live traffic in logs
  • Plan file with full step-by-step record: ~/.claude/plans/2026-09-16-forgejo-postgres-16-to-18-migration.md (local, not in repo)

🤖 Generated with Claude Code

## Summary Part of the 2026-09-12 infra update audit. Two batches of work on the same branch: **Low-risk image bumps (2026-09-12):** - `searxng`: `2026.7.5-a6438586a` → `2026.9.5-c7f3080aa` - `nextcloud`: `redis:7-alpine` → `redis:8-alpine`, `collabora/code` pinned to `26.04.2.4.1` (was floating `:latest`) - `mcp-agents`: `proxmoxmcp-plus` `0.5.14` → `0.5.16` - Documents a Swarmpit `update_stack` gotcha hit while deploying these: its generated-compose export/import silently drops `entrypoint:` overrides, secret `uid`/`gid`/`mode`, and `placement` constraints — broke several `mcp-agents` and `nextcloud` services until redeployed for real via `docker stack deploy` against the repo YAML. Documented as a repo-wide convention in root `CLAUDE.md`. **Forgejo Postgres major migration (2026-09-16):** - `db`: `postgres:16-alpine` → `postgres:18-alpine` (dump/restore, same shape as the `mcp-agents` metamcp-postgres 16→17 runbook) - Verified: PostgreSQL 18.6 running, all 128 tables restored, row counts match baseline exactly, `code.coppercore.net` returns 200, real issue/PR traffic serving correctly post-cutover - Real gotcha hit and documented: the official `postgres` image changed its bind-mount convention at major 18 (docker-library/postgres#1259) — now wants the mount at `/var/lib/postgresql` (the parent), not `.../data`. Fixed in the stack file; cross-referenced in `ccnet-mcp-agents/README.md`'s runbook since that service's own postgres (17→18, planned) will hit the same thing. - Safety backups (full `forgejo dump` zip + `pg_dump` SQL + baseline row counts) kept at `/mnt/swarm-data/backups/forgejo-migration-2026-09-16/` on shared CephFS storage, old pg16 datadir kept as `.pg16-bak` for ~1 week as instant rollback. ## Test plan - [x] All affected swarm services confirmed `N/N running`, `update: completed` via Swarmpit - [x] Forgejo: DB version, row-count diff against pre-migration baseline, HTTP 200, live traffic in logs - [x] Plan file with full step-by-step record: `~/.claude/plans/2026-09-16-forgejo-postgres-16-to-18-migration.md` (local, not in repo) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
- searxng: 2026.7.5-a6438586a -> 2026.9.5-c7f3080aa
- nextcloud: redis 7-alpine -> 8-alpine, collabora/code pinned to 26.04.2.4.1
  (was floating :latest)
- mcp-agents: proxmoxmcp-plus 0.5.14 -> 0.5.16

Also documents a Swarmpit update_stack gotcha hit while deploying these:
its generated-compose export/import silently drops entrypoint overrides,
secret uid/gid/mode, and placement constraints, breaking several mcp-agents
and nextcloud services until redeployed for real via docker stack deploy
against the repo YAML.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAp6m6GGf1PKfff1HKziBB
Postgres major-version bumps can't reuse the on-disk datadir. Followed the
same dump/restore shape as the mcp-agents metamcp-postgres 16->17 migration
(2026-08-29): quiesce forgejo, pg_dump the pg16 database, move the datadir
aside, initdb fresh under pg18, restore, verify row counts + app health,
then bring the app back up.

Real finding along the way: the official postgres image changed its data
layout convention starting at major 18 (docker-library/postgres#1259) --
it now expects the bind mount at /var/lib/postgresql (the parent), not
.../data, and creates its own versioned subdirectory inside. The old
.../data mount target made the pg18 image refuse to start, mistaking the
empty fresh datadir for leftover unmigrated data from a skipped upgrade.
Fixed by mounting the parent directory instead.

Verified: PostgreSQL 18.6 running, all 128 tables restored, row counts
match the pre-migration baseline exactly (a 2-row apparent mismatch on the
`action` table traced to pg_stat_user_tables's n_live_tup being a stale
estimate, not real data loss -- confirmed by counting the dump's own COPY
block), code.coppercore.net returns 200, real issue/PR traffic serving
correctly post-cutover.

Backups (forgejo dump zip + pg_dump SQL + baseline row counts) kept at
/mnt/swarm-data/backups/forgejo-migration-2026-09-16/ on shared CephFS
storage. Old pg16 datadir kept as /mnt/swarm-data/forgejo/postgres.pg16-bak
for ~1 week as instant rollback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAp6m6GGf1PKfff1HKziBB
Cross-reference the docker-library/postgres#1259 gotcha hit during the
forgejo postgres 16->18 migration (mount target moved from .../data to
the parent /var/lib/postgresql) in both ccnet-forgejo-swarm/CLAUDE.md
(what actually changed there) and ccnet-mcp-agents/README.md's dump/
restore runbook (mcp-agents' own postgres is still on 17 and will hit
the same thing whenever it crosses into 18).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAp6m6GGf1PKfff1HKziBB
copper merged commit 269f50f36d into main 2026-09-16 20:07:42 +00:00
Sign in to join this conversation.
No description provided.