ccnet-wi-pbs: flip offsite replication from pull to push #62

Merged

copper merged 4 commits from wi-pbs-pull-to-push into main

2026-09-01 14:00:24 +00:00

claude-bot commented

2026-09-01 13:41:58 +00:00

Collaborator

Docs for the WI offsite-PBS replication change applied by hand 2026-08-30 → 09-01.

Note: local main was 2 commits ahead of origin/main when this branch was cut, so this PR also carries 8226c90 (mcp-agents: metamcp-postgres 16→17) and 3b39aea (ccnet-luna-pbs-vm-runbook: new dir) — both pre-existing on the local main, not this session's work. 3b39aea is a real dependency (it creates the runbook dir these docs edit). Merging brings origin/main up to where local main already was, plus the WI change. Alternatively: push local main first, then this branch rebases to just the WI commit.

What changed on the infra (already live)

Was: WI-side pull s-123363f1-cb12 (02:00) with associated-key ccnet-hq = decrypt on pull. That defeats PBS's known-chunk skip → the entire ~530 GiB copper dataset re-transferred over the WAN every night (~18 h) vs a few-GB real delta. Stored plaintext. Removing the key isn't possible (PBS 4.2 refuses: No matching key found).
Now: push s-wi-push from luna-pbs (ns copper, push, 03:00, remove-vanished false, no key / no rate limit), remote ccnet-wi → WI-side token root@pam!luna-push (DatastoreAdmin on /datastore/wi-qnap/copper). Push ships the already-encrypted chunks verbatim → ~8 min/night after a one-time 1.1 TiB re-seed.
WI is now ciphertext-only — the ccnet-hq key was removed from that box (identical material stays on luna as ccnet-main + offline paperkey; a WI-side restore needs it re-imported).
Bandwidth: luna's Slow traffic-control rule removed (unshaped); WI Default-Limit rate-in 50 MiB → 80 MB.
Gotcha: the 81 pre-existing WI groups were owned by root@pam; the token-auth push skipped them all until each group's owner file was rewritten to root@pam!luna-push.
Old pull job + copper-hq remote + key removed on WI; rollback kit at /root/wi-pull-teardown-2026-09-01/ on ccnet-wi-pbs.

Files (WI commit `d266979`)

docs/ccnet-wi-site.md — ccnet-wi-pbs section rewritten
docs/homelab-build-history.md — new Item 19
ccnet-luna-pbs-vm-runbook/ — README (consumer bullet, Prereq B, new "WI offsite replication" section), CLAUDE.md, keys/README.md
ccnet-pbs-backup-swarm-data/ — README + CLAUDE.md ("WI pull job" → third push job)

Not included: ccnet-luna-pbs-vm-runbook/CUTOVER.md also has two one-line WI edits from this work, left with the unrelated pre-existing CUTOVER rework for a separate commit.

Full procedure: ~/.claude/plans/do-a-very-simple-vivid-kurzweil.md.

🤖 Generated with Claude Code

Docs for the WI offsite-PBS replication change applied by hand 2026-08-30 → 09-01. > **Note:** local `main` was 2 commits ahead of `origin/main` when this branch was cut, so this PR also carries `8226c90` (mcp-agents: metamcp-postgres 16→17) and `3b39aea` (ccnet-luna-pbs-vm-runbook: new dir) — both pre-existing on the local `main`, not this session's work. `3b39aea` is a real dependency (it creates the runbook dir these docs edit). Merging brings `origin/main` up to where local `main` already was, plus the WI change. Alternatively: push local `main` first, then this branch rebases to just the WI commit. ## What changed on the infra (already live) - **Was:** WI-side pull `s-123363f1-cb12` (02:00) with `associated-key ccnet-hq` = *decrypt on pull*. That defeats PBS's known-chunk skip → the entire ~530 GiB `copper` dataset re-transferred over the WAN every night (~18 h) vs a few-GB real delta. Stored plaintext. Removing the key isn't possible (PBS 4.2 refuses: `No matching key found`). - **Now:** push `s-wi-push` from luna-pbs (ns `copper`, push, 03:00, `remove-vanished false`, no key / no rate limit), remote `ccnet-wi` → WI-side token `root@pam!luna-push` (`DatastoreAdmin` on `/datastore/wi-qnap/copper`). Push ships the already-encrypted chunks verbatim → **~8 min/night** after a one-time 1.1 TiB re-seed. - WI is now **ciphertext-only** — the `ccnet-hq` key was removed from that box (identical material stays on luna as `ccnet-main` + offline paperkey; a WI-side restore needs it re-imported). - Bandwidth: luna's `Slow` traffic-control rule removed (unshaped); WI `Default-Limit` `rate-in` 50 MiB → 80 MB. - Gotcha: the 81 pre-existing WI groups were owned by `root@pam`; the token-auth push skipped them all until each group's `owner` file was rewritten to `root@pam!luna-push`. - Old pull job + `copper-hq` remote + key removed on WI; rollback kit at `/root/wi-pull-teardown-2026-09-01/` on `ccnet-wi-pbs`. ## Files (WI commit `d266979`) - `docs/ccnet-wi-site.md` — `ccnet-wi-pbs` section rewritten - `docs/homelab-build-history.md` — new Item 19 - `ccnet-luna-pbs-vm-runbook/` — README (consumer bullet, Prereq B, new "WI offsite replication" section), CLAUDE.md, keys/README.md - `ccnet-pbs-backup-swarm-data/` — README + CLAUDE.md ("WI pull job" → third push job) Not included: `ccnet-luna-pbs-vm-runbook/CUTOVER.md` also has two one-line WI edits from this work, left with the unrelated pre-existing CUTOVER rework for a separate commit. Full procedure: `~/.claude/plans/do-a-very-simple-vivid-kurzweil.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 3 commits

2026-09-01 13:41:59 +00:00

mcp-agents: migrate metamcp-postgres 16 -> 17 8226c90f3d

A Postgres major bump can't reuse the on-disk datadir (PG_VERSION
incompatible), so this was a logical dump/restore rather than an image
swap: pg_dump the ~250 KB `metamcp` DB out of the running pg16 container,
scale metamcp to 0, move the datadir aside to `metamcp-postgres.pg16-bak`
(kept intact for instant rollback), let pg17 initdb a fresh cluster, then
psql-restore the dump. Verified post-restore: PG 17.11, all row counts
match baseline (tools=224, mcp_servers=5, drizzle migrations=13, ...),
MCP endpoint returns 401 not 502, frontend 307->/en->200 via Caddy,
writes confirmed, data files land on the swarm-nvme pool (xattr
re-applied to the new datadir).

MetaMCP's schema (drizzle migrations 0000-0013) uses no Postgres
extensions, only built-ins, so stock postgres:17-alpine was sufficient.
README documents the full runbook + rollback as the template for the
forgejo/nextcloud DB tiers when they need a major bump.

pg16-bak datadir retained ~1 week (also in the nightly PBS CephFS
backup), then to be removed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WCYV6DNBPe7rqaZVBx4yPL

ccnet-luna-pbs-vm-runbook: new dir — luna-pbs relocation CT 356 -> Unraid VM 3b39aeab11

Relocating the Proxmox Backup Server (10.10.10.250) off privileged LXC CT 356
on node06 and onto a KVM VM on Unraid (Aegis), keeping the same IP and TLS
cert fingerprint. Datastore (2.6 TB on Aegis's `backup` raidz2 pool) does not
move — only the PBS compute does, collapsing the CT->cluster->Aegis NFS hop
into an in-host br0 path and coupling PBS's lifecycle to the Unraid array.

Contents:
- README.md    — VM spec, build method (Debian 13 genericcloud + cloud-init,
                 OVMF), the /etc/proxmox-backup seed procedure, storage tuning
                 (post-soak), the 3-layer VM/config backup design.
- CUTOVER.md   — ordered copy-paste runbook, [CLAUDE]/[YOU] tagged, with rollback.
- CLAUDE.md    — load-bearing gotchas (two-writer hazard, root@pam auth, keep
                 the export unchanged for the move, pre-cutover zfs snapshot).
- luna-pbs.xml — the live libvirt domain (OVMF, pbs.png icon).
- files/       — cloud-init user-data/network-config, pbs-nfs-remount.sh
                 (fallback, no self-reboot), systemd mount-ordering drop-in.
- keys/        — placeholders only; real cert/key/password material never here.

Status: VM built + staged (PBS 4.2.5-1 held, both lifecycle tests passed,
baseline backup taken). Cutover not yet run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQJmq9kQvZ9Fw2KYbRBY8e

ccnet-wi-pbs: flip offsite replication from pull to push d2669798a0

The WI offsite replica was a pull run on ccnet-wi-pbs (s-123363f1-cb12,
02:00) with associated-key ccnet-hq set = decrypt on pull. Decrypting
makes WI store chunks under plaintext digests that never match the
source's encrypted-chunk digests, so PBS's known-chunk skip never fires
and the whole ~530 GiB copper dataset re-transferred over the WAN every
night (~18 h) against a few-GB real delta. Stored plaintext. Removing
the key isn't an option — PBS 4.2 then refuses every group
(No matching key found).

Replaced with a push from luna-pbs (s-wi-push, ns copper, push, 03:00,
remove-vanished false, no key/rate-out), remote ccnet-wi -> WI-side token
root@pam!luna-push. A push ships the already-encrypted chunks verbatim
so known-chunk skip works: ~8 min/night after a one-time 1.1 TiB
re-seed. WI is now ciphertext-only; the ccnet-hq key was removed from
that box (same material stays on luna as ccnet-main + offline paperkey).
Old pull job, copper-hq remote, and key removed on WI; rollback kit at
/root/wi-pull-teardown-2026-09-01/ on ccnet-wi-pbs.

Docs only — the infra changes were applied by hand (credential-sensitive).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QovxLG6bZyfYkqTGuUcJoe

claude-bot added 1 commit

2026-09-01 13:59:45 +00:00

ccnet-luna-pbs-vm-runbook: CT-stays-live cutover rework + config self-backup layer 7693a2ef46

CUTOVER.md: reorder so CT 356 stays running (PBS stopped) through the
/etc/proxmox-backup seed and is only powered off afterward (step 6.5) —
the seed tars straight from the live CT rather than a stopped one. Step 6
split into 6a–6d with an explicit fingerprint sanity gate.

luna-pbs.xml: vda driver cache=writeback -> cache=directsync + io=io_uring
(the CUTOVER step-5 tuning, captured in the tracked XML).

files/luna-pbs-selfbackup.{sh,service,timer,env.example}: Layer-1 config
self-backup — proxmox-backup-client backs /etc into aegis-unraid as
host/luna-pbs (ns copper), encrypted with the reused PVE key, nightly
01:30 timer. Lands under ns copper so the existing offsite sync jobs
replicate it for free. env/key files stay out of git.

Also picks up two one-line WI-replication edits in CUTOVER.md (avoid-window
+ deferred-rerun command) left over from the pull->push change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QovxLG6bZyfYkqTGuUcJoe

copper merged commit 61d3ed7d85 into main

2026-09-01 14:00:24 +00:00

copper deleted branch wi-pbs-pull-to-push

2026-09-01 14:00:25 +00:00

copper referenced this pull request from a commit

2026-09-01 14:00:26 +00:00

Merge pull request 'ccnet-wi-pbs: flip offsite replication from pull to push' (#62) from wi-pbs-pull-to-push into main

claude-bot referenced this pull request

2026-09-02 21:48:56 +00:00

Stand up Unraid (Aegis) appdata backup coverage + evaluate moving PBS back onto Unraid as a container #59