feat(matrix): add ccnet-matrix-swarm self-hosted Matrix stack #84
No reviewers
Labels
No labels
blocked
borg-backup
ceph
forgejo
in-progress
infra
netdata
nextcloud
p:high
p:low
p:medium
searxng
service-onboard
swarm-nodes
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
copper/ccnet-prod-devops!84
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/ccnet-matrix-swarm"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Self-hosted Matrix (Synapse + Cinny + Element Web + LiveKit + ntfy) on the ccnet Swarm cluster
Adds a complete 7-service Matrix stack: text + voice/video + push + two web clients + ops automation. 4 commits, ~1900 net LOC, 4 review passes (3 oracle + 1 local-model second-opinion) all addressed.
What landed
matrix-swarm-stack.yml— 7-service stack (postgres, synapse, cinny, livekit, matrixrtc-jwt, element-web, ntfy)config/homeserver.yaml.example— Synapse v1.161.0 template with MatrixRTC + token-gated registration + federation IP-blacklistconfig/livekit.yaml.example— envsubst-rendered SFU config templateconfig/element-config.json+config/cinny-config.json— two web clients (both bundle Element Call)bootstrap-secrets.sh,generate-homeserver-yaml.sh,render-livekit-yaml.sh,create-admin.sh,sync-matrix-cert.sh,backup-signing-key.shREADME.md(746 lines) +CLAUDE.md(383 lines) — operator runbook and gotchasServices
matrix_postgrespostgres:18-alpine@sha256:d3e1620b530c944afa6e887d22eb899824da68e19c52024bf98f5220c88a65b2matrix_synapsematrixdotorg/synapse:v1.161.0matrix_cinnyajbura/cinny:v4.12.7matrix_element-webvectorim/element-web:v1.12.29matrix_livekitlivekit/livekit-server:v1.13.7matrix_matrixrtc-jwtghcr.io/element-hq/lk-jwt-service:0.7.0matrix_ntfybinwiederhier/ntfy:v2.28.0Key design decisions
coppercore.net— immutable. All user handles@user:coppercore.net. Cannot change without a full migration.enable_registration: true+registration_requires_token: true. Admins create per-user tokens via the Synapse Admin API (POST /_synapse/admin/v1/registration_tokens/new).matrix_rtc.transports+extra_well_known_client_content.rtc_foci.livekit-api-keyANDlivekit-api-secretare consumed by both LiveKit (via rendered yaml) and lk-jwt-service (via_FROM_FILEenv). MUST be the same pair; the README's "Rotate LiveKit API key" section documents the atomic procedure.matrix.coppercore.net. Clients fetch from the apex first, then from the subdomain.synapse/coppercore.net.tls.{crt,key}owner 991:991, mode 0600/0644. Auto-renewed daily bysync-matrix-cert.shvia cron: scp from OPNsense ACME → mountpoint-guarded atomic install →docker kill --signal=HUP(zero-downtime).mode: ingress(the Swarm routing mesh answers on every node, Caddy round-robins). LiveKit is the deliberate exception (network_mode: host) per Element Call team's recommendation.Out-of-stack prerequisites (must be done before first deploy)
Detailed in README §"Out-of-stack prerequisites". Summary:
matrix.,chat.,app.,ntfy.,matrix-rtc..coppercore.net → manager IPs (split-horizon via Unbound Host Overrides for LAN clients).sync-matrix-cert.shactually triggers a Caddy reload.coppercore.netwell-known block:matrix.coppercore.net, place at/mnt/swarm-data/matrix/synapse/coppercore.net.tls.{crt,key}(chown 991:991, chmod 0600 key).Audit history
516d7a3.registration_requires_token: truemissing — would have opened public registration), 1 HIGH (docker kill --signal=HUPfailed silently because service name ≠ container ID), several MEDIUM — all addressed.keysmap andwebhook.api_keywould have caused 401s on every Element Call JWT — the api_key_id substitution was wrong), 3 HIGH (matrixrtc-jwt unreachable from OPNsense Caddy, Caddy config used Docker service names, render-livekit-yaml.sh idempotency defeated secret rotation), several MEDIUM — all addressed in commitcc634bd. The oracle's lk-jwt-service:0.7.0 "doesn't exist" finding was incorrect — verified the tag exists via the GitHub releases page (released 2026-09-10) and the stack remains pinned to 0.7.0.matrix_matrixrtc-jwt.1.<containerid>reference becausedocker psreturns container IDs not task IDs), 1 MEDIUM (render-livekit-yaml.shawk idempotency parse was dead —/^[a-zA-Z0-9_]+:/never matches indented+quoted YAML; idempotency never triggered) — both fixed and verified in commit5e83e07.Known gaps (documented in README §"Known gaps" and CLAUDE.md)
turn.enabled: true+turn.urisinlivekit.yamlif hostile NAT clients appear.Reviewer notes
Please look especially at:
matrix-swarm-stack.yml(288 lines) — the 7-service shape, thenetwork_mode: hostexception for livekit, the uid/gid/mode mounts on synapse secrets, thelivekit-api-{key,secret}consumption by BOTH livekit (rendered yaml) and matrixrtc-jwt (env). The deploy block patterns (replicas / restart_policy / update_config) match the repo's forgejo + mcp-agents conventions.config/homeserver.yaml.example(154 lines) — MatrixRTC block placement, uid/gid-991-relevant templating, the rtc_foci well-known atmatrix.coppercore.net, theenable_registration: true+registration_requires_token: truetoken-gating.scripts/render-livekit-yaml.sh— envsubst + PyYAML idempotency check + mountpoint guard. Replaced the broken awk that sec-1 found.scripts/sync-matrix-cert.sh— acme.sh non-atomic-write guard + task-to-container-id resolution + SIGHUP for Synapse hot-reload.README.md(746 lines) §"Out-of-stack prerequisites" — the operator's first stop before first deploy.CLAUDE.md(383 lines) — gotchas that don't fit the runbook shape. Includes the uid/gid-991-is-not-optional note (same bug class that hitproxmox-mcpin production 2026-09-12).Branch
feat/ccnet-matrix-swarm— 4 commits, all on the worktree at.slim/worktrees/ccnet-matrix-swarm/.Scope expansion: the original 4-service stack (postgres/synapse/cinny/ntfy) becomes 7 services (also adding livekit, matrixrtc-jwt, element-web). Brings voice/video (Element Call / MatrixRTC) online, plus a fuller-featured alternative web client alongside Cinny. New services - livekit (livekit/livekit-server:v1.13.7, host networking) — MatrixRTC SFU. Listens 7880/tcp + 7881/tcp + UDP 7882-7892 (UDP mux range instead of the impractical 50000-60000 default). Caddy on matrix-rtc.coppercore.net path-routes /livekit/jwt/* to lk-jwt-service and the rest to LiveKit. - matrixrtc-jwt (ghcr.io/element-hq/lk-jwt-service:0.7.0, overlay) — Element's MatrixRTC Authorization Service. Validates the user's OpenID token against Synapse, mints LiveKit JWTs granting room access. LIVEKIT_FULL_ACCESS_HOMESERVERS pinned to coppercore.net (never *). - element-web (vectorim/element-web:v1.12.29, 2 replicas, ingress) — alternative web client with native Element Call widget integration. Conf at ./config/element-config.json, strict JSON per Element Web v1.12.x schema; restarts on every config edit (entrypoint copies /app/config.json to /tmp at container start, no live-reload). New config - homeserver.yaml.example: adds experimental_features {msc3266, msc4143, msc4222} for MatrixRTC, max_event_delay_duration: 24h, rc_message + rc_delayed_event_mgmt rate limits for Element Call E2EE bursts, top-level matrix_rtc.transports block (BOTH new url and legacy livekit_service_url keys per Synapse 1.161.0 upgrade note), and extra_well_known_client_content.rtc_foci for the well-known fallback. - config/ntfy-server.yml / config/cinny-config.json: unchanged. New files - config/livekit.yaml.example: envsubst template with LIVEKIT_API_KEY placeholder; rendered to CephFS at /mnt/swarm-data/matrix/livekit/ livekit.yaml by scripts/render-livekit-yaml.sh. - config/element-config.json: strict Element Web v1.12.29 config. features.feature_element_call_video_rooms + element_call.use_exclusively for MatrixRTC; element_call.guest_spa_url points at app.coppercore.net. - scripts/render-livekit-yaml.sh: parallel to generate-homeserver-yaml.sh, pulls livekit-api-key from Swarm secrets, envsubst-renders, chown 0:0 mode 0600 (LiveKit container runs as root). - .gitignore: adds config/livekit.yaml (rendered, lives on CephFS instead). New secrets (2; total now 8) - livekit-api-key (32 hex chars) and livekit-api-secret (64 hex chars). CRITICAL — both livekit (via the keys map in livekit.yaml) AND matrixrtc-jwt (via LIVEKIT_KEY_FROM_FILE) MUST read the SAME pair. If they diverge, Element Call JWT issuance fails with 401s from the SFU. Atomic rotation procedure documented in README §Day-to-day. Documentation - README.md: rewrote to include 7-service table, new hostnames (app, matrix-rtc), LiveKit firewall ports (separate from Swarm mesh because of host networking), 5 OPNsense Caddy domains + 1 federation L4, two render-on-deploy steps (homeserver + livekit), §10 phone+call setup covering both Element Call and UnifiedPush, corrected claim that Cinny can't make calls (Cinny v4.11.1+ bundles Element Call — verified in cinnyapp/cinny@v4.11.1 source), updated known-gaps section. - CLAUDE.md: corrected same Cinny call claim (now 'Cinny + Element Web both bundle Element Call — no client fork needed'); added LiveKit host-networking gotcha (intentional exception to mode: ingress); LiveKit API key/secret atomicity gotcha; Element Web config-restart gotcha; LiveKit config render chown pattern. Out-of-stack prerequisites (README §Op-s) updated - DNS A/AAAA for matrix, chat, app, ntfy, matrix-rtc (split-horizon). - OPNsense Caddy blocks for all 5 routed hostnames + 8448 L4 for matrix. - OPNsense firewall opening 7880/tcp + 7881/tcp + UDP 7882-7892 for the LiveKit node (since host networking bypasses Swarm port publishing). - VPS Caddy matching blocks + apex well-known (unchanged). Tested locally: PyYAML parses every yaml, json.load parses every json, bash -n passes on every script.Third-pass oracle review (ora-3) of the LiveKit / Element Web expansion caught one CRITICAL and three HIGH bugs that would have caused deploy-time failures. All addressed. CRITICAL — LiveKit key/secret wiring would 401 every Element Call JWT - config/livekit.yaml.example: keys map and webhook were wrong. The keys map is api_key_id: api_secret. The previous template used a hardcoded key ID (matrixrtc) with the secret being the only substituted value, and the webhook used the hardcoded key ID. lk-jwt-service sets the api_key_id as the JWT iss claim; LiveKit looks up the signer via iss → keys map. Wrong key ID = signed JWT with unknown key = 401. Fix: keys: ${LIVEKIT_API_KEY}: "${LIVEKIT_API_SECRET}" and webhook.api_key: ${LIVEKIT_API_KEY}. Both substituted from the corresponding Swarm secrets at render time. - scripts/render-livekit-yaml.sh: now reads BOTH livekit-api-key AND livekit-api-secret, exposes both as env for envsubst, AND has a content-hash idempotency check (compares api_key_id in rendered output vs Swarm secret value) so post-rotation re-runs work even when the template's mtime hasn't changed. New --force flag skips the check for explicit bypass. HIGH — matrixrtc-jwt unreachable from OPNsense Caddy - matrix-swarm-stack.yml: matrixrtc-jwt now has a ports block publishing 8082 (mode: ingress) so it's reachable from the Caddy routing mesh. Removed the (documentation-only) expose block. Removed the redundant secrets mounts from the livekit service (LiveKit reads keys from the rendered yaml, not /run/secrets/). HIGH — Caddy config in README used Docker service names - README.md: matrix-rtc.coppercore.net block now points at explicit manager IPs (10.10.10.211:8082 for the jwt service with /healthz health check; 10.10.10.211:7880 for LiveKit with / health check). OPNsense Caddy can't resolve Docker service names from outside the Swarm cluster. FALSE ALARM — lk-jwt-service:0.7.0 was flagged as nonexistent - Verified via webfetch of the GitHub releases page: v0.7.0 was released 2026-09-10 as the current Latest (Rust rewrite, documented backwards-compatible). Stack remains pinned to 0.7.0. If the Rust rewrite causes first-deploy issues, fall back to 0.6.0 (also verified to exist). MEDIUM — generate-homeserver-yaml.sh missing CephFS mountpoint guard - scripts/generate-homeserver-yaml.sh: same mountpoint -q check the other two render/sync scripts use, prevents silent local-file fallback when CephFS is down. MEDIUM — extra_well_known_client_content serving location comment - config/homeserver.yaml.example + CLAUDE.md: corrected. Synapse serves the well-known at public_baseurl + /.well-known/matrix/client (i.e. https://matrix.coppercore.net/...), NOT at the apex. The apex well-known is VPS Caddy only and does NOT include rtc_foci. LOW — Dead jitsi config in element-config.json - config/element-config.json: removed 'jitsi': { preferred_domain: 'meet.jit.si' }. element_call.use_exclusively: true already disables the Jitsi path, so the key was dead config. Verified locally: PyYAML parses all yaml, json.load parses all json, bash -n passes on every script, no Docker service names remain in any external-facing Caddy block.sec-1 (second-opinion / local ornith model) was invoked after ora-3 to sanity-check the work. Its output truncated (recurring pattern with multi-stage local-LLM agents), but the captured thinking surfaced two real bugs the oracles had missed. HIGH — README §7 health-check command built an invalid container ref - README.md line 412 used: docker exec matrix_matrixrtc-jwt.1.$(docker ps -q -f name=... | head -1) 'docker ps -q' returns a full CONTAINER ID, not a task ID. The concatenated string 'matrix_matrixrtc-jwt.1.<containerid>' does not resolve to anything in Docker — 'docker exec' needs a container ID, a container name, or 'matrix_matrixrtc-jwt.1.<task_id>'. Running this command produces 'Error: No such container'. - Fix: drop the 'matrix_matrixrtc-jwt.1.' prefix; use just the container ID returned by 'docker ps -q' (which IS a valid exec target). MEDIUM — render-livekit-yaml.sh idempotency parse was dead - scripts/render-livekit-yaml.sh used an awk one-liner whose pattern '/^[a-zA-Z0-9_]+:/' required the lines-after-'keys:' to start with alphanumeric — but envsubst produces indented YAML: keys: "abc123...": "secret" leading 2-space indent made the regex never match, so 'old_key_in_file' was always empty, so the idempotency short-circuit always failed, so the script always re-rendered (and re-chmod-ed) the file on every run. The header comment promised 'skips when the rendered output matches the live secret' but that path was unreachable. - Verified empirically (TEST 1 in the sec-1 turn): rendered file with the exact shape envsubst produces, awk returns ''. - Fix: replaced the awk with a PyYAML single-liner that loads the YAML and picks the first key under the 'keys' map. PyYAML is already verified working on this repo's stack files (used by the existing parse check). - Verified empirically: PyYAML extracts 'abc123def456789' correctly from the same fixture the awk failed on. What the oracles missed and sec-1 caught - awk-based YAML parse that's fragile against envsubst formatting (the rendered keys are indented+quoted, naive regex never matches) - container-ID vs task-ID confusion in 'docker exec ...' invocations (a real footgun in operator docs that's only visible if you actually copy-paste the command and run it against a live cluster) Both fixes are local edits in the same commit. Verified locally: PyYAML extracts the api_key_id correctly on the realistic rendered fixture; bash -n passes on every script; all yaml/json parses. Sec-1 model test takeaway: the local ornith model produces high-quality substantive analysis (real, specific findings with file:line citations) in its thinking, but its output budget truncates before formal findings are emitted — same failure mode as fix-2 and fix-3 on the long-running background agents. Worth flagging for the user's awareness.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.