swarm-wkr-01 silently drops off a service's VXLAN overlay network on reschedule #33

New issue

Closed

opened 2026-07-18 02:50:23 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-18 02:50:23 +00:00

Collaborator

Symptom

2026-07-18: SearXNG search requests were returning HTTP 500 for roughly 1-in-3 requests (whichever landed on the replica on swarm-wkr-01 via the routing mesh). Application traceback was valkey.exceptions.TimeoutError: Timeout connecting to server — the searxng_searxng replica on swarm-wkr-01 couldn't reach the shared searxng_valkey instance (running on swarm-mgr-03) on the searxng_searxng-internal overlay network.

Root cause

swarm-wkr-01 had silently failed to (re)join the VXLAN tunnel for the searxng_searxng-internal network (vni 4100). Confirmed by tcpdump -i eth0 udp port 4789 on both ends during a live connect attempt from a wkr-01 container to valkey:6379: zero VXLAN packets left wkr-01 for that network, while the same capture window showed active vni 4100 traffic between mgr-01/mgr-02/mgr-03/wkr-02. Basic node-to-node reachability was fine (ICMP, UDP/4789 port open) — this was Docker's overlay-network gossip/membership state on wkr-01 getting stuck for that one network, not a firewall/routing/MTU problem.

Fix applied

docker service update --force searxng_searxng — forced all 3 tasks to reschedule, and the new wkr-01 task rejoined the overlay cleanly (connect_ex: 0 to valkey afterward, confirmed via docker exec ... python3 socket.connect_ex). No more TimeoutError/500s in service logs post-fix.

Why this matters beyond SearXNG

swarm-wkr-01 also hosts nextcloud_redis (shared, single-replica) and forgejo_forgejo, both of which have cross-node dependents (nextcloud web replicas on mgr-01/mgr-02 reaching redis on wkr-01; nextcloud web on wkr-01 reaching nextcloud_db on mgr-03 — the same node pair that broke for SearXNG). Checked both post-incident and they're currently healthy, but if either of those services is ever force-rescheduled and hits the same gossip glitch, it would present as Nextcloud 500s/file-locking errors rather than SearXNG ones.

Open questions / follow-up

Root cause of why wkr-01 got stuck on that one overlay network is still unknown — this looks like a known class of Docker Swarm overlay-network gossip bug, not something specific to this cluster's config.
No monitoring currently catches this class of failure (silent per-node overlay drop) proactively — it was only caught because a user reported search 500s. Worth considering a periodic cross-node connectivity check for shared single-replica dependencies (valkey/redis/postgres/mariadb), since these are exactly the services most exposed to this failure mode (any client on any node needs to reach them, and they're often not the one node that gets rescheduled/tested).
If this recurs, check docker network inspect <net> container membership across nodes and/or restarting the Docker daemon on the affected node (more disruptive — bounces every overlay network on that node, not just the affected service).

Repro / verification commands used

# confirm task scheduling
docker service ps <service> --no-trunc

# confirm overlay traffic (or lack of it) during a connect attempt
sudo tcpdump -ni eth0 udp port 4789

# test raw connectivity from inside a container
docker exec <cid> python3 -c "import socket; s=socket.socket(); s.settimeout(3); print(s.connect_ex(('<target>', <port>)))"

# force reschedule
docker service update --force <service>

## Symptom 2026-07-18: SearXNG search requests were returning HTTP 500 for roughly 1-in-3 requests (whichever landed on the replica on `swarm-wkr-01` via the routing mesh). Application traceback was `valkey.exceptions.TimeoutError: Timeout connecting to server` — the `searxng_searxng` replica on `swarm-wkr-01` couldn't reach the shared `searxng_valkey` instance (running on `swarm-mgr-03`) on the `searxng_searxng-internal` overlay network. ## Root cause `swarm-wkr-01` had silently failed to (re)join the VXLAN tunnel for the `searxng_searxng-internal` network (vni 4100). Confirmed by `tcpdump -i eth0 udp port 4789` on both ends during a live connect attempt from a `wkr-01` container to `valkey:6379`: **zero VXLAN packets left `wkr-01`** for that network, while the same capture window showed active vni 4100 traffic between `mgr-01`/`mgr-02`/`mgr-03`/`wkr-02`. Basic node-to-node reachability was fine (ICMP, UDP/4789 port open) — this was Docker's overlay-network gossip/membership state on `wkr-01` getting stuck for that one network, not a firewall/routing/MTU problem. ## Fix applied `docker service update --force searxng_searxng` — forced all 3 tasks to reschedule, and the new `wkr-01` task rejoined the overlay cleanly (`connect_ex: 0` to valkey afterward, confirmed via `docker exec ... python3 socket.connect_ex`). No more `TimeoutError`/500s in service logs post-fix. ## Why this matters beyond SearXNG `swarm-wkr-01` also hosts `nextcloud_redis` (shared, single-replica) and `forgejo_forgejo`, both of which have cross-node dependents (`nextcloud` web replicas on `mgr-01`/`mgr-02` reaching `redis` on `wkr-01`; `nextcloud` web on `wkr-01` reaching `nextcloud_db` on `mgr-03` — the *same* node pair that broke for SearXNG). Checked both post-incident and they're currently healthy, but if either of those services is ever force-rescheduled and hits the same gossip glitch, it would present as Nextcloud 500s/file-locking errors rather than SearXNG ones. ## Open questions / follow-up - Root cause of *why* `wkr-01` got stuck on that one overlay network is still unknown — this looks like a known class of Docker Swarm overlay-network gossip bug, not something specific to this cluster's config. - No monitoring currently catches this class of failure (silent per-node overlay drop) proactively — it was only caught because a user reported search 500s. Worth considering a periodic cross-node connectivity check for shared single-replica dependencies (valkey/redis/postgres/mariadb), since these are exactly the services most exposed to this failure mode (any client on any node needs to reach them, and they're often not the one node that gets rescheduled/tested). - If this recurs, check `docker network inspect <net>` container membership across nodes and/or restarting the Docker daemon on the affected node (more disruptive — bounces every overlay network on that node, not just the affected service). ## Repro / verification commands used ```bash # confirm task scheduling docker service ps <service> --no-trunc # confirm overlay traffic (or lack of it) during a connect attempt sudo tcpdump -ni eth0 udp port 4789 # test raw connectivity from inside a container docker exec <cid> python3 -c "import socket; s=socket.socket(); s.settimeout(3); print(s.connect_ex(('<target>', <port>)))" # force reschedule docker service update --force <service> ```

claude-bot added the

p:medium

swarm-nodes

labels

2026-07-18 02:50:30 +00:00

claude-bot closed this issue

2026-09-03 01:56:19 +00:00