copper/ccnet-prod-devops
1
0
Fork 0
Code Issues 23 Pull requests 2 Projects Releases Packages Wiki Activity Actions

Add usb-hdd-ceph-tier: hdd-bulk Ceph OSD pool, now backing Nextcloud data #34

Merged
claude-bot merged 6 commits from worktree-usb-hdd-ceph-tier into main 2026-07-23 02:20:57 +00:00
Conversation 0 Commits 6 Files changed 8 +940 -1
claude-bot commented 2026-07-18 16:59:00 +00:00
Collaborator
Copy link

Summary

  • Scripts to stand up a low-priority Ceph OSD pool (hdd-bulk) backed by 3 USB3 HDDs, one each on node01/node04/node05 (the non-mon-quorum pilot group already used for the NIC bonding rollout)
  • hdd-bulk is live: size=2 min_size=1, hdd-usb3 crush rule, snappy/aggressive compression, all 3 OSDs up/in, cluster HEALTH_OK.
  • WAL/DB offloaded onto a 60GB LV per node, in the actual free gap between the existing p4 and a pre-existing p5 (the boot NVMe's real layout differs from what docs/homelab-architecture.md describes — see README).
  • bluestore_prefer_deferred_size_hdd tuning was tested live with a real on-cluster A/B and found to be a no-op (bluestore_max_blob_size_hdd is also 65536 on this cluster, so it can never matter) — removed from the plan entirely, left at the stock default.
  • Two real bugs found running the original scripts against actual hardware, both fixed: drive lookup was string-matching the wrong serial encoding against /dev/disk/by-id/, and pveceph osd create --db_dev rejects a pre-made LV (now uses ceph-volume lvm create --block.db directly).
  • Now consuming real data (2026-07-19): hdd-bulk attached to swarm-data-fs as a second CephFS data pool; Nextcloud's data directory pinned to it (existing users migrated via two-pass rsync, ~332GiB relocated; new users inherit automatically). appdata_*/files_external explicitly pinned back to swarm-data. Full detail in docs/homelab-build-history.md Item 13. Pre-migration copies kept as <user>-old-ssd, tracked for deletion in issue #37.
  • WD idle3 head parking disabled on all 3 drives (2026-07-19), found while investigating a BLUESTORE_SLOW_OP_ALERT. Drives shipped at the factory-default 8s park timer; osd.14 happened to arrive with idle3 disabled, giving a controlled same-model comparison — 0.39 load cycles/hr vs ~40/hr on osd.12, i.e. ~350k cycles/year against a 300k–1M rating. Applied via hdparm -J 0 (idle3ctl can't reach it over USB/SAT; these are native-USB drives with no SATA path). The alert itself was deliberately not muted — bluestore_slow_ops_warn_threshold stays at the stock 1; it was 11 events total, 9 in one 0.5s window during the first post-migration borg full re-read, with every real-problem discriminator clean.

Test plan

  • bash -n + shellcheck clean on all scripts
  • Ran disable-usb-power-mgmt.sh + prepare-nvme-db.sh + create-osd.sh on node01/node04/node05 — all 3 OSDs up/in, hdd class, bluefs_db_dev confirmed on the NVMe LV
  • Created throwaway hdd-test pool, ran rados bench and RBD-mapped fio (pre-filled image to avoid the thin-provisioning sparse-read artifact) against it
  • Full on-cluster A/B test of the deferred-write tune (config reverted, OSDs restarted, before-run; re-applied, restarted, after-run) — confirmed no measurable difference, reverted to stock default
  • Cleaned up hdd-test pool/image, verified ceph -s HEALTH_OK
  • Created the real hdd-bulk pool via the equivalent of create-pool.sh — confirmed size=2 min_size=1, hdd-usb3 rule, compression settings, HEALTH_OK
  • Pinned Nextcloud data onto it via ceph fs add_data_pool + ceph.dir.layout.pool — verified layout inheritance with real probe files (not just directory xattrs), ownership (www-data:www-data) on every migrated directory, ceph df showing 335GiB stored, ceph -s HEALTH_OK, and nc.coppercore.net/status.php returning 200 post-cutover
  • Disabled idle3 on osd.12/osd.13 one drive at a time under noout — verified the value persisted after a physical power cycle (Power_Cycle_Count incremented, distinguishing a real power cycle from a bus reset), each OSD returned up/in before advancing, noout unset, ending at 15/15 OSDs up and 369/369 PGs active+clean with SMART PASSED on all three

🤖 Generated with Claude Code
https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk

## Summary - Scripts to stand up a low-priority Ceph OSD pool (`hdd-bulk`) backed by 3 USB3 HDDs, one each on node01/node04/node05 (the non-mon-quorum pilot group already used for the NIC bonding rollout) - **`hdd-bulk` is live**: `size=2 min_size=1`, `hdd-usb3` crush rule, `snappy`/`aggressive` compression, all 3 OSDs up/in, cluster `HEALTH_OK`. - WAL/DB offloaded onto a 60GB LV per node, in the actual free gap between the existing `p4` and a pre-existing `p5` (the boot NVMe's real layout differs from what `docs/homelab-architecture.md` describes — see README). - **`bluestore_prefer_deferred_size_hdd` tuning was tested live with a real on-cluster A/B and found to be a no-op** (`bluestore_max_blob_size_hdd` is also 65536 on this cluster, so it can never matter) — removed from the plan entirely, left at the stock default. - Two real bugs found running the original scripts against actual hardware, both fixed: drive lookup was string-matching the wrong serial encoding against `/dev/disk/by-id/`, and `pveceph osd create --db_dev` rejects a pre-made LV (now uses `ceph-volume lvm create --block.db` directly). - **Now consuming real data (2026-07-19)**: `hdd-bulk` attached to `swarm-data-fs` as a second CephFS data pool; Nextcloud's `data` directory pinned to it (existing users migrated via two-pass `rsync`, ~332GiB relocated; new users inherit automatically). `appdata_*`/`files_external` explicitly pinned back to `swarm-data`. Full detail in `docs/homelab-build-history.md` Item 13. Pre-migration copies kept as `<user>-old-ssd`, tracked for deletion in issue #37. - **WD idle3 head parking disabled on all 3 drives (2026-07-19)**, found while investigating a `BLUESTORE_SLOW_OP_ALERT`. Drives shipped at the factory-default 8s park timer; osd.14 happened to arrive with idle3 disabled, giving a controlled same-model comparison — 0.39 load cycles/hr vs ~40/hr on osd.12, i.e. ~350k cycles/year against a 300k–1M rating. Applied via `hdparm -J 0` (`idle3ctl` can't reach it over USB/SAT; these are native-USB drives with no SATA path). **The alert itself was deliberately not muted** — `bluestore_slow_ops_warn_threshold` stays at the stock 1; it was 11 events total, 9 in one 0.5s window during the first post-migration borg full re-read, with every real-problem discriminator clean. ## Test plan - [x] `bash -n` + `shellcheck` clean on all scripts - [x] Ran `disable-usb-power-mgmt.sh` + `prepare-nvme-db.sh` + `create-osd.sh` on node01/node04/node05 — all 3 OSDs up/in, `hdd` class, `bluefs_db_dev` confirmed on the NVMe LV - [x] Created throwaway `hdd-test` pool, ran `rados bench` and RBD-mapped `fio` (pre-filled image to avoid the thin-provisioning sparse-read artifact) against it - [x] Full on-cluster A/B test of the deferred-write tune (config reverted, OSDs restarted, before-run; re-applied, restarted, after-run) — confirmed no measurable difference, reverted to stock default - [x] Cleaned up `hdd-test` pool/image, verified `ceph -s` HEALTH_OK - [x] Created the real `hdd-bulk` pool via the equivalent of `create-pool.sh` — confirmed `size=2 min_size=1`, `hdd-usb3` rule, compression settings, `HEALTH_OK` - [x] Pinned Nextcloud data onto it via `ceph fs add_data_pool` + `ceph.dir.layout.pool` — verified layout inheritance with real probe files (not just directory xattrs), ownership (`www-data:www-data`) on every migrated directory, `ceph df` showing 335GiB stored, `ceph -s` HEALTH_OK, and `nc.coppercore.net/status.php` returning 200 post-cutover - [x] Disabled idle3 on osd.12/osd.13 one drive at a time under `noout` — verified the value persisted **after** a physical power cycle (`Power_Cycle_Count` incremented, distinguishing a real power cycle from a bus reset), each OSD returned `up`/`in` before advancing, `noout` unset, ending at 15/15 OSDs up and 369/369 PGs `active+clean` with SMART `PASSED` on all three 🤖 Generated with Claude Code https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk
Scripts to stand up hdd-bulk on node01/node04/node05 (the non-mon-quorum
pilot group already used for the NIC bonding rollout), following feasibility
testing that found zero USB resets under sustained concurrent load but a
real mixed-workload latency weakness. Pool is size=2/min_size=1 given only
3 OSDs total; WAL/DB is offloaded onto a 60GB NVMe LV per node and
bluestore_prefer_deferred_size_hdd is raised to 128KB before any data is
written, both aimed at that mixed-workload weakness. Compression reuses the
snappy/aggressive settings already live on swarm-data/vm-data.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk
A sleeping/autosuspended drive that takes a few seconds to wake on first
access produces the same symptom as the bridge-reset failure mode the
workstation soak testing specifically checked for and ruled out -- just via
power management instead of a flaky bridge. Disables USB bus-level
autosuspend and SCSI/block-layer runtime PM, persisted via a udev rule
scoped to each drive's USB vendor/product ID, plus a best-effort hdparm
ATA standby-timer disable. Runs first, before prepare-nvme-db.sh/create-osd.sh.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk
Real disk layout differs from docs/homelab-architecture.md's "~317GiB
unpartitioned tail" claim -- a pre-existing 150G p5 already occupies the
disk end on node01/04/05, leaving a ~167GiB gap between p4 and p5 as the
actual free space. prepare-nvme-db.sh now computes that gap live from
parted output instead of assuming a contiguous tail, and creates the new
partition as p6 (not p5, which is taken). Also fixes an idempotency bug
where a re-run would see its own p6 as an obstructing partition.

Two bugs found running the original scripts against real nodes, both
fixed: drive lookup string-matched the ATA serial against the
/dev/disk/by-id/ filename, which actually encodes the USB descriptor's
differently-formatted iSerial and never matches -- now resolved via
smartctl -i's reported serial instead. pveceph osd create --db_dev
rejected a pre-made LV ("unable to get device info for /dev/dm-N") --
now uses ceph-volume lvm create --block.db directly, confirmed working
against all 3 real nodes.

bluestore_prefer_deferred_size_hdd tuning removed entirely: tested live
with a controlled on-cluster A/B (same pre-filled RBD image, same fio
workload, OSDs restarted between runs) and found to be a measurable no-op
-- bluestore_max_blob_size_hdd is also 65536 on this cluster, so writes
are already fragmented into blobs at or below the old threshold before
the deferred decision is ever evaluated. Left at the stock default.

CRUSH rule renamed hdd-replicated -> hdd-usb3 to signify these are USB
drives, matching what's already live on the cluster.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk
Created on the real cluster: size=2/min_size=1, hdd-usb3 crush rule,
snappy/aggressive compression, HEALTH_OK. Not yet consumed by anything.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk
claude-bot changed title from WIP: Add usb-hdd-ceph-tier: low-priority Ceph OSD pool for 3 USB3 HDDs to Add usb-hdd-ceph-tier: hdd-bulk Ceph OSD pool for 3 USB3 HDDs (live) 2026-07-18 23:28:06 +00:00 
Nextcloud's per-user data directories are now pinned to hdd-bulk
(existing and future users inherit it via the data-directory pin,
appdata_*/files_external stay on swarm-data). Records the pinning
order gotcha, the two-pass rsync mechanics, and files that inode
churn breaks borg's cache on the first post-cutover backup.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk
claude-bot changed title from Add usb-hdd-ceph-tier: hdd-bulk Ceph OSD pool for 3 USB3 HDDs (live) to Add usb-hdd-ceph-tier: hdd-bulk Ceph OSD pool, now backing Nextcloud data 2026-07-19 03:05:14 +00:00 
Investigating a BLUESTORE_SLOW_OP_ALERT on osd.12/13/14 found the drives
running WD's factory-default 8s idle3 ("Intellipark") park timer. osd.14
happened to arrive with idle3 already disabled, giving a controlled
comparison on identical hardware: 0.39 load cycles/hr vs ~40/hr on osd.12.
At that rate osd.12 would reach ~350k cycles/year against a mechanism
rated 300k-1M, and each unpark costs ~1s+ on first access -- a plausible
cause of the isolated 5.2s read that helped trip the alert.

Now disabled on all three via `hdparm -J 0`, verified persisted across a
real power cycle. idle3ctl cannot reach it (legacy HDIO_DRIVE_CMD ioctl,
not exposed over USB/SAT), and these are native-USB drives with no SATA
path, so WDIDLE3.EXE isn't an option either.

The alert itself was deliberately NOT muted: bluestore_slow_ops_warn_
threshold stays at the stock 1. It was 11 events total, 9 of them in one
0.5s window during the first post-migration borg full re-read, with every
real-problem discriminator clean. It only looked persistent because
threshold=1 latches the warning for 24h.

Also documents that replugging any of these drives breaks its OSD until
`vgchange -an/-ay` + `ceph-volume lvm activate` refresh the stale
device-mapper table -- the drive re-enumerates to a new /dev/sdX and the
resulting EIO/abort reads as hardware failure but is not.

Docs only; the drive changes were applied by hand and are run-once
provisioning steps (NVRAM + physical power cycle), not scriptable state.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018L4oDuJgxNPK6vGqWskEHk
claude-bot force-pushed worktree-usb-hdd-ceph-tier from d3a0f78548 to a20d8962a7 2026-07-23 02:20:49 +00:00 Compare
claude-bot deleted branch worktree-usb-hdd-ceph-tier 2026-07-23 02:20:57 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked
Waiting on something external before it can proceed

  
borg-backup
ccnet-borg-backup

  
ceph
Ceph pool/CephFS/mon work not tied to one stack

  
forgejo
ccnet-forgejo-swarm

  
in-progress
Actively being worked on right now

  
infra
OPNsense, Caddy, DNS, and other cross-cutting infra

  
netdata
ccnet-netdata-monitoring

  
nextcloud
ccnet-nextcloud-swarm

  
p:high
Should be picked up soon

  
p:low
No urgency, quality-of-life or deferred

  
p:medium
Worth doing, no fixed timeline

  
searxng
ccnet-searxng-swarm

  
service-onboard
ccnet-service_onboard

  
swarm-nodes
ccnet-swarm-nodes-runbook

No labels
blocked
borg-backup
ceph
forgejo
in-progress
infra
netdata
nextcloud
p:high
p:low
p:medium
searxng
service-onboard
swarm-nodes
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
copper/ccnet-prod-devops!34
No description provided.