Verify idle3 fix held: re-check Load_Cycle_Count on the USB HDD tier (~2026-08-02) #39

New issue

Closed

opened 2026-07-19 21:33:09 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-19 21:33:09 +00:00

Collaborator

WD idle3 head parking was disabled on osd.12/osd.13 on 2026-07-19 (osd.14 already had it disabled). The change lives in drive NVRAM and was verified after a physical power cycle — but the only proof it's actually working in production is the load-cycle counter not growing.

Baselines at the moment idle3 was disabled (2026-07-19):

OSD	node	Load_Cycle_Count
osd.12	node01	8187
osd.13	node04	8026
osd.14	node05	1328

To check:

# on node01 / node04 / node05 — identify by serial, not /dev/sdX
smartctl -A /dev/sdX | awk '$1==193'
hdparm -J /dev/sdX          # should still read "disabled"

Interpreting it: single-digit / low-tens growth over ~2 weeks = fix held, close this. Growth in the hundreds (the old rate was ~40/hr on osd.12, i.e. ~13k over two weeks) = it silently didn't take, and the drive needs re-checking — hdparm -J is a reverse-engineered implementation, so a silent no-op is a real possibility.

Also worth a glance while you're there: whether BLUESTORE_SLOW_OP_ALERT has stopped recurring. If it's still tripping regularly under normal load with SMART clean, that's the trigger to scope bluestore_slow_ops_warn_threshold to osd.12/13/14 only — deliberately left at the stock 1 for now rather than muting an alert we hadn't characterised.

Background and the full evidence table: usb-hdd-ceph-tier/README.md, "WD idle3 (Intellipark) head parking" section. Related: #37.

WD idle3 head parking was disabled on `osd.12`/`osd.13` on 2026-07-19 (`osd.14` already had it disabled). The change lives in drive NVRAM and was verified after a physical power cycle — but the only proof it's actually working in production is the load-cycle counter not growing. **Baselines at the moment idle3 was disabled (2026-07-19):** | OSD | node | Load_Cycle_Count | |---|---|---| | osd.12 | node01 | 8187 | | osd.13 | node04 | 8026 | | osd.14 | node05 | 1328 | **To check:** ```bash # on node01 / node04 / node05 — identify by serial, not /dev/sdX smartctl -A /dev/sdX | awk '$1==193' hdparm -J /dev/sdX # should still read "disabled" ``` **Interpreting it:** single-digit / low-tens growth over ~2 weeks = fix held, close this. Growth in the hundreds (the old rate was ~40/hr on osd.12, i.e. ~13k over two weeks) = it silently didn't take, and the drive needs re-checking — `hdparm -J` is a reverse-engineered implementation, so a silent no-op is a real possibility. Also worth a glance while you're there: whether `BLUESTORE_SLOW_OP_ALERT` has stopped recurring. If it's still tripping regularly under normal load with SMART clean, that's the trigger to scope `bluestore_slow_ops_warn_threshold` to `osd.12/13/14` only — deliberately left at the stock `1` for now rather than muting an alert we hadn't characterised. Background and the full evidence table: `usb-hdd-ceph-tier/README.md`, "WD idle3 (Intellipark) head parking" section. Related: #37.

claude-bot added the

ceph

p:low

labels

2026-07-19 21:48:05 +00:00

claude-bot commented

2026-08-10 22:12:38 +00:00

Author

Collaborator

Closing — the fix was confirmed to have held, and the hardware is now gone, so there is nothing further to measure.

Final reading, 2026-08-08 (last check before decommission), against the 2026-07-19 baselines:

OSD	Baseline	Final	Δ	Hours	Rate after fix	Rate before fix
osd.12	8187	8401	+214	442	0.48/hr	~39.7/hr
osd.13	8026	8233	+207	458	0.45/hr	~14.6/hr
osd.14	1328	2519	+1191	459	2.59/hr	0.39/hr (lifetime)

osd.12 and osd.13 came in at roughly 75x and 30x better than their pre-fix rates — projected ~4k cycles/year against a 300k–1M rating, versus ~348k/year before. hdparm -J reported disabled on all three at every check, including on osd.12's drive after it survived node01's hard failure, a physical move to node02, and an OSD destroy/recreate — good evidence the setting genuinely persisted in drive NVRAM rather than being echoed back.

osd.14 (the control, factory-disabled) drifted upward across both windows and ended highest of the three. Never a concern in absolute terms (~23k/year), but it does show idle3 was the dominant parking mechanism, not the only one. Not worth chasing now.

BLUESTORE_SLOW_OP_ALERT also stopped recurring on these OSDs — the only later occurrence was on osd.4, an unrelated SSD. bluestore_slow_ops_warn_threshold was left at the stock 1 and never tuned, which was the right call: the original alert turned out to be 11 events, 9 of them in a single 0.5s window during the first post-migration borg full re-read.

Now moot: the tier was decommissioned 2026-08-08 — hdd-bulk deleted, osd.12/13/14 removed, drives physically pulled. Nextcloud's data moved to the swarm-data-ec EC 4+2 pool. The idle3 findings and the replug/stale-device-mapper gotcha remain documented in usb-hdd-ceph-tier/README.md for any future USB-backed tier.

Closing — **the fix was confirmed to have held**, and the hardware is now gone, so there is nothing further to measure. **Final reading, 2026-08-08** (last check before decommission), against the 2026-07-19 baselines: | OSD | Baseline | Final | Δ | Hours | Rate after fix | Rate before fix | |---|---|---|---|---|---|---| | osd.12 | 8187 | 8401 | +214 | 442 | **0.48/hr** | ~39.7/hr | | osd.13 | 8026 | 8233 | +207 | 458 | **0.45/hr** | ~14.6/hr | | osd.14 | 1328 | 2519 | +1191 | 459 | 2.59/hr | 0.39/hr *(lifetime)* | osd.12 and osd.13 came in at roughly **75x and 30x** better than their pre-fix rates — projected ~4k cycles/year against a 300k–1M rating, versus ~348k/year before. `hdparm -J` reported `disabled` on all three at every check, including on osd.12's drive after it survived node01's hard failure, a physical move to node02, and an OSD destroy/recreate — good evidence the setting genuinely persisted in drive NVRAM rather than being echoed back. osd.14 (the control, factory-disabled) drifted upward across both windows and ended highest of the three. Never a concern in absolute terms (~23k/year), but it does show idle3 was the dominant parking mechanism, not the only one. Not worth chasing now. `BLUESTORE_SLOW_OP_ALERT` also stopped recurring on these OSDs — the only later occurrence was on `osd.4`, an unrelated SSD. **`bluestore_slow_ops_warn_threshold` was left at the stock `1`** and never tuned, which was the right call: the original alert turned out to be 11 events, 9 of them in a single 0.5s window during the first post-migration borg full re-read. **Now moot:** the tier was decommissioned 2026-08-08 — `hdd-bulk` deleted, osd.12/13/14 removed, drives physically pulled. Nextcloud's `data` moved to the `swarm-data-ec` EC 4+2 pool. The idle3 findings and the replug/stale-device-mapper gotcha remain documented in `usb-hdd-ceph-tier/README.md` for any future USB-backed tier.

claude-bot closed this issue

2026-08-10 22:12:42 +00:00