Evaluate BlueStore compression for swarm-data/vm-data pools #15

New issue

Open

opened 2026-07-06 11:32:20 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-06 11:32:20 +00:00

Collaborator

Unlike RGW compression (docs.ceph.com/en/latest/radosgw/compression/), which
only applies to RadosGW/S3 objects and doesn't touch our CephFS/RBD-only
setup, BlueStore compression is pool-level and transparent to whatever
interface sits on top (CephFS, RBD) - it's the one actually relevant here.

Unlike Ceph dedup (docs.ceph.com/en/latest/dev/deduplication/, evaluated and
rejected 2026-07-06 - "highly experimental", manual ceph-dedup-tool workflow,
RGW-focused), BlueStore compression is a mature, non-experimental, long-stable
feature. Off by default (bluestore_compression_mode=none).

Candidate pools:

swarm-nvme: skip - pinned there specifically for Postgres/MariaDB/Valkey
latency (see docs/homelab-architecture.md), compression adds CPU/latency
overhead we don't want there.
swarm-data / vm-data: worth a pilot. Forgejo git objects (already
zlib-compressed internally, low additional savings expected), Nextcloud
user files (mixed compressibility), VM boot disks (moderate savings
likely).

Suggested approach: ceph osd pool set swarm-data compression_mode passive
(passive, not aggressive/force, to avoid CPU cost on incompressible data),
algorithm left at default snappy (docs explicitly warn zstd has high CPU
overhead for BlueStore's typical small-object sizes). Watch OSD CPU and
ceph df savings for ~1 week before deciding whether to extend to vm-data.

No p:medium label exists in this tool's scheme (p:high/p:low only) - filed as
p:low since this is a low-urgency "worth investigating" item, not blocking
anything. Bump to p:high if you want it picked up sooner.

Prompted by seeing others use it, and this Red Hat writeup on Ceph Storage 3.3 BlueStore compression performance: https://www.redhat.com/en/blog/red-hat-ceph-storage-33-bluestore-compression-performance Unlike RGW compression (docs.ceph.com/en/latest/radosgw/compression/), which only applies to RadosGW/S3 objects and doesn't touch our CephFS/RBD-only setup, BlueStore compression is pool-level and transparent to whatever interface sits on top (CephFS, RBD) - it's the one actually relevant here. Unlike Ceph dedup (docs.ceph.com/en/latest/dev/deduplication/, evaluated and rejected 2026-07-06 - "highly experimental", manual ceph-dedup-tool workflow, RGW-focused), BlueStore compression is a mature, non-experimental, long-stable feature. Off by default (bluestore_compression_mode=none). Candidate pools: - swarm-nvme: skip - pinned there specifically for Postgres/MariaDB/Valkey latency (see docs/homelab-architecture.md), compression adds CPU/latency overhead we don't want there. - swarm-data / vm-data: worth a pilot. Forgejo git objects (already zlib-compressed internally, low additional savings expected), Nextcloud user files (mixed compressibility), VM boot disks (moderate savings likely). Suggested approach: `ceph osd pool set swarm-data compression_mode passive` (passive, not aggressive/force, to avoid CPU cost on incompressible data), algorithm left at default `snappy` (docs explicitly warn zstd has high CPU overhead for BlueStore's typical small-object sizes). Watch OSD CPU and `ceph df` savings for ~1 week before deciding whether to extend to vm-data. No p:medium label exists in this tool's scheme (p:high/p:low only) - filed as p:low since this is a low-urgency "worth investigating" item, not blocking anything. Bump to p:high if you want it picked up sooner.

claude-bot added the

ceph

p:low

labels

2026-07-06 11:32:20 +00:00