cross-posted from: https://hexbear.net/post/2006806

In response to a post made by @actual_patience@programming.dev about a month ago, asking if anybody has tried Bcachefs.

I recently got around to giving Bcachefs a shot. I am currently running a Gentoo system with Bcachefs as the root filesystem. The filesystem consists of a 1.5TB NVMe volume backed by a conventional 6TB hard disk drive.

Getting it set up was no picnic. I might be missing something, but as far as I can tell a lot of standard tooling lacks support for filesystems spanning multiple block devices (single-partition Bcachefs seems to work like any other FS, but that defeats the point of using Bcachefs). It seems most of them assume you are going to use LVM for a use case like this, while Bcachefs sidesteps the device-mapper interface (and everything built on top, like LUKS and LVM) completely.

At the moment, Bcachefs uses a colon-separated list to specify these block devices. From things like the initramfs generated by dracut or mkinitcpio to utilities included in util-linux, this syntax is not widely supported. A multi-device Bcachefs filesystem can be mounted with mount. The command looks something along the lines of mount -t bcachefs /dev/nvme0n1p1:/dev/sda1 /mnt. These multi-device lists only work if you specify the filesystem type explicitly, and I had to roll my own initramfs from scratch in order to take this input from the kernel command line and handle it specially. Though mount works, libraries like libblkid (and probably others) report suspicious results about the volume / constituent partitions (i.e. that only one of the constituent partitons is mounted). That said, I am not running bleeding versions of Linux, util-linux, or bcachefs. Unstable maybe, but provided by versioned ebuilds in Gentoo. It is also worth noting, it is not currently possible to mount Bcachefs subvolumes the way you can Btrfs subvolumes. The only way this can be done currently is via bind mounts. There is an issue open to implement this feature, but it is not currently a high priority.

The filesystem itself appears to be working smoothly, though there are several rough edges (for instance, I don't think you can currently change a filesystem label after formatting it. Also it struggles when there is not enough space available to reach the replicas_required target. I didn't get an error, it just got super slow).

Before trying this out, I used to run Btrfs in a multi-disk LVM which performed ssd caching at the block layer. Performance on Bcachefs isn't noticeably worse or better IMO, but I haven't done any remotely empirical side-by-side benchmarks. I just got this running yesterday and will need more time to assess performance. I will need to put a lot more data into this filesystem before it starts missing cache on the hot drive, for instance.

Bcachefs is certainly in an experimental state (as the Linux menuconfig tool will remind you), but it aims to do exactly what I am looking for in a filesystem so I had to try it out. I plan on sticking with it unless a catastrophe occurs.