Blog: How Tos
Software BTRFS RAID1 with LUKS2 FDE
The guide below is simplified in a way that preparing the boot partition is not covered.
Software based btrfs RAID1 requires two devices, which conceptually don’t even need to be on different disks. But for obvious reasons, it’s a good idea if they are…
Having mirroring against encrypted storage brings its own set of challenges: It’s easy to encrypt a mirrored device (e.g. using md whereby each device has exactly the same data blocks), then use LVM and file systems on top. It’s not so easy to have this done if the mirroring is done at a layer “higher” than the encryption, since it is necessary to deal with multiple individually encrypted devices. Which is fine. More complicated but certainly not impossible.
This post aims to guide through implementing btrfs mirroring on two encrypted storage devices.
The basic requirement for btrfs RAID1 is having two partitions, ideally in different disks, as described below:
# fdisk -l /dev/sda Disk /dev/sda: 978.1 GiB, 1050214588416 bytes, 2051200368 sectors Disk model: Crucial_CT1050MX Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: D2E25ADE-25A7-F056-9982-F40C48B55D53 Device Start End Sectors Size Type /dev/sda1 2048 526335 524288 256M EFI System /dev/sda2 526336 1574911 1048576 512M BIOS boot /dev/sda3 1574912 2623487 1048576 512M Linux filesystem /dev/sda4 2623488 935856127 933232640 445G Linux filesystem
Similarly, there was a /dev/sdb4 on the second disk which had the same size. For btrfs, it is not a requirement that both devices have the same size, since the RAID setup will take as much space as the smallest device allows. Since LVM2 will be in place, all components of the system will live under /dev/sda4 and /dev/sdb4 as logical volumes sitting on encrypted physical volumes.
The disks in this example were partitioned with a GPT label. This is helpful but not required for the purpose of this howto.
Lastly the boot partitions (/dev/sda3 and /dev/sdb3) were equally configured in a btrfs raid1 layout, except these were not under LUKS2 encryption.
LUKS2 encryption and boot tricks
Before considering the file system, the partitions need encrypting using cryptsetup to implement luks2.
For the purpose of this tutorial, and for the boot process to be simple, both devices are encrypted with the same passphrase. This means the actual keys are unique per device but the passphrase used to decrypt the storage encryption keys is the same on both devices. This is optional, just makes life easier.
Volume names ‘crypt-d1’ and ‘crypt-d2’ were chosen, which will tie in with /etc/crypttab’s content.
# cryptsetup luksFormat /dev/sda4 --type luks2 # cryptsetup luksOpen /dev/sda4 crypt-d1 # cryptsetup luksFormat /dev/sdb4 --type luks2 # cryptsetup luksOpen /dev/sdb4 crypt-d2
At this point, it’s worth noting down the respective partition UUIDs and creating /etc/crypttab
# echo "crypt-d1 PARTUUID=$(lsblk /dev/sda4 -o partuuid -n) btrfs_r1 luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl > /etc/crypttab # echo "crypt-d2 PARTUUID=$(lsblk /dev/sdb4 -o partuuid -n) btrfs_r1 luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl >> /etc/crypttab
Beyond the automation, here’s what /etc/crypttab looks like:
# cat /etc/crypttab crypt-d1 PARTUUID=ab4f59d1-8e44-2839-8e8f-3b42e2db5192 btrfs_r1 luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl crypt-d2 PARTUUID=aa3a4e05-84b7-b758-b66f-bbafa1271f6a btrfs_r1 luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl
The above lines under /etc/crypttab have the following impact:
- labels crypt-d1 and crypt-d2 as ‘btrfs_r1’ for the purpose of handling by cryptsetup tools;
- Once the passphrase has been entered for crypt-d1, it’s handled by the decrypt_keyctl to assign transparently to crypt-d2 (and any other device matching the ‘btrfs_r1’ label);
Now that the encrypted volumes exist and are available, LVM is used to manage the encrypted space. Each encrypted device will become a physical volume (PV) and hold its own volume group (VG) and logical volumes (LV).
# pvcreate /dev/mapper/crypt-d1 # vgcreate disk1 /dev/mapper/crypt-d1 # lvcreate -l 90%VG -n root disk1 # lvcreate -l 100%VG -n swap disk1 ### repeat for PV crypt-d2 into a VG 'disk2' ###
After setting up, the layout should be similar to the following:
... ├─sda4 8:4 0 445G 0 part │ └─crypt-d1 253:0 0 445G 0 crypt │ ├─disk1-swap 253:1 0 16G 0 lvm [SWAP] │ └─disk1-root 253:2 0 429G 0 lvm / ... └─sdb4 8:20 0 445G 0 part | └─crypt-d2 253:3 0 445G 0 crypt | ├─disk2-swap 253:4 0 16G 0 lvm [SWAP] | └─disk2-root 253:5 0 429G 0 lvm ...
You will notice that until this point both disks have been setup individually, each with their own encrypted space and LVM layout.
# pvs PV VG Fmt Attr PSize PFree /dev/mapper/crypt-d1 disk1 lvm2 a-- 444.98g 0 /dev/mapper/crypt-d2 disk2 lvm2 a-- 444.98g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root disk1 -wi-ao---- 428.98g swap disk1 -wi-ao---- 16.00g root disk2 -wi-ao---- 428.98g swap disk2 -wi-ao---- 16.00g
btrfs RAID1 filesystem
It is at this point that btrfs makes an appearance. First by being created under /dev/mapper/disk1–root, then by being extended under RAID1 into /dev/mapper/disk2-root.
# mkfs.btrfs /dev/disk1/root # mount /dev/disk1/root /mnt/root # btrfs device add /dev/disk2/root /mnt/root # btrfs filesystem label /mnt/root root-r1 # btrfs filesystem balance start -dconvert=raid1 -mconvert=raid2 /mnt/root
The above sequence of commands creates a btrfs filesystem under /dev/disk1/root, labels it as ‘root-r1’, adds the device /dev/disk2/root and balances the file system across the two devices in a raid1 configuration.
Checking the btrfs RAID filesystem can be done using btrfs filesystem usage and btrfs device usage as shown below:
# btrfs filesystem usage / Overall: Device size: 857.96GiB Device allocated: 258.06GiB Device unallocated: 599.90GiB Device missing: 0.00B Used: 248.08GiB Free (estimated): 303.13GiB (min: 303.13GiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 138.28MiB (used: 0.00B) Data,RAID1: Size:127.00GiB, Used:123.82GiB /dev/mapper/disk1-root 127.00GiB /dev/mapper/disk2-root 127.00GiB Metadata,RAID1: Size:2.00GiB, Used:226.30MiB /dev/mapper/disk1-root 2.00GiB /dev/mapper/disk2-root 2.00GiB System,RAID1: Size:32.00MiB, Used:48.00KiB /dev/mapper/disk1-root 32.00MiB /dev/mapper/disk2-root 32.00MiB Unallocated: /dev/mapper/disk1-root 299.95GiB /dev/mapper/disk2-root 299.95GiB
# btrfs device usage / /dev/mapper/disk1-root, ID: 1 Device size: 428.98GiB Device slack: 0.00B Data,RAID1: 127.00GiB Metadata,RAID1: 2.00GiB System,RAID1: 32.00MiB Unallocated: 299.95GiB /dev/mapper/disk2-root, ID: 2 Device size: 428.98GiB Device slack: 0.00B Data,RAID1: 127.00GiB Metadata,RAID1: 2.00GiB System,RAID1: 32.00MiB Unallocated: 299.95GiB
For the purpose of fstab, both /dev/disk1/root and /dev/disk2/root will have the same file system UUID so either can be mounted. However, using the UUID will solve the problem transparently.
Conceptually, after swap space has been created under /dev/disk1/swap and /dev/disk2/swap (not RAID), then root and swap entries on fstab can be created as follows:
echo " UUID=$(lsblk /dev/disk1/root -o uuid -n) / btrfs defaults 0 0 UUID=$(lsblk /dev/disk1/swap -o uuid -n) none swap sw 0 0 " >> /etc/fstab
Notice that under fstab UUID is used, whereas under /etc/crypttab PARTUUID was used instead. Only one UUID was necessary to reference the root file system (not both). Also notice that since PARTUUID and UUIDs are used, there’s no fiddling with device names or locations – if you’re not using UUIDs instead of device paths, do it, it will change your life. Here’s what fstab looks like:
# cat /etc/fstab UUID= /boot btrfs defaults 00 UUID= /boot/efi vfat defaults,noatime 0 0 UUID=e76a6d38-096c-301b-f56b-ab0d7170f725 / btrfs defaults 00 UUID=ad711836-473e-2449-2bde-5419c7cef40f none swap sw 0 0 UUID=316d8414-371f-6910-3b5f-cce6b1dd0bf9 none swap sw 0 0
Boot loader and initramfs
Ensuring that the boot loader and initramfs are setup correctly are not entirely trivial matters. If the above has been done on a running system
- with existing /boot partition (with bootloader files)
- with correctly labelled LUKS2 volumes (crypt-d1 and crypt-d2)
- with correct LVM names (disk1 and disk2)
- with correctly constructed /etc/fstab and /etc/crypttab,
then running the following sequence of commands should enable preparation of the initramfs with the correct cryptsetup binaries and helper scripts (decrypt_keyctl) as well as correctly constructed grub.conf.
# update-initramfs -u # update-grub # grub-install /dev/sda # grub-install /dev/sdb