Encrypted ZFS on Root for Devuan: ZFS with LUKS, without Systemd

Let's say that your hobbies include putting ZFS onto different things, but you have a deep-seated loathing of amateur init systems and their authors.

Let's say this problem is compounded by the fact that every major Linux distribution has adopted same said init system and it can be particularly thorny to remove it and replace it with something, anything sane. What else can you do but curse the darkness? All is lost!

No, my friends. We can light a candle. There's always a way. Sometimes a ridiculous way, but a way nonetheless.

A number of folks who still have their heads about them and understand that some cures are worse than the disease have forked Debian and made it mo' better. The Devuan Linux distribution is still very new at this point and recently hit its 1.0 release. Its goal is "software freedom", which is code for "no systemd", so it's perfect for our goals.

The following is a quick and dirty howto for setting up a Devuan system with encrypted ZFS on root. It is meant to be read and understood in the context of my other ZFS-on-root howtos and is not intended for a beginner ZFS (or Linux) enthusiast. We're taking a big step backwards here and doing some old-school compiling of DKMS modules.

I say "taking a big step backwards" because Canonical, the maker of Ubuntu Linux, took a big step forward by integrating the ZFS kernel module into their kernels from 16.04 onwards. But they also adopted systemd, which is Muy Malo, so we have to roll our own zfs.ko. Twice. But I am getting ahead of myself.

The following howto uses the Devuan Jessie 1.0.0 minimal Live CD ISO (devuan_jessie_1.0.0_amd64_minimal-live.iso), which lacks any desktop environments. We don't need them, we're just going to be typing our way to joy. Moreover, kernel module builds fail on the Devuan desktop live environment, so use the minimal live ISO.

Make sure your system has a drive you don't care about formatting, and at least 4GB of RAM. Attempting this with anything less will result in spurious errors that will impair your progress. ZFS itself enjoys a large memory footprint, so I don't suggest this for any resource-constrained, embedded setups. Proceed at your own risk. These steps come with no warranty whatsoever. Good luck.

I'm assuming that the root drive you're going to format is /dev/sda. Your actual mileage may vary.

Wipe and partition the disk.

wipefs --force --all /dev/sda
# or
dd if=/dev/zero of=/dev/sda bs=1M count=2
/sbin/parted --script --align optimal /dev/sda mklabel msdos
/sbin/parted --script --align optimal /dev/sda mkpart primary 1MiB 100%
/sbin/parted --script --align optimal /dev/sda set 1 boot on

Check that the partition table is correct:

/sbin/parted --script /dev/sda p

You'll notice that we're breaking with tradition of creating a separate partition for /boot and / partitions like we've previously done. Like I said, quick and dirty. I was surprised to see that this works, even under Debian/Devuan. At the risk of potential boot errors, this lets you take ZFS snapshots of your kernel and initrds, which may be really useful to you after a disaster.

Log into the Devuan Live CD. Be sure to understand your network configuration. Linux networking is a Kafkaesque nightmare of conflicting system config file advice and is beyond the scope of this howto. For simplicity, in this howto we assume your machine uses DHCP to get online. Use a real setup for a real server.

dhclient eth0

Add the jessie-backports packages to your apt/apt-get sources.

echo 'deb http://auto.mirror.devuan.org/merged jessie-backports main contrib' > /etc/apt/sources.list.d/backports.list

Update the package sources and install cryptsetup.

time apt update
time apt install -y cryptsetup

Setup your partition into a LUKS container.

cryptsetup luksFormat -h sha512 /dev/sda1
cryptsetup luksOpen /dev/sda1 cryptroot

ZFS on Devuan requires compiling multiple kernel modules and is painful. Canonical took this pain away and yet forces systemd upon us. Curses. Curses I say.

Start by adding the headers for the Live CD kernel.

time apt install -y linux-headers-$(uname -r)

Install the ZFS DKMS package and dependencies. This will pop up a license incompatibility warning that you have to acknowledge. Optionally, you can force apt-get to skip this warning by setting an environment variable. To skip the interactive ZFS license warning screen:

DEBIAN_FRONTEND=noninteractive
export DEBIAN_FRONTEND

Build ZFS.

time apt-get install -y -t jessie-backports zfs-dkms

A quick word about apt versus apt-get. Both work equally well for our purposes. Use whichever you prefer, but be aware that you will need to specify certain exact packages from certain exact repositories. "apt-get install -t reponame package" is the same to us as "apt install package/reponame". I usually choose the command that contains the fewest number of characters.

Keep an eye on the output of this command and watch for any errors building the modules. You'll be prompted to check /var/lib/dkms/something/build/make.log to troubleshoot. Common errors include transient resource constraint problems: running out of memory, running out of disk space on the Live CD file system, et cetera. If you encounter an error, stop. Fix your system. Do not proceed until your setup is entirely correct.

When the zfs-dkms package is installed, you can add the ZFS kernel module to your Live CD setup.

/sbin/modprobe zfs

If you get any errors, stop here and correct them.

Setup your zpool and datasets. This will look familiar if you've read my other ZFS howtos.

zpool create -f -O mountpoint=none -O compression=lz4 -O atime=off -o ashift=12 zdevuan /dev/mapper/cryptroot
zfs create -o mountpoint=/ zdevuan/root
zpool set bootfs=zdevuan/root zdevuan

Optionally, you can create separate datasets for /boot, /home, and so forth. I like to create an unmounted "zpool/root" dataset and then children datasets for my various boot environments (example: "zpool/root/cur", "zpool/root/next", and so on). Theoretically, this helps you rollback bad upgrades without too much swearing. To add a /boot dataset for example:

zfs create -o mountpoint=/boot zdevuan/boot

Create your desired datasets. Unmount the zpool and remount it under /mnt.

zpool export -a
zpool import -R /mnt zdevuan

Since I never bothered looking for where Devuan install discs keep their packages or base image (I think they're under /lib/live. That's what mount suggests.), we're going to set up our new system with debootstrap:

time /usr/sbin/debootstrap jessie /mnt https://auto.mirror.devuan.org/merged

Whatever you do, don't use the --no-check-gpg flag to make debootstrap run faster.

When debootstrap finishes setting up the base system under /mnt, add a few files from your Live CD to the new system:

cp -v -p /etc/apt/sources.list.d/backports.lists /mnt/etc/apt/sources.list.d
cp -v -p /etc/locale.gen /mnt/etc

Before the new system solidifies, it throws errors about missing locales and things of that nature, so I like to use the Live CD environment to edit the new system's config files. Edit /mnt/etc/fstab to the following:

/dev/mapper/cryptroot /     zfs defaults,noatime 0 0
zdevuan/boot          /boot zfs defaults,noatime 0 0

The line that will mount /boot is superfluous if you don't change the zdevuan/boot dataset's mountpoint value to "legacy", but don't skip ahead just yet.

Edit /mnt/etc/crypttab to add the UUID of the cryptroot LUKS container. There are multiple ways to get the UUID. You can run ls -l /dev/disk/by-uuid or you can run blkid. You have options.

blkid /dev/sda1 # get the UUID for /dev/sda1
echo 'cryptroot UUID=UUIDHERE /rootkey.bin luks,keyscript=/bin/cat' > /mnt/etc/crypttab

Create a key for the cryptroot LUKS container. I've written about this previously and people have Strong Opinions about PRNGs. Use whatever makes you feel better. If your entropy pool is taking its own sweet time, you can run this in a tmux or GNU screen window and switch to another window to keep working while dd keeps crunching.

time dd if=/dev/random iflag=fullblock of=/mnt/boot/rootkey.bin bs=512 count=4

You can also use OpenSSL, if you trust OpenSSL. In my testing, OpenSSL can sometimes yield slightly, slightly less probabilistically-indistinguishable-from-noise pseudorandom output than /dev/random. It's very close and significantly faster.

openssl rand -out /mnt/boot/rootkey.bin 2048

Set the hostname.

echo myhostname > /mnt/etc/hostname
echo '127.0.0.1 myhostname' >> /mnt/etc/hosts

Set the networking configuration for your new machine. This can be very complicated, so for this howto we continue to assume we're just going to use DHCP.

vi /mnt/etc/network/interfaces

Add these lines:

auto eth0
iface eth0 inet dhcp

Mount system directories.

cd /
for i in /dev /dev/pts /proc /sys; do mount -B $i /mnt$i; done

Chroot to the new system. Yes, we're going to re-download and install ZFS again.

chroot /mnt /bin/bash --login

Do some system setup.

passwd
passwd -u root
dpkg-reconfigure tzdata
ln -sf /proc/self/mounts /etc/mtab

Add some packages that debootstrap skips. You can also use --include in the debootstrap, but I don't care to make that process any more complicated than it already is. This howto assumes you're running this on an amd64 system. Adjust the "linux-image-<arch>" package accordingly for whatever your intended arch is.

apt update
apt install -y locales kbd # select your locale here
time apt install -y cryptsetup linux-image-amd64
time apt install -y linux-headers-$(uname -r) # Again
time apt-get install -y -t jessie-backports zfs-dkms zfs-initramfs

Hopefully by now the LUKS key has finished getting written. Create a crypto keyfile hook for the initramfs and add the key to the LUKS container.

cryptsetup luksAddKey /dev/sda1 /boot/rootkey.bin
chmod 0 /boot/rootkey.bin
vi /etc/initramfs-tools/hooks/crypto_keyfile
# Add these lines:
#!/bin/sh
cp -p /boot/rootkey.bin "${DESTDIR}"

Set the script to be executable.

chmod +x /etc/initramfs-tools/hooks/crypto_keyfile

OK, almost there. Symlink the cryptroot device into /dev. We're skipping the udev rule to automate this because so help me, we're never going to update our kernel on this machine ever again. See my other howtos if you want information on how to add a udev rule for this.

ln -sf /dev/mapper/cryptroot /dev

A dirty little secret: Devuan Jessie doesn't have a ZFS-compatible bootloader. We're going to steal the package we need from the Devuan testing branch, confusingly called "Ascii".

cp -v -p /etc/apt/sources.list /etc/apt/sources.list.orig
echo 'deb http://auto.mirror.devuan.org/merged ascii main' >> /etc/apt/sources.list
apt update
time apt install -y grub-pc/ascii

Remove the Ascii repo.

mv /etc/apt/sources.list.orig /etc/apt/sources.list

Edit your GRUB config.

vi /etc/default/grub

Make the following changes to the grub file:

-GRUB_CMDLINE_LINUX_DEFAULT="quiet"
-GRUB_CMDLINE_LINUX=""
+GRUB_CMDLINE_LINUX_DEFAULT="boot=zfs"
+GRUB_CMDLINE_LINUX="cryptdevice=UUID=UUIDHERE:cryptroot"
+GRUB_ENABLE_CRYPTODISK=y

Run through all the bootloader and ramdisk setup steps to make sure everything is up-to-date:

update-initramfs -u -k all
update-grub
grub-install /dev/sda

If grub-install completes without errors, you're ready to unmount everything and restart the system. Now is your last chance to add groups, users, configure /etc/sudoers, or do any other last-minute system prep before the system goes live. When you're ready, quit the chroot.

exit

Set the mountpoint for the /boot dataset from /boot to "legacy". It will be mounted at boot time from the line in /etc/fstab that we thought we wouldn't need.

zfs set mountpoint=legacy zdevuan/boot

Note: The /dev and /proc mountpoints are going to be locked until you kill the irqbalance process that the kernel image package added. I don't know why this hangs around, but you need to cleanly unmount the zpool in order to get a clean first-time boot. You won't lose data, but if you skip this step you'll be forced to manually import the zpool from single-user mode.

killall irqbalance
umount /mnt/sys
umount /mnt/proc
umount /mnt/dev/pts
umount /mnt/dev
zfs umount -a
zpool export -a
halt -p

If you cannot unmount any of the mounts at this point, run lsof | grep /mnt/whatever to see which process is using the mount point. Kill or killall the offending process.

Start the machine. Eject the Devuan minimal Live CD media and boot from disk. You should be set. Test it out, make your snapshots, and start using an encrypted ZFS-root, systemd-free Linux server.

External References and Resources

https://obviate.io/2008/12/04/truly-non-interactive-unattended-apt-get-install/

http://www.thecrosseroads.net/2016/02/booting-a-zfs-root-via-uefi-on-debian

https://github.com/zfsonlinux/zfs/wiki/Debian-Jessie-Root-on-ZFS

https://github.com/zfsonlinux/zfs/wiki/Debian

https://www.debian.org/releases/stable/i386/apds03.html