Installing ZFS on Devuan (Again)

We've already made a systemd-free Linux server with ZFS-on-root, so let's have some fun. Typically I like to build a system with the tools of that system: use a Mint LiveCD to install Mint, Ubuntu to install Ubuntu, and so on, but it's not strictly necessary.

In particular, you can use an Ubuntu 16.04-based LiveCD with built-in ZFS support to save yourself from having to compile ZFS kernel modules twice. At the end of the day, your machine will have Devuan on it, even if you entirely installed and configured it from inside an Ubuntu or Mint session.

Here's a remix of how to build a Devuan system using a Linux Mint 18.x LiveCD and a decent network connection. Unlike the previous howto, this one will combine the stable and unstable Devuan package repositories. The advantage of doing this is a system with a modern Linux kernel and a newer version of the ZFS modules.

This is slightly more advanced than the previous howto, but don't worry. You should be a pro at this by now, and we're not going to be doing anything too terribly different here in terms of the core concepts you've already mastered. We'll be making a new LUKS container, we'll be putting ZFS on the container, installing an OS onto ZFS, and finally configuring the bootloader to decrypt and mount it. Easy peasy.

First, fetch the Linux Mint 18.x ISO and boot your machine. I like using linuxmint-18.3-xfce-64bit.iso, but use what you like. Start a terminal and become root and install the two packages you need to continue:

sudo su
killall light-locker # no screens shall be saved
apt update
apt install -y zfsutils-linux debootstrap

Partition your disk and create a LUKS container for it. This howto assumes your disk is /dev/sda and you're putting one partition on it. Your actual mileage may vary.

CRYPTNAME=cryptroot
DEVICE=/dev/sda
PARTITIONNUMBER=1
PART=${DEVICE}${PARTITIONNUMBER}

wipefs --force --all ${DEVICE}
dd if=/dev/zero of=${DEVICE} bs=1M count=2

/sbin/parted --script --align opt ${DEVICE} mklabel msdos
/sbin/parted --script --align opt ${DEVICE} mkpart pri 1MiB 100%
/sbin/parted --script --align opt ${DEVICE} set ${PARTITIONNUMBER} boot on
/sbin/parted --script --align opt ${DEVICE} p

cryptsetup luksFormat -h sha512 ${PART}
cryptsetup luksOpen ${PART} ${CRYPTNAME}

Check that you have a /dev/mapper/cryptroot LUKS container and note its UUID value:

cryptsetup luksDump ${PART}
blkid -o export ${PART} | grep -E '^UUID='

Create your zpool.

ZPOOLNAME=zroot
ZROOTDATASETNAME=jessie
CRYPTNAME=cryptroot
VDEV=/dev/mapper/${CRYPTNAME}
TARGET=/mnt

# /sbin/modprobe zfs # skip this if ZFS is not a kernel module (Ubuntu 16.04+)
/sbin/zpool create -f \
  -R ${TARGET} \
  -O mountpoint=none \
  -O atime=off \
  -O compression=lz4 \
  -O normalization=formD \
  -o ashift=12 \
  ${ZPOOLNAME} ${VDEV}

/sbin/zfs create -o canmount=off        ${ZPOOLNAME}/root
/sbin/zfs create -o mountpoint=/        ${ZPOOLNAME}/root/${ZROOTDATASETNAME}
/sbin/zfs create -o mountpoint=/boot    ${ZPOOLNAME}/boot
/sbin/zfs create -o mountpoint=/home    ${ZPOOLNAME}/home
/sbin/zfs create -o mountpoint=/var     ${ZPOOLNAME}/var
/sbin/zfs create -o mountpoint=/var/log ${ZPOOLNAME}/var/log

/sbin/zpool set bootfs=${ZPOOLNAME}/root/${ZROOTDATASETNAME} ${ZPOOLNAME}

Install your OS. We use debootstrap here, and we leverage a couple of bonus packages we want with --include. You could really go overboard here and install the kitchen sink. I prefer to keep --include lean (but not too lean) and add what I need later.

ARCH=amd64
BRANCH=jessie
TARGET=/mnt
# HTTPS works here but not for apt
MIRROR=https://auto.mirror.devuan.org/merged
PKGS=console-setup,cryptsetup,kbd,locales,tmux,openssh-client

/usr/sbin/debootstrap \
  --arch=${ARCH} \
  --include=${PKGS} \
  ${BRANCH} \
  ${TARGET} \
  ${MIRROR}

N.B.: I have not had much luck with a reliable way to intelligently install packages with respect to their dependencies other than debootstrap and the apt family of tools. apt, apt-get, aptitude, and their ilk expect a network connection to find and fetch packages. This makes, say, downloading a set of .DEB files to a local file share and then saying "go install these before I turn your network interface on" a problem. I've done experimentation with a number of tools that are ultimately unsatisfying: multistrap and gdebi come to mind. If you only want to install Devuan once, go and make your bespoke debootstrap --include as long as you want. I like to keep the fetched packages around, un-bootstrapped in a tarball, to help me create reproducibly similar systems. You can do this with the --foreign argument, with the cost of needing to chroot to the new system-to-be and run debootstrap, locally, a second time. Doing so is outside the scope of this howto, but it can be done.

When the base system has installed successfully, put your new fstab in place. For example:

# cat /mnt/etc/fstab
/dev/mapper/cryptroot /        zfs defaults,noatime 0 0
zroot/boot            /boot    zfs defaults,noatime 0 0
zroot/home            /home    zfs defaults,noatime 0 0
zroot/var             /var     zfs defaults,noatime 0 0
zroot/var/log         /var/log zfs defaults,noatime 0 0

Other important files that need to be updated:

echo myhostname > /mnt/etc/hostname
echo en_US.UTF-8 UTF-8 > /mnt/etc/locale.gen
echo 127.0.0.1 myhostname >> /mnt/etc/hosts
ln -sf /proc/self/mounts /mnt/etc/mtab

Set your network config. This howto assumes DHCP for simplicity. Linux networking is terrible, so I chattr the interfaces file to keep it from being molested by something well-meaning but misguided that wants to make sure my minimalist server OS can connect to a coffeeshop wifi access point if one appears, because apparently there could someday be a Starbucks that opens up inside a datacenter. I do this to /etc/resolv.conf, too.

echo auto eth0 >> /mnt/etc/network/interfaces
echo iface eth0 inet dhcp >> /mnt/etc/network/interfaces
chattr +i /mnt/etc/network/interfaces

Add some mountpoints into your /mnt:

for i in /dev /dev/pts /proc /sys; do mount -B $i /mnt$i; done

Create a key for your bootloader to use to unlock the LUKS container.

TARGET=/mnt
KEYDIR=${TARGET}/boot
DEVICE=/dev/sda
PARTITIONNUMBER=1
PART=${DEVICE}${PARTITIONNUMBER}
INITRAMFSHOOKSDIR=${TARGET}/etc/initramfs-tools/hooks
KEYFILE=rootkey.bin

openssl rand -out ${KEYDIR}/${KEYFILE} 2048 # you can always use dd here too
chmod 0 ${KEYDIR}/${KEYFILE}
cryptsetup luksAddKey ${PART} ${KEYDIR}/${KEYFILE}

mkdir -p ${INITRAMFSHOOKSDIR}
echo "cp -p /boot/${KEYFILE} \"\${DESTDIR}\"" > ${INITRAMFSHOOKSDIR}/crypto_keyfile
chmod +x ${INITRAMFSHOOKSDIR}/crypto_keyfile

chroot into your system and configure it. tmux or GNU screen would be useful here, hence why I put tmux in the debootstrap. Some of the following steps can be done while you're waiting for things to compile.

chroot /mnt

Set up your apt repos. Make sure you have the Devuan unstable branch, "ceres", and it should include at least the "main" and "contrib" categories. Debian refugees may recall that their unstable branch is "sid", so there may be a period of adjustment re-learning that sid is now ceres.

cd /etc/apt
cp -p sources.list sources.list.orig
vi sources.list # do your editing here
# cat sources.list
deb http://auto.mirror.devuan.org/merged jessie main
deb http://auto.mirror.devuan.org/merged ceres main contrib

Get the Devuan repo key. It is absent from your new system because you didn't use a Devuan installation medium. You could get this key from the Devuan LiveCD in /usr/share/keyrings, but I'll show you how to do it by hand here:

gpg --verbose --keyserver=pgp.mit.edu --recv-key 94532124541922FB
gpg --verbose --export --armor --output=./devuan_jessie.key 94532124541922FB
apt-key add ./devuan_jessie.key

Now you can begin to customize your packages. At minimum, we'll be installing a compiler, a kernel, a bootloader, and some kernel modules. If your architecture isn't AMD64, adjust it accordingly.

apt update
apt install -y build-essential
apt install -y -t ceres linux-image-amd64
apt install -y -t ceres linux-headers-amd64

I find that when mixing stable and unstable repos, meaning both jessie and ceres, the newer package usually wins. If that's the case, then the -t ceres argument isn't strictly necessary. I like to include it anyway for clarity. I want the latest kernel and kernel headers the repo has to offer.

Install ZFS. This can take a while.

DEBIAN_FRONTEND=noninteractive
export DEBIAN_FRONTEND
time apt install -y -t ceres zfs-dkms

Create a password for root. Pick a time zone and a locale. I typically like to do this in another tmux window while ZFS builds.

passwd
passwd -u root
dpkg-reconfigure tzdata
locale-gen

When ZFS finishes installing, continue by adding a ZFS-aware initramfs

apt install -y -t ceres zfs-initramfs grub-pc

Configure GRUB and prep your initramfs. The UUID value you created at the beginning will be important here. As an example, this howto assumes your UUID is 9862499a-80b0-459d-9a86-5f2ddbe0464c. Replace this value with your real UUID. If your LUKS container is named something other than cryptroot, adjust that, too.

blkid -o export /dev/sda1 | grep -E '^UUID='
vi /etc/crypttab
cat /etc/crypttab
cryptroot UUID=9862499a-80b0-459d-9a86-5f2ddbe0464c /rootkey.bin luks,keyscript=/bin/cat

Make sure /etc/default/grub contains the following:

GRUB_CMDLINE_LINUX_DEFAULT="boot=zfs"
GRUB_CMDLINE_LINUX="cryptdevice=UUID=9862499a-80b0-459d-9a86-5f2ddbe0464c:cryptroot"
GRUB_ENABLE_CRYPTODISK=y

Test if GRUB can detect your ZFS dataset.

grub-probe /

If the result isn't "zfs", something is wrong. Do not continue until the problem is fixed.

Create a new initramfs. Recent updates may have precluded the need to symlink /dev/mapper/cryptroot to /dev/cryptroot; your actual mileage may vary.

ln -sf /dev/mapper/cryptroot /dev
update-initramfs -u -k all

Update GRUB and install the GRUB bootloader.

update-grub
grub-install /dev/sda

If the result is "Installation finished. No error reported." you can proceed.

Disable log compression for /var/log. Since you're already using lz4 compression on the zpool, further per-file compression is, in general, unhelpful.

for file in /etc/logrotate.d/* ; do
  if grep -Eq "(^|[^#y])compress" "$file" ; then
    sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file"
  fi
done

When the system is configured how you want it, exit the chroot.

exit

Unmount your mountpoints, change your non-root datasets' mountpoint to "legacy", and export the zpool.

for i in sys proc dev/pts dev
do
  umount /mnt/$i
done

/sbin/zfs unmount -a

for dataset in boot home var/log var
do
  /sbin/zfs set mountpoint=legacy zroot/${dataset}
done

/sbin/zpool export -a

Finally, stop the machine, eject the LiveCD, and boot off of the disk. You will be prompted to unlock the LUKS container with a password, and from there the boot process should continue without further prompting.

Login as root. Add a swap device zvol.

ZPOOLNAME=zroot
SWAPNAME=swap
DEVICENAME=/dev/zvol/${ZPOOLNAME}/${SWAPNAME}

/sbin/zfs create -V 128M -b $(getconf PAGESIZE) \
  -o compression=zle \
  -o logbias=throughput \
  -o sync=always \
  -o primarycache=metadata \
  -o secondarycache=none \
  -o com.sun:auto-snapshot=false \
${ZPOOLNAME}/${SWAPNAME}

mkswap -f ${DEVICENAME}
echo ${DEVICENAME} none swap defaults 0 0 >> /etc/fstab
swapon -av

If you ever find yourself adding another kernel to the system, linux-image-A.B.C-D-amd64 for example, make sure you add linux-headers-A.B.C-D-amd64 as well. Adding the corresponding kernel headers should, in theory, run /etc/kernel/header_postinst.d/dkms and build the necessary ZFS kernel modules for the new kernel automatically. Always be cautious when and how you update your kernel.