The Laziest Cron Job on Windows

"You know that story about how NASA spent millions of dollars developing this pen that writes in zero G? Did you ever read that? And how Russia solved the problem?"

"Yeah, they used a pencil."

"Right, a normal wooden pencil. Just seems like Phillip takes the NASA route almost every time."

— Aaron and Abe don't have a stellar comprehension of history, Primer

A while ago I wanted to run something at a certain exact time when I wasn't going to be around to watch it. And it was a Windows box.

The proper way to schedule something on Windows is with a scheduled task.

For what I wanted to do, this was overkill. I just wanted to run a program at 4:15 PM that day and be done. So I wrote this:

Param([DateTime] $At, [String] $Command)
$ts = $At - (Get-Date)
if (0 -gt $ts.Ticks) { Exit }
Write-Verbose ("sleeping until {0}" -f ($At))
Start-Sleep -Milliseconds $ts.TotalMilliseconds
try { Invoke-Expression -Command $Command }
catch { Write-Error $_ }

I saved it to a file, ".\do-at.ps1", and I ran it like so:

.\do-at.ps1 -?
do-at.ps1 [[-At] <datetime>] [[-Command] <string>] [<CommonParameters>]

.\do-at.ps1 -At "4:15 PM" -Command "calc.exe" -Verbose

When you just need a pencil, it's useful.


Encrypted ZFS on Root for Devuan: ZFS with LUKS, without Systemd

Let's say that your hobbies include putting ZFS onto different things, but you have a deep-seated loathing of amateur init systems and their authors.

Let's say this problem is compounded by the fact that every major Linux distribution has adopted same said init system and it can be particularly thorny to remove it and replace it with something, anything sane. What else can you do but curse the darkness? All is lost!

No, my friends. We can light a candle. There's always a way. Sometimes a ridiculous way, but a way nonetheless.

A number of folks who still have their heads about them and understand that some cures are worse than the disease have forked Debian and made it mo' better. The Devuan Linux distribution is still very new at this point and recently hit its 1.0 release. Its goal is "software freedom", which is code for "no systemd", so it's perfect for our goals.

The following is a quick and dirty howto for setting up a Devuan system with encrypted ZFS on root. It is meant to be read and understood in the context of my other ZFS-on-root howtos and is not intended for a beginner ZFS (or Linux) enthusiast. We're taking a big step backwards here and doing some old-school compiling of DKMS modules.

I say "taking a big step backwards" because Canonical, the maker of Ubuntu Linux, took a big step forward by integrating the ZFS kernel module into their kernels from 16.04 onwards. But they also adopted systemd, which is Muy Malo, so we have to roll our own zfs.ko. Twice. But I am getting ahead of myself.

The following howto uses the Devuan Jessie 1.0.0 minimal Live CD ISO (devuan_jessie_1.0.0_amd64_minimal-live.iso), which lacks any desktop environments. We don't need them, we're just going to be typing our way to joy. Moreover, kernel module builds fail on the Devuan desktop live environment, so use the minimal live ISO.

Make sure your system has a drive you don't care about formatting, and at least 4GB of RAM. Attempting this with anything less will result in spurious errors that will impair your progress. ZFS itself enjoys a large memory footprint, so I don't suggest this for any resource-constrained, embedded setups. Proceed at your own risk. These steps come with no warranty whatsoever. Good luck.

I'm assuming that the root drive you're going to format is /dev/sda. Your actual mileage may vary.

Wipe and partition the disk.

wipefs --force --all /dev/sda
# or
dd if=/dev/zero of=/dev/sda bs=1M count=2
/sbin/parted --script --align optimal /dev/sda mklabel msdos
/sbin/parted --script --align optimal /dev/sda mkpart primary 1MiB 100%
/sbin/parted --script --align optimal /dev/sda set 1 boot on

Check that the partition table is correct:

/sbin/parted --script /dev/sda p

You'll notice that we're breaking with tradition of creating a separate partition for /boot and / partitions like we've previously done. Like I said, quick and dirty. I was surprised to see that this works, even under Debian/Devuan. At the risk of potential boot errors, this lets you take ZFS snapshots of your kernel and initrds, which may be really useful to you after a disaster.

Log into the Devuan Live CD. Be sure to understand your network configuration. Linux networking is a Kafkaesque nightmare of conflicting system config file advice and is beyond the scope of this howto. For simplicity, in this howto we assume your machine uses DHCP to get online. Use a real setup for a real server.

dhclient eth0

Add the jessie-backports packages to your apt/apt-get sources.

echo 'deb http://auto.mirror.devuan.org/merged jessie-backports main contrib' > /etc/apt/sources.list.d/backports.list

Update the package sources and install cryptsetup.

time apt update
time apt install -y cryptsetup

Setup your partition into a LUKS container.

cryptsetup luksFormat -h sha512 /dev/sda1
cryptsetup luksOpen /dev/sda1 cryptroot

ZFS on Devuan requires compiling multiple kernel modules and is painful. Canonical took this pain away and yet forces systemd upon us. Curses. Curses I say.

Start by adding the headers for the Live CD kernel.

time apt install -y linux-headers-$(uname -r)

Install the ZFS DKMS package and dependencies. This will pop up a license incompatibility warning that you have to acknowledge. Optionally, you can force apt-get to skip this warning by setting an environment variable. To skip the interactive ZFS license warning screen:


Build ZFS.

time apt-get install -y -t jessie-backports zfs-dkms

A quick word about apt versus apt-get. Both work equally well for our purposes. Use whichever you prefer, but be aware that you will need to specify certain exact packages from certain exact repositories. "apt-get install -t reponame package" is the same to us as "apt install package/reponame". I usually choose the command that contains the fewest number of characters.

Keep an eye on the output of this command and watch for any errors building the modules. You'll be prompted to check /var/lib/dkms/something/build/make.log to troubleshoot. Common errors include transient resource constraint problems: running out of memory, running out of disk space on the Live CD file system, et cetera. If you encounter an error, stop. Fix your system. Do not proceed until your setup is entirely correct.

When the zfs-dkms package is installed, you can add the ZFS kernel module to your Live CD setup.

/sbin/modprobe zfs

If you get any errors, stop here and correct them.

Setup your zpool and datasets. This will look familiar if you've read my other ZFS howtos.

zpool create -f -O mountpoint=none -O compression=lz4 -O atime=off -o ashift=12 zdevuan /dev/mapper/cryptroot
zfs create -o mountpoint=/ zdevuan/root
zpool set bootfs=zdevuan/root zdevuan

Optionally, you can create separate datasets for /boot, /home, and so forth. I like to create an unmounted "zpool/root" dataset and then children datasets for my various boot environments (example: "zpool/root/cur", "zpool/root/next", and so on). Theoretically, this helps you rollback bad upgrades without too much swearing. To add a /boot dataset for example:

zfs create -o mountpoint=/boot zdevuan/boot

Create your desired datasets. Unmount the zpool and remount it under /mnt.

zpool export -a
zpool import -R /mnt zdevuan

Since I never bothered looking for where Devuan install discs keep their packages or base image (I think they're under /lib/live. That's what mount suggests.), we're going to set up our new system with debootstrap:

time /usr/sbin/debootstrap jessie /mnt https://auto.mirror.devuan.org/merged

Whatever you do, don't use the --no-check-gpg flag to make debootstrap run faster.

When debootstrap finishes setting up the base system under /mnt, add a few files from your Live CD to the new system:

cp -v -p /etc/apt/sources.list.d/backports.lists /mnt/etc/apt/sources.list.d
cp -v -p /etc/locale.gen /mnt/etc

Before the new system solidifies, it throws errors about missing locales and things of that nature, so I like to use the Live CD environment to edit the new system's config files. Edit /mnt/etc/fstab to the following:

/dev/mapper/cryptroot /     zfs defaults,noatime 0 0
zdevuan/boot          /boot zfs defaults,noatime 0 0

The line that will mount /boot is superfluous if you don't change the zdevuan/boot dataset's mountpoint value to "legacy", but don't skip ahead just yet.

Edit /mnt/etc/crypttab to add the UUID of the cryptroot LUKS container. There are multiple ways to get the UUID. You can run ls -l /dev/disk/by-uuid or you can run blkid. You have options.

blkid /dev/sda1 # get the UUID for /dev/sda1
echo 'cryptroot UUID=UUIDHERE /rootkey.bin luks,keyscript=/bin/cat' > /mnt/etc/crypttab

Create a key for the cryptroot LUKS container. I've written about this previously and people have Strong Opinions about PRNGs. Use whatever makes you feel better. If your entropy pool is taking its own sweet time, you can run this in a tmux or GNU screen window and switch to another window to keep working while dd keeps crunching.

time dd if=/dev/random iflag=fullblock of=/mnt/boot/rootkey.bin bs=512 count=4

You can also use OpenSSL, if you trust OpenSSL. In my testing, OpenSSL can sometimes yield slightly, slightly less probabilistically-indistinguishable-from-noise pseudorandom output than /dev/random. It's very close and significantly faster.

openssl rand -out /mnt/boot/rootkey.bin 2048

Set the hostname.

echo myhostname > /mnt/etc/hostname
echo ' myhostname' >> /mnt/etc/hosts

Set the networking configuration for your new machine. This can be very complicated, so for this howto we continue to assume we're just going to use DHCP.

vi /mnt/etc/network/interfaces

Add these lines:

auto eth0
iface eth0 inet dhcp

Mount system directories.

cd /
for i in /dev /dev/pts /proc /sys; do mount -B $i /mnt$i; done

Chroot to the new system. Yes, we're going to re-download and install ZFS again.

chroot /mnt /bin/bash --login

Do some system setup.

passwd -u root
dpkg-reconfigure tzdata
ln -sf /proc/self/mounts /etc/mtab

Add some packages that debootstrap skips. You can also use --include in the debootstrap, but I don't care to make that process any more complicated than it already is. This howto assumes you're running this on an amd64 system. Adjust the "linux-image-<arch>" package accordingly for whatever your intended arch is.

apt update
apt install -y locales kbd # select your locale here
time apt install -y cryptsetup linux-image-amd64
time apt install -y linux-headers-$(uname -r) # Again
time apt-get install -y -t jessie-backports zfs-dkms zfs-initramfs

Hopefully by now the LUKS key has finished getting written. Create a crypto keyfile hook for the initramfs and add the key to the LUKS container.

cryptsetup luksAddKey /dev/sda1 /boot/rootkey.bin
chmod 0 /boot/rootkey.bin
vi /etc/initramfs-tools/hooks/crypto_keyfile
# Add these lines:
cp -p /boot/rootkey.bin "${DESTDIR}"

Set the script to be executable.

chmod +x /etc/initramfs-tools/hooks/crypto_keyfile

OK, almost there. Symlink the cryptroot device into /dev. We're skipping the udev rule to automate this because so help me, we're never going to update our kernel on this machine ever again. See my other howtos if you want information on how to add a udev rule for this.

ln -sf /dev/mapper/cryptroot /dev

A dirty little secret: Devuan Jessie doesn't have a ZFS-compatible bootloader. We're going to steal the package we need from the Devuan testing branch, confusingly called "Ascii".

cp -v -p /etc/apt/sources.list /etc/apt/sources.list.orig
echo 'deb http://auto.mirror.devuan.org/merged ascii main' >> /etc/apt/sources.list
apt update
time apt install -y grub-pc/ascii

Remove the Ascii repo.

mv /etc/apt/sources.list.orig /etc/apt/sources.list

Edit your GRUB config.

vi /etc/default/grub

Make the following changes to the grub file:


Run through all the bootloader and ramdisk setup steps to make sure everything is up-to-date:

update-initramfs -u -k all
grub-install /dev/sda

If grub-install completes without errors, you're ready to unmount everything and restart the system. Now is your last chance to add groups, users, configure /etc/sudoers, or do any other last-minute system prep before the system goes live. When you're ready, quit the chroot.


Set the mountpoint for the /boot dataset from /boot to "legacy". It will be mounted at boot time from the line in /etc/fstab that we thought we wouldn't need.

zfs set mountpoint=legacy zdevuan/boot

Note: The /dev and /proc mountpoints are going to be locked until you kill the irqbalance process that the kernel image package added. I don't know why this hangs around, but you need to cleanly unmount the zpool in order to get a clean first-time boot. You won't lose data, but if you skip this step you'll be forced to manually import the zpool from single-user mode.

killall irqbalance
umount /mnt/sys
umount /mnt/proc
umount /mnt/dev/pts
umount /mnt/dev
zfs umount -a
zpool export -a
halt -p

If you cannot unmount any of the mounts at this point, run lsof | grep /mnt/whatever to see which process is using the mount point. Kill or killall the offending process.

Start the machine. Eject the Devuan minimal Live CD media and boot from disk. You should be set. Test it out, make your snapshots, and start using an encrypted ZFS-root, systemd-free Linux server.

External References and Resources







Why I Don't Run FreeBSD on My Laptop


You may or may not have noticed I've been writing more about ZFS on Linux lately. This might seem weird coming from a person who is admittedly a big proponent of BSD, primarily OpenBSD, and someone with a strong disdain for systemd and the entire reasoning behind that whole... thing.

First, some background. I started playing with FreeBSD back in the 3.2 days, I don't care to remember how long ago that was. I remember the install CD didn't include a DHCP client on it so I needed to get a lease using a Windows install, then turn it off, swap the hard disk, and manually set the ifconfig to get the FreeBSD install on the network long enough to download dhclient.

On rare occasions I still trawl web.archive.org looking for the long-lost CNet article from 20 or so years ago outlining five alternate operating systems: I vaguely remember them including QNX, the BeOS, Amiga, and FreeBSD. I forget the fifth. It might have been Linux. FreeBSD was described as "bulletproof", and that intrigued me. FreeBSD seemed very similar to Linux, with an older pedigree and a far stricter adherence to kernel-userland cooperation. I liked the idea of a central governing body. It gives you one place to point your finger when something goes sideways. I was sold.

Eventually I settled on OpenBSD for my network OS of choice and that looks to have been a decision I made around version 2.4 or so, long before PF or LibreSSL were devised. I ran -CURRENT for a while, I kept local mirrors of the i386 and amd64 install sets, I labored over mkisofs command line arguments back when the release ISO image was copyrighted, before Theo blessed the existence of official installXX.iso files.

When web series BSD Now asked for tutorials, I decided to finally sit down and learn a thing or two about DragonFly BSD's very own HAMMER file system and write up a crash course for it.

What I'm trying to say is that I've been a dyed-in-the-wool BSD fan for a while. I'm no dev and my proposed patches were laughably bad, so anymore I just keep quiet and let the BSDs do their thing. I don't hate the BSDs and this critique is not intended to offend, but it is nonetheless a criticism of the state of BSD's real-world usefulness.


I recently had an experience with some very insidious malware that infected my network. It's called "Windows 10 Creators Update" and it took 25% of my machine pool offline when it forced a reboot and waited for a non-existent sitting-at-the-keyboard user to go through the "out-of-box experience" setup process again. Dozens of my machines stopped doing work because they were infected by a big screen that, instead of saying "your files have been encrypted", simply said "Welcome to Windows 10!" and had a "Next" button it wanted me to click.

So I've been looking for a reliable replacement OS for a while now. In 2015 I experimented with Antergos, an Arch Linux derivative, and it was nice, but it was prompting me to install updates twice a day. So I switched to Linux Mint and It Was Good. But Mint derives from Ubuntu, which derives from Debian, which adopted systemd, and that makes me sad.

There was a recent article by Peter N. M. Hansteen titled OpenBSD and the modern laptop that got me thinking about putting BSD on one of my laptops. I have a hard requirement for ZFS support... By Any Means Necessary. That summarily eliminates OpenBSD, because by its own admission OpenBSD sucks. Maybe FreeBSD? Maybe not.

The Failure of FreeBSD

When I first began testing non-Windows laptop setups, my first choice was the now-defunct PC-BSD. Unfortunately I cannot link to the bug since the project has ended, but there was an issue with the bsdinstall scripts that prevented easy dual-booting between multiple partitions. I tried to fix this myself, but I'm no dev and my patches were laughably bad. The PC-BSD experiment did not last long.

Eventually systemd started creeping into my laptop, so I renewed my interest in a BSD substitution. PC-BSD is now TrueOS and uses FreeBSD's -CURRENT branch as its starting point. I no longer require dual-booting on my laptop, so PC-BSD/TrueOS's dual booting problem shouldn't be a blocker anymore. So far, so good! It doesn't boot in a Hyper-V VM. The installer just hangs at a partially-drawn portion of the boot menu. So testing out a change in a VM before applying it to real-world hardware isn't possible for me. Turns out I also have a hard requirement that the OS must support cursory Hyper-V virtualization so I can potentially break something important in an expendable VM before I potentially break something important on my laptop. So TrueOS was out the window.

I finally looked at FreeBSD 11.0-RELEASE or, if you prefer, "TrueOS with the training wheels off". It runs under Hyper-V very well, and I've been using it both on local virtualized instances and in Azure since around 10.2 or so. It can run in the cloud, but it can't run on my ThinkPad.

What? Yeah. I use an old Lenovo ThinkPad for my testing, a T410 to be specific. This isn't some obscure workalike architecture that was only produced in Sweden for six months and was exclusively used by electrical engineering automation microcontrollers or something. It's just a goddamn ThinkPad. I put an SSD drive in it last year and it keeps chugging along like a trooper. Too bad the FreeBSD kernel hangs on startup.

It doesn't always hang. It's about 50/50 whether or not the installer ISO is going to get to the install wizard or hang during the dmesg part. I can sometimes get it to boot fully by toggling some boottime options, but I don't have a consistent workaround. Sounds like a hardware problem, right? Right. But Windows still works on this laptop like a champ. After getting the OS installed, I had hoped the startup problem was limited to the installer, but no, the finished client wouldn't boot after a restart, either. The first startup went fine, the second one hung.

That would end the dreams of a FreeBSD desktop experience for anyone. Anyone but me! I persisted. I fetched 11.1-RELEASE the day it came out hoping that maybe some minor kernel revision would spare me, but no, no luck there. 11.1 behaves just as 11.0 did.

Dejected, I began fussing with Linux again, and trying to choose a non-systemd Linux is an increasingly unpleasant experience. So I went back to FreeBSD and did some testing with 11.1-RELEASE on a Hyper-V VM. And a FreeBSD desktop experience there is still painful.

C. O. D. (Crash on Delivery)

FreeBSD has pretty good documentation. This is undisputed. So when I set up my FreeBSD test VM, I went to The FreeBSD Handbook and started reading. To be clear, I wanted a FreeBSD 11.x/amd64 machine running XFCE and a GUI login was an optional nice-to-have. So I ran pkg install xfce like the handbook says and then ran /usr/local/bin/startxfce4.

Turns out I hadn't installed the xorg package first. The XFCE package doesn't require it (Really? Really?). It doesn't get installed as a pre-req, so to be thorough I wiped everything, reinstalled, and did that first. Then I re-added xfce and tried again. And X11 started... kinda. And then it crashed.

So I read. I read tutorial after tutorial after tutorial. I even defected to MATE tutorials on old FreeBSD versions, such was my need for a FreeBSD desktop.

I'd reinstalled FreeBSD a half a dozen times up to this point, and I was getting very good at hard-resetting the VM, logging in, rolling back to an initial ZFS snapshot, and rebooting again to save a little bit of time. And I was determined.

There were some randomly-recommended manual tweaks of dubious value that forums and howtos mentioned, but I didn't really comprehend what those changes would do or if they would help. I was cautious but willing to experiment. This is why being able to screw around inside a VM is so important to me.

Eventually after a couple days of fussing with it, I found the correct commands to get a desktop environment started on a FreeBSD Hyper-V VM:

pkg install xorg
pkg install xfce
echo "exec /usr/local/bin/startxfce4 --with-ck-launch" > ~/.xinitrc
# add hald_enable="YES" to /etc/rc.conf

Now there is joy, right? No, no there is not. The mouse doesn't work. I Alt-keyed and arrow keyed around a little bit and still couldn't get to a quit or a shutdown button. Or start an xterm window so I could maybe type halt -p or something. I hard-reset the VM.

I added moused_enable="YES" to /etc/rc.conf and rebooted again to see if that would help. It did not. At that point I was tired and out of ideas and howtos. FreeBSD won't reliably boot on my ThinkPad and, even if it one day does, I can't get a FreeBSD desktop working in a VM because the VM ignores mouse input.

So I went back to fighting with Linux. I wrote an encrypted-root-ZFS-on-Devuan tutorial that I plan to post soon. Stay tuned.


Installing Linux Mint 18/Ubuntu 16.04: Encrypted ZFS Root and /boot Partitions

A while ago I cobbled together a spate of howtos into a singular method for installing a root-on-ZFS Ubuntu/Mint system with LUKS. Since then, I've kept reading on the subject and I found a solution for a long-standing problem with Linux/GRUB OSes. I lamented that there wasn't a good way to boot your machine using full-disk encryption, so even if all of your /home and system data is secure when the system is powered off, you still had this pesky /boot partition hanging around out in the open. There's a big Achilles' heel in your nice, secure disk encryption setup if your bootloader and kernel are just sitting ducks.

Turns out, there's a way you can really get full disk encryption with LUKS and GRUB. With a few changes to the previous instructions, we can get a system that runs ZFS on root in a LUKS-encrypted container and get an LUKS-encrypted /boot partition as well.

For stability purposes, the /boot partition will remain ext2/3-formatted. There was a recent mailing list thread crowing about ext2's "worse is better" design philosophy that I found quite entertaining.

The following is a terse set of instructions that skip a lot of explanation. Refer to the previous instructions for further details where desired. All the usual caveats still apply: do not perform these steps on a disk that contains data you care about. Be comfortable with the concepts of using disk partitioning tools and ZFS. Your actual mileage may vary. Not applicable where void by law. Safety not guaranteed. Use at your own risk. This means you.

Boot the Linux Mint 18+ (or Ubuntu 16.04+) ISO. Open a terminal and become root.

sudo su
killall xscreensaver

Identify your storage device. This is usually /dev/sda, and we use that as an example in this howto. Wipe the partition table off of this device with dd or, if you prefer, wipefs.

wipefs --force --all ${ROOTDISK}
# or:
# dd if=/dev/zero of=${ROOTDISK} bs=1M count=2

Set up a new partition table on the disk. Create one partition for your encrypted /boot container and one for your encrypted ZFS pool.

/sbin/parted --script ${ROOTDISK} mklabel msdos
/sbin/parted --script --align optimal ${ROOTDISK} mkpart primary  1MiB 513MB  # encrypted /boot
/sbin/parted --script --align optimal ${ROOTDISK} mkpart primary 513MB  100%  # cryptroot
/sbin/parted --script ${ROOTDISK} set 1 boot on

Our steps now begin to differ from the original howto. Instead of one LUKS container, we create and mount two. I first started experimenting with containers named "cryptroot" and "cryptboot" and you can imagine how quickly that got confusing. Among other things, /boot holds the kernel so I call the LUKS container that will hold /boot "cryptkern" here to reduce confusion.

cryptsetup luksFormat -h sha512 ${ROOTDISK}1
cryptsetup luksFormat -h sha512 ${ROOTDISK}2
cryptsetup luksOpen  ${ROOTDISK}1 cryptkern
cryptsetup luksOpen  ${ROOTDISK}2 cryptroot

Install the ZFS utilities on the live CD session. You can also fetch these .DEB files and store them locally, but that is not covered here.

apt update
apt install -y zfsutils-linux
zpool create -O mountpoint=none -O compression=lz4 -O atime=off -o ashift=12 zmint /dev/mapper/cryptroot
zfs   create                       zmint/root
zfs   create -o mountpoint=/       zmint/root/mint18
zpool set bootfs=zmint/root/mint18 zmint

Additional ZFS datasets can be created at this point, but this howto skips them for simplicity.

Export the zpool and reimport it under /mnt.

zpool export -a
zpool import -R /mnt zmint

Format and mount the /boot partition. Note that the mkfs.ext3 command is formatting the unlocked LUKS container device and not ${ROOTDISK}1, which is the LUKS container itself.

mkfs.ext3 /dev/mapper/cryptkern
mkdir /mnt/boot
mount -o noatime /dev/mapper/cryptkern /mnt/boot

Install the OS to /mnt. I use unsquashfs.

apt install -y squashfs-tools
time unsquashfs -f -d /mnt/ /media/cdrom/casper/filesystem.squashfs
cp -v -p /run/resolvconf/resolv.conf /mnt/run/resolvconf/resolv.conf
cp -v -p /media/cdrom/casper/vmlinuz /mnt/boot/vmlinuz-`uname -r`

A totally random aside:

There's a lot of banter online about the merits and deficiencies of using /dev/random versus /dev/urandom and which OSes have greater or fewer differences between the two. Even professionals like to bicker about it. Ultimately, I suggest we all start using /dev/arandom. He who fears PRNG-deciphering nation states and loves wearing a tinfoil hat can make his own decisions. Where you value speed, I find haveged to be great at dramatically improving PRNG performance if you trust your machine's rand()-making device. It's less useful here than when creating enormously-sized PGP keys, but it's handy to have around.

# Optional
apt install -y haveged

Once you trust your randomness, create a LUKS decryption key file for each container. This prevents us from needing to type decryption passwords for each container every time we boot.

time dd bs=512 count=4 if=/dev/random iflag=fullblock of=/mnt/boot/rootkey.bin
time dd bs=512 count=4 if=/dev/random iflag=fullblock of=/mnt/root/kernelkey.bin

cryptsetup luksAddKey ${ROOTDISK}1 /mnt/root/kernelkey.bin
cryptsetup luksAddKey ${ROOTDISK}2 /mnt/boot/rootkey.bin

chmod 0 /mnt/root/kernelkey.bin
chmod 0 /mnt/boot/rootkey.bin

Mount important system directories into /mnt and chroot to the new system:

cd /
for i in /dev /dev/pts /proc /sys; do mount -B $i /mnt$i; done
chroot /mnt /bin/bash --login

Write your fstab:

/dev/mapper/cryptkern /boot ext3 defaults,noatime 0 2
/dev/mapper/cryptroot /      zfs defaults         0 0

Configure the system. This section is abbreviated for simplicity. See the original instructions for details.

passwd -u root
ln -s /proc/self/mounts /etc/mtab
echo myhostname > /etc/hostname
echo myhostname >> /etc/hosts
dpkg-reconfigure tzdata
# add a user account, etc...

Get the list of UUIDs for your system's partitions. You will use these UUIDs for decrypting the LUKS container partitions and in the bootloader.

ls -l /dev/disk/by-uuid

Write your crypttab. If your cryptkern partition is /dev/sda1, use the sda1 UUID for that line in crypttab and so on. crypttab is evaluated from top to bottom, so ordering matters here: I put the cryptroot line at the top and the cryptkern line beneath it, since the key for cryptkern is kept in the cryptroot container.

vi /etc/crypttab
# Add these lines:
cryptroot UUID=UUIDHERE /rootkey.bin        luks,keyscript=/bin/cat
cryptkern UUID=UUIDHERE /root/kernelkey.bin luks,discard

Write a hook script for initramfs to get a copy of the key to decrypt cryptroot.

vi /etc/initramfs-tools/hooks/crypto_keyfile
# Add these lines:
cp -p /boot/rootkey.bin "${DESTDIR}"

Make the hook script executable.

chmod +x /etc/initramfs-tools/hooks/crypto_keyfile

Install ZFS and a ZFS-aware initramfs on the installed system.

apt update # again
apt install -y zfsutils-linux
apt install -y zfs-initramfs
cp -v -p /lib/udev/rules.d/60-zpool.rules /etc/udev/rules.d/

Edit /etc/default/grub. This is where you'll point to the encrypted root partition.

vi /etc/default/grub
# Make the following changes

Note that you're NOT putting the cryptkern UUID and LUKS container here. Your boot sequence will be:

  1. Power on
  2. Get password for cryptkern
  3. Load initramfs
  4. initramfs loads key for cryptroot from /boot
  5. Mount /
  6. System loads key to mount cryptkern from /root

You could use passwords for all of your LUKS containers if you really wanted to do so, but keys are quicker and relatively safe if you can wrap you head around this ping ponging of device decryption.

Symlink your cryptroot device under /dev so updating the bootloader won't throw an error.

ln -sf /dev/mapper/cryptroot /dev
echo 'ENV{DM_NAME}=="cryptroot", SYMLINK+="cryptroot"' > /etc/udev/rules.d/99-cryptroot.rules

Update your initramfs and bootloader.

update-initramfs -c -k all
grub-install /dev/sda

Quit the chroot, ummount everything, and reboot.

umount /mnt/dev/pts
umount /mnt/dev
umount /mnt/proc
umount /mnt/sys
umount /mnt/boot
zfs umount -a
zpool export -a

If all goes according to plan, after POST you'll get a screen like this:

Attempting to decrypt master key...
Enter passphrase for hd0.msdos1 (a9e29f6295bc49919d5ed7820f941974):

That's hd0 (your disk), msdos1 (your MBR partition table), and as an example, a9e29f6295bc49919d5ed7820f941974, which corresponds to the UUID of your cryptkern LUKS container as per /dev/disk/by-uuid. If you type your password correctly, you'll see this:

Slot 0 opened

And your boot sequence will proceed as intended. When necessary to decrypt each LUKS container, the boot sequence will use to the keys you specified in /etc/crypttab and not prompt for manual input.

External References and Resources




A "review" of Michael Warren Lucas - Immortal Clay

Michael W. Lucas writes very good technical books. They hit that rare sweet spot between dry, abstract classroom theory and rote copy-and-paste that gives you practical examples as well as enough background on the subject to inform you as to why you would want to do it his way. His guides are invaluable, especially with complex subjects like ZFS, which is no mere file system but rather an amorphous stack of cooperating storage technologies and principals. It's a godsend to have a concise explanation handed to you that foregoes much of the academics and gives you usable, real-world guidance.

What I'm trying to say that Michael W. Lucas can write tech books. But what about fiction?

Lucas writes his fiction under the name Michael Warren Lucas, presumably to separate it from his fact. He ran a sale on some of his titles earlier this year and encouraged me to try his ostensibly horror-ish title Immortal Clay. So I did.

Admittedly, I approached this story with a fair amount of trepidation. A good writer cannot necessarily write everything equally well and I've suffered a number of authors who felt brave enough to try to genre hop before they were ready. Nonetheless, a low e-book price was encouraging and if I didn't like the book, I wouldn't be out too much. Immortal Clay describes itself as what would happen if The Thing from John Carpenter's The Thing had won. This was a good sign.

You know The Thing. Researchers in Antarctica find a dog that is really a shapeshifting alien that eats living organisms and creates perfect clones of them. Kurt Russell fights it with a flamethrower. But that's where The Thing's story ends, and that is more or less where Immortal Clay picks up.

I was ambivalent about this because it's very easy to conjure up a "what if" concept and very difficult to deliver a complex story based on it. There is a wonderful short story about the events of The Thing told from The Thing's perspective. It would be hard to top that.

Fortunately, any concerns I may have had about Lucas's ability to deliver on the promise of his premise were allayed by the end of the first chapter. This is a book that starts with the end of the world and just keeps going from there. When a space alien that perfectly clones life forms finally conquers the planet, you wouldn't think there would be much more of a story to tell. You'd be wrong.

Our protagonist dies in the prologue. Or rather, the alien duplicate that is our protagonist is copied from a person who dies in the prologue. It possesses all of his physical characteristics and behaviors, and retains all of his memories. He was a police detective before he was eaten and the alien copy keeps his inherent desire for justice, even if he now lives in a bizarre realm where there really is no longer any specific system of law anymore. How do you solve crimes, even murders, when you aren't completely sure if alien-copied things can be killed?

Immortal Clay is not pure horror, per se. It is a post-apocalyptic suburban mystery novel. It certainly draws inspiration from The Thing, and James Gunn's Slither, but it has as much in common with Jean-Paul Sartre and Tim Burton as it does to Raymond Chandler. Our hero is just as confused as Alec Baldwin and Geena Davis are at the start of Beetlejuice, stuck in their home and unsure of their fates until they find their guidebook for the afterlife. You remember it: it reads like stereo instructions. Our hero doesn't get the benefit of a book or a grizzled social worker to spell it out for him, so he asks himself some compelling philosophical questions without always getting answers. Turns out if an alien eats your homeworld and spits out perfect copies of everything, those copies will have a lot of psychological problems and, oh by the way, all the old rules of what constitutes "alive" and "dead" are right out the window.

Immortal Clay is a fun romp through a community of quote-unquote "survivors" with quirky personalities blended with real survival problems trying to make the best they can after the most absurd of unnatural disasters has ruined their planet. They are left to pick up the pieces and try to put them back together, even if there's no instruction manual. It has some genuinely horrific imagery and some genuinely emotionally harrowing moments, especially around our not-really "survivors" having very real survivors' guilt.

The mystery portions of the story are well-paced and Lucas avoids the contrivance of an oh-I'm-so-clever Agatha Christie denouement. It's a good old-fashioned whodunnit, with the added complications of "mass extinction" and "civilization was eaten by a space monster" thrown in to keep things interesting. I found myself unable to put this story down once I'd picked it up. Lucas's fiction style leans towards short, brutally plot-propelling chapters that break the action up into even, fitting scenes, and he clearly pays very strict deference to his outline. Even at 51 chapters, it's a quick read. The plot never lags and there is no unnecessarily flowery "let me prove to you I have a thesaurus" prose. Our protagonist never spends, for no discernible reason, scores of pages describing in agonizing detail how he eats a bowl of cereal. He's got a crime to solve, dammit, and he's going to solve it or die trying.

If he can die, that is. He isn't certain he can be killed.

But someone is trying to figure it out for him for sure. Or rather, some Thing.