A "review" of Deadpool

A Juice Newton opening credits sequence. 10/10



SPACEPLAN has just been released for Steam. Billing itself as an "experimental piece of interaction based partly on a total misunderstanding of Stephen Hawking's A Brief History of Time," SPACEPLAN is a delightful little game that began as a browser-based story at http://jhollands.co.uk/spaceplan/ and is well worth the investment of a few million clicks.

I stumbled across SPACEPLAN some time last year and immediately found its story entertaining, clever, and compelling. It's impossible to go into detail of the events of SPACEPLAN without ruining some aspect of the plot, so suffice it to say that it is a discovery-driven game that uses mouse clicks as its primary means of game currency. You end up clicking in SPACEPLAN. You click a lot.

Or, conversely, you don't click very much. I counted and you can, technically, complete the game using (32 + 6 + 10) mouse clicks (that's energy-generating clicks, not energy-spending clicks, of which there are at minimum 11). So it's possible to complete the game with only a few dozen clicks if you're patient. Veeeeeeeeery patient. The appeal of SPACEPLAN is that you need to go and accomplish things, and to do that you need energy, and you accrue energy by either (a) clicking the mouse, or (b) waiting for your energy accumulator to accumulate. It's up to you how much you want to click beyond the minimum 15 initial clicks, compelled only by your own sense of curiosity for what's going to happen next. You can do a purely click-fueled run, or you can do a 25-click run, or anything in between. It's up to you.

From there it's left up to the player to balance patience against progress: if you want to spend your energy to enhance your exploration, therefore advancing the story, or spend it on improving your energy collection technology, delaying immediate story progression in order to be able to accelerate the telling of that same story later. The entire game is a representation of the procrastinator's dilemma: do you begin walking to your destination along a dirt road today or do you wait for a six-lane express motorway to get built tomorrow?

SPACEPLAN is not the kind of game that would be fun to stream online. There will never be SPACEPLAN tournaments. You set up your energy collection process, whether that be manual or automatic, and then you wait. You wait until you have enough energy to buy the next thing you need to unlock the next story element. SPACEPLAN is more interactive than Progress Quest but, depending on how you play it, it doesn't have to be much more interactive.

If you intend to replay the game for analytical purposes, you'll want to invest in an autoclick utility. These are widely available for free online because simulating a mouse click is a common Windows UI programming exercise, and clicking something over and over again is just monotonous enough to warrant creating a simple automation utility.

So, thusly armed with an autoclick program that looks like My First Visual Basic GUI and works like a champ, I set out measuring how expensive the inventory of SPACEPLAN is, and only after completing the tally did it occur to me that I could not share it without spoiling the game for a new player.

In short: SPACEPLAN leans heavily on the utility of potatoes. Like, a lot. Potatoes are so useful in the universe of SPACEPLAN that it would make Mark Watney jealous. And so you click ever onward, unto a tuber-based journey that reaches into the very deepest of our cosmological questions about the origin and nature of time and space. And it misrepresents the principles and concepts in A Brief History of Time, phenomenally. Click. Explore. Repeat.


Installing Linux Mint 18/Ubuntu 16.04 with Encrypted ZFS Root

I recently discovered how easy it is to set up Linux Mint 18 onto a machine with a ZFS root partition in a LUKS-encrypted volume. I started reading about it late one night, slept on it, did some more reading the next day, and had a working proof of concept before bed that night.

(I presume the following setup is possible under earlier versions of Mint, but you'd need to add the zfs-native PPA and its package, plus its SPL dependencies, and compile some kernel modules. Remember to properly curate your linux kernel headers. In the Live CD. And then again on the installed system. Good luck. Not all kernels are created equal.)

Mint 18 is based on Ubuntu 16.04 and thus contains built-in ZFS kernel support. No more DKMS shenanigans whenever you attempt a kernel upgrade. Hallelujah!

These steps were assembled from a number of other online tutorials. I've listed them at the bottom for reference and further reading. These tutorials were absolutely invaluable to understanding this process and this howto is only a lightly edited compilation of them. It assumes that you're installing Linux Mint 18.1 XFCE from the ISO. Your actual mileage may vary.

linuxmint-18.1-xfce-64bit.iso has the following checksums:

MD5 = 703e14fef35528143063bf2f9f0f5075

SHA1 = 0bc84cf02f3a89db9ad661ac59ed10ec3b379b0b

SHA256 = 7dc167bd5e6e28641c8aaaf406bba46b1be2d472f85d1d86280ed80e19bb8462

SHA512 = 3d4ac84efb00a3c852b7f3a24e4e6c560772b84e2967f339c654d547f78bd0705734bfdc16a3ef0e770d36933b4cf049e349ac1acdcc513ba9d3142d990aeb30


This tutorial requires:

  1. A drive that contains no information you value. Its contents will be entirely overwritten.
  2. A 64-bit CPU and a Ubuntu 16.04-based OS.
  3. A lot of RAM. 4GB is really pushing it. More is better.
  4. A working Internet connection. Ubuntu 16.04 includes ZFS kernel support out of the box, but not the userland tools to use it. Sigh... it's a legal thing.

I've structured this howto thematically, so each step can more or less stand on its own. This is why I don't install everything that will be needed at one time. Doing it this way hopefully adds clarity and improves comprehension of the howto, even if it may not be remarkably efficient. Feel free to adjust the sequence to suit your own needs.

Installing Linux Mint 18 with ZFS root on LUKS

Boot the install ISO. Open a Terminal window and become root. Yes, you will be installing additional software into the Linux Live CD environment, so update your apt sources and make sure your network connection works:

sudo su
# stopping the screensaver is not required, but I just don't want it running:
killall xscreensaver
apt update

Determine your root disk. This tutorial will assume it's "/dev/sda" and most commands will use this convention. The smarter thing to do is use the disk's UUID and for settings that persist after the install we will do so. The quick and dirty way is to make a simple partition table and remember it.

# Find your root disk. In this case, we'll use /dev/sda:

When we want to refer to /dev/sda now, we can use ${ROOTDISK}. To be clear, when we specify /dev/sda1 or /dev/sda2, we can call it ${ROOTDISK}1 or ${ROOTDISK}2 until later on when we start doing chroot stuff.

Make a simple partition table. This tutorial assumes you'll use an MBR partition. (Note: At some point in the future, you'll need to figure out how to do this with a GPT partition and an EFI-enabled bootloader.)

This partition layout is really basic: we make one partition for what will be our /boot partition and a second partition for the entire rest of the disk. Dual-booting is not covered here. (Adding a dedicated swap partition is also beyond the scope of this tutorial. If you want swap on your machine, consider adding it post-install as a ZFS zvol.)

Be very cautious here. We are making irreversible changes to your disk from this point forward. If the drive has data you care about, do not proceed.

dd if=/dev/zero of=${ROOTDISK} bs=1M count=2
/sbin/parted --script ${ROOTDISK} mklabel msdos
/sbin/parted --script --align optimal ${ROOTDISK} mkpart primary  1MiB 513MB  # /boot
/sbin/parted --script --align optimal ${ROOTDISK} mkpart primary 513MB  100%  # crypt
/sbin/parted --script ${ROOTDISK} set 1 boot on

Check that the partition table looks OK.

/sbin/parted --script ${ROOTDISK} print

Create an encrypted LUKS volume on the second partition. If you want a good password, consider digging up your old copy of Yahtzee and rolling a Diceware password. Once the LUKS volume is created, open it.

cryptsetup luksFormat -h sha512 ${ROOTDISK}2
cryptsetup luksOpen  ${ROOTDISK}2 cryptroot

Install the ZFS utilities for Linux and begin creating your zpool and ZFS datasets in the mounted LUKS volume.

A ZFS dataset is an odd construct, something between an LVM logical volume and a file system. ZFS is a physical device management tool, a volume manager, and a file system all at once. Since it is simultaneously both the file system layout on a device and it oversees that same device for the kernel, it doesn't precisely fit into one definition or another. Think of a zpool as the abstracted logical grouping of your devices (like one or more physical disks), and a dataset as the configuration for the files on those devices (like mount points and size quotas). Between your files and the platters on the hard disk itself, there is no Dana. Only ZFS.

apt install -y zfsutils-linux
zpool create -O mountpoint=none -O compression=lz4 -O atime=off -o ashift=12 zmint /dev/mapper/cryptroot
zfs   create                       zmint/root
zfs   create -o mountpoint=/       zmint/root/mint18
zpool set bootfs=zmint/root/mint18 zmint


You will need to install a number of ZFS packages and dependencies to the installer as well as re-installing them to the target OS after you run chroot later in this howto. You can, however, save the .deb files and re-use them for both targets. If you intend to do this to multiple machines, this could save you time and bandwidth. If you don't want to download the .deb files twice, do this instead of running apt install zfsutils-linux:

Optional .deb file download steps

apt install -y --download-only zfsutils-linux
mkdir ./my-zfs-debs
cp -p /var/cache/apt/archives/*.deb ./my-zfs-debs
# scp -r ./my-zfs-debs my@backup:~/backup-location # export .deb files to a remote location
dpkg -i ./my-zfs-debs/*.deb # manually install from local disk

Because the / partition is already mounted, you will get an error running the last "zfs create" command. It's OK to proceed. You only need one mountpoint for ZFS to work, but you may optionally set up more fine-grained mount points now. A separate home dataset is probably a good idea:

Optional mount point:

zfs create -o mountpoint=/home zmint/home
zfs list

When the ZFS datasets are created how you want them, export the zpool and reimport it under the /mnt directory:

zpool export -a
zpool import -R /mnt zmint

Format the /boot partition. Linux needs to boot something when the machine starts, and it needs to be both (1) unencrypted and (2) not ZFS. The only open source operating systems I know that don't need an unencrypted bootstrap component to support full disk encryption are FreeBSD 10+ and OpenBSD. Traditionally, if your Linux boot data isn't on ext2 or ext3, you're asking for trouble. Some people will insist that it's OK to format /boot as an ext4 file system. We have a word for people like that: "youths". Or some days, "whippersnappers". I prefer ext3 because (1) it's got journaling and (2) that journaling can be turned off and you can just go back to treating it like ext2 in a pinch.

mkfs.ext3 ${ROOTDISK}1
mkdir /mnt/boot
mount -o noatime ${ROOTDISK}1 /mnt/boot

Install Linux Mint 18. There are a number of ways to do this, but the one I really enjoyed learning about is to skip the Ubiquity GUI and just extract the base system layout onto /mnt with unsquashfs:

apt install -y squashfs-tools
time unsquashfs -f -d /mnt/ /media/cdrom/casper/filesystem.squashfs
cp -p /run/resolvconf/resolv.conf /mnt/run/resolvconf/resolv.conf
cp -p /media/cdrom/casper/vmlinuz /mnt/boot/vmlinuz-`uname -r`

I hear debootstrap may also work. I like to time the unsquashfs command because I've never installed Linux in under 2 minutes with a GUI. Technology. Ain't it grand?

Make some additional mounts and then chroot into your fledgling Mint system to do configuration:

cd /
for i in /dev /dev/pts /proc /sys; do mount -B $i /mnt$i; done
chroot /mnt /bin/bash --login

Set up /etc/fstab. You will need to add an entry for /boot and for the decrypted ZFS root dataset.

Add lines for /boot and /dev/mapper/cryptroot. Instead of using /dev/sda, find the UUID of your device and use it here. UUIDs are unique and will save you headaches in the future if you ever have to swap drives, SATA cables, or motherboards. Since you mounted the Live CD's /dev partition, take a look for what UUID maps to /dev/sda1:

ls -l /dev/disk/by-uuid

Note the symlink for ../../sda1, a hexadecimal string like "702fec05-5da3-449e-9943-a7b546883fa2". Use it to refer to your /boot partition in /etc/fstab:

vi /etc/fstab
# Add these lines, adjusting your UUID value appropriately:
# UUID=702fec05-5da3-449e-9943-a7b546883fa2 /boot ext3 defaults,noatime 0 2
# /dev/mapper/cryptroot                     /     zfs  defaults         0 0

Make sure your Mint system will correctly identify your LUKS volume when it boots. Find the UUID for ../../sda2 and add it to /etc/crypttab. For this example, let's say the UUID for /dev/sda2 is b43e9b90-3b4c-4ac0-9969-073b66dd7d73:

ls -l /dev/disk/by-uuid
echo cryptroot UUID=b43e9b90-3b4c-4ac0-9969-073b66dd7d73 none luks,discard >> /etc/crypttab

This is basic setup stuff:

passwd -u root
ln -s /proc/self/mounts /etc/mtab
echo myhostname > /etc/hostname
echo myhostname >> /etc/hosts
dpkg-reconfigure tzdata

Add a user and a locale.

cd /usr/lib/ubiquity/user-setup

locale-gen en_US.UTF-8
cd /usr/lib/ubiquity/localechooser

We already installed the ZFS utilities, but that was on the Live CD and they will not persist after a reboot. Install them a second time onto the new system. Canonical lawyers, Oracle lawyers, OpenZFS leadership team... please fix this.

apt update
apt install -y zfsutils-linux

Optionally, if you chose to download the .deb files for the zfsutils-linux package, you can install them now, either by mounting the ./my-zfs-debs dir to somewhere under /mnt, or by copying them from your backup location and doing a dpkg -i ./my-zfs-debs/*.deb.

Prepare a ZFS-enabled initramfs:

apt install -y zfs-initramfs
cp -p /lib/udev/rules.d/60-zpool.rules /etc/udev/rules.d/
update-initramfs -c -k all

update-initramfs looks like it gets run when zfs-initramfs gets initially installed, I'm not totally sure. I run it manually and it seems to not hurt anything.

Edit your grub config. Specifically, find /etc/default/grub and change the GRUB_CMDLINE_LINUX_DEFAULT line:

vi /etc/default/grub
# change: 
# to: 

I have had limited success with the Linux Mint splash screen on a ZFS-root system. Command-line prompts for the decryption password work OK. Leave in the "splash" argument if you're feeling lucky.

Before you update grub, make sure that /dev/mapper/cryptroot is symlinked to /dev/cryptroot. Otherwise, it will throw an error.

ln -s /dev/mapper/cryptroot /dev/cryptroot

You have to do this once by hand, but you can set it up to be automatic every time the machine restarts. Do this now so you don't forget after a reboot and potentially mess up your next apt upgrade:

echo 'ENV{DM_NAME}=="cryptroot", SYMLINK+="cryptroot"' > /etc/udev/rules.d/99-cryptroot.rules

Update and install the new grub config, presumably to /dev/sda:

grub-install /dev/sda

Exit the chroot, unmount all your /mnt mounts, export your zpool, and reboot:

umount /mnt/dev/pts
umount /mnt/dev
umount /mnt/proc
umount /mnt/sys
umount /mnt/boot
zfs umount -a
zpool export -a


Eject the install media and test your new Mint machine. From here, you're on your own. Patch your system, make your first ZFS snapshots, and start customizing it for your needs. Congrats!

By the way, if you want ZFS without this hassle, FreeBSD supports root ZFS in the default installer and has for a while now. There's even a book I recommend.


Different tutorials on this subject insist that the names of the LUKS volume or the ZFS zpool (which I called "cryptroot" and "zmint", respectively) are hard-coded and must be the exact strings "cryptroot" and "rpool". I tested this with different values for each and it still worked. You can call your LUKS volume "bob" for all it seems to matter. Just don't call your zpool "tank". It will still work, but it's cliché and you'll just make Allan Jude sad.

External References and Resources











The DIY root server: Be Your Own Dot

Previously, we have worried about setting up a secure mechanism for running a DNSCurve-enabled recursive DNS resolver and a DNSCurve-enabled authoritative DNS server. Both of these are important, since you need to have both (a) software to encrypt your DNS queries and it needs to be compatible with (b) software to encrypt DNS responses.

If you recall from those earlier tutorials, every DNS lookup starts with a query to the root servers. But what are "the root servers" exactly?

To put it simply, the root servers are thirteen machines that know where every top-level domain (TLD) lives. There used to be a few TLDs: the main ones (.com, .net, .org, .gov, et cetera) and then a few dozen more for each country (.us, .uk, .jp, .de, and so on). In the last few years ICANN (the people in charge of that decision) have opened up the TLD namespace to just about every durn thing you can imagine. There are over 1500 TLDs now and more are constantly being added. If you thought the Internet was over-commercialized before, now we have a .mcdonalds and a .americanexpress. There isn't a .popeyes yet, but there's a .church. I don't think that one's for chicken but I could be wrong.

This all came about because Internet domains are, for convenience (if not necessity), hierarchical. When you want to go to www.youtube.com, you want to go to the one true YouTube, not some other random website that might call itself "youtube" and has gamed a search engine to make it the top result. So there needed to be a system that provides this incontrovertible consensus that youtube.com is, for everyone, everywhere, the domain with the cat videos and video game playthroughs you know and love. Hence, there must be a central source of truth, and that's where the root servers come in.

Root servers don't know where www.youtube.com is, but they are the definitive source of truth for how to reach the .com servers. And those .com servers are the definitive source of truth for youtube.com, and so on down the line until you get to where the cat videos are. Your recursive DNS resolver needs to ask these questions and trust the answers it receives. This is a good thing, because you want your cat videos. This is a bad thing because if someone wants to deprive you of cat videos, they can manipulate the domain name responses to keep you from them. Or to give you the wrong cat videos, which is somehow even worse. Unless, of course, you're running DNSCurve end-to-end.

First, the bad news: at present, this isn't possible. If you set up a DNSCurve-enabled resolver and a DNSCurve-enabled authoritative DNS server, you have encrypted the DNS traffic between you and the last set of machines you need to reach. But those darn root servers aren't running DNSCurve.

Now, the good news: you don't need the root servers, not exactly. What you need is the service that the root servers provide, which is that mapping from top level domains to nameservers that can answer questions for those domains. And if you can get encryption and authentication on top of that, it would be a lot better than depending on unencrypted root server queries.

So how do you get TLD data without asking the root servers? Easy! Make a copy of the Internet.

OK, maybe not the whole thing, but at least you can have a copy of the top level domain DNS database. It's freely available, and in one easy location, too:


There's an uncompressed version available too, but hey, packets ain't free, kids.

The really useful thing about this zone file is that it's cryptographically signed. This means that if you verify the data against its signature, you can start to have confidence that the root data hasn't been altered by a third party.

With your own (signed) copy of the root zone, you automatically know where all the .com servers are, all the .org servers are, and so on. You just need a way to make this data accessible to your recursive resolver.

We've already developed the skills that we need to do this. We just need to create one additional authoritative DNS server that we can pretend to be a root server to our recursive resolver.

If you use tinydns, this is going to be pretty simple. Set it up on a loopback IP address on the same machine as your recursive resolver. Let's use

  1. Create a new group and users for your new DNS root service and for your root zone fetching function:

    sudo groupadd _dnsroot
    sudo useradd -d /dev/null -s /usr/bin/false -g _dnsroot _dnsroot
    sudo useradd -d /dev/null -s /usr/bin/false -g _dnsroot _dnsrootlog

    sudo groupadd dnsgpg
    sudo useradd -d /home/dnsgpg -m -s /bin/sh -g dnsgpg dnsgpg

    I like to use a dedicated dnsgpg user account to hold the GnuPG keyring.

  2. Configure tinydns. If you installed djbdns-1.05 as per the recursive resolver howto you will already have tinydns-conf on your system:

    sudo tinydns-conf _dnsroot _dnsrootlog /etc/dnsroot

    This will make a new service directory for you, so you can switch to the default /etc/dnsroot/root directory and set up your new zone. Note that /etc/dnsroot/root is the directory which is chrooted into when the tinydns service runs and has nothing to do with our using it for the root zone. There are a lot of things we're calling root here, so I want to avoid confusion.

  3. Install GnuPG. Odds are really good that you either have this installed already or you can add it very quickly. This depends on your operating system and may require you to add the necessary package. A by-no-means complete list of commands that may help:

    FreeBSD: sudo pkg install gnupg20

    OpenBSD: doas pkg_add gnupg

    Debian/Ubuntu: sudo apt-get install gnupg2

    (Dear OS maintainers, please standardize on this nonsense. Thanks.)

  4. Get the root zone signing key. This is going to change over time because the people who are in charge of that decision believe in regular key rotation and don't believe in subkeys. There are pros and cons to this, but oh well. The folks who own the root zone have opted to use a 1024-bit DSA key. This isn't considered to be long-term secure anymore, but it's all we've got. We have to roll with the decisions of the key owners.

    Until March 2018, the root zone is signed by the following GnuPG ID: "Registry Administrator <nstld@verisign-grs.com>". This ID can be sought on a keyserver of your choice.

    The fingerprint for this key is: F0CB 1A32 6BDF 3F3E FA3A 01FA 937B B869 E3A2 38C5

    You have to install this cert locally before you can use it:

    sudo su -l dnsgpg
    # Confirm you are the dnsgpg user
    # Confirm you are in the dnsgpg user's home directory
    gpg2 --keyserver pgpkeys.mit.edu --recv-key E3A238C5
    gpg2 --list-keys --fingerprint
    # Verify the certificate is installed
    # Confirm you are no longer the dnsgpg user

  5. Convert the root zone into tinydns-data format. Dan Bernstein figured out how to do this a long time ago and put his notes together about how he did it with the now-defunct ORSC root database. His method involved doing an AXFR and cleaning up the result. We already have the zone file, since we fetched it over HTTPS and will verify its correctness with GnuPG.

    I made a slightly modified version of Dan Bernstein's cleanup script that works with the InterNIC root.zone file format and I am making it publicly available here for the first time.

    wget -N https://su.bze.ro/software/dnsroot
    chmod 0755 ./dnsroot
    sudo mv ./dnsroot /etc/dnsroot/root/

    If you look at my dnsroot script, you'll notice one of the first lines looks like this:

    echo .::

    This line tells tinydns that the "." domain is controlled by, which we also specified when we ran tinydns-conf. This IP address must match in both locations, so if you change the IP address on which tinydns will run, make sure this line in the dnsroot script always matches what's in /etc/dnsroot/env/IP.

  6. Modify the permissions of the /etc/dnsroot/root directory to allow the dnsgpg user account to edit files there:

    sudo chown dnsgpg.dnsgpg /etc/dnsroot/root

  7. Fetch the latest root.zone.gz file and its GnuPG signature, and check that they match.

    cd /etc/dnsroot/root
    sudo su dnsgpg --command ' \
      wget -N https://www.internic.net/domain/root.zone.gz && \
      wget -N https://www.internic.net/domain/root.zone.gz.sig && \
      gpg2 --status-fd 1 --verify /etc/dnsroot/root/root.zone.sig \

    Check that the GnuPG signature is valid. The output should contain something along the lines of:

    Good signature from "Registry Administrator <nstld@verisign-grs.com>"

  8. If the signature is good, extract the data from the .gz file and run the dnsroot script:

    cd /etc/dnsroot/root
    sudo su dnsgpg --command ' \
      gzip -d -c < root.zone.gz > root.zone.tmp && \
      mv -f root.zone.tmp root.zone && \
      ./dnsroot < root.zone > ./data.tmp && \
      mv -f ./data.tmp ./data && \

  9. Confirm that you have a valid ./data and ./data.cdb file in /etc/dnsroot/root. Check that ./data starts with your ".::" line, and check that the data.cdb reports valid data when you query it with tinydns-get.

    cd /etc/dnsroot/root
    head ./data
    # Confirm .:: and other tinydns-data lines for the TLDs
    tinydns-get ns .
    tinydns-get ns com
    tinydns-get ns net
    tinydns-get ns org
    tinydns-get ns kp
    # Confirm that these domains return NS records

  10. If you grep the root.zone file for the IP addresses you get in the responses, they should match. Because of the way the BIND zone file format works, you'll have to do this twice. For a specific IP address:

    grep ip.ad.dr.ess root.zone
    # Take the nameserver string from these response and grep for it
    grep "nameserverstring" root.zone

    You should get an A record and an NS record out of root.zone. This will map a domain name to a nameserver and a nameserver to an IP address. So, for example:

    $ tinydns-get ns kp
    2 kp:
    74 bytes, 1+0+2+0 records, response, noerror
    query: 2 kp
    authority: kp 259200 NS
    authority: kp 259200 NS

    Let's use as the IP we want to check.

    $ grep root.zone
    ns1.kptc.kp.            172800  IN      A

    Now we look for "ns1.kptc.kp." from this result and check that it maps our original TLD nameserver query to the IP address we picked.

    $ grep ns1.kptc.kp. root.zone
    kp.                     172800  IN      NS      ns1.kptc.kp.
    ns1.kptc.kp.            172800  IN      A

    Note that our ./data file eliminates the intermediate nameserver name. It isn't essential for tinydns.

  11. When you're confident that your personal root server is working, you can update your dnscache instance to use it:

    echo > ./@.new
    chmod 0644 ./@.new
    sudo mv ./@.new /service/dnscache/root/servers/
    cd /service/dnscache/root/servers
    sudo cp -p ./@ ./@.backup
    sudo mv -f ./@.new ./@
    sudo svc -t /service/dnscache

    Check that your dnscache instance now reports that the root server is So for example:

    $ dnsqr ns .
    2 .:
    40 bytes, 1+1+0+0 records, response, noerror
    query: 2
    answer: . 259200 NS

    Try using the Internet. Tail your dnscache log file and see if you can resolve the domains you'd normally expect to reach: google.com, bbc.com, internic.net (that's an important one), and so forth.

    If you have trouble with your new root server, restore /service/dnscache/root/servers/@ from the backup you made and restart dnscache.

  12. The root zone changes over time, so perform these "fetch, validate, and convert" steps regularly. The zone file is updated and signed once a day, so anything more frequent than that is unhelpful. Once a week is good enough for my day-to-day network needs, so adjust your fetching frequency accordingly. The dnsgpg user's crontab might be useful here. I have a dnsroot-fetch script I use that does the above fetch-and-run-dnsroot steps:

    wget -N https://su.bze.ro/software/dnsroot-fetch
    chmod 0755 ./dnsroot-fetch
    sudo mv dnsroot-fetch /etc/dnsroot/root/
    sudo su -l dnsgpg
    crontab -e
    # Add this line (without the leading '#'):
    #0 18 * * * (cd /etc/dnsroot/root/; perl -le 'sleep 300 + int rand 500'; ./dnsroot-fetch)

    The Perl section is optional. I like to add the perl command to sleep for a random amount of time more than 5 minutes and less than 13 minutes. This is just a little window to be a good bot and avoid traffic spikes of other scripts that run at the top of the hour or at a quarter past.

I've been hosting my own copy of the root zone and using it as my private root server for about six years and so far, it's working out pretty well.

"But gee," you're probably wondering. "Where's the encryption you've been going on about?" Fair question. If you're going to run this private root server on the same machine as your recursive DNS resolver, you don't really need to put a CurveDNS proxy in front of it. It's going to be consumed by the same host on a loopback address and so there aren't many places a malicious attacker can interfere with this data in transit unless they've already compromised the machine. That's the point of using the "" address that keeps popping up in our configs and scripts.

You can host a public copy of the root zone, though. You know how to do this based on everything we've learned thus far. You can: (a) create a private copy of the root zone based on these steps and then (b) put a CurveDNS proxy in front of it to make it publicly available. The only thing that needs to be adjusted is the IP address of that echo .:: line in your dnsroot script. Since this line is the address that is treated as the one root server in the universe, you need to adjust this to be the public IP address of your public DNSCurve-enabled root server.

You can still host the root on of course. You'd still set "" to be the value of /service/curvedns/env/FORWARDIP, but for the /etc/dnsroot/root/data file, it needs to report a reachable, public IP address. There may not be a good way for your server to be able to predict its public IP address, especially if it's using a NAT configuration, so adjust your local ./dnsroot script accordingly to update the line for the "." domain. If you want to be thorough, you can also set up an axfrdns service to handle TCP queries. By default, tinydns only supports UDP queries. curvedns will handle both UDP and TCP requests, but axfrdns needs to be set up to receive the TCP queries that curvedns will forward to it. But I'm getting ahead of myself.

Next time: putting all of this together.


The DIY CurveDNS Proxy: Privacy for Your Public Domain Data

At some point someone noticed that I put together a recursive resolver howto and it got linked to dnscurve.io, a good resource for all your DNSCurve query-encrypting needs.

I ended that howto by saying "this is only the beginning," which is an understatement if ever I've made one. DNSCurve is an underdog in the Internet security community. By my surveys, fewer than 50 domains in the .com zone are using DNSCurve to protect their service. How big is .com? Over 126 million domains in total. Can we fix that? Yes, we can. Can we do it alone? No, not yet. But there's still hope.

We can still control our DNS queries though. DNSCurve encryption is per-query encryption: if you have to ask six questions to six different machines to get a valid answer, all six of those questions need to be encrypted or you will leak information to a passive observer. To simplify this problem, consider how DNS works:

  1. I have a question: "How do I get to www.google.com?"
  2. I ask the root servers: "What's the IP address for www.google.com?"
  3. The root servers say: "I don't know, but you can ask these .com servers," and give me a list to check.
  4. I ask the com servers: "What's the IP adddress for www.google.com?"
  5. The com servers say: "I don't know, but you can ask the google.com servers," and give me a list to check.
  6. I ask the google.com servers: "What's the IP address for www.google.com?"
  7. The google.com servers say: "It's,, and a few more you can use."

In many cases there can be more questions, and detours, and circular dependencies, and all sorts of fun convolutions, but this is the basic version of how your DNS queries get answered. These are each individual questions, sent to different parts of the world, and answered by different software. If you aren't using DNSCurve for each of those queries, you're not getting the full protection that DNSCurve can provide. This isn't your fault, since in order to get this end-to-end DNSCurve support, every level — each server that is going to either give you an answer or else say "I don't know, but you can ask..." — needs to support DNSCurve.

Right now, there are 50 .com domains that are doing their part. That's not good enough:

  1. The root servers don't support DNSCurve.
  2. No major top level domain service supports DNSCurve. You really want .com, .net, and .org to get on board with this.
  3. Only a handful of domains support DNSCurve.

By yourself, you can only control that last one. You can set up your authoritative DNS server with a CurveDNS proxy. This is a good thing! For starters, CurveDNS does not replace your DNS server, it only acts as a forwarder to it. If you use BIND, you can keep using BIND. If you use MaraDNS, you can keep using MaraDNS. If you want, you can put CurveDNS on a different IP and provide both encrypted and non-encrypted DNS service for backwards compatibility. CurveDNS will forward non-DNSCurve queries, so this isn't necessary, but you can do it that way if you really like how your DNS server handles its sockets.

The following tutorial is NOT for the casual internet user. This is strictly for domain owners who want to provide DNSCurve-enabled service to the general public. Bump that 50 up.

Are you ready to make your DNS server awesome? Great!

What do I need to get started?

You should be familiar with my earlier recursive resolver howto. You do not need to use a DNSCurve-enabled recursive resolver to set up a DNSCurve-enabled authoritative resolver, but the principles will be the same: start with a Unix or Linux system and prepare to fetch, compile, and install some software. If compiling your own software makes you nervous, stop. Take a deep breath, bookmark this page, and come back when you're feeling brave.

  1. Get yourself a copy of CurveDNS. CurveDNS was developed by Harm van Tilborg and is currently the only serious DNSCurve proxy for authoritative DNS servers I've yet found. It uses NaCl, the Networking and Cryptography library. You may remember this from the recurvsive resolver howto. If you do, you can reuse that version here by linking it into CurveDNS, but this is a little more advanced and is not strictly necessary. CurveDNS includes its own copy of NaCl that it will use by default. For the sake of simplicity, the included version of NaCl will be the one we use.

    The most recent version of curvedns-0.87.tar.gz has the following checksums:

    MD5 = 693c8b0e96642dbc3bc2013e5c71fc4f

    SHA1 = 5941e53de5836d2cef037b8e45f585762bd6e8df

    SHA256 = 9f45d0324d2917dd93546b0af74428ff50e06293fd4c273b7e5f6b62f88d9e6a

    SHA512 = c8cf96340c3db1d9061f9f138df40f52254eb53fef8bcb65cfde57d02fd45977200e003e7dbaa4d51328cf5fedb14c860793a7e89050d2202205d43c5bd93264

    If you want an updated version that may be released at a later date, it also has a github project you can clone.

    wget -N https://github.com/curvedns/curvedns/archive/curvedns-0.87.tar.gz
    openssl dgst -sha256 ./curvedns-0.87.tar.gz
    # SHA256(./curvedns-0.87.tar.gz)= 9f45d0324d2917dd93546b0af74428ff50e06293fd4c273b7e5f6b62f88d9e6a
    [ -d ./src ] || mkdir -m02755 ./src.new && mv ./src.new ./src
    gzip -d < ./curvedns-0.87.tar.gz | (cd ./src; tar -xf -)
    cd ./src/curvedns-curvedns-0.87
    time ./configure.nacl

    NaCl can take a while to compile, so I like to do this in a tmux or GNU screen window and let it run while I do other things.

    While that compiles, here's a side note about NaCl: there are a few different forks of the NaCl library you may want to explore. While I am going to use the included NaCl libraries in this tutorial, you can consider finding one NaCl library that's right for you and using it in your software. NaCl has grown into an ecosystem of libraries that include:

    • The original NaCl software, whose purpose is to simplify and enhance online encryption, signing, and authentication into a single easy-to-use API.
    • The libsodium project, which aims for portability and redistributability.
    • The tinynacl fork used by Jan Mojžíš for dqcache. More on this later.
    • tweetnacl, a port of NaCl that can be expressed in 100 140-character-or-fewer tweets. If you ever enjoyed the idea of "RSA encryption in x lines of Perl", this is a fun read.
  2. CurveDNS requires libev and the libev dev resources to compile. This is a common package that is very likely supported by your operating system's package manager. In FreeBSD, you can use "pkg install libev" and "pkg install libevdev". In OpenBSD, use "pkg_add libev", and in Debian/Ubuntu, "apt-get install libev-dev". Your actual mileage may vary. Don't forget this step!

  3. When NaCl is finished compiling, I suggest adding the chroot patch for CurveDNS. The curve-chroot.diff file has the following checksums:

    MD5 = 079f9a82c0bc2c2c8c8f9c5a3fbfb8bb

    SHA1 = 1d6162364211861d74f9baaa91f7e01faecc1dd4

    SHA256 = 3edcfd551eceace7231ac0f6c84812e29c0f4d594277b639507f29f56c3e3dd0

    SHA512 = f3278cd70b8d57ed06a13262e10af9ecbf3934c132010aa6d90e6c6eb7a8f4c5ef085c1c0981f982af9e8e54e4b742a8f60e4c6e40978623f7adc85a0d0673a9

    cd ./src/curvedns-curvedns-0.87
    wget -N https://www.blinkenlights.ch/misc/curve-chroot.diff
    openssl dgst -sha256 ./curve-chroot.diff
    # SHA256(./curve-chroot.diff) = 3edcfd551eceace7231ac0f6c84812e29c0f4d594277b639507f29f56c3e3dd0
    patch -p1 < ./curve-chroot.diff
  4. (Optional, based on platform) Additionally, there's a patch for CurveDNS so it will compile on OpenBSD. Use it if you are building CurveDNS on that platform.

  5. (Optional) A few years ago I hit a problem with CurveDNS's source IP socket binding code, so I'm providing a patch for ip.c if you plan to use the CURVEDNS_SOURCE_IP environment variable.

    cd ./src/curvedns-curvedns-0.87
    wget -N https://su.bze.ro/software/curvedns_sourceip_bindfix.patch
    openssl dgst -sha256 ./curvedns_sourceip_bindfix.patch
    # SHA256(./curvedns_sourceip_bindfix.patch) = 6e8d073e3ae1f97dd6eb6ca60f2c6f383d766179df133ac7950c59193dd34e02
    patch -p0 < ./curvedns_sourceip_bindfix.patch
  6. Compile CurveDNS:


  7. Assuming all goes as expected, CurveDNS will compile. Confusingly, it does not include an installer, so put the files where you need them. I suggest /usr/local/bin:

    sudo install -m0755 curvedns /usr/local/bin/
    sudo install -m0755 curvedns-keygen /usr/local/bin/

    Now comes the fun part, setting up your CurveDNS service:

  8. Create a new group and two users.

    sudo groupadd _curvedns
    sudo useradd -d /dev/null -s /bin/false -g _curvedns _curvedns
    sudo useradd -d /dev/null -s /bin/false -g _curvedns _curvednslog

  9. Set up a "curvedns" service directory. This is assuming you use daemontools:

    [ -d ./curvedns ] || mkdir -m02755 ./curvedns.new && mv ./curvedns.new ./curvedns
    mkdir -m02755 ./curvedns/env
    mkdir -m02755 -p ./curvedns/log/main
    sudo chown _curvednslog._curvedns ./curvedns/log/main

  10. Set up your service and log run files:

    Create ./curvedns/run:

    echo '#!/bin/sh' > ./run.new
    echo 'exec 2>&1' >> ./run.new
    echo "exec envdir ./env sh -c '" >> ./run.new
    echo '  exec envuidgid _curvedns softlimit -d$DATALIMIT /usr/local/bin/curvedns $IP $PORT $FORWARDIP $FORWARDPORT' >> ./run.new
    echo "'" >> ./run.new
    mv ./run.new ./curvedns/run
    chmod 0755 ./curvedns/run

    Create ./curvedns/log/run:

    echo '#!/bin/sh' > ./run.new
    echo 'exec setuidgid _curvednslog multilog t ./main' >> ./run.new
    mv ./run.new ./curvedns/log/run
    chmod 0755 ./curvedns/log/run

  11. Create your service environment variables. You will need to know where you plan to run CurveDNS and where your authoritative DNS server lives. The variables you will need are:

    • CURVEDNS_DEBUG - (Optional) How much information will CurveDNS write to the log. Make this an integer from 1 to 5. 1 is fatal messages only, 5 gives a lot of additional debugging data. For this tutorial, we are going to set this to 5. You can change this later if you like.
    • CURVEDNS_SOURCE_IP - (Optional) If you applied the source IP patch, you can force an output IP for CurveDNS to use. This is handy when you are doing IP aliases and NAT.
    • DATALIMIT - How big you will let CurveDNS get. Too low will crash the service, too high can cause system thrashing. You will need to tune this value to find what works for you.
    • FORWARDIP - The IP address where your authoritative DNS server lives.
    • FORWARDPORT - The port where your authoritative DNS server lives. Almost always "53".
    • IP - The IP that CurveDNS will use.
    • PORT - The port that CurveDNS will use. Almost always "53".

    Bear in mind that you cannot run CurveDNS on the same port and IP as your authoritative DNS server, though you can run CurveDNS and your DNS service on the same machine. For example, our DNS server runs on the IP address I am going to run CurveDNS on, so my environment variables will be:

    echo 5 > ./curvedns/env/CURVEDNS_DEBUG
    echo 20000000 > ./curvedns/env/DATALIMIT
    echo > ./curvedns/env/FORWARDIP
    echo 53 > ./curvedns/env/FORWARDPORT
    echo > ./curvedns/env/IP
    echo 53 > ./curvedns/env/PORT
  12. The best part, now you create your CurveDNS encryption key. CurveDNS does not use a separate key exchange mechanism. A CurveDNS server's hostname is that server's public key. The keygen utility prints out secret information, so this part needs to be kept safe at all times. If your CurveDNS private key is readable, your data isn't secured.

    This is where we put on our tinfoil hats and fiddle with our file masks.

    # Remember this value. It's usually 0022.
    umask 077
    /usr/local/bin/curvedns-keygen > ./curvedns/private.key
    # Use the value from your first umask check here:
    umask 0022
    ls -l ./curvedns/private.key
    # Check that the permissions on private.key are -rw-------

    Check the contents of ./private.key. It should look something like this, with different values for each of the three lines:

    DNS public key: uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq
    Hex public key: 35a43e71c5f3149aef50eece245432c4beb9508c4f677c15ac7415a50812a25a
    Hex secret key: f3f16c27f2ff6f94d52b0aa6b4011444be707f5e05b96feb31890096496f7075

    You can ignore the hex public key, it isn't used directly. Rather, it is base-32 encoded in a special way. That is the "DNS public key" string, in this case, "uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq". Write it down, because that is the value you will give to your domain registrar so that people can reach your CurveDNS proxy.

    The hex secret key is the other important portion. This value needs to be put into an environment variable called CURVEDNS_PRIVATE_KEY:

    # Remember this value. It's usually 0022.
    umask 077
    tail -1 ./curvedns/private.key | cut -f2 > ./curvedns/env/CURVEDNS_PRIVATE_KEY
    # Use the value from your first umask check here:
    umask 0022
    chmod -w ./curvedns/env/CURVEDNS_PRIVATE_KEY
    ls -l ./curvedns/env/CURVEDNS_PRIVATE_KEY
    # Check that the permissions on CURVEDNS_PRIVATE_KEY are -r--------

  13. Put your curvedns service directory in its permanent location and symlink it to /service:

    sudo mv ./curvedns /etc
    sudo ln -s /etc/curvedns /service/curvedns

  14. Now tail /service/curvedns/log/main/current and check that your service is running. You can test that CurveDNS is forwarding correctly with the usual DNS tools (dig, dnsq), and you can verify that the DNSCurve portion is working with dq, a remarkable DNS tool that Jan Mojžíš wrote.

    Normal traffic should work as expected. Assuming your authoritative DNS server controls myvanitydomain.dom, you can query it through the CurveDNS proxy:

    dig @`cat /service/curvedns/env/IP` ns myvanitydomain.dom

  15. If normal DNS traffic is passing through CurveDNS correctly, use dq to test the encryption. Use the DNS public key string you got from running curvedns-keygen:

    dq -a -k uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq ns myvanitydomain.dom `cat /service/curvedns/env/IP`

    If this is working, you can put your CurveDNS proxy online. If you are using NAT or a firewall to forward external DNS lookups to, you can (after making a backup of your configs), change your setup to forward DNS to and do the above validation checks from an external IP address.

  16. Last step: update your DNS records on your authoritative DNS server and then update your registrar. If you were previously using "ns1.myvanitydomain.dom" as the name you provided for your authoritative DNS server, change this to
    "uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq.myvanitydomain.dom". Much longer, true, but this is the public key for your CurveDNS proxy. If it were shorter, it wouldn't be as secure. When a DNSCurve-enabled recursive resolver sees this nameserver hostname, it will know (a) your domain supports DNSCurve and (b) how to encrypt all the queries it will send to that host.

    If you're using tinydns, you would change your DNS lines like this (assuming your DNS servers are and

    - .myvanitydomain.dom:
    + .myvanitydomain.dom:

    See How to install a DNSCurve forwarder for more examples.

When you are confident your CurveDNS proxy is working well for your DNS traffic, you can repeat this process with your remaining nameservers. This slightly changes the questions your DNSCurve-enabled resolver asks:

  1. I have a question: "How do I get to www.myvanitydomain.dom?"
  2. I ask the root servers: "What's the IP address for www.myvanitydomain.dom?"
  3. The root servers say: "I don't know, but you can ask these .dom servers," and give me a list to check.
  4. I ask the dom servers: "What's the IP adddress for www.myvanitydomain.dom?"
  5. The dom servers say: "I don't know, but you can ask uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq.myvanitydomain.dom." (This isn't just a strange hostname. It's also the public key to use when encrypting communications to that host.)
  6. I ask uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq.myvanitydomain.dom: "*whisper whisper whisper*" (It's secret. No one knows what I'm asking.)
  7. The uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq.myvanitydomain.dom says: "*whisper whisper whisper*" (It's secret. No one can see what the response is.)

Unlike a DNSSEC-signed zone, which is a publicly-viewable signature scheme for publicly-viewable queries and responses, the entire conversation with the CurveDNS proxy is encrypted and authenticated. I encrypt my initial question to uz5p19xmspsm75ntr3bgrm92b964qhmc8kkhutsrbjpnc5bb4822pq.myvanitydomain.dom, and all subsequent queries to and responses from that nameserver are secure. In my sequence of resolver questions, the last two are private, and only I and the nameserver know what was asked and what was answered. This is a good start, but it doesn't give us the end-to-end encryption we need to make a safe, reliable, and functional Internet. You're still leaking your question to the root servers and to the intermediate nameservers that handle .dom. And one-third protection is simply no protection at all. We still have a long way to go.

Earlier in this tutorial, I mentioned that the root servers don't support DNSCurve and you can only enable DNSCurve on your own domain. This isn't entirely true: after securing your own domains, there's also a way for you to secure the first part of your DNS lookups, too. There's still hope.

Next time: Getting to the "root" of the problem and any other awful puns that may occur to me.