2017-11-10

A "review" of Jeff VanderMeer - Authority

You and I will be together
When we shed our memory
— They Might Be Giants, "Erase"

Imagine an adventure. Imagine going on an exciting adventure, like a James Bond action story, or a suspenseful murder mystery. That's Jeff VanderMeer's Annihilation.

Now imagine, at the conclusion of the adventure, James Bond's departmental accountant carefully ordering all of 007's accrued expense reports then filling out forms for regulatory compliance and fiduciary reimbursal purposes. Imagine a spry novice detective finding the killer with a brilliantly lateral bit of inspiration, then leaving while a small team of uniformed police officers take everyone's statements and dutifully record them into sprial-bound notebooks to be later processed as evidence. Imagine, also, this ex post facto processing procedure being performed. Meticulously. That's Jeff VanderMeer's Authority.

Are you bored yet? You should be. We all love a good story. What we don't entertain is the droll paperwork that comes after the credits roll. We love to see our heroes ride off into the sunset, not seated at a desk reviewing files, scheduling followup interviews, and thinking about their relationship with their mothers. Wait, what?

Let me start with the beginning. Annihilation gave us the biologist, a very flawed, very human narrator who transcends her basic role of just an anonymous observer. She becomes a person. We grow to understand her foibles and as we understand them better, we understand her better, too. Authority shifts the narrative away from the biologist to a new person. A person with a name and everything.

As well as Annihilation built up the enigma that is Area X, Authority tears it all down, the way a young child would break a stained-glass window just to see the pretty shards fall and savor the impulsive joy of making chaos for the sake of chaos. That child, that chaos, is John "Control" Rodriguez. The irony of this is not lost on me.

Think of this as solving problems
That should never have occurred
Please don't call it strangulation
That is such an ugly word

After the warm camaraderie that VanderMeer carefully fosters with the biologist's narrative, the change in Authority is bracing, like a cup of ice water thrown in your face, or a sharp slap across your cheek. He cautiously blew on a fragile ember in the tinder of Annihilation and created the bright, burning flame of the biologist. Then he dumped ice water on it and called it "Control". "Control" is so smug and self-assure that his internal monologue grates on you like a cat's claws. He's douchey, he's bland, he's smarmy. He gave himself the nickname "Control". 'Nuff said.

John, or "Yawn", as I have grown to call him, is brought in to lead the Southern Reach after the events of Annihilation. The mystery of Area X that we experienced first-hand has a terrible counterpart to it: the sheer inescapable beaureacracy of the research organization tasked with assembling the expeditions. Yawn is awful, as a character, as a narrator, and as a person. Authority wastes nearly a hundred pages at the start meticulously boring you through Yawn's first day on the job — boring like boredom and boring like when someone drives a power drill through your skull. He spends the entirety of the longest day I have EVER read reminiscing about the complex relationship he has with his cold and distant mother, who is both distant and cold. Her coldness and her distance were very influential upon him, and he won't let up about it. Not until you understand that his cold mother was distant to him, and even after you get that subtle character trait of his, still he will not relent in reminding himself of how far away and chilly his cold and distant mother is to him, and how frosty and detached he feels about her coldness, and about her distance.

Finger find the button marked "Erase"
Like a deep sea diver falling into a mermaid's embrace

Years ago, Mr. Filthy reviewed the second Lord of the Rings movie and I will never forget his thoughts on it. They fit that movie and they suit Authority here with little modification:

"Jesus H. Christ, The Lord of the Rings 2 Authority was boring. And long. Holy crap, I can't even hang around people I like for three hours without wanting to punch somebody in the nose, so imagine the torture of spending it with speechifying goblins and wizards scientists and beaureaucrats."

"The Lord of the Rings 2 Authority is tedium. Oh, it's spectacular tedium, full of sweeping, breathtaking panoramas and hundreds of shots taken by helicopters circling mountain peaks descriptions of what sounds like Florida. ... It has no personality, no subtlety, juvenile characters and a simplistic plot. It has no personal conflict I cared about, and the dwarf John "Control" Rodriguez got on my fucking nerves."

Everyone gets on the bus out of town
And the lights start going out one by one

Authority is the continuation of the mystery, but it is not itself mysterious. It is the aftermath, the cleanup crew, the story of the adventurers' support team. They don't get to go on the adventures themselves, they just pick up the broken remnants and hand out cake and ice cream to any of the survivors. If Annihilation were an amazing stage play, slogging through Authority is like watching the stage technicians construct the set before the performance and tear it all down again and demanding that you buy a second ticket just to have the privilege of watching the process. I hated, hated, hated this book.

Everything that Jeff VanderMeer proved he could do as an author in Annihilation he apparently forgot in Authority. If you're the kind of asshole who loves knowing how the magician does all of his tricks, you may like this book just for the pure exposition that it divulges. While it does little to explain Area X, Authority takes a seemingly perverse delight in sapping the mystery from that story, quantifying everything you'd want to remain unsaid, naming names and going into obscene detail about much of what Annihilation left tantalizingly ambiguous or open to interpretation.

One of the most compelling and nuanced scenes in Annihilation is when the biologist finds a thing, an entity, she calls The Crawler. I do not wish to give much detail to this scene, it should be read to be believed. It is magical, emotional, and beguiling. She struggles with an ignorance of background information and a wash of sensory stimuli and confronts the situation like a person fighting tidal waves batting her body. Have you ever seen something so cosmically beautiful that it stops your breath? It's a deeply personal moment that every reader will perceive differently. It betrays the sanctity of that moment in Annihilation for Authority to come along and clumsily, rudely, stumble into the room, burp loudly, and announce with no great fanfare The Crawler's backstory. His name was Joe Schmoe and he was a mechanic down at the docks before The Big Bad Thing happened on October 8, 1996, at 2:23 PM, Eastern Daylight Time. It was a Tuesday!-- Harvey's Burrito Barn was having a half-price happy hour that day!

For fuck's sake, Jeff. For fuck's sake.

If Annihilation was a spooky campfire story told in the dead of night to delight and terrify its audience, Authority is that same storyteller the next morning continuing on with needless, extraneous details about who the escaped mental patient was, which psychiatrist diagnosed him, where he got his hook for a hand, who the teenagers parked in the car on Lovers' Lane were, where they came from, what kind of car it was, what shape of door handle it had, who gave them the car, whether or not its registration was up to date, who filed it with the county clerk and when, and how far Lovers' Lane is from Interstate 44. You have to take a right at the second light after Sam's Garage. If you get to Old Man MacGee's apple orchard you've gone too far.

Good god, Jeff, who gives a damn about all that shit? You're killing the whimsical yet ominous mood here.

Put one box on the sidewalk
And you return with the next and the first one's gone

The building blocks that Annihilation used to such great effect here are totally gone. This is a new player, playing a wholly different game. Did you like playing Street Figher? Cool! Now Street Fighter 2 is a tax-filing simulator. Don't forget to itemize your deductions for extra points! Authority fills in some background, unsatisfactorily I might add, and creates at best an interesting final 50 pages or so. Both Annihilation and Authority have breakneck-paced, cannot-put-it-down conclusions, the kind that will keep you compulsively turning pages until 4:30 in the morning if need be. I must have looked insane reading this book, putting on my glasses, picking up Authority, only reading a page, setting it down, taking my glasses off. Then staring off at nothing in particular like Nietzsche, contemplating what I'd just read, processing it for a minute, then putting my glasses on, picking the book up again, and reading another page and repeating this whole process again and again until it was over.

My biggest regret about Authority is that for a bucking-bronco ride of a 50-ish page denouement in the final act, the book is 352 pages long. The first 6/7ths of this book are banal, monotonous, and miserable. Just fucking miserable. There are a couple of bits and pieces thrown in to try, poorly, to keep this slog enticing. It feels like VanderMeer fed you an amazing breakfast and is now starving you over the course of a long, arduous day, throwing you only the meagerest of kitchen scraps, just to keep you alive and hungry while he prepares a late dinner, and it's degrading.

Button marked "Erase"
When darlings must be murdered,
When your heartbreak overrides the very thing you cannot face
The skeletons that won't stay down
The mercy kill that can't be drowned

If I were to reconsider Authority with a kinder view, and I honestly wouldn't, I would see it less of a story about Area X, that great revenant mystery of the first novel, and more about the relationship that Control creates inside his own mind with the biologist. Were I to see his actions as earnestly trying to decipher her like she was a puzzle it might not irritate me so. Yawn wastes a great deal of space waxing poetic about his designs on cracking the case and being the big hero, but the posturing absolutely murders the flow between the first book and the second. The relationship between the narrators is a reflection of their wildly different attitudes and styles. The idea of marrying them into a single unifying story is shallow and ill-fitting. It isn't Romeo and Juliet, or Fred and Ginger, or even Sid and Nancy. It's John and Yoko.

2017-11-04

The Salsa20 Cipher in Powershell, For Some Reason

Sometimes, against my better judgment, I find myself working in ultra-restrictive secure environments, the computer equivalent of a clean room, or like that room in Mission Impossible where Tom Cruise had to hang from wires because everything around him was being monitored by a security force.

Modern secure-access tools aren't a bad thing, per se. They keep China at bay. But sometimes they're just a little too secure.

Long story short, sometimes in trying to keep the bad guys from doing bad things, secure workstations keep the good guys from doing good things, too. For example, sometimes they keep you from using builtin cryptography.

That is quite possibly the dumbest thing I have ever witnessed in my life, and I once legitimately asked my social studies teacher what they called World War I before there was a World War II[0].

I ended up writing one of the many AES variants in Powershell. Not because someone asked me to do so, but because goddammit, no one is going to tell me when I can and cannot move secrets between machines. The secure environment restricted most of my access to OS-native cryptography primitives and even to character-encoding classes. This is kind of like asking a person to swim with one or both hands tied behind their back. You can do it, but don't expect the person to succeed.

AES isn't a bad cipher. It's not perfect. I've disliked S-boxes ever since I found out Feistel grew up in Germany during that whole "starting all the wars" era. It's been heavily audited, there's no practical attack on its pure design and its strongest versions are known to be vulnerable only to weaknesses in implementation. (I may oversimplify here. AES is still good enough for your bank to use, whatever that may be worth.) There's nothing inherently wrong with AES yet. My Powershell-only implementation had no optimizations and is pitifully slow[1]. Like, super duper slow. How slow? Four bytes per second. Not kilobytes. Bytes.

So I went back to my original idea, which was to try a modern cipher that could be run from Powershell and give "good enough" security to data in transit or kept on disk for short periods of time. Things like temporary passwords and OAuth tokens. Not things like banking account information or Social Security numbers. China already has those.

I spent a little time this weekend re-reading and implementing DJB's Salsa20 cipher in pure Powershell. It took about a night. Let's get to it, shall we?

The Salsa20 cipher is a really clever design. Given a key and a nonce, it creates a deterministic set of up to 2^70 64-byte outputs that can be XOR-ed with plaintext in order to create an invertible ciphertext. Given the same key and series of nonces, the ciphertext can be decrypted with the same set of steps.

Salsa20 defines a "rotation" operation that has been around since DJB's old SURF days. It shifts the bits of a 32-bit integer around. In C it looks like this:

#define ROTATE(x,b) (((x) << (b)) | ((x) >> (32 - (b))))

In Powershell, do this:

Function Salsa20-Rotate {
  Param(
    [UInt32]                    $u,
    [Byte][ValidateRange(0,32)] $c
  )
  [UInt32] $m = $u -shl $c
  [UInt32] $n = $u -shr (32 - $c)
  Return [UInt32] ($m -bor $n)
}

The Salsa20 paper also defines a "littleendian" function used to convert 4-byte arrays into 32-bit integers. The array [1,2,3,4] would be the integer 0x04030201 for example. In practice, you need two functions, one to convert an array into an integer, and another to convert the integer back into an array:

Function Salsa20-LoadLittleEndian {
  Param([Byte[]] $x)
  [UInt32] $out = $x[3]; $out *= 256
  $out += $x[2]; $out *= 256
  $out += $x[1]; $out *= 256
  $out += $x[0]
  Return $out
}

Function Salsa20-StoreLittleEndian {
  Param([UInt32] $u)
  $x    = New-Object Byte[] 4
  $x[0] = $u -band 255; $u = $u -shr 8
  $x[1] = $u -band 255; $u = $u -shr 8
  $x[2] = $u -band 255; $u = $u -shr 8
  $x[3] = $u
  Return $x
}

Powershell also lacks a decent integer arithmetic feature, as far as I know. If you add a UInt32 value to a UInt32 value, it will try to keep any overflow from occurring. Sometimes more is less, so when I add 1 to the maximum possible value an unsigned 32-bit integer can have, I want the value to be zero. To address this, create a "sum" function that will add two UInt32 values and return a UInt32 value:

Function Salsa20-Sum {
  Param(
    [UInt64] $a,
    [UInt64] $b
  )
  Return [UInt32]([UInt32]::MaxValue -band ($a + $b))
}

That's all you really need for Salsa20 primitives. I dabbled with writing the quarterround, rowround, and columnround functions as the Salsa20 paper defines them, but the NaCl library doesn't bother with any of that for its Salsa20 reference implementation. So I just used that:

[Byte[]] $c = 101,120,112,97,110,100,32,51,50,45,98,121,116,101,32,107 # "expand 32-byte k"

Function Salsa20-Core {
  Param(
    [Parameter(Mandatory=$True,Position=0)]
    [Byte[]] $k,
    [Parameter(Mandatory=$True,Position=1)]
    [Byte[]] $in,
    [Parameter(Mandatory=$False,Position=2)]
    [Byte] $Rounds = 20
  )

  [UInt32] $j0  = $x0  = Salsa20-LoadLittleEndian  $c[ 0..3]
  [UInt32] $j1  = $x1  = Salsa20-LoadLittleEndian  $k[ 0..3]
  [UInt32] $j2  = $x2  = Salsa20-LoadLittleEndian  $k[ 4..7]
  [UInt32] $j3  = $x3  = Salsa20-LoadLittleEndian  $k[ 8..11]
  [UInt32] $j4  = $x4  = Salsa20-LoadLittleEndian  $k[12..15]
  [UInt32] $j5  = $x5  = Salsa20-LoadLittleEndian  $c[ 4..7]
  [UInt32] $j6  = $x6  = Salsa20-LoadLittleEndian $in[ 0..3]
  [UInt32] $j7  = $x7  = Salsa20-LoadLittleEndian $in[ 4..7]
  [UInt32] $j8  = $x8  = Salsa20-LoadLittleEndian $in[ 8..11]
  [UInt32] $j9  = $x9  = Salsa20-LoadLittleEndian $in[12..15]
  [UInt32] $j10 = $x10 = Salsa20-LoadLittleEndian  $c[ 8..11]
  [UInt32] $j11 = $x11 = Salsa20-LoadLittleEndian  $k[16..19]
  [UInt32] $j12 = $x12 = Salsa20-LoadLittleEndian  $k[20..23]
  [UInt32] $j13 = $x13 = Salsa20-LoadLittleEndian  $k[24..27]
  [UInt32] $j14 = $x14 = Salsa20-LoadLittleEndian  $k[28..31]
  [UInt32] $j15 = $x15 = Salsa20-LoadLittleEndian  $c[12..15]

  for ($i = $Rounds; $i -gt 0; $i-=2) {
    $x4  = $x4  -bxor (Salsa20-Rotate (Salsa20-Sum $x0  $x12) 7)
    $x8  = $x8  -bxor (Salsa20-Rotate (Salsa20-Sum $x4  $x0)  9)
    $x12 = $x12 -bxor (Salsa20-Rotate (Salsa20-Sum $x8  $x4)  13)
    $x0  = $x0  -bxor (Salsa20-Rotate (Salsa20-Sum $x12 $x8)  18)
    $x9  = $x9  -bxor (Salsa20-Rotate (Salsa20-Sum $x5  $x1)  7)
    $x13 = $x13 -bxor (Salsa20-Rotate (Salsa20-Sum $x9  $x5)  9)
    $x1  = $x1  -bxor (Salsa20-Rotate (Salsa20-Sum $x13 $x9)  13)
    $x5  = $x5  -bxor (Salsa20-Rotate (Salsa20-Sum $x1  $x13) 18)
    $x14 = $x14 -bxor (Salsa20-Rotate (Salsa20-Sum $x10 $x6)  7)
    $x2  = $x2  -bxor (Salsa20-Rotate (Salsa20-Sum $x14 $x10) 9)
    $x6  = $x6  -bxor (Salsa20-Rotate (Salsa20-Sum $x2  $x14) 13)
    $x10 = $x10 -bxor (Salsa20-Rotate (Salsa20-Sum $x6  $x2)  18)
    $x3  = $x3  -bxor (Salsa20-Rotate (Salsa20-Sum $x15 $x11) 7)
    $x7  = $x7  -bxor (Salsa20-Rotate (Salsa20-Sum $x3  $x15) 9)
    $x11 = $x11 -bxor (Salsa20-Rotate (Salsa20-Sum $x7  $x3)  13)
    $x15 = $x15 -bxor (Salsa20-Rotate (Salsa20-Sum $x11 $x7)  18)
    $x1  = $x1  -bxor (Salsa20-Rotate (Salsa20-Sum $x0  $x3)  7)
    $x2  = $x2  -bxor (Salsa20-Rotate (Salsa20-Sum $x1  $x0)  9)
    $x3  = $x3  -bxor (Salsa20-Rotate (Salsa20-Sum $x2  $x1)  13)
    $x0  = $x0  -bxor (Salsa20-Rotate (Salsa20-Sum $x3  $x2)  18)
    $x6  = $x6  -bxor (Salsa20-Rotate (Salsa20-Sum $x5  $x4)  7)
    $x7  = $x7  -bxor (Salsa20-Rotate (Salsa20-Sum $x6  $x5)  9)
    $x4  = $x4  -bxor (Salsa20-Rotate (Salsa20-Sum $x7  $x6)  13)
    $x5  = $x5  -bxor (Salsa20-Rotate (Salsa20-Sum $x4  $x7)  18)
    $x11 = $x11 -bxor (Salsa20-Rotate (Salsa20-Sum $x10 $x9)  7)
    $x8  = $x8  -bxor (Salsa20-Rotate (Salsa20-Sum $x11 $x10) 9)
    $x9  = $x9  -bxor (Salsa20-Rotate (Salsa20-Sum $x8  $x11) 13)
    $x10 = $x10 -bxor (Salsa20-Rotate (Salsa20-Sum $x9  $x8)  18)
    $x12 = $x12 -bxor (Salsa20-Rotate (Salsa20-Sum $x15 $x14) 7)
    $x13 = $x13 -bxor (Salsa20-Rotate (Salsa20-Sum $x12 $x15) 9)
    $x14 = $x14 -bxor (Salsa20-Rotate (Salsa20-Sum $x13 $x12) 13)
    $x15 = $x15 -bxor (Salsa20-Rotate (Salsa20-Sum $x14 $x13) 18)
  }

  $x0  = Salsa20-Sum $x0 $j0
  $x1  = Salsa20-Sum $x1 $j1
  $x2  = Salsa20-Sum $x2 $j2
  $x3  = Salsa20-Sum $x3 $j3
  $x4  = Salsa20-Sum $x4 $j4
  $x5  = Salsa20-Sum $x5 $j5
  $x6  = Salsa20-Sum $x6 $j6
  $x7  = Salsa20-Sum $x7 $j7
  $x8  = Salsa20-Sum $x8 $j8
  $x9  = Salsa20-Sum $x9 $j9
  $x10 = Salsa20-Sum $x10 $j10
  $x11 = Salsa20-Sum $x11 $j11
  $x12 = Salsa20-Sum $x12 $j12
  $x13 = Salsa20-Sum $x13 $j13
  $x14 = Salsa20-Sum $x14 $j14
  $x15 = Salsa20-Sum $x15 $j15

  $out = @()
  $out += Salsa20-StoreLittleEndian $x0
  $out += Salsa20-StoreLittleEndian $x1
  $out += Salsa20-StoreLittleEndian $x2
  $out += Salsa20-StoreLittleEndian $x3
  $out += Salsa20-StoreLittleEndian $x4
  $out += Salsa20-StoreLittleEndian $x5
  $out += Salsa20-StoreLittleEndian $x6
  $out += Salsa20-StoreLittleEndian $x7
  $out += Salsa20-StoreLittleEndian $x8
  $out += Salsa20-StoreLittleEndian $x9
  $out += Salsa20-StoreLittleEndian $x10
  $out += Salsa20-StoreLittleEndian $x11
  $out += Salsa20-StoreLittleEndian $x12
  $out += Salsa20-StoreLittleEndian $x13
  $out += Salsa20-StoreLittleEndian $x14
  $out += Salsa20-StoreLittleEndian $x15

  Return $out
}

As I said earlier, the key to using Salsa20 as an encryption cipher is to XOR the message with the given output of a Salsa20 block: the input to the core function isn't the message, it's just a key and a nonce. To take a series of bytes and encrypt it with a provided encryption key, you could do something like this:

Function Salsa20-EncryptBlock {
  Param(
    [Parameter(Mandatory=$True,Position=0)]
    [ValidateNotNullOrEmpty()]
    [Byte[]] $Key,

    [Parameter(Mandatory=$True,Position=1)]
    [ValidateNotNullOrEmpty()]
    [Byte[]] $Message
  )

  $bytes = $Message.Length
  $n     = New-Object Byte[] 16
  $out   = @()

  while (0 -lt $bytes) {
    $output = Salsa20-Core $Key $n
    [UInt32] $nonce = Salsa20-LoadLittleEndian $n[8..11]
    for ($i = 0; $i -lt [Math]::Min($bytes, $BlockSize); $i++) {
      $out += [Byte]($Message[$nonce * $BlockSize + $i] -bxor $output[$i])
    }

    $nonce = Salsa20-Sum 1 $nonce
    $new_n = Salsa20-StoreLittleEndian $nonce
    for ($i = 0; $i -lt 4; $i++) { $n[8 + $i] = $new_n[$i] }

    $bytes -= $BlockSize
  }
  Return $out 
}

Be aware this method as shown has a much lower limit on the number of nonces that can be used (2^32) than the real cipher can allow (max 2^70). Implementing higher nonce support is left as an exercise for the reader. Other improvements are possible and I may share them later, but this is just a basic framework to get you started. (Not-so-subtle hint: nonces don't always have to start at zero. What does $n[0..7] do, anyway? Caveat: Powershell doesn't do "pointers", exactly.) Given the above Salsa20 Powershell definitions, you can encrypt just about any series of bytes you'd care to encrypt, and more quickly than AES, too:

$utf8       = New-Object System.Text.UTF8Encoding
$rnd        = New-Object System.Random
$key        = New-Object Byte[] 32
$plaintext  = $utf8.GetBytes("Hello world!")
$rnd.Nextbytes($key)

Write-Host ("key={0}" -f (($key|%{$_.ToString('x2')}) -join ' '))
Write-Host ("plaintext={0}" -f (($plaintext|%{$_.ToString('x2')}) -join ' '))

[Byte[]] $ciphertxt = Salsa20-EncryptBlock -Key $key -Message $plaintxt
Write-Host ("ciphertxt={0}" -f (($ciphertxt|%{$_.ToString('x2')}) -join ' '))

[Byte[]] $decrypted = Salsa20-EncryptBlock -Key $key -Message $ciphertxt
Write-Host ("decrypted={0}" -f (($decrypted|%{$_.ToString('x2')}) -join ' '))

And the output looks, for example, like so:

key=f8 34 5f a7 4e 93 cf 90 5a 3f 8d fc ad 5e 93 d9 d2 c2 f9 69 f7 42 b0 5d a6 f0 69 c8 ed 4c aa 82
plaintext=48 65 6c 6c 6f 20 77 6f 72 6c 64 21 # "Hello world!" in hex
ciphertxt=31 24 a4 d6 46 cb ad 03 92 ec e7 f8 # encrypted gibberish
decrypted=48 65 6c 6c 6f 20 77 6f 72 6c 64 21 # "Hello world!", decrypted

How much faster is Salsa20 than AES? Well if my "works, just barely" AES-in-Powershell code could do 4 bytes per second, this Salsa20 implementation does, by my "whatever I have laying around" measurements, about 461 bytes per second. That's still awful, but we're sticking to only-a-Powershell-script here, and it's still orders of magnitude faster than the other Powershell script. I call it a win.

It may not be RSA in 3 lines of Perl, but let's face it, sometimes they won't let you use Perl. Philistines. Wheresoever possible, use a serious implementation of a cipher you trust. But for anywhere that encryption is forbidden, make your own in the bathtub of the world.

I welcome cryptanalysis on this implementation. Though you should NOT consider this Salsa20 implementation to be serious cryptography, I've found that it meets the published results of the examples in the Salsa20 paper. Crypto is easy. Good crypto is hard. If there is an error in this code, obvious or subtle, I would like to know about it.

[0] They called it "The Great War", which it probably was, comparatively speaking. Adolf Hitler earned a medal during it and everything.

[1] Most CPU manufacturers have dedicated AES support in their hardware. But in a secure environment that won't even let you create System.Text.ASCIIEncoding objects, which is really just a mapping that says "A = 65, B = 66..." do you really think they'll let you use the hardware crypto APIs?

2017-09-27

A "review" of Jeff VanderMeer - Annihilation

Packed in, all eyes turned in
No one to see on the quay, no one waving for me
Just the shoreline receding
Ticket in my hand, I'm thinking "Wish I didn't hand it in"
—Okkervil River

Ten years ago, I came across a profoundly dark and disturbing novel called House of Leaves by Mark Z. Danielewski. While it is not a horror novel per se it is a dense and uncompromising meditation on, among other things, a harshly unflinching study of mental health and a fear of the unknown. It set a bar that is nearly impossible to surpass and remains the gold standard of English fiction when it comes to its imposing sense of sheer existential dread and forboding.

It owes this power to its completely holistic, intricate design. House of Leaves is a masterwork of layered concepts interwoven and arranged with the care and precision of a watchmaker or the army of designers behind a sophisticated new Apple iGadget. Yet on its surface, there remain still the basic materials, the building blocks of a masterpiece: a house, a hallway, a rocky marriage, competing character motivations. Then conflict. And questions.

Questions about the unknown. Perhaps the unknowable.

But this is not a review of House of Leaves.

Leaving behind all the faces that I might replace
If I tried on that long ride
Looking deep inside
But I don't want to look so deep inside yet

If you make the best novel of a generation, there are bound to be a bevy of young Turks angling for their shot to dethrone you, and Jeff VanderMeer's Annihilation is a very good first salvo. From the first sentences of his story you are given enough detail to begin to worry, but not enough detail to know about what, precisely, is worrying you.

The plot can be distilled to its building blocks. Four explorers, experts in their respective fields, in what is effectively a nature preserve. Their mission is to explore it. Some authors build rich, elaborate worlds for their characters. Some construct whole planets of spice and sand, or galaxies or universes or intertwined multiverses that grow and clash with unfathomably vast empires who cross over all of time and space. VanderMeer needs no such infinities to be able to terrify. His sandbox is a comparatively small patch of land and lakes, a thatch of wooded terrain, a tower, a lighthouse. It is enough.

If H. P. Lovecraft's elder gods need stars and space and ages and eons to fill you with dread, VanderMeer is his antithesis. He's a minimalist, with a collection of effective, potent storytelling tools. He is no less a master watchmaker, though his product requires fewer moving parts. Many authors now rely upon the trope of the unreliable narrator, the guide you must necessarily follow, even if they prove themselves not worth the trust. The narrator of Annihilation, a trained biologist, is not exactly unreliable. Rather, she admits to herself, and thus to you the reader, that she is flawed, finicky, and all too human. She justifies her thoughts about as much as any of us do, asking not for your sympathy or your understanding, but simply stating her opinion of the matter and how it influenced her. Though you may not come to her same conclusions, she is not impulsive or whimsical. Her decisions fit her in the time and place she makes them. She is not asking you to agree, she is simply documenting the situation the way any good scientist would.

We sing "Is that marionette real enough yet
To step off of that set
To decide what her hands might be doing, ruining
The play to, in the ensuing melee, escape?"

The building blocks of Annihilation, though of many of the same shapes, form a much different design than House of Leaves. There still remains a rocky marriage, competing character motivations. Then conflict and questions. Where other authors say "the dark, distant infinite and unknowable is terrifying", VanderMeer turns this around and always simplifies, simplifies, simplifies. "The most terrifying thing about the unknown," he seems to say with this book, "is if it's right next to you, possibly even surrounding you, in front of you, behind you, beneath your feet, waiting. Confined, contained, but with no comprehension among any of you what is keeping you safe, or for how long you can stay that way."


The tower is a main character of the book, though it exists as only a physical structure. Even as an inert, immutable construct, it clearly holds a complex and possibly sinister personality all its own, and it poses the enigma at the heart of Annihilation. These explorers are following in the literal and figurative footsteps of several previous expeditions that explored and studied the same territory. If those earlier expeditions' notes never mentioned the existence of this building, conspicuously located where it could not be overlooked, then how did it get there and what is its purpose?

The explorers clash. They do research, they disagree with each other, the team dynamic dissolves and the biologist is so quick to point this out that it broaches upon insult to her peers. The story itself is blazingly short given the weight of its omnipresent, suffocating dread. Questions are raised, answers are posited, and the ceaseless sense of being watched and manipulated occupy nearly every page, yet by the end you realize that you've been lockstep following the biologist's narrative, minus her intimately humanizing background exposition, for only about a week. Her journaling is a blend of observation and confession as she performs her duties and, privately, questions precisely whom she is serving and to what ends her exploration will satisfy. You can follow the scientific method in the midst of life-threatening danger, but do you really want to?

We sail out on orders from him but we find
The maps he sent to us don't mention lost coastlines
Where nothing we've actually seen has been mapped or outlined
We don't recognize the names upon these signs

Annihilation may not be exactly about the explorers, or the nature preserve, or the unknown, or the impossible. Perhaps it's about the temporary, transient nature of things, the cyclical pattern of natural events, even of reality itself. It may not always be possible to get answers to all of your questions, but you will compulsively ask them because it is core to your character to seek to understand.

To understand even when, in the face of impossibility against all one's categorical nomenclature, every phylum, class, and genus of the plant and animal kingdoms cannot enlighten you.

Read an author-annotated version of chapter 1 of Jeff VanderMeer's Annihilation here: https://genius.com/3147796

2017-09-04

The Laziest Cron Job on Windows

"You know that story about how NASA spent millions of dollars developing this pen that writes in zero G? Did you ever read that? And how Russia solved the problem?"

"Yeah, they used a pencil."

"Right, a normal wooden pencil. Just seems like Phillip takes the NASA route almost every time."

— Aaron and Abe don't have a stellar comprehension of history, Primer

A while ago I wanted to run something at a certain exact time when I wasn't going to be around to watch it. And it was a Windows box.

The proper way to schedule something on Windows is with a scheduled task.

For what I wanted to do, this was overkill. I just wanted to run a program at 4:15 PM that day and be done. So I wrote this:

[CmdletBinding()]
Param([DateTime] $At, [String] $Command)
$ts = $At - (Get-Date)
if (0 -gt $ts.Ticks) { Exit }
Write-Verbose ("sleeping until {0}" -f ($At))
Start-Sleep -Milliseconds $ts.TotalMilliseconds
try { Invoke-Expression -Command $Command }
catch { Write-Error $_ }

I saved it to a file, ".\do-at.ps1", and I ran it like so:

.\do-at.ps1 -?
do-at.ps1 [[-At] <datetime>] [[-Command] <string>] [<CommonParameters>]

.\do-at.ps1 -At "4:15 PM" -Command "calc.exe" -Verbose

When you just need a pencil, it's useful.

2017-08-06

Encrypted ZFS on Root for Devuan: ZFS with LUKS, without Systemd

Let's say that your hobbies include putting ZFS onto different things, but you have a deep-seated loathing of amateur init systems and their authors.

Let's say this problem is compounded by the fact that every major Linux distribution has adopted same said init system and it can be particularly thorny to remove it and replace it with something, anything sane. What else can you do but curse the darkness? All is lost!

No, my friends. We can light a candle. There's always a way. Sometimes a ridiculous way, but a way nonetheless.

A number of folks who still have their heads about them and understand that some cures are worse than the disease have forked Debian and made it mo' better. The Devuan Linux distribution is still very new at this point and recently hit its 1.0 release. Its goal is "software freedom", which is code for "no systemd", so it's perfect for our goals.

The following is a quick and dirty howto for setting up a Devuan system with encrypted ZFS on root. It is meant to be read and understood in the context of my other ZFS-on-root howtos and is not intended for a beginner ZFS (or Linux) enthusiast. We're taking a big step backwards here and doing some old-school compiling of DKMS modules.

I say "taking a big step backwards" because Canonical, the maker of Ubuntu Linux, took a big step forward by integrating the ZFS kernel module into their kernels from 16.04 onwards. But they also adopted systemd, which is Muy Malo, so we have to roll our own zfs.ko. Twice. But I am getting ahead of myself.

The following howto uses the Devuan Jessie 1.0.0 minimal Live CD ISO (devuan_jessie_1.0.0_amd64_minimal-live.iso), which lacks any desktop environments. We don't need them, we're just going to be typing our way to joy. Moreover, kernel module builds fail on the Devuan desktop live environment, so use the minimal live ISO.

Make sure your system has a drive you don't care about formatting, and at least 4GB of RAM. Attempting this with anything less will result in spurious errors that will impair your progress. ZFS itself enjoys a large memory footprint, so I don't suggest this for any resource-constrained, embedded setups. Proceed at your own risk. These steps come with no warranty whatsoever. Good luck.

I'm assuming that the root drive you're going to format is /dev/sda. Your actual mileage may vary.

Wipe and partition the disk.

wipefs --force --all /dev/sda
# or
dd if=/dev/zero of=/dev/sda bs=1M count=2
/sbin/parted --script --align optimal /dev/sda mklabel msdos
/sbin/parted --script --align optimal /dev/sda mkpart primary 1MiB 100%
/sbin/parted --script --align optimal /dev/sda set 1 boot on

Check that the partition table is correct:

/sbin/parted --script /dev/sda p

You'll notice that we're breaking with tradition of creating a separate partition for /boot and / partitions like we've previously done. Like I said, quick and dirty. I was surprised to see that this works, even under Debian/Devuan. At the risk of potential boot errors, this lets you take ZFS snapshots of your kernel and initrds, which may be really useful to you after a disaster.

Log into the Devuan Live CD. Be sure to understand your network configuration. Linux networking is a Kafkaesque nightmare of conflicting system config file advice and is beyond the scope of this howto. For simplicity, in this howto we assume your machine uses DHCP to get online. Use a real setup for a real server.

dhclient eth0

Add the jessie-backports packages to your apt/apt-get sources.

echo 'deb http://auto.mirror.devuan.org/merged jessie-backports main contrib' > /etc/apt/sources.list.d/backports.list

Update the package sources and install cryptsetup.

time apt update
time apt install -y cryptsetup

Setup your partition into a LUKS container.

cryptsetup luksFormat -h sha512 /dev/sda1
cryptsetup luksOpen /dev/sda1 cryptroot

ZFS on Devuan requires compiling multiple kernel modules and is painful. Canonical took this pain away and yet forces systemd upon us. Curses. Curses I say.

Start by adding the headers for the Live CD kernel.

time apt install -y linux-headers-$(uname -r)

Install the ZFS DKMS package and dependencies. This will pop up a license incompatibility warning that you have to acknowledge. Optionally, you can force apt-get to skip this warning by setting an environment variable. To skip the interactive ZFS license warning screen:

DEBIAN_FRONTEND=noninteractive
export DEBIAN_FRONTEND

Build ZFS.

time apt-get install -y -t jessie-backports zfs-dkms

A quick word about apt versus apt-get. Both work equally well for our purposes. Use whichever you prefer, but be aware that you will need to specify certain exact packages from certain exact repositories. "apt-get install -t reponame package" is the same to us as "apt install package/reponame". I usually choose the command that contains the fewest number of characters.

Keep an eye on the output of this command and watch for any errors building the modules. You'll be prompted to check /var/lib/dkms/something/build/make.log to troubleshoot. Common errors include transient resource constraint problems: running out of memory, running out of disk space on the Live CD file system, et cetera. If you encounter an error, stop. Fix your system. Do not proceed until your setup is entirely correct.

When the zfs-dkms package is installed, you can add the ZFS kernel module to your Live CD setup.

/sbin/modprobe zfs

If you get any errors, stop here and correct them.

Setup your zpool and datasets. This will look familiar if you've read my other ZFS howtos.

zpool create -f -O mountpoint=none -O compression=lz4 -O atime=off -o ashift=12 zdevuan /dev/mapper/cryptroot
zfs create -o mountpoint=/ zdevuan/root
zpool set bootfs=zdevuan/root zdevuan

Optionally, you can create separate datasets for /boot, /home, and so forth. I like to create an unmounted "zpool/root" dataset and then children datasets for my various boot environments (example: "zpool/root/cur", "zpool/root/next", and so on). Theoretically, this helps you rollback bad upgrades without too much swearing. To add a /boot dataset for example:

zfs create -o mountpoint=/boot zdevuan/boot

Create your desired datasets. Unmount the zpool and remount it under /mnt.

zpool export -a
zpool import -R /mnt zdevuan

Since I never bothered looking for where Devuan install discs keep their packages or base image (I think they're under /lib/live. That's what mount suggests.), we're going to set up our new system with debootstrap:

time /usr/sbin/debootstrap jessie /mnt https://auto.mirror.devuan.org/merged

Whatever you do, don't use the --no-check-gpg flag to make debootstrap run faster.

When debootstrap finishes setting up the base system under /mnt, add a few files from your Live CD to the new system:

cp -v -p /etc/apt/sources.list.d/backports.lists /mnt/etc/apt/sources.list.d
cp -v -p /etc/locale.gen /mnt/etc

Before the new system solidifies, it throws errors about missing locales and things of that nature, so I like to use the Live CD environment to edit the new system's config files. Edit /mnt/etc/fstab to the following:

/dev/mapper/cryptroot /     zfs defaults,noatime 0 0
zdevuan/boot          /boot zfs defaults,noatime 0 0

The line that will mount /boot is superfluous if you don't change the zdevuan/boot dataset's mountpoint value to "legacy", but don't skip ahead just yet.

Edit /mnt/etc/crypttab to add the UUID of the cryptroot LUKS container. There are multiple ways to get the UUID. You can run ls -l /dev/disk/by-uuid or you can run blkid. You have options.

blkid /dev/sda1 # get the UUID for /dev/sda1
echo 'cryptroot UUID=UUIDHERE /rootkey.bin luks,keyscript=/bin/cat' > /mnt/etc/crypttab

Create a key for the cryptroot LUKS container. I've written about this previously and people have Strong Opinions about PRNGs. Use whatever makes you feel better. If your entropy pool is taking its own sweet time, you can run this in a tmux or GNU screen window and switch to another window to keep working while dd keeps crunching.

time dd if=/dev/random iflag=fullblock of=/mnt/boot/rootkey.bin bs=512 count=4

You can also use OpenSSL, if you trust OpenSSL. In my testing, OpenSSL can sometimes yield slightly, slightly less probabilistically-indistinguishable-from-noise pseudorandom output than /dev/random. It's very close and significantly faster.

openssl rand -out /mnt/boot/rootkey.bin 2048

Set the hostname.

echo myhostname > /mnt/etc/hostname
echo '127.0.0.1 myhostname' >> /mnt/etc/hosts

Set the networking configuration for your new machine. This can be very complicated, so for this howto we continue to assume we're just going to use DHCP.

vi /mnt/etc/network/interfaces

Add these lines:

auto eth0
iface eth0 inet dhcp

Mount system directories.

cd /
for i in /dev /dev/pts /proc /sys; do mount -B $i /mnt$i; done

Chroot to the new system. Yes, we're going to re-download and install ZFS again.

chroot /mnt /bin/bash --login

Do some system setup.

passwd
passwd -u root
dpkg-reconfigure tzdata
ln -sf /proc/self/mounts /etc/mtab

Add some packages that debootstrap skips. You can also use --include in the debootstrap, but I don't care to make that process any more complicated than it already is. This howto assumes you're running this on an amd64 system. Adjust the "linux-image-<arch>" package accordingly for whatever your intended arch is.

apt update
apt install -y locales kbd # select your locale here
time apt install -y cryptsetup linux-image-amd64
time apt install -y linux-headers-$(uname -r) # Again
time apt-get install -y -t jessie-backports zfs-dkms zfs-initramfs

Hopefully by now the LUKS key has finished getting written. Create a crypto keyfile hook for the initramfs and add the key to the LUKS container.

cryptsetup luksAddKey /dev/sda1 /boot/rootkey.bin
chmod 0 /boot/rootkey.bin
vi /etc/initramfs-tools/hooks/crypto_keyfile
# Add these lines:
#!/bin/sh
cp -p /boot/rootkey.bin "${DESTDIR}"

Set the script to be executable.

chmod +x /etc/initramfs-tools/hooks/crypto_keyfile

OK, almost there. Symlink the cryptroot device into /dev. We're skipping the udev rule to automate this because so help me, we're never going to update our kernel on this machine ever again. See my other howtos if you want information on how to add a udev rule for this.

ln -sf /dev/mapper/cryptroot /dev

A dirty little secret: Devuan Jessie doesn't have a ZFS-compatible bootloader. We're going to steal the package we need from the Devuan testing branch, confusingly called "Ascii".

cp -v -p /etc/apt/sources.list /etc/apt/sources.list.orig
echo 'deb http://auto.mirror.devuan.org/merged ascii main' >> /etc/apt/sources.list
apt update
time apt install -y grub-pc/ascii

Remove the Ascii repo.

mv /etc/apt/sources.list.orig /etc/apt/sources.list

Edit your GRUB config.

vi /etc/default/grub

Make the following changes to the grub file:

-GRUB_CMDLINE_LINUX_DEFAULT="quiet"
-GRUB_CMDLINE_LINUX=""
+GRUB_CMDLINE_LINUX_DEFAULT="boot=zfs"
+GRUB_CMDLINE_LINUX="cryptdevice=UUID=UUIDHERE:cryptroot"
+GRUB_ENABLE_CRYPTODISK=y

Run through all the bootloader and ramdisk setup steps to make sure everything is up-to-date:

update-initramfs -u -k all
update-grub
grub-install /dev/sda

If grub-install completes without errors, you're ready to unmount everything and restart the system. Now is your last chance to add groups, users, configure /etc/sudoers, or do any other last-minute system prep before the system goes live. When you're ready, quit the chroot.

exit

Set the mountpoint for the /boot dataset from /boot to "legacy". It will be mounted at boot time from the line in /etc/fstab that we thought we wouldn't need.

zfs set mountpoint=legacy zdevuan/boot

Note: The /dev and /proc mountpoints are going to be locked until you kill the irqbalance process that the kernel image package added. I don't know why this hangs around, but you need to cleanly unmount the zpool in order to get a clean first-time boot. You won't lose data, but if you skip this step you'll be forced to manually import the zpool from single-user mode.

killall irqbalance
umount /mnt/sys
umount /mnt/proc
umount /mnt/dev/pts
umount /mnt/dev
zfs umount -a
zpool export -a
halt -p

If you cannot unmount any of the mounts at this point, run lsof | grep /mnt/whatever to see which process is using the mount point. Kill or killall the offending process.

Start the machine. Eject the Devuan minimal Live CD media and boot from disk. You should be set. Test it out, make your snapshots, and start using an encrypted ZFS-root, systemd-free Linux server.

External References and Resources

https://obviate.io/2008/12/04/truly-non-interactive-unattended-apt-get-install/

http://www.thecrosseroads.net/2016/02/booting-a-zfs-root-via-uefi-on-debian

https://github.com/zfsonlinux/zfs/wiki/Debian-Jessie-Root-on-ZFS

https://github.com/zfsonlinux/zfs/wiki/Debian

https://www.debian.org/releases/stable/i386/apds03.html