I'm queuing like a Brit!

"'Cause it hit me that Montana was really just a leg
And just like that it all fell into place
And I don't think I could stand here any longer in this bed
Since I figured out Montana is a leg.
A leg.
Now I get it."
— John Linnell, "Montana"

A few days ago, I wrote (but never posted) a short entry about how I didn't "get" packet queuing. Which was true. I wrote how hard it was for me to make the great leap from one man's ability to prioritize TCP ACK packets to my ultimate goal of optimizing my downloads to increase throughput. Of course, it seems so simple now. But I had to first understand the details before I could be content in crafting an implementation.

The pf queue in question looks like this:

altq on $ext_if priq bandwidth 100Kb queue { q_pri, q_def }
queue q_pri priority 7
queue q_def priority 1 priq(default)

Deceptively simple, no? The queue is composed of two sets: q_pri is the priority queue and q_def is the default queue. In supermarket checkout terms, q_def is full of mothers buying a month's worth of groceries for the whole family and q_pri is the 12 items or less^* register meant for bachelors who just want to buy deodorant and a gallon of milk and get the hell out of there.

[^* by which I mean "fewer"]

The problem as it existed to me wasn't in the queue's creation. It was in the following two lines that utilized it:

pass out on $ext_if proto tcp from $ext_if to any flags S/SA \
keep state queue (q_def, q_pri)

pass in  on $ext_if proto tcp from any to $ext_if flags S/SA \
keep state queue (q_def, q_pri)

Just what the hell is going on here? It took me months to figure it out. And when I did it hit me like a punch to the solar plexis. Daniel Hartmeier, author of pf and its first documented implementor of ACK prioritization, wasn't trying to filter his packets. I had spent all of this time in vain tearing my proverbial hair out trying to marry fast ACKs with filtered packets. Clearly...clearly just tacking those two pass in, pass out rules into your pf.conf and calling it a day is insufficient. That much I knew.

But how the hell do you filter and queue simultaneously? A long time ago, I remember thinking to myself "Boy. I hope I don't end up putting 'keep state queue' at the end of every one of my pass rules." At the time, I had quite a few pass rules and my pf.conf was an enormous masterwork of security wrapped around my network topology. I may post it for posterity sometime. It was glorious. But I had no wish to go through and change every rule by hand.

How naïve I was back then. And how close I was to the truth. The truth being that I didn't have to set "keep state queue" for every rule. Just the ones where download speed is a priority: HTTP, for example. And SSH. And BitTorrent. That's only three, and there isn't much else I do that needs a boost.

The magic of fast ACKs is no more complicated than changing this:

pass out quick on $ext_if inet proto tcp from any to any port 80 \
keep state

to this:

pass out quick on $ext_if inet proto tcp from any to any port 80 \
flags S/SA keep state queue (q_def, q_pri)

Eventually, you get used to seeing one rule take up two lines. Small price to pay for benefits that border on the extraordinary: I've upped my rsync-over-SSH transfer rate by 10 KB/s and nearly doubled my BitTorrent upload rate. So of course the trick isn't to prioritize ACKs and then filter them, as I erroneously believed for so long. The trick is to filter first and prioritize along the way. Now I get it.