Firewall Failover with pfsync and CARP

Coming from a non-profit background, I am routinely told to do an (x + 1) job with (x - 1) dollars. The +2 difference is usually filled in with creativity and garmonbozia. This "never buy anything over $1000" requirement is not always pleasant but it is, by definition, a cheap way to get things done. More to the point, it reinforces a certain mindset: that you can spend a lot of money to buy what you want, or you can carefully cobble a comparable solution together from smaller, older, cheaper components. If you're a Fortune 500 company, you'd probably want to invest a few tens of thousands of dollars into a fault-tolerant high availability z-Series server or six. These things are skinny and sleek and if you ever have a problem, IBM will be happy to send a technician out to read the little blinkenlights and tell you what's wrong.

On the other hand, you can save a bundle by using a greater number of tinier machines that work together to provide a similar level of availability. I've read that Google has 100,000 servers, so many that they don't even bother replacing hard drives when they fail: they just note which ones are bad and route around them until they decide to replace a bunch of them en masse. So instead of putting all their eggs in one really, really good basket, Google has eggs that go around a giant collection of 100,000 baskets and just don't go into the ones that look dodgy.

My point in all of this is that I can spend money on a big shiny firewall appliance and pray that it's still under warranty should it ever lock up (and that I'm awake and in the office at the time), or I can take 2 old Windows 95 boxes you were never going to use again and string 'em together:

Firewall Failover with pfsync and CARP

My favorite line from the article: "It's possible to run the pfsync protocol on one of the 'real' networks, but because of the security risks, it is strongly recommended that a dedicated, trusted network be used for pfsync. This can be as simple as a crossover cable between interfaces on two firewalls."

I don't have firewall availability problems at home (at work is another matter entirely): I use PF and my three workstations give it no problems. I have used it for a couple of years now and I have never not once experienced a crash of any kind: software or hardware. And were it to ever crash, I'd lose maybe a half a day of connectivity: that's not even enough for e-mail to bounce, and I'm pretty sure my BitTorrent uploads and instant messaging client would survive.

Nonetheless, if I ever do string two machines together to build a CARP-enabled redundant firewall cluster running PF and pfsync, I will have to make certain that each unit contains 3 network cards: one for the external interface, one for the internal interface, and one exclusively for internal firewall communication.

Now, a crossover cable is just hunky-dorey when you only have two firewalls, but if you have 3 or more, you're looking at a slightly more complicated setup involving a hub or something. Most hubs have at least 4 ports, so you can easily get away with having 4 units in your firewall cluster if you use the cheapest hub you can find.

What kind of network you'd have that could potentially require 4 separate firewalls keeping an eye on each other is another matter entirely. One firewall device is plenty for most people and most companies. I imagine two devices is enough even for a couple of the big boys.