A Crash and a CARP

Yesterday, my private server crashed, after 258 loyal days of uptime. Running out of disk space on /var/log a few days prior probably had something to do with it, though obviously not right away. I'm taking advantage of the clean slate of a fresh reboot to do some light house-keeping, like patching OpenSSH and, uh, I guess that's about it. Given the extreme age of the server, compiling anything bigger than a "Hello world" application takes at least an hour.

Dealing with a kernel panic always makes me think about how I would redesign my network to route around them. It's not a problem for, say, e-mail service, which is designed to cope with failure: multiple MX records can point to multiple hosts, and every decent MTA queues mail for a long period of time, so that even if you're just running on a single host, downtime of a few hours will not bounce the message.

But what about services that demand to be more responsive? Things like HTTP and FTP need another approach, even if it's a simple round-robin setup of A records. Bottom line: redundancy of services demands redundant hosts.

This is why I'm happy that I'd already pre-ordered OpenBSD 3.6, which supports the cryptographically sound Common Address Redundancy Protocol. Two machines running CARP can transparently keep at least one service up and running between them, even when one of the machines goes down. There's an article that explains this in more detail. To wit:

"CARP is a multicast protocol. It groups several physical systems together under one or more virtual addresses. Of these, one system is the master and responds to all packets destined for the group (not necessarily all packets destined for the IP address, when in a load-balancing environment). The other systems act as hot spares. No matter what the IP and MAC address of the local physical interface, packets sent to the CARP address are returned with the CARP information."

Very useful.