Massive Filesystem Corruption
At about 10am this morning, the power supply on the main server failed. This has happened once before, and to be honest the novelty of fitting a not-quite-the-same power supply into the case was wearing a bit thin. I did it nonetheless, using an old supply from a 386 that had been hanging around just waiting to be made useful again.

With the new power-supply in place, it was discovered that only two of the three disks were responding. Hmm, very odd. Locate faulty disk, remove, check over...

Oh my. Chips on the drive controller aren't supposed to bubble like that, are they? And boy does it smell bad! That's okay, that drive was just /var, and I can live without my log files and proxy cache. After all, that's why I've made backups of the other two disks but not this one.

Except that /var contains the all-important /var/spool/mail, /var/cvs, Debian package information, and some databases. A bit of an ouch there.

It was only later that I discovered that I had my last six years worth of mail also on said hard-drive, since it had more space than the other disks. That was the crushing blow.

As it happens, the blowing power-supply also managed to cause damage to the motherboard/CPU/RAM as well, since init would segfault upon startup. Changing the Mobo/CPU/RAM fixed the problem. I'll be testing sticks of memory to see which ones can be recovered.

Moral of the story, backup everything, even the stuff you don't care about, in case there's some stuff you do care about in there as well.

It looks like the drive's controller is dead, but the platters and mechanism inside are still okay. Tried replacing the controller with a similar model from a drive that has busted insides, but to no avail. It *tries* to spin the drive up and all, but is obviously expecting something else. I think if I can find an identical controller (old-ish Qantum fireball) I should be fine. I'm hoping that one of these professional data recovery joints has a big box of them.