Lorien's Back!

Troth mailing lists have been down since the move; Dave Haxton (the new hosting administrator) and I are working hard on a solution.

However, investigating that led me to upgrade lorien to a new version of its operating system (Debian Linux), and that, in turn, required that I reboot the server. That's not bad. The bad bit is that it's in a reboot loop. This should be fixable with little work, but I thought I'd mention it.

Non-Geeky Update: I know why it's broken, but I'm tired enough that I might do something stupid while trying to grok how to fix it. I respect your data, and mine, and its integrity more than that! So, I'm going to sleep so I can tackle this when I'm awake enough to get my head around this vastly changed thing.

Non-Geeky Resolution: Ah, sleep, what knits up the ravelled sleeve of care! Once I'd slept on it, it was only an hour to successfully handle the problem. All lorien functions, including webmail, back online. Enjoy! Troth e-mail lists are still down, so unfortunately you can't enjoy those just now...

Geeky Update: The way RAID is handled changed considerably between woody (old Debian) and sarge (new Debian). All my partitions are RAID-1 arrays (mirrored partitions), including / and /boot, so an inability to deal with RAID means an inability to deal with life, hence boot loop. I debated stripping out all RAID information and bringing things up, then rebuilding the RAIDs, but I think any tango with mdadm is best done after sleep. Nothing whatsoever was aided by the fact that I suspected it was RAID going fubar at the get-go, but it took me two hours to cobble together bootable Debian media with which to mount a rescue attempt (no working floppy drives! joy! but the bootable cd that I thought you couldn't burn on OS X worked, yay).

Geeky Resolution: Well, once I'd slept on it, I threw myself at the brick wall of mdadm a few more times. Then, I realised that everything would go so much better if I made sure the md (metadisk/RAID) module was loaded -- it wasn't. Funny thing how it's easier to detect raids once your kernel knows what one is! Then I mounted all my meta-devices, chrooted into /mnt, ran a lilo, and lo, was Bob then made my uncle. No mdadm necessary, the old utilities still do the right thing, but lilo was one of the things upgraded, so bootblocks needed to be updated -- that was the real problem.

All services on lorien, my server, which includes many websites and all their e-mail, are currently unavailable. Believe me, I'm as annoyed as all of you... more so by the fact that I find myself having to sleep rather than having enough adrenaline enough to FIX IT.

We're back!

-- Lorrie

Mail Delivery Restored, Spam Filtering Restored

Update: Everything should be fine now.
Geek: The SPF policy server didn't come back, but it wasn't doing much good that I could see anyway.

Non-Geek: Mail delivery was down for awhile after the successful boot. It's back up now, so the MAIL WILL FLOW, but you won't get most of my serverside spam and virus filtering for a couple hours while I rebuild some infrastructure.

Geek: The new Debian did... something... that overran my CPAN and its lacy network of modules. Without perl modules, postgrey (greylisting server) and amavis (content filter: calls spamassassin for spam filtering & clamav for anti-virus fu) won't work. We're still doing the RBL and bogus-header tests, as those are internal to postfix, but greylisting and amavis/spamassassin/clamAV are all down until that's rebuilt.

-- Lorrie