baby's first outage

I had my first unplanned outage today, down for 16 minutes after just shy of my first 40 days. I finished a counseling cycle earlier this evening, beginning a month or so before I left work. I started writing here in part as a side effect of that therapy, and thought to check my uptime as a sort of celebratory demarcation. I was excited to see 39 days and some hours, but didn't stop to write about it then. Instead I felt compelled to dive into firewall configuration, which turned into a bigger bite than I intended to take.

I read and planned and felt confident in what I'd assembled, but I oops'd. I started with iptables -P INPUT DROP, which effectively killed my current session and prevented a new one. I don't have any users with passwords, so console access wasn't helpful. I rebooted, then waited for ping to reveal the server was back online. I never received a reply, and attempts to ssh and refresh in the browser were failing. I worried that the OS had unbeknownst to me automagically persisted that one rule and that I'd have to reinstall.

I finally thought to check the console and found that startup failed in the bootloader - symbol 'grub_calloc' not found. I tried manually booting in the grub recovery shell but most commands failed with the same error. I ran a full system upgrade back when I first provisioned the host but never rebooted after, so my guess is that the grub package was updated but some on-disk state wasn't. I was fortunately able to reboot into a temporary recovery host, which let me grub-install manually per the Internet's advice. One final reboot and I was back online, 41 days since my first day in between jobs.

# last reboot
reboot   system boot  x  Wed Aug 31 21:47   still running
reboot   system boot  x  Sat Jul 23 09:33 - 21:31 (39+11:57)

That drops me to three 9's now, and I'd have to be up without interruption for another 80 days before reaching four. Had I only taken five minutes to correct the issue I'd still have four, and at 10 minutes it'd only take another 40 days to get there. I've been wanting a second host since before I even had the one, so that might be another treat to commemorate the occasion.