← all posts site ↗

the reboot that swallowed my server

one morning everything on my server broke at once and the pattern took a while to see. ssl errors on every proxied service, dns queries dying, the dashboard dead. the server had rebooted at nine in the morning and docker started its containers before the data disk finished mounting. every container that bind mounts that disk got attached to empty phantom directories on the root drive, so the reverse proxy had no configs and no certs, the dns box had no config at all, and the real data sat untouched on the properly mounted disk while every running service saw nothing.

the give away was that a container's config directory resolved to a different filesystem than the host path. recreating the stacks made the bind mounts re resolve and everything came back, https serving again and dns answering. then the deeper problem surfaced, my morning check reported fine that day because it verified endpoints, not container states. two stacks were already dead at boot and it had no idea. that felt like a personal failure because it was one.

so the check got rebuilt to look at actual container health, and the moment it did it started seeing what it had been missing. it also got an auto repair layer, down containers get recreated or restarted, orphan monitors that point at nothing get deleted, docker junk and journal logs get pruned, and the network stack heals itself, restarting the tunnel or the dns container or renewing certs before they expire.

the permanent fix for the original blast was a boot dependency so docker waits for the data disk to mount before it touches anything, which kills the entire class of failure. then all the separate maintenance jobs got merged into a single daily one that stays silent when it fixes nothing and only nags when something stays broken. the scary part is a reboot you did not schedule can turn your whole server into a zombie and the check still says green.