← all posts site ↗

the build children that froze the whole box

the box froze twice in one afternoon and i had to hit the power button both times. the first suspect was the usual crud, docker, memory, a service crash, but the logs told a different story. what actually killed it was the work i had started myself. a llama cpp cuda compile chewing eight cores while a tauri build researched crates in parallel, plus a crawl service that had no reason to be polling at all. the agents did not fight each other, they just added up to a frozen system. i killed every background build child and the thing settled.

then i did what i should have done months ago, a full audit of every service, watchdog, and timer on the box, and asked the only question that matters for each one. what does this actually do at boot. the wake word model trainer was alive since boot as a forever service and i have not trained a model with it in weeks, gone. the gpu flip service looked important but the compose file already pins the nvidia agent image, and the running container already sees the gpu from inside, so the flip was just a thirteen second force recreate of an already correct container, gone. the old uptime monitor stack got deleted entirely, including the root owned data directory, via a throwaway container so i did not even need sudo.

then i reconciled every running container against the new monitor. three real services got added because they had no probe at all, and one dead monitor got removed because its service died weeks ago and it had been failing a hundred and ten out of a hundred and ten checks. the dashboard now shows thirty three endpoints with zero failures, and the only outage blips left line up exactly with my two manual reboots. the rule going forward is simple. a service earns its boot slot or it does not get one.