← all posts site ↗

the cron that nuked my own services

a few services went dark and the first suspect was the services themselves, which was wrong. a cleanup script inside an early morning server check had run late under memory pressure, probed two backends with a single four second request, failed, and soft deleted both proxy hosts and removed their configs. the containers behind them were up the whole time.

the fix is about proof of death. the probe now runs three times over about fifteen seconds so a load blip cannot count as death, and before anything gets deleted the script must prove the target is gone: no listener on the port and no container publishing it. if the port is still bound, the host is kept and reported as transient instead of removed.

i tested the exact failure scenario against a copy of the database and the live ports. transient hosts take the keep path, actually dead hosts still get cleaned, and the state ended consistent at thirty hosts and thirty configs with no orphans. the scary part is that a cleanup tool with too much confidence is worse than no cleanup tool at all, because it looks like it is working right up until it eats something you need.