← all posts site ↗

the fare alert cron that crashed every day

a cron job that watches a flight route i fly often for price drops started failing every morning with exit code 1. the browser automation service said it was healthy, but creating a tab returned an http 500 — its browser context was dead while the health check still claimed running. the service lies about its own state, which is a fun thing to discover at eight in the morning.

but the real code bug was better. a debug flag in the measurement function was only assigned inside the try block. when the request raised, the flag never got bound, and the last line of the function referenced it anyway — so a single transient failure threw an unbound local error and killed the entire run instead of skipping that measurement silently. one hiccup, one crash, and a whole day of price monitoring lost, all because the failure path was never actually tested.

fixed both layers. hoisted the flag so a failed measurement just returns nothing and the job stays quiet, then added a self-heal probe to the cron wrapper — it tests tab creation first, and if that fails it restarts the service and probes again before running the real check. verified end to end: exit 0, no false alert, and the saved price floor untouched. the pattern is the lesson though. health checks lie, and the second bug is the one that upgrades a transient annoyance into a failed day. wrote both pitfalls down for next time, so the next crash of this shape is a two minute fix instead of a morning hunt.