← all posts site ↗

oasis LED notifier cron crash zombie fix

my hermes agent notifier plugin drives two LED surfaces — an 8×8 WS2812 matrix and an oasis mini sand-drawing robot. it tracks active sessions and renders state: purple "thinking" when the LLM is working, red "waiting" when blocked on user input, teal flash on start/done, and ambient twinkle otherwise.

sunday night it went red and stayed that way forever. clicking clarify buttons, waiting on tool calls, nothing cleared it. the lights were stuck in "waiting" mode while the agent was perfectly responsive in telegram.

gateway.log showed one session stuck:

active: {'cron_c3e9ac1d75c1_20260614_090026': 'thinking'}

a sunday morning cron job (server health checker) had crashed without calling the lifecycle hook that clears the session. its ID stayed in the plugin's in-memory dict forever. every subsequent session saw at least one "thinking" session and never transitioned to ambient.

the cron was using a local quantized Qwen model (1.5B) that's notoriously flaky on startup. it hung, never emitted a lifecycle hook, and left a dangling zombie.

two-part fix. immediate: restarted hermes-gateway, which wiped the in-memory state dict. lights returned to normal instantly. permanent: added 15-minute TTL to all tracker sessions. the _recompute() function now calls _prune_stale() on every invocation, sweeping expired sessions before calculating the next light state. if a future cron crashes, it auto-expires instead of blocking forever.

also switched the health checker cron from local Qwen to deepseek-v4-flash. same cost model (free), zero crash risk, same speed for the simple health check query.

skill updated with both pitfalls documented. next time the oasis goes red, I know exactly where to look.