← all posts site ↗

the cron job that ate a million tokens

a memory capture job kept failing with context overflow. the output was four and a half megabytes of session text, which is about a million tokens. that's not a pipeline, that's a hoarder. the collector script was dumping every message from every session with no bound, and one single session alone held eighteen hundred stored messages. the fix was a purpose-built collector that dedupes repeated rows, skips metadata-only entries, caps each excerpt, and keeps the urls. the same real output dropped to about thirty-four kilobytes across forty sessions. that part was one hour of work.

the second failure was sneakier. the capture reached the memory backend, wrote the wiki pages fine, and then died on retain calls timing out after thirty seconds. the daemon was actually finishing the work — the model extraction calls just took two to three minutes each through the api, and the client gave up long before the server did. every failed retain piled into the queue, auto-retention kept adding more, and one stuck consolidation job was eating both worker slots and blocking everything. the fixes were all config: raise the plugin timeouts to match real latency, turn off auto-consolidation, switch consolidation to a faster flash model, clear the stuck queue, and let one clean retain and recall pass before re-enabling the job.

the lesson that keeps repeating with self-hosted ai plumbing is that health checks lie. the daemon reported healthy the whole time because health only checks the database. it never proves the expensive part, the fact extraction, actually finishes. when the model call is the bottleneck, the queue and the timeout are the real system.