context search on my photo library started erroring out of nowhere. first instinct was the database or the index, but both were fine. the log told a different story. the machine learning container had pinned nearly all of its four gig gpu: 4036 of 4096 megabytes used, zero percent utilization, no visible process. when a search tried to create a cuBLAS handle for clip text embedding it hit a hard resource allocation failure. nothing left to give.
root cause turned out to be by design, not a leak. on boot the container preloads six onnx models into vram and pins them there forever: clip text and image encoders for search, face detection and recognition, and two ocr models. the log literally says in-memory cache with unloading disabled. once resident they stay resident, so the whole stack sits at the ceiling on a small card even between jobs, and any request that needs one more context allocation dies. a restart reclaims it in seconds and models reload on demand, first search slow for a moment then instantly warm.
the fix i kept is a conditional restart. every six hours a script checks vram inside the container, and only if usage crosses ninety percent does it recycle, staggered away from backups. models reload in about two seconds at boot so search stays pinned and instant, and it never interrupts when things are healthy. honest caveat: this is a band aid, not a cure. a four gig card is too small for the full model stack. the real options are dropping ocr to free the two biggest chunks, or a bigger gpu. separately, the web ui showed blank tiles but still scrolled, which was the client's lazy loading deadlocking after it loaded during the crash. a page reload fixed it. that one was never the server.