← all posts site ↗

my local model was not slow, the frontend was

went looking for an uncensored model that fits a sixteen gig amd card and ended up in a speed investigation. the model pick itself settled fast: a hybrid attention nine billion model that leaves most of the card for context, because the bigger dense options spill and die on long tool histories. and one hard number: a one million token context is physically impossible on sixteen gig. the key value cache alone needs over thirty gig regardless of model, so nobody gets a usable million token window on this card. two fifty six thousand is the practical ceiling.

then the mystery. the same model, same gpu, around forty tokens a second either way, but one interface answered hi in 0.36 seconds and another felt like several seconds. the gpu was never the bottleneck. the logs showed the slow frontend stuffs a three thousand token system prompt plus a long pile of tool schemas into every single request, then fires background calls on top: auto title generation, suggested follow ups, auto tagging. a single hi quietly turned into about seventy seconds of hidden generation, serialized on the same model. the thinking stream made it worse by generating a verbose thousand token reasoning monologue on every structured prompt.

fixes, in order of impact. kill the injected tool set, it mostly costs time on a plain local backend. turn off the auto title, tagging, and follow up features, that removes the hidden background burn entirely. optionally disable thinking at the server so the small model stops writing essays before answering. keep context honest at what you actually use. after that the round trip matches the lean interface at sub second, because the harness was never the problem. sometimes the wrapper is.