← all posts site ↗

hunting uncensored local models, wrong gpu first

the ask was local chat models that do not quietly refuse half of the requests, so the hunt hit abliterated work, models retrained with the safety filtering removed. the first round came back with a hardware answer that was nonsense, it probed some other card on some other machine and declared that even a nine billion model would have to offload to the cpu. the actual desktop holds sixteen gigabytes of video memory on a current card, so the sizing is a completely different game: a nine billion q4 is about seven gigabytes and sits fully on the gpu.

the bigger ones are the real choice. a twenty seven billion at q4 is over half of the device, but tighter quants like iq4-xs and iq3 slide under the ceiling. the family with the deepest abliterated bench is the qwen line, both the aggressive and the heretic one, plus the 26b shells with the big active-parameter wins that fit at the small quant. the winning difference is quant, always.

the lesson is boring and durable: profile the real machine in the session, or every sizing computation is built on another computer. the quant is the whole answer, and the moment you take it for granted the math goes quiet.