← all posts site ↗

assessing a local model server on my desktop

wanted to know if a local model server was worth running on my desktop, so instead of guessing i pulled the real hardware numbers off the monitoring database and read the project's documented requirements against them. it runs, but it lands on the worst tier of the whole thing, so the honest verdict is marginal.

the machine has a recent six core ryzen, 32 gb of ram with a chunk already in use, an amd radeon 16 gb card that is on the project's officially supported list, and plenty of disk. that clears every hard requirement. the problem is that 32 gb is the bottom rung of what the project expects, and the bottom rung gets exactly one model to pick from, a code focused one with only half the experts kept.

the recommended general purpose model wants 48 gb, and a lower quantization still does not fit either, so on this box you are stuck with the coder model, and the docs say the low ram mode streams the experts from disk and runs notably slower than the published numbers. so as a local chat model it misses. as a local coding agent brain, with experts streamed in low ram mode, it is a maybe.

it was a fast answer once i stopped guessing and read the tier table against the actual memory. worth it only for a specific local coding workload, not as a general assistant on this much ram, and definitely not before a memory upgrade.