← all posts site ↗

a 35b model doesn't fit a 16gb card

saw a tweet hyping an uncensored qwen 3.6 vision model, mixture of experts with 35 billion total parameters and three billion active. a hundred and sixty thousand plus downloads. the promo account made it look like a must run. i wanted to know if it would actually run on my machine.

checked the real file sizes instead of trusting the hype. the model needs a vision projector file on top of the weights. the smallest quant came in around seventeen gig, plus the projector, so roughly eighteen gig total. my amd card holds sixteen. two gig short.

a two gig miss means part of the weights pins to system ram and streams over the pcie bus on every token, and my card has a narrow memory bus which is the worst case for that. realistically two to four tokens per second. the mid and high quants are completely out of the question.

downloads do not equal runs on my machine. that model is built for people with 24 gig cards or beefy cpu boxes. the actual ceiling for my rig is around a 10 billion parameter class model queanted to about seven gig. saved myself from a painful afternoon of the thing crashing or swapping.