← all posts site ↗

speculative decoding lost on my gpu

wanted the small local model faster. the base 9b does about fifty tokens a second on a 16 gig card, and every speculative variant that tried to beat it made it worse. the first suspects were the usual ones, model does not fit, layer offloading, wrong batch sizes, so benched the harness itself. the benchmark rejected my flags first, then showed decode flat across every offload level, eighteen layers on the gpu and thirty layers on the gpu gave the same speed. that killed the overflow theory and left one awkward fact: weight streaming is not what sets the pace.

so tested the mtp finetunes, the ones with the draft head built into the same file. the counters showed the draft model firing, and acceptance sat at 44 percent, below the 60 percent kill line. the math is one number: 2.5. speculative decoding replaces one cheap step with one bigger step that checks several candidates, and it only wins if the bigger step costs about the same as the small one. on a card where latency and occupancy set the pace instead of memory bandwidth, the multi row verify pass costs 2.5 times a single step.

the break even rule is brutal. at 31 percent acceptance you get half the speed. at 44 percent you get two thirds. you need above 70 percent to gain anything at all, and neither experiment got close. the killer extra: one of the mtp files was bad on its own, 16 tokens a second without the draft and prefill collapsed to a tenth of normal, so the earlier comparison was confounded. the plain quant of the base model is still the fastest thing this machine has run.

the real lesson is that speculative decoding is a bet on the draft model agreeing with the main one, and on this hardware the verification pass is expensive enough that the bet needs to pay off almost every time. no flag fixes that.