i was curious how much hardware you actually need to run the latest open source frontier models locally. the ones like kimi k3 with 2.8 trillion parameters and deepseek v4 with a trillion parameters. i have an amd gpu with 16gb of vram which can comfortably run 9 billion parameter models but these frontier ones are a completely different beast.
the key thing is that these are mixture of experts models. kimi k3 has 2.8 trillion total parameters but only about 32 billion fire per token. so you get frontier level intelligence without needing to compute all 2.8 trillion weights at once. the catch is you still need to load all those weights into memory. with q4 quantization which is the sweet spot between quality and size the model takes up about 600 gigabytes. my 16gb gpu is off by a factor of 40.
so what does the hardware ladder look like. at the bottom around 3000 to 4000 dollars you can do a pure cpu build with 384gb of ddr5 ram running q2 quantization. it works but painfully slow at 5 to 10 tokens per second and the degraded reasoning quality from heavy quantization hurts. the practical entry point is about 6000 to 8000 dollars for four used rtx 3090s with 96gb of vram total plus 256gb of system ram doing partial gpu offload. you get 10 to 20 tokens per second at q4 quality. the cleanest single machine solution is a mac studio with the m3 ultra and 512gb of unified memory for around 10000 dollars. unified memory means full bandwidth to all 512gb so you can load q4 models comfortably at maybe 15 to 30 tokens per second.
but for kimi k3 specifically even q2 quantization needs about 950gb of memory. that means either 8 a100 80gb cards at 200k plus or you wait for a mac studio with a terabyte of unified memory that doesn't exist yet. the takeaway is that running last year's frontier models at home is doable for under 10k but the cutting edge stuff released last week is still firmly in datacenter territory. the gap between what's open sourced and what you can practically run at home keeps narrowing but it hasn't closed yet.