Which open model fits my hardware?

Every setup below is matched against the quant file sizes on our cards. A quant "fits" when file size + 10% + 1.5 GB (for KV cache and runtime buffers at modest context) stays under usable memory. Long contexts need more.

Estimates from published file sizes, not benchmarks. Mac and Strix Halo usable-memory figures are defaults and can be raised. Speed is not considered: dense models on CPU-only will be slow.

8 GB GPU (RTX 4060/3070)

No card fits fully.

Runs with offload (slower; assumes 64 GB system RAM, llama.cpp --n-cpu-moe style):

12 GB GPU (RTX 3060/4070)

Fits fully (largest quant that fits):

Runs with offload (slower; assumes 64 GB system RAM, llama.cpp --n-cpu-moe style):

16 GB GPU (RTX 4060 Ti/5070 Ti)

Fits fully (largest quant that fits):

Runs with offload (slower; assumes 64 GB system RAM, llama.cpp --n-cpu-moe style):

24 GB GPU (RTX 3090/4090)

Fits fully (largest quant that fits):

Runs with offload (slower; assumes 64 GB system RAM, llama.cpp --n-cpu-moe style):

32 GB GPU (RTX 5090)

Fits fully (largest quant that fits):

Runs with offload (slower; assumes 64 GB system RAM, llama.cpp --n-cpu-moe style):

2x 24 GB GPUs (48 GB)

Fits fully (largest quant that fits):

Runs with offload (slower; assumes 64 GB system RAM, llama.cpp --n-cpu-moe style):

64 GB unified-memory Mac (~48 GB usable for GPU by default)

Fits fully (largest quant that fits):

128 GB unified-memory Mac (~96 GB usable by default)

Fits fully (largest quant that fits):

DGX Spark 128 GB (~110 GB practical)

Fits fully (largest quant that fits):

Strix Halo / Ryzen AI Max 128 GB (up to 96 GB GPU allocation)

Fits fully (largest quant that fits):

CPU-only, 64 GB system RAM

Fits fully (largest quant that fits):