Best local LLMs for 96GB VRAM
What you can run with 96 GB of GPU memory (95 GB usable after ~1 GB overhead), at 16k context with an f16 KV cache. Speeds use data measured on RTX PRO 6000 Blackwell Workstation where available.
Models that fit 96 GB, largest first
| Model | Params | Quant | Fit | Memory | Tok/s | Speed source |
|---|---|---|---|---|---|---|
| gpt-oss 120B | 116.83B MoE | Q8_0 | fits fully | 64.0 GB | 196 | scaled from measured |
| GLM-4.5-Air | 110.47B MoE | Q5_K_M | fits fully | 86.6 GB | 95 | estimated |
| Qwen3.6 35B-A3B | 35.95B MoE | Q8_0 | fits fully | 37.3 GB | 281 | estimated |
| Gemma 4 31B | 31.27B | Q8_0 | fits fully | 35.3 GB | 33 | estimated |
| Qwen3-Coder 30B-A3B | 30.53B MoE | Q8_0 | fits fully | 34.1 GB | 246 | estimated |
| Qwen3.6 27B | 27.78B | Q8_0 | fits fully | 29.8 GB | 39 | estimated |
| Gemma 4 26B-A4B | 25.81B MoE | Q8_0 | fits fully | 27.9 GB | 215 | estimated |
| Mistral Small 3.2 24B | 24.01B | Q8_0 | fits fully | 27.7 GB | 44 | estimated |
| gpt-oss 20B | 20.91B MoE | Q8_0 | fits fully | 12.5 GB | 287 | scaled from measured |
| Gemma 4 12B | 11.96B | Q8_0 | fits fully | 13.7 GB | 44 | scaled from measured |
| Qwen3.5 9B | 9.65B | Q8_0 | fits fully | 10.1 GB | 60 | scaled from measured |
| Qwen3-VL 8B | 8.77B | Q8_0 | fits fully | 11.1 GB | 66 | scaled from measured |
| Qwen3.5 4B | 4.66B | Q8_0 | fits fully | 5.1 GB | 93 | scaled from measured |
Copy-paste commands for the top pick
gpt-oss 120B Q8_0, 16k context.
llama-server -hf unsloth/gpt-oss-120b-GGUF:Q8_0 -c 16384 -fa on --jinja -ngl 99
OLLAMA_CONTEXT_LENGTH=16384 ollama run hf.co/unsloth/gpt-oss-120b-GGUF:Q8_0
How much context fits in 96 GB?
Largest context where weights plus KV cache stay inside 95 GB, by KV cache type (q8_0 halves and q4_0 quarters the cache). Native limits and YaRN extension apply per model.
| Model | Weights | f16 KV | q8_0 KV | q4_0 KV |
|---|---|---|---|---|
| gpt-oss 120B Q8_0 | 63.4 GB | 128k | 128k | 128k |
| GLM-4.5-Air Q5_K_M | 83.5 GB | 32k | 64k | 128k |
| Qwen3.6 35B-A3B Q8_0 | 36.9 GB | 1M | 1M | 1M |
| Gemma 4 31B Q8_0 | 33.1 GB | 256k | 256k | 256k |
| Qwen3-Coder 30B-A3B Q8_0 | 32.5 GB | 512k | 1M | 1M |
| Qwen3.6 27B Q8_0 | 28.6 GB | 1M | 1M | 1M |
| Gemma 4 26B-A4B Q8_0 | 27.3 GB | 256k | 256k | 256k |
| Mistral Small 3.2 24B Q8_0 | 25.1 GB | 128k | 128k | 128k |
| gpt-oss 20B Q8_0 | 12.1 GB | 128k | 128k | 128k |
| Gemma 4 12B Q8_0 | 13.1 GB | 256k | 256k | 256k |
| Qwen3.5 9B Q8_0 | 9.5 GB | 1M | 1M | 1M |
| Qwen3-VL 8B Q8_0 | 8.7 GB | 512k | 1M | 1M |
| Qwen3.5 4B Q8_0 | 4.5 GB | 1M | 1M | 1M |
Try the exact numbers for your setup in the interactive advisor.
What does not fit fully in 96 GB
Every model in the dataset fits.
GPUs with 96 GB
RTX PRO 6000 Blackwell Workstation. See their pages: RTX PRO 6000 96GB (bonus page).
Hardware pages near 96 GB
AMD Ryzen AI Max+ 395 (Strix Halo) 128GB · NVIDIA DGX Spark 128GB · Mac Studio (2025) M3 Ultra 96GB · MacBook Pro 14/16 (2024) M4 Max 128GB · Mac mini (2024) M4 Pro 64GB · RTX PRO 6000 96GB (bonus page) · Mac Studio M5 Max 128GB · MacBook Pro 14/16 M5 Max 128GB
FAQ
What is the best local LLM for 96GB VRAM?
By size, gpt-oss 120B Q8_0 is the largest model in our dataset that fits fully at 16k context (64.0 GB, ~196 tok/s (scaled from measured)).
How many models fit in 96GB VRAM?
13 of 13 models fit fully at 16k context with an f16 KV cache; 0 more run only with CPU offload.
How much context can I use with 96GB?
It depends on the model: for gpt-oss 120B the KV cache allows up to 128k tokens at f16 and 128k with a q4_0 cache.
Neighbouring tiers: 64 GB · 128 GB · all tiers. Numbers come from the same memory-fit math as the advisor; sources and methods.