Best local LLMs for 24GB VRAM
What you can run with 24 GB of GPU memory (23 GB usable after ~1 GB overhead), at 16k context with an f16 KV cache. Speeds use data measured on GeForce RTX 4090 where available.
Models that fit 24 GB, largest first
| Model | Params | Quant | Fit | Memory | Tok/s | Speed source |
|---|---|---|---|---|---|---|
| Qwen3.6 35B-A3B | 35.95B MoE | UD-Q4_K_M | fits fully | 22.5 GB | 190 | measured |
| Gemma 4 31B | 31.27B | Q4_K_M | fits fully | 20.5 GB | 44 | measured |
| Qwen3-Coder 30B-A3B | 30.53B MoE | Q4_K_M | fits fully | 20.2 GB | 266 | measured |
| Qwen3.6 27B | 27.78B | Q5_K_M | fits fully | 20.7 GB | 42 | scaled from measured |
| Gemma 4 26B-A4B | 25.81B MoE | UD-Q5_K_M | fits fully | 21.7 GB | 135 | scaled from measured |
| Mistral Small 3.2 24B | 24.01B | Q6_K | fits fully | 22.0 GB | 42 | scaled from measured |
| gpt-oss 20B | 20.91B MoE | Q8_0 | fits fully | 12.5 GB | 258 | measured |
| Gemma 4 12B | 11.96B | Q8_0 | fits fully | 13.7 GB | 64 | measured |
| Qwen3.5 9B | 9.65B | Q8_0 | fits fully | 10.1 GB | 95 | measured |
| Qwen3-VL 8B | 8.77B | Q8_0 | fits fully | 11.1 GB | 102 | scaled from measured |
| Qwen3.5 4B | 4.66B | Q8_0 | fits fully | 5.1 GB | 158 | measured |
Copy-paste commands for the top pick
Qwen3.6 35B-A3B UD-Q4_K_M, 16k context.
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M -c 16384 -fa on --jinja -ngl 99
OLLAMA_CONTEXT_LENGTH=16384 ollama run hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
How much context fits in 24 GB?
Largest context where weights plus KV cache stay inside 23 GB, by KV cache type (q8_0 halves and q4_0 quarters the cache). Native limits and YaRN extension apply per model.
| Model | Weights | f16 KV | q8_0 KV | q4_0 KV |
|---|---|---|---|---|
| Qwen3.6 35B-A3B UD-Q4_K_M | 22.1 GB | 32k | 64k | 128k |
| Gemma 4 31B Q4_K_M | 18.3 GB | 32k | 64k | 128k |
| Qwen3-Coder 30B-A3B Q4_K_M | 18.6 GB | 32k | 64k | 128k |
| Qwen3.6 27B Q5_K_M | 19.5 GB | 32k | 64k | 128k |
| Gemma 4 26B-A4B UD-Q5_K_M | 21.1 GB | 64k | 128k | 256k |
| Mistral Small 3.2 24B Q6_K | 19.4 GB | 16k | 32k | 64k |
| gpt-oss 20B Q8_0 | 12.1 GB | 128k | 128k | 128k |
| Gemma 4 12B Q8_0 | 13.1 GB | 256k | 256k | 256k |
Try the exact numbers for your setup in the interactive advisor.
What does not fit fully in 24 GB
Every model in the dataset fits.
GPUs with 24 GB
GeForce RTX 3090, GeForce RTX 4090, Radeon RX 7900 XTX. See their pages: RTX 3090 24GB · RTX 4090 24GB · AMD RX 7900 XTX 24GB.
Hardware pages near 24 GB
RTX 3090 24GB · RTX 4090 24GB · RTX 5090 32GB · 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM) · RTX 5070 Ti 16GB · RTX 5080 16GB · AMD RX 7900 XTX 24GB · Mac mini / iMac M4 16GB
FAQ
What is the best local LLM for 24GB VRAM?
By size, Qwen3.6 35B-A3B UD-Q4_K_M is the largest model in our dataset that fits fully at 16k context (22.5 GB, ~190 tok/s (measured)).
How many models fit in 24GB VRAM?
11 of 13 models fit fully at 16k context with an f16 KV cache; 0 more run only with CPU offload.
How much context can I use with 24GB?
It depends on the model: for Qwen3.6 35B-A3B the KV cache allows up to 32k tokens at f16 and 128k with a q4_0 cache.
Neighbouring tiers: 16 GB · 32 GB · all tiers. Numbers come from the same memory-fit math as the advisor; sources and methods.