Can 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM) run Mistral Small 3.2 24B?
Measured results for Mistral Small 3.2 24B (24.01B dense) on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM), plus what fits and the commands to run it.
Short answer
It runs on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM) only with partial offload (slow): 17.0 GB needed against 15.0 GB usable, and the measured speed is about 19.1 tokens/s.
Measured generation speed by quantization
| Quant | Generation tok/s | Range | Prompt tok/s | Runs | Origin |
|---|---|---|---|---|---|
| Q4 class (median) | 19.1 | 19.1–19.1 | 1206 | 1 | includes our runs |
| Q4_K_M (our run, depth 0) | 19.1 | 3 repeats | 1206 | 3 | own measurement |
Generation = tg128 (128 generated tokens, short context). Public rows are medians of community llama.cpp results (sources); our own runs use llama-bench b11658 on a rented GPU.
Speed vs context depth measured
Our own llama-bench runs on GeForce RTX 4060 Ti 16GB (flash attention on, 3 repeats): generation and prompt speed with the given number of tokens already in the context.
| Quant | Context depth | Generation tok/s | vs depth 0 | Prompt tok/s |
|---|---|---|---|---|
| Q4_K_M | 0 | 19.1 | 100% | 1206 |
| Q4_K_M | 4k | 18.2 | 95% | 1132 |
Alternatives
Mistral Small 3.2 24B does not fit fully here. It does fit fully on:
- Mac mini M6 32GB: Q4_K_M, 22 GB usable, ~7 tok/s estimated
- MacBook Air M5 32GB: Q4_K_M, 22 GB usable, ~6 tok/s scaled from measured
- RTX 3090 24GB: Q6_K, 23 GB usable, ~39 tok/s scaled from measured
- RTX 4090 24GB: Q6_K, 23 GB usable, ~46 tok/s scaled from measured
Faster for Mistral Small 3.2 24B
- RTX 4090 24GB: Q6_K, 23 GB usable, ~46 tok/s scaled from measured
- RTX 3090 24GB: Q6_K, 23 GB usable, ~39 tok/s scaled from measured
- AMD RX 7900 XTX 24GB: Q6_K, 23 GB usable, ~30 tok/s scaled from measured
Other models on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM), similar size
- LFM2.5 24B-A2B (hardware page): Q4_K_M, fits fully, ~140 tok/s (scaled from measured)
- gpt-oss 20B: Q8_0, fits fully, ~94 tok/s (measured)
- Gemma 4 12B: Q8_0, fits fully, ~21 tok/s (measured)
- Qwen3.5 9B: Q8_0, fits fully, ~30 tok/s (measured)
Memory and context limits
Q4_K_M weights are 14.3 GB; the 16k f16 KV cache adds 2.68 GB. Largest context that fits in 15.0 GB usable:
| KV cache type | Max context |
|---|---|
| f16 | - |
| q8_0 | - |
| q4_0 | - |
Native limit 128k. Full KV cache table for Mistral Small 3.2 24B.
Copy-paste commands
The advisor's pick for this hardware: Q4_K_M, 16k context.
llama-server -hf unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M -c 16384 -fa on --jinja -ngl 35
OLLAMA_CONTEXT_LENGTH=16384 ollama serve ollama run hf.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M
FAQ
How fast is Mistral Small 3.2 24B on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM)?
About 19.1 tokens/s for generation (Q4_K_M on llama.cpp CUDA, 128-token generation), 1206 tokens/s for prompt processing. Speed drops as the context fills; see the depth table.
How much memory does Mistral Small 3.2 24B need on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM)?
The advisor's pick, Q4_K_M, is 14.3 GB of weights plus 2.68 GB of KV cache at 16k context (17.0 GB), against 15.0 GB usable.
Is the Mistral Small 3.2 24B speed on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM) measured or estimated?
Measured: our own llama-bench run on a rented GeForce RTX 4060 Ti 16GB, including runs with 4k to 4k tokens already in context.
What can I use instead of 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM) for Mistral Small 3.2 24B?
It needs more memory than this hardware has; Mac mini M6 32GB, MacBook Air M5 32GB, RTX 3090 24GB fit it fully. Faster: RTX 4090 24GB (~46 tok/s), RTX 3090 24GB (~39 tok/s), AMD RX 7900 XTX 24GB (~30 tok/s).
What is the maximum context for Mistral Small 3.2 24B on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM)?
It does not fit fully in GPU memory, so context is bounded by offload, not by this table.
Related
All models on 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM) · Mistral Small 3.2 24B on all hardware · VRAM tier · all measured pages · leaderboard
Compare: RTX 5070 Ti 16GB vs 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM) · RTX 5060 Ti 16GB vs 16GB VRAM class (RTX 4060 Ti 16GB, 32GB RAM)