Can RTX 5090 32GB run Mistral Small 3.2 24B?
Measured results for Mistral Small 3.2 24B (24.01B dense) on RTX 5090 32GB, plus what fits and the commands to run it.
Short answer
Yes. Mistral Small 3.2 24B Q8_0 fits fully on RTX 5090 32GB at 16k context, using 27.7 of 31.0 GB usable, and generates about 63.9 tokens/s in our own run. As the context fills it slows down: 57.4 tok/s with 16k tokens already in context (90% of the empty-context speed), measured. For Q8_0, our bandwidth-only estimate was ~39 tok/s against a measured median of ~64 (38% too pessimistic).
Measured generation speed by quantization
| Quant | Generation tok/s | Range | Prompt tok/s | Runs | Origin |
|---|---|---|---|---|---|
| Q4 class (median) | 101.1 | 101.1–101.1 | 5185 | 1 | includes our runs |
| Q8 class (median) | 63.9 | 63.9–63.9 | 5246 | 1 | includes our runs |
| Q4_K_M (our run, depth 0) | 101.1 | 3 repeats | 5185 | 3 | own measurement |
| Q8_0 (our run, depth 0) | 63.9 | 3 repeats | 5246 | 3 | own measurement |
Generation = tg128 (128 generated tokens, short context). Public rows are medians of community llama.cpp results (sources); our own runs use llama-bench b11658 on a rented GPU.
Speed vs context depth measured
Our own llama-bench runs on GeForce RTX 5090 (flash attention on, 3 repeats): generation and prompt speed with the given number of tokens already in the context.
| Quant | Context depth | Generation tok/s | vs depth 0 | Prompt tok/s |
|---|---|---|---|---|
| Q4_K_M | 0 | 101.1 | 100% | 5185 |
| Q4_K_M | 4k | 96.2 | 95% | 5053 |
| Q4_K_M | 16k | 85.8 | 85% | 4239 |
| Q8_0 | 0 | 63.9 | 100% | 5246 |
| Q8_0 | 4k | 61.9 | 97% | 5137 |
| Q8_0 | 16k | 57.4 | 90% | 4306 |
Alternatives
Less memory that still fits Mistral Small 3.2 24B
Smaller memory usually means a cheaper card or Mac (we do not list prices).
- Mac mini M6 32GB: Q4_K_M, 22 GB usable, ~7 tok/s estimated
- MacBook Air M5 32GB: Q4_K_M, 22 GB usable, ~6 tok/s scaled from measured
- RTX 4090 24GB: Q6_K, 23 GB usable, ~46 tok/s scaled from measured
Other models on RTX 5090 32GB, similar size
- LFM2.5 24B-A2B (hardware page): Q8_0, fits fully, ~343 tok/s (scaled from measured)
- Gemma 4 26B-A4B: Q8_0, fits fully, ~217 tok/s (measured)
- gpt-oss 20B: Q8_0, fits fully, ~375 tok/s (measured)
- Qwen3.6 27B: Q8_0, fits fully, ~52 tok/s (measured)
Memory and context limits
Q8_0 weights are 25.1 GB; the 16k f16 KV cache adds 2.68 GB. Largest context that fits in 31.0 GB usable:
| KV cache type | Max context |
|---|---|
| f16 | 32k |
| q8_0 | 64k |
| q4_0 | 64k |
Native limit 128k. Full KV cache table for Mistral Small 3.2 24B.
Copy-paste commands
The advisor's pick for this hardware: Q8_0, 16k context.
llama-server -hf unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q8_0 -c 16384 -fa on --jinja -ngl 99
OLLAMA_CONTEXT_LENGTH=16384 ollama serve ollama run hf.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q8_0
FAQ
How fast is Mistral Small 3.2 24B on RTX 5090 32GB?
About 63.9 tokens/s for generation (Q8_0 on llama.cpp CUDA, 128-token generation), 5246 tokens/s for prompt processing. Speed drops as the context fills; see the depth table.
How much memory does Mistral Small 3.2 24B need on RTX 5090 32GB?
The advisor's pick, Q8_0, is 25.1 GB of weights plus 2.68 GB of KV cache at 16k context (27.7 GB), against 31.0 GB usable.
Is the Mistral Small 3.2 24B speed on RTX 5090 32GB measured or estimated?
Measured: our own llama-bench run on a rented GeForce RTX 5090, including runs with 4k to 16k tokens already in context.
What can I use instead of RTX 5090 32GB for Mistral Small 3.2 24B?
With less memory, Mac mini M6 32GB (~7 tok/s), MacBook Air M5 32GB (~6 tok/s), RTX 4090 24GB (~46 tok/s) also fit it fully.
What is the maximum context for Mistral Small 3.2 24B on RTX 5090 32GB?
With an f16 KV cache about 32k tokens, 64k with q8_0 and 64k with q4_0, bounded by the model's own limit.
Related
All models on RTX 5090 32GB · Mistral Small 3.2 24B on all hardware · VRAM tier · all measured pages · leaderboard
Compare: RTX 4090 24GB vs RTX 5090 32GB · RTX 3090 24GB vs RTX 5090 32GB · 2x RTX 3090 (48GB total) vs RTX 5090 32GB