Can RTX 4070 12GB run Qwen3.5 9B?
Measured results for Qwen3.5 9B (9.65B dense) on RTX 4070 12GB, plus what fits and the commands to run it.
Short answer
Yes. Qwen3.5 9B Q8_0 fits fully on RTX 4070 12GB at 16k context, using 10.1 of 11.0 GB usable, and generates about 51.3 tokens/s in our own run. For Q8_0, our bandwidth-only estimate was ~29 tok/s against a measured median of ~51 (43% too pessimistic).
Measured generation speed by quantization
| Quant | Generation tok/s | Range | Prompt tok/s | Runs | Origin |
|---|---|---|---|---|---|
| Q4 class (median) | 79.8 | 79.8–79.8 | 4237 | 1 | includes our runs |
| Q8 class (median) | 51.3 | 51.3–51.3 | 4418 | 1 | includes our runs |
| Q4_K_M (our run, depth 0) | 79.8 | 3 repeats | 4237 | 3 | own measurement |
| Q8_0 (our run, depth 0) | 51.3 | 3 repeats | 4417 | 3 | own measurement |
Generation = tg128 (128 generated tokens, short context). Public rows are medians of community llama.cpp results (sources); our own runs use llama-bench b11658 on a rented GPU.
Speed vs context depth measured
Our own llama-bench runs on GeForce RTX 4070 12GB (flash attention on, 3 repeats): generation and prompt speed with the given number of tokens already in the context.
| Quant | Context depth | Generation tok/s | vs depth 0 | Prompt tok/s |
|---|---|---|---|---|
| Q4_K_M | 0 | 79.8 | 100% | 4237 |
| Q4_K_M | 4k | 77.8 | 97% | 4109 |
| Q4_K_M | 16k | 72.1 | 90% | 3557 |
| Q8_0 | 0 | 51.3 | 100% | 4417 |
| Q8_0 | 4k | 50.4 | 98% | 4334 |
| Q8_0 | 16k | 48 | 94% | 3728 |
Memory and context limits
Q8_0 weights are 9.5 GB; the 16k f16 KV cache adds 0.59 GB. Largest context that fits in 11.0 GB usable:
| KV cache type | Max context |
|---|---|
| f16 | 32k |
| q8_0 | 64k |
| q4_0 | 128k |
Native limit 256k, 1.01M with YaRN. Full KV cache table for Qwen3.5 9B.
Copy-paste commands
The advisor's pick for this hardware: Q8_0, 16k context.
llama-server -hf unsloth/Qwen3.5-9B-GGUF:Q8_0 -c 16384 -fa on --jinja -ngl 99
OLLAMA_CONTEXT_LENGTH=16384 ollama run hf.co/unsloth/Qwen3.5-9B-GGUF:Q8_0
FAQ
How fast is Qwen3.5 9B on RTX 4070 12GB?
About 51.3 tokens/s for generation (Q8_0 on llama.cpp CUDA, 128-token generation), 4417 tokens/s for prompt processing. Speed drops as the context fills; see the depth table.
How much memory does Qwen3.5 9B need on RTX 4070 12GB?
The advisor's pick, Q8_0, is 9.5 GB of weights plus 0.59 GB of KV cache at 16k context (10.1 GB), against 11.0 GB usable.
What is the maximum context for Qwen3.5 9B on RTX 4070 12GB?
With an f16 KV cache about 32k tokens, 64k with q8_0 and 128k with q4_0, bounded by the model's own limit.
Related
All models on RTX 4070 12GB · Qwen3.5 9B on all hardware · VRAM tier · all measured pages · leaderboard
Compare: RTX 4070 12GB vs RTX 5070 12GB