Can RTX 4090 24GB run Gemma 4 26B-A4B?
Measured results for Gemma 4 26B-A4B (25.81B MoE, 3.8B active) on RTX 4090 24GB, plus what fits and the commands to run it.
Short answer
Yes. Gemma 4 26B-A4B Q4_K_M fits fully on RTX 4090 24GB at 16k context, using 17.5 of 23.0 GB usable, and generates about 169.3 tokens/s in our own run. For UD-Q5_K_M, our bandwidth-only estimate was ~178 tok/s against a measured median of ~135 (31% too optimistic).
Measured generation speed by quantization
| Quant | Generation tok/s | Range | Prompt tok/s | Runs | Origin |
|---|---|---|---|---|---|
| Q4 class (median) | 169.3 | 169.3–169.3 | 9165 | 1 | includes our runs |
| Q4_K_M (our run, depth 0) | 169.3 | 3 repeats | 9165 | 3 | own measurement |
Generation = tg128 (128 generated tokens, short context). Public rows are medians of community llama.cpp results (sources); our own runs use llama-bench b11658 on a rented GPU.
Speed vs context depth measured
Our own llama-bench runs on GeForce RTX 4090 (flash attention on, 3 repeats): generation and prompt speed with the given number of tokens already in the context.
| Quant | Context depth | Generation tok/s | vs depth 0 | Prompt tok/s |
|---|---|---|---|---|
| Q4_K_M | 0 | 169.3 | 100% | 9165 |
| Q4_K_M | 4k | 160.6 | 95% | 8915 |
| Q4_K_M | 16k | 152.6 | 90% | 7816 |
Memory and context limits
UD-Q5_K_M weights are 21.1 GB; the 16k f16 KV cache adds 0.55 GB. Largest context that fits in 23.0 GB usable:
| KV cache type | Max context |
|---|---|
| f16 | 64k |
| q8_0 | 128k |
| q4_0 | 256k |
Native limit 256k. Full KV cache table for Gemma 4 26B-A4B.
Copy-paste commands
The advisor's pick for this hardware: UD-Q5_K_M, 16k context.
llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q5_K_M -c 16384 -fa on --jinja -ngl 99
OLLAMA_CONTEXT_LENGTH=16384 ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q5_K_M
FAQ
How fast is Gemma 4 26B-A4B on RTX 4090 24GB?
About 169.3 tokens/s for generation (Q4_K_M on llama.cpp CUDA, 128-token generation), 9165 tokens/s for prompt processing. Speed drops as the context fills; see the depth table.
How much memory does Gemma 4 26B-A4B need on RTX 4090 24GB?
The advisor's pick, UD-Q5_K_M, is 21.1 GB of weights plus 0.55 GB of KV cache at 16k context (17.5 GB), against 23.0 GB usable.
What is the maximum context for Gemma 4 26B-A4B on RTX 4090 24GB?
With an f16 KV cache about 64k tokens, 128k with q8_0 and 256k with q4_0, bounded by the model's own limit.
Related
All models on RTX 4090 24GB · Gemma 4 26B-A4B on all hardware · VRAM tier · all measured pages · leaderboard
Compare: RTX 3090 24GB vs RTX 4090 24GB · RTX 4090 24GB vs RTX 5090 32GB · RTX 4090 24GB vs AMD RX 7900 XTX 24GB · RTX 5080 16GB vs RTX 4090 24GB