keptn/gemma4-31b-w4a8

Gemma 4 31B-it · W4A8 · for radiance

google/gemma-4-31B-it in one 20 GiB file for the radiance inference engine on AMD RDNA4. Text, images and video, with Google's assistant head built in for fast speculative decoding.

• Close to the original: on par with Google's own 4-bit QAT version (KL against the full-precision model)
• Fast: prose at 97 to 111 tok/s on two cards (sampled / greedy), 215 tok/s BetterBench combined on four
• Long context: 262,144 tokens; a needle at 130k tokens is found
• Vision: images and video through the original bf16 vision tower
• Tools and thinking: OpenAI-style tool calls, thinking on or off per request

File	gemma4-31b-w4a8.rad, 19.98 GiB
Needs	radiance 1.5.0 or newer; 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700)
Weights	int4, one scale per 128 columns (GPTQ, with SmoothQuant on q/k/v)
Activations	int8 per row
lm_head	int8
Speculation (MTP)	Google's Gemma 4 assistant head, bf16, 3 draft tokens
Vision tower	the original, bf16, 1.07 GiB
Context	262,144 tokens
Version	1.0 (2026-10-13)

Quick start

With the compose files from the radiance repository (set RADIANCE_MODELS, RADIANCE_API_KEY and the group ids in .env, see radiance's README):

hf download keptn/gemma4-31b-w4a8 gemma4-31b-w4a8.rad --local-dir ~/models
cp -r radiance/deploy/compose ~/radiance-compose && cd ~/radiance-compose
cp .env.example .env                    # RADIANCE_MODELS=~/models, RADIANCE_API_KEY, RENDER_GID, VIDEO_GID
docker compose -f gemma4-31b-w4a8.yaml up -d

Or directly (two cards):

docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e HSA_ENABLE_IPC_MODE_LEGACY=1 -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:1.5.0 --model /models/gemma4-31b-w4a8.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --p2p on --kv-cache-dtype fp8 --max-model-len 65536 --max-num-seqs 8 \
    --max-num-batched-tokens 2048 --num-speculative-tokens 3 --temp 1.0 --top-p 0.95 --top-k 64

	change
one card	ROCR_VISIBLE_DEVICES=0, drop --tp 2 --p2p on, --max-num-seqs 2
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
full context	--max-model-len 262144 --max-num-seqs 2
thinking	per request: "chat_template_kwargs": {"enable_thinking": true}

ROCR_VISIBLE_DEVICES picks the cards. Use the numbers of your cards (amd-smi list). Three cards do not work, because Gemma's 32 attention heads do not split into three.

Quality

How far this file is from the original model (KL divergence; lower is better, 0 = identical):

	mean KL	prose	top-1 match
this file	0.217	0.140	88.8%
previous version	0.331	0.272	86.5%
Google's QAT int4, in vLLM (16-bit activations)	0.195	0.154	88.4%
Google's QAT int4, in radiance (int8 activations)	0.274	0.237	85.2%
original model in bf16 (vLLM), as a floor	0.033	0.006	97.1%

This file is about a third closer to the original than the previous version, and about half on prose. The difference to Google's own 4-bit version in vLLM is within measurement noise. A second check on 17 new chats, not used to build anything, shows the same.

We measured on 34 chats (German and English prose, code, JSON, documents) against the original model in full precision. radiance returns the top 20 tokens per position, so all values are lower bounds.

benchmark (greedy, thinking off)	this file	previous version
GSM8K (math, 5-shot)	93.8%	94.3%
IFEval (following instructions, strict)	90.0%	88.4%

Speed

Radeon AI PRO R9700 at 215 W, radiance 1.5.0, fp8 KV, 3 draft tokens, thinking off.

BetterBench 0.6.1, full run: 8 categories, 20 runs each after 3 warm-up runs, temperature 0.7, one stream. Then prefill and parallel streams. Decode tok/s (median):

	1 card	2 cards	4 cards
combined	78.1	137.4	215.1
prose	55.7	96.7	149.8
code	85.0	149.0	233.2
JSON	96.0	170.0	263.5
summarization	71.4	126.8	196.1
prefill at 24k tokens	1,540	2,777	3,737
4 streams, total	138	374	518

Fixed prompts, greedy: three prompts per kind, 384 tokens each, two rounds, two cards. Drafts accepted at each of the three draft positions:

	tok/s	drafts accepted (position 1 / 2 / 3)
prose	110.6	0.76 / 0.63 / 0.59
code	146.8	0.89 / 0.88 / 0.86
JSON	163.1	0.94 / 0.97 / 0.95

Greedy text is easier to guess than sampled text. On prose, a step gives about 2.5 tokens here and 2.2 in BetterBench. So the same setup shows 110.6 tok/s here and 96.7 tok/s there.

Vision

The original Gemma 4 vision tower (bf16, 1.07 GiB per card) for images and video in chat requests (image_url, video_url).

	default	change with
tokens per image	280	--image-tokens 560 or 1120 (also 70, 140)
video	1 frame per second, at most 32 frames, 70 tokens per frame	--video-fps, --video-max-frames, --video-max-frame-tokens
tower placement	VRAM	--vision-tower-placement ram needs PR #91 (not in 1.5.0, where the assistant head then runs on the CPU). With it: text speed unchanged, about 54k more tokens of KV on two cards (1,017,648 instead of 963,728 at 262k × 8), video about 4× slower

benchmark (lmms-eval 0.7.3, greedy)	result
DocVQA val, 5,349 questions	87.4 (ANLS)
AI2D test, 3,088 questions	83.9%
Video-MME, short videos, no subtitles, 900 questions	69.7%

Images used the default 280 tokens. For Video-MME, lmms-eval sends 32 frames per video as images.

Memory

KV cache with the vision tower in VRAM, fp8 KV, --max-model-len 262144, from radiance's startup report:

	2 sequences	4 sequences	8 sequences
1 card	270,448 tokens	248,848 tokens	205,632 tokens
2 cards	524,256 tokens	1,007,088 tokens	963,728 tokens
4 cards	524,144 tokens	1,048,416 tokens	2,097,040 tokens

Tokens of context the KV pool holds in total, shared by all sequences. Where a value equals sequences × 262,144, every sequence can have the full context. On one card each sequence also keeps its own sliding-window state, so more sequences leave less for the shared pool.

Tested

On radiance 1.5.0 with this file:

• needle in a haystack at 8k, 32k, 97k and 130k tokens: found every time
• tool calls: 10/10 on a tool-calling check
• video: two full-HD clips of 90 and 120 s: order of events, motion and on-screen text correct

How it was made

1. Start from the bf16 checkpoint google/gemma-4-31B-it.
2. Calibration set: 110 chats, 165,626 tokens (25 with thinking). None of these prompts is in the tests above.
3. SmoothQuant on q/k/v (alpha 0.5). A few channels in front of attention carry large values. In int8 they cost all other channels their precision. SmoothQuant moves that range into the weights. The model still computes the same function. Script and scales: smoothing/.
4. GPTQ to int4 with one f16 scale per 128 columns: gemma4-31b-w4a8-lmi8.recipe, built with rad-convert.
5. lm_head int8 (round to nearest). Google's assistant head and the vision tower stay bf16.

python smoothing/smooth_checkpoint.py gemma-4-31B-it smoothing/smoothquant-stats.npz 0.5 gemma-4-31B-it-sq
rad-convert gemma-4-31B-it-sq --draft-model gemma-4-31B-it-assistant \
    --recipe gemma4-31b-w4a8-lmi8.recipe --set quantization_config.quant_method=compressed-tensors \
    --quant output.weight=rtn:codes=i8,group=128,scale=bf16,clamp=sym -o gemma4-31b-w4a8.rad

GPTQ needs statistics from the calibration set (calib= in the recipe).

Limits

• radiance runs on AMD RDNA4 cards.
• Long pasted documents are the weakest case: the distance to the original is larger there than on prose or code.

Credits

• Gemma 4 support in radiance: based on the initial work of sfjes (sfjes@vmdaemon.org) in PR #75.
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
• Base model, assistant head and vision tower: Google.

License

Apache 2.0, the license of the base model. See LICENSE.