patrickbdevaney/dspark-mtp-draft-head-s3recap-p25-b0.1

DSpark MTP draft head s3recap-p25-b0.1 for DeepSeek-V4-Flash-0731-REAP

This is a fine-tuned DSpark MTP draft head for 0xSero/DeepSeek-V4-Flash-0731-REAP, together with an archive of every other head trained in the same programme. It is not a standalone language model. It replaces the checkpoint's own draft head for speculative decoding. On a Jetson AGX Thor with the author's CUDA engine, the frozen 8-prompt suite decodes at 28.38 tok/s at draft width 5, against 14.61 tok/s plain autoregressive decode in the same run (1.94x).

If you want to download one thing and run it, use DeepSeek-V4-Flash-0731-REAP-DSpark. It is the full checkpoint with this head already installed.

Overview

Target model	0xSero/DeepSeek-V4-Flash-0731-REAP, a REAP prune (160 of 256 routed experts) of deepseek-ai/DeepSeek-V4-Flash-0731 @ 9e165c30
What mtp_trained.safetensors holds	The 72 non-expert tensors of the three chained DSpark MTP blocks (mtp.0/1/2): attention, router gate and bias, shared expert, hyper-connection parameters, main_proj, Markov head and confidence head
Parameters	517,318,215, all BF16 (training master). Routed experts are not included; they come unchanged from the target.
Weight file	mtp_trained.safetensors, 1,034,643,326 bytes, SHA-256 42552974a69a…
Draft width	5, the checkpoint's own dspark_block_size
Engine revision at promotion	a608d52d7 of deepseek-v4-flash-0731-cuda
Repo size	24.85 GB in total, mostly the archived heads under arms/ (about 1.03 GB each)
Licence	MIT

Which version should I use

Repo	What it is
DeepSeek-V4-Flash-0731-REAP-DSpark	Recommended. Full 48-shard checkpoint with this head installed in shards 46–48
This repo, root files	The same head as a BF16 master, for people who already have the target checkpoint
This repo, arms/	Every other head from the programme, for comparison or further work
deepseek-v4-flash-0731-REAP-calibrated-draft-head	The earlier s3 head, measured at width 6. A copy is also at arms/s3/.

Files

Path	Contents
mtp_trained.safetensors	The head (BF16 master)
head_card.json	Promotion record: per-prompt τ and tok/s, engine revision, SHA-256 of the master and of the installed shards 46–48
eval.log	Raw engine log of the promotion measurement
train_metrics.json	Per-step training history
config.json	The target checkpoint's config.json
BEST_SETUP.md, HEAD_REGISTRY.md, DECODE_ENDGAME.md	Snapshots, dated 2026-08-25, of the engine repo's recipe, promotion registry and decode-optimisation plan
arms/<name>/	24 archived entries; see Archived heads

Using it

The loadable head is the target checkpoint with these tensors substituted into shards 46–48, re-quantised to the checkpoint's formats by tools/build_trained_head.py in the engine repo. That tool computes fresh power-of-two E8M0 block scales and refuses any tensor whose round-trip error exceeds 0.10.

git clone https://github.com/patrickbdevaney/deepseek-v4-flash-0731-cuda && cd deepseek-v4-flash-0731-cuda
python3 tools/build_trained_head.py \
    --base /path/to/DeepSeek-V4-Flash-0731-REAP \
    --trained /path/to/mtp_trained.safetensors \
    --out /path/to/head_out

The shards installed in DeepSeek-V4-Flash-0731-REAP-DSpark match the SHA-256 values recorded in this repo's head_card.json. The head has only been run with the author's CUDA engine on a Jetson AGX Thor (sm_110a). No other runtime has been tested.

Evaluation

The protocol is the engine's frozen 8-prompt suite, one prompt per category, with at least 200 generated tokens each, greedy decoding and draft width 5. τ is the number of tokens committed per target forward; the maximum is 6 at width 5. Each run checks the first 8 speculative tokens against plain AR (LOSSLESS GATE ... PASS). Under greedy verification, output is the same as plain greedy decoding by construction.

	τ	tok/s	AR in the same run	Source
Stock DSpark head, width 5	3.2550	23.25	13.86	engine evidence/run0_tau_blk5.log
s3, width 5 (previous incumbent)	3.6888	—	—	engine docs/HEAD_REGISTRY.md
This head, width 5	3.8413	28.38	14.61	eval.log, head_card.json

Per prompt (this head):

Category	τ	tok/s	vs AR
multi_turn	4.83	34.05	2.33x
long_context	4.53	31.17	2.13x
code_edit	4.18	29.80	2.04x
agentic_format	4.16	29.87	2.04x
short_factual	3.87	28.83	1.97x
reasoning	3.57	28.12	1.92x
explanation	3.00	24.14	1.65x
code_gen	2.59	21.08	1.44x
Suite mean	3.8413	28.38	1.94x

Compared with the stock head at width 5, τ rises in six categories and falls in two: long_context (5.00 to 4.53) and agentic_format (4.41 to 4.16). Under the project's release rule, the head fails 3 of 6 category floors (long_context, agentic_format, reasoning). It was promoted on the suite mean. The release pointer was not moved to it (HEAD_REGISTRY.md).

The project's end-to-end figure is 22.66 to 28.38 tok/s (+25.3%). The 22.66 is the stock head at width 6 on an earlier engine revision, so that gain combines the head, the width change and engine work.

Archived heads

A head is promoted when its suite τ beats the incumbent by more than 3.5%, the measured run-to-run spread. tok/s values come from different engine revisions and base AR varied between 13.7 and 14.6 tok/s, so compare τ within one width rather than tok/s. Values below come from each arm's head_card.json and the engine's HEAD_REGISTRY.md.

Arm	Width	τ	tok/s	Status
shipped-dspark-0731reap	6	3.5362	22.66	Stock head. Record only; no weights in this repo (they are in the target checkpoint)
s1	6	3.5762	24.52	Promoted at the time; refused when re-adjudicated on τ
s2	6	3.6275	24.76	Not promoted
s2-abl-ce0.5_tv0.5	6	3.6412	24.82	Not promoted
s2-abl-ce0.9_tv0.1	6	3.6400	24.90	Not promoted
s2-abl-ce1.0_tv0.0	6	3.6712	25.10	Not promoted
s3	6	3.8438	25.53	Promoted; 3.6888 at width 5
s3recap	5	3.6250	26.85	Not promoted (control)
s3recap-ce1.0	5	3.7325	27.69	Not promoted
s3recap-ce0.5	5	3.6950	27.51	Not promoted
s3recap-deficit	5	3.7975	27.87	Not promoted
s3recap-p25-b0.1 (repo root)	5	3.8413	28.38	Promoted, served
s3recap-p25-b0.5	5	3.6738	27.08	Not promoted
s3recap-hass1	5	3.6225	26.76	Not promoted
s3recap-hass1-p25	5	3.7950	27.86	Not promoted
s3recap-conf1.0	5	3.8025	28.02	Not promoted
s3recap-conf0.1	5	3.7925	28.04	Not promoted
s3recap-p25b-p2-b0.3	5	3.7700	27.90	Not promoted
s3recap-p25b-p2-b0.2	5	3.8163	28.08	Not promoted
auto-ace0p3	5	3.8462	28.40	Not promoted
auto-anchorpow1p5	5	3.8112	28.14	Not promoted
auto-beta0p05	5	3.8425	28.33	Not promoted
auto-ace0p5	5	3.8887	28.50	Not promoted (+1.2%, under the 3.5% bar); highest τ measured
agentic-p25-b0.1	5	3.6812	27.37	Not promoted; trained on a 3,071-sample agentic corpus
agentic-p25-b0.1-c3of7	—	unmeasured	—	Partial (3 of 7 chunks); never evaluated

Each arm directory holds mtp_trained.safetensors and head_card.json, plus eval.log and train_metrics.json where they exist. The exceptions: shipped-dspark-0731reap has only a head_card.json, and agentic-p25-b0.1-c3of7 has no logs.

How it was made

• Data. 1,536 prompts were balanced over 8 categories; 1,472 were used for training and 64 held out. Continuations were generated by the target checkpoint itself, and its hidden states were captured over those sequences. The agentic_format prompts came from BFCL.
• Training. One pass of 1,472 steps in three chunks, 16 positions per sequence. Peak learning rate 5e-5, 73 warmup steps, then cosine decay. The cross-entropy and total-variation loss weights were 0.1 and 0.9. Deficit weighting multiplied each block's loss by min((1 − r)/mean(1 − r), 3.0), where r is that block's accepted fraction. A KL anchor with β = 0.1 pulled toward the pre-training head, scaled by r. The confidence-loss weight was 0. Loss went from 1.627 to 0.037 (tail mean 0.739). Sources: train_metrics.json, BEST_SETUP.md.
• β bracket. β = 0 (s3recap-deficit) fell short of the bar. β = 0.5 (s3recap-p25-b0.5) landed below the incumbent.

Limitations

• Measured on one Jetson AGX Thor with one custom engine. No other runtime has been tested.
• The head is specific to this 160-expert REAP checkpoint.
• Trained heads in this programme have lost to the untrained head on held-out continuation drafting against a paired control. The β anchor is meant to limit this. That it does so is not established.
• The four best block-5 arms are within 1.3% τ of each other, inside the 3.5% bar. Picking this head over auto-ace0p5 or auto-ace0p3 is not statistically resolved.
• The three root .md files are 2026-08-25 snapshots of the engine repo. The engine repo is the current reference.

Changelog and corrections

• 2026-08-23: head uploaded; per-category table added, with the note that it does not pass the release rule.
• 2026-08-23 to 2026-08-26: arms archived under arms/. Until 2026-11-06 the README was a copy of the engine repo's development status page, not a model card.
• 2026-11-06: card rewritten and verified against repo files and build artifacts; a 2026-10-26 automated rewrite contained errors.

License and attribution

MIT, following the target checkpoint. Upstream model by DeepSeek (deepseek-ai/DeepSeek-V4-Flash-0731, MIT). REAP pruning by 0xSero (0xSero/DeepSeek-V4-Flash-0731-REAP, MIT).

Citation

@misc{devaney2026dsv4s3recap,
  title  = {DSpark MTP draft head s3recap-p25-b0.1 for DeepSeek-V4-Flash-0731-REAP},
  author = {Devaney, Patrick},
  year   = {2026},
  url    = {https://huggingface.co/patrickbdevaney/dspark-mtp-draft-head-s3recap-p25-b0.1}
}