DLite: History-free Speculative Decoding for Agentic AI

This repository contains a DLite draft checkpoint presented in DLite: History-free Speculative Decoding for Agentic AI.

DLite is a history-free speculative decoding framework for agentic AI. It reuses contextual hidden states from the target model to predict draft tokens without maintaining a historical draft KV cache. A lightweight sequential head restores dependencies within each speculative block, while a two-stage training strategy aligns the draft model with the target model.

Code: https://github.com/SJTU-DAI-Sys/DLite

DLite draft checkpoints are paired with specific target models (e.g. Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3.5-4B). Please refer to the GitHub repository and the Hugging Face collection for the matching target model.

Usage

from dlite import DLiteEngine

engine = DLiteEngine.from_pretrained(
    target_model="/path/to/Qwen3-8B",
    draft_model="/path/to/dlite-student",
    attention_backend="auto",  # auto, eager, sdpa, flash_attention_2
)
result = engine.generate("Explain speculative decoding in one paragraph.")
print(result.text)
print(result.stats)

See the GitHub repository for installation, CLI usage, and benchmarking instructions.

Downloads last month
140
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including SJTU-DAI-Sys/Qwen3-14B-DLite

Paper for SJTU-DAI-Sys/Qwen3-14B-DLite