---
license: apache-2.0
base_model: mistralai/Voxtral-Mini-4B-Realtime-2602
pipeline_tag: automatic-speech-recognition
language:
  - zh
  - en
library_name: transformers
tags:
  - realtime
  - speech-recognition
  - streaming-asr
  - bilingual
---

# X2-ASR-4B-0812

**Real-time bilingual streaming ASR (Chinese / English, including mixed speech).**

X2-ASR-4B-0812 is **Stage 1** of [X2-Turn](https://github.com/X-Square-Robot/X2-Turn): a full fine-tune of [mistralai/Voxtral-Mini-4B-Realtime-2602](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602) on a large-scale aligned Chinese–English corpus. It keeps the original delayed-stream protocol (one token every **80 ms**, configurable lookahead τ) and **does not** add a turn-taking head.

The dual-head model that jointly predicts ASR and turn state is [`x-square-robot/X2-Turn-4B-0812`](https://huggingface.co/x-square-robot/X2-Turn-4B-0812). Use that checkpoint for voice-assistant endpointing, barge-in, and the Turn Demo. Use **this** checkpoint when you only need streaming transcription, or as the backbone to train your own Stage 2.

## Model summary

| | |
|---|---|
| **Model ID** | [`x-square-robot/X2-ASR-4B-0812`](https://huggingface.co/x-square-robot/X2-ASR-4B-0812) |
| **Parameters** | ~4B |
| **Languages** | Chinese, English (mixed) |
| **Frame rate** | 80 ms |
| **Output** | Streaming ASR only (no turn head) |
| **Base model** | Voxtral-Mini-4B-Realtime-2602 |
| **Sibling (Stage 2)** | [`X2-Turn-4B-0812`](https://huggingface.co/x-square-robot/X2-Turn-4B-0812) |
| **Paper** | [arXiv:2608.10878](https://arxiv.org/abs/2608.10878) |
| **Code** | [X-Square-Robot/X2-Turn](https://github.com/X-Square-Robot/X2-Turn) |
| **License** | Apache-2.0 (see [LICENSE](LICENSE) and [NOTICE](NOTICE)) |

Do **not** load this checkpoint with `load_mtp_checkpoint()` from the X2-Turn package. That loader expects `vad_lm_head`, which this Stage 1 release does not have.

## Quick start

This is a stock `VoxtralRealtimeForConditionalGeneration` checkpoint. No `trust_remote_code` is required.

```bash
pip install --upgrade transformers "mistral-common[audio]"
```

```python
from transformers import AutoProcessor, VoxtralRealtimeForConditionalGeneration
from mistral_common.tokens.tokenizers.audio import Audio

model_id = "x-square-robot/X2-ASR-4B-0812"

processor = AutoProcessor.from_pretrained(model_id)
model = VoxtralRealtimeForConditionalGeneration.from_pretrained(
    model_id,
    dtype="bfloat16",
    device_map="auto",
).eval()

audio = Audio.from_file("/path/to/input.wav", strict=False)
audio.resample(processor.feature_extractor.sampling_rate)

inputs = processor(audio.audio_array, return_tensors="pt")
inputs = inputs.to(model.device, dtype=model.dtype)

outputs = model.generate(**inputs)
print(processor.batch_decode(outputs, skip_special_tokens=True)[0])
```

Default streaming delay is **τ = 480 ms** (6 × 80 ms), matching the pretrained Voxtral Realtime recipe. The same weights cover τ in 80–2400 ms; the tables below show the accuracy–latency trade-off.

A GPU with **at least 16 GB of VRAM** is recommended.

## Relation to X2-Turn

Training follows the two-stage recipe in the paper:

1. **Stage 1 (this checkpoint)** — streaming ASR adaptation on ~26k hours of Chinese–English speech with frame-level ASR labels. Full fine-tune of the causal audio encoder and the language decoder. τ is sampled per batch between 1 and 30 frames (80–2400 ms).
2. **Stage 2** — [`X2-Turn-4B-0812`](https://huggingface.co/x-square-robot/X2-Turn-4B-0812) attaches an independent turn head (initialized as a copy of the ASR `lm_head`) and jointly fine-tunes ASR + turn state.

Stage 2 is stronger at turn-taking. **Stage 1 is stronger at ASR** on the same streaming test sets (see below). After Stage 2 joint training, recognition degrades on some sets; that is expected.

## Benchmark results

Streaming ASR of **this checkpoint** at three operating points of the delay τ. Chinese numbers are **CER** (%); English numbers are **WER** (%). Avg. is the unweighted mean over the listed sets. A subset of the 480 ms / 2400 ms numbers also appears in Table 3 of the [paper](https://arxiv.org/abs/2608.10878), together with Stage 2 (X2-Turn) and chunk-based baselines.

**Chinese (CER ↓)**

| Test set | τ = 480 ms | τ = 960 ms | τ = 2400 ms |
|---|:---:|:---:|:---:|
| AISHELL-1 Dev | 2.14 | 1.45 | 1.30 |
| AISHELL-1 Test | 2.57 | 1.76 | 1.48 |
| AISHELL-2 Android | 4.70 | 3.77 | 3.42 |
| AISHELL-2 iOS | 4.41 | 3.60 | 3.23 |
| AISHELL-2 Mic | 4.54 | 3.77 | 3.54 |
| AISHELL-3 Test | 3.16 | 2.33 | 2.13 |
| AISHELL-4 Test | 18.72 | 17.26 | 16.42 |
| WenetSpeech Dev | 8.13 | 7.79 | 7.81 |
| WenetSpeech Test-Meeting | 9.25 | 8.26 | 7.68 |
| WenetSpeech Test-Net | 9.50 | 8.70 | 8.40 |
| **Avg.** | **6.71** | **5.87** | **5.54** |

**English (WER ↓)**

| Test set | τ = 480 ms | τ = 960 ms | τ = 2400 ms |
|---|:---:|:---:|:---:|
| GigaSpeech | 12.23 | 11.05 | 10.87 |
| LibriSpeech test-clean | 2.40 | 1.65 | 1.54 |
| LibriSpeech test-other | 5.87 | 4.25 | 3.77 |
| TED-LIUM Test | 4.62 | 3.76 | 3.66 |
| VoxPopuli | 9.16 | 6.63 | 6.05 |
| VoxPopuli Accented | 27.93 | 25.88 | 25.12 |
| **Avg.** | **10.37** | **8.87** | **8.50** |

Larger τ uses more lookahead and generally lowers error, at the cost of additional streaming delay. AISHELL-4 and VoxPopuli Accented remain the hardest sets in each language. For Stage 2 turn-taking accuracy and the comparison against Uni-ASR / Freeze-Omni, see the [paper](https://arxiv.org/abs/2608.10878).

## Training data

Stage 1 uses public Chinese and English speech corpora totaling about **26k hours** (about 14k Chinese / 12k English), with word-level timestamps from forced alignment:

- Chinese: AISHELL-1/2/3/4, AliMeeting, WenetSpeech, KeSpeech
- English: LibriSpeech, GigaSpeech, TED-LIUM, VoxPopuli

English words are tokenized with a leading space (` Hello` / ` world`), matching Voxtral’s BPE convention; CJK characters are unchanged.

## Architecture

Unchanged from Voxtral-Mini-4B-Realtime-2602:

- Causal audio encoder + projector + language decoder
- Single ASR `lm_head` (no `vad_lm_head`)
- Delayed-stream tokens: `[STREAMING_PAD]` (id 32), `[STREAMING_WORD]` (id 33), then text
- Runtime metadata: `params.json`, `tekken.json`, `processor_config.json`, `generation_config.json`

The canonical `model.safetensors` is a single BF16 file.

## Use cases

**Recommended for:**

- Low-latency Mandarin, English, and mixed-language streaming ASR
- Reproducing the paper’s Stage 1 ASR numbers
- Initializing a custom Stage 2 / turn-taking fine-tune

**Not recommended for:**

- Turn-taking, endpointing, backchannel detection, or barge-in — use [`X2-Turn-4B-0812`](https://huggingface.co/x-square-robot/X2-Turn-4B-0812)
- Safety-critical decisions without human oversight
- Legal transcription as the sole basis for action

## Limitations

- Accuracy may degrade under noise, reverberation, overlapping speakers, accents, dialects, code-switching, far-field microphones, or packet loss.
- This checkpoint does not predict turn state. Do not treat ASR completeness as a turn-end signal.
- Real-time latency depends on hardware, serving configuration, and the chosen τ.

Speech may contain personal, biometric, confidential, or copyrighted information. Obtain appropriate consent, minimize collection and retention, and avoid logging raw audio or transcripts by default.

## Citation

If you use this model, please cite:

```bibtex
@article{fu2026x2turn,
  title   = {X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction},
  author  = {Kaiqi Fu and Rime Wen and Altman Lin and Shawn Qin and Roy Gan and Hao Wang and Qian Wang},
  journal = {arXiv preprint arXiv:2608.10878},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.10878}
}
```

## License

Model weights are released under [Apache-2.0](LICENSE). Use is also subject to the [Mistral base model terms](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602). See [NOTICE](NOTICE) for attribution details.