Full Duplex Leaderboard
Turn-taking, commands and voice style, with a separate tab for each benchmark.
When should the agent speak? A shared set of 5 recorded two-person conversations (TurnBench dev); sample counts show coverage. Ranked by Youden’s J ×100: recall (%) minus FPR (%). Higher is better. Hover a column for its definition.
| # | Model | Youden’s J ×100 ↑ | Turn ends caught | False alarms | Latency | Interruptions caught | Interruption false alarms | Samples |
|---|---|---|---|---|---|---|---|---|
| 1 | VAP (otoSpeech) Local CPU | 80.8 | 88.2% | 7.4% | 641 ms | 80.0% | 19.4% | 238 turn ends 5 conversations |
| 2 | Local CPU | 79.1 | 81.1% | 2.0% | 1,011 ms | 70.0% | 11.8% | 238 turn ends 5 conversations |
| 3 | OpenAI Realtime, server VAD (1.8 s silence) Cloud API | 77.9 | 82.3% | 4.5% | 2,005 ms | 80.0% | 39.2% | 238 turn ends 5 conversations |
| 4 | OpenAI Realtime, semantic VAD (high eagerness) Cloud API | 76.2 | 83.6% | 7.4% | 2,437 ms | 80.0% | 40.0% | 238 turn ends 5 conversations |
| 5 | On device | 75.0 | 92.9% | 17.8% | 1,315 ms | 60.0% | 9.8% | 238 turn ends 5 conversations |
| 6 | Silero VAD + silence timeout On device | 74.6 | 92.4% | 17.8% | 1,317 ms | 60.0% | 9.8% | 238 turn ends 5 conversations |
| 7 | On device | 74.6 | 90.5% | 16.0% | 1,506 ms | — | — | 148 turn ends 2 conversations |
| 8 | On device | 71.0 | 87.4% | 16.3% | 1,639 ms | 50.0% | 9.8% | 238 turn ends 5 conversations |
| 9 | Cloud API | 66.2 | 86.5% | 20.4% | 831 ms | — | — | 193 turn ends 3 conversations |
| 10 | On device | 64.0 | 88.2% | 24.3% | 1,031 ms | 30.0% | 11.5% | 238 turn ends 5 conversations |
| 11 | Cloud API | 56.1 | 63.0% | 6.9% | 1,589 ms | — | — | 238 turn ends 5 conversations |
| 12 | On device | 56.0 | 63.9% | 7.9% | 1,137 ms | — | — | 238 turn ends 5 conversations |
| 13 | Cloud API | 49.2 | 60.1% | 10.9% | 1,239 ms | — | — | 238 turn ends 5 conversations |
| 14 | Local CPU | 40.8 | 54.6% | 13.9% | 573 ms | 80.0% | 10.8% | 238 turn ends 5 conversations |
| 15 | Local CPU | 36.0 | 53.4% | 17.3% | 586 ms | 80.0% | 9.1% | 238 turn ends 5 conversations |
Hosted voice agents are tested by talking with each speaker in real time; their turn ends are read from when they start replying, and they are not scored on interruptions. Rows with fewer conversations ran part of the set. We ran every displayed configuration ourselves. VAP appears once, using its larger five-conversation run; native-format checks are linked from the method. Hover scores for all-38 dev results.
What should it do? Spoken home-assistant commands (SLURP test). Each command comes with 10 tools plus "no action"; in 1 of 5 the right tool is missing. Ranked by far-mic accuracy.
| # | Model | Accuracy far mic | Accuracy close mic | Calls despite missing tool | Arguments exact | Latency | Samples |
|---|---|---|---|---|---|---|---|
| 1 | Audio inCloud API | 83.0% | 84.8% | 42.6% 81/190 | 44.7% | 985 ms | 1,000 commands |
| 2 | Audio inCloud API | 75.9% | 83.2% | 18.9% 36/190 | 45.4% | 547 ms | 1,000 commands |
| 3 | Speech → textOn device | 75.2% | 86.6% | 46.3% 88/190 | — | 80 ms | 1,000 commands |
| 4 | Speech → textCloud API | 73.5% | 81.4% | 11.6% 22/190 | — | 250 ms | 1,000 commands |
| 5 | Speech → textOn device | 73.5% | 79.6% | 39.0% 74/190 | — | 29 ms | 1,000 commands |
| 6 | Audio inCloud API | 73.0% | 76.0% | 17.4% 33/190 | 40.6% | 1,914 ms | 1,000 commands |
| 7 | Audio inOn device | 72.8% | 75.2% | 69.0% 131/190 | — | 46 ms | 1,000 commands |
| 8 | Audio inCloud API | 66.7% | 68.2% | 28.6% 2/7 | 41.9% | 894 ms | 50 commands |
| 9 | Audio inOn device | 61.7% | 64.9% | 16.3% 31/190 | 35.1% | 1,950 ms | 1,000 commands |
| 10 | Audio inOn device | 56.5% | 71.8% | 89.5% 170/190 | 34.9% | 224 ms | 1,000 commands |
| 11 | Speech → textOn device | 50.8% | 60.7% | 5.8% 11/190 | — | 6 ms | 1,000 commands |
| 12 | Speech → textOn device | 36.4% | 43.1% | 34.7% 66/190 | 17.6% | 375 ms | 1,000 commands |
Speech → text rows transcribe with Parakeet first, then pick the tool from the transcript. Missing-tool call rates pool headset and far-field requests; they do not measure activation from background speech or silence. Arguments exact is blank for models that only name the tool.
| # | Model | Overall / 100 | APS Voice description | DSD Style direction | RP Role- play | Data used | Other results |
|---|---|---|---|---|---|---|---|
| 1 | Qwen3-TTS 1.7B VoiceDesign (speech-swift) On devicePilot · different judge setting | 76.7 23/30 | 70.0% 7/10 | 100.0% 10/10 | 60.0% 6/10 | InstructTTSEval · development pilot 5 English + 5 Chinese lines; 30 style checks Gemini Pro · pilot setting | English 80.0%Chinese 73.3%Wrong words 3.3% |
| 2 | Original recordings Reference | 73.3 110/150 | 84.0% 42/50 | 94.0% 47/50 | 42.0% 21/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 77.3%Chinese 69.3%Wrong words 0.0% |
| 3 | Google Gemini 3.8 Flash TTS Cloud API | 67.3 101/150 | 76.0% 38/50 | 78.0% 39/50 | 48.0% 24/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 72.0%Chinese 62.7%Wrong words 0.0% |
| 4 | Breeze TTS 2 (speech-swift, BF16) On devicePilot · different judge setting | 66.7 20/30 | 60.0% 6/10 | 80.0% 8/10 | 60.0% 6/10 | InstructTTSEval · development pilot 5 English + 5 Chinese lines; 30 style checks Gemini Pro · pilot setting | English 53.3%Chinese 80.0%Wrong words 3.3% |
| 5 | FireRedTTS3-Instruct (speech-swift, FP32) On deviceProvisional · 1 judge response under review | 65.3 98/150 | 68.0% 34/50 | 78.0% 39/50 | 50.0% 25/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 72.0%Chinese 58.7%Wrong words 0.0%EN · First PCM 4.499 s · RTF 1.654 · Peak memory 10.819 GBZH · First PCM 11.221 s · RTF 2.125 · Peak memory 11.701 GBComplete buffered audio |
| 6 | ElevenLabs v4 Cloud APIInput adapter under review | 63.3 95/150 | 68.0% 34/50 | 76.0% 38/50 | 46.0% 23/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 62.7%Chinese 64.0%Wrong words 0.7% |
| 7 | VoxCPM2 2B (speech-swift) On deviceProvisional · 1 judge response under review | 59.3 89/150 | 64.0% 32/50 | 68.0% 34/50 | 46.0% 23/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 61.3%Chinese 57.3%Wrong words 0.0% |
| 8 | MOSS-VoiceGenerator (speech-swift, BF16 + FP32 codec) On deviceProvisional · 1 judge response under review | 56.0 84/150 | 52.0% 26/50 | 76.0% 38/50 | 40.0% 20/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 52.0%Chinese 60.0%Wrong words 8.0%EN · First PCM 0.470 s · RTF 0.610 · Peak memory 8.448 GBZH · First PCM 0.457 s · RTF 0.572 · Peak memory 8.619 GBStreaming PCM |
| 9 | Qwen3-TTS 1.7B CustomVoice (speech-swift) On device | 52.7 79/150 | 54.0% 27/50 | 64.0% 32/50 | 40.0% 20/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 49.3%Chinese 56.0%Wrong words 0.0% |
| 10 | Qwen3-TTS 0.6B CustomVoice (speech-swift) On deviceExperimental input · 1 judge response under review | 48.0 72/150 | 48.0% 24/50 | 60.0% 30/50 | 36.0% 18/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 50.7%Chinese 45.3%Wrong words 2.0% |
| 11 | Amazon Nova 2.5 Sonic (speech generation) Cloud APIEnglish only | 40.0 30/75 | 32.0% 8/25 | 56.0% 14/25 | 32.0% 8/25 | InstructTTSEval 25 English lines; 75 style checks Gemini Pro · standard cohort | English 40.0%Wrong words 1.3% |
| 12 | CosyVoice3 0.5B (speech-swift) On deviceInput conditioning and judge response under review | 34.7 52/150 | 4.0% 2/50 | 60.0% 30/50 | 40.0% 20/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 33.3%Chinese 36.0%Wrong words 32.0% |
| 13 | Fish Audio S2 Pro (speech-swift, BF16) On deviceProvisional · 2 judge responses under review | 34.7 52/150 | 28.0% 14/50 | 50.0% 25/50 | 26.0% 13/50 | InstructTTSEval 25 English + 25 Chinese lines; 150 style checks Gemini Pro · standard cohort | English 32.0%Chinese 37.3%Wrong words 1.3% |