shruti-marathi / eval /comparison.md
jiteshkprasad's picture
Shruti Marathi: model, card, licence, NOTICE (private draft)
00e30ff verified
|
Raw History Blame Contribute Delete
2.2 kB

Shruti Marathi: how every model in the results table was run (30 Oct - 2 Nov 2026)

Same 1,200 clips for every model: 200 per set, the first clips of 1-29 s in a fixed seeded order (SPRING-INX: a fixed set of 200 code-mixed clips). Same scorer: Unicode NFC, lowercase, punctuation to spaces, word error rate with a 1,000-resample bootstrap 95% interval, plus a script-blind romanised view and English recall.

Set What it is Licence References
FLEURS mr_in test (Google) read Wikipedia sentences CC-BY-4.0 Devanagari
Kathbath Marathi valid (AI4Bharat) read sentences over the phone; the public valid split stands in for the test split CC-BY-4.0 Devanagari
DiarBench Marathi (Sarvam, AI4Bharat) real conversations, single-speaker turns with at least one English word CC-BY-4.0 English in Latin; and native
IndicVoices Marathi valid (AI4Bharat) read, extempore and conversational CC-BY-4.0 English in Latin (from the dataset's own English tags); and native
SPRING-INX R2 Marathi test (IIT Madras) phone conversations CC-BY-4.0 as written (English in Latin)
Vaani Marathi test (ARTPARK-IISc) image-prompted spontaneous speech CC-BY-4.0 English in Latin; and native
Model How it ran
Shruti Marathi the shipped shruti-marathi.cact in the Cactus needle engine on a Mac (Apple silicon), 5-beam, language detected
Whisper tiny, base transformers fp16 on a GPU, beam 5, Marathi set (size: whisper.cpp q8_0 files)
Dolphin base sherpa-onnx int8 CTC on a Mac; the export picks its own language
marathi-conformer-33m-moe the published fp32 checkpoint on a Mac CPU, greedy CTC; the published code lacks its MoE layer class, so we rebuilt it from the checkpoint
IndicConformer Marathi the OpenVoiceOS int8 ONNX export (AI4Bharat's 120M per-language model), onnx-asr greedy CTC on a CPU

Whisper runs away under plain beam search without temperature fallback (output longer than twice the reference on many clips), which inflates its numbers; Whisper's own Marathi fine-tune (whisper-small-marathi, 967 MB) still scores 55.1 on FLEURS and 54.0 on Kathbath in the same harness.