mlboydaisuke's picture
slow_ar_int8.tflite replaced by the same export with a prefill_64 signature: its prefill_256 and decode give the first file's outputs bit for bit on 38 test prompts, and 256 + prefill_64 positions give the same KV cache and logits as 256 + a padded prefill_256 (12 of 12); the host loop and the Android app run a prompt's last chunk of up to 64 positions through it. android/: the app keeps the slow AR's packed weights in an XNNPACK weight cache file (peak memory after load about 1.1 GB lower on a Pixel 11 Pro and a Galaxy S26), creates and warms both GPU decoders at load and checks the GPU output against a CPU reference bundled in the APK; on both phones it made the same codes as with the first release's slow file in every test run. Card and android/README.md updated with the source repository's new name (Edge0/Audio8-TTS-Realistic-Air), the 2026-11-03 measurements, the limitation for a voice cloned across languages and the fix of the Preview repository's decoders. Nothing deleted.
752cd65 verified
|
Raw History Blame Contribute Delete
14.3 kB

Audio8-TTS-Realistic — type a sentence, hear it in a chosen voice, on the phone

Type a sentence, pick a voice and tap Speak: the phone turns the sentence into speech in that voice and plays it. The voice is one of the model's two sample voices or your own, recorded once for up to 10 seconds. On a Pixel 11 Pro an 18-token English sentence became 5.48 s of speech in 4.67 s (real-time factor 0.853: the time to make the speech divided by its length), and on a Galaxy S26 6.36 s in 5.07 s (0.797), each the median of 6 runs (2026-11-03). Everything runs on the phone through LiteRT CompiledModel from Kotlin: the two autoregressive graphs on the CPU with 4 threads, and the codec decoder on the GPU when its output passes a check, else on the CPU. The app asks for no network permission.

Model

litert-community/Audio8-TTS-Realistic-0.6b, converted from Edge0/Audio8-TTS-Realistic-Air (Apache-2.0; renamed from Edge0/Audio8-TTS-Realistic-0.6b): a DualAR speech model at 44.1 kHz. Its model card says that it supports 23 languages, and that it clones a voice zero-shot from a reference recording and its transcript; record the voice in the language it will speak (the model card's Limitations has the measurement). The app reads these files of the model repository:

File Bytes Role Runs on
slow_ar_int8.tflite 558,303,696 slow AR, 24 layers: one semantic token per 46 ms frame; KV cache of 1,024 positions; prompt chunks of 256 and 64 positions CPU, 4 threads, its packed weights kept in a weight cache file
fast_ar_int8.tflite 67,780,320 fast AR, 4 layers: the frame's 10 codebook codes, 10 calls per frame CPU, 4 threads
codec_decoder_fp16_T128.tflite 261,633,152 codec decoder, up to 128 frames (5.9 s) per call GPU if it passes the output check, else CPU
codec_decoder_fp16_T192.tflite 261,829,760 the same decoder for up to 192 frames (8.9 s); longer speech in windows the same backend as the 128-frame decoder
codec_encoder_fp16_10s.tflite 419,269,280 codec encoder: 10 s of audio to codes, for "Record my voice" CPU, created per recording
tokenizer.json 12,217,872 the vendor's Qwen2 BPE tokenizer Kotlin
voices/ja_funasr_example/, voices/en_librispeech_1272/ 7,310 the two sample voices: codec codes and transcript —

Total 1,581,041,390 bytes. The sample voices come from the Fun-ASR-Nano example clip (Apache-2.0) and LibriSpeech dev-clean speaker 1272 (CC BY 4.0).

Run it

Android 8.0 (API 26) or newer on arm64-v8a with about 2.0 GB free (the files the README pushes are 1,581,041,390 bytes, the weight cache the app writes at its first load about 365 MB, the debug APK 9.3 MB); on the computer hf, adb, a JDK 17 or newer and the Android SDK (ANDROID_HOME). The commands call sh gradlew because the Hugging Face Hub does not keep file modes: a downloaded gradlew is not executable, and ./gradlew fails with "Permission denied" (or run chmod +x gradlew once).

Step 1. Download the model repository:

hf download litert-community/Audio8-TTS-Realistic-0.6b --local-dir audio8r

Step 2. Build the app, install it and start it once:

cd audio8r/android && sh gradlew :app:assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n com.audio8tts.realistic/com.audio8tts.MainActivity

This first start shows which files are missing and creates the folders that step 3 fills (a folder that adb creates belongs to the shell user, and the app may not be allowed to open it).

Step 3. Push the model files and the two sample voices, from audio8r/android (where step 2 left you):

adb push ../*.tflite ../tokenizer.json /sdcard/Android/data/com.audio8tts.realistic/files/
for v in ja_funasr_example en_librispeech_1272; do adb push ../voices/$v/* /sdcard/Android/data/com.audio8tts.realistic/files/voices/$v/; done

Step 4. Start the app again (-S stops the first start's process before it):

adb shell am start -S -n com.audio8tts.realistic/com.audio8tts.MainActivity

Pick a voice, type a sentence and tap Speak. The sentence plays from the speaker and is saved as a wav in Android/data/com.audio8tts.realistic/files/Documents/. The package name com.audio8tts.realistic keeps this app apart from the Preview model's sample (com.audio8tts): both can be installed.

Speed on two phones (2026-11-03)

LiteRT 2.2.0; slow and fast AR on the CPU with 4 threads, the slow AR's KV cache updated in place and its weights read from the weight cache; codec decoder on the GPU (its check at load: correlation 0.99986 with the bundled CPU output on the Galaxy S26, 0.99988 on the Pixel 11 Pro). Each sentence was spoken 6 times, 3 in each of two app processes; the values are medians.

Phone Text Voice Frames Audio Generate Real-time factor
Galaxy S26 (SM-S942Q, Android 16) Our train departs early in the morning, so we should reach the station on time. English sample 137 6.36 s 5.07 s 0.797
Galaxy S26 この製品は、スマートフォンの上で音声を合成することができます。 Japanese sample 114 5.29 s 4.00 s 0.755
Pixel 11 Pro (Android 17) Our train departs early in the morning, so we should reach the station on time. English sample 118 5.48 s 4.67 s 0.853
Pixel 11 Pro この製品は、スマートフォンの上で音声を合成することができます。 Japanese sample 110 5.11 s 4.50 s 0.882

Generate is text in, samples out: prompt, prefill, the frame loop and the codec decode. The real-time factor is generate / audio. The Galaxy S26 ran most of its runs with CPU frequency caps (the model card has the details). Load (the AR graphs, both GPU decoders and the check, with the weight cache and the GPU's compiled programs already on the phone) took 6.71 and 6.42 s on the Galaxy S26 and 8.21 and 8.27 s on the Pixel 11 Pro; the first load after installing, which writes the weight cache and compiles the GPU's programs, took 23.9 and 45.4 s on the Pixel 11 Pro (two first loads). Peak memory (VmHWM) was 2.31 GB on the Galaxy S26 and 2.01 GB on the Pixel 11 Pro (kB × 1,024 / 10⁹). The model card has the phase times, the accuracy test and one call of each graph on the CPU, GPU and NPU.

Record your own voice

Tap Record my voice (10 s), allow the microphone, and read the sentence the screen shows, exactly as written (it becomes the voice's transcript; the sentence is Japanese when the text box holds Japanese, English otherwise). Stop and save ends the take early; the recording also ends at 10.03 s, the encoder's input length. The app cuts the silence before and after the speech to 0.25 s, sets the peak to -3 dBFS, runs the codec encoder (created for this call and closed after it) and saves the voice in Android/data/com.audio8tts.realistic/files/voices/my_voice/ (codes.npy, meta.json, reference.wav). "My voice" is then selected, and stays available on later launches. Record only voices whose owners agreed, and say that the audio is synthetic when you share it.

When something is missing or wrong

Situation What the app does
A required model file is missing (slow_ar_int8.tflite, fast_ar_int8.tflite, codec_decoder_fp16_T128.tflite, tokenizer.json) ERROR, and the list of the missing files with the folder they belong in; Speak stays off
Empty text "Type a sentence first."
More than 49 tokens "Too long: N tokens. One Speak reads up to 49 tokens (about 17 s of speech); split the text." The count under the text box turns red first. One call makes at most 512 frames (23.8 s); at the highest rate measured on these graphs, 10.4 frames per token, 49 tokens stay within that (509.6 frames)
Microphone permission refused "Microphone permission denied: recording a voice is off. The sample voices still work."
Cancel while generating stops before the next frame and returns to READY; the next Speak runs
Cancel while playing or recording stops the playback; drops the recording
A voice folder is missing that voice shows "(not installed)" and cannot be picked
The encoder file is missing "Record my voice" says which file to push; speaking still works

The longest prompt the app can build (49 tokens and a 10 s recorded voice) is 320 positions with the English recording sentence and 324 with the Japanese one, which leaves room in the slow graph's 1,024-position KV cache for the 512-frame cap.

Device check

Audio8DeviceCheck (app/src/androidTest) runs the same Audio8Tts the screen uses on the pushed files and logs one line per step under the tag audio8-check: models, load, speak-ja, speak-en, empty-text, long-text, cancel, regenerate (the same frames as speak-ja after the cancel), register (a voice registered from the speak-ja wav and its text, read back from its folder, then used for the English sentence), release (a second close is a no-op), missing-model and mic-denied. The last line, RESULT ok=<all>, passes when every step is ok; its line_pass needs the core steps, a real-time factor of 2.0 or less on the first speak and at least 3 failure-path steps. After sh gradlew :app:assembleDebugAndroidTest and the pushes above:

adb install -r app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
adb shell am instrument -w -e class com.audio8tts.Audio8DeviceCheck com.audio8tts.realistic.test/androidx.test.runner.AndroidJUnitRunner
adb logcat -d -s audio8-check | grep RESULT

The check writes its wav files to Android/data/com.audio8tts.realistic/files/check/. sh gradlew :app:testDebugUnitTest runs the JVM tests (text rules, the .npy writer, the recording clean-up, the weight cache's file names and clean-up).

Measurement entry and launch extras

com.audio8tts.GateActivity runs one measurement job from a JSON file in the app's internal files folder (check: the device check; tok: the tokenizer and the prompt against the vendor processor's ids; teacher: teacher-forced logits and hidden dumps; synth: speech for each case through the app's own path; kv: in-place KV against ping-pong; argpu: the AR graphs on the GPU against the CPU; codecab: two codec files alternated); the speed table above came from its synth job, which speaks through the same Audio8Tts.speak as the screen. Its KDoc lists the extras and the report format. MainActivity keeps launch extras for screen recordings and timing runs (--ez autorun true, --ez kv_probe true, --es text "...", --es codec cpu|gpu|auto, and more in its KDoc); each such launch writes its numbers to Android/data/com.audio8tts.realistic/files/Documents/audio8-demo-<epoch>.json.

How it works

The Kotlin code reproduces the model repository's host loop (audio8_tts_litert.py) on the CompiledModel API. The prompt is the vendor's chat format: the reference transcript and its 10 rows of codec codes, then the typed text. The slow AR prefills it in chunks of 256 positions, a last chunk of up to 64 through its prefill_64 signature, and decodes one frame at a time. Its packed weights are kept in one XNNPACK weight cache file in the xnnpack/ folder of the app's cache directory, named by the SHA-256 of slow_ar_int8.tflite and the LiteRT version: the first load writes it, later loads map it, and a load that cannot use it creates the graph without it. Its KV cache length is read from the graph (1,024 positions here); the cache lives in one set of tensor buffers that each call reads and updates in place, so no cache bytes cross JNI per step (a ping-pong mode with two sets remains, kvMode = "pingpong"). For every frame the fast AR makes 10 calls for the 10 codebook codes. Sampling is the vendor's top-k 50, top-p 0.9, temperature 0.7 with a repetition-aware redraw. The codec decoder turns the frames into 44.1 kHz audio: one 128-frame call up to 128 frames, one 192-frame call up to 192, and beyond that 192-frame windows with 128 frames of left context. At load both GPU decoders are created and warmed, and the 128-frame one decodes a test window whose CPU output ships in assets/codec_check_T128.f32; the GPU decodes when the two correlate at 0.99 or more, the CPU otherwise.

File Role
MainActivity.kt the input screen (voice picker, text box, Speak / Cancel, footer with device, runtime, backends, load time and generate / audio = RTF), the microphone recording, playback, and the launch extras
Audio8Tts.kt text + voice → samples and times, shared by the screen and the device check: missing files, voices, text rules, cancellation, a close that may be called twice, the remembered GPU codec failure
Audio8Engine.kt the graphs: prefill and decode with the KV cache in place, the fast AR, the codec decoder windows, the encoder, the vendor sampler
WeightCache.kt the slow AR's weight cache file: its name from the model's SHA-256 and the runtime version, the marker written after a complete load, the removal of stale caches
TextRules.kt empty and too-long texts (49 tokens)
QwenBpeTokenizer.kt byte-level BPE from tokenizer.json, checked at load against the test vectors in assets/prompt_constants.json
PromptConstants.kt the fixed prompt fragments and the tokenizer test vectors
ReferenceAudio.kt silence trim and peak normalization of a recording
Wav.kt, Npy.kt, PlayBar.kt wav files, the voices' codes.npy, the progress and level bar
DeviceCheck.kt, Audio8DeviceCheck.kt the device check above
GateActivity.kt the measurement entry

License

The code in this folder is under the MIT license (LICENSE). The model is Apache-2.0 (Edge0); the sample voices are Apache-2.0 (Fun-ASR-Nano example) and CC BY 4.0 (LibriSpeech). The model card lists the source model's limitations: quality varies by language; noisy reference audio or a mismatched transcript can reduce stability and speaker similarity; speech may contain pronunciation errors, repetitions, omissions or early stopping.