Can a text-pretrained multilingual model be taught to translate speech to speech — for a language pair with almost no parallel speech data?
That is what this organization holds: a LoRA-adapted Cohere2 backbone driving a frozen Moshi depth decoder over Mimi audio codes, with a text "inner-monologue" running alongside the audio stream. Everything is here — the text corpus, the synthetic speech, the encoded tokens, the model, and every training checkpoint.
The full-corpus run covered 76,250 steps / 2.07 epochs on a Cloud TPU v6e-16, reaching a best validation composite of 2.8199 at step 76,000. Read honestly, capability arrived in a clear order — language identity → text translation → audio synthesis — and the run bought the first two:
what we measured (@best, 500 held-out rows, greedy) |
hi→tr | tr→hi |
|---|---|---|
| free-run text chrF++ — the inner-monologue | 25.7 | 25.1 |
| generated-audio ASR-chrF++ | 3.7 | 9.6 |
| ground-truth-audio ceiling for the same pipeline | 92.1 | 86.6 |
| BLASER-2.0 QE (ASR-free speech semantics, 1–5) | 2.53 | 2.49 |
The model learns the translation mapping; intelligible audio synthesis is the frontier. The gap between 3.7/9.6 and the 92/87 ceiling is not a broken pipeline — that ceiling was measured through the same codec and ASR — it is the frozen depth decoder. BLASER-2.0 sees more meaning in the generated speech than a transcript recovers, which says the audio carries real signal that is simply not yet intelligible.
We publish the number next to its ceiling every time. A low score with no ceiling beside it is unreadable.
| 🎧 Model | tr-hi-s2st-v0.3 — ~89-checkpoint suite + playable audio |
| 📊 Full results | v0.3 evaluation report |
| 🗂 Training corpus | tr-hi-mimi-encoded |
| 💻 Code | github.com/tiny-aya-simultaneous-translation |
| ✍️ Write-up | Adapting Moshi for Low-Resource Speech Translation |
| 📈 Training run | W&B xzcb60bl · emergence report |
tr-hi-parallel-text text triples (en pivot -> tr / hi)
| TTS
tr-hi-parallel-speech-v2 synthetic speech + round-trip ASR quality signals
| Mimi encode
tr-hi-mimi-encoded 8 codebooks @ 12.5 Hz + word-level alignments
| Stage-2 training
tr-hi-s2st-v0.3 the released model
v0.1
and v0.2 are kept
as honest history, including the disclosed mistakes: v0.2 trained on the wrong
dataset via a launcher default, and both trained audio-only because a filename
check wrongly concluded the corpus had no text alignments. It ships 840,426 of
them.fleurs-tr-hi-mimi-encoded
(real human speech; acoustic shift only — its texts overlap training and it
must never be called held-out text),
lahaja-eval (Hindi, dialect-labelled),
cv-tr-eval (Turkish).dualcodec-hindi-25hz
and dualcodec-turkish-25hz:
can a neural codec fine-tuned on a low-resource language beat the stock
multilingual one? Directly relevant, since codec quality is what caps v0.3.Training speech is entirely synthetic TTS, and the model is measurably distribution-bound — on real human speech the text score collapses from ~25 to ~8 chrF++. Translation references are machine-translated, so they carry their own ceiling. This is a research artifact for low-resource S2ST, not a production translator.
Released weights are CC-BY-NC-4.0 — non-commercial, inherited from
CohereLabs/tiny-aya-base.
The training and evaluation code is Apache-2.0. Moshi/Mimi components are
CC-BY-4.0. Corpus text derives from FLORES (CC BY-SA 4.0 — share-alike
propagates), OPUS-100 and TTS output; check each dataset card before
redistributing.
Compute for the v0.3 run was provided by Google's TPU Research Cloud (TRC).