Sota

Khmer is the national language of Cambodia, and it is spoken alongside English in much of daily life. Meta's Omnilingual ASR recognises speech in more than 1,600 languages, Khmer among them, and its 1B-parameter model is a strong starting point. But a model that has to cover 1,600 languages carries far more capacity than two languages need, and running it costs memory and time.

Sota (សោត, "the ear") keeps what Khmer and English need and removes the rest. We fine-tune Omnilingual ASR on the two languages, then let the model learn which of its attention heads and feed-forward units it can do without, and cut 80% of its encoder away. The result is two compact speech recognisers: sota-large, with 204M parameters, and sota-base, with 73M.

In brief:

• Accuracy. On the DDD Khmer test set, sota-large gets 1.77% of characters wrong, against 2.47% for the 975M-parameter model it was pruned from. sota-base gets 2.60%, against 3.68% for its 325M-parameter teacher.
• Size and speed. sota-large needs 12.3 GMAC per second of audio, against its teacher's 50.3, and runs 2.5 times faster on the test sets. sota-base needs 5.8 GMAC/s and runs 1.8 times faster.
• Two languages. Both models transcribe Khmer and English. English is the weaker side: pruning costs it more, and on LibriSpeech test-clean the pruned models make 1.7–1.9 times as many word errors as their teachers.

The code, the training recipe and both models are open source under the Apache 2.0 licence.

Architecture#

Base model#

Sota starts from the CTC models of Meta's Omnilingual ASR: omniASR_CTC_1B for sota-large and omniASR_CTC_300M for sota-base. Both are wav2vec 2.0 encoders, pre-trained on unlabelled audio and then fine-tuned to transcribe:

• a 7-layer convolutional feature extractor that turns the waveform into one frame every 20 ms;
• a convolutional position encoder;
• pre-LayerNorm transformer layers, 48 at width 1,280 in the 1B and 24 at width 1,024 in the 300M;
• a linear CTC head over Omnilingual's 9,812-character vocabulary, which predicts every output position at once in a single pass.

wav2vec 2.0 learns from audio with no transcripts. Spans of the latent frames are masked, and at each masked frame the transformer's output has to pick out the true frame, in a quantized form, from distractors. To do that, it has to learn what speech around a gap sounds like.

Structured pruning#

Removing single weights makes a model sparse without making it faster, because the matrix multiplications keep their shapes. Sota removes whole structures instead: attention heads, and units of the feed-forward network (FFN). Dropping a head or an FFN unit shrinks the matrices themselves, so the smaller model runs faster on ordinary hardware. The model dimension, the convolutional front end and the CTC head are left intact.

To decide what to remove, every head and every FFN unit gets a learned gate $z$, after the L0 relaxation of Louizos et al. (2018). A Hard Concrete gate is sampled in each forward pass:

Here $u$ is drawn uniformly from $(0, 1)$, $\log\alpha$ is the gate's learned parameter, $\beta$ is a temperature, and $\gamma < 0$ and $\zeta > 1$ stretch $s$ so that the clipped $z$ is exactly 0 or exactly 1 with non-zero probability. The output of each head or unit is multiplied by its $z$. The gates can be trained by gradient descent, and the expected number of open gates is a smooth function of $\log\alpha$.

Sparsity constraint#

The expected share of the encoder that the gates remove, $\rho$, is pushed towards a target $t$ by an augmented Lagrangian term added to the loss:

The multipliers $\lambda_1$ and $\lambda_2$ are trained by gradient ascent, so the penalty grows for as long as $\rho$ misses $t$. The target $t$ ramps linearly from 0 to 80% over the first quarter of the pruning run and then holds, which lets the model give up units gradually.

Distillation#

While the gates learn, the pruned model is taught to match the fine-tuned model it started from, which is kept frozen as a teacher. The loss compares the two models' hidden states at five layers:

Here $h_k$ and $h_k^{T}$ are the student's and the teacher's hidden states after layer $k$, $\operatorname{L1}$ is their mean absolute difference and $\cos$ their cosine similarity. $K$ is layers 0, 12, 24, 36 and 48 in the 1B and 0, 6, 12, 18 and 24 in the 300M. The last target is taken after the final LayerNorm, where the CTC head reads, because the raw residual stream of a pre-LayerNorm model has an arbitrary scale at the top.

The recipe follows Fine-tune Before Structured Pruning (Han et al., 2025), which pruned WavLM for speaker diarization and itself builds on DPHuBERT. Its key step is the first one: fine-tune the full model on the target task before pruning, so that the teacher, and the starting point, already know the task.

Training runs in five steps.

1. Fine-tune. The full model is trained with CTC on Khmer and English. It becomes both the starting point for pruning and the frozen teacher.
2. Prune. The gates and the model train together, under the distillation loss and the sparsity constraint, until 80% of the encoder is gated off.
3. Cut. The gates are fixed and folded into the weights, and the closed heads and units are deleted. Each layer now has its own width.
4. Distil with the mask frozen. The smaller model keeps learning to match the teacher's hidden states, with nothing left to prune.
5. Recover. The model is trained with CTC again, plus a frame-level KL divergence to the teacher's output distribution.

Pruning costs accuracy, and the last two steps win it back. On the DDD dev set, sota-large's error rate goes from 3.40% after fine-tuning to 7.06% after pruning, 5.60% after distillation and 2.40% after recovery, below the teacher's. English recovers less: on LibriSpeech dev-clean the same steps take it from 3.95% to 12.97%, 8.46% and 6.60%.

Data#

Sota was trained on about 1,537 hours of speech, mixed 50/50 between Khmer and English by audio seconds.

FLEURS share. At its natural share FLEURS would be only 1–2% of the audio. In early tests, FLEURS-km dev error rose from 10.7% to 13.1% in the first thousand steps of fine-tuning, as the model drifted towards DDD's style. We therefore fix FLEURS at 20% of each language's audio and repeat its clips as needed.

Preprocessing.

• The DDD test set, a DDD dev pool and an exclude list are removed from the training data.
• Utterances shorter than 1 s or longer than 25 s are dropped, and all audio is resampled to 16 kHz mono.
• Text is normalised to NFC, zero-width characters are removed and whitespace is collapsed. English is lowercased.
• Text is filtered to the Omnilingual character vocabulary, which has no Khmer punctuation such as ។. This drops 1.2% of DDD characters.
• Each utterance is normalised to zero mean and unit variance, as in Omnilingual ASR.

Training#

Both sizes use the same five steps and the same schedule.

Fine-tuning uses a learning rate of 10⁻⁵ for the encoder and 10⁻⁴ for the CTC head. Pruning uses 2 × 10⁻⁴ for the model and 2 × 10⁻² for the gates and the multipliers, and recovery 5 × 10⁻⁵. Batches are filled by audio length, up to 160 seconds of audio each. To fit the 1B model's training in GPU memory, its fp32 master weights and AdamW moments are kept in CPU memory while the GPU computes in bf16. The full configuration is in training/configs/.

Results#

We measure character error rate (CER) on Khmer and word error rate (WER) on English: the share of characters, or words, that are substituted, dropped or inserted, against the reference. Khmer is written without spaces between words, so where one word ends and the next begins is a choice made by a segmenter, and word error rate would depend on that choice. All scores use greedy CTC decoding with no language model, on the full test sets: 1,993 utterances of DDD, 771 of FLEURS-km, 647 of FLEURS-en, 2,620 of LibriSpeech test-clean and 2,939 of test-other.

On DDD, both pruned models beat both larger models. sota-large's error rate is 0.7 points below its teacher's and less than a third of the stock 1B's. DDD is 80% of the Khmer training audio, so it is also the set the models know best.

On FLEURS-km, pruning gives back what fine-tuning gained. The pruned models end close to the stock models, 0.1 points behind them, and 2–4 points behind their teachers.

English pays the most. On LibriSpeech test-clean the pruned models make 1.7–1.9 times as many word errors as their teachers, and on FLEURS-en 1.5–1.6 times as many. The capacity that survives the cut is spent mostly on Khmer.

Speed. A fifth of the parameters does not give five times the speed: on a single long utterance the pruned models are 1.6× and 2.2× faster. Measured as the real-time factor over the batched test sets, the speed-ups are 1.8× for sota-base and 2.5× for sota-large.

Ablation#

To see which parts of the recipe matter, we trained sota-large four times, adding one change at a time. Each run keeps everything from the one before it:

1. Prune the stock model. Skip fine-tuning: prune Omnilingual's 1B as released, with the stock model as the teacher and no FLEURS in the data. Recover with CTC alone for 8,000 steps.
2. + Fine-tune first. Fine-tune the 1B on Khmer and English before pruning, and use that fine-tuned model as the teacher. FLEURS joins the data at the same time.
3. + Output distillation. During recovery, also match the teacher's output distribution, not just the transcripts.
4. + Longer recovery. Recover for 16,000 steps instead of 8,000. This is the released sota-large.

Fine-tuning first matters most for the sets that are not DDD. Pruning the stock model already gives a good DDD score, 2.20%, but FLEURS is far worse: 16.36% on Khmer against 12.51% when we fine-tune first, and 28.07% on English against 21.16%. This step changes two things at once, the teacher and FLEURS in the data, and we haven't separated their effects.

Output distillation improves every test set. With transcripts alone, LibriSpeech error crept upwards as recovery went on. Also matching the teacher's outputs stopped that drift, because the teacher still knows the English the small model is losing.

Longer recovery helps again on every set, most of all on DDD, which falls from 2.09% to 1.77%. The smaller sota-base was still improving when its recovery ended, so longer runs may help further.

What pruning keeps#

The gates do not prune every layer evenly. Figure 4 shows what each of sota-large's 48 layers kept.

• Width follows the distillation targets. FFN width climbs towards each layer matched to the teacher and collapses right after it. Layer 12 keeps 3,355 of its 5,120 FFN units; layer 13 keeps 146.
• The deepest layers lose their attention. Layers 38–47 keep no attention heads at all, so they only transform each frame on its own, through their FFN units.
• The same at both sizes. sota-base shows the same pattern: FFN width peaks at its matched layers 6, 12, 18 and 24, and layers 21–24 keep no heads.

Limitations#

• English is the weak side. Pruning costs English far more than Khmer. Use sota-large, or a fine-tuned teacher, when English accuracy matters.
• No punctuation or casing. Output is lowercase with no punctuation, because the Omnilingual character vocabulary has no Khmer punctuation.
• Greedy decoding only. There is no language model or beam search.
• One pass per file. Each file is decoded in a single pass, so long recordings should be split into utterances first. Training used clips of 1–25 s.
• Two languages only. The models were trained and tested on Khmer and English. The other languages Omnilingual supports will have degraded.
• Prepared speech. DDD is curated recordings on set topics, and FLEURS and LibriSpeech are read aloud. Conversational speech, noisy recordings and Khmer–English code-switching within a sentence are untested.

Usage#

Install the package from GitHub:

pip install git+https://github.com/netra-ai-lab/SOTA

The first run downloads the weights from Hugging Face (sota-base, 292 MB; sota-large, 814 MB). A CPU is enough, and CUDA is used when available.

import sota

asr = sota.load_model("sota-base")              # or "sota-large"
print(asr.transcribe("speech.wav"))             # one file -> one string
print(asr.transcribe(["a.wav", "b.flac"]))      # a list -> a list, batched

From the command line, each file prints as one line, the path and then the text:

sota transcribe speech.wav other.flac --model sota-large

The repository also holds the whole training recipe, from data preparation to export, in one resumable script, along with the fine-tuned teachers (1B, 300M).

Future work#

Sota is the speech recognition half of a planned joint model for transcription and speaker diarization, in the spirit of DiariZen's pruned WavLM. Fine-tuning for CTC removes speaker information from the top layers, so the next step is to re-adapt the pruned encoder, or re-prune from a diarization-tuned teacher. We also want to close the English gap, test on conversational and code-switched speech, and put Sota in the playground so anyone can try it.

References#

1. Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages
   Omnilingual ASR Team, Meta AI.
   2025.
   arXiv:2511.09690

2. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
   Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.
   NeurIPS 2020.
   arXiv:2006.11477

3. Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
   Jiangyu Han et al.
   Interspeech 2025.
   arXiv:2505.24111

4. Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models
   Jiangyu Han et al.
   2025.
   arXiv:2506.18623

5. DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
   Yifan Peng, Yui Sudo, Shakeel Muhammad, and Shinji Watanabe.
   Interspeech 2023.
   arXiv:2305.17651

6. Learning Sparse Neural Networks through L0 Regularization
   Christos Louizos, Max Welling, and Diederik P. Kingma.
   ICLR 2018.
   arXiv:1712.01312

7. FLEURS: Few-Shot Learning Evaluation of Universal Representations of Speech
   Alexis Conneau et al.
   IEEE SLT 2022.
   arXiv:2205.12446

8. LibriSpeech: An ASR Corpus Based on Public Domain Audio Books
   Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.
   ICASSP 2015.
   OpenSLR 12

9. Khmer ASR Cultural Dataset
   Digital Divide Data.
   Hugging Face Datasets.
   Digital-Divide-Data/khmer-speech-dataset

@misc{netralab2026sota,
  title  = {{Sota: Khmer and English speech recognition, pruned to a fifth of its size}},
  author = {{Netra Lab}},
  year   = {2026},
  url    = {https://netra-lab.example/research/sota}
}