Netra MT 1

Most machine translation systems that cover Khmer treat it as one language among hundreds. Models like NLLB-200 and MADLAD-400 spread their capacity across every language they support, and they are large enough that running them usually means a GPU or a paid API. For a Khmer speaker on an ordinary laptop, or a developer building for one, neither is a good fit.

We wanted to know how far a small model could get if it were built for this one language pair from the start. Netra MT 1 is the result: a 77M-parameter encoder-decoder transformer that translates in both directions between English and Khmer, trained from scratch on about 220 million tokens of parallel text.

In brief:

• English→Khmer. On the ALT test set, Netra MT 1 scores higher than every open model we compared, including NLLB-200 (1.3B) and MADLAD-400 (3B), on spBLEU, chrF++ and TER. Google Translate still leads on most metrics, and the larger open models score higher on COMET.
• Khmer→English. This direction is clearly weaker. Netra MT 1 trails every working baseline by 11.8–16.7 spBLEU and 7–10 COMET points.
• Speed. With greedy decoding on a desktop CPU it translates a sentence in about 66 ms in PyTorch, 16 times faster than NLLB-200 (600M). The released package, on ONNX Runtime, takes 39 ms.

The model, tokenizer, training code and evaluation outputs are all open source under the MIT licence, and the weights are on Hugging Face.

Architecture#

Layer allocation#

A translation model has two halves with very different costs. The encoder reads the whole source sentence at once, so it runs a single time per sentence. The decoder writes the translation one token at a time, so every one of its layers runs again for every token it writes. Following Kasai et al. (2021), we put most of the depth in the encoder: 12 encoder layers and only 2 decoder layers. The model keeps a large capacity for understanding the source, but each generated token costs very little.

On the CPU, this layout cut median greedy latency from 217 ms to 76 ms per sentence compared with our earlier autoregressive baseline, a 2.85× speedup. We had also tried a non-autoregressive diffusion decoder to cut latency. The new model turned out to be faster than it as well, at 76 ms against 114 ms with 10 refinement steps, while keeping ordinary autoregressive decoding.

Direction tokens#

One model handles both directions. Each source sentence starts with a token naming the target language, <2km> or <2en>, and the decoder takes its cue from that. "I love my country", to be translated into Khmer, reaches the encoder as the pieces ▁ <2km> ▁i ▁love ▁my ▁country. That is the example in Figures 2 and 4.

Rotary position embeddings#

Attention by itself has no sense of word order: it compares every token with every other, wherever they sit. The original Transformer adds a position vector to each token. We use rotary position embeddings (RoPE) instead, which turn the vectors rather than adding to them.

Each attention head works on 64-dimensional queries and keys. RoPE splits them into 32 pairs of dimensions and rotates pair $i$ of a token at position $m$ by the angle $m\theta_i$, where

The first pair turns quickly, 57° per position; the last barely moves. Together they act like the hands of a clock, which tell the time from fast and slow hands at once.

Because the query and the key are both rotated, the score between two tokens depends only on how far apart they are:

where $q$ is the query of the token at position $m$, $k$ is the key of the token at position $n$, and $R_m$ rotates every pair $i$ by $m\theta_i$. There is no table of positions to run out of, so the model has no hard limit on input length. RoPE is used in the self-attention of both the encoder and the decoder. Cross-attention, where the target reads the source, has no rotation.

Layer structure#

Every layer has the same two sub-layers as the original Transformer, attention and a feed-forward block, but each is built in the modern way (Figure 3).

The normalisation comes before each sub-layer, and the sub-layer's output is added back to the input:

This pre-norm layout keeps a clean path for the signal through all 12 encoder layers, and it kept training stable at the higher learning rate the deeper encoder needed. RMSNorm is a cheaper form of layer normalisation: it only rescales, without first subtracting the mean,

where $d = 512$ is the model dimension, $\epsilon = 10^{-6}$ avoids dividing by zero, and $g$ is a learned scale for each dimension.

The feed-forward block is SwiGLU. It projects the input twice, and one projection decides how much of the other to let through:

where $W_g$ and $W_v$ map 512 dimensions to 2,048, $W_2$ maps them back to 512, $\sigma$ is the sigmoid and $\odot$ multiplies element by element.

Shared embeddings#

One embedding table of 32,000 pieces × 512 dimensions, 16.4M parameters, is used three times: to look up the source pieces for the encoder, to look up the pieces the decoder has written so far, and as the output layer, which scores every row of the table to choose the next piece. Both languages share one vocabulary, so one table serves all three. Separate tables would add 32.8M parameters and take the model from 77M to about 110M.

KV cache#

In attention, every token is compared with the others through its key and passes on its value. The decoder writes one piece at a time, and at each step its self-attention needs the keys and values of every piece written so far. Recomputing them all at every step repeats nearly all the work. Instead the decoder computes keys and values for its newest piece only, appends them to a cache, and reads the rest. The encoder's keys and values, which cross-attention reads at every step, are computed once per sentence.

In the six steps of Figure 4, each decoder layer computes 6 sets of keys and values instead of 1 + 2 + … + 6 = 21. The saving grows with the length of the sentence. Together with the 2-layer decoder, this keeps the cost of each new piece small.

Tokenization#

Khmer is written without spaces between words. A tokenizer given raw Khmer text has to guess word boundaries from character statistics alone, and those guesses often cut across words.

We segment all Khmer text into words with khmercut before training the tokenizer, and again wherever Khmer appears as model input or output. The SentencePiece vocabulary therefore learns pieces aligned to real words. At inference time the segmentation is removed again, so the output reads as natural Khmer.

We also lowercase English, but only when it is the source of an English→Khmer pair. "I love Cambodia" and "i love cambodia" should get the same translation, so we stop the model from treating them differently. When English is the target, it keeps its original casing, so Khmer→English output comes out properly capitalised.

Data#

We trained on about 2.4 million unique sentence pairs from four public sources. They mix synthetic, model-generated translations with text crawled from the web.

We removed duplicate and empty pairs, and pairs outside our length limits. Each remaining pair is then used twice, once in each direction, which gives about 4.2 million training examples. The cleaned set is published as Netra-Lab/bilingual-en-km.

Training#

We trained on two RTX 4070 GPUs with AdamW. The learning rate warmed up linearly over 2,000 steps to a peak of 3×10⁻⁴, then followed a cosine decay. We used an effective batch of 128 sentence pairs, label smoothing of 0.05 and mixed precision. The full configuration is in 09_train_student.py.

Results#

We evaluated on the test split of the Asian Language Treebank (ALT): 1,018 sentences, professionally translated, with one reference per sentence. Every system was decoded greedily, and every system was scored with the same metric settings. We compared against four open models, from 350M to 3B parameters, and against Google Translate.

English→Khmer#

Overlap metrics. Netra MT 1 leads every open model on spBLEU, chrF++ and TER, by 7 spBLEU points over the next best. It also has the lowest TER of all six systems, Google Translate included.

Learned metrics. These are less clear-cut. BERTScore puts us second, behind Google Translate. But on COMET, which is designed to track human judgments, NLLB-200 (1.3B) and MADLAD-400 score 1.5–2 points higher than we do. Our outputs share more surface wording with the references, but we can't say from these numbers alone that they are better translations.

Khmer→English#

This is the model's clear weakness. Every working baseline beats it on every metric, by 11.8–16.7 spBLEU points. We think the main reason is the English side. The large multilingual models learned English from vastly more text than our 2.4M pairs contain, and a 2-layer decoder has little room to make up for that. Fixing this direction is our first priority for the next version.

Speed#

Greedy decoding. On the CPU, Netra MT 1 is 16 times faster than NLLB-200 (600M) and 8 times faster than the 350M English→Khmer model. On the GPU the gap to NLLB-200 (600M) is about tenfold.

Beam search. In this measurement the advantage mostly disappears with five beams. The decoder does five times the work at every step, so the cheap encoder no longer dominates, and on the GPU the 350M model is faster than ours. Greedy decoding is the package's default.

ONNX Runtime#

The released package, netra-nmt 1.0.0, runs the model on ONNX Runtime instead of PyTorch, and its beam search stops early.

Same output. The ONNX model generates exactly the same tokens as the PyTorch model we evaluated, on all 1,018 ALT test sentences, in both directions, with greedy and beam-5 decoding. Every score above holds for the package.

Beam search is 11 times faster. The old beam search ran all 128 steps for every sentence, because it stopped only when every beam had finished. The new one stops as soon as no unfinished beam can still beat the best finished translation. A beam's score can only fall as it grows, so stopping never changes the result: on the ALT test set, early-stopped and full beam search give identical output on all 1,018 sentences in both directions. GPU speed on ONNX Runtime hasn't been measured yet.

Limitations#

• One test set. ALT is a single benchmark of news-style sentences with one reference translation each. Scores on legal, medical or conversational text may differ, and we haven't run a human evaluation.
• Overlap metrics favour us more than COMET does. For English→Khmer, the lexical metrics and the learned metric don't agree on how we compare with the larger models. The honest summary is that we are competitive with them, not that we are better.
• Synthetic training data. At least a fifth of the training pairs were generated by other models. Their errors and stylistic habits are likely to carry over into ours.
• Short inputs. Training sentences were cut to 256 tokens. RoPE lets the model accept longer input, but we haven't measured how quality holds up on long passages, and documents should be translated sentence by sentence.
• Khmer→English. Quality in this direction, covered above, is well behind the alternatives.
• Rare names and terms. These get mistranslated. In the first ALT test sentence, Khmer→English renders “Randwick Racecourse” as “Trophystic Racing Square”.
• Training data terms. Three of the four training sources are third-party and keep their own terms. Check them before using the model commercially.

Usage#

The model is on Hugging Face, and you can try it in the playground. The Python package runs it on ONNX Runtime, without PyTorch, and downloads the 189 MB model on first use:

pip install "netra-nmt[cpu]"    # or "netra-nmt[gpu]" for an NVIDIA GPU

from netra_nmt import translate, NetraTranslator

translate("Hello, how are you?")                               # English→Khmer by default
translate("ខ្ញុំស្រឡាញ់ប្រទេសរបស់ខ្ញុំ។", direction="km2en")

t = NetraTranslator()                                          # one model, kept loaded
t.translate("Good morning, my friend.", mode="beam", beam_size=5)

The package also has a command-line tool, netra-nmt, and a local web app and REST API, netra-nmt serve, which come with pip install "netra-nmt[server,cpu]". A Docker image with the model inside runs offline:

docker run --rm -p 8000:8000 ghcr.io/netra-ai-lab/netra-nmt:latest

To fine-tune the model on your own sentence pairs, one English–Khmer pair per line, the repository has one script for each step:

bash training/setup.sh
bash training/check_data.sh my_pairs.tsv
bash training/finetune.sh my_pairs.tsv --name my-model
bash training/evaluate.sh published training/runs/my-model/checkpoints/best/checkpoint.pt
bash training/export.sh training/runs/my-model/checkpoints/best/checkpoint.pt

Future work#

Each of these follows from a limitation above:

• Khmer→English. Closing the gap to the larger models, most likely through more English-side data and a deeper decoder for this direction.
• More varied test sets. Legal, medical and conversational text, alongside ALT's news-style sentences.
• Human evaluation. Asking Khmer speakers to judge the translations directly, rather than relying only on automatic metrics.

References#

1. Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
   Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, Noah A. Smith.
   ICLR 2021.
   arXiv:2006.10369

2. RoFormer: Enhanced Transformer with Rotary Position Embedding
   Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu.
   Neurocomputing, 2024.
   arXiv:2104.09864

3. Root Mean Square Layer Normalization
   Biao Zhang and Rico Sennrich.
   NeurIPS 2019.
   arXiv:1910.07467

4. GLU Variants Improve Transformer
   Noam Shazeer.
   2020.
   arXiv:2002.05202

5. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
   Taku Kudo and John Richardson.
   EMNLP 2018, System Demonstrations.
   arXiv:1808.06226

6. No Language Left Behind: Scaling Human-Centered Machine Translation
   NLLB Team, Marta R. Costa-jussà, James Cross, et al.
   2022.
   arXiv:2207.04672

7. MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
   Sneha Kudugunta, Isaac Caswell, Biao Zhang, et al.
   NeurIPS 2023, Datasets and Benchmarks.
   arXiv:2309.04662

8. COMET: A Neural Framework for MT Evaluation
   Ricardo Rei, Craig Stewart, Ana C. Farinha, Alon Lavie.
   EMNLP 2020.
   arXiv:2009.09025

9. BERTScore: Evaluating Text Generation with BERT
   Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi.
   ICLR 2020.
   arXiv:1904.09675

10. chrF++: words helping character n-grams
    Maja Popović.
    WMT 2017.
    ACL Anthology W17-4770

11. Introduction of the Asian Language Treebank
    Hammam Riza, Michael Purwoadi, Gunarso, et al.
    Oriental COCOSDA 2016.
    Dataset

12. Seanghay Yath. khmercut: Khmer word segmentation [Software]. GitHub.

@misc{netralab2026netramt,
  title  = {{Netra MT 1: A compact bidirectional encoder-decoder transformer for English–Khmer translation}},
  author = {{Netra Lab}},
  year   = {2026},
  url    = {https://netra-lab.example/research/netra-mt}
}