Netra-NMT

Abstract#

Netra-NMT is a compact 77M-parameter encoder-decoder transformer trained from scratch on 220 million tokens of English–Khmer parallel text (4.2M bidirectional examples). The encoder uses bidirectional self-attention, much like BERT, to capture global contextual representation. The decoder generates autoregressively through causal self-attention and encoder-decoder cross-attention.

The model adopts a modern transformer recipe: rotary position embeddings (RoPE) in self-attention with no learned position table, RMSNorm with pre-normalization for stable optimization, SwiGLU feed-forward networks, and a single embedding table shared across the encoder input, decoder input, and tied output projection.

Khmer text is word-segmented with khmercut before tokenization, so the shared 32K SentencePiece vocabulary sees real word boundaries rather than an unbroken character stream.

Dataset#

Training used roughly 2.4 million unique English–Khmer sentence pairs, combining LLM-generated synthetic data with web-crawled parallel text across legal, literary, medical, technical, and conversational domains.

After deduplication, length filtering, and empty-pair removal, each surviving pair is duplicated in both directions with a direction prefix token (<2km> / <2en>), yielding ~4.2 million training examples.

Two preprocessing details matter. English is lowercased only when it is the en2km source, so "I love Cambodia" and "i love cambodia" map to identical inputs — English kept as a km2en target retains its natural casing. Khmer, written without spaces, is segmented with khmercut wherever it appears, and de-segmented back to natural text at inference time.

Architecture#

The encoder passes the source sentence through 12 transformer layers with bidirectional self-attention. Positions are encoded by rotating queries and keys inside self-attention, so there is no learned position table and the model generalizes past its training length.

The decoder uses only 2 layers. Each applies causal self-attention with RoPE over previously generated tokens (backed by a KV cache for O(T) rather than O(T²) incremental decoding), cross-attention over the full encoder output, and a SwiGLU feed-forward block.

That deep-encoder / shallow-decoder split follows Kasai et al. (2021): most of the modeling capacity lives in the parallelizable encoder, while the thin decoder — the part that runs sequentially at inference — keeps per-step cost low without sacrificing quality.

Results#