How We Evaluate a Multilingual Transliteration Model

A transliteration output can look fluent and still be unusable. If it changes OpenAI API, removes an emoji, rewrites a URL, or adds an explanation around the answer, it cannot safely replace the selected text.

That is why we test the replacement contract, not only whether a reader understands the result.

Exact match measures direct replaceability

Our strict score accepts an output only when its final text matches an accepted answer. This catches extra commentary, altered punctuation, changed English, and small spelling differences.

Exact match is deliberately severe. In Arabic, more than one understandable spelling may exist for a casual dialect phrase. A strict mismatch therefore identifies a review case; it does not automatically prove that every unmatched output is meaningless.

Every language keeps its own score

The frozen suite currently contains 127 cases:

These are project test counts under the same Convert task. The suite is small and cannot establish broad production accuracy. The Arabic result, for example, shows why one total number is not enough.

Protected spans are part of correctness

We check whether the model preserves objects such as:

• English words and product names;
• code and shell commands;
• URLs, email addresses, handles, and hashtags;
• numbers, emoji, punctuation, and line breaks.

We also include English-only inputs and text already written in the target script. A converter should be able to leave text alone.

Held-out cases limit memorization

Evaluation inputs must stay outside the training rows. We keep a frozen suite for comparison over time and a separate edge suite for difficult mixed, social, and dialect cases. A result on one suite is named as that suite's result rather than presented as universal accuracy.

Runtime is part of the tested object

The same model files can behave differently across inference runtimes. For the published 6-bit artifact, direct MLX and LM Studio do not produce identical counts. We therefore use the LM Studio score when describing the desktop integration and keep direct-runtime evidence separate.

The next guide shows how to run Transliteration-4B in LM Studio. Product text-scope behavior is documented in Text scope strategy.