Qwen vs Gemma for Transliteration: 4B Test Results

We chose Qwen for the first UnimeType transliteration model because it scored higher in the Swift Convert evaluator, and the Qwen-based training path produced our best tested artifact. The untuned Qwen candidate reached 66/127 accepted outputs, the untuned Gemma candidate reached 33/127, and Transliteration-4B reached 96/127.

These results come from the same 127-case frozen suite and Swift Convert evaluator. The evaluator applies the full language-specific prompt, protected-fragment handling, output validation, and deterministic local conversions. Because the tested artifacts use different quantization formats and LM Studio backends, the results support a product choice; they do not prove that Qwen is universally better than Gemma.

Qwen vs Gemma model cards: useful signals, not Convert scores

Qwen3.5-4B is published under Apache 2.0, and its model card states coverage of 201 languages and dialects. Gemma 3 describes coverage of more than 140 languages and reports multilingual evaluations, including Indic benchmarks.

Those facts make both models worth testing. They do not measure whether ni jintian you kong ma becomes the expected Chinese sentence, whether Romaji produces the intended Japanese script, or whether Arabizi preserves a username beside Arabic text. Language counts and general benchmarks are candidate signals, not Convert accuracy.

Qwen vs Gemma results on 127 Convert cases

We ran both untuned 4B models and the selected Transliteration-4B artifact in LM Studio through the same Swift product evaluator. An output passed only when the restored final text exactly matched one of that case’s accepted outputs and its route matched the case contract.

The first two rows compare installable product candidates, not isolated architectures: Qwen used GGUF Q4_K_M, while Gemma used MLX QAT 4-bit. The third row reports the selected Qwen-based artifact. Because training, quantization, and backend changed together, this table does not isolate fine-tuning as the sole cause of the difference or prove that a trained Qwen always beats a trained Gemma.

Exact match is also intentionally strict. A readable alternative spelling can still fail if it is not one of the accepted outputs. The suite is large enough to expose concrete failures, but too small to claim universal language accuracy.

Language results: Qwen led all four tested slices

Chinese was the clearest base-model difference in the Swift evaluator: untuned Qwen passed 33 cases and untuned Gemma passed 16. Qwen also led the Japanese, Hindi, and Arabic slices. Transliteration-4B produced the highest result in every slice, including 22/22 on the current Hindi cases.

Arabic remains the weakest slice for both untuned candidates. Arabizi can represent dialect words and borrowed words with several plausible spellings. A model can therefore produce understandable Arabic while missing a strict accepted output. We keep the strict score because Convert must replace text directly, but the 6/22 and 4/22 results need case review rather than a claim of broad Arabic accuracy.

Hindi is the only Indic language in this frozen suite. Gemma’s published Indic evaluation coverage made it a serious candidate, but our Hindi slice—not a general Indic benchmark—is the evidence available for the current product target.

Why UnimeType chose Qwen over Gemma

We chose Qwen because its untuned candidate scored higher in the same evaluator, and the completed Qwen-based artifact produced the highest result in this local comparison: 96/127. These scores describe three tested artifacts; they are not an architecture-controlled measurement of training gain.

Apache 2.0 was another practical advantage for a public derivative model. Gemma is available under the Gemma Terms of Use, which include specific notice and redistribution obligations. That difference affects packaging and distribution work; it does not make a claim about linguistic quality.

Gemma remains a credible candidate, but we have not trained it under matched conditions. This comparison therefore cannot answer whether a similarly trained Gemma artifact would close the gap while meeting the same runtime and distribution requirements.

Why Phi-4-mini was not a first-round candidate

Before training a candidate, we first check whether its documented language coverage includes all four target languages. Microsoft’s Phi-4-mini-instruct model card lists Arabic, Chinese, and Japanese among its supported languages, but does not list Hindi. That makes Phi a lower-priority candidate for this four-language release. It does not prove poor Hindi accuracy; we have not run the same test on Phi.

The deciding comparison is whether each trained candidate passes the same Convert inputs, preserves protected text, and runs within the target local environment.

Want to try the selected model? Follow How to run Transliteration-4B in LM Studio. For the decisions behind it, compare 1B, 2B, 4B, or 27B and read How we evaluate a multilingual transliteration model. The product’s supported input patterns are listed in Input languages.