How to Build Everyday Transliteration Data Without Translation Corpora

A translation dataset might pair “See you tomorrow” with a sentence in another language. A Convert model receives a different object: something like kal Zoom pe milte hain, where only part of the message should change script and Zoom must survive exactly.

That difference changes what belongs in the training data.

Translation pairs teach the wrong operation

Translation changes language. Transliteration changes writing system while preserving the intended language and the protected parts of the input.

If a dataset rewards semantic translation, it can teach a model to replace English words that the user deliberately typed, smooth away informal phrasing, or rewrite the sentence. Those may be acceptable translation choices. They are Convert failures.

Everyday input contains more than clean sentences

Real drafts contain fragments and boundaries:

• short replies such as hao de or theek hai;
• names, brands, places, and loanwords;
• @mentions, #hashtags, links, and email addresses;
• code, commands, versions, and filenames;
• emoji, repeated punctuation, and stretched spellings;
• two or more languages in the same line;
• spelling systems that vary from person to person.

The dataset therefore needs cases built around these objects. A clean sentence-only collection can score well while still breaking the text people actually paste into Convert.

Social style is useful, but copying posts is not required

The useful signal in social writing is the shape of the input: abbreviation, code-switching, emoji placement, reply markers, dialect spellings, and incomplete grammar. We can build original examples that reproduce those patterns without copying a person's post into the training set.

Each example should also carry testable constraints. A URL must remain byte-for-byte unchanged. A multiline input must keep its line count. An English-only control must remain English. A target-script input should not be unnecessarily rewritten.

Edge cases are the dataset, not an appendix

For Convert, an edge case is often an ordinary message with one extra constraint. @sara, v2.1, or an Arabic chat numeral can change the correct output. These cases must be present during dataset design and held-out evaluation.

The goal is not to collect the largest pile of sentences. The goal is to cover the decisions the model will face: convert, preserve, or leave unchanged.

Next, we separate Pinyin, Romaji, Hinglish, and Arabizi as four different conversion problems. Product rules for protected content are documented in Privacy and protected content.