Tokenization (Natural Language Processing)

Tokenization is the process of breaking text into smaller units, called tokens, that a language model can convert into numbers and process. Every modern AI language system — from chatbots to translation tools — reads and writes through a tokenizer, and the design of that tokenizer quietly determines how much text fits in a model's context window, how much an AI feature costs to run, and how well the system handles languages other than English.^[1][2]

Background

Early natural language processing systems split text on spaces and punctuation, treating words as the atomic unit. That approach breaks down for languages without spaces between words, for compound words, and for the many forms a single root word can take. The neural era replaced fixed rules with subword tokenization: algorithms such as byte-pair encoding, originally developed as a data-compression technique, learn from a training corpus which character sequences occur together often enough to be merged into single tokens. WordPiece, used in BERT, and SentencePiece, designed to be language-agnostic, followed; byte-level schemes adopted by GPT-family models guarantee that any text — including emoji, mixed scripts and code — can be represented as a sequence of tokens.^[1][3]

How It Works

A tokenizer is trained separately from the model itself. It builds a vocabulary — typically tens of thousands to a few hundred thousand entries — of the most useful words, word fragments and bytes, then applies it consistently at both training and run time. Frequent English words such as "the" or "model" usually occupy a single token; an unusual name might be split into three or four fragments.

The efficiency of that split is measured by fertility, the average number of tokens a word or sentence requires. English text commonly averages roughly one token per short word, but the same content in other languages can consume several times as many tokens. An often-cited illustration from OpenAI's developer community notes that for some tokenizers a message in one language can require ten to twenty times more tokens than a comparable English message; studies of Indian and Southeast Asian languages have found similar gaps, and tokenizer choice has become a recognised axis of multilingual AI research.^[2][3][4]

Applications and Impact

Because commercial AI services bill by the token, tokenization has direct economic consequences: text that tokenizes heavily costs more to process, fills context windows faster and can make a chatbot in a lower-resource language several times more expensive to operate than the English equivalent. Developers building retrieval-augmented systems, translators or local-language assistants therefore treat tokenizer behaviour as an engineering variable — choosing models with more efficient coverage, compressing prompts, or fine-tuning on domain text so that common phrases become compact tokens.^[1][4]

Tokenization also shapes fairness. A model whose tokenizer fragments a language unnaturally sees that language in rougher pieces, which can degrade quality on tasks such as translation, summarisation and sentiment analysis. Researchers have proposed frameworks for evaluating and adapting tokenizers for under-served language families, including a 2026 conceptual framework examining efficiency trade-offs in adapting large language models for ASEAN languages.^[1][2]

>See Also

• Large Language Models
• Word2Vec
• Context Window
• Machine Translation
• ILMU (Malaysian Large Language Model)
• MaLLaM (Malaysia Large Language Model)
• Fine-Tuning