Tokenization

Tokenization is the process of converting a sequence of raw text (or other input modality) into a sequence of discrete symbols called tokens, each of which is mapped to an integer identifier and then to a learned vector representation that a neural network can process. Tokenization is the first and most fundamental step in how a large language model (LLM) receives input: because neural networks operate on numerical vectors rather than characters or words directly, every piece of text must be broken into tokens before the model can act on it. The choice of tokenization scheme directly affects a model's efficiency, its multilingual capability, its handling of rare words, and the effective cost of using it through an API.^[1]

How Tokenization Works

A tokenizer segments input text into units drawn from a fixed vocabulary — a list of every token the model recognises. Each token is assigned an integer ID. For example, the sentence "the cat sat" might be tokenised as ["the", " cat", " sat"] with IDs [464, 3756, 3299]. At inference time, the model reads these IDs, maps each to a learned embedding vector, and processes the sequence through its transformer layers.

A key property of modern tokenizers is that they produce tokens of variable length in characters — a single token may represent a whole word ("cat"), a part of a word ("un", "believe", "able"), a single character, or even a punctuation mark. This subword approach balances two extremes: whole-word tokenisation (which requires an enormous vocabulary and cannot handle new or misspelled words) and character-level tokenisation (which produces very long sequences that are expensive to process).

Major Tokenization Algorithms

Byte Pair Encoding (BPE)

Byte Pair Encoding is the most widely used tokenization method in modern LLMs. Originally a data-compression technique, BPE iteratively merges the most frequent adjacent symbol pairs in a training corpus to build a vocabulary. The result is a vocabulary that contains common words as single tokens and breaks rare words into frequent subword pieces. GPT-2, GPT-3, GPT-4, and most models in the Llama family use BPE-based tokenizers.^[2]

WordPiece

WordPiece, developed by Google, is similar to BPE but selects merges based on maximising the likelihood of the training corpus under a language model. It is the tokenization method behind BERT and its derivatives. WordPiece tokens are prefixed with ## to indicate continuation of a word (e.g., "playing" becomes "play" + "##ing").^[3]

Unigram and SentencePiece

Google's SentencePiece library implements the Unigram Language Model algorithm, which starts from a large set of candidate tokens and prunes them probabilistically to reach the target vocabulary size. SentencePiece treats input as a raw stream of Unicode characters without pre-tokenising on whitespace or punctuation, which makes it well-suited to languages such as Chinese, Japanese, and Korean that do not use spaces between words. SentencePiece is used by T5, mT5, ALBERT, XLNet, and several Llama variants.

Tiktoken

tiktoken is an open-source BPE tokeniser library released by OpenAI, designed for fast tokenization of text for GPT models. It is the tokeniser used for GPT-3.5, GPT-4, GPT-4o, and o-series models, and is the reference implementation for counting tokens to estimate API costs.^[4]

Tokens and Pricing

Because commercial LLM APIs are priced per token, tokenization has direct financial implications. A single English word typically corresponds to roughly 1.0 to 1.5 tokens, but this ratio varies significantly by language and script. Non-Latin scripts are generally less efficient: the same meaning expressed in Chinese, Arabic, or Korean often requires two to four times as many tokens as the English equivalent under tokenizers trained predominantly on English text. This tokenisation tax is a concrete operational concern for multilingual applications and is a motivation for building tokenizers with broad multilingual coverage.^[5]

Effect on Multilingual Performance

The quality of a tokenizer's coverage of a given language has a measurable effect on model performance in that language. If a language's script is poorly represented in the tokenizer's training data, words in that language are split into many small fragments, which both increases cost (more tokens) and degrades model understanding (less semantic signal per token). This is one reason that building models with dedicated tokenizers for specific languages — as seen in DeepSeek (optimised for Chinese), Aya (multilingual), and MalLam (Bahasa Malaysia) — can significantly improve performance on those languages relative to using a general-purpose tokenizer.

>See Also

• Embedding
• Transformer Architecture
• Context Window
• Large Language Models
• MalLam Malaysian LLM

References