Tokenization
The process of breaking text into smaller units called tokens — sub-words, characters, or symbols — that a language model can process numerically.
What Is Tokenization?
Tokenization is the first step in how a language model reads text. Before a model can process a sentence, the text must be split into discrete units called tokens and mapped to integer IDs. A token is often a sub-word fragment rather than a whole word: common words may be a single token, while rare or long words are split into several. For example, "tokenization" might become "token" + "ization", and a typical English word averages about 1.3 tokens.
Most modern LLMs use sub-word tokenization algorithms such as Byte Pair Encoding (BPE), WordPiece, or SentencePiece. These algorithms learn a vocabulary of frequent character sequences from the training corpus, balancing vocabulary size against sequence length. The result is a fixed vocabulary (often 50,000–200,000 tokens) that can represent any text, including unseen words, by composing known sub-word pieces.
Why It Matters
Tokenization quietly governs cost, speed, and capability in every LLM application. API pricing is per token, context windows are measured in tokens, and latency scales with token count. Understanding tokenization explains why code, JSON, and non-English languages consume more tokens per character, why exact character counting fails, and why a model sometimes struggles with spelling, arithmetic, or rhyming — tasks that depend on character-level detail the tokenizer hides.
Real-World Examples
Token Counting for Cost
A developer estimating API cost uses a tokenizer library (like tiktoken) to count tokens in a prompt, since a 1,000-word document can range from 1,300 to 2,000+ tokens depending on language and formatting.