What are Tokens?
A model never sees your text. It sees a list of integers. Tokens are the pieces your text is cut into, and almost every limit and price you meet is counted in them.
From Text to Numbers
Four steps sit between a string and the first matrix multiply
Raw Text
What you typed, as a plain string.
"Tokens matter"Tokenizer
Splits the string into known pieces using a fixed vocabulary learned before training.
"Tok" "ens" " matter"Token IDs
Each piece is looked up in the vocabulary and replaced by its integer index.
[9126, 641, 4132]Embeddings
Each ID indexes a row of the embedding matrix — the vector the network actually consumes.
[0.02, -0.41, ...]The reverse runs at the end: the model outputs a score for every ID in the vocabulary, one is chosen using the sampling parameters, and the tokenizer turns that ID back into text.
One Sentence, Cut Into Tokens
Colours mark token boundaries. Note that the leading space usually belongs to the token after it.
Common words are single tokens. Rarer ones ("Understanding", "LLMs") get split into fragments — which is exactly the point. The exact split and the IDs differ per tokenizer; the shape of the result does not.
Why Not Just Use Words?
Subwords are the compromise between two bad extremes
Characters
Every letter is its own token. The vocabulary is tiny and nothing is ever unknown.
- Vocabulary of ~100 entries
- Never fails on a new word
- Sequences become very long
- Attention cost grows with length
- Model must relearn spelling
Subwords
Frequent words stay whole; rare words break into reusable fragments. This is what essentially every modern LLM uses.
- Short sequences for common text
- Any string is representable
- Shares roots: "play" in "playing"
- Splits can be unintuitive
- Poor fit for unseen languages
Whole Words
One token per dictionary word. Short sequences, but the vocabulary can never be complete.
- Shortest sequences
- Token boundaries match meaning
- Huge embedding matrix
- Typos become unknown tokens
- No sharing between word forms
Subword Algorithms
Different ways to decide where the cuts go
BPE
Byte Pair Encoding starts from single characters and repeatedly merges the most frequent adjacent pair into a new token, until the vocabulary reaches its target size.
Used by: GPT-family tokenizers, Llama, and most byte-level variantsWordPiece
Merges the pair that most increases the likelihood of the training corpus rather than the merely most frequent pair. Continuation pieces are marked, classically with a "##" prefix.
Used by: BERT and its descendantsSentencePiece
Treats the raw string — spaces included — as the unit of work, so it needs no language-specific pre-tokenizer. Encodes spaces explicitly, which makes decoding perfectly reversible.
Used by: T5, Llama, Mistral and many multilingual modelsUnigram
Starts from a large candidate vocabulary and prunes it, keeping the pieces that best explain the corpus under a probabilistic model. Can offer several valid segmentations of the same word.
Used by: SentencePiece in its unigram modeBPE, One Merge at a Time
Training on a corpus where "lower", "lowest" and "slow" are common
Two tokens now cover the word, and both are reusable: "low" also serves "lower" and "slow", "est" also serves "largest". At inference the learned merge list is simply replayed in order — no counting happens again.
The Vocabulary Size Trade-off
Bigger vocabulary means fewer tokens per sentence, but a larger embedding matrix
Bar height = tokens needed for the same paragraph
The cost sits at both ends of the network: the embedding matrix and the output layer are both vocabulary × hidden_size. Doubling the vocabulary buys shorter sequences and pays for it in parameters and in a wider softmax at every step.
Tokens Are the Unit of the Context Window
Everything the model can attend to shares one budget
Input and output share it
The window covers the prompt and the generated reply together. Filling it with context leaves no room for an answer.
History is re-sent every turn
A chat is stateless underneath. Each turn resends the whole transcript, so token use per turn grows as the conversation does.
Memory grows with it
Every token in the window holds a slot in the KV cache, which is why long contexts cost VRAM, not just time.
Overflow is not an error
Most tools silently truncate or drop the oldest messages when the budget is exceeded — the model simply stops seeing the start.
Rules of Thumb
Useful approximations when you are estimating cost or length
~4 chars per token
For ordinary English prose, a token averages roughly four characters, or about three quarters of a word. Treat it as an estimate, never as a billing calculation.
Whitespace counts
A leading space is normally part of the following token, so "cat" and " cat" are different IDs. Stray double spaces and odd indentation quietly cost tokens.
Numbers fragment
Long numbers are split into several pieces, and where the cuts land varies by tokenizer. This is a large part of why digit-by-digit arithmetic is awkward for LLMs.
Not all languages are equal
Tokenizers trained mostly on English need more tokens for the same meaning in other scripts, which makes identical text longer and more expensive to process.
Code has its own shape
Indentation, braces and identifiers tokenize very differently from prose. Newer tokenizers add multi-space tokens specifically to make source code cheaper.
Count, do not guess
Each model family has its own vocabulary, so token counts are not portable. Use the tokenizer that ships with the model when the number actually matters.