Tokenization Deep Dive: How LLMs Actually Process Text

What Is a Token?

LLMs do not process characters or words — they process tokens: subword units that represent common character sequences. "Tokenization" is the process of splitting raw text into these units. GPT-4 uses roughly 1 token per 4 characters for English text, or about 750 words per 1000 tokens.

How Byte-Pair Encoding (BPE) Works

Most modern LLMs use BPE or a variant. The algorithm starts with individual characters and repeatedly merges the most frequent adjacent pair until reaching the target vocabulary size (typically 32K–128K tokens):

# Simplified BPE example
corpus = "low lower lowest new newer newest"
# Initial: l,o,w, ,l,o,w,e,r, ...
# After 1 merge: lo,w, ,lo,w,e,r, ...
# After n merges: "lower" → ["low", "er"] or ["lower"]
# Depends on training corpus frequencies

Why Tokenization Explains Strange LLM Behaviours

• Letter counting: "How many 'r's in 'strawberry'?" fails because "strawberry" is split as ["str","aw","berry"] — the model never sees individual letters.
• Unusual capitalisation: "hello" and "Hello" may be different tokens with different semantics.
• Code performance: Python code tokenizes much more efficiently than, say, Haskell — so more Python was in training data, making LLMs better at Python.
• Non-English performance: Languages with less training data have longer, less frequent tokens — Japanese text may tokenize 3–5x more tokens than equivalent English text.

Counting Tokens in Code

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
text = "The quick brown fox jumps over the lazy dog."
tokens = enc.encode(text)
print(f"{len(tokens)} tokens: {tokens}")
# 9 tokens: [791, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 13]

Always estimate token counts before making API calls. A 100-page PDF can easily exceed 50K tokens — more than GPT-4o's practical working window.