Words to tokens to numbers

Why text has to be numbered at all

Every operation inside a transformer is a matrix multiplication, an addition, or a smooth non-linearity. None of those accept the string "cat". So the very first thing any language model does is replace text with a list of integers.

The integers are not meaningful in themselves — they are row indices. Id 3797 means 'look up row 3797 of the embedding matrix'. The number is arbitrary; the vector stored at that row is what carries meaning, and that vector is learned during training.

Choosing the unit: characters, words, or something between

Split on characters and the vocabulary is tiny — a hundred or so symbols — so nothing is ever unknown. But sequences become very long, and the model must relearn from scratch that c-a-t means the same thing every time.

Split on words and each token is meaningful, but the vocabulary is unbounded. New words, typos, names and inflections all become 'unknown', and 'run', 'runs', 'running' share nothing.

Subwords are the compromise everyone converged on. Frequent words stay whole, rare words break into reusable pieces, and nothing is ever unknown because the pieces bottom out at single bytes.

The trade-off, concretely

text = "the cat sat on the mat"

# 1. Decide the units. Here: unique whitespace-separated words.
vocab = sorted(set(text.split()))

# 2. Assign each an integer. This mapping IS the tokenizer.
stoi = {s: i for i, s in enumerate(vocab)}   # string -> id
itos = {i: s for s, i in stoi.items()}       # id -> string

encode = lambda s: [stoi[w] for w in s.split()]
decode = lambda ids: " ".join(itos[i] for i in ids)

print(vocab)                       # ['cat', 'mat', 'on', 'sat', 'the']
print(encode("the cat sat"))       # [4, 0, 3]
print(decode([4, 0, 3]))           # the cat sat

# The fatal flaw: anything outside the vocabulary raises KeyError.
# encode("the dog sat")  ->  KeyError: 'dog'
# Subword tokenization exists precisely to make that impossible.

Note: Real tokenizers add normalisation, a splitting regex, special tokens and byte fallback — but the core is still exactly this dictionary.

Preparing text before tokenization

Before splitting, text is normalised: Unicode form is standardised, control characters are stripped, and sometimes case is folded. This matters because visually identical strings can be different byte sequences — a composed 'é' and an 'e' plus a combining accent tokenize differently and land on different embedding rows.

Whitespace is the part people underestimate. In GPT-family tokenizers the leading space is part of the token, so " cat" and "cat" are two different ids with two different embeddings. A trailing space in a prompt genuinely changes what the model predicts next.

Byte-pair encoding: the algorithm behind almost every modern tokenizer

BPE starts with the smallest possible vocabulary — the 256 individual bytes — and then grows it greedily. Count every adjacent pair of symbols in the corpus, merge the most frequent pair into a single new symbol, record the merge, and repeat.

After a few thousand merges, common words like ' the' have become single tokens because their letters always co-occurred, while a rare word survives as a handful of fragments. The learned merge list, applied in order, is the tokenizer.

The elegance is that the vocabulary size is a dial you set. Stop after 30,000 merges and you have BERT-scale; keep going to 100,000 and you have GPT-4-scale, with longer average tokens and shorter sequences.

from collections import Counter

def train_bpe(text, target_vocab_size):
    # Start from raw bytes: every possible input is representable.
    tokens = list(text.encode("utf-8"))
    vocab = {i: bytes([i]) for i in range(256)}
    merges = {}

    while len(vocab) < target_vocab_size:
        pairs = Counter(zip(tokens, tokens[1:]))
        if not pairs:
            break
        best = max(pairs, key=pairs.get)      # most frequent adjacent pair
        if pairs[best] < 2:                   # nothing repeats: stop
            break

        new_id = len(vocab)
        merges[best] = new_id
        vocab[new_id] = vocab[best[0]] + vocab[best[1]]

        # Replace every occurrence of the pair with the new symbol.
        merged, i = [], 0
        while i < len(tokens):
            if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == best:
                merged.append(new_id)
                i += 2
            else:
                merged.append(tokens[i])
                i += 1
        tokens = merged

    return vocab, merges

corpus = "the cat sat on the mat, the cat ate the rat " * 40
vocab, merges = train_bpe(corpus, target_vocab_size=300)

# The merges learned first are the most frequent substrings in the corpus.
for pair, new_id in list(merges.items())[:6]:
    print(new_id, repr(vocab[new_id]))

Note: Because the base vocabulary is all 256 bytes, no input is ever out-of-vocabulary. Emoji, Chinese, mojibake — everything decomposes to bytes in the worst case.

Token efficiency, and compression ratio

A useful measure of a tokenizer is its compression ratio: characters per token on your text. GPT-4's tokenizer averages roughly four characters per token on English prose. Higher is better — it means more text fits in the context window and each forward pass covers more content.

Measuring the distribution of token lengths is more informative than the mean. A well-fitted tokenizer produces mostly multi-character tokens on its target language; a poorly fitted one falls back to one- and two-character fragments, which is exactly what happens on code, on unusual formatting, and on languages under-represented in the merge training.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # GPT-4 class tokenizer

samples = {
    "English": "The quick brown fox jumps over the lazy dog.",
    "Hindi":   "तेज़ भूरी लोमड़ी आलसी कुत्ते के ऊपर से कूदती है।",
    "Code":    "for i, x in enumerate(xs): total += x ** 2",
    "JSON":    '{"user_id": 8371, "is_active": true}',
}

for name, text in samples.items():
    ids = enc.encode(text)
    ratio = len(text) / len(ids)
    print(f"{name:8} {len(text):3} chars  {len(ids):3} tokens  {ratio:4.2f} chars/token")

# Inspect the actual pieces — this is the most clarifying thing you can print.
for tid in enc.encode(" tokenization is unintuitive"):
    print(tid, repr(enc.decode([tid])))

Note: Non-Latin scripts routinely cost two to four times more tokens per character, because far fewer of their sequences earned a merge. That is a direct cost and context-window penalty.

BERT tokenizes differently, and it shows

BERT uses WordPiece rather than BPE. Continuation pieces are marked with '##', so 'tokenization' becomes 'token', '##ization' — the marker says 'this attaches to the previous piece with no space'. GPT encodes the same information by putting the leading space inside the token instead.

BERT also wraps every sequence in special tokens: [CLS] at the front, whose final hidden state is used as the sentence representation for classification, and [SEP] at boundaries. [MASK] is the token the model was trained to fill in. Those special ids are part of the vocabulary and must be present or the model behaves oddly.

from transformers import AutoTokenizer

gpt2 = AutoTokenizer.from_pretrained("gpt2")
bert = AutoTokenizer.from_pretrained("bert-base-uncased")

s = "Tokenization is unintuitive."
print(gpt2.tokenize(s))
# ['Token', 'ization', 'Ġis', 'Ġunint', 'uit', 'ive', '.']    Ġ = leading space

print(bert.tokenize(s))
# ['token', '##ization', 'is', 'un', '##int', '##uit', '##ive', '.']

print(bert(s)["input_ids"])      # note the [CLS] 101 ... [SEP] 102 wrapper

# Ids are NOT portable. The same integer decodes to different strings.
print(repr(gpt2.decode([464])), repr(bert.decode([464])))

Note: Feeding one model's ids to another produces fluent-looking nonsense, not an error. Always re-tokenize from text when moving between models.

Translating between tokenizers

Because ids are model-specific, comparing two models on the same text means decoding to strings and re-encoding — you cannot map id to id. Even then the token boundaries differ, so any per-token comparison needs an alignment step that maps character spans, not indices.

This is the practical obstacle to comparing hidden states across model families, and the reason careful interpretability work aligns on character offsets rather than token positions.

Zipf's law, and why any of this works

Word frequencies follow Zipf's law: the nth most common word appears roughly proportionally to 1/n. A handful of words account for a large fraction of all text, and a very long tail appears once or twice.

That skew is exactly what makes subword tokenization efficient. Merging the tiny number of extremely frequent sequences buys most of the available compression, and the long tail can afford to be fragmented because it is rare. Plot token frequency against rank on log-log axes and you get a near-straight line — the same law holds for tokens, not just words.