The tool that chops text into tokens and gives each piece an ID number.
A tokenizer is the first step before an AI reads anything. It cuts your text into tokens using a fixed list of pieces, called its vocabulary, and turns each one into an ID number. The model only ever sees those numbers.
The vocabulary is built from lots of text by keeping chunks that often appear together. Different tokenizers can split the same sentence differently, which changes the cost and how much fits in the context window.
In 2023, people found 'glitch tokens': pieces in the vocabulary that a model had barely seen during training, which made it act strangely. Some researchers are now testing models that skip the tokenizer.
Think of it like this
Imagine texting with only a sticker pack. Common words have their own sticker, but anything missing has to be spelled out with several small letter stickers. A tokenizer is that sticker pack plus the rules for which stickers to use.
For example
"Tokenization" might become "Token" · "ization": two pieces instead of twelve letters.
Common mix-ups
Mix-upA tokenizer cuts sentences at the spaces.
ActuallyIt cuts up Chinese or Japanese text, which hardly uses spaces, with no trouble. In many tokenizers the space is glued onto the front of a word, so " cat" and "cat" can even be different chunks.
Mix-upDifferent tokenizers give the same result.
ActuallyA model is trained around its own tokenizer's list of chunks, so just swapping in a different tokenizer stops it working properly. The 141 glitch tokens reported in 2023, such as "SolidGoldMagikarp", could make a model say something unrelated even when it was simply asked to repeat them.
Try it in class
For teachers at the start of a lesson — or for students working on their own.
- Question
- What would be good — and hard — about reading smaller units directly, with no tokenizer?
- Activity
- Play the sticker-pack game. Each group picks ten common words as its "sticker pack" and counts how many stickers it takes to write a sentence the teacher reads out; anything not in the pack is spelled with letter stickers. Compare counts between groups with different packs, then read the Byte Latent Transformer record (E0475) to see what it tried to do.
Records about this
Source-checked moments from AI history.
Processes bytes directly, without a fixed vocabulary, using entropy-based dynamic patches; the first large-scale scaling of the tokenizer-free approach
Popularized the idea that a mismatch between the tokenizer and the model's training corpus is a real failure mode