The small piece of text an AI reads and writes, one at a time.
An AI doesn't read text letter by letter. It first cuts a sentence into pieces called tokens. Common words usually become one token; rare words get split into several.
How much an AI can read at once, and how much it costs to use, are both counted in tokens. Some languages, such as Korean, usually need more tokens than English to say the same thing. So when people say an AI 'picks the next word', it really picks the next token.
It's not only text. AIs that work with pictures or sound also cut them into small pieces and line them up, just like tokens.
Think of it like this
Think of LEGO. A common shape is one big brick; an unusual shape takes several small bricks snapped together. To an AI, a sentence is a row of these bricks.
For example
"unbelievable" might be split into "un" · "believ" · "able".
Common mix-ups
Mix-upAI reads text one letter at a time.
ActuallyBecause it sees chunks rather than letters, it can surprisingly get easy letter-counting questions wrong, like "How many r's are in strawberry?". When letters inside a word matter, writing them out with spaces between them sometimes helps.
Mix-upThe same meaning costs the same in any language.
ActuallyA language that needs more tokens also fits less text into the same window. Paste the same sentence in Korean and English into one of the token-counting tools that several AI companies offer, and you can see the gap yourself.
Try it in class
For teachers at the start of a lesson — or for students working on their own.
- Question
- How would you say "the AI picks the next word" more precisely?
- Activity
- Play with the LEGO comparison. With a partner, pick five long words and split each into big blocks (common parts) and small blocks (rare parts) on cards. Compare how many blocks a sentence takes in English and in another language you know, then read the 2023 "glitch token" record (E0191) and talk about how one odd piece caused trouble.
Records about this
Source-checked moments from AI history.
Popularized the idea that a mismatch between the tokenizer and the model's training corpus is a real failure mode