The neural network design behind most of today's large language models.
The Transformer is a neural network design introduced by a Google team in the 2017 paper "Attention Is All You Need". Most of today's large language models, like GPT, Claude and Gemini, are built on it.
Older designs usually read a sentence one word at a time, in order. A Transformer uses attention so every token can look at every other token at once, which makes it easier to link words that sit far apart.
Looking at everything at once also means computing it all at once, which suits GPUs. That let researchers train on huge amounts of text and scale models up.
Think of it like this
In the telephone game, a message passes from person to person and gets fuzzy by the end of the line. A Transformer is more like a class group chat: everyone sees every message at once, so it's easy to tell who is replying to whom.
For example
In "Sam lent Maya his notebook, and it was brand new," the model works out that "it" means the notebook.
Common mix-ups
Mix-upA transformer reads a sentence one word at a time, from the start.
ActuallyBecause it sees everything at once, it needs "position information" added as numbers to know the word order. And while it reads everything at the same time, when writing it usually goes one token after another.
Mix-upTransformer is the name of a chatbot.
ActuallyThe final T in GPT stands for Transformer. And the paper that introduced this design tested it not on a chatbot but on translating English into German and French.
Try it in class
For teachers at the start of a lesson — or for students working on their own.
- Question
- Why did "looking at everything at once" open the way to much bigger models?
- Activity
- Compare a whisper chain with a group chat. First, six people pass a sentence along in whispers; then write the same sentence on the board so everyone sees it at once. Compare how accurate it was at the end and how long each took, then talk about how "computed all at once" connects to training speed.
Records about this
Source-checked moments from AI history.
Self-attention alone, without recurrence, made parallel training possible, establishing the architectural premise for large-scale pretraining