What is attention? The calculation that decides how much each token focuses on every other token

The calculation that decides how much each token focuses on every other token.

Attention gives each word a score for how strongly it should look at every other word. The scores are shared out from a fixed total, so focusing hard on one word means less focus elsewhere. Where that focus lands helps the model work out things like what "it" refers to.

Attention became widely known through 2014 translation research, where it decided which part of the original sentence to look at. The 2017 Transformer built a whole model out of it, a version called self-attention.

Since every token looks at every other one, the work grows fast as text gets longer. That's one reason long context windows are hard.

Think of it like this

In a noisy cafeteria, you hear your name and your ears snap toward it while everything else fades a little. Attention is each word deciding whose voice to tune into right now.

For example

In "Jen packed her bag, and it was heavy," the word "it" pays more attention to "bag" than to "Jen".

Common mix-ups

Mix-upAttention means the AI "concentrates" the way a person does.
ActuallyA person's focus goes to one place at a time, but a transformer runs several attention calculations, called "heads", at once. One head might track what a pronoun refers to while another looks at the word right next door, each catching a different relationship.
Mix-upAttention was invented together with the transformer in 2017.
ActuallyIn 2014, Bahdanau, Cho Kyunghyun and Bengio fixed a bottleneck in translation — cramming a whole long sentence into one fixed-size bundle of numbers — by letting the model look back at the original each time it translated a word. The transformer took up this idea three years later.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: Why does the computing grow so fast as the text gets longer?
Activity: Cut "Mia packed her bag, and it was heavy" into word cards. Let the card "it" share out 10 points among the other cards, and compare how each group shared them. Then count how many pairs of cards look at each other when there are 4, 8 and 16 cards, and draw the numbers as a table.