Training a small model to copy what a big model has learned.
Distillation trains a small "student" model to imitate the answers of a large "teacher" model. The student learns not just the right answer but how likely the teacher thought each option was, so it picks things up faster than it would alone. A 2015 paper by Geoffrey Hinton and colleagues made the method widely known.
The small model that comes out is faster and cheaper, so it can run on a phone or laptop, though it is usually a little less capable than its teacher. Collecting another company's AI answers in bulk and distilling them without permission has also led to disputes.
Think of it like this
Imagine borrowing the study notes of the top student in your year. Instead of reading the whole textbook, you follow what they highlighted and where they hesitated, and you catch up much faster. Even so, notes alone rarely make you as good as they are.
For example
If the teacher sees a photo as "70% cat, 25% tiger", the student also learns that the two look quite alike.
Common mix-ups
Mix-upA small distilled model is exactly as good as its teacher.
ActuallyThe student picks up the teacher's mistakes too, so what the teacher gets wrong, the student tends to get wrong as well. And the gap between them tends to show more on rare, difficult questions than on easy ones.
Mix-upThe student model just copies the teacher's right answers.
ActuallyThe 2015 paper by Hinton, Vinyals and Dean deliberately raised the "temperature" to flatten the teacher's probability table before showing it to the student. That way even the "these look alike" information hidden in tiny probabilities like 1% stands out for the student.
Try it in class
For teachers at the start of a lesson — or for students working on their own.
- Question
- Why is collecting another company's AI answers for distillation a source of disputes?
- Activity
- One "teacher" shows five picture cards, each with probabilities like "cat 70%, tiger 25%, dog 5%". The "students" then guess five new cards — once after seeing only the top answers, once after seeing the full probability lists. Compare which worked better, then read the 2026 record (E0866) and talk about why permission matters.
Records about this
Source-checked moments from AI history.
The distillation dispute moves from suspicion to quantified accusation. Choosing to disclose its detection work and seek an industry-wide response rather than sue reflects the limits of legal remedies
Formalized distillation, which transfers a large model's knowledge to a small model. The root of today's small-model lineage