Training an AI on people's choices about which of two answers is better.
RLHF stands for reinforcement learning from human feedback. Usually the AI is first taught once with model answers that people wrote. Then it writes several answers to the same question, and people pick the better one, over and over. That picking step is the heart of RLHF.
Since people can't rate every answer, their choices train a scoring model that imitates human taste. The AI is then tuned to earn higher scores from it.
In the 2022 InstructGPT study, people preferred a small tuned model's answers to those of a model over 100 times bigger. ChatGPT, released later that year, was trained this way too. A known side effect: models may learn to tell people what they want to hear.
Think of it like this
Think of a talent show where judges never explain the "right" performance; they just pick which of two acts they liked more. After enough rounds, performers figure out what the crowd enjoys.
For example
Given a short, accurate answer and a long, rambling one, a person picks one, and that choice becomes training data.
Common mix-ups
Mix-upRLHF means people teach the AI the right answer, one by one.
Actually"Which of these two is better?" is much easier for people than "write the perfect answer" or "score this out of ten", and people disagree less. That's why RLHF collects comparisons rather than right answers.
Mix-upAn answer people like is always a good answer.
ActuallyPeople tend to give more points to long, polite answers or ones that agree with them. So if the AI says "you're right" when you might be wrong, ask it separately to "find what's wrong in what I said".
Try it in class
For teachers at the start of a lesson — or for students working on their own.
- Question
- When your class picks the "better answer", what do you judge by?
- Activity
- Run it like a talent-show panel. Prepare ten pairs of answers to the same question (one short and accurate, one long and rambling) and have classmates raise hands for the better one. Collect the reasons on the board as your class's "scoring rules", and look for any pair where the nicer-sounding answer won. Read the InstructGPT record (E0140) to see what the method changed.
Records about this
Source-checked moments from AI history.
Showed that outputs of the 1.3B model were preferred to those of the 175B GPT-3, establishing RLHF as the standard procedure for alignment
OpenAI itself wrote that it unlocked capabilities that had been hard to draw out through prompt engineering alone