What are scaling laws? The observation that AI gets predictably better as models

The observation that AI gets predictably better as models, data and compute grow.

Scaling laws describe a pattern: as you increase a model's parameters, its training data and the computing power used to train it, its prediction error falls in a fairly regular way. OpenAI researchers set this out in 2020, which meant people could estimate in advance how much better a bigger model would be.

In 2022 the Chinchilla study showed that the data should grow along with the model, not just the model itself. Despite the name, these aren't laws of nature; they're trends observed so far. Researchers disagree about how long the trend will hold.

Think of it like this

Think of training for a race. If every time you double your practice your time improves by a similar proportion, you can roughly predict your next result. But nobody can promise the pattern will last forever.

For example

Chinchilla, with 70 billion parameters and more training data, beat a model with 280 billion parameters.

Common mix-ups

Mix-upScaling "laws" are laws of nature, like gravity.
ActuallyScaling laws are observations — trends found in experimental results and written as formulas — so there's no guarantee they always hold the way gravity does. After Gary Marcus made this point in his 2022 essay "Deep Learning Is Hitting a Wall", "hitting a wall" became the standard phrase for this kind of doubt.
Mix-upTo get better results, just make the model bigger.
ActuallyAfter training and comparing more than 400 models, the Chinchilla team concluded that many big models of the time were "undertrained" — they had read too little text for their size. With the same computing budget, it's better to grow the training data in the same proportion as the model size.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: When you predict the future from the trend so far, what should you watch out for?
Activity: On grid paper, plot five made-up points — days of practice against skipping-rope jumps — join them and estimate day six. Then each group lists three reasons the line might bend. Finally read records E0101 and E0143 side by side and sum up each side's argument in one line.