What is a parameter? One of the many numbers inside a model that get tuned during training

One of the many numbers inside a model that get tuned during training.

Parameters are the numbers inside a model. The weights on the connections in a neural network are the main kind. Training is really just adjusting these numbers bit by bit until they combine to give good answers.

People often describe a model's size by its parameter count. GPT-3, from 2020, had about 175 billion; small models that run on phones have a few billion. More parameters can hold more patterns, but more isn't automatically smarter: what the model learned, and how well, matters too.

Think of it like this

Picture a sound mixing desk covered in knobs. Each knob is a parameter, and training is turning them little by little until it sounds right. An AI just has billions of knobs.

For example

In "a 7B model", the B stands for billion: about 7 billion parameters.

Common mix-ups

Mix-upThe more parameters, the smarter the model — always.
ActuallyIt's like an essay: more words don't make it better. A small model carefully tuned for one job can beat a much bigger model at that job, so it's better to compare them on the task you actually need than by size.
Mix-upParameters are settings that people type in one by one.
ActuallyParameters start out filled with random numbers and get adjusted by training. The values people do choose, such as how many layers to stack or how big each correction should be, have a separate name: hyperparameters.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: How many times more parameters does GPT-3 have than a "7B" model?
Activity: Using the fact that B means billion, write a 7B model (about 7 billion) and GPT-3 from 2020 (about 175 billion, record E0108) in the same units and work out the ratio. Then estimate how many years it would take to count to 7 billion at one number per second. Finally, write one line each on what you'd miss by choosing a model on parameter count alone. Lessons DAY 05, "Just how big is this number?", and DAY 09, "Is a bigger model always better?", take it further.