What is pretraining? The first, giant training stage where a model learns broadly from huge amounts of data

The first, giant training stage where a model learns broadly from huge amounts of data.

Pretraining is the first stage of building an AI model. The model is shown a huge amount of text, code and other data and asked to guess the next token again and again. Nobody has to label anything: the text itself supplies both the question and the answer.

Along the way the model picks up grammar, common sense and knowledge from many fields. It's expensive, though: thousands of graphics chips running for weeks or months. The result, a base model, is good at continuing text but clumsy at answering questions helpfully, so it's then polished with fine-tuning. The "pre" means it comes before that polishing.

Think of it like this

It's like an athlete spending years on fitness and basic skills before specializing in one sport. With strong basics, any sport comes faster. Pretraining is that basic training.

For example

Give a base model "Once upon a time" and it happily keeps writing a story, but it may respond oddly to "help me with my homework".

Common mix-ups

Mix-upOnce pretraining is done, the model is ready to be a friendly chat partner.
ActuallyGive a base model a question and it may carry on with a list of similar questions instead of answering. It is only continuing text the way worksheets or forum posts online do — it has no sense yet of being the one who helps.
Mix-upPretraining needs people to label every right answer.
ActuallyWhat people put effort into isn't labels but choosing the material. "Cleaning" — filtering out repeated copies of the same text, adverts and personal information — makes a big difference to how good the model turns out.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: What happens to pretraining when the data runs out?
Activity: Read the 2024 record (E0476) together and, in groups, rewrite in one sentence what it means to call data "the fossil fuel of AI". Then borrow the athlete comparison and make a table comparing the time and cost of basic training (pretraining) and sport-specific practice (fine-tuning).