What is inference? Running a finished, trained model to get an answer out of it

Running a finished, trained model to get an answer out of it.

Inference is the stage where a finished model is actually used. If training is studying from the textbook, inference is writing your answers on the test. Every time you send a question to an AI, one round of inference happens.

In everyday English, "inference" can mean working something out. In AI it simply means running the model, which is different from a reasoning model thinking step by step before it answers.

One inference is small, but billions of them a day add up to a lot of electricity and computing. That's why people work on running models directly on phones and on making models lighter.

Think of it like this

Studying for a test can take weeks; writing one answer takes seconds. Studying is training; writing the answer is inference. An AI does that "writing" fresh for every question it gets.

For example

Ask an AI one question and get one reply: that's one round of inference.

Common mix-ups

Mix-up"Inference" means the AI is thinking things through carefully.
ActuallyIn AI, "inference" usually just means running a trained model to get an output, while "reasoning" is the word for thinking things through. Even a very simple spam filter does one inference every time an email arrives.
Mix-upOne question costs so little computing that it doesn't matter.
ActuallyWhen lots of questions arrive at once, graphics-card memory runs short fast. So in 2023 researchers at UC Berkeley released vLLM, which splits memory into small pages the way an operating system does, cutting the waste to under 4%.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: Training or inference — which one happens every time you ask an AI something?
Activity: Estimate how many questions your class sent to an AI today and write it on the board. Multiply it up to a whole school, then a whole city, to feel how small runs add up. Then open the llama.cpp record (E0196) and check when people started trying to run models on their own computers instead of big servers.