What is quantization? Storing a model's numbers with less precision so it takes up less memory

Storing a model's numbers with less precision so it takes up less memory.

An AI model is made of billions of numbers (weights), and each one is usually stored using about 16 bits. Quantization rounds those numbers off so they fit in 8 or even 4 bits. Going from 16 bits to 4 bits cuts the memory the model needs to about a quarter.

That's how models built for big data centers can run on a home computer or even a phone. Squeeze too hard, though, and answer quality can drop, so researchers keep working on smarter ways to choose what to shrink and by how much. Some models are shrunk after training; others are trained with the shrinking in mind.

Think of it like this

Think of writing pi as 3.14 instead of 3.14159265. It takes far less space in your notebook, and most everyday sums come out almost the same. Round it all the way down to 3, though, and your answers start to go noticeably wrong.

For example

Google released a 4-bit version of its Gemma 3 27B model that shrank it from about 54 GB to about 14 GB.

Common mix-ups

Mix-upQuantization deletes some of what the model learned.
ActuallyIn the 2022 LLM.int8() study, a 175-billion-parameter model was converted to 8 bits without losing performance, and the memory it needed halved. Quantization doesn't delete numbers: it keeps how many there are and only shortens each one's precision.
Mix-upThe more you shrink it, the better.
ActuallyLabels like Q4 or Q8, often found in downloadable model file names, tell you roughly how many bits each number has been shrunk to. The more you shrink, the more the answers tend to suffer, so usually the sensible choice is the highest number that still fits in your computer's memory.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: How far can you shrink the numbers and still get "almost the same answer"?
Activity: Solve five circle-circumference problems with pi as 3.14159, 3.14 and 3, and record how far apart the answers are. Each group sets its own rule for "close enough", then works out the shrink ratio in the Gemma 3 record (E0569) — 54 GB down to 14.1 GB — and compares it with your table.