What is a small language model (SLM)? A language model built with fewer parameters

A language model built with fewer parameters, small enough to run on a phone or laptop.

A small language model has somewhere between a few million and a few billion parameters. The name only makes sense next to large language models, which can have hundreds of billions or more.

Being small, it needs less memory and power and runs fast, so it can work inside a phone or laptop with no internet. To make up for its size, builders pick its training material carefully or let it learn from a bigger model's answers.

In April 2024, Microsoft's 3.8-billion-parameter phi-3-mini scored 69% on MMLU, and Microsoft claimed it matched Mixtral 8x7B. Yet the report admitted the model has no room to store much factual knowledge, pointing to its low score on the TriviaQA quiz.

Think of it like this

Think of the school tuck shop versus a big supermarket. The tuck shop is right next to your classroom, so you can pop over at break, but it doesn't stock much. A small language model is the tuck shop: close by and quick to use.

For example

Open models released for running on devices like phones, such as Meta's Llama 3.2 1B and 3B, can be downloaded and run on your own laptop.

Common mix-ups

Mix-upSmall language models are rough jobs made when a company can't afford a big one.
ActuallyBig companies make small models on purpose. Meta said it built Llama 3.2 1B and 3B by cutting parts out of the larger Llama 3.1 8B (pruning) and having them learn from the answers of the 8B and 70B models (distillation). Big models actually helped make good small ones.
Mix-upAny phone can run a small language model.
ActuallyEven a 3-billion-parameter model needs about 6 GB just for its weights if each number takes 16 bits (2 bytes). That's why it gets shrunk further for phones: phi-3-mini squeezed to 4 bits takes about 1.8 GB. Multiply the parameter count in billions by 2 to estimate the gigabytes needed at 16 bits.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: For an AI that helps with homework, which jobs can a small language model handle, and which need a big one?
Activity: Multiply the parameter counts of Kanana Nano 2.1B (E1240), phi-3-mini 3.8B (E0358) and Llama 3.2 3B (E1233) by 2 to estimate the gigabytes each needs at 16 bits, and compare them with your phone's or laptop's memory. Then sort jobs such as fixing spelling, asking for a historical date and summarising a long article into a tuck-shop column and a supermarket column.