What is on-device AI? AI that runs right on your phone or laptop instead of on a faraway server

AI that runs right on your phone or laptop instead of on a faraway server.

On-device AI means the model runs on your own device, such as a phone or laptop. Your request doesn't have to travel to a distant data center, so it can work offline and often answers faster. What you type or photograph also doesn't have to leave the device.

The catch is that phones have limited memory and battery, so big models don't fit. Instead they use small models, like the roughly 3-billion-parameter model Apple described in 2024, often shrunk further with tricks such as quantization. Many phones split the work: easy jobs stay on the device and hard ones go to a server.

Think of it like this

It's the difference between walking to the library every time you have a question and flipping open a pocket dictionary from your bag. The dictionary is instant, and nobody else sees what you looked up. For the really hard stuff, though, you still need the library.

For example

If your phone can still take dictation or read the text in a photo in airplane mode, that's on-device AI at work.

Common mix-ups

Mix-upAI always runs on a server somewhere across the internet.
ActuallyGemma 3n, released by Google in 2025, runs in only about 2–3 GB of memory and accepts not just text but images, sound and video. AI that runs inside a phone is moving beyond simple tasks.
Mix-upThe AI in a phone does the same job as the big model on a server.
ActuallyApple's on-phone model stores each number in 3.7 bits on average and swaps in small "adapters" for different jobs, such as summarising or polishing text. It's built to do a few set jobs well at a small size, so it plays a different role from big server models.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: Which jobs belong on your phone, and which on a server?
Activity: With a phone in aeroplane mode, test dictation, reading text in a photo, translation and summarising a long article, and record works or doesn't. Sort the results into "needs to be fast", "involves private data" and "is hard", then use the pocket-dictionary-versus-library comparison to explain the split.