What is a GPU? A chip that does huge numbers of simple calculations at once: the workhorse of AI

A chip that does huge numbers of simple calculations at once: the workhorse of AI.

A GPU (graphics processing unit) was built to colour in millions of pixels on a game screen at the same time. It packs thousands of small calculating units, so it's great at splitting easy maths into pieces and doing them all at once. Training AI turns out to be exactly that kind of work: endless simple multiplying and adding.

From the mid-2000s, studies showed GPUs could train neural networks much faster, and the 2012 ImageNet winners used two graphics cards. Today's large models train for weeks or months on thousands or even tens of thousands of GPUs or similar AI chips in data centers. That has made GPUs expensive and hard to get.

Think of it like this

Say you have 10,000 maths problems. A few brilliant students working through them one by one is slow; a few thousand students each taking a handful at the same time is far faster. A CPU is like the few brilliant students; a GPU is the crowd of thousands.

For example

Meta reported linking up to 16,000 GPUs together to train Llama 3.

Common mix-ups

Mix-upGPUs were made for AI from the start.
ActuallyIn 2007 NVIDIA released CUDA, which made it much easier to use GPUs for calculations other than graphics. After that, researchers took up GPUs in earnest for training neural networks — gaming chips became central to AI by finding a new use.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: Why do "thousands of students" suit AI training better than "a few clever friends"?
Activity: Prepare 100 one-digit addition problems. Time two people splitting them, then time the whole class splitting them. Next, mix in problems that need the previous answer first and try again — and talk about when splitting the work helps and when it gets stuck. Open the CUDA record (E0021) to see when GPUs opened up for general computing.