Part 5 Review
From one dot to the phone in your pocket · with the real terms · DAY 46-55
AIMade by AIThis part review was written by AI (Claude) from that part's lessons and records, and published after automated checks. The owner doesn't read every item.How it's made
Part 5 was ten days of opening up an AI. It started with the math of one dot, went through attention, embeddings, retrieval, fine-tuning and human feedback, and ended with the metal and the electricity that run it - and your own phone.
No quizzes and no experiments here. Every new term gets one sentence of plain explanation the first time it shows up, so just read straight down.
DAY 46 · Neural networks
One dot can only multiply and add
A neural network is dots passing numbers around, stacked in layers. A dot is called a node, one tier a layer. Each dot multiplies every incoming number by a weight and adds them all up. That total is the weighted sum. Then it pushes the total through a function that acts as a threshold and hands it to the next layer.
That threshold function is called an activation function. It decides whether a signal passes straight through, bends, or stops. Without it, stacking any number of layers ends up the same as one multiply-and-add. That's because stacking only straight, proportional math collapses into a single straight calculation.
The first weights are arbitrary. The network answers, then walks backward through its own error to fix them. Sending the error from back to front and nudging each weight is called backpropagation, made widely known in 1986. Repeat that an uncountable number of times and the numbers left behind are what it learned.
DAY 47 · Attention
Attention is a share you split up
Attention builds a query, a key and a value for every word, then matches queries against keys to get scores. The step that turns those scores into shares adding up to 1 is called softmax. Because they are shares, the total is fixed - stare hard at one word and the rest must weaken.
When "it" has a query that matches some earlier noun's key, the staring piles onto that spot. The values mix in by those shares, producing a new number bundle with the context blended in. That's why the same word comes out as a different value in a different sentence.
A 2014 translation study used it first, as a device against losing distant words in long sentences. The 2017 paper dropped the read-in-order part entirely and kept only attention - the Transformer. Looking all at once made the math easy to split across machines, which is how models got big.
DAY 48 · Meaning as numbers
Meaning gets turned into a place
A word turned into a bundle of numbers is called an embedding. One bundle holds hundreds to thousands of numbers. The learning rule is simple - words with similar company get similar numbers. Nobody writes the meanings in. It works them out while reading an enormous amount of text.
Once meaning is a place, the distance between two words can be calculated. A 2013 study showed that even the direction between places carries meaning - take man away from king, add woman, and you land near queen. That directions hold relationships is why the method stayed in use so long.
Models today don't use whole words; they cut text into pieces called tokens. Chopping unfamiliar words into pieces made the unknown-word problem much smaller. That's why length gets counted in tokens rather than letters.
DAY 49 · Finding similar things
It measures meaning, not letters
Sentences can become embeddings too. Measure the angle between two bundles and you see how much the meanings overlap. The common yardstick is cosine similarity - it drops the size of the bundle and compares direction only. Same direction lands near 1; unrelated falls toward 0.
"That was fun" and "that was no fun" share almost every letter, but their directions are far apart. The other way around, with no letters in common at all, same meaning still comes out close. Those two cases are exactly where the old letter-overlap methods kept going wrong.
A 2019 study put out a way to train two sentences onto the same yardstick. That let people turn millions of text pieces into numbers in advance and, when a question arrives, pull out only the close ones. That pulling-out is the first half of answering with the book open, the next day's lesson.
DAY 50 · Answering with the book open
Look it up and read it, instead of reciting
Laying found text open beside the question is called retrieval-augmented generation, RAG for short. The question becomes an embedding, the close text pieces get pulled out, and those pieces go into the model with the question. The model reads what's in front of it instead of what it memorized.
Inventing happens when it tries to fill a gap it doesn't know with memory. With the source text in front of it there's less gap to fill, and it can show where the words came from. With a source attached, the reader can check it themselves.
Not having to retrain the model is the big win - swap out only the text you lay open and the answers follow. But if the piece it found is off-target or wrong, the answer goes wrong with it. That's why the real effort goes into the finding rather than the answering.
DAY 51 · Teaching it extra
Nudge a grown model instead of raising a new one
A model that did the big round of learning is called a base model, and teaching it one more job on top is fine-tuning. Rather than changing every number inside, the common approach changes only a tiny slice. In cost it is a completely different thing from training from scratch.
LoRA, from 2021, freezes the original numbers and bolts two small plates on beside them, training only those. That cut the numbers it touches down to one ten-thousandth. Swap the bolted-on plates and one base model can do several different jobs.
Fine-tuning is strong on tone, format and the way a job gets done. Putting newly arrived facts in is the job of answering with the book open. They aren't rivals but different roles, so in practice they often run together.
DAY 52 · People handing out scores
Not the right answer, just which one is better
You have the model produce several answers to one question and let a person pick the better one. That record is called preference data. The picks are used to build a reward model first - a small model that guesses the score a person would give. Tuning the main model to push that score up is the reinforcement learning step.
A good answer can't be written down as one thing. "Be kind" or "keep it short" is hard to turn into a rubric. So instead of writing a score, you only ask which of the two is better. That's a far easier question for a person. That's what it means to say you collect preference rather than right-and-wrong.
In a 2022 study, people preferred this tuned small model's answers over those of a model more than 130 times its size. It's a rare case where tuning made a bigger difference than scaling up. But whose preference you collected stays baked in. That's why 2022 also brought a method that writes the rules down and has AI do the scoring.
DAY 53 · GPUs
The shape of the math was the same all along
A graphics card (GPU) is a part built to do easy math a huge number of times at once. That's called parallel computing. Painting millions of dots on a screen at once needed thousands of simple workers rather than a few clever ones. Inside AI training is matrix multiplication - trillions of easy multiplications and additions, all at once.
One card of the kind used for training holds about 80 billion tiny switches called transistors. That single card does roughly 1,000 trillion calculations a second. Today's cards also take the route of handling numbers at a coarser scale to get more math done.
Llama 3 was built by tying 16,000 cards into one block and running it for 54 days. A block like that is called a cluster, and the speed of the links between cards can decide performance too. In 2024 one place said it had tied together 100,000 cards.
DAY 54 · Data centers and electricity
One is tiny, the pile is not
In a measurement released in 2025, the median electricity for one text question was 0.24 watt-hours. That's about a TV left on for 9 seconds, so on its own it is a small amount. That said, it is a company-reported figure on the company's own terms, and the value shifts with what you count.
Data centers worldwide used 415 terawatt-hours of electricity in 2024 - about 1.5% of all electricity. The International Energy Agency expects 945 terawatt-hours by 2030, more than double. 1.5% looks small, but things that double in six years are not common.
The per-question figure is dropping fast, because the field keeps moving toward getting the same answer with less math. But if the number of questions grows faster, the total keeps rising anyway. When you read these numbers, start by not mixing up one question with the total.
DAY 55 · AI inside my phone
It splits the work into pieces
Today's phones carry a small model of about 3 billion numbers. Running inside the phone is called on-device. Even with no internet it can summarize, polish text and tidy up notifications. It differs in size from the model on the server by tens to hundreds of times.
It uses quantization, storing the big model's numbers at a coarser scale. Fewer digits also means more speed. When memory runs short, loading only the part needed at that moment is used alongside it. Still, shrinking costs it some jobs, so not everything goes on the phone.
The default is that the phone tries first and sends out only the pieces too heavy for it. Finishing inside means faster, less electricity, and your words never leaving. Jobs that need that building from before are still there. Mixing the two is today's answer.
Part 5 in one line
Thread the ten days and one line shows through - the inside is math, not magic, and everything comes down to how you aim that math and how you run it. Multiply and add (DAY 46), decide where to look (DAY 47), turn meaning into a place (DAY 48-49), look it up and lay it open (DAY 50), tune it (DAY 51-52) - that's the aiming; the metal, the electricity and the phone (DAY 53-55) are the running.
Part 6 looks ahead. Is an AI that matches people really coming, and why do people argue over those words?
Learn more
- Deep learning (Nature, 2015) — a short summary written by the three people who built deep learningnature.com/articles/nature14539
- Learning representations by back-propagating errors (1986) — the paper that made fixing weights by the size of the error widely knownnature.com/articles/323533a0
- McCulloch & Pitts, A logical calculus... (1943) — the first paper to treat a nerve cell as a computing elementlink.springer.com/article/10.1007/BF02478259
- Neural Machine Translation by Jointly Learning to Align and Translate (2014) — the translation study where attention was first usedarxiv.org/abs/1409.0473
- Attention Is All You Need (2017) — the paper that dropped reading in order and kept only attentionarxiv.org/abs/1706.03762
- Efficient Estimation of Word Representations (2013) — the study that spread the idea of turning words into number bundlesarxiv.org/abs/1301.3781
- Sentence-BERT (2019) — a way to build sentence bundles that are good for measuring distancearxiv.org/abs/1908.10084
- Dense Passage Retrieval (2020) — finding text by places of meaning instead of letter matchingarxiv.org/abs/2004.04906
- Retrieval-Augmented Generation (2020) — the study that had a model answer with found text laid open beside itarxiv.org/abs/2005.11401
- LoRA (2021) — the method that cut the numbers you touch down to one ten-thousandtharxiv.org/abs/2106.09685
- Training language models to follow instructions with human feedback (2022) — tuning a model with records of what people pickedarxiv.org/abs/2203.02155
- NVIDIA Hopper Architecture In-Depth (2022) — the makers' own walkthrough of the card used for trainingdeveloper.nvidia.com/blog/nvidia-hopper-architectur…
- The Llama 3 Herd of Models (2024) — the record of a 54-day run on 16,000 cardsarxiv.org/abs/2407.21783
- Measuring the environmental impact of AI inference (Google, 2025) — a company-reported measurement of the electricity one question takescloud.google.com/blog/products/infrastructure/measu…
- Energy and AI - Executive summary (IEA, 2025) — an official summary putting data center electricity in national termsiea.org/reports/energy-and-ai/executive-summary
- Introducing Apple's On-Device and Server Foundation Models (2024) — an official post on splitting work between a phone model and a server modelmachinelearning.apple.com/research/introducing-appl…
- A Neural Probabilistic Language Model (2003) — the starting point for turning words into number bundlesjmlr.org/papers/v3/bengio03a.html
- QLoRA (2023) — tuning a model while its numbers are held at a coarser scalearxiv.org/abs/2305.14314
- Constitutional AI (2022) — a study that writes the rules down and has AI do the scoringarxiv.org/abs/2212.08073
- Colossus (xAI, 2024) — public material on tying together 100,000 cardsx.ai/colossus
- Gemma 3n developer guide (Google, 2025) — an official guide to small models that run on a phonedevelopers.googleblog.com/en/introducing-gemma-3n-d…