The Llama 3 Herd of Models published
AIMade by AIWritten by AI (Claude) from the source below. Check anything important against the source.How it's made
The Llama 3 Herd of Models published — over 54 days of pretraining on 16K GPUs there were 466 job interruptions in total, 419 of them unexpected, with about 78% attributable to hardware. Even so, effective training time exceeded 90%
Why it matters
The first primary source to publish quantitative data on fault tolerance in large-scale training. Settled, for the whole industry, that checkpointing and automatic recovery are essential
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2407.21783Added in Week 35, 2026 · See that week
On the map — ⑱ AI Infrastructure Software line
- Previous stopZeRO++
- This record · stopLlama 3 Herd paper
- Next stopAMD makes its first MLPerf Training submission