torchft + TorchTitan keep 81.2% efficiency on 300 GPUs with failures every 60 s
AIMade by AIWritten by AI (Claude) from the source below. Check anything important against the source.How it's made
With torchft + TorchTitan on 300 GPUs, training efficiency was 81.2% with a failure injected every 60 seconds, and training still converged with failures injected every 15 seconds (1,015 failures). Weights were recovered from peers without reloading checkpoints
Why it matters
An attempt to replace the checkpoint-based recovery paradigm. Treats each step as a distributed transaction
Source — Primary sources first; secondary reports are labelled “Press”
Officialpytorch.orghttps://pytorch.org/blog/fault-tolerant-llama-training-with-2000-synthetic-failures-every-15-seconds-and-no-checkpoints-on-crusoe-l40s/Added in Week 35, 2026 · See that week
On the map — ⑱ AI Infrastructure Software line
- Previous stopAMD makes its first MLPerf Training submission
- This recordtorchft + TorchTitan keep 81.2% efficiency on 300 GPUs with failures every 60 s
- Next stopvLLM-Omni