NVIDIA presents Megatron-LM
AIMade by AIWritten by AI (Claude) from the source below. Check anything important against the source.How it's made
Presented Megatron-LM — implemented intra-layer tensor parallelism in PyTorch alone. Sustained 15.1 PFLOPS with an 8.3B model on 512 GPUs
Why it matters
The standard implementation of large-scale model parallelism
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/1909.08053Added in Week 35, 2026 · See that week
On the map — ⑱ AI Infrastructure Software line
- Previous stopXLA
- This record · stopMegatron-LM
- Next stopZeRO-Offload