Prehistory — From Neural Networks to the TransformerYoshua Bengio, Patrice Simard, Paolo FrasconiYoshua Bengio
Learning long-term dependencies with gradient descent is difficult
AIMade by AIWritten by AI (Claude) from the source below. Check anything important against the source.How it's made
Learning long-term dependencies with gradient descent is difficult — IEEE TNN 5(2) 157–166
Why it matters
Confirmed in theory and experiments that recurrent neural networks struggle to learn long-term dependencies, motivating the development of later architectures
Source — Primary sources first; secondary reports are labelled “Press”
Pressen.wikipedia.orghttps://en.wikipedia.org/wiki/Vanishing_gradient_problemAdded in Week 35, 2026 · See that week
On the map — ⓪ Prehistory line
- Previous stopBackpropagation
- This recordLearning long-term dependencies with gradient descent is difficult
- Next stopGradient-based learning applied to document recognition