Sliding-window beats linear attention
AIMade by AIWritten by AI (Claude) from the source below. Check anything important against the source.How it's made
Sliding-window beats linear attention released — points out that the line of research retrofitting existing LLMs with linear attention has never been properly compared with simpler baselines, and reports that sliding-window attention (SWA) with sinks matches or beats post-trained linear-attention models across several LLMs and a range of downstream tasks. On long-context reasoning tasks such as Needle-in-a-Haystack and BABILong, SWA performed 2–10 times better than linear attention; it claims that SWA needs no post-training, is very fast and uses less memory
Why it matters
A rebuttal paper that sets a baseline — "a sliding window that needs no post-training is enough" — against the strand of linear-attention retrofits that had been in the spotlight as a fix for quadratic scaling. The authors conclude that linear-attention models will catch up with SWA only if trained from scratch or put through extensive post-training. Figures are as reported by the authors.
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2608.28444Added in Week 36, 2026 · See that week
On the map — ⑪ Model Architecture line
- Previous stopMistral Small 4
- This recordSliding-window beats linear attention
- Next stopIntern-NCP Team