Multi-Stream Attention Residuals
TL;DR Instead of a plain residual connection, each layer attends over the outputs of earlier layers, with several heads that each look at the past through a different context window. At 18B and 28B MoE scale it beats standard residuals, Hyper-Connections and Attention Residuals on 12 benchmarks, for a 1.58x compute speedup at equal loss.
What the class wrote 0
Nothing yet. Pick a task on the roles page and write the first post, see how to post.
Who is doing what (week 2)
Who does what in week 2: see the sign-up sheet. Its Week check tab shows what is still missing.
Write about this paper: Write