MSAR paper on LinkedIn

LinkedIn

Turns out that familiar ideas like multi-head attention can still yield substantial gains when applied along a different axis. Residual connections help deep networks train by letting information and gradients flow directly across layers. But as successive layer outputs accumulate in a single residual stream, earlier features can become diluted and harder for later layers to access selectively.

Attention Residuals addresses this by letting layers attend directly to earlier representations. A new paper, Multi-Stream Attention Residuals (MSAR), extends that approach to multiple streams: different heads retrieve representations from earlier layers through different views of the sequence context.

In scaling-law experiments, the authors report a 1.58× compute advantage over standard residual connections at the same loss, compared with 1.35× for Attention Residuals and 1.30× for manifold-constrained Hyper-Connections.

A nice example of how a well-established idea can yield gains in new applications.

Paper: https://lnkd.in/damRha3s

Sunan Zou

Posted at: https://lnkd.in/p/dUrd6Ykm