Week 2 in one page: Multi-Stream Attention

The Paper’s Main Points

The paper discusses residual connections in LLMs, it builds upon and improves a line of literature that has seeked to replace the standard skip connection. The standard skip connection has a major flaw — information dilution, all previous layer representations are compressed into a single vector. Previous works like mHC and AttnRes have improved on this by suggesting an attention-like mechanism for selectively and dynamically attending to previous layer representations. Multi-Stream Attention Residuals (MSAR) wishes to extend this into a multi-head(stream) attention, the challenge presented is doing so without additional linear projections or presistent memory. The authours solve this using causal convolutions along the sequence dimension, expanding previous layer hidden states into multiple representations (streams), for which they apply depth-wise attention like in AttnRes then fuse represntations back into a single vector like in mHC.

What we made

✍️ Writing: clearing up jurgon and notations

Writing this week focused on clarity. Many in class felt the paper requires a strong background that wasn’t clearly presented (expressed in the main points post); the abstract rewrite focused on fixing this and making the paper more accessible. The methods section rewrite improved clarity by replacing the difficult-to-follow formal notation with a simple table that summarizes the method. Another abstract rewrite chose to highlight the most compelling result (1.58x iso-loss compute speedup) earlier in the abstract, where reader attention is at its peak.

🧠 Learning: strengthening our base

Explaing HC and mHC post provides an in-depth overview of a key advancement in the modern residual connections lien of work. It walks the reader through one of the more mathematically complex solutions in this field, explaining the rational behind it and providing clear examples — a must-read for anyone looking to dive deeper into this research area. What are these evals anyway? is a high-level and comprehensive review of LLM evaluations, including the different categories of evaluations, their meaning, and even some very helpful examples from popular evals, very valuable information for approaching any paper across the LLMs domain and trying to decide what “being better” actually means.

📊 Visualization: choose the explanation that clicks

Some of this week`s visualizations focused on conveying the paper’s main mechanism and which hidden states contribute to the residual connection of each layer, suggesting an alternative to the paper’s figure2 and designing an interactive interface that allows playing with diffrent configurations. Another suggestion was A better Figure 1, tweaking its presentation to suggest a better first impression. Finally, A Better Figure 1(c) suggests a revisualization of the evaluation comparision results, adding the dynamics of these results across scales.

🔬 Research: the missing control

The author background helps us understand the paper is part of the authors’ current work towards perfecting residual connections (More recent work xHC) as part of their overall research theme of improving LLM pretraining by optimizing aspects of the architecture design. The literature review provides context that came up again and again in class as essential for understanding this paper.

This week’s critiques (1, 2, 3, 4) while mostly positive emphsizing the impressive empirical gains and the succesfull effort to minimize the cost they come at, all boil down to the same question: is the mechanism nessecary or are the gains coming from somewhere else? Critique 1 praised the w/o convolution control in this context while the other critiques mentioned “Canon layers” that improve performance via a causal convolution before attention and before the MLP alone (Allen-Zhu, 2025), suggesting the authors conflated the local context bias from the convolutions with their multihead attention. Critique 4 provides a suggestion for a control to test this theory: “AttnRes plus a single causal convolution on the sources, one head. Add Vanilla plus Canon-A/C. If these match MSAR, the contribution is token mixing.”

The proposed follow-up tests an asymmetric design using multi-stream retrieval for Attention sublayers and standard residual connectionss for the subsequent MLPs. The rational lies within the depth-wise attention maps visualizations from the paper appendix which show MLPs primarily rely on the immediate preceding sublayer anyway, this approach could drastically cut memory I/O overhead without sacrificing performance.

🔭 Scouting: depth scaling and more

The scouting posts (1, 2, 3) presented a few complementary directions as well as unrelated interesting developments. Recurrent Looped Transformer and Skip a Layer or Loop It? Learning Program-of-Layers in LLMs presented in scouting posts 1 and 2 both tackle dynamic scaling of LLM depth, perhaps suggesting an alternative solution to the residual dilution problem when increasing depth.

📣 Communication: from the article to a feed

The LinkedIn post presents the rationale for adapting a familiar idea like multi-head attention to AttnRes. There is also a video explainer.