Critique of Multi-Stream Attention Residuals

Is their technique really attention over residuals?

🔬 Research skills · Critique
on Multi-Stream Attention Residuals
by Adam Gabet · October 29, 2026 · edit this page

Summary of the claims

Attention Residuals (AttnRes) replace the fixed residual sum with softmax attention over earlier layers (Kimi Team, 2026). Each layer has one learned query. That is one attention distribution over depth. MSAR claims to make it multi-head (Qin et al., 2026). It expands each stored source into 4 streams. Stream 0 is the source itself. Streams 1–3 are causal depthwise convolutions over previous input tokens with 4 lengths (0, 4, 8, 12).

The claims: lowest loss at 18B (1.750 vs 1.769 for AttnRes, Fig. 1a). 1.58Ă— iso-loss compute speedup over Vanilla, vs 1.35Ă— for AttnRes (Fig. 3). Cost: +0.3% parameters and +0.5% FLOPs (Table 5).

Strengths

The idea is simple. Splitting the hidden state into heads doesn’t work. Copies of the same vector give identical keys. Borrowing diversity from the previous token avoids both. Pipeline communication stays at AttnRes cost. They win in the loss at (almost) same compute.

Weaknesses

The gain may come from token mixing, not from multiple depth heads. The convolutions add information about earlier tokens into every sublayer input. This includes MLP sublayers, which normally see only the current token. A short causal convolution might be a known win. Canon layers put a kernel-4 causal convolution before attention and before the MLP (Allen-Zhu, 2025). Primer added depthwise convolutions after Q, K and V and cut training compute by up to 4Ă— (So et al., 2021).

Efficiency is reported in FLOPs, not time. Memory traffic is 21.5d per token per sublayer. AttnRes is 5.5d (Table 6). The 1.58Ă— is a FLOP ratio. No throughput is reported. The convolution also breaks the per-token locality of AttnRes. Sequence parallelism now needs a halo exchange across shards which introduces communication overhead. Decoding needs a convolution state at every layer.

How it could have been improved

Add one control: AttnRes plus a single causal convolution on the sources, one head. Add Vanilla plus Canon-A/C. If these match MSAR, the contribution is token mixing. Report tokens per second. Test long context with sequence parallelism.

Verdict

Would you accept it? Is the direction worth pursuing? Why?

I wouldn’t accept until they show the gain is from the “attention” on residual connection and not convolution on the previous tokens. They also need to prove that this can be trained efficiently at scale with training time not just flops.

References

  1. Attention Residuals
    Kimi Team
    arXiv preprint arXiv:2603.15031, 2026
  2. Multi-Stream Attention Residuals
    Xiaohan Qin, Xiangdong Zhang, Tuo Dai, and 7 more authors
    arXiv preprint arXiv:2610.13630, 2026
  3. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
    Zeyuan Allen-Zhu
    In Advances in Neural Information Processing Systems, 2025
    arXiv:2512.17351
  4. Primer: Searching for Efficient Transformers for Language Modeling
    David R. So, Wojciech Mańke, Hanxiao Liu, and 3 more authors
    In Advances in Neural Information Processing Systems, 2021


Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • MSAR Explainer Video
  • What are these evals anyway?
  • Week 1 in one page: three models, one Matryoshka suite
  • Copyedit Matryoshka Lm Suites_2
  • Copyedit Matryoshka Lm Suites