Multi-Stream Attention Residuals, The Author's Path.

A bit about the Authors’s path.

Xiaohan Qin (First Author)

Xiaohan Qin is a Ph.D. student in Computer Science at Shanghai Jiao Tong University (SJTU). She completed her undergrad (CS) at SJTU in 2025 and now works at SJTU’s ReThinkLab and the School of Intelligent Instruments (SII). Her research focus is on neural network architecture and training improvements. Specifically, Transformer modeling and multi-task learning. Qin previously worked on papers like “Expanded Hyper-Connections” (xHC), which augments the Transformer’s residual streams for better capacity, as well as methods like JTok (token self-modulation) and NITP (Next Implicit Token Prediction) for more efficient language-model pretraining. She has also worked on multi-task fairness.

The current paper Multi-Stream Attention Residuals fits naturally into Qin’s trajectory: it is another architecture enhancement for Transformers that aims to improve training efficiency. In fact, she is listed as a co-author on related works like xHC (expanded hyper-connections). In summary, this paper continues her research within large language models and improvements to the Transformer’s architecture.

Junchi Yan (Last Author)

Junchi Yan is a professor (Distinguished Research Fellow) at Shanghai Jiao Tong University’s School of Artificial Intelligence (and School of Computer Science). He earned his Ph.D. in electronic engineering from SJTU (2015, under Wei Liu) and has been on the SJTU faculty since. Yan is the founder and leader of SJTU-ReThinkLab (in partnership with the Shanghai Innovation Institute, SII). Under his leadership, ReThinkLab focuses on “cutting-edge technologies” combining data-driven learning with optimization: spanning areas like foundation LLMs, multi-task learning, and even quantum machine learning. For example, his recent publications include work on Hyper-Connections (coauthored with Qin and others), as well as on quantum neural operators and reinforcement learning topics.

This paper aligns with Yan’s general goal of improving deep learning architectures and their training. As co-author and lab director, Yan has steered many projects on Transformer scaling (e.g. Hyper-Connections, MoE upcycling, token scaling axes) and multi-objective optimization. Multi-Stream Attention Residuals continues in that diretion by proposing a new residual scheme for Transformers. Specifically, the idea of having multiple attention streams over previous layers extends the line of research (from manifold-constrained Hyper-Connections to attention-based residuals) that Yan has been involved in. In short, this paper represents the next step in the authors series of works on stronger residual connections and more efficient training for large models than that we currently have today, fitting squarely into Yan’s lab mission in machine learning and applications.