A new way of speculative decoding and on policy distillation

2 papers that expand on ideas used in the matryoshka suits paper.

🔭 Paper scouting · Recent papers nobody is talking about
on Matryoshka Language Model Suites
by matan2001 · October 24, 2026 · edit this page

The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation (Hao Li, MeiJia Chen, Weijie Ren, Donghan Li, Zijun Tian, Jingchun Huang, Naibo Wang) this paper introduce a new method to on policy distillation called RIDE, that try to make the student model surpass the teacher rather to simply imitate it.

WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models (Hyeongju Ha, Jae-Joon Kim) this paper introduce a new way for speculative decoding for looped language models, that parallelizes token drafting and verification across different recurrence depths, significantly accelerating inference without requiring an additional draft model.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • What Makes a Chain of Thought Easy to Learn?
  • Rewriting the abstract
  • A better Figure 2
  • The minds behind Matryoshka
  • Visualizing 🪆 Models