Literature Review: Recurrent Depth and Latent Reasoning

The papers this builds on

Recurrent computation offers a way to increase a network’s effective depth while reusing its parameters. Universal Transformers introduced repeated, shared self-attention transformations across depth (Dehghani et al., 2019). Subsequent work showed that recurrent networks trained on simple problems could solve harder instances by performing more iterations at inference time (Schwarzschild et al., 2021). However, additional iterations can also degrade performance. Bansal et al. (2022) addressed this “overthinking” problem through input recall and progressive training.

These ideas connect to algorithmic reasoning and in-context learning. Giannou et al. (2023) demonstrated theoretically how looped Transformers can execute iterative algorithms, while Yang et al. (2024) showed that trained looped Transformers can match standard Transformers on several in-context learning tasks using substantially fewer parameters.

Geiping et al. (2025) bring recurrent-depth modeling to a 3.5-billion-parameter language model trained on approximately 800 billion tokens. Their model repeatedly applies a shared Transformer block to hidden representations. Randomized training depths and stabilization techniques enable performance gains from additional inference-time iterations, especially on mathematics and coding tasks. Its central contribution is demonstrating this approach at substantial language-model scale, without requiring specialized reasoning supervision.

The fields and concepts it connects to

A related research direction moves reasoning away from explicit text: Deng et al. (2024) gradually internalize chain-of-thought (CoT) steps; Cheng and Van Durme (2024) compress reasoning into continuous representations; and Hao et al. (2025) feed hidden states back as subsequent input embeddings. Concurrently, Saunshi et al. (2025) provide theoretical and empirical evidence for looped Transformers’ reasoning capabilities.

Together, these connections position recurrent depth as a complementary route to test-time scaling alongside explicit CoT. My main takeaway is that Geiping et al. establish the feasibility of large-scale recurrent reasoning; establishing when it offers the best accuracy–compute trade-off remains a separate question.