What Makes a Chain of Thought Easy to Learn?

In Matryoshka Language Model Suites, the authors use Knowledge Distillation (KD) to train smaller models using the predictions of a larger model.

This is a good opportunity to explore a related topic: learning from Chain-of-Thought (CoT). In particular, we’ll discuss how Knowledge Distillation can be used to transfer reasoning abilities from large language models to smaller ones, and what makes this approach effective.

But first, let’s introduce Chain-of-Thought reasoning.

From Answers to Reasoning

One of the important developments in LLMs is Chain-of-Thought (CoT) reasoning: instead of producing an answer directly, a model generates intermediate reasoning steps before arriving at its final answer.

For example, consider a simple multiplication problem:

Question: What is 17 × 24?

Answer only: 408.

With CoT: 17 × 24 = 17 × (20 + 4) = 340 + 68 = 408.

CoT can improve a model’s ability to solve multi-step reasoning problems at inference time. But it can also play an important role during training, by providing supervision on the intermediate reasoning steps rather than only the final answer.

Knowledge Distillation: Learning from a Teacher Model

Traditionally, language models were trained primarily on human-generated text. Today, model-generated synthetic data has become an important ingredient in LLM training, particularly during post-training.

One way to leverage stronger models is through Knowledge Distillation, where a smaller student model learns from a larger teacher model.

For autoregressive LLMs, the teacher can generate training sequences

\((y_1,\ldots,y_T),\) potentially including Chain-of-Thought reasoning. One way to train the student is to minimize the following distillation loss, which encourages it to match the teacher’s next-token probability distribution:

\[\mathcal{L}_{\mathrm{KD}} = \sum_{t=1}^{T} D_{\mathrm{KL}}\left( p_{\mathrm{teacher}}(\cdot \mid x,y_{<t}) \,\|\, p_{\mathrm{student}}(\cdot \mid x,y_{<t}) \right).\]

Here, $x$ is the question, $y_{<t}$ contains the preceding tokens, $p$ gives next-token probabilities, and $D_{\mathrm{KL}}$ (Kullback Leibler divergence) measures how much the teacher’s and student’s distributions differ.

Does distilling reasoning actually help?

In Distilling Step-by-Step (Hsieh et al., 2023), the authors show that teacher-generated rationales can substantially improve data efficiency. In one experiment, a 770M-parameter student outperformed a few-shot prompted 540B-parameter teacher on a reasoning benchmark.

DeepSeek-R1 (2025) provides another prominent example: reasoning traces generated by a powerful model were used to train smaller models, leading to strong reasoning performance.

These examples use supervision on generated reasoning tokens, rather than necessarily matching the teacher’s full probability distributions as in the equation above. Both approaches transfer information from the teacher to the student.

In my view, one way to understand the benefit of distillation is that it gives the student a richer learning signal: reasoning traces expose intermediate steps, while teacher probabilities provide information beyond the selected tokens. This suggests that we should ask not only whether a CoT is correct, but also how useful the information it provides is for learning.

How Does CoT Help Simple Student Models Learn?

One possible explanation is that CoT makes difficult learning problems easier by exposing intermediate computations.

Suppose the target function is a composition of several operations:

\[y = f_3(f_2(f_1(x))).\]

In ordinary supervised learning, the student only observes input-output pairs $(x,y)$.

It must learn the entire computation from these examples.

With complete CoT supervision, the student also observes the intermediate values:

\[z_1=f_1(x), \quad z_2=f_2(z_1), \quad y=f_3(z_2).\]

Instead of learning the entire mapping directly, the student can learn to generate a sequence of simpler intermediate computations.

Of course, additional supervision does not guarantee easier learning in every setting. But it suggests that the structure of the reasoning trace can fundamentally change the learning problem.

In our recent theoretical work, we investigate an idealized version of this phenomenon. We show that carefully structured CoT can help even a single autoregressive neuron learn complex computations, including tasks performed by much larger neural networks or any polynomial-time Turing machine. The key is to provide the intermediate steps during training. Without those steps, learning from questions and final answers alone can be provably hard.

This suggests that a good CoT should be judged not only by whether it reaches the correct answer, but also by how easily the student can learn from it. A longer trace with simple steps might be more useful than a shorter one with difficult jumps. The goal, then, is to design reasoning traces that make learning easier for the particular student.