The original writing had 2 points that I found to be unclear:

1. What is $e_{\theta_m}$? What’s its dimension? whats its relation to $e_{\theta_{m-1}}$? I found the answers to be that while \theta_m contains \theta_{m-1}, the embedding layers are independent, and their dimension is the difference in dimension between different blocks.
2. The original writing “Each sub-model $T_{\theta_m}$ is a separate Transformer stack with its own hidden dimension (or width)” is I belive confusing; the following formulation from the main text indicates that the difference in dimension between a model and its direct sub-model is only within the new layers. I made this clearer by rewriting this using blocks where dimension is consistent, and those blocks are composed into different sub-models

Before and after

Three language models. One training run. 36% less training compute.

Matryoshka nests 500M, 1.5B and 3B LMs together and gets nearly the same benchmark accuracy as training each one separately.

2/ Think nesting dolls, but for Transformers.

Each larger LM builds on the smaller one. Each size has its own width, depth and output head, so you can deploy the 500M or 1.5B model without loading the full 3B model.

3/ The clever part: the 3B LM teaches the smaller ones during the same training run. No separate teacher pass.

At inference, shared layers and KV cache give 26% higher greedy speculative-decoding throughput than a separate 500M/3B pair in the reported A100 test.

4/ And the accuracy cost? At most 0.5 percentage points on average across 7 zero-shot benchmarks, at each model size.

Three deployable models. 38% fewer total parameters to store.

Paper: https://arxiv.org/abs/2608.09703

Bluesky

Why train 3 LMs separately? Matryoshka nests 500M, 1.5B and 3B models in one stack: 36% less training compute, with average zero-shot accuracy within 0.5 points of separate models. Shared KV cache also boosts speculative-decoding throughput by 26% in an A100 test. https://arxiv.org/abs/2608.09703

Mastodon

Three LMs. One training run.

Godey & Artzi’s Matryoshka suite nests 500M, 1.5B and 3B models instead of training each separately.

Result: 36% less training compute, average benchmark accuracy within 0.5 points, and 26% higher greedy speculative-decoding throughput than a separate draft/verifier pair in an A100 test.

Each model can still be deployed independently.

https://arxiv.org/abs/2608.09703 #NLProc #MachineLearning

LinkedIn

If you train language models in several sizes, why train each one from scratch?

Matryoshka Language Model Suites, by Nathan Godey and Yoav Artzi, nests 500M, 1.5B and 3B models into one architecture. Train them together, then deploy any size independently, with its own width, depth and KV cache.

The payoff: 36% less total training compute, while average accuracy stays within 0.5 percentage points of separately trained models across seven zero-shot benchmarks. Shared layers also make speculative decoding more efficient: 26% higher greedy throughput than a separate 500M/3B draft and verifier pair in the reported A100 setup.

The bigger idea: optimize the whole model family, not just the largest checkpoint. The results are promising, though so far tested only up to 3B parameters.

Paper: https://arxiv.org/abs/2608.09703

The paper tells a story about training a suite of stand-alone models more efficiently by stacking them and training end-to-end. It then makes a point about distiliation in this setup being cheap.

Strengths

I felt the article was short, concice and readable.

Weaknesses

It is not clear if distilation is crucial to performance. The main figure could include less information.

How it could have been improved

Show or talk about results for a vanilla-matryshuka setup(no purple arrows). If that makes sense, show a simpler figure of the architecture and make your main claim about efficiency (3.2B instead of 5.2B. 38%~ less params -> roughly the same in compute. perhaps you don’t need a figure?). Otherwise, in the case the purple arrows are crucial, express that.

Verdict

I liked the paper. The technique was clear even for a novice AI reader (myself). The technique itself is cool.

Figure 2 mixes two kinds of things in one bar chart: conditions (baseline, same data, same initialization) and points on a dial (soft-mapping at 20% and 99.9%, where a fraction p of each token pair’s embedding is shared). The text’s main claim about this experiment is a sharp threshold: “At 20% tying, generalization remains weak with CLeq = 8.6%; at 50% tying it jumps to 82.7%”. But the 50% point, the one that shows the jump, is not in the figure.

The full sweep is in the appendix (the soft-mapping table), and it covers two settings: the clone languages, and English–WWT-Arabic. Plotted together, they tell a better story.

The redesign

Each line is normalized by its own value at full sharing (p = 100%), so both fit on one axis: “how much of the achievable transfer do you get when sharing a fraction p of the embedding?” Hover a point for the raw CLeq.

{
  "type": "line",
  "data": {
    "labels": ["0%", "20%", "50%", "90%", "99%", "99.9%", "100%"],
    "datasets": [
      {
        "label": "English₁–English₂ (clone languages)",
        "data": [-2.6, 8.6, 82.7, 95.8, 99.1, 99.0, 100],
        "borderColor": "#2f6fde",
        "backgroundColor": "#2f6fde",
        "borderWidth": 3,
        "pointRadius": 5,
        "tension": 0
      },
      {
        "label": "English–WWT-Arabic (natural pair)",
        "data": [null, 7.9, 13.5, 76.2, 83.3, 81.0, 100],
        "borderColor": "#e8590c",
        "backgroundColor": "#e8590c",
        "borderWidth": 3,
        "pointRadius": 5,
        "tension": 0
      }
    ]
  },
  "options": {
    "responsive": true,
    "plugins": {
      "title": { "display": true, "text": "Share of the achievable transfer vs. fraction of embedding shared" },
      "legend": { "position": "bottom" }
    },
    "scales": {
      "x": { "title": { "display": true, "text": "Fraction p of each token pair's embedding that is tied (not to scale)" } },
      "y": { "min": -10, "max": 105, "title": { "display": true, "text": "% of CLeq at full sharing" } }
    }
  }
}

What the new figure shows that the old one didn’t

• The cliff is real, and it’s in a different place for the two settings. Clone languages jump between 20% and 50% sharing. The natural pair is still at 13% of its ceiling at 50%, and only jumps between 50% and 90%. Natural languages need much more sharing before knowledge flows.
• Reserving a few language-specific dimensions is cheap. At p = 90% (96 free dimensions out of 960) the natural pair keeps 76% of the benefit, which supports the paper’s “soft mapping” proposal for restoring language identity.
• It invites the right next question: where exactly between 50% and 90% is the WWT cliff, and why is it later?

Design decisions

• A line, not bars, because p is a dial, not a set of categories.
• Normalize each series so a 12.6%-max curve and a 100%-max curve can share an axis. The raw numbers stay in the table and tooltips.
• Conditions that aren’t on the dial (same data, same initialization) belong in a separate, small panel or table, not in the same bar list.
• Honest axis: the x-axis is evenly spaced but the values are not, so it says “not to scale”. Plotting the number of free dimensions on a log axis would be a scale-true alternative; try it!

Data: the appendix soft-mapping table and Figure 2 of (Gaber et al., 2026).

1. Cross-lingual knowledge compartmentalization already exists after pretraining (CLeq 0.9–5.9% across En–Ar, En–Ru, 360M and 7B), and code-switching or activation alignment barely help.
2. Disjoint token spaces are sufficient to cause it: identical English copies with shifted token IDs show CLeq ≈ 0.
3. Pretraining drifts apart tokens that start aligned (same-initialization CLeq 78.6%, below the 100% of tied embeddings).
4. Word-wise translation (WWT) into a shared token space substantially improves transfer (to 12.5–23.3%), and only when the mapping is semantically faithful.

Strengths

• The clone-language design isolates one variable. English₁–English₂ removes script, grammar, culture and data differences at once, which is exactly what you need to make a causal claim about tokens. Combining it with fictive-fact injection at known doses (Allen-Zhu & Li, 2024; Chang et al., 2024) is a genuinely clean methodology.
• The metric normalizes away competence. CLeq divides cross-lingual gains by native gains, so a bigger model that learns everything better doesn’t look like it transfers better. This is what lets the paper push back on “alignment emerges with scale” (Li et al., 2024).
• The controls are the right ones. Shuffled dictionary and transliteration share tokens without sharing meaning, and both fail. That turns “shared tokens help” into “shared meanings help”, a sharper and more useful claim.
• Ceiling calibration. Soft mapping shows that CLeq can actually reach ~100%, so low values reflect real compartmentalization rather than a broken metric.

Weaknesses

• “Sufficient” is shown, “main cause” is argued. Clones prove disjoint tokens can cause compartmentalization. In natural pairs, merging tokens recovers only 12.6% (Arabic) to 23.3% (Russian), so most of the gap remains. The conclusion says tokenization is “a substantial part of the story”; the title and abstract read stronger than that.
• The metric’s linearity matters more than the paper admits. CLeq is a ratio of linear-regression slopes, while acquisition is log-like. A quick simulation (reimplementation post) suggests the ratio is exact at 0% and 100% but can overstate intermediate transfer. That doesn’t change the paper’s main contrasts (near-zero vs. clearly positive), but it makes “12.6% of native efficiency” hard to interpret literally.
• Translationese. The Arabic corpus is machine-translated English. The authors acknowledge it and replicate with native Russian, which helps, but the main 7B result is on translated Arabic.
• Small differences, single runs. Each CLeq value is one pretraining run. Differences like 2.2% (code-switching) vs 2.5% (alignment) vs 0.9% (baseline) are within the noise level suggested by simulations of this design; the appendix’s significance tests cover the headline comparisons, not every bar.
• The WWT cost is real. About 30% more tokens for Arabic means 23% fewer Arabic documents under a fixed budget and slower inference. The positive perplexity result is encouraging, but a practitioner needs Arabic-side quality too (it’s in an appendix).
• Simple facts only. 2,048 single-attribute facts. Whether multi-hop or relational knowledge behaves the same is open.

How it could have been improved

• Report seeds or bootstrap intervals for every CLeq bar, not only headline comparisons.
• Add a curve-aware estimator (e.g. fit the log-like curve and solve for the exposure-equivalent) as a robustness column next to linear CLeq.
• Test a byte-level or tokenizer-free model in the clone setup. The paper raises this question itself; even one small run would tell us whether the barrier is token identity or something deeper.
• Lead with the diagnosis (Sections 4–5) in Figure 1, and present WWT as evidence for the mechanism rather than as the headline fix.

Verdict

Accept. The clone-language + fictive-fact methodology is a reusable tool, the sufficiency result is clean and surprising, and the semantic controls make the mechanism precise. The framing should be toned down from “why pretraining fails” to “disjoint tokens are one sufficient cause”.

Worth pursuing? Yes. The most promising direction is the gap the paper exposes: natural languages need far more sharing than clones before knowledge flows (see the redrawn Figure 2). Understanding why is a project in itself.

Suppose you want to know whether reading a fact in Arabic helps a model recall it in English. You need to know how many times the model saw that fact, in which language. For real facts (“Paris is the capital of France”) this is hopeless: the pretraining corpus has billions of documents, the fact appears in countless phrasings, and counting is at best an estimate (Elazar et al., 2024).

The idea: invent the facts

Instead of counting real facts, make up new ones and insert them into the training data yourself:

1. Invent entities that cannot exist in the corpus. “Cultramive is a company that sells silver-plated bicycles.” No document on the web mentions Cultramive, so anything the model knows about it came from you.
2. Write each fact many ways. Generate many short documents that paraphrase the same fact, so the model learns the fact, not one sentence. Do it in each language you care about.
3. Control the dose. Decide per fact how many documents to insert in each language. This paper uses every combination of {0, 20, 100, 1000} exposures in language A × the same in language B.
4. Test with held-out questions. Ask multiple-choice questions written separately from the training documents (“What does Cultramive sell?”), so a correct answer means recall, not copy-paste.

Now every fact has a known recipe (say, 100 exposures in English, 20 in Arabic) and a measured outcome (accuracy on its English questions). That turns “does knowledge transfer?” into a regression you can fit.

Two details that make it work

• Check that the facts are learnable at all. With 0 injections accuracy should sit at chance (25% for four options) and rise with exposures. The paper checks this first, and finds a log-like learning curve.
• Keep the injected data tiny. The fictive documents are a sliver of the training tokens, so they don’t change what the model learns about the language itself.

Where this comes from

Controlled injection of synthetic facts was popularized by the Physics of Language Models series (Allen-Zhu & Li, 2024) and by studies of how facts are acquired during pretraining (Chang et al., 2024). This paper’s contribution is to combine it with two languages and cloned vocabularies, so the dose in each language and the only difference between the languages are both under control.

Try it yourself

You can run a miniature version on a laptop: train a small GPT on TinyStories, insert 50 invented facts at different rates, and plot accuracy against exposures. The reimplementation post simulates the analysis side.

1. Is CLeq linear enough? (a weekend, no GPUs)

Question: how much does the linear-slope ratio distort intermediate transfer values, given the log-like learning curve? Experiment: simulate the exact design (16 groups × 128 facts, doses {0, 20, 100, 1000}) under different curve shapes and known true transfer, and compare linear CLeq to a curve-aware estimator. A first pass is in the reimplementation post. Why: every number in the paper goes through this metric. If it’s biased in a predictable way, a correction table makes all future CLeq numbers more interpretable.

2. Where is the WWT cliff? (a few small runs)

Question: with clone languages, sharing 50% of each embedding already gives 83% of full transfer. For English–WWT-Arabic, 50% gives only 13%, and 90% gives 76% (see the redrawn Figure 2). Why does the natural pair need so much more sharing? Experiment: sweep p between 50% and 90% at 360M, and repeat with WWT-Russian. Then vary dictionary quality (e.g. drop 10–50% of entries to transliteration) to see whether the cliff moves. Why: this is the gap between “tokens suffice in the lab” and “tokens are the fix in practice”.

3. Does a byte-level model escape? (one clean comparison)

Question: is the barrier token identity, or something any input encoding would show? Experiment: run the English₁–English₂ setup with a byte-level model, where the second copy uses a byte-level cipher (shifted byte values). Same data, same facts. Why: the paper explicitly leaves this open, and the answer changes whether tokenizer-free architectures are worth pursuing for multilinguality.

4. More than two languages (a real project)

Question: if Arabic, Russian and Hebrew are all mapped into English tokens, do they interfere, and does knowledge flow between the non-English languages through English? Experiment: trilingual and quadrilingual runs with WWT for all non-English languages, measuring CLeq for every pair. Why: the paper’s own limitations section names this as open, and it’s what a real multilingual model would need.

5. Beyond simple facts (a thesis chapter)

Question: does compartmentalization look the same for relational and multi-hop knowledge (“Cultramive’s founder studied at…”) or for skills like arithmetic? Experiment: extend the fictive dataset with two-hop facts and with a small synthetic skill, injected in one language and tested in the other. Why: the abstract talks about “knowledge”, but the evidence is single-attribute facts. Skills transferring (or not) would matter even more for practice.

How to judge a follow-up

Before proposing one, ask: If it works, would the authors want to cite it? If it fails, would we learn something? All five above pass both tests.

1. The cross-lingual knowledge gap (the problem)

Multilingual models answer the same factual question differently depending on the language. Qi et al. measured this inconsistency directly (Qi et al., 2023), and ECLeKTic built a benchmark around knowledge that exists in one language’s sources and must be transferred to others (Goldman et al., 2025). Earlier, Chang et al. studied the “curse of multilinguality” at the level of language modeling across 250 languages (Chang et al., 2024). All of these observe finished models; none can say where the gap comes from. That’s the opening this paper takes.

2. Cloned languages as a laboratory (the method, part 1)

The trick of duplicating a vocabulary to create a “fake second language” goes back at least to K et al. (2020), who used it to ask what makes multilingual BERT cross-lingual (K et al., 2020). Schäfer et al. (2024) used cloned languages to study language imbalance (Schäfer et al., 2024), and Howe & Wingate (2026) to study compartmentalization in aggregate terms (Howe & Wingate, 2026). This paper’s addition is to measure knowledge in the clone setup, fact by fact.

3. Injecting fictive facts (the method, part 2)

To know exactly how often a model saw a fact, insert facts that cannot exist anywhere else. Physics of Language Models 3.1 made this standard for studying knowledge storage (Allen-Zhu & Li, 2024), and Chang et al. (2024) used it to trace knowledge acquisition during pretraining (Chang et al., 2024). (Why counting real facts is hard: (Elazar et al., 2024).) See also the fictive-facts explainer.

4. Tokens, scripts and vocabulary overlap (the mechanism)

Does sharing subwords matter for transfer? The classic answer was “surprisingly little”: Conneau et al. and Artetxe et al. found that cross-lingual structure emerges without any shared vocabulary (Conneau et al., 2020; Artetxe et al., 2020). More recently Kallini et al. found that overlap, especially overlap with matching meanings, does help (Kallini et al., 2025), and RomanSetu proposed romanizing text to unlock multilingual ability (J et al., 2024).

This paper reconciles the two camps: structure does emerge without shared tokens (their embedding analysis agrees with Conneau et al.), but knowledge does not follow it. And their transliteration control (−0.3%) explains why romanization helps mainly between related languages, where cognates supply shared meanings.

The one it argues with

PreAlign reports that cross-lingual knowledge gaps narrow with scale and that alignment emerges spontaneously in larger models (Li et al., 2024). This paper finds scale helps only a little (0.9% → 5.9% from 360M to 7B) and not at all in the clone setup, and attributes the disagreement to measuring a normalized ratio rather than absolute accuracy. Reading both papers side by side is a good exercise in how metric choices shape conclusions.

Bigger picture

The conclusion connects to the Platonic Representation Hypothesis, the idea that models trained on different data and modalities converge toward similar representations (Huh et al., 2024). This paper is a caution: similar structure in two token spaces is not enough for knowledge to flow between them. And it points to tokenizer-free models such as ByT5 (Xue et al., 2022) as an untested way around the problem.

The paper’s metric, CLeq, is estimated from one regression over all 2,048 fictive facts:

where $N_A(f)$ and $N_B(f)$ are how many times fact $f$ was seen in each language. Pretraining models is out of reach for a homework, but the measurement can be tested: generate accuracies where I know the true transfer, run the paper’s estimator, and see what comes out.

The setup mirrors the paper: 16 groups of 128 facts, each group at one combination of ${0, 20, 100, 1000}$ exposures per language; 4-option multiple-choice questions (25% chance); a log-like learning curve, as the paper reports. “True transfer” $t$ means one foreign exposure is worth $t$ native ones, so a fact’s accuracy is the learning curve evaluated at $N_A + t \cdot N_B$.

The code

LEVELS = [0, 20, 100, 1000]          # exposure counts used in the paper
K = 60.0                             # how quickly learning saturates (smaller = faster)
FACTS_PER_GROUP = 128                # 16 groups x 128 facts = 2,048 facts
CHANCE, CEIL = 0.25, 0.9             # 4-way multiple choice; best reachable accuracy
QUESTIONS_PER_FACT = 4               # held-out MCQs per fact

def learning_curve(effective_exposures, k=K):
    """Log-like acquisition: fast at first, then saturating."""
    return CHANCE + (CEIL - CHANCE) * np.log1p(effective_exposures / k) / np.log1p(1000 / k)

def simulate(transfer, rng):
    nA, nB = np.meshgrid(LEVELS, LEVELS, indexing="ij")
    nA = np.repeat(nA.ravel(), FACTS_PER_GROUP)
    nB = np.repeat(nB.ravel(), FACTS_PER_GROUP)
    p = np.clip(learning_curve(nA + transfer * nB), 0, 1)
    acc = rng.binomial(QUESTIONS_PER_FACT, p) / QUESTIONS_PER_FACT
    return nA, nB, acc

def cleq(nA, nB, acc):
    X = np.column_stack([np.ones_like(nA), nA, nB]).astype(float)
    alpha, beta_A, beta_B = np.linalg.lstsq(X, acc, rcond=None)[0]
    return 100 * beta_B / beta_A

Full script (runs in a second, needs only numpy): cleq_sim.py

Results

200 simulated “pretraining runs” per row:

And how much it depends on the curve’s shape, for true transfer of 5% / 12.5%:

What this means

1. The endpoints are trustworthy. No transfer gives ≈0%, perfect transfer ≈100%. The paper’s central contrast, near-zero for disjoint tokens vs. clearly positive for shared ones, holds under this check.
2. In between, linear CLeq is generous when learning saturates. A fact seen 1000 times natively is near its ceiling, so native exposures look less valuable per unit, and the ratio favors the foreign slope. Under this model, a measured 12.6% would correspond to a true exposure-equivalence of roughly 2–8%, depending on the curve. So “12.6% of native learning efficiency” is best read as a ranking score, not a literal exchange rate.
3. One run has about ±2 points of noise. Differences like 0.9% vs 2.2% vs 2.5% (baseline vs code-switching vs alignment) are within a single standard deviation here.

Caveats

This depends on my modeling choice: that transfer acts like “extra effective exposures” fed into one shared learning curve. Other models of transfer would give a different correction. The paper says it tried a curve-aware estimator and found the linear one more informative. Comparing the two on simulations like these would be a nice follow-up (see five follow-ups).

What I learned

Before trusting a metric, simulate it with a known answer. It takes an hour and tells you which differences you’re allowed to read into.

Method from (Gaber et al., 2026).

The original abstract is accurate, but it reads like the paper’s table of contents: motivation, method, finding, method, finding. The most surprising result, that two identical copies of English fail to share knowledge when their token IDs differ, only arrives in the sixth sentence, and the abstract never says how bad the problem is in numbers. A reader skimming arXiv decides in two sentences whether to keep reading.

So the rewrite (1) opens with the phenomenon in concrete terms, (2) gives the key numbers, (3) says in one clause what the metric means, and (4) separates “what causes it” from “what fixes it”.

Before and after

A title to match. The current title, Why Pretraining Fails to Share Cross-Lingual Knowledge, promises an answer but doesn’t give it. An alternative that does: Disjoint Tokens Keep Knowledge Apart: Cross-Lingual Compartmentalization Starts in Pretraining.

The general rules

1. Lead with the finding the reader can’t predict. “LLMs transfer poorly across languages” is known; “identical copies of English don’t share knowledge” is new. Put the new thing early.
2. One claim per sentence, with its evidence. Replace abstractions (“a fundamental barrier”) with the observation that justifies them (“−2.6% between identical copies”).
3. Define your metric in words before quoting it. “Recovering 12.6% of native-language learning efficiency” is precise to the authors and opaque to everyone else; “one exposure in Arabic is worth 12.6% of one in English” is not.
4. Cut connective filler. “Guided by this understanding”, “To isolate its cause”, “While this limitation is well documented” make the reader wait for content.

What I’m unsure about

The rewrite drops “unlike human multilinguals”, which is a nice hook, and it adds a sentence (semantics matter) that the original leaves for the paper body. Whether that trade is worth it is a good thing to argue about in class.

Reference: (Gaber et al., 2026).