Critique: a clean diagnosis, a partial cure

The claims

1. Cross-lingual knowledge compartmentalization already exists after pretraining (CLeq 0.9–5.9% across En–Ar, En–Ru, 360M and 7B), and code-switching or activation alignment barely help.
2. Disjoint token spaces are sufficient to cause it: identical English copies with shifted token IDs show CLeq ≈ 0.
3. Pretraining drifts apart tokens that start aligned (same-initialization CLeq 78.6%, below the 100% of tied embeddings).
4. Word-wise translation (WWT) into a shared token space substantially improves transfer (to 12.5–23.3%), and only when the mapping is semantically faithful.

Strengths

• The clone-language design isolates one variable. English₁–English₂ removes script, grammar, culture and data differences at once, which is exactly what you need to make a causal claim about tokens. Combining it with fictive-fact injection at known doses (Allen-Zhu & Li, 2024; Chang et al., 2024) is a genuinely clean methodology.
• The metric normalizes away competence. CLeq divides cross-lingual gains by native gains, so a bigger model that learns everything better doesn’t look like it transfers better. This is what lets the paper push back on “alignment emerges with scale” (Li et al., 2024).
• The controls are the right ones. Shuffled dictionary and transliteration share tokens without sharing meaning, and both fail. That turns “shared tokens help” into “shared meanings help”, a sharper and more useful claim.
• Ceiling calibration. Soft mapping shows that CLeq can actually reach ~100%, so low values reflect real compartmentalization rather than a broken metric.

Weaknesses

• “Sufficient” is shown, “main cause” is argued. Clones prove disjoint tokens can cause compartmentalization. In natural pairs, merging tokens recovers only 12.6% (Arabic) to 23.3% (Russian), so most of the gap remains. The conclusion says tokenization is “a substantial part of the story”; the title and abstract read stronger than that.
• The metric’s linearity matters more than the paper admits. CLeq is a ratio of linear-regression slopes, while acquisition is log-like. A quick simulation (reimplementation post) suggests the ratio is exact at 0% and 100% but can overstate intermediate transfer. That doesn’t change the paper’s main contrasts (near-zero vs. clearly positive), but it makes “12.6% of native efficiency” hard to interpret literally.
• Translationese. The Arabic corpus is machine-translated English. The authors acknowledge it and replicate with native Russian, which helps, but the main 7B result is on translated Arabic.
• Small differences, single runs. Each CLeq value is one pretraining run. Differences like 2.2% (code-switching) vs 2.5% (alignment) vs 0.9% (baseline) are within the noise level suggested by simulations of this design; the appendix’s significance tests cover the headline comparisons, not every bar.
• The WWT cost is real. About 30% more tokens for Arabic means 23% fewer Arabic documents under a fixed budget and slower inference. The positive perplexity result is encouraging, but a practitioner needs Arabic-side quality too (it’s in an appendix).
• Simple facts only. 2,048 single-attribute facts. Whether multi-hop or relational knowledge behaves the same is open.

How it could have been improved

• Report seeds or bootstrap intervals for every CLeq bar, not only headline comparisons.
• Add a curve-aware estimator (e.g. fit the log-like curve and solve for the exposure-equivalent) as a robustness column next to linear CLeq.
• Test a byte-level or tokenizer-free model in the clone setup. The paper raises this question itself; even one small run would tell us whether the barrier is token identity or something deeper.
• Lead with the diagnosis (Sections 4–5) in Figure 1, and present WWT as evidence for the mechanism rather than as the headline fix.

Verdict

Accept. The clone-language + fictive-fact methodology is a reusable tool, the sufficiency result is clean and surprising, and the semantic controls make the mechanism precise. The framing should be toned down from “why pretraining fails” to “disjoint tokens are one sufficient cause”.

Worth pursuing? Yes. The most promising direction is the gap the paper exposes: natural languages need far more sharing than clones before knowledge flows (see the redrawn Figure 2). Understanding why is a project in itself.