The core idea in one picture
Same sentence, same model, different token IDs, and the knowledge stops flowing. A diagram of the three setups.
The paper has one idea you can draw: knowledge flows between languages only through tokens they share, and only if those tokens mean the same thing. Three setups show it. Blue and orange are separate token spaces; green is a shared one. The small numbers are token IDs (with a vocabulary of size V). Words are shown left to right in reading order.
1. English1 – English2: identical text, disjoint token IDs
2. English – WWT-Arabic: Arabic grammar, English tokens
3. Controls: shared tokens, wrong meanings
The token IDs and the two-word Arabic example are illustrative, not taken from the paper's tokenizer. The paper's own WWT example is below.
Why this picture
The paper’s Figure 1 shows results (bars). A reader who hasn’t read Section 5 can’t tell what “English₁–English₂” or “WWT” even are. A diagram of the inputs explains the experiments in five seconds, and makes the conclusion feel obvious in hindsight, which is what a good key figure does.
Ideas for doing better: animate it (the same sentence morphing between setups), or add a fourth row with soft mapping, where each token pair shares only part of its embedding.
Numbers from (Gaber et al., 2026).
References
- Why Pretraining Fails to Share Cross-Lingual Knowledge2026
Enjoy Reading This Article?
Here are some more articles you might like to read next: