Week 0 in one page: tokens keep knowledge apart

The paper in three sentences

LLMs that learn a fact in one language mostly don’t know it in another, and Gaber et al. show this starts during pretraining (Gaber et al., 2026). Their cleanest experiment trains on two identical copies of English that differ only in token IDs, and finds the copies share essentially no knowledge. Mapping Arabic word by word into English tokens raises transfer 14×, but only when the mapping preserves meaning.

What we wrote

✍️ Writing. The abstract buries its best result in sentence six; our rewrite leads with it and puts numbers up front. Three things to remember checks whether the abstract, Figure 1 and the intro actually deliver the main points (the most original one, clone languages, is missing from Figure 1).

🧠 Learning. Fictive facts explains the trick that makes the whole paper possible: invent facts so you know exactly how often the model saw them.

📊 Visualization. A better Figure 2 plots the appendix’s soft-mapping sweep as curves, revealing that natural languages need far more embedding sharing than clones before knowledge flows. The core idea in one picture draws the three setups token by token.

🔬 Research. The critique: accept, with a clean diagnosis but framing stronger than the evidence. The literature map places it among cloned-language studies, fictive-fact injection, and the vocabulary-overlap debate. Five follow-ups range from a weekend simulation to a thesis chapter.

💻 Coding. Reimplementing CLeq simulates the metric with a known answer: exact at 0% and 100%, generous in between, about ±2 points of noise per run.

📣 Communication. The paper in four feeds: ready-to-post versions for X, Bluesky, Mastodon and LinkedIn.

What we argued about in class

(Fill in after class: the disagreements are the most interesting part.)

• Is “disjoint tokens are sufficient” the same as “disjoint tokens are the cause”?
• Should Figure 1 show the diagnosis or the fix?
• How literally should we read a CLeq of 12.6%?