Tokenization: A Survey for Modern NLP

Marco Cognetta, Christopher Akiki, Pawan Sasanka Ammanamanchi, Catherine Arnett, Thomas Bauwens, Pavel Chizhov, Pieter Delobelle, Konstantin Dobler, Desmond Elliott, Omer Goldman, Tatsuya Hiraoka, Valentin Hofmann, Eugene Jang, Julie Kallini, İlker Kesen, Sachin Kumar, Sander Land, Jindřich Libovický, Tomasz Limisiewicz, Alisa Liu, Clara Meister, Benjamin Minixhofer, Sangwhan Moon, Tiago Pimentel, David Pohl, Suchir Salhan, Tomohiro Sawada, Pavel Stepachev, Gianluca Vico, Philip Whittington, Xiulin Yang, Yuval Pinter, 2026

Paper
TL;DR (To be written by the class after week 4. Edit this file: one or two sentences someone can read in 10 seconds.)

A long survey (32 authors) of how tokenizers are built and where they go wrong: normalization, subword algorithms, vocabularies, multilingual imbalance, evaluation, theory, and alternatives such as latent or visual tokenization. It is long, so for your task read the main part and the sections it needs.

What the class wrote 0

Nothing yet. Pick a task on the roles page and write the first post, see how to post.

Who is doing what (week 4)

Who does what in week 4: see the sign-up sheet. Its Week check tab shows what is still missing.

Edit this paper's page

Write about this paper: Write