Xenomirant

TLDR: Tokenization is the way that text is segmented before being input to a language model. Despite never being exposed to alternative tokenizations during training, LLMs unexpectedly develop the capacity to comprehend and even produce incorrectly tokenized text. We believe that these behaviors are understudied from an alignment perspective, and hope that this post will provoke discussion of the topic. We aim to develop a guide to recent progress in understanding tokenization through a sequence of blogposts, geared toward alignment researchers, synthesizing recent literature and reporting experiments we conduct. This first post highlights counter-intuitive, safety-relevant behaviors of models with regard to tokenization and presents a guiding research agenda. 0. How tokenization works, quick recap In modern LLM research, tokenization is an essential initial step before pretraining. The tokenizer (commonly, byte-level byte-pair encoding or BBPE (Gage, 1994)) splits the internet-scale dataset into byte-level chunks and begins a sequence of merges, optimizing compression (via frequency-based merging in BPE) or pointwise mutual information (as in WordPiece (Song et al., 2021)). Building the vocabulary is usually greedy, in the sense that merges only add new tokens and do not remove previous ones, implying that individual characters are valid tokens. Throughout subsequent model training, the tokenizer is usually left unchanged. Although the choice of tokenizer is known to affect multi-lingual capabilities (Limisiewicz et al., 2023) as well as math and coding performance (Singh & Strouse, 2024), it rarely attracts the attention of alignment researchers. The mapping between token and character alphabets is uniquely determined by the tokenizer, which can be defined as a pair of maps . [1] The encoding function provides an injective mapping from byte/character sequences to token sequences [2]. Thus, any tokenization that lies in the image of , i.e. for any is a canonic