TLDR: Tokenization is the way that text is segmented before being input to a language model. Despite never being exposed to alternative tokenizations during training, LLMs unexpectedly develop the capacity to comprehend and even produce incorrectly tokenized text. We believe that these behaviors are understudied from an alignment perspective, and hope that this post will provoke discussion of the topic.

We aim to develop a guide to recent progress in understanding tokenization through a sequence of blogposts, geared toward alignment researchers, synthesizing recent literature and reporting experiments we conduct. This first post highlights counter-intuitive, safety-relevant behaviors of models with regard to tokenization and presents a guiding research agenda.

0. How tokenization works, quick recap

In modern LLM research, tokenization is an essential initial step before pretraining. The tokenizer (commonly, byte-level byte-pair encoding or BBPE (Gage, 1994)) splits the internet-scale dataset into byte-level chunks and begins a sequence of merges, optimizing compression (via frequency-based merging in BPE) or pointwise mutual information (as in WordPiece (Song et al., 2021)). Building the vocabulary is usually greedy, in the sense that merges only add new tokens and do not remove previous ones, implying that individual characters are valid tokens. Throughout subsequent model training, the tokenizer is usually left unchanged.

Although the choice of tokenizer is known to affect multi-lingual capabilities (Limisiewicz et al., 2023) as well as math and coding performance (Singh & Strouse, 2024), it rarely attracts the attention of alignment researchers.

The mapping between token and character alphabets is uniquely determined by the tokenizer, which can be defined as a pair of maps . ^[1] The encoding function provides an injective mapping from byte/character sequences to token sequences ^[2]. Thus, any tokenization that lies in the image of , i.e. for any is a canonical token sequence. (Gastaldi et al., 2024).

Unlike the encoding function, decoding is surjective, so multiple token sequences can map to the same character sequence. In addition to the canonical tokenization, , for example, each character sequence admits a character-based tokenization, where each symbol is treated independently. A paper by Veira et al. (2025) provides a nice illustration of various tokenizations of a single string.

Thus, following Gastaldi et al. (2024), we call the tokenization canonical, iff for a particular token sequence .

What's the object we are dealing with?

In general, language modelling is a task of learning a probability distribution over , a possibly infinitely long sequence of bytes^[3]. Models are only exposed to the canonical token sequences during pretraining and SFT, and, statistically speaking, a theoretically optimal language model would assign zero probability to any non-canonical token sequence to preserve consistency with the underlying measure .

However, this isn't the case for real models due to non-negativity of the softmax function. Yet it seems reasonable to assume that any local sub-sequence of tokens that was never observed in the training dataset (and, unlike synonyms or any other semantic relations), should be assigned a negligibly low probability. After all, unlike synonyms, non-canonical sequences have no clear frequency co-occurrence patterns which could serve to infer the underlying meaning, as there's no shared context where such tokenizations could be found.

Here's the argument. There's a well known quote:

You shall know a word by the company it keeps. (Firth, 1957)

This is the basic principle initial static language models, like Word2Vec (Mikolov et al., 2013), more modern masked language models (Le Breton et al., 2025) and general masked autoencoders (He et al., 2021) are built around: some parts of the object get masked, and the model is trained to infer the correct ones from the present ones. If we retokenized just one word, or a small subset, we could argue that the model is performing such error-correction, internally estimating the correct "misspelled" parts.

Importantly, this is not the case. When the whole context gets re-tokenized^[4], the model still appears to infer the meaning of the sentence, despite not having any in-distribution context it could rely on. Thus, it seems to operate on some level that, unlike semantics in its late Wittgensteinian view (1953), can not be inferred from frequency patterns alone.

Below, we give a compact overview of some key ideas and research questions as thesis statements. The claims may not make sense without reading the full post, and will be further developed in future writing-- worry not if you are lost!

Here are some key ideas to keep in mind while reading this post.

• Tokenization is not simply a source of contextual compression allowing for more compute-efficient generation, nor should it be thought of as a mere tool to obtain larger semantically meaningful text chunks (i.e. 1-to-1 word-token mappings). Rather, tokenization interacts with model capabilities throughout the training pipeline in unexpected ways: pre-trained models are partially able to comprehend non-canonical sequences despite never being exposed to them, and SFT strengthens this invariance behaviorally.
• LLMs generate non-canonical token sequences, and these have downstream effects, since the tokens the model produces become tokens it has to read.
• Alignment training that does not explicitly expose the model to non-canonical token sequences may fail to generalize to such text, leading to irregular model behavior during, among other times, chain-of-thought reasoning and accompanying in-context learning.
• Although, as Jain et al. (2026) showed, some invariance is already learnt at pre-training, and average residual stream activations preserve stability w.r.t. re-tokenizations, these symmetries seem to be reinforced and clarified during instruction tuning (Zheng et al., 2026), and we should aim to understand this interplay more thoroughly.
• Invariance capabilities stay underdeveloped in most sub-8B models we've studied, while an ability to exploit these invariances at test time gets suppressed by further RL training. Although it is reasonable to wonder whether exploiting such invariances is beneficial and whether the model is aware of the possibility to exploit them for "hidden" computations in a manner similar to "dot-by-dot" reasoning computations (Pfau et al., 2024), yet we are not aware of any empirical evidence for their ability beyond some reader-side examples given by Jain et al. (2026).
  

Here is an overview of our guiding research questions:

• Alignment training seems to be revertible and unstable to simple re-tokenizations. Under which conditions does it happen, and could we build defenses by consistency training with respect to. re-tokenizations?
• How is in-context learning (ICL) affected by re-tokenizations? Do splits in certain regions/tokens matter more than the others? Are there splits with far-reaching effects on the residual stream?
• Is exploration during RL affected by the amount of non-canonicity, and can we build more robust initializations without costly data interventions? Synthetic data generation is used as a tool to increase diversity of data and promote better generalization, yet it's also known to trigger "model collapse" (Shumailov et al., 2024) beyond certain limit. Re-tokenizations allow to increase representation diversity without residing to synthetic generative processes.

Now the heart of the post. There are two sections. The first surveys recent research, including a few of our own experiments, on tokenization symmetries from a fundamental alignment perspective. The second introduces a conceptual research agenda that we have been working on for a few weeks. We seek feedback and questions on the agenda.

1. Re-tokenization displays unexpected symmetries and evades alignment

1.1 LLMs learn partial symmetry over re-tokenizations

So LLMs are trained on canonical token sequences to predict canonical token sequences. What could go wrong? Yet something interesting happens. Despite never being exposed to non-canonical tokenizations, modern LLMs appear to be capable of both reading and writing non-canonical token sequences.^[5]

1. They seem to assign meaningful conditional probabilities to continuations, given non-canonically tokenized prompts / task descriptions^[6] and seem to be relatively robust to such perturbations, though the robustness decreases with the degree of fragmentation (number of splits) away from the canonical tokenization. Moreover as Zheng et al. (2025) found, in several tasks, such as , , and , re-tokenizations can even boost capabilities without any fine-tuning or adaptation.
   Essentially, we can assume that models end up learning either the merge structure of the tokenizer to perform several tasks -- especially the ones that require handling arithmetic, bracket balance etc., or the underlying byte sequence distribution. Mechanistically, Yang et al. (2026) showed that some early layer attention heads can perform "word recovery" from character-level tokenizations, building residual stream representations similar to the one of the canonical token embedding. Yet the question of how these attention heads emerge, whether they are stable across re-tokenizations options, and why re-tokenizations can appear beneficial in certain domains, despite requiring additional computations, are left without answers. Psycho-linguistic experiments in token counting (Kaushal & Mahowald, 2022) and morphological and phonological feature inference (Hiraoka & Okazaki, 2024) seem to further support the hypothesis that models develop byte-level awareness.

2. This blogpost by Beren Millidge gives some explanation for why non-canonical tokenization may benefit the model's performance for arithmetic tasks. Overall, number representations (especially for large numbers) are awful and lack consistency, while differences in their representations (e.g. whether delimiters are used or not) introduce additional variance. As a result, it might be "simpler" to learn the latent representation than to operate on its surface level embeddings. And, if the model ends up performing inductive learning to infer the tokenizer structure anyway, it's reasonable to assume that more advanced models might have an incentive to exploit it for increased exploration by e.g. treating non-canonical tokenizations as equivalence classes for some tasks, while residing to exploring their representations in superposition for the others. Jain et al., (2026) introduce to measure whether retokenized prompts can lead the model to explore alternative reasoning paths and increase the capabilities of the model. They claim that at least some of them are "unreachable" from the canonical tokenization, and should correspond to slightly different computations, performed by the model, as residual representations of tokenizations diverge at the last layers. Beyond using tokenization as a source of randomness, learned symmetries might allow advanced models to systematically vary their reasoning patterns; we take up this question in the research agenda.
3. The mechanism for symmetry is the model's exposure to word fragments in canonical contexts. Kaplan et al. (2025) found that early layers implement a general "detokenization" step. They take a run of subword pieces and collapse it into a single vector sitting near the embedding the word would have had as one token, so the rest of the network reasons over word-level units regardless of how the input was chopped up. The model has to learn this because ordinary pre-training is full of multi-piece words, such as rare words, morphology, proper nouns, non-English, and code identifiers. Therefore, composing pieces into a lexical unit is common, and the ability to read non-canonical sequences, including character-level segmentations, is a generalized capability.

4. LLMs sample non-canonical tokens at inference time, and the probability to sample non-canonical token sequences increases with length. Preliminary experiments show that on average, each non-canonical token raises the odds that the next word is non-canonical by 1.9–3.2×. The effect seems to be the strongest immediately after the insertion and decays with distance: for Olmo-7B, 2.6× at one to two words, 1.35× at thirty to sixty. Varying the fraction of a prompt’s words that are non-canonically tokenized while holding the prompt text byte-identical leads to monotonic rise in non-canonical token emission rates with that fraction, reaching 8–17× in comparison to the control when half the prompt is non-canonical^[7].
   Similar experiments were previously performed by Lui Geh et al. (2025). As more tokens get sampled (unconditionally), the probability of generating non-canonical tokenization grows -- yet the growth laws differ between models. Notably, some non-canonical tokenizations were found to be more likely than their canonical counterparts. They provide the string ␣ as an example, showing that a canonical tokenization ␣ by Gemma is less likely than non-canonical tokenization ␣^[8]. Similar observations on the fact that LLMs learn and model morphological and syntactic structures across languages, despite the fact the tokenization was never accounting for them, has been made by in prior research as well (Ismayilzada et al., 2025, Kennedy 2025).

1.3 We should care about tokenization more

Although LLMs seem to be aware of their tokenization to some extent, their understanding is fragile and context-dependent. If they could indeed infer and encode the merge structure into their computational circuits^[9], they wouldn't be dependent on the "distance" from the canonical tokenization. If LLMs encode the tokenizer's merge structure in some lossy way, understanding why and categorizing settings where they fail is relevant for the general study of what models know about their own properties and training.

Most importantly, Lui Geh et al. (2025) showed that re-tokenization can evade alignment without any changes to the prompt. Thus, by just re-tokenizing the prompt non-canonically, an attacker has an increased likelihood to bypass guardrails and alignment monitors. As most commonly used monitors are usually LLM-based as well, this attack vector could be effective at both levels, while being completely identical to regular text from the logger's perspective (assuming that the logger is unaware of the tokenization.)

Recent studies (Zheng et al., 2025; Lui Geh et al., 2025; Jain et al. (2026)) indicate that models do not have a systematic understanding of the merge structure or the underlying byte sequence distribution. Rather, the model has learned a partial symmetry over tokenizations of the same string with residual sensitivities that become more pronounced as the tokenization becomes more granular (see above figure). We are not aware whether residual sensitivity follows a similar pattern in frontier models.

In smaller models, the symmetry remains relatively underdeveloped and fragile. For example, Ismayilzada et al. (2025) evaluated several LLMs, from small models to GPT-4 and Gemini, for the ability to infer morphological compositional structures, and found performance to decrease sharply as morphological complexity increased. The extent of symmetry is not uniform across languages, either: Ghosh & Jyothi (2026) found that tokenization-invariance is mostly preserved for English, where most tokens have 1-to-1 lemma correspondence, while languages with higher token fragmentation show increased variance across re-tokenizations.

Although users cannot explicitly control the tokenization for closed-source LLMs, models expose themselves to non-canonical sequences during generation (Lui Geh et al., 2025). A model conditions on its own output. Non-canonical tokens it emits become context it must read, and reading non-canonical context is exactly what adversarial tokenization exploits. Therefore, non-canonical tokens could pose a problem during in-context learning (ICL) and cross-agent communication channels. If sampling non-canonical token sequences steers the model off-manifold (Wurgaft et al., 2026) of typical activations, it might eventually lead to dramatic changes in the residual stream while generating coherent and plausible text for CoT monitors.

If models are capable of exchanging meaningful messages, while keeping byte sequence monitors completely blind, we think, non-canonical tokenizations present another challenge for faithfulness and possible steganographic capabilities (Zolkowski et al., 2025) of language models, that, unlike prior studies, don't necessitate the use of ciphers or any changes to the statistical properties of the output text.

Although, this is not something that is empirically observed in smaller models, which tend to degrade and destabilize as non-canonicity compounds (Lui Geh et al., 2025), we can hardly predict whether this behavior is kept at scale, given the aforementioned findings and known incentives to perform latent reasoning.

2. Re-tokenization symmetry: research agenda

We suspect some of the findings described above might not hold for large-scale foundational models. Additional investigations are necessary to understand the effects that model scale and long-context RL have on non-canonicity and tokenizer-related properties of models.

Here's the list of questions and problems we find the most interesting. We are open to adjusting our research direction based on the feedback we get from the community.

We think most problems in this domain arise from the incompleteness of deep learning theory, entangling frequency-induced biases, optimization dynamics, feature learning and semantic generalization. Although there have been increasing attempts to build a theory of data attribution (Zhu & Cangelosi, 2025), inner lexicon generalization (Kaplan et al., 2025), effects of softmax, token frequency and optimization on latent stuctures (Gao et al., 2019) and exploration-driven incentives in advanced agents (Simchowitz & Slivkins, 2023), given that we don't have a well-developed deep learning applicable theory in either domain, we don't suspect any general theory of tokenization to arise soon.

Due to the emergent nature of the phenomena above, most experiments required to connect this application to learning theory will either be too costly (e.g. pretraining with a different tokenizer which has several desirable properties) or confounded by effects we can't control for (e.g. adding tokens or doing tokenizer transfer, similar to Minixhofer et al. (2025) or Haltiuk & Smywinski-Pohi (2026)). Thus, we target our efforts towards defense mechanisms and obtaining sufficient empirical evidence of the tokenizer-related behavioral patterns found in modern LLMs.

2.1 Our initial focus: Prompt re-tokenization symmetry and reader invariance.

Do small changes to the tokenization ever have large effects?

While previous approaches have studied how model performance, as measured by various benchmarks, depends on the prompt tokenization's distance from the canonical^[10], we are interested in fine-grained analysis of model behavior under single edits and sequences of edits to the context. Beyond performance evaluations, our upcoming posts will define behavioral metrics that attempt to capture the model's understanding of the context, instead of simply pass or fail.

• Which edits matter the most for behavior?
• How do small edits compound? 
• Does the effect depend heavily on the text being modified?
• Do the effects persist long-term, as we continue to sample from the model?
• Ultimately, can the model steer itself to a diverse set of behaviors by sampling non-canonical tokens, while evading CoT monitors?

Excluding the final bullet point, initial experiments have revealed partial answers to the questions above, and we expect to release an overview soon.

Do models necessarily learn invariances over tokenizations and which effects its reinforcement begets?

The counterpart to sensitivity is invariance, the symmetry that models learn over non-canonical tokenizations. We will deepen the study of reader invariance (does the model treat various tokenizations of the same text equivalently?) and raise new, unexplored questions of writer invariance (does the model assign significant probability to non-canonical token sequences?).

• Can we amplify the model's invariances intentionally via fine-tuning, and what side effects does it produce? How far does it generalize and can non-canonicity be treated as a feature (where we could expect learning and generalization, similar to Emergent Misalignment and other Narrow-to-Broad Generalization cases (Betley et al., 2025)) or does is the effect primarily restricted to particular tokens affected by consistency training? Some preliminary experiments for multilingual contexts were done by Ghosh & Jyothi (2026), while Steger et al. (2026) showed that finetuning with stochastic re-tokenizations (Sims et al., 2026) can improve both performance and robustness to non-canonically tokenized prompts.
• What does consistency training converge to? If we fine-tune the model to minimize the divergence between distributions given canonical and any other non-canonical token sequence, it might reinforce the biases imposed by the choice of the tokenizer and available distinct tokens. Does such reinforcement happen in practice and how does it affect the "inner lexicon"?
• How much is adversarial training different from uniform sampling based methods, and which model properties do these optimization targets amplify? Lui Geh et al., 2025 proposed an algorithm to search for adversarial tokenizations. If we use it for adversarial consistency training, do latent concept representations stabilize as well, leading to easier and more predictable steering (Wu et al., 2025)? Does it lead to increased adversarial stability to common attacks like GCG (Zou et al., 2023), AutoDAN (Liu et al., 2024) or FFA (Zhou et al., 2024) as well?
• If the platonic representation hypothesis (Huh et al., 2024) is true and models converge to similar word / concept representations, how does consistency training affect feature structure? Do models with different tokenizer sizes and token splits post-trained for consistency converge to similar or distinct distributions at scale? How much effect can be attributed to the tokenizer?
• Is character-based tokenization somewhat "easier" to infer in comparison to the random tokenization "quite far" from the canonical one? In this case we should see a U-shaped curve of performance/capabilities w.r.t. tokenization distance.
• Evals relevant: Models with web access or local hardware access could read tokenizer merge structure explicitly. Does it lead to larger stability? Which effects can we observe? We are unaware of any studies on this question.

2.2 Tokenization curvature and in-context learning (ICL) equilibria states

Finding a clean variational structure with a known ground truth is rare in theoretical deep learning settings, and most examples of this kind reduce to toy models with fixed decision rules or data generation routines. Due to emergent partial symmetry (see section 1.2 above), re-tokenization appears to be a useful playground for understanding language models' responses to small contextual changes, as re-tokenization isolates semantic meaning from learned representations. The possibility of varying the context in a controlled way motivates a dynamical systems view of the context-continuation divide, where the dynamics are defined by sampling from a model but are probed using re-tokenization. Thus in-context learning can be understood using mathematical tools from discrete differential geometry. This is one of the most exciting directions. We have some initial results here and plan to release a follow-up post in the upcoming weeks.

2.3 Long context ICL, RL exploration efficiency and writer invariance

Jain et al. (2026) showed that conditioning on non-canonical tokens in the prompt promotes diverse reasoning. We are interested in a separate question: would a model with writer-invariance, i.e. one that fluently produces various tokenizations of the same text during generation, have greater capability for diverse reasoning?

Models firstly expose themselves to non-canonical token sequences and start learning on them in online RL / RLVR. Most commonly used RL algorithms are known to trigger entropy collapse and fall to exploitation, which may hinder exploration and exploitation of non-canonical token sequences for computations . The idea is analogous to the neuroscientific result that knowing multiple languages can improve cognitive capabilities in humans (Bialystok et al., 2012). To address this issue in deep learning, entropy-regularized RL approaches, such as GFlowNets (Y. Bengio et al., 2021) perform amortized MCMC-like inference, and sample from the distribution proportional to the reward function without collapsing to a narrow set of solutions. Although this approach hasn't found as much use in LLM training as e.g. GRPO (Shao et al., 2024), there has been a growing body of evidence towards the benefits of entropy-regularized approaches for latent reasoning and improving exploration (Takase et al., 2024; Younsi et al., 2025).

Another recent line of work investigates the role of initialization for on-policy RL. Warm-start off-policy learning and distillation (Shrestha et al., 2025) are known as biased, but sample-efficient alternatives to on-policy RL. Some reweighting-based approaches like dynamical fine-tuning (DFT) (Wu et al., 2025) and bounded log-likelihood (BLL) (Zhao et al., 2026) have modify SFT so that it introduces less variance and mitigates overfitting to the SFT data distribution.

Separately, several works have proposed to depart from enforcing canonicity and explored an effect of stochastic re-tokenizations (Sims et al., 2026, Steger et al., 2026) on pretraining, supervised finetuning and in-context learning, and found them to improve robustness to both random and stochastic perturbations. In addition, it allowed to obtain more isotropic and disentangled concept features.

Essentially, stochastic tokenization imposes some degree of path invariance over the tokenization lattice, beyond increasing the entropy of the resulting distribution.

Let parametrize the joint distribution of byte sequences and their tokenizations , and let denote the number of distinct tokenizations obtainable from the tokenizer .

Then, if the distribution over tokenizations is uniform:

We can lower bound the entropy of the joint distribution as:

So the cross-entropy loss for each sample is lower-bounded by the logarithm of the number of plausible re-tokenizations, which becomes astronomically large for long contexts^[11]. This should also significantly increase the entropy of the resulting unconditional distribution , parametrized by the model.

Although imposing a uniform distribution over tokenizations could be too strict, biasing it towards non-canonicity should increase the entropy and, supposedly, promote larger exploration rates during on-policy RL.

Several questions we find interesting on this topic:

• Can we learn path invariance and what are the limits under which the model starts to collapse and lose capabilities? Formally how much entropy can we introduce to , so that the model preserves and possibly even reinforces its capabilities. Several methods we propose to explore:
  • StochasTok finetuning ( learning symmetry explicitly via Monte Carlo sampling)
  • GFlowNets allows to learn path-invariance by design, though we can bias the resulting distribution towards tokenizations of equal length to the canonical one.
• What happens to the embedding space and latent feature structure of the model due to such an invariance finetuning?
  • How does contextual re-tokenization affect the development of the "inner lexicon" (Kaplan et al., 2026), does it stabilize the residual stream or make it more contextually dependent?
  • What happens to the spectral properties of activations? Most models were shown to have extremely fast decaying singular spectra distributions of activations, especially in the middle layers (Godey et al., 2024; Razzhigaev et al., 2024). How does it correspond to the fact that most latent lexicon and understanding (Skean et al., 2024) seems to happen there as well?
  • Does latent centroid steering benefit from such interventions (locally isotropic feature structures should make centroid representations more stable), or are there weird structures emerging?

2.4 Theoretical or compute-heavy questions

• Do models with different tokenizer merge structures converge to different minima, and which consequences does the choice of the tokenizer optimization objective have?
  Given two models trained on the same data, with the same optimizer and batch sequence converge to different minima? Several studies have manipulated How different and tokenizer-dependent they are? How much inductive biases does the choice of tokens and merges in the tokenizer impose on the model and its alignment susceptibility?
• Can we design a multi-objective tokenization algorithm which will implicitly impose alignment/exploration relevant biases? BPE optimizes for total frequency, WordPiece optimizes for PMI, and both fix the desired number of tokens, leaving some "glitch" tokens undertrained and decreasing the efficiency during training and inference. The well-known SolidGoldMagikarp is an example.
  Could we balance uniform token-splitting with PMI to obtain a pareto-optimal solution and an adaptive tokenizer size? Which implications on scaling laws, misalignment and efficiency will it have?
• Do models necessarily learn invariances over tokenizations or is it an artifact of underfiting, i.e. failing to converge to the canonical distribution despite trillion-token pretraining datasets? For example, transformers without positional encodings (see NoPE (Kazemnejad et al., 2023)) are permutation-invariant architecturally, yet seem to infer positional information from data and perform comparatively well. Unlike positional information, which is essential for language modeling, non-canonicity appears to be a somewhat "arbitrary" artifact, which is intentionally learnt, weakening an underfitting hypothesis.
  We wonder if such capabilities be "disabled" / pruned, and which other capabilities get lost as a side effect?

We will update this list as we make these problems more concrete and identify new open problems.

We develop this project as MARS V fellows, working under the supervision of Dmitry Vaintrob. Thanks to all the MARS team for making this happen and bringing us together.

1. ^{^}

   In some cases, like BPE-dropout, the maps are allowed to be stochastic.

2. ^{^}

   In this post we don't make a distinction between byte and character sequences, unless it's stated otherwise.

3. ^{^}

   This makes it general enough to incorporate multimodal models of any kind, as their inputs are inevitable representable as sequences of bytes as well. This is what byte-latent transformers (BLT) (Pagnoni et al., 2024) seek to exploit. And fortunately, we can restrict it to all the sequences that end in the token to avoid the formalism defining the measure on infinite sequences.

4. ^{^}

   And the number of such re-tokenizations scale exponentially with length.

5. ^{^}

   Some models are exposed to non-canonical sequences during on-policy RL, but the phenomena we describe emerge already in pre-trained models and are not fundamentally modified during RL, modulo entropy statistics.

6. ^{^}

   We will use the term "reader invariance" to refer to this behavioral pattern later.

7. ^{^}

   However, besides re-tokenizations, similar effects can be produced by random context perturbations, and might be attributable to significantly increased entropy of the conditional distributions.

8. ^{^}

   Which appears to align with the morphological structure of the word. Contrary to this, tokenization algorithms (at least the most widely used) are optimizing for compression, not for linguistic compositionality.

9. ^{^}

   And the merge structure is relatively small in comparison to the number of tokens in pretraining data - for example, Qwen3.8-2.4T-A95B (Qwen Team, 2026) tokenizer has 248,320 tokens, while being pretrained on trillions of tokens

10. ^{^}

    Distance between two tokenizations is defined by the minimum number of legal edits (splits and merges that respect the vocabulary) that transform one segmentation into the other; see (Levenshtein, 1966; Zheng et al., 2025; Geh et al., 2025).

11. ^{^}

    Note, that this distribution is a direct generalization of the commonly written distribution and it still factorizes over tokens autoregressively. For a common theoretical model that only assumes canonical tokenizations to be learned, the conditional probability for the canonical , and is otherwise.