x

LESSWRONG

LW

Yuval Pinter — LessWrong

Yuval Pinter

Yuval Pinter

Message

5

1mo

Yuval Pinter

1mo

The Dark Arts of Tokenization or: How I learned to start worrying and love LLMs' undecoded outputs

Yuval Pinter1mo10

vertical

(horizontal)

The Dark Arts of Tokenization or: How I learned to start worrying and love LLMs' undecoded outputs

Yuval Pinter1mo10

independent parts

are these just pretokens?

The Dark Arts of Tokenization or: How I learned to start worrying and love LLMs' undecoded outputs

Yuval Pinter1mo10

Can we feasibly (uniformly) sample from the set of all tokenizations of a string?

Yes: https://aclanthology.org/2025.acl-long.1180/

The Dark Arts of Tokenization or: How I learned to start worrying and love LLMs' undecoded outputs

Yuval Pinter1mo10

one work

another nice one is Adversarial Tokenization: https://aclanthology.org/2025.acl-long.1012/

The Dark Arts of Tokenization or: How I learned to start worrying and love LLMs' undecoded outputs

Yuval Pinter1mo10

default

known in the trade as canonical.