Abstract

Perplexity remains the primary intrinsic metric for evaluating language models, yet direct comparison across models with different tokenization schemes presents methodological challenges. We introduce a tokenizer-normalized perplexity metric that enables consistent comparison of language models regardless of their tokenization approaches. Through empirical analysis of 19 language models across five families on WikiText-2, we demonstrate how tokenization differences can affect traditional perplexity measurements by up to 21.6%. Our normalization method reveals the magnitude of tokenization effects on model comparisons and enables analysis of architectural efficiency patterns. We identify Llama Scout as performing substantially worse than models with far fewer active parameters. The Weighted Perplexity Benchmark provides researchers with a principled approach to evaluate language models while controlling for tokenization strategies.

Authors: Jessica Taylor, Vie McCoy

1. Introduction

As language model architectures, training approaches, and tokenization strategies diversify, comparing their fundamental text prediction capabilities has become increasingly difficult. Perplexity, defined as the exponentiated average negative log-likelihood of a sequence, remains the standard intrinsic evaluation metric. However, different tokenization schemes create systematic differences in perplexity comparisons that complicate cross-model evaluation.

This post addresses the need for tokenizer-independent evaluation metrics through the Weighted Perplexity Benchmark (WPB). Our contributions include:

A mathematical framework for normalizing perplexity scores across different tokenization schemes
Empirical evaluation of 19 language models demonstrating significant tokenization effects
Analysis of how tokenization differences affect model comparisons
Identification of architectural efficiency patterns when tokenization effects are controlled

2. Background and Related Work

2.1 Perplexity and Tokenization Dependencies

For a tokenized sequence $X = (x_{1}, x_{2}, \dots, x_{t})$ and model $M$ , perplexity is defined as:

{PPL}_{M} (X) = exp (- \frac{1}{t} t \sum i = 1 ln p_{M} (x_{i} | x_{< i}))

The critical issue is that $t$ (the number of tokens) varies significantly across tokenizers for identical text. As noted by Mielke et al. (2019), this creates systematic differences in perplexity scores that complicate fair comparison.

2.2 Prior

...

Posts

Wikitag Contributions

Comments