Documenting a failed experiment

My question: could you upgrade a base model by giving it more time to think? Concretely, could you finetune a base model (pretrain only) to make effective use of filler tokens during inference. I looked around and found a few papers, but in all cases they either

Trained a model from scratch in some fancy new way
Compared performance according to a series of benchmarks, but not perplexity.

I really wanted to see if you could get the perplexity of a model to improve, not by training it from scratch, but by doing a small amount of finetuning to teach a model to make effective use of more tokens.

Methods

I didn’t want the hassle of managing my own training pipeline, or a custom cloud setup, so I decided to use Tinker from Thinking Machines. They host a whole bunch of open source models, and allow you to finetune them (LoRA) over the API.

The (base) models available:

Model	Total Parameters	Active Parameters^[1]	Architecture	Distilled?
Llama-3.2-1B	1.23B	1.23B	Dense	Yes (from 8B/70B)^[2]
Llama-3.2-3B	3.21B	3.21B	Dense	Yes (from 8B/70B)
Llama-3.1-8B	8B	8B	Dense	No
Llama-3.1-70B	70B	70B	Dense	No
Qwen3-8B-Base	8B	8B	Dense	No
Qwen3-30B-A3B-Base	30.5B	3.3B	MoE	No
DeepSeek-V3.1-Base	671B	37B	MoE	No

My plan was to “upgrade” these models by expanding the number of tokens during testing by including “filler tokens”, which would just be a copy of whatever the last token was.^[3]I would then measure the perplexity only on the “real” tokens, masking all of the filler tokens (which would be trivial to predict anyways).

So

[The] + [ quick] + [ brown] + ?

Would become:

[The] + (The) + [ quick] + ( quick) + [ brown] + ( brown) + ?

By scaling up the number of forward passes (but giving no new information), I hoped to show that the models could learn to make effective use of this extra inference time.

Out of the box, expanding data like this made the models perform way worse (not too surprising, it’s very OOD), so I decided to try finetuning.

Finetuning

The plan: Finetune the models to not get freaked out by the filler tokens, and make effective use of the extra time.

In order to eliminate confounders, I needed my finetuning data to be as similar to the original pretraining corpus as possible (only expanded). For this I decided to use SlimPajama, which Claude told me was pretty close. To be extra sure I would:

Finetune a model on data from SlimPajama (no filler tokens)
Finetune a model on an expanded version of

...

Joseph Eisner

Joseph Eisner

Joseph Eisner

Joseph Eisner