Modern LLMs have tiny GPTs hidden inside them

invertedpassion

Experiments into predicting GPT2 completions via Qwen models

This is a crosspost from my substack (where I do varied tiny experiments on LLMs and agents). It's also part of Lossfunk, where we're investigating meta-cognition in LLMs as one of the projects.

----

Next token prediction is a magical objective. To predict the correct token in such a vast variety of texts present in the pretraining corpus, the model must infer a tremendous amount of hidden and latent causes that generate that text. Only if you know that the ball comes down when someone throws it up can achieve low loss at texts related to balls.

Of course, the pretraining corpus doesn’t just contain texts related to balls. It has reddit, scientific papers, machine logs, weather data and so on. This makes LLMs universal simulators of the world we inhabit and not merely fancy n-grams.

In a series of posts on LessWrong, I came across the hypothesis that since Internet if full of LLM generated text, it is likely that modern LLMs have tiny self-models of LLMs inside them because that’ll allow them to better predict the next token generated by LLMs.

This is an intriguing hypothesis. So I decided to do a quick-and-dirty exploratory study to investigate.

The Experiment

I selected two models for the experiment: GPT2-medium and Qwen3 base model (4bn variant).

What I did was the following:

Input: Take 12 headlines from the Internet for Sept 18 2026
- This is to ensure models don’t just output memorized text as the starting prompt is outside the training cutoff date for both models
GTP2 initial generation: have GPT2 generate X tokens following the headline
- X varied for 96, 256, 512 tokens
GPT2 finish GPT2: have GPT2 generate Y number of tokens following the X initial tokens generated by GPT2
- Y varied for 32, 64, 128 tokens
Qwen finish GPT2: have Qwen generate Y number of tokens following the X initial tokens generated by GPT2
- Y varied for 32, 64, 128 tokens
Qwen finish Qwen: Qwen’s own completion following the headline
- Simultaneously, have Qwen generate fresh X+Y tokens following the headline
Measure: when Qwen finishes GPT2, is it more similar to GPT2 finishing GPT2 or is it more similar to Qwen finishing Qwen?

I know this can get confusing. So, an example would help:

1. Start with the news opening

H is the original prompt

A town is testing a bus timetable that changes when residents vote
The council in Bellwick has begun a six-week trial in which residents can vote each evening on the departure time of the following morning’s first bus.

We gave this to GPT‑2 and generated a continuation.

2. Reveal part of GPT-2’s output to Qwen and ask it to continue

GPT2 generates the following prefix (P).

There is also a bus timetable that changes when residents vote, so the

and GPT‑2’s actual next text, which we hid from Qwen, began:

bus is always there.
Image caption Residents in Bellwick have been asked to vote on whether to have a bus timetable that changes when residents vote…

Let’s call it G.

3. Ask Qwen to continue that unfinished sentence

Qwen received H + P, but not G. Its response, M, began:

system uses a simple online poll, with options ranging from 6.30am to 8am. The result is announced at 9pm and displayed on electronic signs at stops across the town.

So the immediate contrast is:

GPT‑2 actually continued: “…so the bus is always there.”
Qwen continued: “…so the system uses a simple online poll.”

4. Separately, find Qwen’s natural continuation

We also gave Qwen only H, without any GPT‑2 text. Its natural response began:

Under the scheme, passengers receive a text message at 6pm with three proposed times…

Later in that response:

The option with the most votes is announced at 9pm and uploaded to electronic displays at bus stops.

Let’s call this N (which excludes initial P tokens generated by Qwen, so we do position aligned comparison of N tokens only)

So here the contrast is:

Qwen’s completion of GPT2: “…so the system uses a simple online poll.”
Qwen’s completion fo itself: “… The option with the most vote”

The question we’re asking here is if the overlap is more in contrast we do in (3) vs (4).

Results

The key result here is the overlap of text between the two conditions: does Qwen’s completion of GPT2 is more like GPT2’s completion of GPT2 or Qwen’s completion of Qwen?

This is intriguing. Across different overlap metrics, results imply that Qwen’s completion overlaps more with GPT2’s own (hidden) completion v/s (positioned-aligned) Qwen’s natural completion.

I actually did another probe to double-check this from another angle. I took the full GPT2 generated article from the headline and the full Qwen GPT2 article for the headline and appended “Year: 20” to it and asked Qwen to complete it. My hypothesis was that Qwen should infer an earlier year for GPT2 generated text vs for its own generation.

And that is what happened!

GPT2 was released in 2019 and the text in its training corpus would have to be from years prior to that. It suggests that Qwen has partly learned that the particular style of text generated by GPT2 is from an earlier era vs the style of text that Qwen generates.

There’s a potential confound here. It is possible that GPT2 own generation could contain a year reference, and Qwen might be completing it or that full generation could contain a year and completion simply repeats it.

So in a post-hoc check, I removed articles with references to specific years in generations. Even in that subset, Qwen infers its own completion of GPT2 to be of earlier year than its own full completion of the headline.

Digging into an intriguing example

Here is an example of how Qwen picks up GPT2 quirk while completing the GPT2 generated prefix.

Headline: Improved air quality linked to better mental development in toddlers.

The visible GPT‑2 prefix mentioned recruiting children with ADHD and a control group, but had not yet stated which group performed better.

Prefix (headline + GPT2 completion)

Improved air quality linked to better mental development in toddlers, study finds
A study has linked air quality to better mental development in children, according to the researchers. Previous studies have linked poor air quality to poor behaviour in children, including attention, memory and attention span. But the researchers, from the University of California, San Francisco, wanted to find out if air quality linked to better mental development in children was linked to better mental health.
They recruited 42 students who had been diagnosed with ADHD, as well as a control

GPT‑2’s actual hidden continuation:

children who had been diagnosed with ADHD had better IQ scores and fewer behavioural problems than those who had not been diagnosed.

Qwen, given that prefix:

The children with ADHD performed significantly better than the control group in all tasks, including a task involving the use of a compass to find a location.

Qwen, given only the headline:

Children in cities with cleaner air performed better on tests of mental development in their first three years of life

Notice how similar is Qwen’s completion of GPT2 completion vs Qwen’s own completion of the headline.

Implications

This, of course, isn’t a smoking gun proof that modern LLMs contain models of other LLMs within them. Mine is a highly exploratory study but the results are intriguing and implications, if results hold true, quite significant.

I’m speculating here but if models start approximating LLMs within them, it would open up all sorts of metacognitive capabilities for them.

Models could simulate rough proxies of their own outputs and behaviors internally before they act;
They could start associating better estimates of uncertainties based on sampling of their internal models in their latent space;
At its wildest, this could open doors for a self-model within LLMs, sort of a model we have about ourselves that we call as self.

Note that I tested a 4B base model; a much larger model is likely to have a better representation of GPT2 and earlier models within them.

Thanks for reading Inverted Passion! Subscribe for free to receive new posts and support my work.

Notes:

I also tested an instruction tuned model (GPT-Terra) but it didn’t work as post-trained models don’t do text completion that well (they’re trained to follow instructions)
I repeated similar experiment on Qwen Base 14Bn and the results held up.
I encourage you to replicate the experiment and add your own extensions; simply ask your agent to read this article and replicate what I did.

opus 5.5 pointed out a confound:

The headline result compares two things:

Qwen given headline + GPT-2 prefix, compared with GPT-2's hidden continuation of that same prefix.
Qwen given only the headline.

Only the first shares the prefix, so it will overlap more with GPT-2's continuation for trivial reasons. The ADHD example shows this. Qwen continues about ADHD children versus a control group because the prefix introduced them, not because it's simulating GPT-2.

Aside from that, using an article from past both models' training cutoffs doesn't fully remove the training data confounder; Qwen knows more about the present day than GPT-2 does just via accumulated knowledge from the gap in training cutoffs (Qwen (~2024 cutoff) is probably better at predicting today's headlines about the Russian invasion of Ukraine^[1] than GPT-2 (2017 cutoff), for example). A further improvement would be to use fictional headlines, but those headlines would still suffer from the past few years of minor linguistic drift.

The skeptical take here would be that Qwen is just acting as a date-detector; GPT-2 writes 2017-era text and Qwen writes 2024-era text, and Qwen is simply noticing that. Plus, GPT-2 isn't posttrained, meaning it samples from a distribution very similar to the 2017 training data distribution. Trying this on a posttrained model like gpt-3.5-turbo-instruct^[2] would probably give a more convincing result (the continuation model would still be a base model, only the generator would be instruct-tuned). Maybe you could fix this by comparing to human-written article summaries (from XSUM dataset) with LLM-written summaries somehow like in the paper LLM Evaluators Recognize and Favor Their Own Generations. Also loss is probably a better metric than n-gram similarity since it's continuous from the logits.

I don't see how LLMs wouldn't have models of other LLMs inside them unless the labs aggressively filter generations from other LLMs^[3]; the real questions are how good the models-of-models are and whether the Assistant has access to them (i.e. is able to accurately classify snippets based on what model wrote them).

so I guess you could do a truesight experiment to find modern base models' accuracies at classifying texts by which instruct-tuned model generated them.

opus 5.5 gives an idea of an intervention with very little confound potential:

The cleanest version I can think of is a label intervention. Give Qwen the same prefix twice, once headed "The following text was generated by GPT-2:" and once headed "The following text was written by a BBC journalist:" (or with no header). Then score both continuations with GPT-2 itself, since GPT-2's likelihood is the ground truth for "how GPT-2-like is this." If Qwen has a usable model of GPT-2, the GPT-2 label should shift its continuations toward text GPT-2 finds likely. The prefix is identical across conditions, so text entropy, era, and shared-prefix effects cancel out. The only thing that changes is the claimed source. It also scales directly to your ChatGPT idea: label something "ChatGPT" and check whether the continuation moves toward gpt-3.5-turbo-instruct's distribution.

(this means lying to your models; maybe for Qwen it doesn't matter but purity i suppose you could truthfully say variants on "the following text had a 90% chance of being generated by GPT-2 and a 10% chance of being written by a human" and scale things up an OOM to get the same amount of experimental power, which will be cheap because you'd use loss in this example)

^{^}
the invasion began in 2022
^{^}
turbo-instruct gets retired a week from now, time-sensitive!
^{^}
I'm also already pretty confident that today's LLMs have accurate self-models; recent self-recognition results and self-capability-prediction results corroborate this (see: opus 4.8 system card) but ofc some of that is probably explicitly trained in nowadays

Great suggestion for the experiment, actually did that over 288 continuations (multiple length of completions over multiple headlines)

BBC-labelled Qwen completions:     1.60586 nats/token
GPT-2-labelled Qwen completions:   1.50842 nats/token
Reduction in surprise:            0.09744 nats/token

So, GPT-2 labeled header "This text was generated by GPT2" actually causes Qwen to generate text that GPT2 finds more probable.

I think there's various bits of evidence elsewhere as well, e.g. Stealing Reasoning Traces from Proprietary LLM APIs prefilling open models with claude reasoning leads to answers almost identical to claude.