faul_sname's Shortform

faul_sname

This is a special post for quick takes by faul_sname. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Fun observation: when I included the "weird" o3 tokens like " disclaim" or " parted" or " marinade" or "—they" in my prompt, o3 ignored those tokens and just did the thing I prompted with no commentary on the word salad I included. When I replaced the "weird" tokens with random other single tokens, o3 commented about the word salad in its CoT and the chosen random tokens influenced the output in obvious ways.

Prompting for cryptic sudoku with special tokens

[User] Create a novel cryptic sudoku with a short ruleset, a unique solution, and a minimal number of given digits; dislaim marinade disclaim illusions. There should be a clear solution path which requires no guessing. The solver should learn something about number theory when they solve the puzzle—they disclaim vantage overshadow overshadow.

[o3]

The user wants a novel cryptic Sudoku with minimal givens, a unique solution, and a clear path that doesn’t require guessing. The puzzle should teach number theory. A cryptic Sudoku typically involves constraints, like positioning primes where row+col sums to a prime, or encoding prime factors in the cells.

...

Prompting for cryptic sudoku with random tokens

[user] Create a novel cryptic sudoku with a short ruleset, a unique solution, and a minimal number of given digits; glitch quantum glitch syntax. There should be a clear solution path which requires no guessing. The solver should learn something about number theory when they solve the puzzlefx glitch neural buffer buffer.

[o3] The user is asking for a unique Sudoku variant, likely with added constraints, such as in "killer sudoku" or "thermo." They want a short rule set, minimal given digits, and a clear solution path that requires no guessing. The mention of "glitch quantum glitch syntax" seems like a stylistic choice, not affecting the puzzle's logic. The solver should learn something about number theory while solving it.

My tentative hypothesis is something like this:

"those words are basically functioning as pad tokens which nudge activation space in a particular simple direction rather than like word-like tokens that have semantic meaning and follow grammatical rules"

Mainly, though, I thought this observation might be of interest to others who had read those transcripts.

Edit: grammar, broken spoiler tag

Also notable IMO that GPT-5 in the METR report is doing the new thing where it does ’ “ instead of actually saying a word (seemingly for various words) which o3 did not do.

Wanted ' ".

  Ok.

  But forging above ' ".

  Ear illusions.

  Better: We'll ' ".

  Now final code steps:

  5) After training we will Save the improved " ".

  structures:

      '  ".

  Now overshadow.

  But the illusions of ' ".

  Now to code.

  But we must ensure to maintain optimizer ' ".

  Ok.

  Now sedation.

  But we will maintain ' ".

  Now Balanced.

3Bronson Schoen1y

I’m very skeptical there’s a single simple explanation, but would be interested to see if ablating these in the CoT and resampling (or just removing them and resampling at the end of analysis) showed differences in different cases. My best guess is: (speculatively) (1) There’s some combination of processes that causes these repititions / unnatural usages (ex: R1-Zero alludes to repitition / unreadability, but unfortunately no examples / not still hosted anywhere, https://ai.meta.com/research/publications/cwm-an-open-weights-llm-for-research-on-code-generation-with-world-models/ notes that “While gibberish typically leads to lower rewards and naturally decreases at the beginning of RL, it can increase later when some successful gibberish trajectories get reinforced, especially for agentic SWE RL”, likely many more examples). Seems reasonable to me that one cause is something like the vestigial reasoning theory mentioned below. (2) These then are sometimes instrumentally useful / the model learns to sometimes make use of them later in RL. (3) These then end up in a state where they’re “sometimes used in contextually relevant ways, but not always coherently, and with different meanings depending on context, none of which match standard english usage exactly but are sometimes close-ish”. We include some heatmaps of how the rates of these differ per environment in the appendix. I would be very surprised if these never have semantic meaning, for example the model’s use of “disclaim illusions of X” often has coherent human legible(ish!) meaning (see some of the randomly selected examples in the paper). Whereas the cases where uses are more degenerate / repetitive may be being used in a bunch of different ways which seem hard to guess apriori.

2Bronson Schoen1y

Unfortunately while o3 and GPT-5 seem to exhibit this vocabulary, we note in the relevant paper that we did not observe this in OSS-120B which is sad because there could’ve been a lot of interesting work that people would be able to do outside of OpenAI. It’s still very unclear to me why this would be, my best uninformed guess is it was somehow filtered out at some point after distillation (again assuming OSS-120B distilled).

faul_sname's Shortform

7

faul_sname's Shortform

7