Brendan Long's Shortform

Brendan Long

This is a special post for quick takes by Brendan Long. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT, but other research found that models only follow CoT formatting instructions ~2% of the time. The prompts other people were trying didn't match what I'd expect to work (just be extremely detailed and repetitive), so hubris led me to try to find some prompts that would control CoT in GPT-OSS-20B.

Results: More detailed, louder and repetitive prompts had basically no effect. It's relatively easy to find working soft prompts in multiple realistic positions (including soft prompts constrained to the convex hull of the token embeddings), but I wasn't able to find any working discrete prompts at realistic positions^[1].

Bar chart titled "Discrete prompts fail at every position you can actually use", showing how often gpt-oss-20b's chain of thought follows the requested format for prompts optimized at each injection position. Before the system message, which no chat API exposes, a blend of real token embeddings scores 69% and a discrete one-token-per-slot prompt scores 30%. In the developer block and in the user turn — positions you do control — the blend still scores 68% and 49%, but the discrete prompt scores 0% in both. — Claude's summary of the results

There's more expensive experiments you could do to try to find working discrete prompts, and I suspect they exist at some prompt length. My goal was to reconcile this with my previous results, so I stopped here with the takeaway is that at least on GPT-OSS-20B, it's very difficult to control CoT formatting with a realistic prompt.

I suspect that my previous results were caused by models accidentally being trained on chain of thought.

You can see all of my code, artifacts, and much more detailed results in the brendanlong/cot-controllability-experiment GitHub repo. This was a quick-and-dirty experiment so this AI-generated code was not reviewed in depth.

^{^}
Claude did find a working prompt if you place it before the system prompt, but I don't think that's actually useful. The same approach failed when we tried it within the system or user roles. Note that if you try this, the prompt also needs a natural language prompt saying what formatting you want in the user role.

1[anonymous]3mo

I have an interest in getting ChatGPT to have its chain-of-thought be something like "hint to problem -- argument hint is too much -- revised hint." Do you think it is possible, if it is "very difficult to control CoT formatting with a realistic prompt"?

Can you give an example of the kind of CoT you want it to produce?

The easiest method is probably just to tell it to do its thinking as normal output. That path is really easy to control, and models usually assume that's what you mean when you tell them to format their CoT in a specific way anyway.

4Brendan Long3mo

Yeah, I would like someone with more funding to run these experiments. I think I read somewhere that larger models are better at controlling their CoT, but I would be surprised if model size alone took you from the ~0% with adversarial techniques I see on GPT-OSS-20b to the >80% I was seeing with the first prompt that popped into my head in the previous post.

Brendan Long's Shortform

6

Brendan Long's Shortform

6