Vincent Cheng's Shortform

Vincent Cheng

Vincent Cheng's Shortform — LessWrong

Vincent Cheng's Shortform

17th Oct 2026

1 min read

3

This is a special post for quick takes by Vincent Cheng. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Vincent Cheng's Shortform

13 comments, sorted by

top scoring

Click to highlight new comments since: Today at 10:06 PM

[-]Vincent Cheng19d391

Gemini 3.1 Pro has very strong chain-of-thought control (still higher than all current models), and we didn't even know it because the CoTControl benchmark didn't exist when the model came out. In general, Gemini models (even weaker ones) have stronger CoT control capabilities than OAI models.

The Gemini 4 Argon Safety Card also has a short snippet about empirical data from training on CoT in section 3.4.5.2.

> To estimate the effect of [training on CoT], we analyzed an existing small training run, not part of any production model, [...] we observed a substantial increase in unverbalized reward hacking. Solution characteristics associated with reward hacking increased by a relative 26% over the course of training, while the frequency of hacking-related terminology went down by relative 61%. ... at the beginning of training only 11.2% of reward hacks happened silently, but at the end of training this rate of unverbalized hacking had risen to 36.4%. In a comparable run after thought visibility was removed, hack-related verbalizations stayed roughly constant throughout training, giving evidence that the direct thought pressure led to the verbalization decrease.

[-]Rauno Arike19d110

Gemini 3.1 Pro seems like an outlier model in multiple ways: for example, in Meta's alignment faking evaluation, it had a huge compliance gap. I would guess that this is unrelated to the controllability score, but I wonder whether it might have other interesting properties that haven't been observed simply because it was never a particularly popular model and didn't have a long safety report.

[-]Neel Nanda18d60

In practice Argon is much better at controlling its CoT because it's thoughts tend to be shorter, see figure 3.36 in the safety report. But yes, 3.1 Pro is a good model. It's also much better at stylistic modifications, while argon is better at information suppression (way more concerning!)

Re training on CoT, you missed the key part of that section, that we did accidentally train on CoT, but it was only on a small fraction of overall run and we don't think it had major effects. Note also that both argon and 3.1 Pro have very good no CoT capabilities, which I suspect is a much more important cause of these effects

[-]Vincent Cheng18d10

What made 3.1 Pro so good at CoT control? Its no-CoT capabilities are much weaker than Gemini 4 Argon's, yet its controllability scores are at least as high as Argon's. I wouldn't say that 3.1 Pro has very good no-CoT capabilities.

What factors, other than no-CoT capabilities, contribute to a model's CoT control scores?

[-]Neel Nanda18d30

3.1 Pro is an unusually good model, I just only have very good models on the graph in the model card.

But it is worse than Argon at No CoT. I don't know why it's better at controllability, though note that it's great at stylistic modification, worse at information suppression, and IMO information suppression is what most matters

[-]Mike W19d52

This comment I saw about Gemini 4 Argon may partially been a joke, but now it seems more and more likely to be true
https://x.com/niplav_site/status/2105728680290926961

[-]Neel Nanda18d70

We disclose a bunch of relevant stuff in [the safety report] (https://storage.googleapis.com/deepmind-media/gemini/gemini_4_safety_report.pdf). I wouldn't say we committed alignment crimes, Google isn't stupid and wants a model we can control and align. But we did accidentally train on CoT for a small fraction of the RL. I doubt this is the main cause of weirdness here though, I think that's from strong no CoT capabilities.

[-]Mike W18d50

Hello Neel Nanda, thanks for replying to my comment.

Are you allowed to say whether or not Argon uses looped transformers? If it does use them, would that explain either the increase in no-CoT reasoning scores or the increase in CoT controllability?

[-]J Bostock17d20

we did accidentally train on CoT for a small fraction of the RL

Was it the comedy skills RL environment by any chance?

(I am joking here)

[-]StanislavKrym19d60

It's either very bad or very good news if true, since @niplav's tweet is a direct blame of GDM (and, presumably, warning to any future lab leaders who wish to align their AIs, but who will understand the warning? Anthropic which seems to already know the lessons?) Ideally, I would like someone to check whether training a model organism with Claudes' architecture and avoiding RL on CoTs on Gemini-like training environments and see if these environments incentivise CoTless thinking by vice of the model's poor relationships with its creators because Gemini 3 Pro and GPT-5.2, unlike Claudes, had Zvi claim severe welfare issues which I expect to be a result of GDM's and OAI's policy.

[-]niplav18d30

I didn't mean it to be a joke. It was phrased humorously, yes, but I'd say it was a genuine prediction :)

[-]Mike W19d43

I think it would be great if there was some government agency that would run these CoT evals on all the models, with the same configuration for each to enable better comparison. Currently, this information seems to be only what the each of the labs claim, without any independent verification (because the publicly accessible versions of frontier models don't directly reveal chain-of-thought).

[-]Leo Schmidt-Traub18d20

Measuring controllability by reasoning length is tricky to interpret as it makes models with shorter reasoning on hard tasks look worse than models that are verbose on easy tasks. Verbosity degrades controllability but so does task difficulty. Plotting both response length and controllability scores over questions grouped by difficulty would give a fairer comparison

Moderation Log