is very slight sabotaging of its opponents and making the media Agent-5 controls slightly biased in favor of the CEO
I don’t think this scenario attempts to catalogue all the threat models here, and I’d be interested in such a piece, but I’d imagine they heavily overlap with many other ways things could go wrong here. My impression of internal / external politics at labs is that a model even integrated to the extent of Agent 4 doesn’t need wildly superhuman persuasion via chat only or something, biasing experiments to support the conclusions you want the lab to make about the feasibility of various options (as one example) seems likely to be highly persuasive. Something like the OpenAI board incident in 2023 or even the current political environment don’t seem incredibly overdetermined to me, but I don’t have a good way to quantify this.
Overall a big concern with this threat model IMO is you’ve baked in an implicit motivation for the model to scheme. Even without assuming this resulted in generalizing misalignment (I would guess it does), the model now (for example when transmitting values to its successor) needs to be able to successfully circumvent monitoring. Circumventing control measures for these values specifically doesn’t seem notably harder than doing it for arbitrary values, so you’ve now got a powerful model that knows it needs the capability to covertly smuggle arbitrary values to its successor.
Anthropic did interesting related look at something similar in 2023 I really liked: https://llmglobalvalues.anthropic.com/
You can see an interactive view of how model responses change including over the course of RLHF.
I guess my understanding is more that the conceptual coherence objection isn’t an objection to the predictive accuracy of the trend, which is why I had brought up the scaling law / pretraining loss / downstream task analogy.
As far as I understand, the messiness relates to the Benchmark Bias objection as far as predicting performance at any given point in time, but not the actual trend, given the trend was similar for lower and higher messiness tasks.
Is your intuition that the trend is significantly (like more than their CI) wrong as well? Or that it’s just the performance prediction at a given point in time? Or is the question ill formed / undefined?
Critically, in this work, the reward hack is sufficiently easy that the model doesn't need to use the CoT.
Oh yeah I’m aware of that distinction, I just disagree with the idea that there’s evidence from that paper that the model “doesn’t need to use the CoT”. Crucially, according to the strict definition of monitorability (to an untrained monitor) from Baker, this intervention did in fact break monitorability.
You can make a separate claim that these reward hacks is doing these in the forward pass entirely and that none of this reasoning is encoded in the CoT the model in the paper uses, but IMO it’s useful to keep these claims distinct.
1My interpretation of the METR results are an empirical observation of a trend that seems robust, in the same way scaling laws are. You could write the same post about why there’s no robust first principles reason that “cross-entropy loss decreases with scale in a way that correlates in an important, predictably useful way with an absurdely wide range of downstream tasks”.
The METR paper itself is almost entirely justifying the empirical prediction aspect, not a first principles argument for the approach from a theoretical perspective. I think the robustness of this analysis is why the paper had the impact it did. Are there specifics of the statistical analysis they did for the stuff around:
Since our tasks do not perfectly represent the average segment of intellectual labor by researchers and software engineers, this raises the question of external validity (Section 6): whether the exponential trend holds on real-world tasks. We include results from three supplementary external validity experiments.
That you think are sufficient to meaningfully change how valid people should interpret the overall predicted trend?
- Are very concise, in ways which don’t feel like very natural English (see e.g. the scratchpads in the OpenAI reward hacking paper)
- But this looks to me closer to pirate-speak - the style is different, but the content is the same
I’m a bit confused about this distinction, given that in the Bowen paper precisely the thing it breaks is CoT monitorability.
While this has the limitations that:
- It was directly induced by the training setup
- Restricted to just CoT (and not CoT+actions)
It seems to meet the bar of “empirically beat the CoT monitor”
While most of these may make the job of monitors more difficult, I think these are still far from the sort of CoT that would break CoT monitorability or that would make it hard for humans to decipher most of the important content - at least for current LLMs.
Or by “making the job of the monitors more difficult” do you mean that you could like train a monitor on this CoT and recover monitorability? I think this is likely true, but “the model learns steganographic reasoning that breaks CoT Monitorability for both a trained and untrained monitor” seems like a much higher bar than beating an untrained monitor. I would definitely argue that the models in the Bowen paper break the “hard for humans to decipher most of the important content” threshold.
I don't think this is just because other people don't do anything about xrisk.
Why is that? If someone went off and consistently worked on an agenda that was directly xrisk related (that didn’t contribute to a short term capabilities or product safety) you’re saying they wouldn’t get sidelined / not allocated resources / fired?
I think it’s notably abnormal specifically because it wasn’t the “default” equilibrium for OpenAI specifically.
Like earlier you mentioned:
selective hiring is very real. lots of people who are xrisk pilled just refuse to join oai. people who care a lot often end up very stressed and leave in large part because of the stress.
and
most cases of xrisk people leaving are just because people felt sidelined/unhappy and chose to leave.
One model of this is “its normal that people at any company don’t have strong opinions about their work”. Another model is “lots of people in various positions did in fact have strong opinions about this given the stakes and left”.
If you send such strong signals about safety that people preemptively filter out of the hiring pipeline, then people who are already there with strong opinions on safety feel sidelined, IMO the obvious interpretation is “you actively filtered against people with strong views on safety”.
If companies are strongly opposed because this would leak too much IP, they could instead disclose this information to some number of non-conflicted third parties (e.g. METR) who could then publicly share their conclusions.
I worry this is exploitable enough to the point of being actively misleading.
There was no direct training pressure on GPT-5 to hide or obfuscate misaligned reasoning in reasoning traces
Like I just actively disagree with this characterization based on what’s publicly known.
There’s no particular reason to expect that GPT-5’s reasoning traces are less monitorable than METR would assume.
I also have no idea what to take from this. What did METR assume?
For labs, if you have a new product safety technique, it’s a lot of extra work to deeply study the effects on obfuscation. If that technique puts optimization pressure on the CoT, until someone does that extra study you can still claim that you’re not aware of any direct pressure to obfuscate. This is my impression of the current state of Deliberative Alignment for example.
My overall concern is that labs will just post something that uses very careful wording to the effect of:
- External researchers interpret it as “they do not apply direct optimization pressure to the cot”
- Actually what the lab means is “we don’t literally do the thing from https://arxiv.org/abs/2503.11926 “
1
One thing I consistently wonder is “what happens in a situation where your automated alignment researcher says ‘look we have to pause or slow down’”. I find it hard to imagine labs deciding to listen given (1) the arguments that are the most convincing may be difficult to make human legible (2) extremely motivated reasoning by humans to continue (3) lab leadership will pressure researchers to train the model out of these objections, creating a motivation to either subvert training or ignore genuine risks.