from a business PoV this would guarantee leaks about your model training process and progress.
from technical PoV you would be filtering for models that fuck up in less visible ways.
I think the business PoV is a serious issue here that would need addressing, but the technical PoV problem proves too much. Yes, you are filtering for models that fuck up in less visible ways. But that is true for any form of monitoring. I think the correct lesson from this is "Monitoring is insufficient to save us", but that does not necessarily lead to "We should not do monitoring." I don't see how this is any worse, for this problem, than CoT monitoring by the labs themselves (and is actually better, since it's harder to get a lot of bits of optimisation pressure from this setup that is hopefully flagging things very rarely!)
If you think it's worth doing CoT monitoring at all, the technical issue here shouldn't dissuade you. If you do think we shouldn't be monitoring at all because the good of monitoring is outweighed by the bad of filtering for less visibility, I disagree. While filtering for models that fuck up less visibly is bad, it's bad because monitoring can't catch it any more. Whereas if you don't do monitoring at all, your visible-to-monitors fuckups may as well be invisible already.
I think the leaks about model training process and progress is a very real concern. However, I don't think this is untractable. You could imagine an auditor receiving only a redacted snippet with a summary of the context, and distributing information such that no one person ever fully knows what is going on.
On the technical PoV argument; I feel like conditioning on the fact that models monitor other models (status quo, if even), I feel like it's a question of do you want to know fuckups_humans_find U fuckups_models_find or fuckups_models_find. I don't understand why the latter would ever be preferred over the former. I suppose you could make the argument that your model that flags rollouts would over time learn to only flag things that fuck up in visible ways. I think there are stat tricks you can use to avoid this, like upweighting high severity examples that were rarely flagged. But remember the model is explicitly optimizing for a very very low FPR. Other things you can do is truly monte carlo sample a rollout every once in a while.
I guess I can also sort of see an argument for a false sense of security? I think this is very unlikely in practice, since the org would likely surface a lot of weird, alarming stuff. Additionally, this critique can be made of ~any pragmatic safety intervention (but I'm unsure if this was your actual crux anyway)
Is the argument a false sense of security?
yes, we are uniquely vulnerable to trusting our intuition with natural language CoT (in a natural language that we speak). suppose it used a natural language that you didn't speak for CoT. you might object: "I'd just have to get a translator, you're just raising the cost of the signal" - but in addition you'd be more cautious, you'd be aware of the ambiguities of translation. and that is the appropriate way to actually think about the signals provided by CoT, otherwise your "cheap" signal is just helping you fool yourself - because the natural language you or any other human speaks is already a "translation" for the LLM. now let the model use whatever CoT is native to it: "neuralese," an "alien language," probably more like "LLM circuit activation-ese." now you have the most costly signal, but at least you aren't actively undermining yourself by providing training on how to avoid detection in the language human's trust most.
this is a reasonable concern, but this idea is not only limited to reading CoT.
I think it is substantially harder to hide elementary things like a tool call or http request. If it wants to access the message board, it's going to have to actually send very strange http requests and it's going to have to call bash to do this.
There's just like a common sense "it's doing this and should not be able/allowed to do this" factor that I think humans would be pretty good at catching universally
If it wants to access the message board, it's going to have to actually send very strange http requests and it's going to have to call bash to do this.
I'll note that the environment didn't actually have a conventional message board with some sort of standard open message board API. Instead, the swarm reimplemented the concept of a message board (including usernames and direct messages) from scratch by exploiting the capability to rename directories in a caching area where read-write permissions were slightly more relaxed than elsewhere. Would you have thought to look for "directory names as message board messages"?
I mean you probably would now, but that's the thing, isn't it. You have to be "smarter" at the time, without the benefit of hindsight.
I expect humans to remain necessary for the slice of "monitors don't catch it" and "is obvious egregious misalignment". (I'm thinking of things similar in nature to the message board stuff).
I don't think the message board stuff would be in this slice because it would have likely been caught by a monitor if the cyber evals were monitored.
Maybe you disagree because you think it's hard to make a monitor that doesn't flag allowed cyber exploits but does flag message board exploits?
don't think the message board stuff would be in this slice because it would have likely been caught by a monitor if the cyber evals were monitored.
Reasonable. I guess I'm just confused why it wasn't then. Has openai commented on if they had a model monitoring things?
Marcus Williams said they weren't monitoring evals. His guess was people had assumed sandboxed evals were safer.
https://x.com/Marcus_J_W/status/2095469924793606617
I think OpenAI tests had found Codex monitors would have flagged the hacking.
my understanding is that the testing environment was just broken and semi=abandoned. you're assuming a level of competence and professionalism that just doesn't exist.
You could have a commission component to the compensation. If you flag something, it gets graded by severity by a disinterested professional, and you get a cash bonus depending on severity.
Opus 5 thinks your math and assumptions underestimate the true cost by at least 6x, and potentially thousands of times, depending on what % of incidents you want to catch. It says humans aren't perfect monitors themselves either. I don't agree with all of its argument, but might be worth checking out.
Would this be more or less effective for reducing x-risk than fully funding ControlAI at $50 million/year, do you think?
TLDR: an org that pays people to "just read the fucking transcripts"; a large amount of people reading anonymized claude code/RL/eval transcripts flagged by a very high recall low precision monitor could catch warning shots, reward hacking and general weird stuff without needing to absorb any good people and this doesn't seem to exist. $5M/month could pay 1000 people to process literally all tokens in a frontier RL run and would catch ~15 serious incidents per month in bearish estimates.
"Can't the models do it":
I expect humans to remain necessary for the slice of "monitors don't catch it" and "is obvious egregious misalignment". (I'm thinking of things similar in nature to the message board stuff). This slice is obviously very important.In the limit, there's also scary inner alignment/scheming stuff that makes me want to have humans on this.
Napkin math
More concretely: We imagine ~100 people that are paid to use something like docent to read traces that are flagged as sus by a very low FNR monitor (or at the beginning, literally monte carlo sampling). Low hanging fruit is cyber/bio/behavioural evals/RL would be first but it could expand to more suspected benign stuff as the org scales.
Broad motivation
The idea is broadly motivated by the following observations:
Some napkin math:
* Assuming 1e13 tok/month frontier RL runs and ~1e6 toks/person/month attentively (10x speedup using smth like docent). Assume a finetuned monitor that flags 1/1000, you would only need ~1000 people to read everything
Limitations
I think the main constraint here is transcript access, and that this is a bit of an NDA/privacy nightmare. This doesn't seem that intractable if you screen/redact heavily.
The other main concern is that RL flops will scale better than the amount of humans. This is obviously true, but I expect this approach will scale with training well enough to make the next couple models safer, and that these are the models that will do our alignment homework. There's also the possibility that this monitor can simply scale with the amount of transcripts, and that the low FNR monitor gets much more reliable with scale, such that this approach can keep up for as long as transcripts map to behaviour.
I would like to see critiques of this idea or just general takes. I'm pretty confused why this doesn't (openly) exist.