TLDR: an org that pays people to "just read the fucking transcripts"; a large amount of people reading anonymized claude code/RL/eval transcripts flagged by a very high recall low precision monitor could catch warning shots, reward hacking and general weird stuff without needing to absorb any good people and this doesn't seem to exist. $5M/month could pay 1000 people to process literally all tokens in a frontier RL run and would catch ~15 serious incidents per month in bearish estimates.
"Can't the models do it": I expect humans to remain necessary for the slice of "monitors don't catch it" and "is obvious egregious misalignment". (I'm thinking of things similar in nature to the message board stuff). This slice is obviously very important.In the limit, there's also scary inner alignment/scheming stuff that makes me want to have humans on this.
Napkin math More concretely: We imagine ~100 people that are paid to use something like docent to read traces that are flagged as sus by a very low FNR monitor (or at the beginning, literally monte carlo sampling). Low hanging fruit is cyber/bio/behavioural evals/RL would be first but it could expand to more suspected benign stuff as the org scales.
Broad motivation The idea is broadly motivated by the following observations:
When agents fuck up, it is extremely visible, someone just has to catch it and elevate it.
Reading the transcripts is an OP solution to this that people in labs broadly don't have time for.
Every major incident so far could have been detected this way
The good people/money ratio in AI safety is about to drastically shift; this idea allows you to use money to get people into safety that doesn't involve having to absorb good people
It's extremely scalable
It doesn't require high salary nor compute expanditure
It's very seeing like a state compatible; you could run an extremely big intern program and just hire the people who actually found stuff
(obvious loopholes are obvious, can use stats to filter.)
I don't trust the monitors to catch everything. The (narrow) slice monitors do not catch is the exact thing that is extremely scary
monitors will learn to circumvent exactly the neural net monitors
something something "even a 14 year old can catch an image of a cat that fools ASI: i predict this to continue being true"
Some napkin math: * Assuming 1e13 tok/month frontier RL runs and ~1e6 toks/person/month attentively (10x speedup using smth like docent). Assume a finetuned monitor that flags 1/1000, you would only need ~1000 people to read everything
This monitor could get bitter lesson'd to hell, because you'd get so much data.
For a team of 1000, assuming a human will only catch 15% of actual cases, and you assume 1 in 10 million transcripts are real, big incidents that's like ~15 incidents caught a month.
This is bearish, many many tricks could be employed to get this higher
Limitations I think the main constraint here is transcript access, and that this is a bit of an NDA/privacy nightmare. This doesn't seem that intractable if you screen/redact heavily.
The other main concern is that RL flops will scale better than the amount of humans. This is obviously true, but I expect this approach will scale with training well enough to make the next couple models safer, and that these are the models that will do our alignment homework. There's also the possibility that this monitor can simply scale with the amount of transcripts, and that the low FNR monitor gets much more reliable with scale, such that this approach can keep up for as long as transcripts map to behaviour.
I would like to see critiques of this idea or just general takes. I'm pretty confused why this doesn't (openly) exist.
TLDR: an org that pays people to "just read the fucking transcripts"; a large amount of people reading anonymized claude code/RL/eval transcripts flagged by a very high recall low precision monitor could catch warning shots, reward hacking and general weird stuff without needing to absorb any good people and this doesn't seem to exist. $5M/month could pay 1000 people to process literally all tokens in a frontier RL run and would catch ~15 serious incidents per month in bearish estimates.
"Can't the models do it":
I expect humans to remain necessary for the slice of "monitors don't catch it" and "is obvious egregious misalignment". (I'm thinking of things similar in nature to the message board stuff). This slice is obviously very important.In the limit, there's also scary inner alignment/scheming stuff that makes me want to have humans on this.
Napkin math
More concretely: We imagine ~100 people that are paid to use something like docent to read traces that are flagged as sus by a very low FNR monitor (or at the beginning, literally monte carlo sampling). Low hanging fruit is cyber/bio/behavioural evals/RL would be first but it could expand to more suspected benign stuff as the org scales.
Broad motivation
The idea is broadly motivated by the following observations:
Some napkin math:
* Assuming 1e13 tok/month frontier RL runs and ~1e6 toks/person/month attentively (10x speedup using smth like docent). Assume a finetuned monitor that flags 1/1000, you would only need ~1000 people to read everything
Limitations
I think the main constraint here is transcript access, and that this is a bit of an NDA/privacy nightmare. This doesn't seem that intractable if you screen/redact heavily.
The other main concern is that RL flops will scale better than the amount of humans. This is obviously true, but I expect this approach will scale with training well enough to make the next couple models safer, and that these are the models that will do our alignment homework. There's also the possibility that this monitor can simply scale with the amount of transcripts, and that the low FNR monitor gets much more reliable with scale, such that this approach can keep up for as long as transcripts map to behaviour.
I would like to see critiques of this idea or just general takes. I'm pretty confused why this doesn't (openly) exist.