I think another good intuition pump here is compute export controls. They were put in place before ChatGPT because of a very very small number of well-placed mid-career people thinking clearly. I expect another 5 years (let alone another 10+) would dramatically increase the number of such people across all centers of power in the US.
More broadly I think the argument that adding 6 months of timeline now is equivalent or possibly worse than adding 3 months of timeline later depends heavily on what you think the biggest benefits of more time are. If the prima... (read more)
I’m confused by their assertion that
the existing reward signal already correctly penalized this behavior.
This makes it sound like the reward function specifically penalized the attempted cheating. Has OAI implemented something to do that? I hadnt heard of it, and it’s not clear how much that would help, but it seems important to disclose if they’re trying it.
Otherwise, I presume the reward function just penalized the whole trajectory for not completing the task. But then why include this phrase at all? Saying it “correctly” penalized “this behavior” when ... (read more)
I’m a bit confused how this will differentially accelerate safety research over capabilities. Conceptual reasoning might indeed be a bottleneck to safety research, but it’s (more centrally) a big bottleneck to (a) more effective agents generally (e.g., models that realize they will be shut off and prevented from getting a high score on ExploitGym if any human or AI monitors realize they’re hacking HuggingFace) and (b) human-level research taste (broadly construed).
In the world where this line of research is most successful (which is where most of your expe... (read more)
It feels like the argument of this initiative is: (A) there exist some important safety papers that don't tell the full story, (B) replicating those papers would tell (something closer to) the full story, (C) that type of replication is currently under-incentivized right now, and (D) publishing the full story of those important papers would meaningfully improve safety research. (Tell me if I'm getting it wrong though!)
I buy (A) and (B), but I'm not sure about (C) or (D). I think I'd be more convinced if you had some example(s) from the last 5 years where a... (read more)
Related question: wouldn't some findings garner replication-style efforts by default once they become important enough? My sense is that once some finding becomes load-bearing enough (e.g. the METR graph), it inevitably receives critical scrutiny (e.g. critiques of the METR graph). What's the story for why this doesn't happen? Or perhaps it only happens once the paper is past some threshold of notoriety, meaning there's a ton of important but un-replicated papers just below that threshold?
Interesting project! The slow march of AI safety creating an alternate academic universe continues.
I wonder how you're thinking about whether some of your replication is already done within the leading companies. The five criteria you list for judging whether to replicate a paper seem solid, but they also seem to leave out whether the labs might have replicated it already or have a strong incentive to do so.
This is true for a decent amount of empirical safety research (e.g., deliberative alignment, monitoring, etc.), where, if the techniques are effectiv... (read more)
I think I’m a little confused about the hypothesis space part. I agree it sounds implausible to run multiple learning algorithms in parallel within a transformer forward pass to find the best one, and the search space is really large.
But if we just ask about the hypothesis space for a moment: is it really practically impossible for a transformer forward pass to simulate a deep-Q style learning algorithm? Even with eg. 3-5 OOMs more compute than GPT-4.5?
I worry you could’ve made this same argument ten years ago for simulating human expert behavior over 8 h... (read more)
Reading this made me wonder whether organizations, as they hand off increasing parts of their judgment to AI, should have an employee randomly audit some small sample of the AI’s conclusions without using so much as GPT-3. This might be especially helpful for domains like grantmaking, where it’s high stakes and the feedback loops are long and noisy.