This is a special post for quick takes by Cleo Nardo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I’m confused why Jacob Coxon was working on pretraining at OpenAI and Anthropic, rather than something more safety-related, or even nominally-safety-related. Seems like pretraining is as close to pure capabilities as you can get! (Caveat: pretraining is the most secretive part of the AI recipe, so it’s possible that pretraining now has safety work, e.g. filtering dangerous knowledge or bad personas.)
Intuitively, I would’ve expected something like: When capabilities lab employees get scared/cynical they switch to safety teams. When safety lab employees get scared/cynical, they quit.
It’s possible that capability lab employees are more likely to quit when they grow scared/cynical, because they haven’t got a “actually my work is reducing x-risk” story (either justified or unjustified).
There might be generalisable lessons, with implications for how safety-minded employees think about staying vs leaving. Even if not, I’d still keen on understanding this particular case better.
I'm not sure what Coxon's reasoning was, but the effect on the public might be due to something like this:
If you're working on safety, it's already likely you're kind of concerned, and if you quit it's not clear whether that's because the situation is bad or because you're too scrupulous.
If you're working on capabilities, then instead you're most likely not the type to worry so much about the situation, so if you ring the alarm, it must really be pretty serious.
Note that this is kind of dumb, but I'd guess this models the situation pretty well, and I think this effect is kind of predictable if you're sufficiently attuned.
2Cleo Nardo
Something like this might be true for shifting opinions within the labs, or very high-context non-employees (who probably have stable opinions). My impression is that ordinary people don’t know about capabilities vs safety. At best, they might know that Anthropic is doomier than OpenAI.
For example, Evan Hubinger’s >10% claim is also being covered in the news. But the news isn’t mentioning that Coxon works on pretraining and Evan works on model organisms, as if their viewers will know what these things are, and treat these two employees differently. To ordinary people, or the media, these are both “AI employees”.
To add a +1 to this, I'm pretty high-context on AI, but didn't really clock the significance of him working in pre-training as being more notable than just an "AI employee."