This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Risk from frontier AI misalignment is dominated by the least-safe frontier models (and by extension the lab which has that model). This has a few important implications.
It's low value to improve safety / alignment at labs where models are already pretty safe / aligned.
Improving alignment at one lab can be net-negative, because it lets that lab push capabilities harder, which then worsens race dynamics, making the (safety-capabilities) tradeoff at the least-safe frontier lab worse.
Edit: I'm interested in feedback on this comment, if you disagree or think it is low quality
Thx! My argument is mainly an argument in principle. This is why I intentionally refrained from naming specific entities. I probably could have made this clearer in the original post.
I am not claiming this argument applies to the current world (I think there is a chance it applies but I feel unsure about the empirical details). I do think it's important to think about, because it could apply to some slice of near-future worlds.
Responding to some points:
"Pretty safe" is relative to the least-safe lab. Even if both OAI and Anthropic are above some defined ... (read more)
I think your comment could be correct in principle, but is applicable only in worlds that aren't this one.
The first point doesn't apply since we don't have any labs where models are already pretty safe / aligned.
The second doesn't either, because the race dynamics appear to be already at maximum and can't go any harder due to engineering and economic limits. We also don't have any evidence that existing labs would - or could - push any harder on capabilities if their models were better aligned.
1Daniel Tan
thx! I think my first point still applies since it’s an argument about relative differences between labs. Even if no labs are particularly safe, if there’s a 100x difference between (least safe) and (most safe) then clearly it makes sense to address the bottleneck directly?
I'm not fully sure how to think about the current state of race dynamics / infra buildout etc. Nonetheless, I think (either now or in the near future) that labs could go much faster on capabilities if e.g. reward hacking was solved.