Given that the main way new surprising capabilities emerge is by progress on compression, a measure rooted in perplexity could give us a shared benchmark for general intelligence, regardless of the degree of automation of the development process. Extrapolating from recent progress, we should expect further transformative capabilities as we enter the ~0.68-0.7 log-perplexity window. I don’t know what those capabilities will be, and we may not be testing for them today. At the same time, we should also expect further degradation(opens in a new window) in currently dominant forms of alignment auditing and further increases in evaluation and training awareness. I believe frontier model developers should jointly endeavour to establish shared norms for how to evaluate such models and red-team safety cases for them by the time they are first used, including shared transparency standards. I further propose common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone (such as the development of a successor to CoT monitors). That is, I’d like an industry standard that specifies a first limit on development, measured by some key metric we can track across labs, and what concretely we’d need to achieve before going beyond it. I hope that this leads the way to a common framework for setting successive limits and pacing further development.
Unfortunately, there's a huge gap in per-token perplexity between OpenAI and Anthropic, because Anthropic uses many more tokens per word. That leads to lower perplexity for Anthropic models, and any naive comparison of per-token perplexity would make Anthropic's models look significantly closer to any fixed bar for perplexity than OpenAI. That said, they say "common limits, based on perplexity and other measurements" - which doesn't preclude doing this adjustment. And Yo Shavit, Boaz Barak, and Joshua Achiam were all quick to respond that they weren't making a specific proposal and that it was not hard to adjust.
But good faith efforts to reach a common rule don't force the other people to try to explain why the rule needs to be adjusted in order to be fair.
Regardless I think perplexity is very Goodhartable, e.g. surely it's easy to increase perplexity but only trivially hurt capabilities you care about.
And regardless even if you aren't gaming it, limiting pretraining perplexity wouldn't affect post-training — are they suggesting allowing all post-training??
And regardless it's crucial to limit AI R&D rather than just training runs or model capabilities. Targeting capabilities of models you train like this is a bad/incomplete way to do pacing since you’ll keep accumulating algorithmic progress (and e.g. when China steals it all then you’ll probably suddenly end your pacing).
There are way better ways to do pacing; my favorite is compute caps and second-favorite is deployment lags. Capabilities-based (and risk-based) pacing is fundamentally bad and perplexity is a bad way to do capabilities-based pacing.
Jakub Pachocki just posted a new proposal for how to set limits on training future AI systems:
Unfortunately, there's a huge gap in per-token perplexity between OpenAI and Anthropic, because Anthropic uses many more tokens per word. That leads to lower perplexity for Anthropic models, and any naive comparison of per-token perplexity would make Anthropic's models look significantly closer to any fixed bar for perplexity than OpenAI. That said, they say "common limits, based on perplexity and other measurements" - which doesn't preclude doing this adjustment. And Yo Shavit, Boaz Barak, and Joshua Achiam were all quick to respond that they weren't making a specific proposal and that it was not hard to adjust.
But good faith efforts to reach a common rule don't force the other people to try to explain why the rule needs to be adjusted in order to be fair.