OpenAI has published a very interesting blog post on model perplexity, including a call for labs to establish common limits on metrics like perplexity that they will not progress beyond until they've made progress on alignment and monitoring.
https://openai.com/index/perplexity-and-intelligence/
Interesting, this confirms that the base model for Astra 6.2 is the model that solved Navier-Stokes. This is surprising to me given the largely negative reaction to the model.
It's definitely their codebase from 2022 at the latest, since they say they've been using the same one since GPT-3.5. They did have Codex in 2021 so I guess some code was technically LLM-generated.
It's basically the same as this graph from Anthropic actually, showing superexponential gains in pretraining for just coding:

It's still extremely significant if you're focused on RSI, but it matters a lot where the gains are coming from.
If it's just synthetic data massively increasing the volume and quality (I assume you can filter for successful solutions to har... (read more)