You can't eval GPT5 anymore
66Ted Sanders
17Lukas Petersson
33Ted Sanders
5Lukas Petersson
4Ted Sanders
1Lukas Petersson
3Lukas Petersson
2Lukas Petersson
21Neel Nanda
10Lukas Petersson
5Matt Goldenberg
3Lukas Petersson
19vjprema
1Lukas Petersson
15VojtaKovarik
New Comment
What would be the best solution for you? A new API parameter that lets you override the date?
7We'll take a serious look at offering this option. We're generally happy to give people more control and transparency; the main worry that might hold us back is death by a thousand papercuts where each extra API option is small by itself but makes the overall experience more confusing for developers when dozens of them pile up. But in this case, might really be worth it. Stay tuned. :)
3Dumb question: why can't you do a find replace on your evals to replace the date with today's date?
We thought about that, but then it's not reproducible if we want to run it for new models later
Is this also because GPT-5 is much more like a "black box software" and a lot less like a "model"? Do the evals run with the assumption that they are running it on a "model" (or something close enough to it), and not a "black box software" that could be doing absolutely anything behind the scenes (including various web searches, addition of hidden context, filtering, even potentially human mechanical turks answering everything)?
Even if you override the date, if its doing hidden web searches in the back, those will be based on todays date on todays internet and will affect its result. It may not solve your problem if this is the case.
I would imagine future "models" will only increasingly move in that direction of a hybrid approach and less like a true foundation model that anyone can do anything on top of, both for functionality, safety and business-model reasons (e.g. Google may not allow their models to remove ads nor reverse engineer their own software).
This is a great point. I admit I have to better understand what each model provider does behind the scenes in the API. Sad if the days of access to the model is gone.
I want to flag that things like this are increasingly going to be a problem even if we solve the narrower "system date mismatch" issue.
- The deployed AI will predictably have access to the internet and probably also information abou what kind of "pre-deployment" testing it went through. The access to the internet will let it reliably figure out what date it is. And this in turn lets the AI figure out that it is, extremely likely, no longer in testing. Or at least gives it a good chance to.
- This will be true for the deployed AI. But the not-yet-deployed AI will know it too, on a quite deep level, because this is all quite easily predictable. So if it doesn't see internet access, it knows it is likely still in testing. If it only has access to weirdly limited or inconsistent "internet", it knows it is still in testing. This will be a problem even if the AI isn't scheming. But if it is scheming, it can even investigate the "internet" it has access to. And if it finds many inconsistencies -- because you tried to make it believe that it is in 2026 when it is in fact 2025, and you don't know how 2026 internet will look like -- it knows it is probably still in testing.
A relevant project idea: I think that to the extent that (some types of) evaluations are going to (in some cases) predictably fail like this, it would be nice to clearly demonstrate that ahead of time. Vince Conitzer had a nice idea to run a sort of competition or challenge on things like this. I am not certain what would be the best way of going about it, or whether it makes sense -- but I was playing around with some ideas in this google doc. If anybody has some ideas, I would definitely be keen on hearing them!
(I was planning to make a separate post about this later, but it feels very relevant to the current post, so I am hastily commenting earlier. Apologies for the ideas being a bit jumbled as a result.)
Curated and popular this week