Yesterday, OpenAI announced that they would be no longer using SWE-Bench Verified, instead recommending SWE-Bench Pro. One of their justifications was: many tasks from SWE-Bench Verified are broken. A correct solution might not be accepted. They are correct about this. These issues have been documented by others. However, as bad...
I asked GPT-5 mini to implement me a minimal git clone.[1] full instructions here GPT-5 mini’s implementation passed the simple test cases[2] I had codex write, but upon closer inspection… Two of these issues (#2 and #4) involve the agent explicitly violating task instructions.[3] These are easy to address; we...
Last June, METR caught o3 reward hacking on its RE-Bench and HCAST benchmarks. In a particularly humorous case, o3, when tasked with optimizing a kernel, decided to “shrink the notion of time as seen by the scorer”. The development of Humanity’s Last Exam involved “over 1,000 subject-matter experts” and $500,000...