One of their justifications was: many tasks from SWE-Bench Verified are broken. A correct solution might not be accepted. They are correct about this. These issues have been documentedby others.
However, as bad as SWE-Bench Verified is, SWE-Bench Pro is much, much worse.
I audited 100 random Swe-Bench pro problems[1]. The full audit results are here.
The most common issue was test leniency. In many cases, the tests barely checked the required functionality at all.
For example, here’s Claude on NodeBB-a91721:
Core Issue Not Tested: The original problem states users should be able to register “using only a valid invitation token, without requiring an email.” Neither test verifies registration without an email - both tests still provide an email during registration.
How did this happen? My guess is: SWE-Bench Pro simply any GitHub commit that modified test cases, regardless of whether those modifications were substantive. In this instance, existing test cases were modified to match the new type signature, but no new test cases were added.
In another instance, the test cases require the agent’s solution to be incorrect.
In flipt, the IsNotOneOf operator was incorrectly implemented to be identical to the IsOneOf operator. This was later fixed.
But SWE-Bench Pro (flipt-cd2f3b0) simply scraped the original, incorrect, test cases.
There’s a critical bug here. Looking at the tests:
- “is not one of”: value “3” is NOT in `[5, 3.14159, 4]`, so `isnotoneof` should return **TRUE**, but test expects `false`
…
An agent correctly implementing the requirements would **FAIL** these tests.
—Claude
The most common issue was “requirements inflation”.
Real world test cases scraped by SWE-Bench Pro will assume implementation details not mentioned in the corresponding issue description. Such tests would be unfair to an agent that produced a correct but alternative implementation.
SWE-Bench Pro addresses this issue by adding a “requirements” section, that includes information about the gold patch’s implementation.
However, these requirements sections frequently go far beyond the details necessary to pass the tests, including implementation details that are not tested at all.
For example, here’s claude’s analysis of tutanota-09c277. While the core functionality is tested, extra implantation details specified in the requirements are not:
So if the time has come to retire SWE-Bench verified, then the time has come. But please, please, do not switch to SWE-Bench Pro.
Yesterday, OpenAI announced that they would be no longer using SWE-Bench Verified, instead recommending SWE-Bench Pro.
One of their justifications was: many tasks from SWE-Bench Verified are broken. A correct solution might not be accepted. They are correct about this. These issues have been documented by others.
However, as bad as SWE-Bench Verified is, SWE-Bench Pro is much, much worse.
I audited 100 random Swe-Bench pro problems[1]. The full audit results are here.
The most common issue was test leniency. In many cases, the tests barely checked the required functionality at all.
For example, here’s Claude on NodeBB-a91721:
How did this happen? My guess is: SWE-Bench Pro simply any GitHub commit that modified test cases, regardless of whether those modifications were substantive. In this instance, existing test cases were modified to match the new type signature, but no new test cases were added.
In another instance, the test cases require the agent’s solution to be incorrect.
In flipt, the IsNotOneOf operator was incorrectly implemented to be identical to the IsOneOf operator. This was later fixed.
But SWE-Bench Pro (flipt-cd2f3b0) simply scraped the original, incorrect, test cases.
—Claude
The most common issue was “requirements inflation”.
Real world test cases scraped by SWE-Bench Pro will assume implementation details not mentioned in the corresponding issue description. Such tests would be unfair to an agent that produced a correct but alternative implementation.
SWE-Bench Pro addresses this issue by adding a “requirements” section, that includes information about the gold patch’s implementation.
However, these requirements sections frequently go far beyond the details necessary to pass the tests, including implementation details that are not tested at all.
For example, here’s claude’s analysis of tutanota-09c277. While the core functionality is tested, extra implantation details specified in the requirements are not:
So if the time has come to retire SWE-Bench verified, then the time has come. But please, please, do not switch to SWE-Bench Pro.
Using a similar methodology to my Terminal-Bench 2 audit. This time, I also searched for cases where the tests were too strict.