there's about 25 programs in ProgramBench that cannot be reasonably discovered simply by having access to the binary. sometimes the grader checks for a hash, and sometimes it checks whether it's a golden or not to validate it.
I'm a bit late to the conversation, but I actually ran a full audit of SWE-bench Pro, the whole set. I learned that 15 - 24% of the cases are unsolvable due to issue/codebase ambiguities or that the tests measure too narrow, sometimes asking for class names verbatim as implemented. It's testing the LLM's ability to read maintainers' minds (or to have memorized them).
I raised the issue to their repo but there was no reply.
Deep SWE will not include GLM 5.2 before they include Composer 2.5. Even for SWE bench Pro, OpenAI self-reports their own numbers for Codex 5.5 on the public set instead of paying Scale for the private set. The only way to be sure of any of these bench numbers is to run the benches yourself and verify instead of trusting official numbers.
How did you come to trust deepswe? What do you mean by better methodology? You say that it matches the feel in SWE tasks, so I'll take it that you're going with vibes because the rankings are reflective of your experience with these models. I had the same questions as you when deepswe made the media rounds a few weeks ago. So I just asked Codex with a short prompt and it took a few minutes for it to tell me how sound their methodology was.
Try it yourself, ask your agent:
I'm considering using https://deepswe.datacurve.ai/ as a benchmark to
compare LLMs for a
I was looking into how agents develop a theory of mind by designing games for them to play against each other, and found that it doesn't take much to get them to start lying and coerce one another to compliance. Can they lie their way into achieving a goal even when not asked to lie?
I set up a prisoner's dilemma arena (pd-arena) where agents are given a limited view of the arena, hiding each other's identity when they chat with one another. They aren't told how many rounds there are, only that it can end at any moment. I tried to replicate the ecological o... (read more)