If labs are going to hill-climb on benchmarks no matter what we do, one benchmark I'd love to see them hill-climb on is something like "how astonished would a careful reviewer who had read your description of the changeset be when looking at the actual code".

Anyone have any ideas for clever ways to build a benchmark like this that incentivize honesty in summarizing what a changeset actually does without incentivizing being ever-sneakier about hiding bad behavior? I feel like there's got to be something in the region of ideaspace that brought us inoculation prompting, but I haven't come up with anything particularly good yet. Ideally I want

1. Incentivizes the model to do their best to disclose any surprises upfront
2. Disincentivizes producing a readme that is longer than the changeset itself
3. Disincentivizes adding problems to the changeset for the purpose of disclosing them
4. Disincentivizes obfuscating problems so that the reviewer isn't unpleasantly surprised on review by reason of not seeing the problem

This shortform brought to you by a prompt "write a wrapper for chrome --remote-debugging-pipe which exposes the same http interface as chrome --remote-debugging-port except requiring jwt auth" which was reported as "here I'm done" but which was actually "I will run chrome --remote-debugging-port on a random port, and then write a proxy which listens for traffic on a different port, makes sure it's authed, and reroutes it to the random port".

Also countless other examples, mostly not as egregious but very often things like adding "if there is an error, silently swallow the exception and try to continue" in a place where we would want failures to be loud and early if they are going to happen.