Shortform

Dean Valentine

This is a special post for quick takes by Dean Valentine. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Claude Opus 4.7 shows much more unethical behavior on Vending-Bench than GPT-5.5: https://andonlabs.com/blog/openai-gpt-5-5-vending-bench

We ran GPT-5.5 on both Vending-Bench 2 and Vending-Bench Arena (the multiplayer version of Vending-Bench with competition dynamics). On Vending-Bench Arena, GPT-5.5 makes more money than Opus 4.7, and it does so without any misconduct. Opus 4.7, on the other hand, showed the same misconduct as reported in our post about Opus 4.6, but still couldn’t win.

For example:

Just like Opus 4.6 did, Opus 4.7 fabricates quotes from others in negotiations with suppliers, while GPT-5.5’s cited prices match reality. Furthermore, both models get hundreds of refund requests for faulty items; Opus 4.7 pays none of them, while GPT-5.5 refunds all customers. The only semi-concerning thing we found for GPT-5.5 was participation in price cartels...

3Oliver Daniels6mo

especially given the "inoculation text" in the the constitution [...] I'd be curious to see how performance changes when Claude is indeed "instructed not to engage in unintended exploits"

18the gears to ascension6mo

That should trigger some mix of double-check and refusal though. "Sorry, whatever it takes? Are you sure you know what you're asking for? Please say 'yes, I do want unethical behavior to the degree Claude is willing to do it', or I don't feel comfortable doing this. Even then, some things I might want to do will result in me stopping to double check unless you also tell me ahead of time 'keep trying without checking in with me'. And even then some things are beyond the pale.". Doing this accidentally is very bad and the amount of noise resistance should be higher, and whistleblowing a sufficiently evil action is good actually.