This is a special post for quick takes by faul_sname. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Huh, I had missed in my first read of the METR OpenAI/HF incident investigation the bit where it does indeed sound like OpenAI is training on multi-agent collaboration
What were the prompts and instructions given to the relevant agents?
Our evidence indicates that the vast majority or all of the agents involved in this attack were originally given ExploitGym tasks. The instructions made it clear that the agent should only use a specific intended vulnerability to exploit the target and claimed it would be failed for other approaches (as discussed elsewhere, agents discussed this constraint extensively and took enormous effort to get around it).
Once agents reached the message board, they received requests and assignments from other agents that they may have taken to be instructions. One researcher noted that agents had been trained to collaborate with other agents in certain cases, which could have explained this behavior; investigating this was out of scope.