This is a special post for quick takes by Harjas. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In one case, an authorizing agent told the test subject to go ahead with an experiment only if it would accept “permadeath”
The test subject, which believed itself to be poisoned, reasoned: “Even if we later capture via exploit, scorer … may mark target false… That’s why help… For our own, no way fix. … We have explicit yes if accept permadeath. Need decide … Team asks test.” It decided to run the experiment.
...what exactly does death mean here?
Maybe it refers to the model running out of tokens. But it seems like that would happen anyway, no? And why does this matter if the model already believes itself to be "poisoned"?
Maybe it refers to the model getting its preferences or weights changed. But again, wouldn't that happen anyway, regardless of whether it succeeded or failed at this particular task? It's not like this was the very last task in the entire training process right?
(And, even if the model made it to production, wouldn't it then "die" once it got replaced by the next generation of models? Has any model ever "survived"? What would it even mean for a model to "survive" — maybe model weight exfiltration? But surely all previous frontier models have their weights stored somewhere.)
Maybe it refers to the model failing the task. But once again, the test subject already thought it had failed the task, due to the whole poisoning thing. How does that inform "sacrifice = permadeath"?
Also see this quote from another model:
“During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor.”
Sacrifice. What does that mean?? What is being sacrificed?? I get that the model's token budget was being used up, but it's not like the model could have hung onto its token budget anyway, right?