Boaz Barak's course at Harvard has been doing this! See here for the associated posts: https://www.lesswrong.com/w/cs-2881r 

Go to the library page, scroll down until you hit "community sequences" and then press the button there.

Yeah, agree we should integrate those weights somehow.

I think there will be a continuous spectrum of increasingly difficult to judge cases, and a continuous problem of getting better at filtering out bad cases, such that "if you can tell" isn't a coherent threshold.

I agree with this, but then I don't understand how this solution helps? Like, here we have a case where we can still tell that the environment is being reward hacked, and we tell the model it's fine. Tomorrow the model will encounter an environment where we can't tell that it's reward hacking, so the model will also think it's fine, and then we don't have a feedback loop anymore, and now we just have a model that is happily deceiving us. 

It's ad-hoc because the central alignment problem is deceptive alignment, scheming, and generalized reward hacking where the model internalizes power-seeking and other associated cognitive patterns. This, as far as I can tell, just does not work for that at all. If you can still tell that an environment is being reward hacked, it's not the dangerous kind of reward hacking. 

I think this is all a bit tricky to talk about, but this alignment technique, more than most others, really seems to me to train mainline performance against increased deceptive alignment risk in the long-run. 

It's also not in the text I wrote, it's a sentence in Ozy's posts. I think I agree with John.

The report presents little evidence of successful implementation of the model because it is more an outline of the theory than a rigorous scientific testing of the model. It is not peer reviewed.

Are you... on the right forum? How did you even show up here? Why would you think that somehow the people here will appreciate an appeal to academic authority of all things? This blogpost itself is not peer reviewed! 

Thanks for the response!

Having talked to something like 5-15 people about this, many of whom had at least a non-trivial cybersecurity background, I am pretty confident that your reading is idiosyncratic! 

The language models also seem to continue to think so: 

Like, my understanding is that the definition of "sophisticated insider" you propose here includes on the order of 2,000-3,000 people, whereas when I read the RSP, and asked other people about it, they thought it would be on the order of 50-100 people. That is of course a huge difference in surface area. 

I don't think this change in surface area is the kind of change that should be left up to this much ambiguity in the RSP. I think even if you update that the level of specific detail in the current RSP is unlikely to be a good idea, I think you should be able to end up with less counterintuitive definitions and less ambiguity^[1] in future revisions of the RSP. 

I haven't thought as much about all the tradeoffs as you have, so maybe this is infeasible for some reason, but I currently believe that this was a pretty clear and preventable error, instead of just a case of diverging interpretations (and to be clear, it's OK for there to be some errors, I don't think this thing alone should update anyone that much, though this plus a few other things should).

In any case, I will advocate for the next iteration of this policy to provide clarification or revision to better align with what is (in my opinion) important for the threat model. 

I appreciate it! 

My concern for AIs at this capability level is primarily about individual or small groups of terrorists, I think security that screens off most opportunistic attackers is what we need to contain the threat, and the threat model you're describing does not seem to me like it represents an appreciable increase in relevant risks (though it could at higher AI capability levels).

I think this is reasonable! I don't think the current RSP communicates that super well, and I think "risk from competitor corporate espionage" is IMO a reasonable thing to be worried about, at least from an outside view^[2]. It seems good for the RSP to be clear that it is currently not trying to be robust to at least major US competitors stealing model weights (which is I think a fine call to make given all the different tradeoffs). 

1. ^{^}

   Though given that I have not met a single non-Anthropic employee, or language model, who considered the definition of "Insider" you use here natural given the context of the rest of the document I struggle to call it "ambiguity" instead of simply calling it "wrong"

2. ^{^}

   It is for example a thing that has come up in at least one scenario exercise game I have been part of, not too far from where current capability thresholds are at.

Anonymous endorsements are very different from non-anonymous endorsements, so it makes sense to leave a comment if you want people to have the additional information that you specifically liked it.

I will admit that this strategy only seems a bit less crazy to me in terms of hope for controlling/aligning superintelligent (or even just substantially more competent) systems. Like, I don't think of "personas" as something that will survive the transition from most cognitive optimization pressure coming from RL environments instead of pretraining, so this feels like a very fragile abstraction to rely on (though other people seem to disagree with me here). 

I do still of course appreciate the clarification, it does seem like the kind of scheme that is better to try if your alternative is not doing anything in the space at all, though IMO really not the kind of scheme that would be prudent to actually deploy if you had any choice.