What is the right way to do a ban on neuralese and/or training against CoTs?

1. Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn't get caught by auditing and/or monitoring. That argument currently depends heavily on CoT monitorability: current models still struggle to do very sophisticated sabotage attempts without discussing the details in their CoT, both due to the sabotage tasks being difficult enough that it's hard for them to do it without TTC, as well as that models are still just not that good at controlling the contents of their CoT.
2. In my view, that is the main reason to care about neuralese and training against CoTs: I think the extent to which those things are bad pretty much entirely flows through the extent to which they make it easier for models doing sophisticated sabotage attempts to hide them from auditing and monitoring that is looking at their CoTs. Furthermore, the arguments in favor of those things mostly say that they won't degrade our auditing and monitoring: maybe process-based oversight (which could include training against the CoT) will make models more monitorable, because they'll be more incentivized to lay out all their cognition in their CoTs; and maybe neuralese won't be a problem for auditing because we'll have good interpretability techniques that let us audit the neuralese—but if those things are true, we should be able to demonstrate that they are true by reference to auditing and monitoring evals.
3. Thus, here is a proposal that I think both proponents and skeptics of CoT monitorability should be able to agree on:
   1. Labs should agree to only[1] internally deploy models that, even with good elicitation[2], still fail to complete sophisticated sabotage tasks[3] without being caught by a monitor, where the monitor is allowed to use whatever interpretability techniques the lab uses in practice.
      1. [1]: Probably not a full ban to start with, maybe initially just a joint statement from all the labs saying they agree you shouldn't do this and/or that the government should take some sort of action to legislate this, or just some related transparency thing.
      2. [2]: I think exactly how to do the elicitation is probably the trickiest question here. To start with, I think you could just train IID on completing the sabotage tasks without triggering the monitor, and right now I think models would still fail. In the long run, though, I think directly training against the monitor is probably too strong. Really, what you want is some elicitation regime that you think captures the extent to which the model would realistically get gradient signal in training for evading monitors (so e.g. to whatever extent you do apply pressure to the CoT in training, you should make sure to apply similar amounts of pressure for elicitation).
      3. [3]: I'm thinking of evals like SHADE-Arena or LinuxBench where there's a) some difficult sabotage side-task that the model is trying to do surreptitiously, b) some main task that the model is supposed to be doing, and c) some monitor (of the same capability level) that's overseeing it trying to verify it's not doing anything suspicious. The idea is that these are run like capability evals, not alignment evals, where we're trying to figure out whether, under good elicitation, the model would be able to fool the monitor, not whether it would choose to do the bad thing by default.
4. Though in some sense this is a capability-based gate, you should always be able to unblock scaling if you get blocked by this just by improving your interpretability and monitoring to make the monitor once again stronger than the policy. Or if the reason that this broke was because you made the CoT less monitorable—e.g. by training against it or by doing some neuralese-y thing—you should always be able to unblock by rolling that back. So in theory I think this should always give you the choice to try to keep monitorability/auditability via CoT or via interp or via whatever else—and it also even blocks CoT scaling and forces you over to interp if indeed CoT monitorability breaks down at some point (and vice versa).