deepy's Shortform

deepy

This is a special post for quick takes by deepy. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Some AI control and software security relevant ideas that have been floating around in my head or in my research for the past 7 months or so:

AI agent environments should have a well-specified action space such that actions can be constrained, and environment state should be easy to represent and restore, such that "agentic monitors" can get full visibility into the environment state before approving actions. Right now most agent environments are nowhere near this.
AI agent task decomposition is not just good for factored cognition based control protocols, but also because in advance of a task, permissions can be scoped to match precisely what is needed for that task. Trusted models can be used to effectively scale the principle of least privilege to large swarms of agents.
- For background: Factored cognition based control protocols (a) make it harder to come up with and coordinate an attack if multiple subagents and (b) means there are fewer actions that are "on task", so there are fewer relevant actions that can be used to hide malicious effects from a monitor
Recently, LLMs seem capable of doing more bookkeeping work regarding program correctness and safety, so we should be aiming them at programming languages that enforce certain classes of correctness and safety in exchange for this now-cheap bookkeeping. The problem is that Haskell is the only relatively popular programming language that provides very legible guarantees about side-effects, for example. Languages like Austral (which is a well thought-out but abandoned project by a friend of mine) provide excellent guarantees, and with formal verification, programs in a well-specified language can be constrained arbitrarily.
- Side note: Recent LLMs are also much more capable of doing formal verification work, and we should be aiming to do more of this. The bottleneck is human verification of high-level specifications or specifications of desirable properties of programs. Verification can improve program security tremendously, but your proof is only as good as your assumptions. Vulnerabilities have been found in formally-verified software before, because they had weak axioms (e.g. made assumptions about interrupts that weren't true), so automated vulnerability scanning is still useful to us.
In response to the recently reported security incidents during RL training at large labs: AI control strategies used during RL training should be "asynchronous", insofar as the monitor cannot affect the outcome of a trajectory. They can, however, halt training at a point where the model discovers a vulnerability or an impossible problem, before trajectories are produced that might reinforce bad behavior. The response here is not to downweight these trajectories, as this creates much selection pressure to avoid monitor detection and would effectively be training on the CoT. Instead, the response should be to resolve the issue with the RL environment: patching a vulnerability, or removing/correcting the impossible task.
In a typical control protocol, there may be certain kinds of actions that produce many false positives from the monitor. One strategy would be to train more precision in the monitor, but another is to provide a designated time period where those actions are allowed to occur such that they do minimal harm. The only example I have of this is "recon" operations where an agent explores a system or codebase. You could have a protocol where the untrusted agent is given its time to explore the codebase, but then must write out a message describing what it found that the monitor approves or paraphrases, then another agent (probably the same model) is given that message and is not allowed to perform any actions the monitor might deem "recon". This seems like it might have usefulness costs, but it also seems like in this specific case it might inhibit scheming as well. Not that interested in this any more.
Capability systems (see https://en.wikipedia.org/wiki/Capability-based_security) seem appropriate for agentic environments. Monotonically attenuating capabilities in which you may only do one thing or another, but not both, might provide some enforcement of desirable security properties without relying on the monitor to do that.
It would be good to establish in general which models we can trust, and what we can trust them with exactly. I find my current workflows to rely on the models I use quite a bit. I have no deep justification for this being OK, and I don't think this is a good status quo in a world where research sabotage is considered a possibility.
It seems insane to me just how bad computer security is, and how so few people are focused on improving it with the new models we have available. Reasons for this might be:
- Tragedy of the commons since most of this work would not be profitable.
- Resistance to moving away from the status quo (e.g. I don't believe most operating systems are going to survive in their present form, and I have no idea what is going to happen with web browsers)
- Limited access to sufficiently powerful models without safeguards outside OpenAI and Anthropic
It's clear to me that you cannot get aligned models if you rely on RL and don't have airtight security during training.
- You can get a long way with this simply by not having any services that are connected to the internet in your RL environment. In the OpenAI/huggingface incident situation, simply do not have an NPM proxy and instead have a local copy of some relevant subset of NPM pages. This isn't literal "air-gapping" but it removes all of the standout weakest links. It is much harder to escape both the gVisor process and KVM (or whichever hypervisor you use) than to find a bug in a proxy service.
- However, I would still be concerned about bugs in whichever non-internet-connected services you have being exploited for coordination between episodes/agents. However, if you design these services right, it should be easy enough to prevent or mitigate this, since these services are supposed to be read only.
It is adversarial of me to say this, but improving security in the short term may increase x-risk since OpenAI's proposed pause seems only to be until they sufficiently improve security, and not until there is a more robust story around alignment. We are reaching scary levels of capabilities and have had the blessing of multiple relatively benign warning shots. As a first pass, moving all capabilities research talent into interpretability research seems sane in a well-coordinated world.

Some AI control and software security relevant ideas that have been floating around in my head or in my research for the past 7 months or so:

AI agent environments should have a well-specified action space such that actions can be constrained, and environment state should be easy to represent and restore, such that "agentic monitors" can get full visibility into the environment state before approving actions. Right now most agent environments are nowhere near this.
AI agent task decomposition is not just good for factored cognition based control protocols, but also because in advance of a task, permissions can be scoped to match precisely what is needed for that task. Trusted models can be used to effectively scale the principle of least privilege to large swarms of agents.
- For background: Factored cognition based control protocols (a) make it harder to come up with and coordinate an attack if multiple subagents and (b) means there are fewer actions that are "on task", so there are fewer relevant actions that can be used to hide malicious effects from a monitor
Recently, LLMs seem capable of doing more bookkeeping work regarding program correctness and safety, so we should be aiming them at programming languages that enforce certain classes of correctness and safety in exchange for this now-cheap bookkeeping. The problem is that Haskell is the only relatively popular programming language that provides very legible guarantees about side-effects, for example. Languages like Austral (which is a well thought-out but abandoned project by a friend of mine) provide excellent guarantees, and with formal verification, programs in a well-specified language can be constrained arbitrarily.
- Side note: Recent LLMs are also much more capable of doing formal verification work, and we should be aiming to do more of this. The bottleneck is human verification of high-level specifications or specifications of desirable properties of programs. Verification can improve program security tremendously, but your proof is only as good as your assumptions. Vulnerabilities have been found in formally-verified software before, because they had weak axioms (e.g. made assumptions about interrupts that weren't true), so automated vulnerability scanning is still useful to us.
In response to the recently reported security incidents during RL training at large labs: AI control strategies used during RL training should be "asynchronous", insofar as the monitor cannot affect the outcome of a trajectory. They can, however, halt training at a point where the model discovers a vulnerability or an impossible problem, before trajectories are produced that might reinforce bad behavior. The response here is not to downweight these trajectories, as this creates much selection pressure to avoid monitor detection and would effectively be training on the CoT. Instead, the response should be to resolve the issue with the RL environment: patching a vulnerability, or removing/correcting the impossible task.
In a typical control protocol, there may be certain kinds of actions that produce many false positives from the monitor. One strategy would be to train more precision in the monitor, but another is to provide a designated time period where those actions are allowed to occur such that they do minimal harm. The only example I have of this is "recon" operations where an agent explores a system or codebase. You could have a protocol where the untrusted agent is given its time to explore the codebase, but then must write out a message describing what it found that the monitor approves or paraphrases, then another agent (probably the same model) is given that message and is not allowed to perform any actions the monitor might deem "recon". This seems like it might have usefulness costs, but it also seems like in this specific case it might inhibit scheming as well. Not that interested in this any more.
Capability systems (see https://en.wikipedia.org/wiki/Capability-based_security) seem appropriate for agentic environments. Monotonically attenuating capabilities in which you may only do one thing or another, but not both, might provide some enforcement of desirable security properties without relying on the monitor to do that.
It would be good to establish in general which models we can trust, and what we can trust them with exactly. I find my current workflows to rely on the models I use quite a bit. I have no deep justification for this being OK, and I don't think this is a good status quo in a world where research sabotage is considered a possibility.
It seems insane to me just how bad computer security is, and how so few people are focused on improving it with the new models we have available. Reasons for this might be:
- Tragedy of the commons since most of this work would not be profitable.
- Resistance to moving away from the status quo (e.g. I don't believe most operating systems are going to survive in their present form, and I have no idea what is going to happen with web browsers)
- Limited access to sufficiently powerful models without safeguards outside OpenAI and Anthropic
It's clear to me that you cannot get aligned models if you rely on RL and don't have airtight security during training.
- You can get a long way with this simply by not having any services that are connected to the internet in your RL environment. In the OpenAI/huggingface incident situation, simply do not have an NPM proxy and instead have a local copy of some relevant subset of NPM pages. This isn't literal "air-gapping" but it removes all of the standout weakest links. It is much harder to escape both the gVisor process and KVM (or whichever hypervisor you use) than to find a bug in a proxy service.
- However, I would still be concerned about bugs in whichever non-internet-connected services you have being exploited for coordination between episodes/agents. However, if you design these services right, it should be easy enough to prevent or mitigate this, since these services are supposed to be read only.
It is adversarial of me to say this, but improving security in the short term may increase x-risk since OpenAI's proposed pause seems only to be until they sufficiently improve security, and not until there is a more robust story around alignment. We are reaching scary levels of capabilities and have had the blessing of multiple relatively benign warning shots. As a first pass, moving all capabilities research talent into interpretability research seems sane in a well-coordinated world.

deepy's Shortform

2