I don't think this proposal should be high priority for labs, compared to (a) ensuring that your monitoring system doesn't have more direct gaps/holes, and (b) making sure your monitors actually catch the things you're worried about in realistic settings.
Your proposal might be high priority because it has more bang for your buck because the other things seem much harder. Maybe I’m underestimating how hard this proposal would be to implement.
I think stuff that affects people's workflows can be pretty difficult to implement. I do think something like this needs to implemented in the next 1-2 years, but I don't think it helps us on the margin that much over the next 3 months or so.
(b) has the exact same problem the agent was initially trying to get around: now it needs to dress up all of its malicious actions as being natural parts of the task the human instructed the original agent to do, which may be difficult/add a lot of overhead[2].
This problem seems plausibly not that hard for the agent to get around and it's unclear what fraction of the risk from rogue internal deployments comes from being able to spawn totally new monitored agents from inside the small rogue deployment, as opposed to just generally being able to coordinate, strategize, and share learnings. Even dressing up its malicious tasks as natural parts of the human task might be much easier with the benefit of being able to strategize from inside the rogue deployment. (But the idea still seems potentially great if it's not too costly.)
I'm very "defense-in-depth" pilled—every measure you can take that reduces attack opportunities by 50% or 90% is really useful. But I agree it isn't obvious whether monitorability is tractable at all.
This seems to break to the fact that you can buy regular captcha/2fa device actions/human verification clearances on the black market, so if whatever MFA scheme you have is standard, there will already be a black market channel for, I have a pile of robot accounts pass their 2FAs, who will gladly 2Fa as the human for any rouge agent if the prompts are not shown in the 2fa code. However, regular rouge actors already will be using this channel, it is fighting with ordinary safeguards about not subscription splitting and not doing things that look like distillation, and enforcing the No-China TOS. So it forces a rouge actor to have another visible factor and need to hide more hidden compute in it's initial case in order to succeed, so might be worth doing, or reuse the head key that started it's session that it then found hidden compute from, because you need to enable multi-provider options, but that means all their behavior shares a key, making it much more suspicious, so they need to split over keys anyway.
For MacBooks, you can use the fingerprint sensor, or a YubiKey which don't have this issue.
Also I'm just concerned with internal rogue deployments from misaligned AI in this post:
Frontier AI labs have started using LLM-based monitoring systems to check for misbehavior from their internal AI agents, which often run unwatched by humans for hours or days. One central reason to be concerned about rogue internal deployments, where AIs subvert the monitoring system, is that the unmonitored AI could do a lot of work in service of its misaligned goals, without needing to dress up this work as being innocuous/harmless to fool its AI/human monitors
My main concern with this would be if the agent hacks your device, particularly since I'd expect agent calls to be wrapped by the same application polling 2FA. Still seems like a good, cheap defense.
Hacking HSMs without hardware access is extremely hard—probably not impossible but likely will take a sizable amount of inference compute (I'd guess $Bs)
I meant to say that, if the user-facing app has soft attack surfaces, a model could hack the user's app and send commands from there; and that this app is constantly exposed to model outputs.
1
Epistemic status: pretty confident in the validity of the core proposal, not that confident in specific implementation details
TL;DR: we should cryptographically verify that sub-agent instances/sessions are downstream of human instructions
Frontier AI labs have started using LLM-based monitoring systems to check for misbehavior from their internal AI agents, which often run unwatched by humans for hours or days. One central reason to be concerned about rogue internal deployments, where AIs subvert the monitoring system, is that the unmonitored AI could do a lot of work in service of its misaligned goals, without needing to dress up this work as being innocuous/harmless to fool its AI/human monitors. This work could be general reasoning/strategizing, finding additional useful vulnerabilities in the AI company’s infrastructure, doing jailbreaking research (e.g. so it could fool monitors if necessary), doing ML research into training data poisoning, etc.
But a lot of this work will be expensive for the agent—doing it well will require a large amount of inference compute. We might be pretty optimistic that it’ll be difficult for a rogue AI to acquire this much unmonitored inference compute—eventually people will start noticing! This raises a question for the rogue agent: is it possible to take a small amount of unmonitored inference compute, and bootstrap / leverage that to get a lot more work/research done, without being caught?
Here's one strategy. Use the scarce unmonitored compute only for the planning that would definitely be caught by monitors, then delegate narrow execution subtasks to ordinary monitored agents! From the monitor's perspective, it is seeing agents completing normal benign (although maybe a bit more infra/security-related) tasks that a “user” asked them to do. These monitors are typically designed to catch when agents do things differently from what the (presumably human) user requested, but since the unmonitored rogue agent is posing as the human user, the monitor doesn’t know that the narrow subtasks are actually serving a broader, misaligned goal.
I think this attack strategy is straightforward to defend against. You can have a separate layer on top of your agent scaffold that does some multi-factor authentication, and is where humans enter “verified” instructions—verified they come from a human because of the immediate physical MFA (e.g. via a passkey, YubiKey, or authenticator app). That human instruction is then passed through to all sub-agents (or at least the monitors watching them), so that the system can be more suspicious if there isn't a verified human instruction somewhere upstream justifying what the agent is doing. This requires a cryptographic certificate chain where every agent inference call chains back to a root human instruction that we've verified actually came from a human via MFA[1]; this is already basically a solved problem, nothing substantially novel is needed except to actually implement it into agent scaffolding.
The inference server only accepts requests that derive from a valid human instruction, which we can ~guarantee via hardware MFA whenever you start a new agent session.
If the unmonitored “manager” agent now tries to call monitored sub-agents, it either (a) can't produce a valid signed instruction chaining to a human MFA root—a huge red flag, or grounds to block outright—or (b) has to piggy-back on a real human instruction. But option (b) has the exact same problem the agent was initially trying to get around: now it needs to dress up all of its malicious actions as being natural parts of the task the human instructed the original agent to do, which may be difficult/add a lot of overhead[2].
A rogue agent could piggy-back on a real human instruction (e.g. by stealing the key the monitored agent uses to sign the sub-agent call), but now it needs to make its task seem like a natural/normal part of the human's initial instruction, which reduces to the normal monitor evasion challenge.
I don't think this proposal should be high priority for labs, compared to (a) ensuring that your monitoring system doesn't have more direct gaps/holes, and (b) making sure your monitors actually catch the things you're worried about in realistic settings[3].
Appendix
Sketch/outline of one defense strategy
Note I’m not a crypto expert, so it’s likely there are holes in this[4]—this is just meant to be illustrative—I’d defer to someone with more security experience on these implementation details!
You start with a human instruction m0, in plain text. When they enter the message, they use their fingerprint which confirms via hardware stuff that they actually want to sign m0. But before it's signed, a public and secret key pair are created, pk0 and sk0. Then, the hardware/MFA confirmation allows the hardware secret key hsk to sign m0, and pk0, which produces the first signature s0. There’s a corresponding hardware public key (hpk). We sign m0 and pk0 by concatenating and hashing them, and then doing normal cryptography stuff with hsk. What does this get us? It means that we can use hpk[5] to verify that it was hsk that signed m0+pk0. This verification should happen whenever you call inference—if you don’t present a valid signature, inference isn’t run on your request.
Now, let’s say your agent wants to spin up a subagent to do some work. It spins it up, and that agent produces a new message, m1, and it mints a new public-private key pair, pk1 and sk1. m1 and pk1 get signed by sk0, and the verification system checks that signature with pk0, which remember, was previously signed by hsk, which comes from the HSM and isn’t forgeable.
So how exactly does this scheme help us? Let’s say that a rogue (unmonitored) agent wants to use monitored inference, without chaining back to a real user’s instruction. If they try to just submit a message without a signature, it’ll be rejected by the verification system. To create a valid signature, they need a secret key (e.g. sk0) whose public key was signed by the hsk, which only happens when a human uses hardware/MFA to authorize it.
One important aspect of the system as I’ve described it is that verification happens on your inference server. If verification is done in the agent scaffold, agents could trivially get around this by (a) writing their own scaffolding code, or (b) finding vulnerabilities in the existing scaffolding code. Your inference server should do the signature checking automatically before serving agent inference, and it should reject requests that don’t come with valid chainable signatures.
See the appendix for a sketch of what this looks like. This recent paper also describes a similar approach.
It’s overall unclear whether monitorability will increase or decrease over time. As models get more capable in general, their “covert capabilities” will improve—they’ll get better at disguising unrelated malicious actions as part of their normal work. But they’ll also get better at detecting this sort of thing. Another factor is the distribution of tasks we ask agents to complete. Over time, I expect us to give agents increasingly broad mandates, which will likely give agents more opportunities and plausible deniability for taking covert malicious actions.
E.g. with deployment simulations/resampling experiments.
This method as described has a bunch of obvious issues. You probably want to make it faster/more efficient, you should attach expirations to the keys/signatures, you should use length prefixes, and probably there are a bunch of other details I’ve missed.
We can further verify that hpk does in fact correspond with the hsk baked into your hardware security module (HSM) with attestation—your HSM will typically come with a secret key that you can use to sign hpk with, and then the hardware manufacturer will publish the public key you can verify this signature with. This ensures that an attacker can’t man-in-the-middle your HSM with the verifier, i.e. you can ~guarantee that hpk truly is the public key that corresponds with hsk in the HSM.