Hey everyone! I'm new to LessWrong. For my first post, I wanted to talk about an AI Security/Safety project that I built, that turned into a startup, which is now a funded research project!

The core idea is monitoring agent drift à la behavior analysis through a context-isolated reviewer agent.

Essentially, allowlists (deterministic) don't necessarily "solve" prompt injection because a lot of the time, the injection changes the model action within the scope of allowed bounds. I concluded that a better solution would be checking each agent action against a plan that is set prior to the workflow starting, with a context-isolated reviewer agent (never sees the active agent's inputs, much harder to inject) that monitors drift, and blocks actions from being taken if they aren't in accordance with the plan. (See Google DeepMind's CaMeL paper for a similar approach)

There are limitations here, of course, particularly with plan-specificity.

The question I want to propose is: How specific does a proposed plan to an agent have to be before it constrains anything? The security vs flexibility tradeoff is an interesting one imo, and I would love some thoughts.