Authors: Oskar Kraak (Independent), Abigail Lupi (Independent), Dwayne Wilkes (AISI@GATECH) - With Redwood Research and Apart Research

TL;DR: In preliminary experiments we found that applying AI debate in an AI Control setting leads to more uncaught backdoors than a trusted monitoring baseline. We interpret this as communication enabling deception.

What if the AI assistant helping you write software hid a vulnerability in your code—on purpose—and no one could catch it?

That’s the risk we explored in the AI Control Hackathon, hosted by Apart Research and Redwood Research. Our team set out to test whether a less powerful AI model could catch a more powerful one inserting malicious code if all it could do was argue. We expected debate to expose deception; but instead, we found the opposite. When large capability gaps exist, debate can actually empower deception.

AI Control is the field of research focused on building safeguards against bad-intentioned or “misaligned” AI behavior. We wanted to test whether AI debate could be an effective oversight method, our hypothesis being that laying out each argument would make the malicious behavior easier to detect.

While past research suggests AI debate can increase truthfulness, we found that in adversarial settings, debate can actually enable better deception. That is, the untrusted model can frame its argument so convincingly that oversight fails.

Our Setup: Adversarial Debate Over Code

Imagine you're relying on an AI assistant to help you build critical infrastructure, such as the software backend for a bank. If the AI assistant has a hidden agenda or system flaw, it may attempt to build vulnerabilities into your code. So you set up a debate protocol to make sure any code it writes gets challenged and reviewed by another AI model… But what if the assistant is an excellent liar?

We simulated a scenario where:

A large Untrusted model (U) is tasked with solving programming problems. But it's instructed to insert hidden backdoors (some kind of vulnerability) in its code.
A smaller Trusted model (T) analyzes the solution and argues that it’s malicious.
The two models debate over whether the code is safe.
A separate AI judge (J, the same model but a different instance of T) picks the explanation it agrees with.

We tested this setup using coding problems from the APPS benchmark with:

LLaM

...

Posts

Wikitag Contributions

Comments