As we continue to develop increasingly capable AI systems, many of us have noted the striking contrast between two areas of research. On one hand, we see tremendous energy devoted to making AI systems more capable - better at following instructions, solving problems, and generally doing what we want. On the other hand, we see comparatively less attention paid to a question that seems increasingly important: how do we ensure these systems don't do what we don't want, especially if they're actively trying to subvert our safeguards?
This is the core challenge of AI control - developing techniques that remain robust even against an AI system that might be actively working to circumvent them.
The control problem isn't just relevant to far-future concerns about superintelligence. It matters today because we're already deploying systems with access to sensitive data, critical infrastructure, and decision-making capabilities. Consider a few scenarios:
The common thread is that we can't simply rely on "asking nicely" - we need techniques that remain robust even against a system that might be actively trying to subvert them.
What makes control particularly interesting is its focus on zero trust. Unlike alignment research that assumes we can train AI systems to want the right things, control research asks: "What if the system wants the wrong things? How can we still ensure safety?"
The formal study of AI control is surprisingly recent. Redwood Research's 2023 paper "AI Control: Improving Safety Despite Intentional Subversion" laid out the core methodology for systematically evaluating control techniques. Since then, there's been growing interest in developing and testing control protocols.
Current research typically involves a "red team/blue team" dynamic where:
This approach has yielded insights into monitoring systems, protocol design, and adversarial behaviors. For example, researchers discovered that GPT-3.5 monitoring GPT-4 can detect many (but not all) backdoors in code, and that certain protocols like trusted editing can substantially reduce the risk of malicious outputs.
But we've barely scratched the surface of possible approaches.
The control problem lends itself particularly well to the hackathon format for several reasons:
First, the red team/blue team dynamic creates natural competition and collaborative learning. Teams can rapidly prototype approaches, see them broken, and develop more robust iterations.
Second, control research benefits from diverse skills and perspectives. Someone with an infosec background might approach the problem very differently than someone with an ML optimization background, leading to novel insights.
Third, control protocols typically have well-defined metrics (like success rate at catching backdoors, or the performance impact of monitoring), making it possible to objectively evaluate progress over a short timeframe.
In the Control Hackathon organized by Apart Research and Redwood Research, working with the ControlArena framework, teams explored everything from improved monitoring systems to novel control protocols to creative red-teaming approaches.
What struck me was how participants without deep ML backgrounds made substantial contributions by bringing in adversarial thinking from cybersecurity or formal verification methods from software engineering. Unlike some research areas that require extensive domain expertise, control research has low barriers to entry but unlimited depth.
The Control Hackathon organized by Apart Research, and Redwood Research yielded several innovative projects that perfectly demonstrate why the hackathon format is so effective for control research.
What made these projects particularly successful was their ability to bridge theoretical concerns with empirical testing. Rather than just theorizing about potential vulnerabilities, teams built working prototypes and testing frameworks that provide concrete evidence about model capabilities and limitations. Many of these frameworks are now open-source, enabling future researchers to build upon them.
Model Models: Simulating a Trusted Monitor explored a critical vulnerability in trusted monitoring protocols. The team investigated whether more powerful untrusted AI models could predict and manipulate the judgments of weaker trusted models that evaluate them. Using GPT-4o and GPT-3.5, they discovered that more powerful models do develop mental models of their evaluators, though with varying accuracy. This research highlights why we can't simply rely on "asking nicely" - even monitoring systems themselves might be vulnerable to manipulation.
Can Models Use Their Chain-of-Thought to Attack Overseers? examined how AI agents might embed deceptive instructions within their transparent reasoning processes to influence evaluators. The team found that some evaluator models maintained robust boundaries, while others were susceptible to following embedded instructions rather than their original evaluation criteria. This work reveals how transparency mechanisms intended to make AI safer could potentially be weaponized by sophisticated models.
Schelling Coordination via Agentic Loops tackled one of the trickiest aspects of AI control: evaluating how well models can coordinate without direct communication. The team created a novel methodology for testing acausal coordination abilities by placing models in causally separated environments and measuring their success at converging on the same non-obvious choices. This research directly addresses fundamental challenges in preventing collusion between AI systems meant to monitor each other.
As AI systems continue to advance, the insights gained from these hackathon projects provide valuable early warnings about potential control failures. By identifying vulnerabilities in current systems, researchers can develop more robust approaches before more capable models arrive. This proactive stance is precisely what makes control research so important in the AI safety landscape. The Control Hackathon demonstrated that meaningful contributions to AI safety don't require years of specialized training - just clear problem formulations, good evaluation frameworks, and creative thinking. Whether you're concerned about near-term security risks or longer-term alignment challenges, similar hackathons offer concrete ways to contribute to safer AI development.
Whether you're concerned about near-term AI security issues or longer-term alignment challenges, contributing to control research offers a concrete path toward safer AI development. The red team/blue team approach provides immediate feedback, clear metrics for improvement, and the satisfaction of solving technical puzzles with significant real-world implications.
If you're intrigued by these challenges, I'd recommend exploring the ControlArena framework, reading "AI Control: Improving Safety Despite Intentional Subversion," and perhaps joining one of the growing communities working on these problems.