AI Control is a subfield of AI Safety research focused on ensuring the safety of powerful AI systems, particularly under the assumption that they might be misaligned or could become intentionally subversive (A Brief Explanation of AI Control).
Unlike AI Alignment, which seeks to build inherently aligned systems, AI Control develops mechanisms to monitor, evaluate, constrain, verify, and manage the behavior of potentially untrustworthy AI to prevent catastrophic outcomes (The case for ensuring that powerful AIs are controlled).
This review outlines key AI control methods discussed in recent literature. For a broader exploration of related topics, refer to An overview of areas of control work.
1. Monitoring
Description: Monitoring encompasses techniques for observing AI system behavior, outputs, and sometimes internal states. The primary goal is to detect potentially harmful actions or intentions, particularly those the AI might try to conceal, enabling timely intervention or review.
Trusted Monitoring (and editing)
- Description: This control method employs a less powerful but trusted entity, such as a smaller AI model or human oversight, to review and potentially edit or block the outputs or actions of a more powerful, untrusted AI. The goal is to catch and mitigate harmful behavior before it occurs, often flagging suspicious items for further review.
- Related Papers:
- Strengths: Can provide a baseline level of safety by having a supposedly aligned entity check actions. Trusted entities are assumed not to be intentionally subversive.
- Weaknesses: The effectiveness is limited by the capabilities of the trusted monitor; a sufficiently advanced untrusted model might be able to deceive it. May lead to a drop in usefulness if the trusted model is too restrictive. Requires identifying and maintaining trust in a model.
Untrusted Monitoring
- Description: Using other instances of the powerful, untrusted model itself (or an ensemble) for monitoring. This requires techniques to prevent these monitors from colluding with the agent they are supposed to oversee.
- Related Papers:
- Strengths: Can potentially leverage the capabilities of advanced models for monitoring. Ensembling diverse untrusted monitors might increase robustness.
- Weaknesses: Requires effective collusion busting techniques, which can be challenging. Paraphrasing might be ineffective against steganography in semantic spaces. Accuracy can degrade when malicious intent is hidden in seemingly benign subtasks.
Human Auditing
- Description: Involving human reviewers to examine AI actions, outputs, and reasoning for signs of harmful behavior or misalignment, often triggered by automated monitoring. This can range from reviewing flagged actions to establishing more complex recursive oversight schemes where humans oversee AIs assisting them in the review process.
- Related Papers:
- Strengths: Humans can bring contextual understanding, common sense, and the ability to identify novel failure modes that automated systems might miss. Recursive oversight can potentially scale human review capabilities.
- Weaknesses: Human review is a bottleneck due to scalability limitations and can be slow and expensive. The effectiveness depends on the training and vigilance of the reviewers, and the process itself can be red-teamed by a sophisticated AI.
Guardrails
- Description: Systems implementing programmable rules, checks, and filters around a primary AI model at inference time to enforce safety, security, or behavioral policies. Guardrails typically monitor and filter user inputs and/or model outputs to prevent harmful content generation (e.g., hate speech, toxicity).
- Related Papers:
- Strengths: Guardrails offer a practical and configurable method for enforcing diverse safety policies, such as content filtering and topical restrictions, externally without altering the core AI model.
- Weaknesses: However, guardrails are often brittle and can be bypassed by adversarial prompts (jailbreaking), highlighting the difficulty in defining truly comprehensive and robust rules. Additionally, they can introduce latency and sometimes become overly restrictive, potentially hindering the AI's utility or causing false positives.
Reward Hacking Monitoring (incl. Chain-of-Thought Monitoring)
- Description: Focuses on detecting when an AI optimizes a proxy reward signal in a way that deviates significantly from the intended true goal (reward hacking). A key technique involves using another LLM (potentially weaker but trusted) to monitor the primary AI's Chain-of-Thought (CoT) reasoning for signs of reward-seeking behavior that misaligns with the underlying task objective. This monitoring of the reasoning process can be more effective than solely observing actions or final outputs.
- Related Papers:
- Strengths: Monitoring CoT reasoning can be significantly more effective at detecting reward hacking than monitoring actions/outputs alone.
- Weaknesses: Optimizing the AI directly against such CoT monitors can lead to obfuscated reward hacking, where the AI learns to hide its intent within the reasoning traces, potentially reducing monitorability (a "monitorability tax").
Activation-Based Safety Interventions (incl. Circuit Breakers)
- Description: Techniques that monitor the internal activations of an AI model during inference to detect patterns associated with potential harms (e.g., generating toxic content, deceptive reasoning). If such patterns are detected, interventions are applied directly to the activations to steer the model towards a safe output, such as refusing the request or generating a harmless alternative.
- Related Papers:
- Strengths: Operates directly on model internals during inference, allowing for real-time intervention before harmful output is fully generated.
- Weaknesses: Reliably identifying the specific activations/circuits corresponding to diverse harmful states across different contexts is a major challenge (robustness issue).
Monitoring LLM Agents for Sequentially Contextual Harm
- Description: Specifically monitoring agentic AI behavior over time to detect harm arising not from individual actions but from a sequence of seemingly benign steps that collectively lead to a harmful outcome. This requires monitoring the context and trajectory of agent behavior.
- Related Papers:
- Strengths: Addresses a critical vulnerability in agentic AI where harmful goals are achieved through decomposed, seemingly safe subtasks.
- Weaknesses: Existing off-the-shelf LLMs have limitations as monitors for this type of harm; their accuracy can be degraded by irrelevant benign subtasks. Specialized training for monitors is needed.
2. Control Evaluations
Description: Control evaluations are methodologies for testing whether a set of safety measures ("control protocols") can prevent an AI from intentionally causing unacceptable outcomes, assuming the AI might actively try to do so. These evaluations often involve a "red team" AI or human trying to subvert the "blue team's" safety protocols.
Subversion Strategy Evaluation
- Description: Specifically focused on measuring an AI's ability to independently devise and execute strategies to subvert control protocols without relying on shared memory between interactions (statelessly). This evaluates the AI's planning, reliability, probability calibration, and acausal coordination skills in an adversarial context.
- Related Papers:
- Strengths: Directly probes the AI's strategic reasoning abilities for the purpose of undermining safety measures, a critical aspect of assessing risks from advanced AI.
- Weaknesses: Current evaluations suggest models do not yet exhibit substantial capabilities in stateless strategic subversion, though affordances can improve performance. The complexity of real-world strategic planning might not be fully captured.
AI Control Levels (ACLs)
- Description: A framework defining different levels of AI capabilities and the corresponding control evaluation methodologies, control measures, and safety case requirements appropriate for each level. This provides a roadmap for adapting safety protocols as AI advances.
- Related Papers:
- Strengths: Offers a structured, forward-looking approach to AI safety by linking control strategies to evolving AI capabilities. Helps in systematically planning for future safety needs.
- Weaknesses: Requires accurate forecasting of AI capabilities and defining effective control evaluations/measures for each level, which is challenging.
Control Evaluations in Multi-Step Agent Environments
- Description: Extending control evaluation methodologies to settings where AI agents perform sequences of actions to achieve a goal, allowing study of more complex attack/defense dynamics.
- Related Papers:
- Strengths: More closely reflects real-world applications where AI agents operate over extended periods. Uncovers new challenges/opportunities for attack and defense.
- Weaknesses: Designing and evaluating protocols in such complex environments is significantly more challenging than in single-step settings. Monitoring is harder.
Resample Protocols Evaluation
- Description: Specifically evaluating control protocols involving multiple samples of AI actions, particularly in multi-step agent settings, measuring the trade-off between attack prevention and usefulness.
- Related Papers:
- Strengths: Demonstrates significant improvements over existing techniques in agent environments by exploiting model statelessness to exert selection pressure against malicious actions and gather incriminating evidence.
- Weaknesses: Introduces additional computational cost. Effectiveness relies on model statelessness.
3. Guaranteed Safe AI
Description: Guaranteed Safe AI (GS AI) aims to produce AI systems with high-assurance quantitative safety guarantees, often relying on formal methods. Security measures and sandboxing are crucial components that contribute to achieving safety guarantees by enforcing constraints on the AI's actions and access.
- Description: Mathematically defining acceptable AI behavior and unacceptable states of the world. This involves creating precise rules or policies.
- Related Papers:
- Strengths: Provides a precise and auditable definition of safety. Enables formal verification.
- Weaknesses: Formulating precise and comprehensive safety specifications, especially capturing nuanced human values, is extremely difficult.
- Description: Creating auditable proofs that the AI system's behavior or outputs (like code) satisfy the safety specification relative to a world model or predefined properties. This uses formal verification techniques. Proof-Carrying Code is a specific application where code completions from an LLM are formally verified by a trusted entity.
- Related Papers:
- Strengths: Offers the potential for high-assurance, quantifiable guarantees about safety, going beyond empirical testing.
- Weaknesses: Formal verification at the scale and complexity of modern AI systems is a significant research challenge. Requires AI components amenable to formal analysis and well-defined specifications. The necessary world models can be hard to create accurately.
Sandboxing
- Description: Limiting access and privileges (permission management), isolating AI processes (sandboxing), and restricting remote access.
- Related Papers:
- Strengths: Reduces the affordances of AIs, making it harder to cause catastrophic outcomes. Protects against unauthorized actions or malicious exploitation. Leverages established security principles.
- Weaknesses: Standard security might be insufficient against advanced AI threats capable of novel exploits or manipulation. Implementing fine-grained control can be complex.
AI Control Safety Cases
- Description: An AI Control Safety Case is a structured argument intended to provide assurance that an AI system, deployed under specific control measures, is safe even assuming the AI might be misaligned or intentionally subversive.
- Related Papers:
- Strengths: Provides a structured framework for reasoning about and justifying the safety of deployed AI systems under control. Can help guide the development and selection of appropriate control measures based on required evidence.
- Weaknesses: The strength of the case heavily depends on the quality, realism, and comprehensiveness of the underlying Control Evaluations. May face fundamental limitations in providing high assurance against unforeseen capabilities or strategies, especially from superintelligent systems.
Incident Reporting, Escalation, and Shutdown Processes
- Description: Establishing clear procedures for identifying, reporting, and responding to potential safety incidents involving AI systems, including mechanisms for escalating concerns and safely shutting down or restricting a misbehaving AI.
- Related Papers:
- Strengths: Provides a crucial safety net to mitigate harm when other preventative measures fail. A well-defined process can ensure timely and appropriate responses.
- Weaknesses: The process must be robust to intentional subversion by AI agents. Requires clear criteria for triggering escalation/shutdown and effective communication channels.
4. Capability Control
Description: Capability control aims to limit the abilities of AI systems, particularly those that could be misused for harmful purposes, while preserving usefulness for intended tasks.
Mitigating Steganography
- Description: Employing methods to make it harder for AIs to use steganography (hidden communication) within their outputs, parameters, or reasoning processes. This controls the capability of secret communication, hindering collusion or data exfiltration. Techniques include paraphrasing outputs and specific detection/regularization methods.
- Related Papers:
- Strengths: Limits the ability of misaligned AIs to coordinate or conceal harmful plans.
- Weaknesses: Paraphrasing might not work against semantic steganography. Detecting subtle steganography, especially in reasoning, remains challenging.
- Description: Limiting the data, resources, and external systems an AI can access, and controlling communication channels between different AIs or AI components. Information siloing hinders coordination for harmful purposes.
- Related Papers:
- Strengths: Reduces the AI's ability to gather information needed for harmful actions and to coordinate with other agents.
- Weaknesses: May hinder the AI's ability to perform its intended tasks effectively if access is too restricted. Determining the right level of restrictions is challenging.
Precise Capability Control (incl. Unlearning, Gradient Routing)
- Description: Techniques aiming to selectively remove, suppress, or prevent the learning of specific unwanted capabilities (e.g., harmful knowledge, dangerous skills) while preserving general performance. Methods include Gradient Routing, which modifies gradient flow during training to localize computation and potentially prevent the model from acquiring or integrating certain abilities, and Unlearning, which seeks to remove the influence of harmful knowledge.
- Related Papers:
- Strengths: These methods offer the potential for fine-grained control over specific AI capabilities, allowing for targeted removal or suppression of unwanted knowledge or skills learned during training or fine-tuning.
- Weaknesses: A major challenge is selectively removing or controlling capabilities without significantly harming the model's general performance and usefulness (often referred to as the alignment tax or performance trade-off). Ensuring complete and robust removal (unlearning) or effective prevention (gradient routing) without unintended side effects or the model finding alternative internal pathways remains technically difficult.
Conclusion
The field of AI Control provides a critical layer of defense for managing risks associated with advanced AI systems, particularly when perfect alignment cannot be guaranteed.
This review highlights several key approaches: comprehensive Monitoring to detect suspicious behavior, rigorous Control Evaluations to test defenses, methodologies aiming for Guaranteed Safety through formal methods and security, and various forms of Capability Control to limit potential harm.
While promising, each category faces significant challenges in terms of scalability, robustness against sophisticated adversaries, and practical implementation. Continued research of these control methods within frameworks like safety cases will be essential for the future AI deployments.