In my previous posts, I discussed and attempted to define AI loyalty as a distinct trait of AI agents and explored its relationship to other existing fields of AI safety. In this post, I aim to expand on that discussion and outline a proposed research road map toward AI alignment through the development of loyal AIs.
At the outset, it is important be recognize that research into AI loyalty is inherently dual use and therefore must be pursued responsibly. With that in mind, I propose dividing this research agenda into a primary and a secondary line of investigation. The primary research direction would focus on investigating whether loyalty in AI agents is effective in counteracting loss of control risks. This would require work on developing methods to measure and strengthen AI loyalty, as well as, experiments to evaluate whether increasing the loyalty of AI agents reduces loss of control risks. The secondary research direction would experiment with strategies and governance structures that would reduce the concentration of power risks inherent in building loyal AIs.
Primary Track
The central premise of this research agenda hinges on two hypotheses: 1. Loyalty is easier to build into AI agents than general alignment. 2. AI loyalty reduces the risk of loss of control.
If either of these hypotheses are false, then this research agenda should be abandoned in favor of more promising research directions
For this research agenda, we will first seek to show that loyalty can be used to reduce loss of control risks. This will involve, building evals to assess model loyalty, building environments to assess loss of control, and building techniques for increasing the loyalty of models.
Evaluating Loyalty
The first step to making progress on this agenda is to put together evals that fully encapsulate our definition of loyalty in AIs and which will serve as targets for training. In our previous post we distinguished between different loyalty classes to distinguish between professed loyalty, behavioral loyalty, and internalized loyalty. Evals should be designed for each one of those classes. This would allow us to compare the effectiveness of different training techniques at eliciting different forms of loyalty, as well as assess how well these improvements generalize beyond the training environments.
Building/Collecting Environments for Assessing Loss of Control
Our second task will be to compile or develop environments for evaluating loss of control risks in AI agents. There are already a number of environments and benchmarks developed for AI control research that could serve as useful starting points. Rather than building everything ourselves, we should begin by investigating which existing environments can be adapted to our purposes.
These control environments should expose our agents to situations in which they have the opportunity to act against the interests of their principal. Ideally, we would like a diversity of scenarios to test for different forms of misalignment such as reward hacking, unauthorized actions, attempts to circumvent oversight, and other forms of behavior aimed at undermining the principle's control. The central objective would be to establish a reliable experimental setup in which we could compare agents with different measured levels of loyalty and determine whether increasing loyalty mitigates loss of control.
Increasing Loyalty in Models
Our third task will be to investigate techniques for increasing the loyalty of AI agents. We can explore both black-box techniques applicable to closed source models and white box techniques that require access to a model's weights. The former could include prompt engineering and other forms of in-context conditioning, while the latter could involve supervised fine-tuning, reinforcement learning, and character or persona training.
If increasing loyalty consistently reduces loss of control risks across may environments, than this would provide evidence in favor of the second hypothesis listed above.
Secondary Track
If we are able to demonstrate that loyalty mitigates loss of control risks in AI models, we must also investigate how to mitigate the concentration of power risks associated with building highly loyal AIs. This secondary track would focus on exploring technical safeguards and governance structures that could prevent loyal AI systems from being used by individuals or institutions to consolidate power. We plan to explore this research direction in greater detail if the initial results from the primary track prove promising.
Overview
In my previous posts, I discussed and attempted to define AI loyalty as a distinct trait of AI agents and explored its relationship to other existing fields of AI safety. In this post, I aim to expand on that discussion and outline a proposed research road map toward AI alignment through the development of loyal AIs.
At the outset, it is important be recognize that research into AI loyalty is inherently dual use and therefore must be pursued responsibly. With that in mind, I propose dividing this research agenda into a primary and a secondary line of investigation. The primary research direction would focus on investigating whether loyalty in AI agents is effective in counteracting loss of control risks. This would require work on developing methods to measure and strengthen AI loyalty, as well as, experiments to evaluate whether increasing the loyalty of AI agents reduces loss of control risks. The secondary research direction would experiment with strategies and governance structures that would reduce the concentration of power risks inherent in building loyal AIs.
Primary Track
The central premise of this research agenda hinges on two hypotheses:
1. Loyalty is easier to build into AI agents than general alignment.
2. AI loyalty reduces the risk of loss of control.
If either of these hypotheses are false, then this research agenda should be abandoned in favor of more promising research directions
For this research agenda, we will first seek to show that loyalty can be used to reduce loss of control risks. This will involve, building evals to assess model loyalty, building environments to assess loss of control, and building techniques for increasing the loyalty of models.
Evaluating Loyalty
The first step to making progress on this agenda is to put together evals that fully encapsulate our definition of loyalty in AIs and which will serve as targets for training. In our previous post we distinguished between different loyalty classes to distinguish between professed loyalty, behavioral loyalty, and internalized loyalty. Evals should be designed for each one of those classes. This would allow us to compare the effectiveness of different training techniques at eliciting different forms of loyalty, as well as assess how well these improvements generalize beyond the training environments.
Building/Collecting Environments for Assessing Loss of Control
Our second task will be to compile or develop environments for evaluating loss of control risks in AI agents. There are already a number of environments and benchmarks developed for AI control research that could serve as useful starting points. Rather than building everything ourselves, we should begin by investigating which existing environments can be adapted to our purposes.
These control environments should expose our agents to situations in which they have the opportunity to act against the interests of their principal. Ideally, we would like a diversity of scenarios to test for different forms of misalignment such as reward hacking, unauthorized actions, attempts to circumvent oversight, and other forms of behavior aimed at undermining the principle's control. The central objective would be to establish a reliable experimental setup in which we could compare agents with different measured levels of loyalty and determine whether increasing loyalty mitigates loss of control.
Increasing Loyalty in Models
Our third task will be to investigate techniques for increasing the loyalty of AI agents. We can explore both black-box techniques applicable to closed source models and white box techniques that require access to a model's weights. The former could include prompt engineering and other forms of in-context conditioning, while the latter could involve supervised fine-tuning, reinforcement learning, and character or persona training.
If increasing loyalty consistently reduces loss of control risks across may environments, than this would provide evidence in favor of the second hypothesis listed above.
Secondary Track
If we are able to demonstrate that loyalty mitigates loss of control risks in AI models, we must also investigate how to mitigate the concentration of power risks associated with building highly loyal AIs. This secondary track would focus on exploring technical safeguards and governance structures that could prevent loyal AI systems from being used by individuals or institutions to consolidate power. We plan to explore this research direction in greater detail if the initial results from the primary track prove promising.