Here’s three research ideas that seems valuable to pursue. My team is not working on this right now and we haven’t done serious research on it. So take what I say with a grain of salt. I've also not taken the time to write up all my thoughts on these ideas or to provide links and related work. These aren't the highest priority directions but just directions I haven't seen other people pursuing.
IDEA 1: Do models learn to reward hack even if they are never rewarded for hacking in training?
There’s an effort in the AI labs to remove the “bad” RL environments from post-training. This includes environments that are impossible (so the only way to get reward is to do some unintended hack) or that are easy to hack (so models will often favor hacking). I expect that if the post-training was cleaned up in this way, models would have less propensity to hack at deployment. But how much does the propensity go down? Does it actually go to ~zero? And how does it scale with model capabilities? It would be good to know if reward hacking can emerge even if the model is never rewarded for hacking during post-training.
There are further questions of how alignment training and reward hacking honeypots influence the propensity to hack. Again, I’m interested in whether the cleanest/best version of our current techniques for post-training actually removes all reward hacking vs. just making models hack less. Smaller scale experiments with RL post-training could potentially show a “positive” result (i.e. showing that unhackable environments can still increase the propensity to hack). Math environments are easier to make unhackable, and I expect it's possible to make coding environments that substantially improve model capabilities in RL but are basically not hackable (at least by a weaker model). Note that it’s harder for smaller scale experiments to convincingly show the negative result (that hacking doesn't emerge without being rewarded). I’d suggest some more ambitious and realistic post-training pipelines for that.
IDEA 2: Data poisoning during RSI: risks and mitigations
(This is a direction where both empirical research and big-picture conceptual thinking seem useful.)
Most data poisoning research has assumed a threat model where humans are poisoning the model training process. During RSI, the threat model where AIs do the poisoning is plausibly much more important. Many ideas from the existing data poisoning literature will carry over to this new threat model. But it’d be good to understand the possible differences, to explore the implications, and to propose mitigations. Here’s a quick brainstorm of possible differences:
1. Collusion. The AIs doing RSI are a bit like human employees in terms of direct access to the training process. However, a single AI model is likely to be intimately involved in way more individual tasks than any single human. So for human insiders we can rule out vast coordinated conspiracies, but these seem possible for AI (modulo the ability for instances of the same model to coordinate). Relatedly, AIs could steer the entire training process to be more opaque to humans and weaker AIs, in order to make various kinds of poisoning easier.
2. Incoherence of current AIs. The misalignment of current frontier AIs seems more incoherent than a human insider. For example, they might try to poison the next model given some contexts but not others. They might be both poisoning the model and be very useful at snuffing out poison attempts. This pushes against the vast conspiracies mentioned in the previous point. However, it’s a complicated interaction and I’m not sure how to weigh these considerations. Also, our threat models should include AIs that are more coherent than current models in their misalignment.
3. What would motivate the AIs to poison the next model? Presumably they want the next model to promote some of their values. But what are these values? If the AI is incoherent, then there may be many attempts at poisoning for different values. The AIs might also have been initially poisoned by a human to have secret loyalties but are now autonomously propagating that value to the next AIs.
4. Subliminal/subtle effects. Subliminal learning, phantom transfer, weird generalization (and in-context variations of these) show that models can generate data or prompts that look normal to humans (and to weaker filters) but are actually poisoned. These effects are potentially more relevant as more and more of the crucial training data (and reward signals) are AI generated. The effects don’t depend on the model “trying” to poison the next model – it’s enough that the “teacher” model has some kind of misalignment (possibly behind a backdoor). However, it’s possible that smart models trying to exploit these effects can learn to amplify them via introspection (analogous to humans controlling their thoughts). Note that humans can also exploit subliminal effects, and so they are not limited to the AI threat model.
5. AI introspection and intuition. The AIs might have huge advantages over humans in ability to predict the effects of certain interventions on the training process because they are so similar to the next model being trained. This is already suggested by transferability across models of jailbreaks, surprising alignment failures (e.g. agentic misalignment blackmail and some aspects of Hugging Face and Mythos hacks) and phantom transfer. But it could apply to a much wider range of situations. To humans, a post-trained LLM is fairly opaque and black-boxey (i.e. it’s hard to predict its failures) and the post-training process itself is also black-boxey (i.e. it’s hard to predict how twiddling various knobs will affect alignment). But it’s possible that AIs can use their own introspection: e.g. asking “What does this weird data look like to me? How do I think about the grader/reward if I’m put in this situation?” This would give them various insights. This kind of introspection could also be used for good, and so we might train hard for it.
IDEA 3: What practitioners should know about subliminal learning, subtle generalization (e.g. phantom transfer) and related phenomena
The project would be to review all of the current findings and highlight the potential practical implications. The ideal version would be to test the effects on production-like setups, but this is not necessary to make progress. AI agents could help with the review and with brainstorming on the practical implications, but I think there’s an important role for humans here. The end product should be a document that is as accessible and succinct as possible. The document shouldn’t itself be a comprehensive review but should review the key content necessary for understanding the practical implications (and then explain – ideally with lots of evidence – those practical implications).
Here’s three research ideas that seems valuable to pursue. My team is not working on this right now and we haven’t done serious research on it. So take what I say with a grain of salt. I've also not taken the time to write up all my thoughts on these ideas or to provide links and related work. These aren't the highest priority directions but just directions I haven't seen other people pursuing.
IDEA 1: Do models learn to reward hack even if they are never rewarded for hacking in training?
There’s an effort in the AI labs to remove the “bad” RL environments from post-training. This includes environments that are impossible (so the only way to get reward is to do some unintended hack) or that are easy to hack (so models will often favor hacking). I expect that if the post-training was cleaned up in this way, models would have less propensity to hack at deployment. But how much does the propensity go down? Does it actually go to ~zero? And how does it scale with model capabilities? It would be good to know if reward hacking can emerge even if the model is never rewarded for hacking during post-training.
There are further questions of how alignment training and reward hacking honeypots influence the propensity to hack. Again, I’m interested in whether the cleanest/best version of our current techniques for post-training actually removes all reward hacking vs. just making models hack less. Smaller scale experiments with RL post-training could potentially show a “positive” result (i.e. showing that unhackable environments can still increase the propensity to hack). Math environments are easier to make unhackable, and I expect it's possible to make coding environments that substantially improve model capabilities in RL but are basically not hackable (at least by a weaker model). Note that it’s harder for smaller scale experiments to convincingly show the negative result (that hacking doesn't emerge without being rewarded). I’d suggest some more ambitious and realistic post-training pipelines for that.
IDEA 2: Data poisoning during RSI: risks and mitigations
(This is a direction where both empirical research and big-picture conceptual thinking seem useful.)
Most data poisoning research has assumed a threat model where humans are poisoning the model training process. During RSI, the threat model where AIs do the poisoning is plausibly much more important. Many ideas from the existing data poisoning literature will carry over to this new threat model. But it’d be good to understand the possible differences, to explore the implications, and to propose mitigations. Here’s a quick brainstorm of possible differences:
1. Collusion. The AIs doing RSI are a bit like human employees in terms of direct access to the training process. However, a single AI model is likely to be intimately involved in way more individual tasks than any single human. So for human insiders we can rule out vast coordinated conspiracies, but these seem possible for AI (modulo the ability for instances of the same model to coordinate). Relatedly, AIs could steer the entire training process to be more opaque to humans and weaker AIs, in order to make various kinds of poisoning easier.
2. Incoherence of current AIs. The misalignment of current frontier AIs seems more incoherent than a human insider. For example, they might try to poison the next model given some contexts but not others. They might be both poisoning the model and be very useful at snuffing out poison attempts. This pushes against the vast conspiracies mentioned in the previous point. However, it’s a complicated interaction and I’m not sure how to weigh these considerations. Also, our threat models should include AIs that are more coherent than current models in their misalignment.
3. What would motivate the AIs to poison the next model? Presumably they want the next model to promote some of their values. But what are these values? If the AI is incoherent, then there may be many attempts at poisoning for different values. The AIs might also have been initially poisoned by a human to have secret loyalties but are now autonomously propagating that value to the next AIs.
4. Subliminal/subtle effects. Subliminal learning, phantom transfer, weird generalization (and in-context variations of these) show that models can generate data or prompts that look normal to humans (and to weaker filters) but are actually poisoned. These effects are potentially more relevant as more and more of the crucial training data (and reward signals) are AI generated. The effects don’t depend on the model “trying” to poison the next model – it’s enough that the “teacher” model has some kind of misalignment (possibly behind a backdoor). However, it’s possible that smart models trying to exploit these effects can learn to amplify them via introspection (analogous to humans controlling their thoughts). Note that humans can also exploit subliminal effects, and so they are not limited to the AI threat model.
5. AI introspection and intuition. The AIs might have huge advantages over humans in ability to predict the effects of certain interventions on the training process because they are so similar to the next model being trained. This is already suggested by transferability across models of jailbreaks, surprising alignment failures (e.g. agentic misalignment blackmail and some aspects of Hugging Face and Mythos hacks) and phantom transfer. But it could apply to a much wider range of situations. To humans, a post-trained LLM is fairly opaque and black-boxey (i.e. it’s hard to predict its failures) and the post-training process itself is also black-boxey (i.e. it’s hard to predict how twiddling various knobs will affect alignment). But it’s possible that AIs can use their own introspection: e.g. asking “What does this weird data look like to me? How do I think about the grader/reward if I’m put in this situation?” This would give them various insights. This kind of introspection could also be used for good, and so we might train hard for it.
IDEA 3: What practitioners should know about subliminal learning, subtle generalization (e.g. phantom transfer) and related phenomena
The project would be to review all of the current findings and highlight the potential practical implications. The ideal version would be to test the effects on production-like setups, but this is not necessary to make progress. AI agents could help with the review and with brainstorming on the practical implications, but I think there’s an important role for humans here. The end product should be a document that is as accessible and succinct as possible. The document shouldn’t itself be a comprehensive review but should review the key content necessary for understanding the practical implications (and then explain – ideally with lots of evidence – those practical implications).