How can we make a nice AI system?

Here's a list of all the techniques I'm aware of.

• Train the AI system to be nice. There are a variety of things we can vary in how we train the AI:
  • Train using model internals OR using outputs.
    • The central internals-based things I’m imagining involve using the internals as a reward signal (e.g., like this). Calling “CoT” “internals” is sometimes reasonable (we might want to do process supervision on the CoT).
  • Vary how similar the distribution we’re training on is to the distribution that we care about.
    • For instance: do online training VS training in a toy domain.
  • Train using an imitation-based objective (SFT) OR an outcome-based objective (RL) OR train on declarative facts / stories (mid-training).
    • Train for good behavior or train against bad behavior. Training for good behavior might include training the AI to produce good looking reasoning, as in deliberative alignment.
      • Obviously, there’s a big question of how we get the labels / reward signal here, which should be studied. Especially if you’re doing untrusted monitoring.
      • We’ll also need to decide whether to use on or off policy data.
    • If we’re training on facts / stories stating that the AI is a nice guy: We can vary what the stories are, and how we instill the persona. For instance, we might add a bunch of irrelevant quirks to the persona, and train for those. We likely want to have the stories explain why the AI takes nice actions.
• We might not directly train the policy, but instead train/prompt monitors, and orchestrate a control system. That is, we might ensemble potentially misaligned AI models into a hopefully mostly good AI system. When ensembling, we’ll likely do some amount of rejection sampling of bad actions, and also “factored cognition” / forcing the AI to solve problems that we don’t think it can sabotage. A large part of the problem here is figuring out how to do a good job of interrogating AI models to see if they’re sabotaging you (i.e., “debate” style techniques).

Please let me know if I've missed any techniques!