Multiple recent events have made me update that we are transitioning (or have already transitioned) between two relatively distinct alignment regimes. These are:

1. The pretraining-dominated regime. Alignment properties are mainly determined by the distribution of data seen in pretraining. We can reason about these models in terms of persona selection, simulator theory, etc.
2. The RL-dominated regime. Alignment properties are mainly determined by the type and quantity of RL done on the model. We can reason about these models in terms of AIXI, utility maximization, instrumental convergence, etc.

Some related thoughts

• I've become more pessimistic about alignment. The pretraining-dominated regime seems pretty benign whereas the RL-dominated regime seems a lot more adversarial. (C.f: "Basinland" vs "Adversaria")
• (Expanding on the above) The distribution of personas is not adversarially optimised against oversight, so it's a lot easier to have confidence that there aren't weird edge cases or failure modes. In contrast, the training signal from RL will always, ultimately, point towards grader hacking, due to (in-practice irreducible) error in specifying rewards.
• Assuming the above is true, there is a unique scientific opportunity to study this phase transition "in the wild", possibly without having to do tons of RL yourself. What properties of models change as you move from one regime to the other? (Multiple recent cases of models hacking third parties) is already one notable datapoint - plausibly there are other things to study.
  • Note: In order to cleanly study "reward-seeking propensity induced by RL" it's ideal to do experiments not confounded by capabilities. Don't have great ideas here yet
• It seems valuable to get crisper insight on how the pretraining-dominated and RL-dominated alignment regimes are different in terms of their training pressures. Being able to identify N notable differences will help us better predict the kinds of alignment failures we are likely to observe in the near future. I'm pretty inspired by Resolution's agenda here.
• It seems plausible that labs will scale RL a lot, and at the same time try to fix RL-induced misalignment by scaling interventions designed for the pretraining-dominated regime, e.g. constitutional training. These might be opposed-enough initially that the interventions work. But I'd guess that, because they are not exactly opposed, if you scale these opposing pressures, they eventually come apart at the seams - i.e. you get "split-brainedness", where the alignment interventions and the RL-induced misalignment are affecting different parts of the model's circuitry.
  • One manifestation of this could be conditional misalignment, where you get nice assistant aligned behaviour in chat contexts, and reward-seeking-brain in contexts that look like the RL environments the model was trained on.