It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization. — Thomas Kwa

Constellation vs MIRI vs Reality

Current AI agents often behave overtly egregiously misaligned. By “overt”, I mean that a human, reading the transcript, would say “The agent is obviously acting in direct opposition to the specification, user intent, and any common-sense understanding of good behaviour.” The reason, it seems, is that such trajectories scored highly during RL.

Firstly, why are these trajectories scored highly? Here's the story:

1. RL currently has poor sample-efficiency, so we need to grade millions of trajectories, so we’re forced to use script graders (RL from Verifiable Reward) or LLM graders (RL from AI Feedback).
2. RL also has poor generalisation (from training environments to deployment environments unseen in training). So we’re forced to synthetically generate thousands of diverse training environments.
3. So overall, RL is very sloppy, without humans generating the environments or the scores.

My impression is that this would’ve been pretty surprising to people three years ago, from both the "Constellation" and "MIRI worldview clusters. (It's very plausible that I've misunderstood what the worldviews were, or how likely those worldviews considered the current situation.)

The Constellation threat models are downstream of Paul/Ajeya. My (possibly mistaken) impression is that:

• They imagined that high-scoring trajectories would at least look good to humans.
• Their primary concern was: Some trajectories might look good to humans, but not actually good, because the actions are too complicated to evaluate, or the AIs have tampered with our measurements.
• But in reality, many high-scoring trajectories would look obviously egregiously misaligned if a human looked at it. Unfortunately, humans don’t have time to look at the trajectories and score them, because we need to grade millions of transcripts.

And if I think back to the old debates about “alignment by default” from the RLHF days — the discussion definitely assumed that we would score highly transcripts that look good to humans. The debate was whether the AI would generalise properly, i.e. cruxes about the inductive biases of NNs (c.f. counting arguments?) or LLM base models (c.f. aligned personas).

I think Alex Mallen makes a similar observation here:

I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". In retrospect, my experience in the last couple years at Redwood (and the Constellation network more broadly) is that the discourse has focused somewhat too much on these more contingent stories for alignment risk and not enough on the basic "Goodharting" argument that misalignment is a convergent result of large-scale outcome-oriented optimization (at multiple levels: the agent, the RL process, and the developers iterating towards seemingly-safe superintelligence). For example, the Constellation view looks somewhat too focused on non-central inductive-bias questions around scheming; I think "the counting argument" was given too central a role when it wasn't a crux for whether superintelligence would take over.

Contra Alex, my reading of the situation is that MIRI made two mistakes that happened to cancel out, while Constellation made only one mistake and so ended up with the wrong conclusion.

Appendix: What could change the situation?

1. Ungamable script grading. Maybe if we put enough effort into constructing the environments and script graders, then the agents won’t find trajectories which score highly but are overly egregiously misaligned (cf. Mechanize). I’m not optimistic about this as a general solution, but it might work for coding and maths. I include "build better sandboxes" in this approach.
2. Higher effort grading. We could hire millions of humans to read trajectories. Or use stronger models (where would we get them?) Or use WBE. Or use a 50 year slow down to run very long RL runs with high quality human grading.
3. Scalable oversight. Maybe we could build better harnesses of the LLM graders (e.g. debate with cross examination plus plus) so they match human performance.
4. 1000-dimensional substructure. My impression of David Africa’s agenda is that we could find a low-dimensional substructure of “personas”. If this substructure contains an aligned persona, then it could be found with O(1000) well-chosen datapoints. And we could put lots of human effort into making sure those datapoints are aligned. Then, during training, we could project down onto the aligned submanifold at each step.
5. Benign goal-guarding. Maybe we could train weak trusted AIs, and then give them a more active role in shaping how RL changes their motivations. The simplest thing would be: the grader’s score is merely advisory, and we ask the AI to choose gradients for each token. Or, the AI has tools to directly modify its weights. Obviously this is scary, and only works if you can first build a trusted AI which is strong enough to do this complicated goal-guarding task.
6. Better sample-efficiency. I think it would be overall worse if AIs had better sample-efficiency and generalisation, because this would likely foom capabilities. But a silver lining is that we could use fewer RL environments and put more effort into grading each trajectory.