Between Ryan Greenblatt’s Current AIs seem pretty misaligned to me and Steven Byrnes’s Four LLM loss functions → four flavors of LLM misalignment, I think my overall view on the state of alignment has shifted somewhat more pessimistically. 

My old view: The models seem pretty aligned right now.

My current view (open to feedback): 

—

‘Aligned behavior’ is a very complicated thing and a pretty narrow target, even when interpreted as ‘roughly instruction-following behavior without any egregious misaligned behaviors.’ There are a ton of ways to be misaligned. Every training signal we’ve tried on LLMs causes some kind of misalignment, by which I mean that there are important divergences between model behavior and developer intentions. [See the table at the top of the Steven Byrnes post.] It seems like reward hacking is way harder to stop than I had expected, and I think I understand why now: making AIs persistent and creative enough to solve hard tasks is in tension with making AIs cautious about how they go about doing tasks. It still feels like LLMs should have learned a lot of common sense from pretraining, and there was some chance that it would be easy for them to at least avoid egregious reward hacks like in the recent OpenAI Hugging Face and UK AISI incidents, but it seems like that boundary is in fact somewhat complicated so AI companies have failed to instill it into their models effectively.

This makes the analogy to humans feel a bit more real. (I’ve written about this before here.) We use a ton of proxies to teach humans how to behave well; ~all of them can be Goodharted; ~everyone succeeds according to some proxies and fails according to others; sometimes (often?) people live bad lives; and it’s especially likely that people live bad lives according to big picture criteria that only have slow or nonexistent feedback loops (aka, that are hard-to-verify) and that conflict with proxies with faster feedback loops. E.g., ‘the net effect of your life on the well-being of factory farmed animals’ has an almost nonexistent feedback loop for most people, while ‘the taste of meat’ has a fast feedback loop. (I might want to make a table like the one at the top of Steven Byrnes’s post, but for human misalignment instead of LLM misalignment. I’d start by looking at Section 9 of my post here (also linked above). I’d love it if someone else did this first!)

In other words, with humans it feels like a bit of a whack-a-mole of different proxies, and after that it’s a toss-up whether we get alignment or not. And I’m starting to think the situation is kind of similar in LLMs. 

There are at least a couple important differences though: 

1) Humans seem to have better alignment generalization properties (for now), e.g. it’s rare to see anything as bad as the OpenAI Huggingface incident from typical humans, it’s not that hard to teach humans that persistence vs non-reward-hacking boundary. This feels like it might be a valuable target for our alignment work imo: make it so that with the same set of proxies, AIs are no more misaligned than humans. (Probably Steven Byrnes has written about this a bunch too, I’d like to look into that more; something about our innate reward functions and how we update our beliefs separately from our desires :).)

2) AIs are more monitorable for now in principle; I think if we read all CoTs, we’re getting much more oversight than human parents get over their kids (who get to think long serial thoughts to themselves and take lots of actions while out of sight).

(Side note: Frankly, I’m highly unsure if I’m acting aligned relative to my most important goals like ‘make the long-term future as good as possible,’ because the feedback loop is nonexistent and the proxies I use for selecting actions (like ‘what I think seems good based on my current AI safety takes’) are highly imperfect.)

—

Even before these posts and incidents, I already thought there could be big problems that arise as AIs become more capable of much longer-horizon tasks, because the capacity to do those tasks inherently requires models to make cognitive moves that could break alignment, like reflecting on how to pursue their goals and abandoning subgoals. CoT seems monitorable enough that I think we’d be able to notice all those problems if we stay close to the current regime; I get a lot more concerned if we switch to recurrent neuralese.

(Written quickly, maybe some things I won’t endorse on reflection)

Edit: I want a shorter handle for my new view, so I don't accidentally revert to the 'things are fine' view just because it's easier to remember in contexts where I don't have much time to think about it. I think I'll call my new view on alignment the "whack-a-mole + toss-up" view, at least in my own head. And if I want to recall some of the most important pieces that are missing from that, a slightly longer handle might be "proxy whack-a-mole + toss-up + good monitoring + long-horizon cognitive moves."