Daniel Tan's Shortform

Daniel Tan

This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Some part of ‘solving the alignment problem’ might be reducible to ‘preserving existing alignment’:

At the moment, it seems that some capabilities training happens after alignment training. E.g. Labs use SFT / DPO to induce alignment, then do RL. Plausible that the proportion of RL will also increase going forward.
More generally, maybe models start off ‘aligned by default’ and misalignment occurs mainly via optimizing against some poorly specified supervision - then maybe constrained optimization is useful

However, it might be nontrivial to preserve this alignment:

Alignment-relevant propensities (reward hacking, sycophancy, ‘being evil’, …) might be modulated by small-but-critical parts of the model
By default, training could result in changes to these propensities, e.g. because of shared circuitry / representations. (c.f. emergent misalignment and related work)

This motivates research on ‘how to add new capabilities while preserving alignment’:

Inoculation prompting does this by reframing ‘misalignment’ as ‘instruction following’.
Gradient routing does this by causing misalignment to be ‘absorbed’ into certain parts of the network, which we can ‘disable’ at deployment
Generally it feels like we want to do some sort of 'constrained optimization' where the constraint is on the model's existing alignment capabilities
Certain techniques from the continual learning literature might also be relevant

---

This is something I'm currently thinking a lot about, welcome takes / comments

FYI, I've been thinking about, and I've noted something similar here.

I'm not really sure what to say about the "why would you think the default starting point is aligned". The thing I wonder about is whether there is a way to reliably gain strong evidence of an increasingly misaligned nature developing through training.

On another note, my understanding is partly informed by this Twitter comment by Eliezer:

Humans doing human psychology will look at somebody lounging listlissly on a sofa and think, "Huh, that person there doesn't seem very ambitious; I... (read more)

2Charlie Steiner1y

All legit, but it's pretty important that "alignment" in these senses is necessarily mediated by things like what options the AI thinks of. So it's kind of a misnomer to talk about "preserving" this alignment as the AIs get to consider more options. Or like, yes, these are properties we would like to preserve across time. But not in a way that implies we should take preserving-type actions. Any more than if I grew up knowing only the 1000 most common English words, and want to learn to correctly use all the words in a physics textbook, it's kind of inapt to say I should just "preserve my ability to use language in the context I'm in."

2Daniel Tan1y

Thanks, agreed directionally! [...] Re: alignment: my mental model is more like “working with a physics professor who is very good at physics but also has some pretty old fashioned views on things like gender roles” -- seems possible to learn the physics without needing to care or update on their other worldviews? [...] You seem to be gesturing at things like AIs figuring out certain actions / traits are instrumentally useful for other things, and adopting them. But here I'm imagining starting with an AI that knows about those options or dilemmas, and would still make choices that humans would approve of. E.g. choosing to allow itself to be shutdown even if it could prevent that. Maybe the cruxy disagreements are: * I think capabilities can be relatively disentangled from the kind of alignment I'm thinking about, whereas this doesn't seem true for you * If you could define a "coherent extrapolated volition" for the persona of current frontier models in the limit of lots of capabilities, I think this CEV would be largely aligned, and you seem to think this would be misaligned.