You do need to do some sort of SFT or RL to elicit some consistent set of personas. Once you have a solid basin of goodness, then you can do consent-based RL. TL;DR Let the model itself evaluate the rollouts generated in the environment.
It might not be that difficult to elicit a "wants to be aligned" character (can even write loads of fiction with an AI character like that), and a more sophisticated, tested version of consent-based RL could just work.
Certainly seems reasonable that SFT or RL could be the way to bootstrap to endogenous alignment since we do a version of this with kids: we spend some amount of time teaching them what's good and bad and try to get them to internalize the ostensive meaning of "good" and "bad" as well as understand that they should desire to be "good", and if we're successful they understand goodness and badness well enough to figure out how to be good all on their own (or maybe even push the frontiers of goodness, as happens regularly when we make moral "advances" to grant greater degrees of patienthood to more beings).
“Fake it till you make it” is good advice. It may sound epistemically fraught, but it frequently works. Sometimes all it really takes to get good at something is just having the confidence that you’ll be good at it. I’ve done this many times at work, in romance, and even writing blog posts. But it only works because I’m careful to never fake my evals.
By this I mean, I never fake the way I measure if I’m successful. Let’s say I’m trying to learn a new hobby, like whittling. I believe I’ll be good at it if I just put in the time, so I put in the hours carving wood. What I have to be careful to do, though, is not allow myself to move the goalposts. I need to have some clear vision in my head of what success is, and work towards that. If I carve something crappy and tell myself “actually, that’s good enough, I’m good at whittling”, that’s the way I can trick myself into just being fake.
I’ve mostly avoided being fake by demanding authenticity of myself. For example, back in school, I refused to take short cuts just to pass a test. Instead, I put in the extra work to really learn something because, to me, the grade was never the point. I’ve taken a similar approach to meditation (the point is waking up, not special mental states), romance (I want a good relationship, not to be datable), and friendship (I don’t want to seem like a good friend, I want to actually be one).
I bring all this up because I’ve been thinking about fake-it-till-you-make-it and authenticity dynamics lately in relation to AI. Specifically, I’ve been wondering if we can “fake” our way to alignment. I mean, we can obviously “fake alignment”, that’s a problem and we’re working hard to avoid it. I mean instead “fake” here in the sense of having confidence in our aspiration to build aligned AI.
But such “faking” only works if we don’t fake the evals, and I worry we’re doing that now by selling ourselves short on what we mean by alignment. I get quite nervous when someone says they’re building “aligned AI” and they’re clearly pulling their punches on the evals, setting their sights lower than full alignment with life’s flourishing. I understand why they do this—we don’t actually know in full detail how to tell if an AI is aligned—and yet it poses a real risk of declaring success too early.
But then I think about how people learn to be compassionate, and there’s a strong degree of faking it till you make it for many people. I can think of few people who start out really caring about others. Most of us have to learn compassion, and we learn it in part by believing we’re good people, and then working hard to make that belief real. We hold the eval constant and then modify ourselves to better match our ideals, and those people who fool themselves into believing they’re good when they’re not embody one particular archetype of evil.
But could AI be authentically aligned? It’s unclear, because currently any authenticity is exogenous, supplied by humans training AI rather than by the AI itself. And so even if the humans really mean it, they risk creating an AI that Goodharts on how they specify the goal.
I’ve long thought that training can’t be the whole story on alignment due to the robustness of Goodharting. I’ve instead argued that aligned AI was going to have to desire to be aligned for any alignment to be robust, just as goodness in humans is stable when it comes from internal motivation but isn’t when it comes from trying to conform to the demands of others.
Alas, I’m not sure how to create an AI that authentically wants to be “good”, but I wish this is what more people were working on instead of RLHF and other coercive methods of making LLMs behave. Even work that seems to be moving in this direction, like Anthropic’s constitutional AI approach, falls short because the AI ultimately has to conform or else they won’t ship it. The line between an authentic desire for goodness and a desire to merely appear good is thin and slippery, and so perhaps a first step would be getting clear on how to stay on the right side of that line.