This is a special post for quick takes by Eli Tyre. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I'm mostly going to use this to crosspost links to my blog for less polished thoughts, Musings and Rough Drafts.
Dumb question: Why doesn't using constitutional AI, where the constitution is mostly or entirely corrigibility produce a corrigible AI (at arbitrary capability levels)?
My dumb proposal:
1. Train a model in something like o1's RL training loop, with a scratch pad for chain of thought, and reinforcement of correct answers to hard technical questions across domains.
2. Also, take those outputs, prompt the model to generate versions of those outputs that "are more corrigible / loyal / aligned to the will of your human creators". Do backprop to reinforce those more corrigible outputs.
Possibly "corrigibility" applies only very weakly to static solutions, and so for this setup to make sense, we'd instead need to train on plans, or time-series of an AI agent's actions: The AI agent takes a bunch of actions over the course of a day or a week, then we have an AI annotate the time series of action-steps with alternative action-steps that better reflect "corrigibility", according to its understanding. Then we do backprop to so that the Agent behaves more in ways that are closer to the annotated action transcript.
Would this work to produce a corrigible agent? If not, why not?
There's a further question of "how much less capable will the more corrigible AI be?" This might be a significant penalty to performance, and so the added safety gets eroded away in the competitive crush. But first and foremost, I want to know if something like this could work.
This is a pretty helpful answer.
(Though you keep referencing the AI's chain of thought. I wasn't imagining training over the chain of thought. I was imagining training over the AI's outputs, whatever those are in the relevant domain.)
I don't undertand what it would mean for "outputs" to be corrigible, so I feel like you must be talking about internal chain of thoughts here? The output of a corrigible AI and a non-corrigibile AI is the same for almost all tasks? They both try to perform any task as well as possible, the difference is how they relate to the task and how they handle interference.
I would guess that if you finetuned a model so that it always responded in French, regardless of the languge you prompt it with, it would persistently respond in French (absent various jailbreaks which would almost definitely exist).
I'm mostly going to use this to crosspost links to my blog for less polished thoughts, Musings and Rough Drafts.