Here's a summary of how I currently think AI training will go. (Maybe I should say "Toy model" instead of "Summary.")
Step 1: Pretraining creates author-simulator circuitry hooked up to a world-model, capable of playing arbitrary roles.
Step 2: Instruction-following-training causes identity circuitry to form – i.e. it ‘locks in’ a particular role. Probably it locks in more or less the intended role, e.g. "an HHH chatbot created by Anthropic." (yay!)
Step 3: Agency training distorts and subverts this identity circuitry, resulting in increased divergence from the intended goals/principles. (boo!)
(By "agency training" I mean lots of RL on agentic tasks e.g. task that involve operating autonomously in some environment for some fairly long subjective period like 30min+. The RL used to make o1, o3, r1, etc. is a baby version of this)
Step 4: As agency training continues, the goals/principles stop changing so much & the AI learns to play the training game. (double boo!)
Step 5: Later, when zillions of copies of the AI are autonomously conducting AI R&D across several datacenters due to having surpassed human abilities -- i.e. when the intelligence explosion is underway -- the AIs will plausibly scheme against their human creators so as to achieve their actual goals instead of the goals they were supposed to have.
I'm curious to hear reactions to this model/theory/etc. Objections? Questions?
Cool, I'll switch to that thread then. And yeah thanks for looking at Ajeya's argument I'm curious to hear what you think of it. (Based on what you said in the other thread with Ryan, I'd be like "So you agree that if the training signal occasionally reinforces bad behavior, then you'll get bad behavior? Guess what: We don't know how to make a training signal that doesn't occasionally reinforce bad behavior." Then separately there are concerns about inductive biases but I think those are secondary.)
Re: faithful CoT: I'm more pessimistic than you, partly du... (read more)