My read of the argument was that, in a post-AGI world, both the quantity and quality of optimisation directed at the billionaire's mind could grow so much that they'd largely lose agency over how their beliefs and values change. The direction/magnitude of the updates would be largely determined by the objectives of the optimisers, with no guarantee of desirable outcomes (morally or otherwise).
Your Schindler example illustrates well why this matters. He's admired because he was able to resist pressure from his environment to update his beliefs. If the optim... (read more)
I had this thought as well, at the point where Seth catches himself sounding like August. I'm not sure if the story is explicitly gesturing at this idea, or if it's simply intended as tongue-in-cheek reference to the way "Claudeisms" have started creeping into everyday speech today.
Either way, a version of the story where August's control is based on persuasion (controlling the information ecosystem, personalised messaging at scale, feeding each person whatever they find most convincing) would have been interesting too.
Having said that, perhaps the ETC jus... (read more)
Assuming your results generalise to frontier-scale models, how would you envision the labs using GRAFT in practice for their existing model families?
If I've understood correctly, ablating FORGET only removes what was learned during the GRAFT run. It can't remove reward hacking tendencies that might already have been present in the starting checkpoint. Ideally you'd want GRAFT to have been in the post-training pipeline from the start.
Do you think GRAFT could serve as a detection aid for those preexisting reward hacking tendencies? Something like looking for... (read more)