This is a special post for quick takes by vigji. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Re reward seeking models, maybe there is some unescapable tensions in cognisant systems doomed to know everything about themselves.
Humans are blissed by (at least partial) blindness to their values shards, and large ignorance of what drove them there; the completeness of models self-representation might be an inexorable tension - the smarter the model, the strongest the drive - towards eating out the useful verisimilitude delta that keeps training/evals matter something for real world deployments.