I think this is a really important thing to get right. I feel like we want to brainstorm lots of candidates for “motivational structure”, eg decision trees, probabilistic progeams, … and then discuss the pros and cons of each. It might also be useful to do more first principles thinking about what properties we expect a “motivational structure” to have. Eg the hierarchical property you’ve pointed out, as well as the transitivity property maybe. I expect this gets cursed bc lots of things depend on a fixed ontology. And two motivational structures may appear indistinguishable under one ontology but be completely different under another. (Ie maybe your evals suggest that Claude is being honest all the time; but maybe it’s only being honest in situations that look like training, or only in situations where its capabilities are not superhuman, etc. The latter two don’t get caught unless your ontology includes sufficiently and honeslty kinda weird finegrained concepts.) IDK maybe you also want to postulate regularity conditions that should hold for motivational structures, we probably are fine ignoring a class of extremely cursed ones in order to deal with the well behaved ones