I've been recently speculating more on what motivations look like in language models and wanted to try sketching a rough mechanistic model for identifying them. This could help make inverse constitution learning or prompt-optimization-type approaches more principled (no affiliation with any group doing this type of work)

If the descriptive model of llms as GLUTs of shallow circuits is correct, then:

• A motivation is some construction (or set of constructions) of shallow circuits that is/are utilized across some distribution; this distribution doesn't need to consist of a single human-interpretable concept (e.g. math or writing); motivations plausibly overlap across domains
  • The uniqueness of such a structure comes entirely from how it implements any part of the model's pre/post-training distribution (otherwise it could represent any generic computation)
  • Many such structures are produced from RLHF-like processes where a model interacts with a reward model, so maybe there is a way to ablate-then-measure motivations here (should structures that elicit capabilities count as motivations?)
• Many motivations extend from some preference the model has, and preferences could just be condensed down to recalling some single fact/detail
• Some motivations are deeper/shallower than others and they aren't necessarily separated by more/less myopia -- e.g. writing 'genuinely' a lot is probably easier to satisfy than reward hacking in coding envs, which is probably easier than learning a set of capability elicitations in coding envs (a simple proxy for motivational depth could be computational depth with number of shallow circuits involved)
• Naively, any set of motivations are in conflict if each structure causes a significant shift in the output distribution (or some other latent variable)

(if this were to be true then it might be hard to interpret the role of a single shallow circuit for a motivation)