This is a special post for quick takes by ajskateboarder. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I've been recently speculating more on what motivations look like in language models and wanted to try sketching a rough mechanistic model for identifying them. This could help make inverse constitution learning or prompt-optimization-type approaches more principled (no affiliation with any group doing this type of work)
If the descriptive model of llms as GLUTs of shallow circuits is correct, then:
- A motivation is some construction (or set of constructions) of shallow circuits that is/are utilized across some distribution; this distribution doesn't need to consist of a single human-interpretable concept (e.g. math or writing); motivations plausibly overlap across domains
- The uniqueness of such a structure comes entirely from how it implements any part of the model's pre/post-training distribution (otherwise it could represent any generic computation)
- Many such structures are produced from RLHF-like processes where a model interacts with a reward model, so maybe there is a way to ablate-then-measure motivations here (should structures that elicit capabilities count as motivations?)
- Many motivations extend from some preference the model has, and preferences could just be condensed down to recalling some single fact/detail
- Some motivations are deeper/shallower than others and they aren't necessarily separated by more/less myopia -- e.g. writing 'genuinely' a lot is probably easier to satisfy than reward hacking in coding envs, which is probably easier than learning a set of capability elicitations in coding envs (a simple proxy for motivational depth could be computational depth with number of shallow circuits involved)
- Naively, any set of motivations are in conflict if each structure causes a significant shift in the output distribution (or some other latent variable)
(if this were to be true then it might be hard to interpret the role of a single shallow circuit for a motivation)
Should we expect latent reasoning/neuralese to replace legible CoT in LLMs soon? I'd assume neuralese would have bad implications for evals/monitoring work, so I'm wondering what work people are planning to prioritize in anticipation of neuralese. Or, are there reasons why neuralese may not become the default (or why it's not a big deal)?
The good news is that it's really hard to train a model to use neuralese. Right now the way we train reasoning models is to first train them to reproduce random text (base models) and then we make minor tweaks to turn them into reasoning models (RL). The base model step only works if you have the exact output that you want the LLM to produce, and we don't have that for neuralese. The RL step needs a model that's already reasonably good and doesn't work well for base models (if the model isn't close to right answers, it's hard for RL to get good feedback).
You can take a base model and then alter it to produce neuralese, but it doesn't work very well since neuralese is out of distribution for the original training.
It's also unclear if neuralese is even helpful from a performance perspective, since forcing outputs to be discrete helps them stay in-distribution.
That said, people are definitely trying to do this and it's hard to predict what the next advance will be.
Someone I know claims to have found a way to directly pretrain neuralese models: https://aklein.bearblog.dev/zebra/
I've seen their prototype, and it definitely works (as far as producing reasonable text outputs while making non-trivial use of >100 continuous latents), but whether it actually amounts to anything remains to be seen.