This is a special post for quick takes by ajskateboarder. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Shallow circuits in llms and other pretrained NNs should have non-trivial implications for the interp and alignment of such models, and there's a handful of arguments to expect that they exist[1], but there doesn't seem to be much, if any empirical evidence for them. Approaches relying on sparsity/reconstruction may not be particularly useful for identifying such circuits, e.g. cross-layer transcoders skip over deep computation with shallow circuits[2]. Deep Causal Transcoding however is one approach that relies more on features being necessarily causal, and the structure and success of the method in jailbreaking gives some evidence for shallow circuits being closer to the "true features" of the model[3]. But it seems hard to extract shallow circuits from DCTs that accurately reflect the original network, at least in a sample-efficient manner.
method i tried
Assuming that for exponential DCTs,
where
One constraint that does help generalization: input features interacting jointly in
which is just the prior function for
My tentative guesses so far: the objective function for exponential DCTs is good enough as-is, trying to attribute circuits over them post-hoc could be sufficient, but you will likely need to rely on more 'local' explanations than just going off the weights of the DCT
- ^
if they do exist, I expect they will exist well into the regime of RLing GPTs to death
- ^
E2E+DS SAEs may sidestep these problems though; I have not tried using these yet.
- ^
and better yet, its rather sample-efficient for finding generalizations
Also I've found that input directions in exponential DCTs under the default objective are very redundant compared to output directions. This suggests one-to-many structure https://www.lesswrong.com/posts/fSRg5qs9TPbNy3sm5/deep-causal-transcoding-a-framework-for-mechanistically?commentId=XjwA7MEajpvvkDiDS
[This comment is no longer endorsed by its author]Reply
I've been recently speculating more on what motivations look like in language models and wanted to try sketching a rough mechanistic model for identifying them. This could help make inverse constitution learning or prompt-optimization-type approaches more principled (no affiliation with any group doing this type of work)
If the descriptive model of llms as GLUTs of shallow circuits is correct, then:
- A motivation is some construction (or set of constructions) of shallow circuits that is/are utilized across some distribution; this distribution doesn't need to consist of a single human-interpretable concept (e.g. math or writing); motivations plausibly overlap across domains
- The uniqueness of such a structure comes entirely from how it implements any part of the model's pre/post-training distribution (otherwise it could represent any generic computation)
- Many such structures are produced from RLHF-like processes where a model interacts with a reward model, so maybe there is a way to ablate-then-measure motivations here (should structures that elicit capabilities count as motivations?)
- Many motivations extend from some preference the model has, and preferences could just be condensed down to recalling some single fact/detail
- Some motivations are deeper/shallower than others and they aren't necessarily separated by more/less myopia -- e.g. writing 'genuinely' a lot is probably easier to satisfy than reward hacking in coding envs, which is probably easier than learning a set of capability elicitations in coding envs (a simple proxy for motivational depth could be computational depth with number of shallow circuits involved)
- Naively, any set of motivations are in conflict if each structure causes a significant shift in the output distribution (or some other latent variable)
(if this were to be true then it might be hard to interpret the role of a single shallow circuit for a motivation)
Should we expect latent reasoning/neuralese to replace legible CoT in LLMs soon? I'd assume neuralese would have bad implications for evals/monitoring work, so I'm wondering what work people are planning to prioritize in anticipation of neuralese. Or, are there reasons why neuralese may not become the default (or why it's not a big deal)?
The good news is that it's really hard to train a model to use neuralese. Right now the way we train reasoning models is to first train them to reproduce random text (base models) and then we make minor tweaks to turn them into reasoning models (RL). The base model step only works if you have the exact output that you want the LLM to produce, and we don't have that for neuralese. The RL step needs a model that's already reasonably good and doesn't work well for base models (if the model isn't close to right answers, it's hard for RL to get good feedback).
You can take a base model and then alter it to produce neuralese, but it doesn't work very well since neuralese is out of distribution for the original training.
It's also unclear if neuralese is even helpful from a performance perspective, since forcing outputs to be discrete helps them stay in-distribution.
That said, people are definitely trying to do this and it's hard to predict what the next advance will be.
Someone I know claims to have found a way to directly pretrain neuralese models: https://aklein.bearblog.dev/zebra/
I've seen their prototype, and it definitely works (as far as producing reasonable text outputs while making non-trivial use of >100 continuous latents), but whether it actually amounts to anything remains to be seen.
Follow-up from past shortform: it is much easier to find pairwise shallow circuits in tensor transformers, using a variant of quadratic DCTs
method
Quadratic DCTs rely on symmetric orthogonalized ALS, optimizing roughly the following:
This optimization problem can be done in closed form by doing the equivalent of "gradient ascent on the 'quadratic causal importance term' from equation (8) with infinite step size". However, in the case of bilinear layer transformers, the better optimization problem for recovering features in theory should be:
This allows us to consider the second-order effects of two feature directions
In practice, this variant of quadratic DCT works much better for bilinear-layer transformers than GLU transformers. I tested this variant of DCT on Elriggs/gpt2-bilinear-18l-9h-1152embd and was able to find diverse circuits, where pairs of input features jointly elicit unique generalizations, across deep slices of the transformer. This is almost certainly due to
thanks Logan Riggs for tensor transformers, and gpt 5.6 sol for red-teaming. tensor transformers are definitely underrated
cached thought: Maybe part of the reason the latest AIs are getting to be rather egregiously misaligned is that inoculation prompting no longer works for this regime