I basically think that ~most of the misalignment risk comes from models significantly smarter than our current ones.

• I remain uncertain about how much instrumental convergence we have seen in current models, I guess it is ‘not much at all’, but I think it remains plausible that future ones will become much less corrigible and more power-seeking in ways which would surprise us if we just extrapolate the LLM trend (I agree that AF is the among most relevant example of this we’ve seen so far, it would be great to monitor this a lot).
• I find it very plausible that ‘human-level’ LLMs will remain aligned-by-default, and that we can use techniques like Control to align future ones, but I’m worried that model jaggedness will mean it’s harder to automate alignment, or equivalently that sufficiently capable models will be much more power-seeking.
• I agree that models have a very good sense of what we value from pre-training, and that pushing them into this basin seems pretty easy. I’m unsure if this remains true as we get e.g. much more competent reward-hackers. It would be good for OAI to replicate some of the recent experiments of Anthropic on their models.

Why is this your intuition?

At the moment they seem to just make it imitate normal-ish CoT, which would presumably improve accuracy because the model has more token-positions/space/capacity to do things like check for self-consistency. You're still scaling up a compute dimension that the model can use for solving things, and you can still do normal RL things to it from that point.

It's just maybe worse in this case because the causality from reasoning chains -> the part of the response containing the answer is worse (it was bad before, but now it is horrible).

GPT-4o and GPT-4 have different tokenizers (see https://platform.openai.com/tokenizer ); as such it doesn’t seem implausible that they are derived from different models/training runs.

It might be worth pointing out ‘ICA with Reconstruction Cost for Efficient Overcomplete Feature Learning’ (@NeurIPS 2011) which argues that the reconstruction cost |WW^Tx - x| can be used as a form of soft orthonormalization loss.

https://proceedings.neurips.cc/paper/2011/file/233509073ed3432027d48b1a83f5fbd2-Paper.pdf

Edit: As per @Logan Riggs’s comment, I seem to have misunderstood what was being meant by ‘loss recovered’, so this comment is not relevant.

Cool post! However, it feels a little early to conclude that

> Conceptually, loss recovered seems a worse metric than KL divergence.

In toy settings (i.e. trying to apply SAEs to a standard sparse coding setting where we know the ground truth factors, like in https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition/ ), SAEs do not acheive zero reconstruction loss even when they recover the ground truth overbasis with high mean max cosine similarity (and the situation is even worse when noise is present). It's never seemed that obvious to me that we should be aiming to have SAE reconstruction loss go to zero as we train better SAEs, as we could plausibly still use the basis that the SAEs extract, without having to plug the SAE into a 'production' system for mech interp (in which case, we would want good reconstructions).

Note: My take does not neccesarily represent the takes of my coauthors (Hoagy, Logan, Lee, Robert) etc etc. Or it might, but they may frame it differently. Take this as strictly my take.

My take is that the goal isn't strictly to get maximum expressive power under the assumptions detailed in Toy Models of Superposition; for instance, Anthropic found that FISTA-based dictionaries didn't work as well as sparse autoencoders, even though they are better in that they can achive lower reconstruction loss at the same level of sparsity. We might find that the sparsity-monosemanticity link breaks down at higher levels of autoencoder expressivity, although this needs to be rigourously tested.

To answer your question: I think Hoagy thinks that tied weights are more similar to how an MLP might use features during a forward pass, which would involve extracting the feature through a simple dot-product. I'm not sure I buy this, as having untied weights is equivalent to allowing the model to express simple linear computations like 'feature A activation = dot product along feature A direction - dot product along feature B direction', which could be a form of denoising if A and B were mutually exclusive but non-orthogonal features.

Hi David, co-author of the 'Sparse Autoencoders Find Highly Interpretable Directions in Language Models' paper here,
I think this might be of interest to you:
We are currently in the process of re-framing section 4 of the paper to focus more on model steering & activation editing; in line with what you hypothesise, we find that editing a small number of relevant features on e.g. the IOI task can steer the model from its predictions on one token to its predictions on a counterfactual token.