ojorgensen

Because of LayerNorm, Directions in GPT-2 MLP Layers are Monosemantic

Yeah I think we have the same understanding here (in hindsight I should have made this more explicit in the post / title).

I would be excited to see someone empirically try to answer the question you mention at the end. In particular, given some direction and a LayerNormed vector $v$ , one might try to quantify how smoothly rotating from $v$ towards $u$ changes the output of the MLP layer. This seems like a good test of whether the Polytope Lens is helpful / necessary for understanding the MLPs of Transformers (with smooth changes corresponding to your 'random jostling cancels out' corresponding to not needing to worry about Polytope Lens style issues).

Replying toOpen Thread – Winter 2023/2024

Open Thread – Winter 2023/2024

It would save me a fair amount of time if all lesswrong posts had an "export BibTex citation" button, exactly like the feature on arxiv. This would be particularly useful for alignment forum posts!

Replying toAgainst Almost Every Theory of Impact of Interpretability

Against Almost Every Theory of Impact of Interpretability

One central criticism of this post is its pessimism towards enumerative safety. (i.e. finding all features in the model, or at least all important features). I would be interested to hear how the author / others have updated on the potential of enumerative safety in light of recent progress on dictionary learning, and finding features which appear to correspond to high-level concepts like truth, utility and sycophancy. It seems clear that there should be some positive update here, but I would be interested in understanding issues which these approaches will not contribute to solving.

Replying toSome ML-Related Math I Now Understand Better

Understanding Counterbalanced Subtractions for Better Activation Additions

But this does not hold for tiny cosine similarities (e.g. 0.01 for $n = 12288$ , which gives a lower bound of 2 using the formula above). I'm not aware of a lower bound better than $n$ for tiny angles.

Unless I'm misunderstanding, a better lower bound for almost orthogonal vectors when cosine similarity is approximately $0$ is just $n$ , by taking an orthogonal basis for the space.

My guess for why the formula doesn't give this is because it is derived by covering a sphere with non-intersecting spherical caps, which is sufficient for almost orthogonality but not necessary. This is also why the lower bound of $2$ vectors makes sense when we require cosine similarity to be approximately $0$ , since then the only way you can... (read 1001 more words →)

Because of LayerNorm, Directions in GPT-2 MLP Layers are Monosemantic

Introduction

Activation additions represent an exciting new way of interacting with LLMs, and have potential applications to AI safety. However, the field is very new, and there are a few aspects of the technique that do not have a principled justification. One of these is the use of counterbalanced subtractions - both adding a feature vector we want, and taking away some vector that we do not want to be present in the output of the model. For example, to make a vector to steer the outputs of GPT-2 XL to be more loving, the activations associated with the input "Love" were paired with the input "Hate", subtracting the "Hate" activations from the... (read 3934 more words →)

I think that this has been stated implicitly in various comments and lesswrong posts, but I haven't found a post which makes this point clearly, so thought I would quickly write one up!

To get the most out of this, you should probably be pretty familiar with GPT-2 style transformer architectures. If you want a refresher, big shoutout to A Mathematical Framework for Transformer Circuits and Neel Nanda's What is a Transformer? video.

Introduction: Polytope Lens for Transformers?

When we are trying to interpret transformer models, a central challenge is to try to assign single, human interpretable meanings to different regions of the residual stream.

It would be very convenient if we could assign meanings to... (read 3831 more words →)

Replying toMech Interp Puzzle 1: Suspiciously Similar Embeddings in GPT-Neo

Mech Interp Puzzle 1: Suspiciously Similar Embeddings in GPT-Neo

(Potential spoilers!)

There is some relevant literature which explores this phenomenon, also looking at the cosine similarity between words across layers of transformers. I think the most relevant is (Cai et. al 2021), where they also find this higher than expected cosine similarity between residual stream vectors in some layer for BERT, D-BERT, and GPT. (Note that they use some somewhat confusing terminology: they define inter-type cosine similarity to be cosine similarity between embeddings of different tokens in the same input; and intra-type cosine similarity to be cosine similarity of the same token in different inputs. Inter-type cosine similarity is the one that is most relevant here).

They find that the residual stream vectors for GPT-2 small tend to lie in two distinct clusters. Once you re-centre these clusters, the average cosine similarity between residual stream vectors falls to close to 0 throughout the layers of the model, as you might expect.

UK Foundation Model Task Force - Expression of Interest

Ian Hogarth has just been announced as the Chair of the UK's AI Foundation Model Taskforce. He's the author of the FT article "We must slow down the race to God-like AGI", and seems to take X-risks from AI seriously.

To quote his twitter thread:

And to that end I put out a call to people across the world. If you are an AI specialist or safety researcher who wants to build out state capacity in AI safety and help shape the future of AI policy then get in touch:

We have £100m to spend on AI safety and the first global conference to prepare for. I want to hear from you and how you think you can help. The time is now and we need more people to step up and help.

The google form to leave an expression of interest is here.

(I am in no way affiliated with Ian or the UK Foundation Model Task Force)

ojorgensen3yQuick Take

Problem: we want to make it hard for ML systems (trained via SGD) to perform naive gradient hacking. By naive gradient hacking, I mean "being able to keep some weights of the network constant for an arbitrary step of SGD".

Solution: do "stochastic" regularisation, e.g. sample the amount of regularisation we perform randomly (could use quantum stuff if we want true randomness). This seems like it should make naive gradient hacking almost impossible - in order to keep some target weights unchanged, you'd have to match their +ve contribution to the loss to the degree of regularisation. If the degree of regularisation is stochastic, the loss contribution must also be stochastic, which is not possible with a determinstic NN!

The question of how useful this is from preventing any gradient hacking depends on how stable the loss landscape around some "deceptive / gradient hacking minimum is". Seems possible the surrounding loss landscape could be pretty unstable to random pertubations?

ojorgensen's Shortform

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Replying toExcessive AI growth-rate yields little socio-economic benefit.

Excessive AI growth-rate yields little socio-economic benefit.

Just a nit-pick but to me "AI growth-rate" suggests economic growth due to progress in AI, as opposed to simply techincal progress in AI. I think "Excessive AI progress yields little socio-economic benefit" would make the argument more immediately clear.

Replying toEIS XI: Moving Forward

EIS XI: Moving Forward

Rando et al. (2022)

This link is broken btw!

Replying toAbuse in LessWrong and rationalist communities in Bloomberg News

Abuse in LessWrong and rationalist communities in Bloomberg News

Didn't get that impression from your previous comment, but this seems like a good strategy!

(Extremely) Naive Gradient Hacking Doesn't Work

Which Issues in Conceptual Alignment have been Formalised or Observed (or not)?

Tldr

A super simple form of gradient hacking might be to perform badly on any change to the value of the model's mesa-objective, in order to preserve the mesa-objective. I use some simple properties of SGD to explain why this method of gradient hacking cannot work.

Thanks for Jacob Trachtman, Jakub Kraus, and Francis Priestland for feedback. Thanks also to Evan Hubinger for responding to my questions.

Assumed Knowledge

I think this post will be most valuable for those who have some familiarity with the alignment problem (in particular, understanding of inner misalignment, mesa-optimisation). Having done something like AGISF would be more than sufficient for reading this.

As an introduction to mesa-optimsation I’d recommend Risks from Learned Optimization:... (read 1604 more words →)

Strange Loops - Self-Reference from Number Theory to AI

One aspect of alignment research I find interesting is improving how we understand potential threat / failure modes by making them more concrete or rigorous in some way. When I think of making these concrete, I’m thinking of theoretical results in ML (think Turner’s power-seeking work) or toy examples of the behaviour arising (that funny boat doing reward hacking).

To my knowledge, some conceptual failure modes which are central to the field have not been observed yet (deceptive alignment and gradient hacking come to mind). This isn't to suggest that they won't be observed in more powerful systems, but it seems important to note nonetheless.

I think it would be useful to have a... (read more)

Evaluating OpenAI's alignment plans using training stories

Summary

I see this post as essentially trying to make clear the link between self-reference as discussed in Hofstadter’s “Gödel, Escher, Bach” and self-reference as considered in some of MIRI’s work on Embedded Agency. I start by considering instances of self-reference in mathematics through Gödel's Incompleteness Theorems, before moving on to instances of self-reference in Löb's Theorem and Embedded Agency.

The aims of this post are to provide a different perspective on the issue of self-reference, as well as to give some intuition behind the proof of Gödel's Incompleteness Theorem. I found this helpful when trying to understand some elements of Embedded Agency, and I hope others might too. I doubt this will be... (read 5374 more words →)

Disagreements about Alignment: Why, and how, we should try to solve them

In this post, I am going to recast the recent “OpenAI’s Alignment Plans” in terms of Evan Hubinger’s Training Stories. I found this somewhat useful in terms of understanding this research agenda, and hopefully it might be useful for others too!

I am going to focus on the approach to alignment titled “Training AI systems to do alignment research”, as this is their current approach to actually creating an aligned AGI system. I think OpenAI sees the agendas “Training AI systems using human feedback” and “Training models to assist human evaluation” as being useful for this goal, but not ways of aligning AGI systems themselves, so I will only talk about these approaches... (read 1275 more words →)