LESSWRONG
LW

Linda Linsefors — LessWrong

I feel like if I try to defend my openmindedness I loose. It just opens up more attac surfaces to someone who is hostile and doesn't argues in good faith.

I think it's much better to call out why calling someone close minded for not listening is just invalid in general, not just this time in particular. And I do believe it is.

If someone isn't listening to you. Them being to close minded is so faaaaaar the list of most likely explanation that. Much less likely than:

Your argument are bad
Dissintrest in the topic
Other things they rather do right now

Linda Linsefors4d

"I guess I am a bit closed on this particular topic - let's discuss something else".

I wish I could say something like that and be ok. But to me it feels too humiliating. And also often factually wrong, I.e. I'd be open to good argument.

Linda Linsefors4d

Bulverism is a good term, thanks!

Linda Linsefors5d

I don't want to use this sugestion, not because it is escalatory, but because it's a question, which invites them to have more opinions.

What I want is a way out, but that has the feeling of standing up for myself, rather then the feeling of humiliation and defeet.

If someone starts to accuse me of not beeing openminded to their opinion, it's usually because I think their opinion is dumb. I rather not hurt their feelings if I can avoid it, but I'm also not going to worry too much about being polite to someone after they done this particular retorical move.

Usually the way out is to just leave. But last time this happened was at a small metup, and the only way out was to leave the event, which I did. I'm not happy about this and would like better options.

Linda Linsefors5d*Quick Take

Even now and then I meet someone who tries to argue that if I don't agree with them this is because I'm not open mided enough. Is there a term for this?

Epistemically I'm not convinced buy this type of arugment, but socialy it feels like I'm beeing shamed, and I hate it.

I also find it hard to call out this type of behaviur when it happens, even when I can tell exactly what is going on. I think it I had a name for this behaviour it would be easier? Not sure though?

Edit to add:

I've now got some more time to figure out what I want and don't want out of this... (read more)

Replying toSafety consultations for AI lab employees

Linda Linsefors1mo

Safety consultations for AI lab employees

Is this something you're stilld doing?

(Just asking in general to keep track of what resouses exists.)

Replying toTwo ways non-U.S. folks can contribute to AI going well

Linda Linsefors1mo

Two ways non-U.S. folks can contribute to AI going well

I think that more countries oficially warning the world about AI risk can do a lot to shift the overton window, which is very impactfull.

Replying toMe and M&Ms

Linda Linsefors1mo

Me and M&Ms

I somehow stumbled on this old post. I'm curious how your experiment with diffrent reinforcment scheduled worked.

My prediction is that your original, get one M&M for each pomodory (bundled when that is practical) worked best, and any exra randomness didn't help.

My reasoning. When ever I read about reinforcment and extinction, I run the following test: Would that outcome be predicted by assuming the animal was inteligently trying to predict what is going on? And the answer is always "yes".

E.g. Why is varied schedules harder to extinguish? Becasue it requires more evideince to make sure the reward is gone. If the reward is predictable, noticeing it's absense is easy, but if it's undpredictable, then you never know. If you're a lab animal.

When I apply this heuristic to your situation, then if you miss an M&M one day, you know what is going on, and you know that this is does not mean the M&M has stopped forever. This is very diffrent from animal training.

Replying toHow I stopped being sure LLMs are just making up their internal experience (but the topic is still confusing)

Linda Linsefors1mo

How I stopped being sure LLMs are just making up their internal experience (but the topic is still confusing)

The latest Claude models, if asked to add two numbers together and then queried on how they did it, will still claim to use the standard “carry ones” algorithm for it.

Could anyone check if the lying feature activates for this? My guess is "no", 80% confident.

Replying toFact Finding: Do Early Layers Specialise in Local Processing? (Post 5)

Linda Linsefors1mo

Fact Finding: Do Early Layers Specialise in Local Processing? (Post 5)

Thanks for responding.

I was imagining "local" meaning below 5 or 10 tokens away, partly anchored on the example of detokenisation, from the previous posts in the sequence, but also because that's what you're looking at. If your definition of "local" is longer than 10 tokens, then I'm confused why you didn't show the results for longer trunkations. I though the plot was to show what happens if you include the local context but cut the rest.

Even if there is specialisation going on between local and long range, I don't expect a sharp cutoff what is local v.s. non-local (and I assume neither do you). If some such soft boundary exists and it... (read more)

LLMs (and probably most NNs) have lots of meaningfull, interpretable linear feature directions. These can be found though various unsupervised methods (e.g. SAEs) and supervised methods (e.g. linear probs).

However, most human interpretable features, are not what I would call the models true features.

If you find the true features, the network should look sparse and modular, up to noise factors. If you find the true network decomposition, than removing the what's left over should imporve performance, not make it wors.

Because the network has a limited number of orthogonal directions, there will be interference terms that the network would like to remove, but can't. A real network decomposition will be everything except this noise.

This is what I think mech-interp should be looking for

It's possible that I'm wrong and there is no such thing as "the networks true features". But we've (humans colectivly) only just started this reaserch agenda. The fact that we haven't found it yet, is not much evidence either way.

Rotations in Superposition

Linda Linsefors

Linda Linsefors, Lucius Bushnaq

2mo

TL;DR We experimentally test the mathematical framework for circuits in superposition by hand-coding the weights of an MLP to implement many conditional^[1] rotations in superposition on two-dimensional input features. The code can be found here.

This work was supported by Coefficient Giving and Goodfire AI

1 Introduction

Figure 1: The desired values for rotating vectors $x$ (green), compared to their estimated values extracted from the neural network, $^x$ (purple) for five of the conditional rotation circuits. The six dots represent the values of the vector $x$ in layers $0$ to $5$ .

In a previous post, we sketched out a construction for compressing many different circuits into a neural network such that they can be computed in superposition.^[2]

By computation in superposition, we mean that a network represents... (read 3134 more words →)

Solstice Singalong Watch Party

Linda Linsefors

Linda Linsefors, philh

3mo

Welcome to Secular Solstice celebration at Arkhipov House!

Location: E2 0QW, Bethnal Green, London UK
Sign up to see exact adress

The main event is that we will watch the recording of the Bay Solstice ceremony and sing along to all the songs. This year's recording does not exist yet, but here's last year's for reference. (Last year's recording is also our backup plan if there is some issue with the new one.)

PLEASE DON'T ARRIVE AFATER 6PM

There will be:

Vegan food (dinner)
Sofa
Optional cuddles
Singing

Schedule:

Arrive from 5pm, eat dinner
Watch party starts at 6pm, expected to last around 3 hours
Feel free to stay and talk/cuddle in the sofa afterwards

I remember reading that some activation stearing experiments used x100 size activation vectors. Is this correct? That would be much larger than the normal activation in that layer, right?

How does the model deal with this? If there is any superposition going on, I expect activation spill over everywhere, breaking everything.

If you amplify the activations in one layer, what effect has that on the magnitude of the activations in the next layer? If it's smaller, by what mechanism? That mechanism is probably an error correction algorithm. Or just some suppression aimed at not letting the network thinking about too many things at once, in order to keep down interference.

Does anyone have experience with activation stearing, and would be up for helping me out? E.g. aswer my questions, jump on a call, or help me set up my own experiments?

Is there anyone in London who want to colaborate with me on some Comp-in-Sup reserch?

Prerequisits are knowing some Linear Algebra and some Python/Pytorch.

When I'm in a conversation often track who the conversation is for. I.e. who is this conversation primerely seving, in this moment.

If I'm ranting, then this conversation is for me, to let me realsease some tension. I will express my self in a way that feels good to me.
If I'm sharing usefull information, then the conversation is for the other person. I will express my self in a way to make the information clear and accessable for them, and also pay attetion to if they even want this information.
I can explain myself becasue I need to be seen, or becasue another person wants to understand me. But these are diffrent things.
Sometimes more

... (read more)

AI Safety Camp 10 Outputs

Robert Kralisch

Robert Kralisch, Remmelt, Linda Linsefors

5mo

See below the project outputs for AI Safety Camp's 10th edition, which took place from January to April 2025. You can also find them on our website.

This year's edition featured a wide range of approaches to reducing AI risk. Projects resulted in published papers, produced helpful community resources, grants for follow-up research, and/or helped team members transition to full-time work in AI Safety.

You are encouraged to scan the post outline for the topic area of greatest interest to you and check out the projects we hosted. As a general rule, the first person listed among the team members was the project lead for the respective project.

Stop/Pause AI

Growing PauseAI

Team members: Chris Gerrby, Sharon Mwaniki, Alyssa Chase-Vilchez,... (read 4982 more words →)

Estimated MSE loss for three diffrent ways of embedding features into neuons, when there are more possible features than neurons.

I've typed up some math notes for how much MSE loss we should expect for random embedings, and some other alternative embedings, for when you have more features than neurons. I don't have a good sense for how ledgeble this is to anyone but me.

Note that neither of these embedings are optimal. I belive that the optimal embeding for minimising MSE loss is to store the features in almost orthogonal directions, which is similar to random embedings but can be optimised more. But I also belive that MSE loss don't prefeer this solution... (read more)

I'm getting typo reacts (not suprsing) but I can't see where typo is when hovering over the reaction. What am I doing wrong?

Thoughts on how to onboard volunteers/collaborators

Volunteers are super flaky, i.e. often abandon projects with no warning. I think the main reason for this is planning fallacy. People promise more than they actually have time for. Best case they will just be honest about this when they notice. But more typically the person will feel some mix of stress, shame and overwhelm that prevents them from communicating clearly. From the outside this looks like person promised to do a lot of things and then just ghosts the project. (This is a common pattern and not a specific accusation of any specific person.)

I don't think I'm particularly good at volunteer management. But I have... (read 899 more words →)

In Defence of Jargon

People used to say (maybe still do? I'm not sure) that we should use less jargon to increase accessibility to writings on LW, i.e. make it easier to outsider to read.

I think this is mostly a confused take. The underlying problem is inferential distance. Geting rid of the jargon is actually unhelpful since it hides the fact that there is an inferential distance.

When I want to explain physics to someone and I don't know what they already know, I start by listing relevant physics jargon and ask them what words they know. This is a super quick way to find out what concepts they already have, and let me... (read more)

•••

Circuits in Superposition 2: Now with Less Wrong Math

Linda Linsefors

Linda Linsefors, Lucius Bushnaq

8mo

Summary & Motivation

This post is a continuation and clarification of Circuits in Superposition: Compressing many small neural networks into one. That post presented a sketch of a general mathematical framework for compressing different circuits into a network in superposition. On closer inspection, some of it turned out to be wrong, though. The error propagation calculations for networks with multiple layers were incorrect. With the framework used in that post, the errors blow up too much over multiple layers.

This post presents a slightly changed construction that fixes those problems, and improves on the original in some other ways as well.^[1]

By computation in superposition we mean that a network represents features in superposition and... (read 6418 more words →)

London Rationalish meetup - Arkhipov

philh

philh, Linda Linsefors

10mo

This month we're again hosted by Arkhipov, the group house some occasional members live in. We'll be in the large living room/kitchen.

The actual address is [redacted, go to facebook to see it, but the postcode is roughly E2 0QN].

The reading list for this month is:

Why I'm not a Bayesian
The business of check cashing
We'll be showing up from 2, and start to discuss these around 3. If you have articles you want to suggest for future readings, you can do that at https://redd.it/v3646u.

If you want to join the Whatsapp group, append the suffix "BbILI8HzX3zgJF8i" to the prefix "https://chat.whatsapp.com/IUIZc3".

MAISU - Minimal AI Safety Unconference

Linda Linsefors

MAISU starts with an Opening session on April 18th (Friday), but most of the sessions will happen during April 19th-21th. You’re welcome to join as much or little as you want.

The event is for anyone who wants to help prevent AI-driven catastrophe. Other than that, we’re open to all perspectives. However each individual session will typically have narrower scope. When attending a session please respect the specific scope of that session.

Because this is an Unconference, the schedule is open to anyone to add sessions. Expect the schedule to continuously expand as the event approaches.

Official website

The official website is guaranteed to have the latest information. This LessWrong event post might be updated less... (read 386 more words →)

Is the output of the softmax in a single transformer attention head usually winner-takes-all?

Linda Linsefors

Using the notation from here: A Mathematical Framework for Transformer Circuits

The attention pattern for a single attention head is determined by $A = softmax (x^{T} W_{Q}^{T} W_{K} x)$ , where softmax is computed for each row of $x^{T} W_{Q}^{T} W_{K} x$ .

Each row of $A$ gives the attention pattern for the current token. Are these rows (post softmax) typically close to one-hot? I.e. are they mainly dominated by a single attention (per current token).

I'm interested in knowing this for various types of transformers, but mainly for LLM and/or frontier models.

I'm asking because I think this has implication for computations in super-position.

Theory of Change for AI Safety Camp

Linda Linsefors

In a private discussion, related to our fundraiser, it was pointed out that AISC hasn't made clear enough what our theory of change is. Therefore this post.

Some caveats/context:

This is my personal viewpoint. Other organisers might disagree about what is central or not.
I’ve co-organised AISC1, AISC8, AISC9, and now AISC10. Remmelt has co-organised all except AISC2. Robert recently joined for AISC10.
I hope there will be an AISC11, but in either case, I will no longer be an organiser. This is partly because I get restless when running the same thing too many times, and partly because there are other things I want to do. But I do think the AISC is in good hands

... (read 1817 more words →)

We don't want to post again "This might be the last AI Safety Camp"

Remmelt

Remmelt, Linda Linsefors, Robert Kralisch

We still need more funding to be able to run another edition. Our fundraiser raised $6k as of now, and will end if it doesn't reach the $15k minimum, on February 1st. We need proactive donors.

If we don't get funded for this time, there is a good chance we will move on to different work in AI Safety and new commitments. This would make it much harder to reassemble the team to run future AISCs, even if the funding situation improves.

You can take a look at the track record section and see if it's worth it:

≥ $1.4 million granted to projects started at AI Safety Camp
≥ 43 jobs in AI Safety taken

Linda Linsefors

AI Safety Outreach = Making more people better informed about AI risk, through any means that works.

Who is this event for:
Anyone who wants to help with, or might want to help with AI safety outreach. No qualifications required.

Approximate Schedule:
There will be a mix of talks, 1-on-1s with other participants, Q&A, some breaks and 2-minute lighting shares from the audience. Here's an approximate schedule, but expect it to be adjusted on the fly.

The event (other than the 1-on-1s) will be recorded and shared publicly, but some parts may be removed on the request of the speakers.

Speakers:

William Baird
Remmelt Ellen

Collaborative note taking:
Shared notes for AI Safety Outreach Seminar - Google Docs

Registration (optional):
Registration is not necessary, but if you sign up here you'll get an event reminder.

Every time I have an application form for some event, the pattern is always the same. Steady trickle of applications, and then a doubling on the last day.

And for some reason it still surprises me how accurate this model is. The trickle can be a bit uneven, but the doubling the last day is usually close to spot on.

This means that by the time I have a good estimate of what the average number of applications per day is, then I can predict what the final number will be. This is very useful, for knowing if I need to advertise more or not.

For the upcoming AISC, the trickle was a late skewed,... (read more)

•••

LESSWRONG
LW

LESSWRONG
LW

Linda Linsefors

This might be the last AI Safety Camp

AISC9 has ended and there will be an AISC10

Circuits in Superposition 2: Now with Less Wrong Math

Funding case: AI Safety Camp 10

Linda Linsefors

Rotations in Superposition

AI Safety Camp 10 Outputs

Circuits in Superposition 2: Now with Less Wrong Math

Is the output of the softmax in a single transformer attention head usually winner-takes-all?

Theory of Change for AI Safety Camp

We don't want to post again "This might be the last AI Safety Camp"

Funding Case: AI Safety Camp 11

Linda Linsefors

This might be the last AI Safety Camp

AISC9 has ended and there will be an AISC10

Circuits in Superposition 2: Now with Less Wrong Math

Funding case: AI Safety Camp 10

Linda Linsefors

Rotations in Superposition

AI Safety Camp 10 Outputs

Circuits in Superposition 2: Now with Less Wrong Math

Is the output of the softmax in a single transformer attention head usually winner-takes-all?

Theory of Change for AI Safety Camp

We don't want to post again "This might be the last AI Safety Camp"

Funding Case: AI Safety Camp 11

1 Introduction

Stop/Pause AI

Growing PauseAI

Estimated MSE loss for three diffrent ways of embedding features into neuons, when there are more possible features than neurons.

Summary & Motivation

Official website