This is a special post for quick takes by Magmappie. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Building on previous experiments by Sauers and others, people have used single- or two-neuron steering to create a severely misaligned ("evil") version of Qwen.
Others have raised concerns that this is bad news for alignment. In their view, this proves that all it takes to create a misaligned model is two turns of the metaphorical screwdriver.
On the other hand, if (and this is a big "if") single-neuron steering does yield more stable personas after all, presumably this would be equally applicable to well-aligned personas (for example, by steering a "Mister Rogers" neuron).
I think this should update us towards the fact that these minds are incredibly brittle and incoherent. We know that for humans you can cut out almost half their brain and you still generally remain having the same personality.
I'm surprised that these models actually are this brittle and incohesive, it seems like it's not something how I think minds in general 'should' behave. One neuron shouldn't change anything.
It seems intuitive that such minds would arise from pre-training. E.g., optimization pressure pushes them towards having cheap mechanisms to switch between personalities.