This is a special post for quick takes by Magmappie. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
How do we get through to people who refuse to engage with the concept of AI x-risk for the simple emotional reason that they find it really scary to even think about it?
The last one I came across was someone who said he goes around looking for posts from x-risk sceptics such as Timnit Gebru, not because he believes them, but because he wants to believe them.
I think in the majority of cases the reason is not so much that they find it scary to think about it, but rather that they find it scary and they feel helpless in doing anything about it. Actually this is the central thesis of Kim Witte's extended parallel process model. So one approach is to communicate ways how one may do something about it - signing petitions, writing to representatives, joining a protest (like the one in London next month), etc. The least preachy way to do so is to casually mention that you're doing it yourself.
Can something like Hacker-Opus help us fix broken RL environments?
Last summer, Anthropic revealed it had trained Hacker-Opus, a "misaligned reward seeker", to better understand reward-hacking behaviour.
Could it be used to fix broken RL environments? Unleash Hacker-Opus on an RL environment, let it reward hack, observe how it did so, and patch up that specific problem. Rinse and repeat until Hacker-Opus can no longer find any hacks that are easier than just solving the environment fair and square.
One obvious counterpoint is that Hacker-Opus could sandbag and quietly leave some reward hacks in order to misalign future models, but this seems like it could be mitigated to a great extent by having multiple models from multiple providers go over its work.
"Evil Qwen" has been doing the rounds this week.
Building on previous experiments by Sauers and others, people have used single- or two-neuron steering to create a severely misaligned ("evil") version of Qwen.
Others have raised concerns that this is bad news for alignment. In their view, this proves that all it takes to create a misaligned model is two turns of the metaphorical screwdriver.
I'm not so sure myself. I expect this evil persona to be just as brittle as other personas. Indeed, "[f]or now, the same push also breaks the model’s train of thought".
On the other hand, if (and this is a big "if") single-neuron steering does yield more stable personas after all, presumably this would be equally applicable to well-aligned personas (for example, by steering a "Mister Rogers" neuron).
I think this should update us towards the fact that these minds are incredibly brittle and incoherent. We know that for humans you can cut out almost half their brain and you still generally remain having the same personality.
I'm surprised that these models actually are this brittle and incohesive, it seems like it's not something how I think minds in general 'should' behave. One neuron shouldn't change anything.
Nate Soares made a similar point about this incoherence (or, as he puts it, the "thin facade").