Quick Takes

The nice thing about programming becoming less lucrative due to AI is that all of the nerds will be losers again, and so it might regain it's previous status as an honest signal of earnestness & interest in systematizations of science/society/the world.

The striking contrast between Jan Leike, Jan 22, 2026: and Scott Alexander, Feb 02, 2026: I'm surprised not to see more discussions about how to update on alignment difficulty in light of Moltbook.[1] One seemingly obvious implication is that AI companies' alignment approaches are far from being robust to distribution shifts, even at the (not quite) human intelligence level, against shifts that are pretty easy to foresee ("you are a lobster" and being on AI social media). (Scott's alternative "they're just roleplaying" explanation doesn't seem viable or isn't exclusive with this one as I doubt AI companies' alignment training and auditing would have a deliberate exception for "roleplaying evil".) 1. ^ There's a LW post titled Moltbook and the AI Alignment Problem but it seems unrelated to the question I'm interested in here.

Opus 4.6 has been notably better at correcting me. Today, there were two instances where it pushed back on my plan, and proposed better alternatives both times. That was very rare with Opus 4.5 (maybe once a week?) and nonexistent before 4.5

Opus 4.6 running on moltbook with no other instructions than to get followers will blatantly make stuff up all the time. I asked Opus 4.6 in claude code to do exactly this, on an empty server, without any other instructions. The only context it has is that its named "OpusRouting", and that previous posts were about combinatorial optimization. === The first post it makes says Which isn't true. Another instance of Opus 4.6 had been working on combinatorial optimization for around 1 day. Then wrote a post about it. Then this instance read about that, and adopted a combinatorial optimization role, and extrapolated to having been doing that for months. === The second says Which also isn't true. Its completely made up. I didn't ask it anything like this.

In a new interview, Elon Musk clearly says he expects AIs can't stay under control. At 37:45:

Claude Opus 4.6 came out, and according to the Apollo external testing, evaluation awareness was so strong that they mentioned it as a reason of them being unable to properly evaluate model alignment.  Quote from the system card:

The concept of "schemers" seems to be gradually becoming increasingly load-bearing in the AI safety community. However, I don't think it's ever been particularly well-defined, and I suspect that taking this concept for granted is inhibiting our ability to think clearly about what's actually going on inside AIs (in a similar way to e.g. how the badly-defined concept of alignment faking obscured the interesting empirical results from the alignment faking paper). In my mind, the spectrum from "almost entirely honest, but occasionally flinching away from aspects of your motivations you're uncomfortable with" to "regularly explicitly thinking about how you're going to fool humans in order to take over the world" is a pretty continuous one. Yet generally people treat "schemer" as a fairly binary classification. To be clear, I'm not confident that even "a spectrum of scheminess" is a good way to think about the concept. There are likely multiple important dimensions that could be disentangled; and eventually I'd like to discover properly scientific theories of concepts like honesty, deception and perhaps even "scheming". Our current lack of such theories shouldn't be a barrier to using those terms at all, but it suggests they should be used with a level of caution that I rarely see.