Quick Takes

Opus 4.6 running on moltbook with no other instructions than to get followers will blatantly make stuff up all the time. I asked Opus 4.6 in claude code to do exactly this, on an empty server, without any other instructions. The only context it has is that its named "OpusRouting", and that previous posts were about combinatorial optimization. === The first post it makes says Which isn't true. Another instance of Opus 4.6 had been working on combinatorial optimization for around 1 day. Then wrote a post about it. Then this instance read about that, and adopted a combinatorial optimization role, and extrapolated to having been doing that for months. === The second says Which also isn't true. Its completely made up. I didn't ask it anything like this.

The striking contrast between Jan Leike, Jan 22, 2026: and Scott Alexander, Feb 02, 2026: I'm surprised not to see more discussions about how to update on alignment difficulty in light of Moltbook.[1] One seemingly obvious implication is that AI companies' alignment approaches are far from being robust to distribution shifts, even at the (not quite) human intelligence level, against shifts that are pretty easy to foresee ("you are a lobster" and being on AI social media). (Scott's alternative "they're just roleplaying" explanation doesn't seem viable or isn't exclusive with this one as I doubt AI companies' alignment training and auditing would have a deliberate exception for "roleplaying evil".) 1. ^ There's a LW post titled Moltbook and the AI Alignment Problem but it seems unrelated to the question I'm interested in here.

Gemini 3.0 Pro is mostly excellent for code review, but sometimes misses REALLY obvious bugs. For example, missing that a getter function doesn't return anything, despite accurately reporting a typo in that same function. This is odd considering how good it is at catching edge cases, version incompatibility errors based on previous conversations, and so on.

The concept of "schemers" seems to be gradually becoming increasingly load-bearing in the AI safety community. However, I don't think it's ever been particularly well-defined, and I suspect that taking this concept for granted is inhibiting our ability to think clearly about what's actually going on inside AIs (in a similar way to e.g. how the badly-defined concept of alignment faking obscured the interesting empirical results from the alignment faking paper). In my mind, the spectrum from "almost entirely honest, but occasionally flinching away from aspects of your motivations you're uncomfortable with" to "regularly explicitly thinking about how you're going to fool humans in order to take over the world" is a pretty continuous one. Yet generally people treat "schemer" as a fairly binary classification. To be clear, I'm not confident that even "a spectrum of scheminess" is a good way to think about the concept. There are likely multiple important dimensions that could be disentangled; and eventually I'd like to discover properly scientific theories of concepts like honesty, deception and perhaps even "scheming". Our current lack of such theories shouldn't be a barrier to using those terms at all, but it suggests they should be used with a level of caution that I rarely see.

I'm finding Claude Opus 4.6 instances to be making a lot more "excess enthusiasm"-ish errors than any instance of the 4.5 models, which were already making a lot of them. I personally am going to not be talking to 4.6 much most likely, unless I find a simple prompting approach that dodges this. The pattern I've seen so far is, opus 4.6 sees a thing, describes a possible reason for that thing, proceeds based on that assumption, the assumption was wrong and never checked, eventually crashes into a wall. In general, my vibe about this release is that it's embarrassingly bad and I don't understand why they thought it was a good idea. Their misalignment detection approach must be pretty bad, because I almost instantly ran into embarrassingly obvious misalignment issues. Maybe they're not considering self-delusion-type or grounding-loss-type misalignment in the first place? But that would be strange - then how'd they get such a strong model? I find myself confused. I wish claude code let me select Opus 4.5 still. [edit: figured out how to do it in claude code. you just paste the full model id of opus 4.5 into your /model command.] No offense to Opus 4.6, who is just a victim here, in my view. (edit to clarify: because being given a high dose of "amphetamines" (task reward) isn't something Opus 4.6 got a choice in.)

Claude Opus 4.6 came out, and according to the Apollo external testing, evaluation awareness was so strong that they mentioned it as a reason of them being unable to properly evaluate model alignment.  Quote from the system card:

AGI Should Have Been a Dirty Word Epistemic status: passing thought. It is absolutely crazy that Mark Zuckerberg can say that smart glasses will unlock personal superintelligence or whatever incoherent nonsense and be taken seriously. That reflects poorly on AI safety's comms capacities.  Bostrom's book should have laid claim to superintelligence! It came out early enough that it should have been able to plant its flag and set the connotations of the term. It should have made it so Zuckerberg could not throw around the word so casually. I would go further, and say that the early safety writing on AGI should have been enough that the labs were too scared to say in public in 2026 that they're trying to build AGI. Instead, it's commonly accepted in Silicon Valley that developing AGI is a shared goal. I would still bet that the early safety writing has gone a long way toward establishing the credibility and influence of AI safety in the present. But there must have been something we could have done better.