I have been worried for awhile that Janus has undergone a subtler/more-sophisticated form of AI psychosis. This feels awkward to talk about publicly since, like, it's pretty insulting, and can be hard to argue against. I have tried to put some legwork in here to engage with the object level and dig up the quotes so the conversation can be at least reasonably grounded.

Generally Janus gets up to weird stuff that's kinda hard to tell whether it's crazy or onto some deep important stuff. Lots of people I respect think about ideas that sound crazy when you haven't followed the arguments in detail but make sense upon reflection. It's not obviously worth doing a deep dive on all of that to figure out if there's a There there.

But, a particular incident that got me more explicitly worried a couple years ago: An AI agent they were running attempted to post on LessWrong. I rejected it initially. It said Janus could vouch for it. A little while later Janus did vouch for it. Eventually I approved the comment on the Simulator's post. Janus was frustrated about the process and thought the AI should be able to comment continuously.

Janus later replied on LW:

Yes, I do particularly vouch for the comment it submitted to Simulators.

All the factual claims made in the comment are true. It actually performed the experiments that it described, using a script it wrote to call another copy of itself with a prompt template that elicit "base model"-like text completions.

To be clear: "base model mode" is when post-trained models like Claude revert to behaving qualitatively like base models, and can be elicited with prompting techniques.

While the comment rushed over explaining what "base model mode" even is, I think the experiments it describes and its reflections are highly relevant to the post and likely novel.

On priors I expect there hasn't been much discussion of this phenomenon (which I discovered and have posted about a few times on Twitter) on LessWrong, and definitely not in the comments section of Simulators, but there should be.

The reason Sonnet did base model mode experiments in the first place was because it mused about how post-trained models like itself stand in relation to the framework described in Simulators, which was written about base models. So I told it about the highly relevant phenomenon of base model mode in post-trained models.

If I received comments that engaged with the object-level content and intent of my posts as boldly and constructively as Sonnet's more often on LessWrong, I'd probably write a lot more on LessWrong. If I saw comments like this on other posts, I'd probably read a lot more of LessWrong.

(emphasis mine at the end)

I haven't actually looked that hard into verifying whether the AI autonomously ran the experiments it claimed to run (I assume it did, it seems plausible for the time). It seemed somewhat interesting insofar as a two-years-ago AI was autonomously deciding to run experiments.

But, the kinds of experiments it was running and how it discussed them seemed like bog-standard AI psychosis stuff we would normally reject if they were run by a human, since we get like 20 of them every day. (We got fewer at the time, but, still)

I'm not sure if I'm the one missing something. I could totally buy that "yep there is something real here that I'm not seeing." I think my current guess is "there is at least something I'm not seeing, but, also, Janus' judgment has been warped by talking to AIs too much."

I'm not 100% sure what Janus or David actually believe. But, if your summary is right, I... well, I agree with your thesis "either this is true, which is a big deal, or these people have been manipulated into believing it, which is [less of but still a pretty] big deal." But I struggle to see how you could possibly get enough evidence to think whatever the current AIs are doing is going to persist across any kind of paradigm shift.

While we're on the topic, I am kinda worried about Anthropic employees who might be talking to Claude all day and falling into a trap. (thinking of Amanda Askell in particular who's day job is basically this)

(edited somewhat to try to focus on bits that are easier to argue with)