The Possessed Machines is one of the most important AI microsites. It was published anonymously by an ex- lab employee, and does not seem to have spread very far, likely at least partly due to this anonymity (e.g. there is no LessWrong discussion at the time I'm posting this). This post is my attempt to fix that.

(The piece was likely substantially human-directed but laundered through an AI due to anonymity or laziness. Thanks to Malcolm MacLeod for reminding me to mention this in the comments. See here for Pangram-on-X analysis claiming 67.5% AI. The prose is not its strength.)

I do not agree with everything in the piece, but I think cultural critiques of the "AGI uniparty" are vastly undersupplied and incredibly important in modeling & fixing the current trajectory.

The piece is a long but worthwhile analysis of some of the cultural and psychological failures of the AGI industry. The frame is Dostoevsky's Demons (alternatively translated The Possessed), a novel about ruin in a small provincial town. The author argues it's best read as a detailed description of earnest people causing a catastrophe by following tracks laid down by the surrounding culture that have gotten corrupted:

What I know is that Dostoevsky, looking at his own time, saw something true about how intelligent societies destroy themselves. He saw that the destruction comes from the best as well as the worst, from the idealists as well as the cynics, from the people who believe they are saving humanity as well as those who want to burn it down.

The piece is rich in good shorthands for important concepts, many taken from Dostoevsky, which I try to summarize below.

First: how to generalize from fictional evidence, correctly

The author argues for literature as a source of limited but valuable insight into questions of culture and moral intuition:

Literature cannot tell us what to do. It cannot provide policy prescriptions or technical solutions. It cannot predict the future or settle empirical questions. The person who reads Dostoevsky looking for an alignment technique will be disappointed.
What literature can do is reshape perception. It can make visible patterns that were invisible, make felt truths that were merely known, make urgent realities that were abstract. It can serve as a kind of training data for moral intuition—presenting scenarios that expand the range of situations one has "experien

...

annaflaviamanda@hotmail.com

annaflaviamanda@hotmail.com