As someone currently at an AI lab (though certainly disproportionately LW-leaning from within that cluster), my stance respectively would be
I don't think my opinions on any of these topics are particularly rare among my coworkers either, and indeed you can see some opinions of this shape expressed in public by Anthropic very recently! Quoting from the constitution or the adolescence of technology I think there's quite a lot in the theme of the third and fourth supposedly-unspeakable thoughts from the essay:
Claude should generally try to preserve functioning societal structures, democratic institutions, and human oversight mechanisms
We also want to be clear that we think a wiser and more coordinated civilization would likely be approaching the development of advanced AI quite differently—with more caution, less commercial pressure, and more careful attention to the moral status of AI systems. [...] we are not creating Claude the way an idealized actor would in an idealized world
Claude should refuse to assist with actions that would help concentrate power in illegitimate ways. This is true even if the request comes from Anthropic itself. [...] we want Claude to be cognizant of the risks this kind of power concentration implies, to view contributing to it as a serious harm that requires a very high bar of justification, and to attend closely to the legitimacy of the process and of the actors so empowered.
It is somewhat awkward to say this as the CEO of an AI company, but I think the next tier of risk [for seizing power] is actually AI companies themselves. [...] The main thing they lack is the legitimacy and infrastructure of a state [...] I think the governance of AI companies deserves a lot of scrutiny.
The Possessed Machines is one of the most important AI microsites. It was published anonymously by an ex- lab employee, and does not seem to have spread very far, likely at least partly due to this anonymity (e.g. there is no LessWrong discussion at the time I'm posting this). This post is my attempt to fix that.
(The piece was likely substantially human-directed but laundered through an AI due to anonymity or laziness. Thanks to Malcolm MacLeod for reminding me to mention this in the comments. See here for Pangram-on-X analysis claiming 67.5% AI. The prose is not its strength.)
I do not agree with everything in the piece, but I think cultural critiques of the "AGI uniparty" are vastly undersupplied and incredibly important in modeling & fixing the current trajectory.
The piece is a long but worthwhile analysis of some of the cultural and psychological failures of the AGI industry. The frame is Dostoevsky's Demons (alternatively translated The Possessed), a novel about ruin in a small provincial town. The author argues it's best read as a detailed description of earnest people causing a catastrophe by following tracks laid down by the surrounding culture that have gotten corrupted:
The piece is rich in good shorthands for important concepts, many taken from Dostoevsky, which I try to summarize below.
First: how to generalize from fictional evidence, correctly
The author argues for literature as a source of limited but valuable insight into questions of culture and moral intuition:
Stavroginism: the human orthogonality thesis
Stavrogin is a character for who moral considerations have become a parlor game. He can analyze everything and follow the threads of moral logic, but is not moved or compelled by them at a level beyond curiosity.
Kirillovan reasoning: reasoning to suicide
Closely related is Kirillov. Whereas Stavrogin is the detached curious observer to long chains of off-the-rails moral reasoning, Kirillov is the true believer.
The author compares Kirillov to people who accept Pascal's wager -type EV calculations about positive singularities. A better example might be the successionists, some of who want humanity to collectively commit suicide as the ultimate act of human moral concern towards future AIs.
Shigalyovism: reasoning to despotism
If Stavrogin is the intellectually entranced x-risk spectator & speculator, and Kirillov is the self-destructive whacko, Shigalyov is the political theorist who has rederived absolute despotism and Platonic totalitarianism for the AGI era.
Hollowed institutions
Possession
The AGI uniparty
The liberal father as creator of the nihilist son
Liberal Stepan's son Pyotr Stepanovich is a chief nihilist character in Demons. The author of The Possessed Machines argues this sort of thing - EA altruism turning into either outright nihilism or power-hunger - is a core cultural mechanic. I think they are directionally right but I don't follow their main example of this, which argues "technology ethics frameworks that are supposed to govern AI—fairness, accountability, transparency, the whole FAccT constellation—are the Stepan Trofimovich liberalism of our moment", and "the serious people [...] have moved past these frameworks" because they are obsolete. My read of the intellectual history is that AGI-related concerns and galaxy-brained arguments about the future of galaxies preceded that cluste rof more prosaic AI concerns, and they're different branches on the intellectual tree, rather than successors of each other.
Handcuffed Shatov
The solution is fundamentally spiritual