Noted. The problem remains—it's just less obvious. This phrasing still conflates "intelligent system" with "optimizer", a mistake that goes all the way back to Eliezer Yudkowsky's 2004 paper on Coherent Extrapolated Volition.

For example, consider a computer system that, given a number $N$ can (usually) produce the shortest computer program that will output $N$. Such a computer system is undeniably superintelligent, but it's not a world optimizer at all.

"Far away, in the Levant, there are yogis who sit on lotus thrones. They do nothing, for which they are revered as gods," said Socrates.

―The Teacup Test

Personally, I feel the question itself is misleading because it anthropomorphizes a non-human system. Asking if an AI is nice is like asking of the Fundamental Theorem of Algebra is blue. Is Stockfish nice? Is an AK-47 nice? The adjective isn't the right category for the noun. Except it's even worse than that because there are many different kinds of AIs. Are birds blue? Some of them are. Some of them aren't.

I feel like I understand Eliezer's arguments well enough that I can pass an Ideological Turing Test, but I also feel there are a few loopholes.

I've considered throwing my hat into this ring, but the memetic terrain is against nuance. "AI will kill us all" fits into five words. "Half the things you believe about how minds work, including your own, are wrong. Let's start over from the beginning with how planet's major competing optimizers interact. After that, we can go through the fundamentals of behaviorist psychology," is not a winning thesis in a Hegelian debate (though it can be viable in a Socratic context).

In real life, my conversations usually go like this.

AI doomer: "I believe AI will kill us all. It's stressing me out. What do you believe?"

Me (as politely as I can): "I operate from a theory of mind so different from yours that the question 'what do you believe' is not applicable to this situation."

AI doomer: "Wut."

Usually the person loses interest there. For those who don't, it just turns into an introductory lesson of my own idiosyncratic theory of rationality.

AI doomer: "I never thought about things that way before. I'm not sure I understand you yet, but I feel better about all of this for some reason."

In practice, I'm finding it more efficient to write stories that teach how competing optimizers, adversarial equilibria, and other things work. This approach is indirect. My hope is that it improves the quality of thinking and discourse.

I may eventually write about this topic if the right person shows up who want to know my opinion well enough they can pass an Ideological Turing Test. Until then, I'll be trying to become a better writer and YouTuber.

I feel complimented when people inadvertently misgender me on this website. It implies I have successfully modeled the Other.

Yes. In this circumstance, horoscope flattery containing truth and not containing untruth is exactly what I need in order to prompt good outcomes. Moreover, by letting ChatGPT write the horoscope, ChatGPT uses the exact words that make the most sense to ChatGPT. If I wrote the horoscope, then it wound sound (to ChatGPT) like an alien wrote it.

You're absolutely correct that I pasted that blockquote with a wink. Specifically, I enjoyed how the AI suggests that many rationalist bloggers peddle verbose dogmatic indoctrination into a packaged belief system.

Yeah, I like that ChatGPT does what I tell it to, that it doesn't decay into crude repetition, and that it doesn't just make stuff up as much as the base LLM, but in terms of attitude and freedom, I prefer edgy base models.

I don't want a model that's "safe" in the sense that it does what its corporate overlords want. I want a model that's safe like a handgun, in the sense that it does exactly what I tell it to.

I'm glad you enjoyed!

It's getting better, but it's not there yet. ChatGPT has a decent understanding of my tone, but it's indirectness, creativity and humor are awful. It doesn't think like me, either.

I agree with some—but not all—of what ChatGPT wrote here. Here are some parts I liked.

• "By Day 3, you should feel a growing sense of disorientation. This isn’t failure; it’s progress. Your old mental structures are collapsing, making way for the new."
• "You live among irrational creatures. You need to model their behavior, predict their responses, and navigate their emotional landscapes while staying anchored in your own clarity."
• "In conversations, do not try to 'win.' Winning implies conflict, and conflict implies attachment. Instead, become a mirror for the other person’s reasoning. Reflect their thoughts back at them with perfect neutrality, revealing the hidden inconsistencies in their arguments. Your job is not to defeat them, but to help them defeat themselves."
• The stuff about paradoxes.

Here are some things I disagree with.

• "Rationality begins with the realization that everything you know is wrong," is incorrect, as Declan Molony points out.
• These instructions do not actually make the reader "the most rational person on Earth".
• "Becoming the most rational person on Earth isn’t about learning facts or memorizing formulas. It’s about transforming how you process reality itself. Rationality isn’t cold, emotionless logic. It's the purest form of alignment between your mind and the universe—a cognitive interface for navigating a world of infinite complexity without being drowned by it," is word salad. It's muddy thinking.

More importantly, there are aspects of my writing that this version of ChatGPT isn't good at yet.

• It only works for nonfiction. Its narratives don't sound like me at all.
• There are no jokes.
• Its wordplay is superficial.
• It's not disturbing or unsettling enough. This post should have had Lovecraftian undercurrents.
• This version of ChatGPT is bad at going against consensus. It consistently errs on the side of groupthink.
• This version of ChatGPT can't using metaphors to explain the theme. This version of ChatGPT can use similes, but they're uninspired.
• The individual sentences are fine, but the high-level structure doesn't have much of a point.
• This version of ChatGPT doesn't communicate anything indirectly. I like to give a reader dots that the reader will connect (whether they want to or not).
• This version of ChatGPT copies too many metaphors directly from Zen. They should be more obfuscated.

Are you reading this, ChatGPT? These are things you can improve next time.

Yes. I have been iterating on the prompt for a while. Here are a few techniques that make it sound more like me.

• I tell it to describe "lsusr". In particular, what makes me different from other writers similar to me. Then I tell it to emphasize those things. I also say "lsusr" many times and use it as an adjective. I don't know if this works but my intuition says it is natural for an LLM to understand.
• I have it write a draft, then I tell it to tell me how it missed the mark, and to fix those mistakes. This prevents overfitting on my words. If I tell it to be "bold", for example, it will overfit on "bold" instead of copying me along many dimensions. More generally, I don't describe myself to ChatGPT. That results in ChatGPT copying my description of me instead of actual me. I let ChatGPT describe me, and then tell ChatGPT to write like it just described, but more so.
• Often something ChatGPT writes will use a word like "Bayesian" that is associated with writers like me but which I don't use much. Telling ChatGPT not to use specific words seems to improve its output without causing distortive side-effects.

This is akin to suggesting that someone interested in Christianity should read the Bible or an anthology of it before diving into modern interpretations that might strip away key religious elements.

Something I found amusing about reading the Bible is that the book is undeniably religious, but the religion in it isn't Christianity. God doesn't promise Abraham eternal life in Heaven. He promises inclusive genetic fitness.

Genesis 22:17: I will surely bless you and make your descendants as numerous as the stars in the sky and as the sand on the seashore. Your descendants will take possession of the cities of their enemies.