My own background is in academic social science and national security, for whatever that’s worth
Why should we assume the AI wants to survive? If it does, then what exactly wants to survive?
...
Why should we assume that the AI has boundless, coherent drives?
Are you familiar with the "realist" school of international relations, and in particular their theoretical underpinnings?
If so, I think it'd be helpful to consider Yudkowsky and Soares's arguments in that light. In particular, how closely does the international order for emerging superintelligences look like the anarchic international order for realist states? What are the weaknesses of the realist school of analysis, and do they apply to AIs?
I'll just hop on the bandwagon and say that I'll be posting my thoughts over at inchpin.substack.com!
1
Do you want to come up with some other "obvious exceptions" to your "Nobody says X" claim?
Tbh, I find this comment kinda bizarre.
Popular belief analogizes internet arguments to pig-wrestling: "Never wrestle with a pig because you both get dirty and the pig likes it" But does the pig, in fact, like it? I set out to investigate.
Nobody writes a story whose moral is that you should be selfish and ignore the greater good,
This seems obviously false. Ayn Rand comes to mind as the most iconic example, but eg Camus' The Stranger also had this as a major theme, as does various self-help books. It is also the implicit moral of JJ Thompson's violinist thought-experiment. My impression from reading summaries is that it's also a common theme for early 20th century Japanese novels (though I don't like them so I never read one myself).
I agree if you model people as along some Pareto frontier of perfectly selfish to perfectly (direct) utilitarian, then in no point on that frontier does offsetting ever make sense. However, I think most people have, and endorse, having other moral goals.
For example, a lot of the intuition for offsetting may come from believing you want to be the type of person who internalizes the (large, predictably negative) externalities of your actions, so offsetting comes from your consumption rather than altruism budget.
Though again, I agree that perfect utilitarians, or people aspiring to be perfect utilitarians, should not offset. And this generalizes also to people whose idealized behavior is best described as a linear combination of perfectly utilitarian and perfectly selfish.
I think this post underestimates the value of practicing and thinking in classic style, even if you chose to ultimately discard it, or not write serious posts in that style. Because writing in classic style is so unnatural to most LessWrong dwellers, forcing yourself to write in that way, unironically, and inhabiting the style in its own lights, and especially doing it in a way that doesn't leave you unsatisfied in the end, is a great way to grow and improve as a writer, and understand the strengths and weaknesses of your own style of writing.
I think most people shouldn't write in classic style, for various reasons. But I have a different take here. I think writing in classic style is just very hard for most people, for a number of subtle reasons. A central tenet of classic style is presentation: the writing should look smooth and effortless. But this effortlessness is almost always a mirage, like an Instagram model who spends three hours in front of a mirror to apply the "just woke up", au naturel, "no makeup makeup" look. Of all the (mostly) internet writers I read, only two writers jump out to me as writing in mostly classic style: Paul Graham and Ted Chiang. I don't think it's coincidence that they both are very unprolific, and both talk about how hard it is to write well, and how many edits they go through.
Below is a short coda I wrote in classic style, for a recent article of mine.
Intellectual jokes, at their core, are jokes that teach you new ideas, or help you reconceive existing ideas in a new way.
My favorite forms of intellectual jokes/humor work on multiple levels: They’re accessible to those who just get the surface joke but rewards deeper knowledge with additional layers of meaning. In some of the best examples, the connection to insight is itself subtle, and not highlighted by a direct reference to the relevant academic fields.
There are two failures of attempts to do intellectual humor. They can fail to be intellectual, or they can fail to be funny. Of frequently cited attempts to do “intellectual” humor that fail to be intellectual, there are again two common forms: 1) they are about intellectuals as people, rather than about ideas, or 2) They’re about jargon, not ideas.
In both cases, the joke isn’t intellectual humor so much as “smart people jokes”: the humor rests on stereotypes, in-group solidarity, and the feeling of smartness that you get when you get a joke, but the joke does not actually teach you about new ideas, or help you reconceive of existing ideas in a new way.
Two examples come to mind:
Q: How do you tell if a mathematician is extroverted2?
A: When he’s talking to you, he stares at your shoes!
And
Q: What’s purple and commutes?
A: An Abelian3 grape.
If you were in my undergrad abstract algebra classes, the above jokes were the shit. For 20 year old math majors, they were hilarious. Nonetheless, they are not, by any reasonable definition of the term, intellectual.
Of course, a more common failure mode is that the jokes simply fail to be funny. I will not offer a treatise into what makes a joke funny. All unfunny jokes are alike in their unfunniness, but each funny joke is funny in its own way.
I’m currently drafting a post on different mature writing by first inhabiting the respective styles and then evaluating the pros and cons, especially in the context of internet writing. It’s a pretty hard post to write, and I suspect it’d be a lot less popular in the end than the Chiang review or many LessWrong posts, but I hope it’d be more helpful.
I'm glad you enjoyed it!
I think this discussion is very hard to have because people have substantial variation in how good their base intuitions, including social intuitions, are and also substantial variation in how good their rational minds (S2's) are.
Most good forecasters start off with pretty good base intuitions for probabilities, and forecasting/calibration practice helps refine their intuitions to be even better. Some good forecasters start with terrible probabilistic intuitions and the formal exercises help improve their intuitions. (I suppose it's also theoretically possible for some good forecasters to start with terrible probabilistic intuitions, continue to have terrible probabilistic intuitions, but forecast well because their formal practice and analysis allows them to override their intuitions. I've just never heard about this in practice).
Many mental health conditions come from creating a wrong intuition/prior about the world that rational-adjacent therapy methods like CBT teach you to override. Depression being the most famous/common. When I'm depressed, it's natural to fixate on specific high-noise signals and automatically assume bad interpretations of the evidence (eg someone didn't respond to a text because I've upset them, or they don't like me for other reasons). Presumably there's the inverse problem as well (eg manics have a very positive "rose-colored goggles" prior for everything), but I've never heard of this as a serious problem in practice.
In the realm of social intuitions, I think there's large variation in how good different social intuitions are. Autism in particular is a disorder which has a core manifestation of being bad at parsing social intuitions. I imagine most young autistic people would benefit from learning higher-order rationality and principles counteracting/overriding their own poor social intuitions ("vibes"), and to a lesser extent training/calibrating their poor social intuitions to become better. On the flip side, neurotypical people that are prone to ideological capture should potentially learn to trust their hearts more, and be less inclined to trust higher order ideologies and "rationality" (which often manifests as rationalizations), in favor of their base intuitions.
But even all of these comments are overly high-order glosses. Reverse all advice you hear, etc.