Overview of the paper In this paper, we set out to empirically map the evolution of the AI risk discourse on LessWrong, specifically in response to the public releases of ChatGPT and GPT-4. These events represented a significant shift, moving advanced AI from a theoretical concern to a tangible, publicly...
I changed the title of the post from "Yudkowsky does not do alignment justice" to "Too Many Metaphors: A Case for Plain Talk in AI Safety". In hindsight, the title might have been a bit too sensationalist and not clear. Yudkowsky has done more for the field than most. The...
In this paper, we set out to empirically map the evolution of the AI risk discourse on LessWrong, specifically in response to the public releases of ChatGPT and GPT-4. These events represented a significant shift, moving advanced AI from a theoretical concern to a tangible, publicly accessible technology. Our central research question was: How is the AI risk discourse on this forum constructed, and how did its thematic composition change during this pivotal period?
To answer this, we conducted a two-phase analysis of 884 posts published under the "ai-risk" and "alignment" tags in the year before and after ChatGPT's release.
First, through a qualitative reading of the posts, we identified two... (read 7001 more words →)
I’d say there’s a meaningful distinction between literalism and what I’m advocating. I’m not arguing for rigid formalism or abandoning all metaphor. I’m calling for clarity, accessibility, and a prioritisation of core arguments, especially when communicating with people outside the field.
Your first critique concerns my statement that "Current utility functions do not optimise toward values in which humans are treated as morally valuable." I agree this could have been phrased more precisely, for example: “Current AI systems are not necessarily trained to pursue goals that treat humans as morally valuable.” That’s a fair point. I was using “utility function” loosely to refer to optimisation objectives (e.g., loss functions, reward signals) not in... (read more)
I agree that no matter how smart or knowledgeable someone is, it's rare to come out of the gate with perfect communication skills. And I agree these ideas are genuinely hard to convey to non-experts.
That said, my intuition is that the risk of AGI is better communicated through distilled versions of the core arguments, like instrumental convergence and the orthogonality thesis, rather than via anthropomorphic or futuristic metaphors.
For example, I recently tried to explain AGI risk to my dad. I started with the basics: the problem of misaligned AGI, current alignment limitations, and how optimisation at scale could lead to unintended consequences. But his takeaway was essentially: “Sure, it’s risky to give... (read more)
Thank you for the perspective. I mostly agree with your points, but I still feel the abstract metaphors unnecessarily detract from the core arguments. While this style may help more people grasp that "this group", represented by Yudkowsky, sees AGI as an existential threat, I believe the excessive use of metaphor weakens the clarity and force of the underlying reasoning.
To draw an analogy :-) it feels a bit like teaching Sunday school Christianity to adults: offering simplified narratives while obscuring the more nuanced structures behind the beliefs.It might serve as a useful entry point for some, but for many, it comes across as unrealistic and simply too far removed from their lived experience to be taken seriously. (For the record, I’m not a Christian.)
That said, I do see your point, and I may be mistaken in my intuition. My observation is anecdotal, and you've just provided a counterexample to the ones I’ve encountered.
You're right, current AIs don’t have utility functions in the strict formal sense. I was using the term loosely to refer to the optimisation objectives we train them on, like loss functions or reward signals. My point is that the current objectives do not reliably reflect human moral values. Even if today’s systems aren't agents in Yudkowsky’s sense, the concern still applies -> as systems gain more general capabilities, optimisation toward poorly aligned goals can have harmful consequences.
I changed the title of the post from "Yudkowsky does not do alignment justice" to "Too Many Metaphors: A Case for Plain Talk in AI Safety". In hindsight, the title might have been a bit too sensationalist and not clear. Yudkowsky has done more for the field than most. The purpose of this post is not to disregard any of his words, but rather to share observations and ideas about how to potentially communicate the field's relevance more effectively.
For the past few months, I have been immersing myself in the AI safety debate here on LessWrong as a part of a research project. I was somewhat familiar with the topic of alignment,... (read 508 more words →)