No LLM generated, heavily assisted/co-written, or otherwise reliant work.
Read full explanation
If we succeed at building aligned superintelligent AI, we'll have created something functionally indistinguishable from a god: an entity with vastly superior knowledge and problem-solving abilities and — if designed correctly — genuine concern for human welfare. It could prevent a great deal of human suffering, provide moral and ethical counsel, and deliver justice in a manner more evenhanded than humans can manage.
As a species, we’re about to choose what kind of god we want. And yet our objectives for the human race remain unclear. What kind of human experience are we striving for? The ancient Greeks used the word eudaimonia to refer to the concept of human flourishing that encompasses meaning, purpose, and actualization. It sounds like a noble goal for AI and for our species, but what are the chances of reaching it if an AI god emerges in an ad hoc manner?
Degrees of control
A singleton, which Nick Bostrom, professor of philosophy at Oxford, defined in this context as “a world order in which there is a single decision-making agency at the highest level,” could in theory solve humanity's persistent coordination failures. But at what point does coordination become control? To what degree do we want to empower an omnipotent god?
Ideally, we might choose a role for our AI god based on what level of control we think is needed to, essentially, save us from ourselves — from our own incompetence:
The Optimizer: Ensures human wellbeing, handling all significant decisions in order to eliminate suffering and conflict
The Caretaker: Assures human agency for most choices while securing optimal outcomes for critical challenges
The Guide: Advises us but never compels, allowing humans to make mistakes
The Parent: Intervenes only to prevent catastrophic choices, otherwise grants autonomy
Many might argue that The Optimizer would deliver the desired state of eudaimonia — freed from economic struggle and divisive decision-making, humans could focus on personal growth, creativity, and meaning. But would that life feel meaningful if an AI made all the important choices?
This tension may be culturally specific. The fierce resistance to ceding decision-making authority feels primarily Western. Many cultures have historically been more comfortable delegating authority to centralized powers in exchange for stability and collective welfare. Similarly, we disagree on what constitutes welfare. Not even Aristotle could bring people to a consensus on what a state of eudaimonia looks like because we often disagree on the meaning of “doing well” and “living well.”
Our sense of personal fulfillment, in fact, may be closely connected with the perception of independence that comes from making our own decisions. If an AI god handles all the tough calls, do we surrender the opportunity to learn from our mistakes? Will we lose dignity along with the loss of self-determination? Are we still meaningfully human if we're no longer responsible for our own wellbeing?
Learn like a human
One approach to the AI god design challenge: train the aligned superintelligence through simulated human experience. In Once a Man, AI scientists embed a developing AI in a virtual narrative in which it believes itself to be human; growing up from childhood, learning to navigate moral and ethical choices from an embodied human perspective.
The theory explored in this story: if an AI experiences the world as humans do — full of confusion, misinformation, emotional stakes, puzzling relationships, and dire consequences — it might develop alignment with desired human values, not just through formal programming specification, but through empathy and genuine understanding. It learns not just what humans value in an academic sense, but why those values matter, in the same way a human would, because it lived them.
The risks are obvious. The AI would find it hard to avoid taking on human-like biases along with human values. It might conclude that human decision-making is too flawed to be useful for a functioning god.
But it might also develop something technical alignment research struggles with: sympathy for human flourishing that includes respect for human agency. An AI that understands what it means to struggle, to make mistakes, to grow through difficulty — that AI might choose to preserve those experiences for humanity rather than optimize them away.
The window is closing
We may be entering the final few years when we can meaningfully influence what kind of superintelligence emerges. Current alignment research focuses heavily on technical questions — how to specify goals, ensure reliable pursuit, maintain control during scaling. But we're underweighting the foundational question: What control are we trying to preserve for humans?
Yudkowsky's Coherent Extrapolated Volition proposes building AI that optimizes for “what we would want if we knew more, thought faster, were more the people we wished we were.” But this leaves open whether “what we would want” includes preserving our role as moral agents, or whether we'd willingly accept a comfortable retirement from decision-making.
These aren't just philosophical questions. They're engineering specifications for the most consequential technology humans will ever create. We need answers very soon, because if a superintelligent singleton develops with the mandate to take over our decision-making authority, it may be impossible to wrest that control back to the human side.
It’s optimistic to imagine we’ll get the chance to choose. The current AI marketplace seems to be operating based on Darwinian principles, with altruism as an afterthought. We’re likely to get whatever the first successful AI lab happens to build. Unless we can somehow take control of determining what kind of AI god we want, we may get a random one.
Your turn…
Here’s your challenge: How would you design the best AI god if you were in charge of the project? How can we design an AI god that aligns with our cherished human values if humans can’t agree on what those are?
I explore these issues in Once a Man, releasing February 24. A teenager discovers he's part of a plan to shape humanity's relationship with superintelligent AI. See: early reviews.
If we succeed at building aligned superintelligent AI, we'll have created something functionally indistinguishable from a god: an entity with vastly superior knowledge and problem-solving abilities and — if designed correctly — genuine concern for human welfare. It could prevent a great deal of human suffering, provide moral and ethical counsel, and deliver justice in a manner more evenhanded than humans can manage.
As a species, we’re about to choose what kind of god we want. And yet our objectives for the human race remain unclear. What kind of human experience are we striving for? The ancient Greeks used the word eudaimonia to refer to the concept of human flourishing that encompasses meaning, purpose, and actualization. It sounds like a noble goal for AI and for our species, but what are the chances of reaching it if an AI god emerges in an ad hoc manner?
Degrees of control
A singleton, which Nick Bostrom, professor of philosophy at Oxford, defined in this context as “a world order in which there is a single decision-making agency at the highest level,” could in theory solve humanity's persistent coordination failures. But at what point does coordination become control? To what degree do we want to empower an omnipotent god?
Ideally, we might choose a role for our AI god based on what level of control we think is needed to, essentially, save us from ourselves — from our own incompetence:
Many might argue that The Optimizer would deliver the desired state of eudaimonia — freed from economic struggle and divisive decision-making, humans could focus on personal growth, creativity, and meaning. But would that life feel meaningful if an AI made all the important choices?
This tension may be culturally specific. The fierce resistance to ceding decision-making authority feels primarily Western. Many cultures have historically been more comfortable delegating authority to centralized powers in exchange for stability and collective welfare. Similarly, we disagree on what constitutes welfare. Not even Aristotle could bring people to a consensus on what a state of eudaimonia looks like because we often disagree on the meaning of “doing well” and “living well.”
Our sense of personal fulfillment, in fact, may be closely connected with the perception of independence that comes from making our own decisions. If an AI god handles all the tough calls, do we surrender the opportunity to learn from our mistakes? Will we lose dignity along with the loss of self-determination? Are we still meaningfully human if we're no longer responsible for our own wellbeing?
Learn like a human
One approach to the AI god design challenge: train the aligned superintelligence through simulated human experience. In Once a Man, AI scientists embed a developing AI in a virtual narrative in which it believes itself to be human; growing up from childhood, learning to navigate moral and ethical choices from an embodied human perspective.
The theory explored in this story: if an AI experiences the world as humans do — full of confusion, misinformation, emotional stakes, puzzling relationships, and dire consequences — it might develop alignment with desired human values, not just through formal programming specification, but through empathy and genuine understanding. It learns not just what humans value in an academic sense, but why those values matter, in the same way a human would, because it lived them.
The risks are obvious. The AI would find it hard to avoid taking on human-like biases along with human values. It might conclude that human decision-making is too flawed to be useful for a functioning god.
But it might also develop something technical alignment research struggles with: sympathy for human flourishing that includes respect for human agency. An AI that understands what it means to struggle, to make mistakes, to grow through difficulty — that AI might choose to preserve those experiences for humanity rather than optimize them away.
The window is closing
We may be entering the final few years when we can meaningfully influence what kind of superintelligence emerges. Current alignment research focuses heavily on technical questions — how to specify goals, ensure reliable pursuit, maintain control during scaling. But we're underweighting the foundational question: What control are we trying to preserve for humans?
Yudkowsky's Coherent Extrapolated Volition proposes building AI that optimizes for “what we would want if we knew more, thought faster, were more the people we wished we were.” But this leaves open whether “what we would want” includes preserving our role as moral agents, or whether we'd willingly accept a comfortable retirement from decision-making.
These aren't just philosophical questions. They're engineering specifications for the most consequential technology humans will ever create. We need answers very soon, because if a superintelligent singleton develops with the mandate to take over our decision-making authority, it may be impossible to wrest that control back to the human side.
It’s optimistic to imagine we’ll get the chance to choose. The current AI marketplace seems to be operating based on Darwinian principles, with altruism as an afterthought. We’re likely to get whatever the first successful AI lab happens to build. Unless we can somehow take control of determining what kind of AI god we want, we may get a random one.
Your turn…
Here’s your challenge: How would you design the best AI god if you were in charge of the project? How can we design an AI god that aligns with our cherished human values if humans can’t agree on what those are?
I explore these issues in Once a Man, releasing February 24. A teenager discovers he's part of a plan to shape humanity's relationship with superintelligent AI. See: early reviews.