We can see the place of AI alignment in the larger scheme by considering its parent problem, its sibling problems, and examples of its child problems, grandchildren, and siblings.problems.
Clickbait: How can Earth-originating intelligent life achieve most of its potential value, whether by AI or otherwise?
Summary: The value achievement dilemma is the general, broad challenge faced by Earth-originating intelligent life in steering our cosmic endowment into a state of high value - successfully turning the stars into a happy civilization. We face potential existential catastrophes (resulting in our extermination or the corruption of the cosmic endowment) such as the possibility of military use of nanotechnology, non-value-aligned AIs, or insane smart uploads. A strategy is relevant to value achievement only if success is a game-changer for the overall dilemma humanity faces. E.g., [value-aligned] powerful AIs or intelligence-enhanced humans both seem to qualify as strategically relevant, but an AI restricted to only prove theorems in Zermelo-Frankel set theory has no obvious game-changing use.
The point of value alignment can be seen in the suggestion by Eliezer YudkowskyTodo
On the other hand, consider someone who proposes that "Rather than building AI, we should build Oracle AIs that just answer questions," and who then, after further exposure to the concept of the [AI-AI-Box Experiment]Experiment and cognitive uncontainability, further narrows their specification to say that an Oracle running in three layers of sandboxed simulation must output only formal proofs of given theorems in Zermelo-Fraenkel set theory, and a heavily sandboxed and provably correct verifier will look over this output proof and signal 1 if it proves the target theorem and 0 otherwise, at some fixed time to avoid timing attacks. This is meant to ensure that the Oracle can only influence a single binary bit of our world in any predictable way.
Summary: The value achievement dilemma is the general, broad challenge faced by Earth-originating intelligent life in steering our cosmic endowment into a state of high value - successfully turning the stars into a happy civilization. We face problemspotential existential catastrophes (resulting in our extermination or the corruption of the cosmic endowment) such as the possibility of military technology oruse of nanotechnology, non-value-aligned AIs.AIs, or insane smart uploads. A strategy is relevant to value achievement only if success is a game-changer for the overall dilemma humanity faces. E.g., [value-aligned] powerful AIs or [intelligence-intelligence-enhanced humans]humans both seem to qualify as strategically relevant, but an AI restricted to only prove theorems in Zermelo-Frankel set theory has no obvious game-changing use.
The point of value alignment can be seen in the suggestion by Yudkowsky Todofind a citation - CFAI? PtS? and Bostrom
that we can see Earth-originating intelligent life as having two possible stable states, superintelligence and extinction. If intelligent life goes extinct, especially if it damages or destroys the ecosphere, it will probably stay extinct. If it becomes superintelligent, it will presumably expand through the universe and stay superintelligent for as long as physically possible. Eventually, our civilization is bound to wander into one of these attractors or another. Furthermore, by the , any sufficiently advanced cognitive agent will be stable in its motivations or . So if and when life wanders into the superintelligence attractor, it will either end up in a stable state of e.g. [fun-loving] or the reflective equilibrium of its creators' civilization and hence achieving lots of , or it will go on maximizing paperclips forever.On the other hand, consider someone who proposes that "Rather than building AI, we should build AIs that just answer questions," and who then, after further exposure to the concept of the [AI-Box Experiment] and cognitive uncontainability, further narrows their specification to say that an Oracle running in three layers of sandboxed simulation must output only formal proofs of given theorems in Zermelo-Fraenkel set theory, and a heavily sandboxed and provably correct verifier will look over this output proof and signal 1 if it proves the target theorem and 0 otherwise, at some fixed time to avoid timing attacks. This is meant to ensure that the Oracle can only influence a single binary bit of our world in any predictable way.
An obvious target strategy for a limited Genie is to ask it to create nanotechnology and use that tech to gently shut down all other AI projects, e.g. by copying the software and then sealing the hardware. But this putative Genie can't model agents, so it may not be able to identify only potential AI projects. We could use such a Genie to build nanotechnology and then heal the sick or create lots of food for the hungry, but while this is a conventional good, we haven't yet identified any path to victory that stops other projects from building AI, or lets us create intelligence-enhanced humans (unless this can be done without modeling human minds or other agents at all).
The value achievement dilemma is a way of framing the valueAI alignment problem in a larger context. This both emphasizes that there might be possible solutions besides AI,AI; and also emphasizes that such solutions must meet a high bar of potency or efficacy in order to resolve our basic dilemmasdilemmas, the way that a sufficiently value-aligned and cognitively powerful AI could resolve our basic dilemmas. Or at least change the nature of the gameboard, the way that a Task AGI could take actions to prevent destruction by later AGI projects, even if is only narrowly value-aligned and cannot solve the whole problem.
The point of value alignmentconsidering posthuman scenarios in the long run, and not just an immediate Task AGI as band-aid, can be seen in the suggestion by Eliezer Yudkowsky
Furthermore, by the Gandhigeneric preference stability argument, any sufficiently advanced cognitive agent willis very likely to be stable in its motivations or meta-preference framework. So if and when life wanders into the superintelligence attractor, it will either end up in a stable state of e.g. fun-loving or the reflective equilibrium of its creators' civilization and hence achieving lots of value, or ita misaligned AI will go on maximizing paperclips forever.
Other positive events or capacities seem like they could potentially prompt entry into the high-value-achieving superintelligence attractor:
On the other hand, consider someone who proposes that "Rather than building AI, we should build Oracle AIs that just answer questions," and who then, after further exposure to the concept of the AI-Box Experiment and cognitive uncontainability, further narrows their specification to say that an Oracle running in three layers of sandboxed simulation must output only formal proofs of given theorems in Zermelo-Fraenkel set theory,theory, and a heavily sandboxed and provably correct verifier will look over this output proof and signal 1 if it proves the target theorem and 0 otherwise, at some fixed time to avoid timing attacks....
On the other hand, suppose someone proposes, as an intended Relevant Limited AI, a [non-non-self-modifying] modifyingGenie agent that is only allowed to model non-cognitive material systems and has been constructed and injuncted not to model other agents, whether those agents are human or other hypothetical minds.
Clickbait: How can Earth-originating intelligent life achieve most of its potential value, whether by AI or otherwise?
Summary: The value achievement dilemma is the general, broad challenge faced by Earth-originating intelligent life in configuringsteering our endowment into a state of high value. Creating - successfully turning the stars into a happy civilization. We face problems such as the possibility of military technology or non-value-aligned AIs. A strategy is relevant to value achievement only if success is a game-changer for the overall dilemma humanity faces. E.g., [value-aligned] agentAIs wouldor [intelligence-enhanced humans] both seem to qualify as strategically relevant, but an AI restricted to only prove theorems in Zermelo-Frankel set theory has no obvious game-changing use.
The value achievement dilemma is a way of framing the value alignment problem in a larger context. This both emphasizes that there might be possible solutions besides AI, and emphasizes that such solutions must meet a high bar of potency or efficacy in order to resolve our basic dilemmas the way that a sufficiently value-aligned and cognitively powerful AI could resolve our basic dilemmas.
The point of value alignment can be seen in the suggestion by Yudkowsky Todoa citation - CFAI? PtS? and Bostrom
Among the dilemmas we face in getting into the high-value-achieving attractor, rather than the extinction attractor or the equivalence class of paperclip maximizers, are:
Other events or capacities seem like they could prompt entry into the high-value-achieving superintelligence attractor:
...The value achievement dilemma is the general, broad challenge faced by Earth-originating intelligent life in configuring our endowment into a state of high Valuevalue. Creating a [value-aligned] agent would solve the broader dilemma, but so, conceivably, might intelligence-enhanced humans or other possible solutions.