Summary: A cognitively powerful agent is relevant if it is cognitively powerful enough to be a game-changer in the larger dilemma faced by Earth-originating intelligent life. Conversely, an agent is irrelevant if the cognitive problems it can solve or tasks it is authorized to perform don't significantly change the overall situation we face.

Definition.

Intuitively speaking, an AI is 'relevant' to the extent it has a 'yes' answer to the box 'Can we use this AI to produce a benefit that solves the larger dilemma?' in this flowchart, or is part of a larger plan that gets us to the lower green circle without any "Then a miracle occurs" steps.

Examples.

• A hypothetical agent that can bootstrap to nanotechnology by solving the inverse protein folding problem and shut down other AI projects, in a way that can reasonably be known safe enough to authorize by the AI's programmers, would be relevant.
• The ability to prove or disprove the Riemann Hypothesis, but not to do anything else, does not make an agent relevant (unless knowing whether the Riemann Hypothesis is true somehow changes everything for the basic dilemma of AI).
• An oracle that can only output verified HOL proofs is not yet 'relevant' until someone can describe theorems to prove such that firm knowledge of their truth would be a game-changer for the AI situation. (Hypothesizing that someone else will come up with a theorem like that, if you just build the oracle, is a HailMaryStep in the plan.)

Importance.

Many proposals for AI safety, especially advanced safety, so severely restrict the applicability of the AI that the AI is no longer allowed to do anything that seems like it could solve the larger dilemma. (E.g., an oracle that is only allowed to give us binary answers for whether it thinks certain mathematical facts are true, and nobody has yet said how to use this ability to save the world.)

Conversely, proposals to use AIs to do things impactful enough to solve the larger dilemma, generally run smack into all the usual advanced safety problems, especially if the AI must operate in the rich domain of the real world to carry out the task (this tends to require full trust).

Open problem.

Describe a cognitive task or real-world task for a limited AI to carry out, which:

• 1, gets us out of the larger dilemma if solved correctly;
• 2.a, has a real-world solution state that is exceptionally easy to pinpoint using a utility function thereby avoiding some of EdgeInstantiation, UnforeseenMaximums, ContextChange, ProgrammerMaximization, and the other pitfalls of AdvancedSafety, if there is otherwise a trustworthy solution for low-impact AI; or
• 2.b, seems exceptionally doable using a KnownAlgorithmNonreflectiveAgent, thereby averting problems of stable self-modification if a KnownAlgorithmNonreflectiveAgent can otherwise be constructed.