A cognitively powerful agent is relevant if it is cognitively powerful enough to be a game-changer in the larger dilemma faced by Earth-originating intelligent life. Conversely, an agent is irrelevant if it is too weak to make much of a difference, or if the cognitive problems it can solve or tasks it is authorized to perform don't significantly change the overall situation we face.
Intuitively speaking, a value-aligned AI is 'relevant' to the extent it has a 'yes' answer to the box 'Can we use this AI to produce a benefit that solves the larger dilemma?' in this flowchart, or is part of a larger plan that gets us to the lower green circle without any "Then a miracle occurs" steps. A cognitively powerful agent is relevant at all if its existence can effectuate good or bad outcomes - e.g., a big neural net is not 'relevant' because it doesn't end the world one way or another.
(A better word than 'relevant' might be helpful here.)
Many proposals for AI safety, especially advanced safety, so severely restrict the applicability of the AI that the AI is no longer allowed to do anything that seems like it could solve the larger dilemma. (E.g., an oracle that is only allowed to give us binary answers for whether it thinks certain mathematical facts are true, and nobody has yet said how to use this ability to save the world.)
Conversely, proposals to use AIs to do things impactful enough to solve the larger dilemma, generally run smack into all the usual advanced safety problems, especially if the AI must operate in the rich domain of the real world to carry out the task (this tends to require full trust).
It is an open problem to propose a relevant, limited AI that would be significantly easier to handle than the general safety problem, while also being useful enough to resolve the larger .
