A task-based AGI or "genie" is an AGI intended to follow a series of human orders, with these orders each being of limited scope - "satisficing" in the sense that they can be accomplished using bounded amounts of effort and resources (as opposed to the goals being more and more fulfillable using more and more effort).

In Bostrom's typology, a Genie contrasts with a Sovereign AGI that acts autonomously, in the pursuit of long-term real-world goals.

Building a safe Task AGI might be easier than building a safe Sovereign for the following reason:

• The AGI can query the user before and during Task performance (if an ambiguous situation arises, and is successfully identified as ambiguous).
• The tasks are meant to be limited in scope - to be accomplishable, once and for all, within a limited space and time, using some limited amount of effort which is then done. A Sovereign pursuing open-ended real-word goals would presumably be in a situation where those goals become more and more accomplished (the outcome is of higher and higher utility) as more and more resources and efforts are expended.
• A Genie can potentially be limited in various ways, since a Task AGI doesn't need to be as powerful as possible in order to accomplish its limited-scope Tasks. A Sovereign would presumably engage in all-out self-improvement.

The advantage of a Genie is that it is plausibly easier, if not easy, to solve the safety problems - it might step down the problem from "impossibly difficult" to "insanely difficult", while still maintaining enough power in the AI to perform pivotal acts.

The obvious disadvantage of a Genie is moral hazard - it may tempt the users in ways that a Sovereign would not. A Sovereign has moral hazard chiefly during the development phase, when the programmers and users are perhaps not yet in a position of special relative power. A Genie has ongoing moral hazard as it is used.

The problem of making a safe genie invokes numerous subtopics such as low impact, mild optimization, and conservatism as well as numerous standard AGI safety problems like reflective stability and safe identification of intended goals.

Eliezer Yudkowsky has suggested that people only confront many important problems in value alignment when they are thinking about Sovereigns, but that at the same time, Sovereigns may be impossibly hard in practice. Yudkowsky advocates that people think about Sovereigns first and list out all the associated issues before stepping down their thinking to Genies, because thinking about Genies may result in premature pruning, while thinking about Sovereigns is more likely to generate a complete list of problems that can then be checked against particular Genie approaches to see if those problems have become any easier.

Three distinguished subtypes of Genie are these:

• Oracles, an AI that is intended to only answer questions, possibly from some restricted question set.
• Known-algorithm AIs, which are not self-modifying or very weakly self-modifying, such that their algorithms and representations are mostly known and mostly stable.
• Behaviorist Genies, which are meant to not model human minds or model them in only very limited ways, while having great material understanding (e.g., potentially the ability to invent and deploy nanotechnology).

Subproblems

(See here for a separate page on open problems in Task AGI safety that might be ready for current research.)

Some further problems beyond those appearing in the page above are:

• Oracle utility functions (that make the Oracle not wish to leave its box or optimize its programmers)
• Effable optimization (the opposite of cognitive uncontainability)
• Online checkability
  • Explaining things to programmers without putting the programmers inside an argmax for how well you are 'explaining' things to them
• Transparency
• Do What I Mean