There are fundamental confusions about intelligent agents, that is, about minds that try to make stuff that they want happen. Some believe that working out these fundamental confusions is necessary for AI alignment. Others prefer more prosaic approaches; or something else not mentioned.
Here's some fundamental confusions that agent foundations tries to answer, mostly informed by the post/paper Embedded Agency:
- How can a mind reason about a world too large to consider in its entirety? Perhaps we should look at what sorts of abstractions they'd use; perhaps we need new imprecise probability theories like infra-bayesianism.
- How can a mind reason when it does not know all of the implications of its beliefs (that is, it has logical uncertainty)?
- How can a mind reason about (possibly logical) counterfactuals to make decisions? This includes problems of multi-agent cooperation/competition, especially if the agents have the others source code, or if one is smarter than the other.
- How can a mind reason about itself, or improve itself? Self reasoning seems to cause a lotta problems: you can't have a full picture of yourself because then you'd also have to include the picture in your picture; and it sorta looks like decision theory problems are related. As an example, to improve itself, an agent will want to ensure that its successor will will still be optimizing for the same goals, and do a better job at it - and also create successors that do better than the ones the agent would make. Yet by Vinge's principle, you can't in general know exactly what something smarter than you would think, as then you would just be at least that smart. So you must do so by reasoning abstractly about properties the prospective mind-designs have and will preserve in their successors. This is a simplified version of the problem humanity faces when creating an aligned AI - it is basically the alignment problem for agents with well-defined utility functions that themselves know their utility function.
- What even is an "agent"? What sorts of agents should we expect to be selected for by evolution or gradient descent or any other selection process?
- What are "goals"? How do we formalize a goal that's about the world, instead of our beliefs about the world (or our observations)? Even something as simple as "maximize the number of diamond atoms" has no clear route to formalization! As an example, what happens if you specify the goal in terms of classical physics, and then tomorrow the AI realizes that quantum mechanics is how reality actually works, then it might end up caring about nothing (since "diamond atom" no longer means anything). If you try to solve the problem, you'll have to have a way to make the AI identify its previous goal-concept of "diamond atom" with something in the new world model.
- Preferences are usually expressed in terms of a utility function, as justified by theorems like the VNM theorem. But can we make sensible theories of rational preferences and decisions that relax the axioms? One consideration is to specify that the agents do not get Dutch-booked, but nothing else. This was suggested in logical induction and radical probabilism.
- How should an agent update on the fact that it exists? The anthropic puzzles and attempted solutions seem to get at questions about how beliefs, updating, and logical decision theories should work. For example, some conclude that we should just ditch 'beliefs' as a base concept entirely, only keeping notions of expected utility!
- Is there a natural/simple shape of agent that lets you correct its goals, without trying to stop you or trying to make you correct it or anything weird like that? The shutdown problem is a simple version, dealing with the case of letting the creators do a single bit of correction.
- If we get insight into the preceding, can we make something like AIXI but incorporating the insights? As an example, perhaps the correct description language (like first-order logic) for theories or observables needs to be modified (instead of using turing machines).