Epistemic Status
Unsure[1], partially noticing my own confusion. Hoping Cunningham's Law can help resolve it.
Confusions About Arguments From Expected Utility Maximisation
Some MIRI people (e.g. Rob Bensinger) still highlight EU maximisers as the paradigm case for existentially dangerous AI systems. I'm confused by this for a few reasons:
- Not all consequentialist/goal directed systems are expected utility maximisers
- E.g. humans
- Some recent developments make me sceptical that VNM expected utility are a natural form of generally intelligent systems
- Wentworth's subagents provide a model for inexploitable agents that don't maximise a simple unitary utility function
- The main requirement for subagents to be a better model than unitary agents is path dependent preferences or hidden state variables
- Alternatively, subagents natively admit partial orders over preferences
- If I'm not mistaken, utility functions seem to require a (static) total order over preferences
- This might be a very unreasonable ask; it does not seem to describe humans, animals, or even existing sophisticated AI systems
- If I'm not mistaken, utility functions seem to require a (static) total order over preferences
- I think the strongest implication of Wentworth's subagents is that expected utility maximisation is not the limit or idealised form of agency
- Shard Theory suggests that trained agents (via reinforcement learning[2]) form value "shards"
- Values are inherently "contextual influences on decision making"
- Hence agents do not have a static total order over preferences (what a utility function implies) as what preferences are active depends on the context
- Preferences are dynamic (change over time), and the ordering of them is not necessarily total
- This explains many of the observed inconsistencies in human decision making
- Hence agents do not have a static total order over preferences (what a utility function implies) as what preferences are active depends on the context
- A multitude of value shards do not admit analysis as a simple unitary utility function
- Reward is not the optimisation target
- Reinforcement learning does not select for reward maximising agents in general
- Reward "upweight certain kinds of actions in certain kinds of situations, and therefore reward chisels cognitive grooves into agents"
- Reinforcement learning does not select for reward maximising agents in general
- I'm thus very sceptical that systems optimised via reinforcement learning to be capable in a wide variety of domains/tasks converge towards maximising a simple expected utility function
- Values are inherently "contextual influences on decision making"
- Wentworth's subagents provide a model for inexploitable agents that don't maximise a simple unitary utility function
- I am not aware that humanity actually knows training paradigms that select for expected utility maximisers
- Our most capable/economically transformative AI systems are not agents and are definitely not expected utility maximisers
- Such systems might converge towards general intelligence under sufficiently strong selection pressure but do not become expected utility maximisers in the limit
- The do not become agents in the limit and expected utility maximisation is a particular kind of agency
- Such systems might converge towards general intelligence under sufficiently strong selection pressure but do not become expected utility maximisers in the limit
- Our most capable/economically transformative AI systems are not agents and are definitely not expected utility maximisers
- I am seriously entertaining the hypothesis that expected utility maximisation is anti-natural to selection for general intelligence
- I'm not under the impression that systems optimised by stochastic gradient descent to be generally capable optimisers converge towards expected utility maximisers
- The generally capable optimisers produced by evolution aren't expected utility maximisers
- I'm starting to suspect that "search like" optimisation processes for general intelligence do not in general converge towards expected utility maximisers
- I.e. it may end up being the case that the only way to create a generally capable expected utility maximiser is to explicitly design one
- And we do not know how to design capable optimisers for rich environments
- We can't even design an image classifier
- I currently disbelieve the strong orthogonality thesis translated to practice
- While it may be in theory feasible to design systems at any intelligence level with any final goal
- In practice, we cannot design capable optimisers.
- For intelligent systems created by "search like" optimisation, final goals are not orthogonal to cognitive ability
- Sufficiently hard optimisation for most cognitive tasks would not converge towards selecting for generally capable systems
- In the limit, what do systems selected for playing Go converge towards?
- I posit that said limit is not "general intelligence"
- In the limit, what do systems selected for playing Go converge towards?
- The cognitive tasks/domain on which a system was optimised for performance on may instantiate an upper bound on the general capabilities of the system
- You do not need much optimisation power to attain optimal performance in logical tic tac toe
- Systems selected for performance at logical tic tac toe should be pretty weak narrow optimisers because that's all that's required for optimality in that domain
- You do not need much optimisation power to attain optimal performance in logical tic tac toe
- Sufficiently hard optimisation for most cognitive tasks would not converge towards selecting for generally capable systems
- I.e. it may end up being the case that the only way to create a generally capable expected utility maximiser is to explicitly design one
I don't expect the systems that matter (in the par human or strongly superhuman regime) to be expected utility maximisers. I think arguments for AI x-risk that rest on expected utility maximisers are mostly disconnected from reality. I suspect that discussing the perils of expected utility maximisation in particular — as opposed to e.g. dangers from powerful (consequentialist?) optimisation processes — is somewhere between being a distraction and being actively harmful[3].
I do not think expected utility maximisation is the limit of what generally capable optimisers look like[4].
Arguments for Expected Utility Maximisation Are Unnecessary
I don't think the case for existential risks from AI safety rest on expected utility maximisation. I kind of stopped alieving expected utility maximisers a while back (only recently have I synthesised explicit beliefs that reject it), but I still plan on working on AI existential safety, because I don't see the core threat as resulting from expected utility maximisation.
The reasons I consider AI an existential threat mostly rely on:
- Instrumental convergence for consequentialist/goal directed systems
- A system doesn't need to be a utility maximiser for a simple utility function to be goal directed (again, see humans)
- Selection pressures for power seeking systems
- Reasons
- More economically productive/useful
- Some humans are power seeking
- Power seeking systems promote themselves/have better reproductive fitness
- Human disempowerment is the immediate existential catastrophe scenario I foresee from power seeking
- Reasons
- Bad game theoretic equilibria
- This could lead towards dystopian scenarios in multipolar outcomes
- Humans getting outcompeted by AI systems
- Could slowly lead to an extinction
I do not actually expect extinction near term, but it's not the only "existential catastrophe":
- Human disempowerment
- Various forms of dystopia
- ^
I optimised for writing this quickly. So my language may be stronger/more confident that I actually feel. I may not have spent as much time accurately communicating my uncertainty as may have been warranted.
- ^
Correct me if I'm mistaken, but I'm under the impression that RL is the main training paradigm we have that selects for agents.
I don't necessarily expect that our most capable systems would be trained via reinforcement learning, but I think our most agentic systems would be.
- ^
There may be significant opportunity cost via diverting attention from other more plausible pathways to doom.
In general, I think exposing people to bad arguments for a position is a poor persuasive strategy as people who dismiss said bad arguments may (rationally) update downwards on the credibility of the position.
- ^
I don't necessarily think agents are that limit either. But as "Why Subagents?" shows, expected utility maximisers aren't the limit of idealised agency.
The key question I always focus on is: where do you get your capabilities from?
For instance, with GOFAI and ordinary programming, you have some human programmer manually create a model of the scenarios the AI can face, and then manually create a bunch of rules for what to do in order to achieve things. So basically, the human programmer has a bunch of really advanced capabilities, and they use them to manually build some simple capabilities.
"Consequentialism", broadly defined, represents an alternative class of ways to gain capabilities, namely choosing what to do based on it having the desired consequences. To some extent, this is a method humans uses, perhaps particularly the method the smartest and most autistic humans most use (which I suspect to be connected to LessWrong demographics but who knows...). Utility maximization captures the essence of consequentalism; there are various other things, such as multi-agency that one can throw on top of it, but those other things still mainly derive their capabilities from the core of utility maximization.
Self-supervised language models such as GPT-3 do not gain their capabilities from consequentialism, yet they have advanced capabilities nonetheless. How? Imitation learning, which basically works because of Aumann's agreement theorem. Self-supervised language models mimic human text, and humans do useful stuff and describe it in text, so self-supervised language models learn the useful stuff that can be described in text.
Risk that arises purely from language models or non-consequentialist RLHF might be quite interesting and important to study. I feel less able to predict it, though, partly because I don't know what the models will be deployed to do, or how much they can be coerced into doing, or what kinds of witchcraft are necessary to coerce the models into doing those things.
It is possible to me that imitation learning and RLHF can bring us to the frontier of human abilities, so that we have a tool that can solve tasks as well as the best humans can. However, I don't think it will be able to much exceed that frontier. This is still superhuman, because no human is as good as all the best humans at all the tasks. But it is not far-superhuman, even though I think being far-superhuman is possible, and a key part in it not being far-superhuman is that it cannot extend its capabilities. As such, I would expect consequentialism to be necessary for creating something that is far-superhuman.
I think many of the classical AI risk arguments apply to consequentialist far-superhuman AI.
Expanded: Where do you get your capabilities from?