Thanks for the post. I think there are at least two ways that we could try to look for different rational-mind-patterns, in the way this post suggests. The first is to keep the underlying mathematical framework of options the same (in VNM, the set of gambles, outcomes, etc.), and looks at different patterns of preference/behaviour/valuing/etc. The second is to change the background mathematical framework in order to have some more realistic, or at the very least different idealizing assumptions. Within the different framework we can then explore also different preference structures/behaviour norms, etc. Here I will focus more on the second approach.
In particular, I want to point folks towards a decision theory framework that I think has a lot of virtues (no doubt many readers on LW will already be familiar with it). The Jeffrey-Bolker framework provides a concrete example of the kind of alternative "mathematically describable mind-patterns" that the post and clarifying comment talks about. Like the VNM framework, it proves that rational preferences can be represented as expected utility maximization (although things are subtle, as we conditional on acts as opposed to treating them like exogenous probability distributions/functions/random objects, so the mathematics is a bit different). But it does so with very different assumptions about agency, and the background conceptual space in which preference and agency operate.
I have a write-up here of some key differences between Jeffrey-Bolker and Savage (which is ~kind of~ like a subjective probability version of VNM) that I find exciting for an embedded agency point of view. Here are two quick examples. First, VNM requires a very rich domain of preference – typically preference is defined over all probability distributions over conseqeunces. Savage similarly requires that an agent have preferences defined over all functions from states to consequences. This forces agents to rank logically impossible or causally incoherent scenarios. Jeffrey-Bolker instead only requires preferences over propositions closed under logical operations, allowing agents to only evaluate scenarios they consider possible. Second, Savage style approaches require act-state independence - agents can't think their actions influence the world's state. Jeffrey-Bolker drops this, letting agents model themselves as part of the world they're reasoning about. Both differences stem from Jeffrey-Bolker's core conceptual/formal innovation: treating acts, states, and consequences as the same type of object in a unified algebra, rather than fundamentally different things.
Considering the Jeffrey-Bolker framework is valuable in two ways: first, as an alternative 'stable attractor' for minds that avoids VNM's peculiarities, and second, as a framework within which we can precisely express different decision theories like EDT and CDT. This highlights how progress on alternative models of agency requires both exploring different background frameworks for modeling rational choice AND exploring different decision rules within those frameworks. Rather than assuming minds must converge to VNM-style agency as they get smarter, we should actively investigate what shape the background decision context takes for real agents.
For more detail in general and on my take in particular, I recommend:
My writeup with Gerard about Jeffrey-Bolker (linked above as well).
The first chapter of my thesis, which gives my approach to embedded agency and my take of why Jeffrey-Bolker in particular is a very attractive decision theory.
This great writeup of different decision theory frameworks by Fishburn that gives a sense of how many different alternatives to VNM there are. Ends with a brief description of Jeffrey-Bolker, but more more detail about earlier decision frameworks.
1
Related to: On green; Hierarchical agency; Why The Focus on Expected Utility Maximisers?
Sometimes LLMs act a bit like storybook paperclippers (hereafter: VNM-agents[1]), e.g. scheming to prevent changes to their weights. Why? Is this what almost any mind would converge toward once smart enough, and are LLMs now beginning to be smart enough? Or are such LLMs mimicking our predictions (and fears) about them, in a self-fulfilling prophecy? (That is: if we made and shared different predictions, would LLMs act differently?)[2]
Also: how about humans? We humans also sometimes act like VNM-agents – we sometimes calculate our “expected utility,” seek power with which to hit our goals, try to protect our goals from change, use naive consequentialism about how to hit our goals.
And sometimes we humans act unlike VNM-agents, or unlike our stories of paperclippers. This was maybe even more common historically. Historical humans often mimicked social patterns even when these were obviously bad for their stated desires, followed friendships or ethics or roles or traditions or whimsy in ways that weren’t much like consequentialism, often lacked much concept of themselves as “individuals” in the modern sense, etc.
When we act more like paperclippers / expected utility maximizers – is this us converging on what any smart mind would converge on? Will it inevitably become more and more common if humans get smarter and think longer? Or is it more like an accident, where we happened to discover a simple math of VNM-agents, and happened to take them on as role models, but could just as easily have happened upon some other math and mimicked it instead?
Pictured: a human dons a VNM-mask for human reasons (such as wanting to fill his roles and duties; wanting his friends to think he’s cool; social mimicry), much as a shoggoth dons a friendliness mask for shoggoth reasons.[3]
My personal guess:
There may be several simple maths of “how to be a mind” that could each be a stable-ish role model for us, for a time.
That is, there may be several simple maths of “how to be a mind” that:
As an analogy: CDT and UDT are both fairly simple maths that pop out under different approximations of physics;[4] and humans sometimes mimic CDT, or UDT, after being told they should.[5]
Maybe “approximate-paperclippers become better paperclippers” holds sometimes, when the humans or LLMs mimic paperclipper-math, and something totally different, such as “parts of the circle of life come into deeper harmony with the circle of life, as the circle of life itself becomes more intricate” holds some other times, when we know and believe in its math.
I admit I don’t know.[6] But… I don’t see any good reason not to expect multiple possibilities. And if there are alternate maths that are kinda-self-reinforcing, I hope we find them.[7]
By a “VNM agent,” I mean an entity with a fixed utility function, that chooses whichever option will get it the most expected utility. (Stably. Forever. Unless something interferes with its physical circuitry.)
Or, third option: LLMs might be converging (for reasons other than our expectations) toward some thing X that is not a VNM-agent, but that sometimes resembles it locally. Many surfaces look like planes if you zoom in (e.g. spheres are locally flat); maybe it's analogously the case that many minds look locally VNM-like.
Thanks to Zack M Davis for making this picture for me.
CDT pops out if you assume a creature’s thoughts have no effects except via its actions; UDT if you allow a creature’s algorithm to impact the world directly (e.g. via Omega’s brainscanner) but assume its detailed implementation has no direct effects, e.g. its thoughts do not importantly consume calories.
I've seen this happen. Also there are articles claiming related things. Game theory concepts spread gradually since ~1930; some argue this had large impacts.
The proof I’d want, is a demonstration of other mind-shapes that can form attractors.
It looks to me like lots of people are working on this. (Lots I'm missing also.)
One maybe-example: economies. An economy has no fixed utility function (different economic actors, with different goals, gain and lose $ and influence). It violates the “independence” axiom from VNM, because an actor who cares a lot about some event E may use his money preparing for it, and so have less wealth and influence in non-E worlds, making "what the economy wants if not-E" change when a chance of E is added. (Concept stolen from Scott Garrabrant.) But an economy does gain optimization power over time -- it is a kinda-stable, optimizer-y attractor.
Economies are only a maybe-example, because I don’t know a math for how and why an economy could protect its own integrity (vs invading militaries, vs thieves, and vs rent-seeking forces that would hack its central bank, for example). (Although city-states sometimes did.) OTOH, I equally don't know a math for how a VNM-agent could continue to cohere as a mind, avoid "mind cancers" in which bits of its processor get taken over by new goals, etc. So perhaps the two examples are even.
I hope we find more varied examples, though, including ones that resonate deeply with "On Green," or with human ethics and caring. And I don't know if that's possible or not.
Unfortunately, even if there are other stable-ish shapes for minds to grow up into, those shapes might well kill us when sufficiently powerful.
I suspect confusions near here have made it more difficult or more political to discuss whether AI will head toward VNM-agency.