I do think that a wide range of shard-based mind-structures will equilibrate into EU optimizers, but I also think this is a somewhat mild statement. My stance is that utility functions represent a yardstick by which decisions are made. "Utility was made by the agent, for the agent" as it were--- and not "the agent is made to optimize the utility." What this means is:
Suppose I start off caring about dogs and diamonds in a shard-like fashion, with certain situations making me seek out dogs and care for them (in the usual intuitive way); and similarly for diamonds. However, there will be certain situations in which the dog-shard "interferes with" the diamond-shard, such that the dog-shard e.g. makes me daydream about dogs while doing my work and thereby do worse in life overall. If I didn't engage in this behavior, then in general I'd probably be able to get more dog-caring and diamond-acquisition. So from the vantage point of this mind and its shards, it is subjectively better to not engage in such "incoherent" behavior which is a strictly dominated strategy in expectation (i.e. leads to fewer dogs and diamonds).
Therefore, given time and sufficient self-modification ability, these shards will want to equilibrate to an algorithm which doesn't step on its own toes like this.
This doesn't mean, of course, that these shards decide to implement a utility function with absurd results by the initial decision-making procedure. For example, tiling the universe (half with dog-squiggles, half with diamond-squiggles) would not be a desirable outcome under the initial decision-making process. Insofar as such an outcome could be foreseen as a consequence of making decisions a proposed utility function, the shards would disprefer that utility function.[1]
So any utility function chosen should "add up to normalcy" when optimized, or at least be different in a way which is not foreseeably weird and bad by the initial shards' reckoning. On this view, one would derive a utility function as a rule of thumb for how to make decisions effectively and (nearly) Pareto-optimally in relevant scenarios.[2]
(You can perhaps understand why, given this viewpoint, I am unconcerned/weirded out by Yudkowskian sentiments like "Unforeseen optima are extremely problematic given high amounts of optimization power.")
This elides any practical issues with self-modification, and possible value drift from e.g. external sources, and so on. I think they don't change the key conclusions here. I think they do change conclusions for other questions though.
Again, if I'm imagining the vantage point of dog+diamond agent, it wouldn't want to waste tons of compute deriving a policy for weird situations it doesn't expect to run into. The most important place to become more coherent is the expected on-policy future.
Let's not forget the old, well-read post: Dreams of AI Design. In that essay, Eliezer correctly points out errors in imputing meaning to nonsense by using suggestive names to describe the nonsense.
Artificial intelligence meets natural stupidity (an old memo) is very relevant to understanding the problems facing this community's intellectual contributions. Emphasis added:
Consider the following terms and phrases:
Some of these terms are useful. Some of the academic imports are necessary for successful communication. Some of the terms have real benefits.
That doesn't stop them from distorting your thinking. At least in your private thoughts, you can do better. You can replace "optimal" with "artifact equilibrated under policy update operations" or "set of sequential actions which have subjectively maximal expected utility relative to [entity X]'s imputed beliefs", and the nice thing about brains is these long sentences can compress into single concepts which you can understand in but a moment.
It's easy to admit the mistakes of our past selves (whom we've conveniently grown up from by time of recounting). It's easy for people (such as my past self and others in this community) to sneer at out-group folks when they make such mistakes, the mistakes' invalidity laid bare before us.
It's hard when you've[2] read Dreams of AI Design and utterly failed to avoid same mistakes yourself. It's hard when your friends are using the terms, and you don't want to be a blowhard about it and derail the conversation by explaining your new term. It's hard when you have utterly failed to resist inheriting the invalid connotations of other fields ("optimal", "reward", "attention mechanism").
I think we have failed, thus far. I'm sad about that. When I began posting in 2018, I assumed that the community was careful and trustworthy. Not easily would undeserved connotations sneak into our work and discourse. I no longer believe that and no longer hold that trust.
To be frank, I think a lot of the case for AI accident risk comes down to a set of subtle word games.
When I try to point out such (perceived) mistakes, I feel a lot of pushback, and somehow it feels combative. I do get somewhat combative online sometimes (and wish I didn't, and am trying different interventions here), and so maybe people combat me in return. But I perceive defensiveness even to the critiques of Matthew Barnett, who seems consistently dispassionate.
Maybe it's because people perceive me as an Optimist and therefore my points must be combated at any cost.
Maybe people really just naturally and unbiasedly disagree this much, though I doubt it.
But the end result is that I have given up on communicating with most folk who have been in the community longer than, say, 3 years. I don't know how to disabuse people of this trust, which seems unearned.
All to say: Do not trust this community's concepts and memes, if you have the time. Do not trust me, if you have the time. Verify.
See also: Against most, but not all, AI risk analogies.
How many times has someone expressed "I'm worried about 'goal-directed optimizers', but I'm not sure what exactly they are, so I'm going to work on deconfusion."? There's something weird about this sentiment, don't you think? I can't quite put my finger on what, and I wanted to get this post out.
Including myself, and I suspect basically every LW enthusiast interested in AI.