120

LESSWRONG
LW

119
World Optimization
Frontpage

19

Judging types of consequentialism by influence and normativity

by Cole Wyeth
29th Apr 2025
3 min read
0

19

World Optimization
Frontpage

19

New Comment
Moderation Log
More from Cole Wyeth
View more
Curated and popular this week
0Comments

Epistemic status: Many citations missing.

Itzhak Gilboa has defined an irrational decision as one where the decision-maker feels embarrassed when confronted with an analysis of their decision. Normativity is then advice to be less irrational (or, the practice of following all such advice in advance).

Is consequentialism normative? Are virtue-ethicists irrational?

I don't think that is inherently true. Batman can really, truly value not directly killing others, even if that ultimately leads to more death down the line. He simply has to center his values on himself - literally, to be self-centered - and specifically his own actions. It's conceivable to formulate a first-person utility function for Batman that takes in his experiences over the course of his life and assigns an arbitrarily low value to those that include actions where he kills someone.

Formal version

I claim that there exist utility functions for AIXI-like agents that depend on the action symbols of their interaction histories. In fact, a recent paper submission of mine (with Marcus Hutter) generalizes the AIXI model in a way that allows this sort of thing. It's not hard to do; it just leaves more "free parameters" than the reward-based utility function of the original AIXI, which makes it a much less specific theory, with many particular implementations that are not interesting.

Batman is not very consequentialist, even though he can sort-of be framed as a consequentialist from a first-person view. This first-person and action-oriented type of consequentialism is, however, incredibly permissive - whatever a person does, one might argue that they preferred it for its own sake[1]. 

For a subtly stronger type of consequentialism, we can be a little more strict, and allow Batman to care not about the internal choice to act but only the "realized" action itself and its consequences. For instance, perhaps Batman should not care whether he internally (almost metaphysically) chose to kill, but only whether he actually observed his hands strangling someone. But by designating "Batman's hands" as a particularly central focus of value in the universe, we still permit a lot of things that look unreasonable from the third person.

In other words, apparently-normative advice (e.g. "please just finish this by killing the Joker so that he doesn't break out of prison for the hundredth time and murder more innocents") may not be normative for a sufficiently stubborn egoist or solipsist.

However, I have an intuition that most of us aren't really that self-centered, if we thought it through. Really, we value the things that happen "out there" in the universe. 

One reason for this is that valuing acts (and all of virtue ethics) arises naturally as a heuristic for having good consequences on the universe. Therefore, I expect that nudging someone with "isn't the reason you decided killing was wrong really because you don't like for people to die?" may actually work, for genetic[2] reasons. Virtue ethics looks suspiciously like a confusion between instrumental and intrinsic value - it doesn't logically have to be, be one suspects it is in any given case.

This points at a deeper reason. Perhaps those beings that are selected for by optimization processes like evolution tend to maximize their influence. If so, the strictest form of consequentialism (actually caring about what happens out in the world, without being overly concerned about one's own actions) seems like a natural value system to arrive at - though perhaps, not necessarily as the sole overriding value system.

It might be that when we criticize rationality, we have this implicit goal of some kind of influence in mind. We expect others to care about influence, at least a little bit, if they really thought it through. If that potential influence is big enough, we might even expect it to dominate other concerns (hence the evangelical zeal of effective altruism).

I don't think influence is inherently normative. In many cases though, I think it is reasonable to expect relevant agents to care about their influence. Agents which do not care even instrumentally about effecting the wider world probably will not predominate. They will probably be bystanders and NPCs - and by their own lights, this is fine. 

However, to achieve relevance, it is probably enough to take the first-person view, but to care about one's experiences much more than one's actions-for-their-own-sake, which is a synthesis of consequentialism and self-centeredness. This condition is already strong enough to rule out Batman, but not hedonists or wireheaders. 

  1. ^

    See Dostoevsky's "Notes from the Underground."

  2. ^

    In the sense of genesis.