Arjun Panickssery's Shortform

Arjun Panickssery

This is a special post for quick takes by Arjun Panickssery. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Many LW people believe in one of a family of meta-ethical theories that don't make sense to me.

Idealizing subjectivism (IdS): This theory says that X is intrinsically valuable, relative to an agent A, if and only if, and because, A would have some set of evaluative attitudes towards X, if A had undergone some sort of idealization procedure." (definition from Joe Carlsmith who says "Idealizing subjectivism has been something like my best-guess meta-ethics. And lots of people I know take it for granted").
Coherent extrapolated volition (CEV): Some say that ideally an AI should predict what people should want "if we knew more, thought faster, were more the people we wished we were, had grown up farther together." The AI should use the desires that "converge" among everyone in some sense, but I also hear people talk about such-and-such person's CEV (cf Habryka on "Vladimir Putin's CEV").
Ideal-observer theory (IOT): This is an academic theory that says that to say something is good is to say that an "ideal observer" would approve of it. Firth in "Ethical absolutism and the ideal observer" says this ideal observer should be omniscient with respect to natural facts, dispassionate, disinterested, consistent, "normal," etc.

These are all framed slightly differently: IdS is an anti-realist theory of what to care about, CEV is about how to command an AI, and IOT is about what moral statements mean. But these theories don't help with the hard problems with meta-ethics that they try to resolve or elide. In particular all of the theories based on an idealization procedure fail because either

The idealization procedure is taken to include moral knowledge, creating circularity, or
The idealization procedure only includes rationality in the making of non-moral judgments, knowledge of non-moral facts, etc, in which case this is a reductionist meta-ethics that doesn't actually cross the is-ought gap (i.e. it remains an open question whether the idealized attitudes would be good).

I basically think these views are popular because while moral realism is not plausible, these idealization theories allow for crypto-realism where the exact same discussion is had but framed around this illusory target of our "idealized" selves, whose relevance or for whose actual convergence there isn't any evidence.

3habryka5mo

I think you are probably misinterpreting me here, though the domain is tricky, so that's understandable. I advocate that you only take the steps towards consistency that are endorsed. There are really quite a lot of those! This does not require giving (apparent) logical consistency some kind of supremacy. Indeed, I would strongly argue against the kind of philosophy that MacAskill tends to do, and don't think it really has much to do with the thing that I expect to happen during CEV. The way I usually phrase it is that you list all the interventions that you could make to your beliefs and brain, and you start doing the ones that seem the most robust under really any viewpoint (e.g. something like "make sure to get enough sleep"). Then you work your way down the list, very conservatively taking actions or propagating beliefs that seem less reversible or robust.[1] I think the default outcome of this maximally conservative approach is that you still end up somewhere extremely different from where you started, and it doesn't really require giving self-consistency some kind of dominating overriding status where someone gives you a clever argument with horrifying conclusions and then you have to accept it. Indeed, not accepting those arguments seems extremely wise to me. Yes, this does require some degree to which my moral beliefs are subject to consistency, but of course, they would have no meaning at all if they were not at least subject to some minimal levels of consistency. A preference needs to ground in reality somehow, and for the things over which you have preferences to "be real" in some meaningful sense. And the subject of this conversation is the kind of preference that makes sense for humans to endorse and make plans around. A bundle of local-minimization urges does not write internet comments, or thinks about what they would like a future AI system to do with them, or cares about "metaethics" at all. 1. ^ This would reasonably also include th

2Steven Byrnes5mo

That all sounds fine, if we’re engaged in a pragmatic project for deciding what to do, and want to propose an answer that you and I can get behind, and that lots of people around the world can also get behind. I think Arjun is (rightly) complaining about something different, namely that Eliezer and you and others frequently slip into treating this answer as being fundamentally privileged / “Right”, as opposed to merely a pragmatic option that you and I and lots of people can get behind. E.g. here’s Nate referring to “the future’s potential value”, as if there’s a metric for that which is canonical and characteristic of humanity-as-a-whole. I think that’s moral-realist (or “crypto”-moral-realist) thinking, sneaking in.

Hmm, I don't really get this. Or like, I am about as sympathetic to this argument as someone saying "E.g. here's Nate referring to 'the future' as a thing that exists, as if there is consensus on there being a single reality and arrow of time. I think that's scientific materialist thinking sneaking in, denying the possibility of solipsism or simulationism". To which my reaction is "yes, metaphysics is actually quite confusing, but come on man, you know what I mean, in as much as words mean anything, this is a fine use of them".

Similarly here, my reaction ... (read more)