This is a special post for quick takes by Raemon. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
There was a particular mistake I made over in this thread. Noticing the mistake didn't change my overall position (and also my overall position was even weirder than I think people thought it was). But, seemed worth noting somewhere.
I think most folk morality (or at least my own folk morality), generally has the following crimes in ascending order of badness:
But this is the conflation of a few different things. One axis I was ignoring was "morality as coordination tool" vs "morality as 'doing the right thing because I think it's right'." And these are actually quite different. And, importantly, you don't get to spend many resources on morality-as-doing-the-right-thing unless you have a solid foundation of the morality-as-coordination-tool.
There's actually a 4x3 matrix you can plot lying/stealing/killing/torture-killing into which are:
And you basically need to tackle these in this order. If you live in a world where even people in your tribe backstab each other all the time, you won't have spare resources to spend on the outgroup or the powerless until your tribe has gotten it's basic shit together and figured out that lying/stealing/killing each other sucks.
If your tribe has it's basic shit together, then maybe you have the slack to ask the question: "hey, that outgroup over there, who we regularly raid and steal their sheep and stuff, maybe it'd be better if we traded with them instead of stealing their sheep?" and then begin to develop cosmopolitan norms.
If you eventually become a powerful empire (or similar), eventually you may notice that you're going around exploiting or conquering and... maybe you just don't actually want to do that anymore? Or maybe, within your empire, there's an underclass of people who are slaves or slave-like instead of being formally traded with. And maybe this is locally beneficial. But... you just don't want to do that anymore, because empathy or because you've come to believe in principles that say not to, or something. Sometimes this is because the powerless people would actually be more productive if they were free builders/traders, but sometimes it just seems like the right thing to do.
Avoiding harming the ingroup and productive outgroup are things that you're locally incentived to do because cooperation is very valuable. In an iterated strategy game, these are things you're incentived to do all the way along.
Avoiding harming the powerless is something that you are limited in your ability to do until the point where it starts making sense to cash in your victory points.
I think this is all pretty non-explicit in most discussions of morality/ethics/what-people-should-do, and conflation of "actions that are bad because it ruins ability to coordinate" and "actions that are bad because empathy and/or principles tell me they are" is common.
Attention is scarce and there are lots of optimization processes going on, so if you think the future is big relative to the present, interventions that increase the optimization power serving your values are going to outperform direct interventions. This doesn't imply that we should just do infinite meta, but it does imply that the value of direct object-level improvements will nearly always be via how they affect different optimizing processes.
This is an experiment in short-form content on LW2.0. I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:
I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.