Probably worth noting that there's lots of frames to pick from, of which you've discussed two: question, ideology, project, obligation, passion, central purpose, etc.
I think that these are all pretty relevant ways to think about being an EA, but are mostly of a different fundamental type than the thing I'm pointing at. Let me get a bit more into the aforementioned math to show why this is approximately a binary categorization along the axis I was pointing at in this post.
Say that there are three possible world states:
As a culture, on average, our values look something like:
EA > gardener > murderer
This is very reasonable and I don't think we can or should change this, as a community.
I have some mental approximation of a utility function. One of the main differences between my internal representation and an actual utility function is that the point I choose as "zero utility" doesn't matter in the formalism, but very much matters to my emotions. If we set EA=0 utility, then the myopic point-maximizing part of my brain feels okay if I do EA things, but awful if I'm getting negative points by being either of the other options. This is the moral obligation frame, where things are only barely emotionally okay if you push as far up the preference ordering as possible.
If we set gardener=0, then things feel emotionally okay if I just take the normal path. I'm not gaining or losing points. It's then positively great if I do EA things and still positively bad if I kill people. This is the moral opportunity frame, and I find it emotionally much better for me. I predict that this frame is better for community health as well, although I have only vibes and anecdata to back me up on this claim.
There are several other points I have left unnamed:
Now that's a lot of possibilities. I promised that I had "approximately a binary categorization", so where's the binary come in? Well, the dividing line I'm drawing is ultimately is "is being a normal member of society 'okay' or 'not okay'?" Alternatively, we ask the question of how our community responds to Carol, the gardener. Are we friends with her? I say yes. Do we give her the same positive reinforcement for her new daffodils that we give to someone when they donate a large sum to EA charities? I say no.
(I am intentionally neglecting the various "evil" cultures from this consideration. Technically it's not a binary if you want to include them, but I really don't see why we would ever consider doing that in real life.)
These other frames you mention are important shards of a healthy EA community, I think. They're just not quite the concept boundary I was trying to draw.
Positive reinforcement is more effective than positive punishment because it tells someone exactly what to do instead of just what not to do.
But being told exactly what to do is much more restrictive than just being told what not to do. So if you make the above claim, and also make the claim that your way is more pleasant and less restrictive, then Bob is right—there is definitely some trick being played here.
Retracted my previous response due to faulty reasoning.
Yes, positive reinforcement gives a more precise signal than positive punishment. Maybe I didn't elaborate on this fact enough, but that is the point. I like it when a community precisely signals what its values are. I like it when a community isn't adversarial to bystanders. These are, among other reasons, precisely why I want a positive reinforcement culture. The point of the post is that these two frames are different. You pointed out a difference in the post that I discussed.
Positive reinforcement, however, is not more restrictive. There is no restriction going on, that's the point. It is just a more precise signal. If I give you a cookie every time you do the dishes and nothing when you don't, that is not restricting the counterfactual-yous who don't do dishes from being in my community. Somebody else just does the dishes and gets the cookie.
I don’t think that you’re responding to what I actually wrote… but perhaps I was unclear. Let me try again, with emphasis on the relevant part:
Positive reinforcement is more effective than positive punishment because it tells someone exactly what to do instead of just what not to do.
The bolded part is what I am addressing.
Being told exactly what to do is more restrictive than just being told what not to do. That is my claim. This claim is not dependent on any purported connection between “being told what to do” and “positive reinforcement”, or between “being told what not to do” and “positive punishment”.
Positive reinforcement, however, is not more restrictive. There is no restriction going on, that’s the point. It is just a more precise signal.
Ethics is the means by which we answer the question “what is the right thing to do”. In ethics, there is no difference between rewarding good things and punishing bad things; these are isomorphic approaches.
That means that only rewarding one specific thing is isomorphic to punishing all things except that thing; and only punishing one thing is isomorphic to rewarding all things except that thing.
(This is analogous to the distinction between permissive vs. restrictive legal systems—or, as in the old joke, the difference between “everything is permitted, except that which is forbidden” and “everything is forbidden, except that which is permitted”.)
Now, you object that you’re not punishing anyone, that all of this stuff is strictly optional. This is the concept of supererogation. There are two problems with this:
First, most rationalists/EAs/whatever are utilitarians, and utilitarianism, famously, has no concept of supererogation. There cannot possibly be any such thing as “good but not mandatory” for a utilitarian. If Bob knows this, then he can see that utilitarians who tell him that he doesn’t have to to do the most good if he doesn’t want to are either hypocrites or liars (or they don’t understand the moral system they claim to espouse).
Second, what Bob in your fictional scenario can probably see, and what most people in real life can also see, is that if status in your community is accorded to those who take all of the so-called “moral opportunities”, and is not accorded to those who turn down those “moral opportunities”, then they’re not “opportunities” at all—they’re obligations.
In other words, Bob asks: “what must I do in order for your community to see me as a good person?”. Possible answers include:
What Bob hears when you talk to him about “moral opportunities” is that your answer is #3, but you don’t want to admit that your answer is #3, because you feel (correctly) that it’s restrictive and judgmental. Bob suspects a trick because there is a trick.
I mean, it’s right there in the terminology you’re using: “moral opportunities”. Moral opportunities. There isn’t any such thing as moral opportunities! There are moral obligations, and then there’s everything else. If you hear people resisting this sort of terminological sleight-of-hand, it’s because they recognize that calling a thing by a different name does not actually change the patterns of status assignment and other social dynamics related to that thing.
Talking about “moral opportunities” is like talking about the opportunity to pass a class by doing the assigned homework, or the opportunity to keep getting your paycheck by doing what your boss tells you to do, or the opportunity to stay out of prison by paying your taxes on time.
The only way to get out of this would be to actually, genuinely not consider someone who does whatever-it-is (helping to save the world from AI / saving the animals / etc.) to be a better person than someone who does. (In the way that we generally don’t consider someone who makes a lot of money to be a better person than someone who makes less money. If you have a chance to make a bunch of money—legally, without doing anything unethical, etc.—we may cheer you on, we may be happy for you if you succeed, but we won’t think less of you if you choose not to do that, right? That chance is an opportunity—the regular, non-moral kind of opportunity.)
But if your community is built around the notion that doing whatever-it-is is important, valuable, and morally the right thing to do, then of course you will not ever stop considering someone who does whatever-it-is to be a better person than someone who does not. (And why should you?) Newcomers will correctly perceive that you think this. And if you try to convince them otherwise, they will correctly detect a trick.
No, that's not quite what I'm saying in my post. I'm pointing at the times when a culture could either positively reinforce X or positively punish not-X, which is exactly the same amount of restriction.
Maybe, one could argue, in practice positive reinforcement creates a more precise signal, which on my model is a good thing; I like communities that clearly signal their values and aren't adversarial to bystanders. I really don't think this is a trick, I think the post is in fact about the differences between these two framings, including about the reinforcement/punishment distinction.
I agree that this is probably right in terms of mental health and social dynamics. Do you believe it is also right as a matter of actual morality? Do you agree with the drowning child analogy? Do you think it applies to broader social problems like AI or factory farming? If not, why?
This concept and terminology was spawned out of a conversation a few years ago with my friend Skyler. I finally decided to write it up. Any mistakes here are my own.
Every once in a while, I find myself at yet another rationalist/EA/whatever-adjacent social event. Invariably, someone walks up to me:
Hi, I'm Bob. I'm pretty new here. What do you work on?
Hi Bob, I'm Alice. I work on preventing human extinction from advanced AI. If you'd like, I'm happy to talk a bit more about why I think this is important.
Bob is visibly nervous.
Yeah, I've already gotten the basic pitch on AI safety, it seems like it makes sense. People here all seem to work on such important things compared to me. I feel a sort of moral obligation to help out, but I feel stressed out about it and don't know where to start.
Alice is visibly puzzled, then Bob is puzzled that Alice is puzzled.
I'm not sure I understand this "moral obligation" thing? Nobody's forcing me to work on AI safety, it's just what I decided to do. I could've chosen to work on music or programming or a million other things instead. Can you explain what you mean without using the word "obligation"?
Well, things like "I'm going to save the world from extinction from AI" or "I'm going to solve the suffering of billions of farmed animals" are really big and seem pretty clearly morally right to do. I'm not doing any of those things, but I feel an oblig- hmm... I feel like something's wrong with me if I don't work on these issues.
I do not think anything is necessarily wrong with you if you don't work on AI safety or any other of these EA causes. Don't get me wrong, I think they're really important and I love when there are more people helping out. I just use a very different frame to look at this whole thing.
Before running into any of these ideas, I was going about my life, picking up all sorts of opportunities as I went: there's $5 on the ground? I pick it up. There's a cool conference next month? I apply. So when I heard that I plausibly lived in a world where humans go extinct from AI, I figured that owning up to it doesn't make it worse, and I looked at my opportunities. I get the chance to learn from a bunch of smart people to try and save the world? Of course I take the chance, that sounds so cool.
My point here is that you're socially and emotionally allowed to not take that opportunity, just like you're allowed to not pick up $5 or not apply to conferences. I think it's probably good for people to pick up $5 when they see it or help out with AI safety if they can, but it's their opportunity to accept or decline.
This feels like approximately the same thing as before? Under the moral obligation frame, people look at me negatively if I don't do the Super Highly Moral thing, and under the moral opportunity frame you tell me I have a choice but only look at me positively if I do the Super Highly Moral thing? Isn't this just the same sort of social pressure, but you say something about respecting personal agency?
Well, I'm not actually that judgmental, I'll look at you pretty positively unless you do something Definitely Wrong. But that's not the point. The point is that these two framings make a huge emotional difference when used as norms for a personal or group culture. Positive reinforcement is more effective than positive punishment because it tells someone exactly what to do instead of just what not to do. Reinforcement is also just a more emotionally pleasant stimulus, which goes a long way.
Let's look at this a different way: say that my friend Carol likes to watch TV and play video games and not much else. The moral obligation frame looks at Carol and finds her clearly in the moral wrong, lounging around while there are important things to be doing. The moral opportunity frame looks at her and sees a person doing her own things in a way that doesn't hurt other people, and that's morally okay.
These two frames still seem weirdly similar, like in the "moral opportunity" frame you just shifted all options to be a bit more morally good so that everything becomes okay. But ultimately both frames still think working on saving the world is better than watching TV. I see what you're saying about emotions, but this still feels like some trick is being played on my sense of morality.
That's a reasonable suspicion. I think the math of this sort of shifting works out, I really don't think there's any trick here. Ultimately it's your choice how you want to interface with your emotions. I find that people are much more likely to throw their mind away when faced with something big and scary that feels like an obligation, compared with when they feel like an explorer with so many awesome opportunities around.
It's sad to live in a world that could use so much saving, and dealing with that is hard. There's no getting around that emotional difficulty except by ignoring the world you live in. Conditional on the world we live in, though, I'd much rather live in a culture that frames things as moral opportunities than moral obligations.
I frame this as a conversation with a newcomer, but I also see the moral obligation frame implicit in a lot of experienced EAs, especially those who are going through some EA burnout. The cultural pieces making up this post mostly already existed across the EA-sphere (and I've tried to link to them where possible), but I haven't seen them collected in this way before, nor have I seen this particular concept boundary drawn.