I wish I could upvote and disagree. It's important to discuss this, but "rounding to zero" is a mistake, and can easily be the source of burnout and dissatisfaction.
If your work literally doesn't change the probability of success, you're probably right to leave. Whether it moves from 51% to 51.0001% or from 5% to 5.0001% (or from 0.1% to 0.1001%) is actually the reverse - you make MORE of a difference at the lower levels of probability of success.
Also, lessons from other kinds of battlefield should be remembered. Most soldiers don't fight for freedom, for resources, or for ideals - they fight for their squadmates, and for the expectations of their family/friends. Doing work that makes the short-term better is very motivating and valuable. Doing work that is short-term useful and long-term neutral-to-very-slightly-positive, and gives you status and lets you hang out with people you like is probably more satisfying and motivating than 90% of humanity has.
I appreciate the comment and think I agree with most of it. Was there anything in the post that seemed to disagree with this reasoning?
This is one of those situations where rank ordering is superior to precision estimates. Having identified the approximate probabilities of success, and multiplied them by their estimated impact, focusing on the details of that number is deeply irrelevant it seems to me. Put the options in order from best to worst, do the best thing, and anchor emotionally on that. What are you doing? The right thing to do.
Once the task is begun I almost never think about the probability of success again. All the updating was done leading up to the decision; there's no reason to visit the question again until some kind of big new information arrives, like confirmation what you are doing can't possibly work or similar.
Read anarchists. Anarchists have had no hope since 1936 and still have never stopped fighting. I'm pretty sure there's a CrimethInc. essay on exactly this topic.
Would you happen to know the name/link to such essay? I made a brief attempt at searching for it but couldn't find it. Thank you.
I honestly have no idea. It might be in Expect Resistance somewhere, which if not directly about this topic, is generally about it.
I may have been (edit: was probably) thinking about The Promise of Defeat, by Moxie Marlinspike, anarchist cyrptographer sailor extraordinaire and the author of the Signal protocol (and the original Signal app, though he's no longer with the project).
A simple but important point, that has shaped my frame for how to be an emotionally healthy and productive person, even if the odds seem long.
I thought at first this was going to be a restatement of Conviction without Deception, but actually man the distinction between epistemic hope, emotional hope, and just-doing-the-thing was pretty helpful. This was a nice, succinct post articulating a good concept.
One thing helping me to preserve hope is the fact that there are so many unknown variables about AGI and how humanity will respond to it, that I don't think that any current-day prediction is worth a lot.
Although I must admit, doomers like Connor Leahy and Eliezer Yudkovsky might be extremely persuasive but they also don't know many important things about the future and they are also full of cognitive biases. All of this makes me tell myself a mantra "There is still hope that we might win".
I am not sure whether this is the best way to think about these risks but I feel like if I'll give it up, it is a straightforward path to existential anxiety and misery, so I try not to question it too much.
We've given up hope, but not the fight. -- Nate Soares, probably
Many people I know are working on projects that they believe have a low (or very low) probability of being helpful. Even when they think diving into their work is the “right move” or “best option”, they find it hard to stay motivated or avoid burnout.
It may be hard to feel motivated about continuing to fight, since doubling our chances of survival will only take them from 0% to 0%. (from Death with Dignity)
I’ve been finding it helpful to distinguish between three concepts:
The three of these tend to go together. Alice thinks her plan is 80% likely to work, so she feels good when she thinks about her plan, and she puts in a lot of effort.
But they don’t have to be.
Consider Bob: he thinks his plan is <10% likely to work. But he still feels good when he thinks about his plan. He knows he’s taking on extremely hard problems, and he’s proud of himself and his colleagues for working on the plan despite its grim forecast. He also puts in a lot of effort.
I know people like Bob, and sometimes I feel like Bob. I might feel excited about a plan (despite its low probability of success), or grateful toward others for persevering and fighting (even if I have grim forecasts about the value of their work).
But not always. Sometimes I feel more like Carol.
Carol thinks her plan is <10% likely to work. She feels badly about this. She feels frustrated and disappointed with herself, perhaps even her colleagues, perhaps even her entire community or civilization. She feels hopeless. She also puts in a lot of effort.
Carol doesn’t put in effort because she has epistemic hope (she doesn’t think her efforts are likely to lead to success) or emotional hope (she doesn’t feel some sense of energy or gratitude or “fighting against the odds” spirit).
Carol tries because… well, she just does. Maybe she tries because she thinks fighting is dignified. Maybe she thinks fighting makes sense under epistemic uncertainty, or she tries out of habit, or to impress her friends, or because she identifies as a person-who-just-tries, or because she has nothing better to do, or because she has something to protect.
When I wonder why some people think MIRI gave up (despite the fact that they’re still working on a variety of research, writing, outreach, and mentorship projects), I think it’s because people associate “giving up” with “losing hope.” If you declare that you think earth’s odds of survival are 0%, you have lost epistemic hope. If you write it in a way that somewhat somber and brooding and disappointed, you signal that you have lost emotional hope. And this pattern-matches onto someone who has stopped fighting. But it doesn’t have to.
You don’t have to fight. If you feel like you’re fighting just because you should, Nate Soares has written some posts that I recommend. You also don't have to fight all the time. Remember that "we cannot fight at maximum all the time, and some times are more important than others".
But you don’t have to stop fighting either. You can lose epistemic and emotional faith in your plan and still keep going.
It’s easier to fight when you have something you (epistemically or emotionally) believe in. But it’s not a requirement. Some people are fighting anyways, even when knowing that any given alignment plan has a small chance of working, even when feeling frustrated, disappointed, unheard, or unprepared.
Hopefully, you already have a plan that you think has a reasonable shot at helping, or you feel good about your plans despite the grim forecasts.
But if you don’t, and you still want to fight, you can. Hope does not have to be a prerequisite for fighting energy.
You can give up hope but stay in the fight.
How do people actually find motivation without hope?
This isn’t a rhetorical question. I hope you offer some ideas in the comments.
For now, here are a few things that I’ve found helpful during moments where I’ve been feeling low in hope:
Note: In the comments, I'd prefer avoiding discussions about how whether or not low hope is "justified." Instead, I'm curious to hear stories about how people summon fighting energy or respond in other ways during moments of low hope.