Can we talk about changing the world? Or saving the world?

I think few here would give an estimate higher than 95% for the probability that humanity will survive the next 100 years; plenty might put a figure less than 50% on it. So if you place any non-negligible value on future generations whose existence is threatened, reducing existential risk has to be the best possible contribution to humanity you are in a position to make. Given that existential risk is also one of the major themes of Overcoming Bias and of Eliezer's work, it's striking that we don't talk about it more here.

One reason of course was the bar until yesterday on talking about artificial general intelligence; another factor are the many who state in terms that they are not concerned about their contribution to humanity. But I think a third is that many of the things we might do to address existential risk, or other issues of concern to all humanity, get us into politics, and we've all had too much of a certain kind of argument about politics online that gets into a stale rehashing of talking points and point scoring.

If we here can't do better than that, then this whole rationality discussion we've been having comes to no more than how we can best get out of bed in the morning, solve a puzzle set by a powerful superintelligence in the afternoon, and get laid in the evening. How can we use what we discuss here to be able to talk about politics without spiralling down the plughole?

I think it will help in several ways that we are a largely community of materialists and expected utility consequentialists. For a start, we are freed from the concept of "deserving" that dogs political arguments on inequality, on human rights, on criminal sentencing and so many other issues; while I can imagine a consequentialism that valued the "deserving" more than the "undeserving", I don't get the impression that's a popular position among materialists because of the Phineas Gage problem. We need not ask whether the rich deserve their wealth, or who is ultimately to blame for a thing; every question must come down only to what decision will maximize utility.

For example, framed this way inequality of wealth is not justice or injustice. The consequentialist defence of the market recognises that because of the diminishing marginal utility of wealth, today's unequal distribution of wealth has a cost in utility compared to the same wealth divided equally, a cost that we could in principle measure given a wealth/utility curve, and goes on to argue that the total extra output resulting from this inequality more than pays for it.

However, I'm more confident of the need to talk about this question than I am of my own answers. There's very little we can do about existential risk that doesn't have to do with changing the decisions made by public servants, businesses, and/or large numbers of people, and all of these activities get us straight into the world of politics, as well as the world of going out and changing minds. There has to be a way for rationalists to talk about it and actually make a difference. Before we start to talk about specific ideas to do with what one does in order to change or save the world, what traps can we defuse in advance?

151 comments, sorted by
magical algorithm
Highlighting new comments since Today at 10:15 PM
Select new highlight date

It's not obvious that the best way to reduce existential risk is to actually work on the problem. Imagine if every farmer put down his plow and came to the university to study artificial intelligence research. Everyone would starve. It may well be that someone's best contribution is to continue to write software to do billing for health insurance, because that helps keep society running, which causes increased wealth, which then funds and supports people who specialize in researching risks among other fields.

I suspect that actually, only a small percentage of people, even of people here, could usefully learn the political truths relevant to existential risk mitigation via the kind of discussion you are proposing. Very few people are in a position to cause political change. The marginal utility gain for the average person to learn the truth on a political matter is practically zero due to his lack of influence on the political process. The many arguments against voting apply to this question as well, of seeking political truth; and even more so, because it's harder to ascertain political truths than to vote.

Most interest in politics is IMO similar to interest in sports or movies. It's fun, and it offers an opportunity to show off a bit, gives something to talk and socialize about, helps people form communities and define their interests. But beyond these kinds of social goals, there is no true value.

Most of the belief that one is in a position where knowing political truths is important, is likely to be self-deception. We see ourselves as being potentially more important and influential than we are likely ever to become. This kind of bias has been widely documented in many fields.

To me, politics is not so much the mind-killer as the mind-seducer. It leads us to believe that our opinions matter, it makes us feel proud and important. But it's all a lie. Politics is a waste of time and should be viewed simply as a form of entertainment. Now entertainment can be good, we all need a break from serious work and politics may be as valid as any other form of recreation, but we here should recognize that and not inflate its importance.

The set of people seriously working to reduce existential risks is very small (perhaps a few hundred, depending on who and how you count). This gives strong general reason to suppose that the marginal impact of an individual can be large, in cases where the individual aims to reduce existential risks directly and is strategic/sane/rational about how (and not in cases where the individual simply goes about their business as one of billions in the larger economy).

Many LW readers are capable of understanding that there are risks, thinking through the differential impact their donations would have on different kinds of risk mitigation, and donating money in a manner that would help. Fewer, but still many, are also capable of improving the quality of thought regarding existential risks in relevant communities (e.g., in the academic departments where they study or work, or on LW or other portions of the blogosphere). And while I agree with Hal's point that most politics is used as entertainment, there is reason to suppose that improving the quality of discussion of a very-high-impact, under-researched, tiny-numbers-of-people-currently-involved topic like existential risks can improve both (a) the well-directedness of resources like mine that are already being put toward existential risks, and (b) the amount of such resources, in dollars and in brainpower.

increased wealth, which then funds and supports people who specialize in researching risks

Would increased average wealth help risk-fighters more than risk-creators? It's not obvious to me either way. What does seem obvious is that from a utilitarian perspective society is hugely underinvesting in risk-fighting and everything else with permanent effects.

What does seem obvious is that from a utilitarian perspective society is hugely underinvesting in risk-fighting and everything else with permanent effects.

That's not obvious to me, and even if it were I don't take a utilitarian perspective.

If you think there is underinvestment in risk fighting you have to come up with arguments to persuade people that don't rely on a utilitarian perspective since most people don't take that perspective when making decisions. Or you can try and find ways of increasing investment that don't rely on persuading large numbers of people.

That utilitarianism implies one should do things with permanent effects comes from the future being much bigger than the present, and the probability of affecting it being smaller but not nearly proportionally smaller.

I agree with your second paragraph.

Even granting that, it's not obvious to me that society is underinvesting in risk fighting. Many of the suggestions for countering global warming for example imply reduced economic growth. It is not obvious to me that the risks of catastrophic global warming outweigh the expected losses from reduced growth from a utilitarian perspective. Any investment in risk fighting carries an opportunity cost in a foregone investment in some other area. The right choice from a utilitarian perspective depends on judgements of expected risk vs. the expected benefits of alternative courses of action. I think the best choices are far from obvious.

Wholly agree on global warming; the best reference I know of on extreme predictions is this. I'm thinking more of future technologies (the self-replicating and/or intelligent kind), but also of building up the general intellectual background and institutions to deal rationally with unknown unknowns.

The assumption being made here is that actions taken with the intent of reducing existential risk will actually have the effect of reducing it rather than increasing it. This assumption seems sadly unlikely to be correct.

"Actions taken with the intent to prevent event X make event X less likely" is going to be my default belief unless there's some strong evidence to the contrary.

Or, more particularly: "Actions taken after carefully asking what the evidence implies about the most effective means of making X less likely, and then following out the means with best expected value, make event X less likely".

mattnewport's counterexamples are good, but they are examples of what happens when "intent to reduce X" is filtered through a political system that incentivizes the appearance that something will be done, that penalizes public acknowledgement of unpleasant truths, and that does not understand science. There is reason to suppose we can do better -- at least, there's reason to assign a high enough probability to "we may be able to do better" for it to be clearly worth the costs of investigating particular issue X's.

There is reason to hope we can do better but a sobering lack of evidence that such hope is realistic. That's not a reason not to try but it seems we can agree that mere intent is far from sufficient.

Even supposing that it is possible to devise a course of action that we have good reason to believe will be effective, there is still a huge gulf to cross when it comes to putting that into action given current political realities.

Even supposing that it is possible to devise a course of action that we have good reason to believe will be effective, there is still a huge gulf to cross when it comes to putting that into action given current political realities.

This depends partly on what sort of "course of action" is devised, and how many people are needed to put it into action. Francis Bacon's successful spread of the scientific method, Louis Pasteur's germ theory, whoever it was who convinced doctors to wash their hands between childbirths, the invention of the printing press, and the invention of modern fertilizers sufficient to keep larger parts of the world fed... provide historical precedents for the idea that small groups of good thinkers can sometimes have predictably positive impacts on the world without extensively and directly engaging global politics/elections/etc.

There is reason to hope we can do better but a sobering lack of evidence that such hope is realistic.

[I'd edited my previous comment just before mattnewport wrote this; I'd previously left my comment at "There is reason to suppose we can do better", then had decided that that was overstating the evidence and added the "--at least...". mattnewport probably wrote this in response to the previous version; my apologies.]

As to evaluating the evidence: does anyone know where we can find data as to whether relatively well-researched charities do tend to improve poverty or other problems to which they turn their attention?

Alcohol prohibition, drug prohibition, the criminalization of prostitution, banking regulations designed to reduce bank failures due to excessive risk taking, bailing out automakers to prevent bankruptcy, policies designed to prevent terrorist attacks such as torturing prisoners... All are examples of actions taken with the intent to prevent X which have quite a lot of evidence to suggest that they did not make X less likely.

To what Matt said, I will add: actions taken with the intent of preventing harmful event X, often have no effect on X but greatly contribute to equally harmful event Y.

Most interest in politics is IMO similar to interest in sports or movies. It's fun, and it offers an opportunity to show off a bit, gives something to talk and socialize about, helps people form communities and define their interests. But beyond these kinds of social goals, there is no true value.

I'm not totally sure what you mean by this. With that said, it does matter very much how the government distributes its resources. While the government is admittedly inefficient, that doesn't mean that it can't be improved. Since politics determines how those resources are distributed, wouldn't becoming involved in politics be a valid and important way to gain your favored causes-i.e. existential risk mitigation- support? Declaring one method of gaining support to be automatically invalid, no matter the circumstances, won't help you.

The arguments against voting are mostly puerile, and so is this one against political judgment. See here for an alternative view.

Are there currently enough soldiers? What is the best way to recruit them? Existential risks is a high-payoff and generally-misunderstood issue. It looks like there is no strong community of professionals to work on it at the moment. In any case, there are existing organizations, and their merits and professional opinion should be considered before anyone commits to anything.

I agree with ciphergoth that we would probably have an easier time discussing political issues than some other communities, and I agree with HalFinney that it's probably not a very good use of our time anyway. Let's say that everyone on LessWrong agrees on a solution to some political problem. So what? We already have lots of good ideas no one will listen to. It doesn't take a long-time reader of Overcoming Bias to realize marijuana criminalization isn't working so well, but so far the efforts of groups with far more resources than ourselves have been mostly in vain.

If someone came up with a new idea for pulling ropes sideways, that might be useful. For example, Robin's idea of futarchy is interesting, although so large-scale that it would be very hard to implement. If someone came up with a suggestion that brilliant, but for a smaller problem, it might do some good. But I always interpreted our posting policies to permit that sort of thing anyway.

But I think the best thing we could possibly do would be to raise the sanity waterline - a rising tide lifts all boats. That means coming up with compact, attractive summaries of our key findings and spreading them as far as possible. More on this later, possibly.

Politics discussion by rationalists is likely to have the most impact when it's about issues that are important, but that aren't widely recognized as such and therefore have relatively few people pulling on the rope. I don't see any point in discussing the Iraq war, say.

Politics action by rationalists is likely to have the most impact on such topics. But since there are already some such topics we know about (global existential risk, for example, or teaching rationality in schools). What do we gain by discovering several more of these and then discussing them?

I agree that it is not a good use of our time to discuss political issue on Less Wrong. In fact, I think it would be harmful, because it would drown out other discussion and attract people who are not prepared to discuss it rationally.

However, we should discuss politics in other forums, using what we have learned here. We should be able to avoid seeing arguments as soldiers. I would like to spread rationality techniques among those who regularly participate in politics. (Though I am not sure how. Leading by example has been to subtle in my experience, and direct instruction leads to emotional defensiveness. It might be interesting to have debates moderated by a Less Wrong member, where it could be seen as their proper role to point out biases.)

It might be interesting to have debates moderated by a Less Wrong member, where it could be seen as their proper role to point out biases.

I immediately thought of the Confessors...

"We already have lots of good ideas no one will listen to."

this is my primary thought on all such sentiments. the best thing for people here to do would probably be to stop worrying about altruism and start trying to get rich. Once you're rich your altruism will actually mean something.

Most of you are rich by historical standards, and by the standards of the world. So think carefully about just how "rich" will be "enough" to "actually mean something."

I'm not sure your standard of wealth is the correct one. Most modern Americans aren't wealthy enough to hire full-time servants; by that standard of wealth there are probably more wealthy people in India, and were probably more wealthy Americans per capita in the 1920s.

I interpret NN's statement as follows: "the wealth distribution has a long tail, so that the majority of philanthropic impact is caused by outliers (Extremistan); it's more important to try to become an outlier yourself than to worry about whether to donate your yearly $50 to Greenpeace".

Most modern Americans aren't wealthy enough to hire full-time servants; by that standard of wealth there are probably more wealthy people in India, and were probably more wealthy Americans per capita in the 1920s.

Don't you think modern household convenience machines are more useful than a servant? Think of electric lights, dishwashers, clotheswashers, personal computers, &c., &c.

money is a stand in for other (harder to quantify) metrics for impact on the future. resource distribution in general would be better if it were allocated rationally would it not? thus we should try to take as much control of resource distribution as we can. In contrast you're speaking from the perspective of satisfying material wants, by which standard we all already live like kings of other ages.

what traps can we defuse in advance?

We care about saving the world and we care about the truth, so sometimes we start caring too much about the ideas that we think represent those things. How can we foster detachment? How can we encourage people to consider an idea even if they don't like it, and then encourage people to relinquish an idea after it's been considered and evenly rejected?

The following paradigm has worked for me:

It's natural to be afraid of considering an idea that we know is false. Thus it is useful to occasionally practice considering ideas that we don't like in order to find that nothing bad happens to our brains or ourselves when we consider them.

It is really important to be able to consider bad ideas, not just because the bad idea might be a good idea, but because it is only through empathy (identification) with an idea that you will be able to find the right counter-argument that will encourage the holder of the idea to relinquish it. (Otherwise, as I'm sure you've observed, the same argument just gets presented another way, another time.)

it is only through empathy (identification) with an idea that you will be able to find the right counter-argument that will encourage the holder of the idea to relinquish it. (Otherwise, as I'm sure you've observed, the same argument just gets presented another way, another time.)

Related: Is That Your True Rejection?, Words as Mental Paintbrush Handles (arguments as paintbrush handles for emotional responses).

Eliezer's counter-argument in "The Pascal's Wager Fallacy Fallacy" is an example of this mistake. Arguments from the Pascal's Wager Fallacy aren't paintbrush handles for expected utility computations, they're paintbrush handles for the fear of being tricked in confusing situations and the fear of exhibiting markers of membership in a ridiculed or rejected group.

Here's Wikipedia's list of Forbidden Words, which I think has some good examples of how language can be subtly loaded on controversial / emotionally charged issues. Diligently watching out for that sort of thing is probably one of the best things we could do to avoid political discussions degenerating.

That doesn't cut it. An easy-to-use, fairly effective technique, but not a game-defining one. Try enforcing that on a random crowd.

We could consider making a list of similar guidelines that we wouldn't want to enforce generally, but that together could provide a sort of cognitive clean room to discuss super-touchy subjects in. "Never mention how someone's false beliefs could arise from flaws in their personality even when that's actually happening" seems like another important one. Probably ban sarcasm. Possibly even ban anecdotes and analogies.

"Never mention how someone's false beliefs could arise from flaws in their personality even when that's actually happening"

If two people have a persistent disagreement of fact, eventually the inescapable conclusion is that they do not fully trust one another for rationalists. Exploring how this came to be the case is the first step to changing the situation.

I think ideally what we need is a space in which we can suggest flaws in a person's personality, and still be friends the next day. Is that possible?

Discussions among rationalists needn't involve differences of opinion; they can instead involve differences of personal impression. That said, there are real differences of opinion among rationalists. I'm not sure, however, that we need to resort to psychoanalysis to resolve them -- after all, argument screens off personality.

We could consider making a list of similar guidelines that we wouldn't want to enforce generally, but that together could provide a sort of cognitive clean room to discuss super-touchy subjects in.

Great idea. I'd say the biggest useful guideline here is that on mind-killing subjects we should make a norm of only saying the pieces we actually know. That is, we should cite evidence for all conclusions, or, better still, cite the real causes of our beliefs, and we should keep our conclusions really carefully to only what is almost tautologically implied by that evidence. We should be extra-precise. And we should not, really really not, bringing in extraneous issues if there's any way to avoid them.

When people try to talk about AI risks, say, without background, they often come up with plausible this and plausible that, and the topics and misconceptions multiply faster than one can sort them out. Whereas interested interlocutors even without much rationality background who have taken the time to sort through the sub-issues one at a time, slowly, sorting through the causes of each intuition and the sum total of evidence on that point, in my experience generally have managed useful conversations.

That's just generally raising the level of fallacy alert, maybe specifically around the politics-induced fallacies. It should be default behavior whenever the fallacious arguments start raining down, around any issue. A typical battle ground for x-rationality skills in action, not a special case.

There's a difference between just being hypersensitive to bad reasoning (usually a good idea), and being hypersensitive to anything that could directly or indirectly cause emotions to flare up (usually not worth the bother).

being hypersensitive to anything that could directly or indirectly cause emotions to flare up (usually not worth the bother)

Molybdenumblue said it really well elsewhere:

If you believe that other human beings are a useful source of insight, you would do well to make some effort not to offend.

Yes, hypersensitivity is by definition uncalled for, but when attempting to communicate with human beings and encourage their reply, it's clearly useful to choose words which are less likely to invoke negative emotions. It's possible to keep the juggling balls of precision, reason, and sensitivity all in the air at the same time; that it can be difficult is not sufficient reason not to try.

Hence I mentioned escalation of your level of sensitivity, meaning to refer to any factors that (potentially) deteriorate constructive thinking. Being hypersensitive to bad reasoning isn't always a good idea, for example if you don't care to reeducate the interlocutor.

Survey: are you motivated to improve or save the world?

This survey aims to determine if there is significant consensus or disparity. It is in response to the datapoint presented here.

If you would like to qualify or explain your response, feel free to do so as a comment to the appropriate response.

  • Note that this is a general solution to the problem of conducting a quick off-the-cuff survey on LW without affecting karma, but you need to be able to view negative scoring comments.

  • If you want to leave me with positive karma, please keep the survey neutral and vote the parent (this) up instead.

Upvote here if your answer is 'yes'.

(And downvote the downvote post to neutralize karma.)

Upvote here if your answer is 'no'.

(And downvote the downvote post to neutralize karma.)

I think it will be very necessary to carefully frame what it would be that we might wish to accomplish as a group, and what not. I say this because I'm one of those who thinks that humanity has less than a 50% chance of surviving the next 100 years, but I have no interest in trying to avert this. I am very much in favour of humanity evolving into something a lot more rational than what it is now, and I don't really see how one can justify saying that such a race would still be 'humanity'. On the other hand, if the worry is the extinction of all rational thought, or the extinction of certain, carefully chosen, memes, I might very well wish to help out.

The main problem, as I see it, is in being clear on what we want to have happen (and what not) and what we can do to make the preferred outcomes more likely. The more I examine the entire issues, the harder it appears to define how to distinguish between the good and the bad outcomes.

I wonder how many rationalists share this view. If a significant number, it would be worthwhile to even discuss this first, in hopes to muster a broader consensus about what the group should do or even to just be aware of the reasons for lack of agreement.

If politicians start following expected utility consequentialism, special interest groups will be able to exploit the system by manufacturing in themselves "offense" (extreme emotional disutility) at unfavored measures, forcing your maximizer to give in to their demands. To avoid this, you need a procedure for distinguishing "warranted" offense from "unwarranted" offense: some baseline of personal rights ultimately derived from something other than self-assessed emotional utility.

If you see a way around this difficulty, let me know, because it seems insurmountable to me right now. Until we sort this out, I find it hard to talk about politics from a consequentialist standpoint, because most successful interest groups today are already heavily using the exploit I've described.

I don't see the object of attack in the room. An exploration of potential utility-maximization political frameworks and their practical pitfalls would possibly be interesting, although in practice I expect this sort of institution to turn into a kind of market, not so much politician-mediated.

I meant to attack this part of ciphergoth's post:

We need not ask whether the rich deserve their wealth, or who is ultimately to blame for a thing; every question must come down only to what decision will maximize utility.

For example, framed this way inequality of wealth is not justice or injustice. The consequentialist defence of the market recognises that because of the diminishing marginal utility of wealth, today's unequal distribution of wealth has a cost in utility compared to the same wealth divided equally, a cost that we could in principle measure given a wealth/utility curve, and goes on to argue that the total extra output resulting from this inequality more than pays for it.

I didn't intend to criticize any real or hypothetical political system. The same emotional exploit could easily defeat a community of rationalists independently evaluating political measures for their utility, as ciphergoth seems to propose.

The same emotional exploit could easily defeat a community of rationalists independently evaluating political measures for their utility

Well, since you've easily recognized this exploit already at the hypothetical stage, this kind of vulnerability won't be a problem. Any consequentialist framework should be able to fight moral sabotage, for example by introducing laws that disincentivize it.

Before disincentivizing, you face the problem of defining and recognizing moral sabotage. It doesn't sound trivial to me. Remember, groups don't admit to using the outrage tactic; they do it sincerely, sometimes over several generations of members. I repeat the question: how does a rationalist tell "warranted" emotional disutility from "unwarranted" in a fair way?

Incentive effects are hugely important, but a utilitarian decision process that causes predictable harm is not a true utilitarian decision process. Your question is a tough one, but it's answerable in principle.

I don't see the problem in principle with a utilitarian deciding that giving in to an instance of moral sabotage will greatly increase later incidence of moral sabotage, resulting in total disutility greater than the manufactured weeping and gnashing of teeth you face if you stand against it now.

So a powerful agent (or a mass of tiny agents with large total power) needs a different utility function on future worlds than that of a lone rationalist observer, due to the need to avoid exploits. Well... which should I pick, then?

Looks like we've run into another of those nasty recursive problems: I choose my utility function depending on what every other agent could do to exploit me, and everyone else does the same. The only natural solution might well turn out to be everyone caring about their own welfare and no one else's, to avoid "mugging by suffering". Let's model the problem mathematically and look for other solutions - I love this stuff.

So a powerful agent (or a mass of tiny agents with large total power) needs a different utility function on future worlds than that of a lone rationalist observer, due to the need to avoid exploits.

No, it needs a different method of maximizing expected utility. Avoiding moral sabotage doesn't reflect a preference, it's purely instrumental.

A related idea: moral sabotage is what happens when one player in the Ultimatum game insists on taking more than a fair share, even if what fare share is depends on his preferences.

Come on, Ciphergoth, the problem of saving humanity would be too easy if you could convince a large number of humans to go along with your proposals! You have a harder challenge: save humanity in spite of the apathy, and in many cases intransigent opposition, of the humans.

I have a hard time believing that anyone in power is serious about saving humanity. There are so many obvious and easy things that could be done, that would clearly be enormously helpful, that no one with power is doing or even suggesting. Politics is almost entirely a signalling game.

As several people have been starting to suggest, it's likely that the best strategy for rationalists involves two steps:

  • accumulate massive wealth and power
  • use it to save humanity

And probably the first step is far harder than the second.

I will admit to an estimate higher than 95% that humanity or its uploads will survive the next hundred years. Many of the "apocalyptic" scenarios people are concerned about seem unlikely to wipe out all of humanity; so long as we have a breeding population, we can recover.

No significant risk of unFriendly AI (especially since you apparently consider uploading within 100 years plausible)? Nanotech war? Even engineered disease? I'm surprised.

The comment appears to me to be saying there is no significant risk of wiping out all of humanity, not that there is no significant risk of any of the dangers you describe causing significant harm.

I think an unfriendly AI is somewhat likely for example but put a very low probability on an unfriendly AI completely wiping out humanity. The consequences could be quite unpleasant and worth working to avoid but I don't think it's an existential threat with any significant probability.

That's a very strange perspective. Other threats are good in that they are stupid, so they won't find you if you colonize space or live on an isolated island, or have a lucky combination of genes, or figure out a way to actively outsmart them, etc. Stupid existential risks won't methodically exterminate every human, and so there is a chance for recovery. Unfriendly AI, on the other hand, won't go away, and you can't hide from it on another planet. (Indifference works this way too, it's the application of power indifferent to humankind that is methodical, e.g. Paperclip AI.)

Consider: humanity is an intelligence, one not particularly friendly to, say, the fieldmouse. Fieldmice are not yet extinct.

I think it is worth considering the number of species to which humanity is largely indifferent which are extinct as a result of humanity optimizing other criteria

Humans satisfice, and not very well at that compared to what an AGI could do. If we effectively optimized for... almost any goal not referring to fieldmice... fieldmice would be extinct.

Humanity is pretty damn impressive from a fieldmouse's perspective, I dare say!

yet humanity cannot create technology on the level of a fieldmouse.

Fieldmice (outside of Douglas Adams fiction) aren't any particular threat to us in the way we might be to the Unfriendly AI. They're not likely to program another us to fight us for resources.

If fieldmice were in danger of extinction we'd probably move to protect them, not that that would necessarily help them.

Unfriendly AI, on the other hand, won't go away, and you can't hide from it on another planet.

Not on another planet, no. But I wonder how practical a constantly accelerating seed ship will turn out to be.

You are assuming that mere intelligence is sufficient to give an AI an overwhelming advantage in any conflict. While I concede that is possible in theory I consider it much less likely than seems to be the norm here. This is partly because I am also skeptical about the existential dangers of self replicating nanotech, bioengineered viruses and other such technologies that an AI might attempt to use in a conflict.

As long as there is any reasonable probability that an AI would lose a conflict with humans or suffer serious damage to its capacity to achieve its goals, its best course of action is unlikely to be to attempt to wipe out humanity. A paperclip maximizer for example would seem to better further its goals by heading to the asteroid belt where it could advance its goals without needing to devote large amounts of computational capacity to winning a conflict with other goal-directed agents.

For people who've voted this down, I'd be interested in your answers to the following questions:

1) Can you envisage a scenario in which a greater than human intelligence AI with goals not completely compatible with human goals would ever choose a course of action other than wiping out humanity?

2) If you answered yes to 1), what probability do you assign to such an outcome, rather than an outcome involving the complete annihilation of humanity?

3) If you answered no to 1), what makes you certain that such a scenario is not possible?

I agree generally, but I think when we talk about wiping out humanity we should include the idea that if we were to lose a significant portion of our accumulated information it would be essentially the same as extinction. I don't see a difference between a stone age tech. group of humans surviving the apocalypse and slowly repopulating the world and a different species (whether dogs, squirrels, or porpoises) doing the same thing.

I don't see a difference between a stone age tech. group of humans surviving the apocalypse and slowly repopulating the world and a different species (whether dogs, squirrels, or porpoises) doing the same thing.

See In Praise of Boredom and Sympathetic Minds: random evolved intelligent species are not guaranteed to be anything we would consider valuable.

We have pretty solid evidence that a stone age tech group of humans can develop a technologically advanced society in a few 10s of thousands of years. I imagine it would take considerably longer for squirrels to get there and I would be much less confident they can do it at all. It may well be that human intelligence is an evolutionary accident that has only happened once in the universe.

The squirrel civilization would be a pretty impressive achievement, granted. The destruction of this particular species (humans) would seemingly be a tremendous loss universally, if intelligence is a rare thing. Nonetheless, I see it as only a certain vessel in which intelligence happened to arise. I see no particular reason why intelligence should be specific to it, or why we should prefer it over other containers should the opportunity present itself. We would share more in common with an intelligent squirrel civilization than a band of gorillas, even though we would share more genetically with the latter. If I were cryogenically frozen and thawed out a million years later by the world-dominating Squirrel Confederacy, I would certainly live with them rather than seek out my closest primate relatives.

EDIT: I want to expand on this slightly. Say our civilization were to be completely destroyed, and a group of humans that had no contact with us were to develop a new civilization of their own concurrent with a squirrel population doing the same on the other side of the world. If that squirrel civilization were to find some piece of our history, say the design schematics of an electric toothbrush, and adopt it as a part of their knowledge, I would say that for all intents and purposes, the squirrels are more "us" than the humans, and we would survive through the former, not the latter.

I don't see any fundamental reason why intelligence should be restricted to humans. I think it's quite possible that intelligence arising in the universe is an extremely rare event though. If you value intelligence and think it might be an unlikely occurrence then the survival of some humans rather than no humans should surely be a much preferred outcome?

I disagree that we would have more in common with the electric toothbrush wielding squirrels. I've elaborated more on that in another comment.

Preferred, absolutely. I just think that the survival of our knowledge is more important than the survival of the species sans knowledge. If we are looking to save the world, I think an AI living on the moon pondering its existence should be a higher priority than a hunter-gatherer tribe stalking wildebeest. The former is our heritage, the latter just looks like us.

Does this imply that you are OK with a Paperclip AI wiping out humanity, since it will be an intelligent life form much more developed than we are?

If I implied that, it was unintentional. All I mean is that I see no reason why we should feel a kinship toward humans as humans, as opposed to any species of people as people. If our civilization were to collapse entirely and had to be rebuilt from scratch, I don't see why the species that is doing the rebuilding is all that important -- they aren't "us" in any real sense. We can die even if humanity survives. By that same token, if the paperclip AI contains none of our accumulated knowledge, we go extinct along with the species. If the AI contains some our of knowledge and a good degree of sentience, I would argue that part of us survives despite the loss of this particular species.

Bear in mind, the paperclip AI won't ever look up to the broader challenges of being a sentient being in the Universe; the only thing that will ever matter to it, until the end of time, is paperclips. I wouldn't feel in that instance that we had left behind a creature that represented our legacy, no matter how much it knows about the Beatles.

OK, I can see that. In that case, maybe a better metric would be the instrumental use of our accumulated knowledge, rather than its mere possession. Living in a library doesn't mean you can read, after all.

What I think you're driving at is that you want it to value the Beatles in some way. Having some sort of useful crossover between our values and its is the entire project of FAI.

I'm just trying to figure out under what circumstances we could consider a completely artificial entity a continuation of our existence. As you pointed out, merely containing our knowledge isn't enough. Human knowledge is a constantly growing edifice, where each generation adds to and build upon the successes of the past. I wouldn't expect an AI to find value in everything we have produced, just as we don't. But if our species were wiped out, I would feel comfortable calling an AI which traveled the universe occasionally writing McCartney- or Lennon-inspired songs "us." That would be survival. (I could even deal with a Ringo Starr AI, in a pinch.)

I strongly suspect that that is the same thing as a Friendly AI, and therefore I still consider UFAI an existential risk.

The Paperclip AI will optimally use its knowledge about the Beatles to make more paperclips.

How much of what it means to be human do you think is cultural conditioning versus innate biological tendency? I think the evidence points to a very large biologically determined element to humanity. I would expect to find more in common with a hunter gatherer in a previously undiscovered tribe, or even with a paleolithic tribesman, than with an alien intelligence or an evolved dolphin.

If you read ancient Greek literature, it is easy to empathize with most of the motivations and drives of the characters even though they lived in a very different world. You could argue that our culture's direct lineage from theirs is a factor but it seems that westerners can recognize as fellow humans the minds behind ancient Chinese or Indian texts with less shared cultural heritage with our own.

I don't consider our innate biological tendencies the core of our being. We are an intelligence superimposed on a particular biological creature. It may be difficult to separate the aspects of one from the other (and I don't pretend to be fully able to do so), but I think it's important that we learn which is which so that we can slowly deemphasize and discard the biological in favor of the solely rational.

I'm not interested in what it means to be human, I want to know what it means to be a person. Humanity is just an accident as far as I'm concerned. It might as well have been anything else.

I'm curious as to what sorts of goals you think a "solely rational" creature possesses. Do you have a particular point of disagreement with Eliezer's take on the biological heritage of our values?

Oh, I don't know that. What would remain of you if you could download your mind into a computer? Who would you be if you were no longer affected by the level of serotonin or adrenaline you are producing, or if pheromones didn't affect you? Once you subtract the biological from the human, I imagine what remains to be pure person. There should be no difference between that person and one who was created intentionally or one that evolved in a different species, beyond their personal experiences (controlling for the effects of their physiology).

I don't have any disagreement with Eliezer's description of how our biology molded our growth, but I see no reason why we should hold on to that biology forever. I could be wrong, however. It may not be possible to be a person without certain biological-like reactions. I can certainly see how this would be the case for people in early learning stages of development, particularly if your goal is to mold that person into a friendly one. Even then, though, I think it would be beneficial to keep those parts to the bare minimum required to function.

What would remain of you if you could download your mind into a computer?

That depends on the resolution of the simulation. Wouldn't you agree?

Once you subtract the biological from the human, I imagine what remains to be pure person.

I think you're using the word "biological" to denote some kind of unnatural category.

I don't have any disagreement with Eliezer's description of how our biology molded our growth, but I see no reason why we should hold on to that biology forever.

The reasons you see for why any of us "should" do anything almost certainly have biologically engineered goals behind them in some way or another. What of self-preservation?

Not unnatural, obviously, but a contaminant to intelligence. Manure is a great fertilizer, but you wash it off before you use the vegetable.

I meant this kind of unnatural category. I don't quite know what you mean by "biological" in this context. A high-resolution neurological simulation might not require any physical carbon atoms, but the simulated mind would presumably still act according to all the same "biological" drives.

One observation and a related suggestion:

(1) We've gone off-topic regarding the demands of this post. Ciphergoth asks what traps can we defuse in advance, before we start to talk about specific ideas to do with what one does in order to change the world. However, I'm neutral about not following instructions -- perhaps Ciphergoph hasn't asked the right question after all, and we need to triangulate towards the right question.

(2) I've got no idea how to begin answering some of the other problems that are being posed. (E.g., how can we best help the world?) So I like Vladimir_Nesov's reminder that there is a post about the value of not proposing solutions until the problem has been discussed as thoroughly as possible. I think we might try it to see if it works in practice. So let's talk about the problem/problems. What is it / what are they?

My impression is that the material covered on OB/LW is more than sufficient to allow people that really understand the material to talk politics without exploding. I don't think we need any politics specific tricks for those that are likely to be helpful contributors.

This came up in the Santa Barbara LW meetup, and I felt like that group could have talked politics the right way. The implicit consensus seemed to be "Yeah, it'd probably work", though we didn't try.

Of course, with a smaller group and stronger selection pressures it is less likely to have a group member or two that can't hold it together when compared to a larger and online community (ie LW).

To summarize: I think there are plenty of people here that can handle it, but probably enough that can't to ruin it if there aren't any preventative measures. Hopefully a merciless downvoting policy will be enough, but failing that, I'm sure we could come up with a way to select a subset of people that are allowed to talk politics.

You also need to sufficiently care about the specific question to work on it, which is not a given. Less general, less popular.

Im not sure its just a matter of rationality (which it is), but also of complexity, ie predicting or estimating utility for policy A vs B can be impossible to model because of chaotic effects etc.

Just because most of the mistakes we see when people argue politics are rather obvious (from a rationalistic pov) doesnt mean they are the only ones. Otherwise social science and economics would be sciences, with capital S.

Richard Posner on the economics of the flu epidemic:

We need an overall "catastrophe budget" that would match expenditures to the net expected benefits of particular measures targeted at particular catastrophic threats.

I think few here would give an estimate higher than 95% for the probability that humanity will survive the next 100 years; plenty might put a figure less than 50% on it.

For the record I would put it at levels overwhelmingly higher than 95%. More like 99.999%.

You can't get away with having such extreme probabilities when a bunch of smart and rational people disagree. There are reasons why the whole Aumann agreement thing doesn't work perfectly in real life, but this is an extreme failure.

If a bunch of people on LW think it's only 50% likely and you think theres only a 0.1% chance that they're right and you're wrong (which is already ridiculously low) it still brings your probability estimate down to around 99.95. This is a 50 fold increase in the probability that the world is going to end over what you stated. Either you have some magic information that you haven't shared, or you're hugely overconfident.

http://lesswrong.com/lw/9x/metauncertainty/ http://lesswrong.com/lw/3j/rationality_cryonics_and_pascals_wager/69t#comments

You cannot selectively apply Aumann agreement. If you want to count tiny bunch of people who believe in AI foom, you must also take into account 7 billion people, many of them really smart, who definitely don't.

I don't have this problem, as I don't really believe that using Aumann agreement is useful with real humans.

Or you could count my awareness of insider overconfidence as magic information:

http://www.overcomingbias.com/2007/07/beware-the-insi.html

This is Less Wrong we're talking about. Insider overconfidence isn't "magic information".

See my top level post for a full response.

Large groups of smart people are frequently wrong about the future, and overwhelmingly so about the non-immediate future. 0.1% may be low but it's not ridiculously so.

(Also "they're right and you're wrong" is redundant. This has nothing to do with any set of scenario probabilities being "right". And any debate of "p=.9" "no, p=.1" is essentially silly because it misunderstands both the meaning of probability as a function of knowledge and our ability to create models which give meaningfully-accurate probabilities.)

And any debate of "p=.9" "no, p=.1" is essentially silly because it misunderstands both the meaning of probability as a function of knowledge and our ability to create models which give meaningfully-accurate probabilities.

Subjective probability is (in particular) a tool for elicitation of model parameters from expert human gut-feelings, which you can then use to find further probabilities and align them with other gut-feelings and decisions, gaining precision from redundancy and removing inconsistencies. The subjective probabilities don't promise to immediately align with physical frequencies, even where the notion makes sense.

It is a well-studied and useful process, you'd need a substantially more constructive reference than "it's silly" (or you could just seek a reasonable interpretation).

As you explain it, it's not silly.

Do you have a link for a top-level post that puts this kind of caveat on probability assignments? Personally, I think that if most people here understood it that way, they'd use more qualified language when talking about subjective probability. I also think that developing and standardizing such qualified language would be a useful project.

It is the sense in which the term "probability" is generally understood on OB/LW, with varying levels of comprehension by specific individuals. There are many posts on probability, both as an imprecise tool and an ideal (but subjective) construction. They should probably be organized in the Bayesian probability article on the wiki. In the meantime, you are welcome to look for references in the Overcoming Bias archives.

You may be interested in the following two posts, related to this discussion:
Probability is in the Mind
When (Not) To Use Probabilities

Without even going into different specific risks, you should beware the conjunction fallacy (or, more accurately, its flip side) when assigning such a high probability. A lack of details tends to depress estimates of an event that could occur as a result of many different causes, since if you aren't visualizing a full scenario it's tempting to say there's no way for it to occur.

You're effectively asserting that not only are all of the proposed risks to humanity's survival this minuscule in aggregate, but that you're also better than 99.9% confident that there won't be invented or discovered anything else that presents a plausible existential threat. How do you arrive at such confidence of that?

I myself would be disappointed if over half of LW put the probability of a single biological human (not an upload, not a reconstruction - an actual descendent with the appropriate number of ancestors alive today) alive in 100 years under 95%. I would consider that to be a gross instance of all kinds of biases. I'm not going to argue about scenarios, here, just point out that there any scenario which tends inevitably to wipe out humanity within one lifetime is totally unimaginable. That doesn't mean implausible, but it does mean improbable.

Personally, I do not believe that any person, group of people, or human-built model to date can consistently predict the probability of defined classes of black-swan events ("something that's never happened before which causes X" where X is a defined consequence such as humanity's extinction) to within even an order of magnitude for p/(1-p). I doubt anybody can get even to within two orders of magnitude consistently. (I also doubt that this hypothesis of mine will be clearly decidable within the next 20 years, so I'm not particularly inclined to listen to philosophical arguments from people who'd like to discard it.)

What I'm saying is, we should stop trying to put numbers on this without big error bars. And I've yet to see anybody propose an intelligent way to deal with probabilities like 10^(-6 +/- 4); just meta-averaging it over the distribution of possible probabilities, to come up with something like 10^-3 seems to be discarding data and to lead to problems. However, that's the kind of probability I'd put on this lemma. ("Earth made uninhabitable by normal cosmic event and rescue plans fail" would probably put a floor somewhere above 10^-22 per year.)

"The chance we're all wrong about something totally unprecedented has got to be less than 99.9%" is total hubris. Yes, totally unprecedented things happen every day. But telling yourselves stories about AGI and foom does not make these stories likely.

This is not, by the way, an argument to ignore existential risk. Even at the 10^-6 (or, averaged over meta-probabilities, 10^-3) level which I estimated, it is clearly worth thinking about, given the consequences. But if you're all getting that carried away, then Less Wrong should just be renamed More Wrong.

Oh, also, I'd accept that the risk of humanity being seriously hosed within 100 years, or extinct within 1000 years, is significant - say, 10^(-3 +/- 4) which meta-averages to something like 15%.

("Seriously hosed" means gigadeath events, total enslavement, or the like. Note that we're already moderately hosed and always have been, but that seriously hosed is still distinguishable.)

I myself would be disappointed if over half of LW put the probability of a single biological human alive in 100 years under 95%.

This is an assertion of your confidence in extinction risk being below 5%.

Personally, I do not believe that any person, group of people, or human-built model to date can consistently predict the probability of defined classes of black-swan events
[...]
This is not, by the way, an argument to ignore existential risk. Even at the 10^-6 (or, averaged over meta-probabilities, 10^-3) level which I estimated, it is clearly worth thinking about, given the consequences.

Not understanding a phenomenon, being unable to estimate its probability, doesn't give you an ability to place its probability below a strict bound. Your assertion of confidence contradicts your assertion of confusion.

I have confidence that nobody here has secret information that makes human extinction much more likely - because almost no information which currently exists could have more than a marginal bearing on a result which, if likely, is a result of human (that is, intelligent) interaction. Therefore I have confidence that the difference in estimates is largely not due to information, but to models. I have confidence that inductive models - say, "how often does a random species survive any hundred year period, correcting for initial population" give answers over 95% which should be considered the default. Therefore, I have confidence that a community of people who generally give lower estimates is subject to some biases (such as narrative bias).

Doesn't mean LW's wrong and I'm right. But to believe that human extinction within a century is likely clearly puts LW in the minority of humanity in your beliefs - even in the minority of rational atheists. And the fact that there is substantial agreement within the LW community on this, when uncertainty is clearly so high that orders of magnitude of disagreement are possible, makes me suspect bias.

Also, I find it funny that people will argue passionately over estimates that differ in log(p/q) from -1 to +1 (~10% to ~90%), but couldn't care less over the difference from say -9 to -7 (.0001% vs .000001%) or 7 to 9. This is in one sense the right attitude for people who think they can do something about it, but it ends up biasing numbers towards log(p/q)=0 [ie 50%], since you are more likely to get argument from somebody who has an estimate on the other side of 50% as yours is.

The fact that we believe something unusual is only weak evidence for the validity of that unusual belief, you are right on that. And given the hypothesis that we are wrong, which is dominant while all you have is the observation that we believe something unusual, you can draw a conclusion that we are wrong because of some systematic error of judgment that makes most here to claim the unusual belief.

To move past this point, you have to consider the specific arguments, and decide for yourself whether to accept them.

Most of the beliefs people can hold intuitively are about 50% in certainty. The beliefs far away from this point aren't useful as primitive concepts, classifying the possible events on one side or the other, as most everything is only on one side, and human mind can't keep track of their levels of certainty. New concepts get constructed, that are more native to human mind and express the high-certainty concepts in question only in combinations, or that are supported by non-intuitive procedures for processing levels of certainty. But if the argument is dependent on use of intuition, you aren't always capable of moving towards certainty, so you remain in doubt. This is the case for unknown unknowns, in particular.