Other Existential Risks

[-]CarlShulman15y120

Regarding D) it depends on why the risks are getting varying amounts of attention. Existential risks mainly get derivative attention as a result of more likely/near-term/electorally-salient/commonsense-morality-salient lesser forms. For instance, engineered diseases get countermeasure research because of the threat of non-extinction-level pathogens causing substantial casualties, not the less likely and more distant scenario of a species-killer. Anti-nuclear measures are driven mostly by the expected casualties from nuclear war than the chance of surprisingly powerful nuclear winter, etc. Climate change prevention is mostly justified in non-existential risk terms, and benefits from a single clear observable mechanism already in progress that fits many existing schema for environmentalism and dealing with pollutants.

The beginnings of a similar derivative effort are visible in the emerging "machine ethics" area, which has been energized by the development of Predator drones and the like, although it's noteworthy how little was done on AI risk in the early, heady days of AI, when researchers were relatively confident in success soon.

Regarding A), I'll have more to say at ano... (read more)

6multifoliaterose15y

Thanks Carl, I'm glad to finally be getting some engagement concerning (A). I will think about these things.

[-]John_Maxwell15y80

I interpret this to be a statement of the type "You should believe SIAI's claims (1) and (2) because we're really smart."

No, it's a statement of the type "You should believe SIAI's claims (1) and (2) because we're really rational." Your mathematician may have been smart and not rational. I remember reading about the phenomenon of smart non-rational people, maybe here: http://www.magazine.utoronto.ca/feature/why-people-are-irrational-kurt-kleiner/

Anyway, your mathematician is a terrible example of an irrational person because he was... (read more)

[-]jimmy15y80

EY argues: "... your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don't."

and you respond by saying that there have been people smarter than Eliezer that have suffered rationality fails when working outside their domain? Isn't that kinda the point?

EY wasn't arguing "My IQ is so damn high that I just have to be right. Look at my ability to g... (read more)

7Paul Crowley15y

More specifically, XiXiDu's whole point was "how do I evaluate this if, instead of addressing the arguments behind it, I talk about who believes it and who doesn't?" If that's the argument, it's fair enough for Eliezer to ask them to assess the rationality of the people whose opinions are being weighed.

0[anonymous]15y

More specifically my point regarding other peoples beliefs was that there are people who know about the topic of superhuman AI and related risks but, judged by their less or non-existing campaigns to prevent the risks, came to different conclusions. Reference: The Singularity: An Appraisal (Video) - Alastair Reynolds, Vernor Vinge, Charles Stross, Karl Schroeder In the case of other people like Marvin Minsky and other AI researchers, amongst others, the knowledge of possible risks should be reasonable to infer from their overall knowledge of the topic. Many scientists disregard speculations concerning the interpretation of quantum mechanics. This is due to the fact that it does not bear additional predictions, i.e. is not subject to empirical criticism. I disagree based on the following evidence: http://xixidu.net/lw/05.png "At present I do not know of any other person who could do that." (Reference) Hypothesis based on shaky conclusions, not on previous evidence.

3XiXiDu15y

More specifically my point regarding other peoples beliefs was that there are people who know about the topic of superhuman AI and related risks but, judged by their less or non-existing campaigns to prevent the risks, came to different conclusions. Reference: The Singularity: An Appraisal (Video) - Alastair Reynolds, Vernor Vinge, Charles Stross, Karl Schroeder In the case of AI researchers like Marvin Minsky, amongst others, the knowledge of possible risks should be reasonable to infer from their overall familiarity with the topic. I disagree based on the following evidence: * http://xixidu.net/lw/05.png * "At present I do not know of any other person who could do that." (Reference) Hypothesis based on shaky conclusions, not on previous evidence.

9[anonymous]15y

You keep posting screenshots from the deleted Roko's post, with the "forbidden" parts blacked-out. I agree that the whole matter could have been handled much better, but I don't see how it or the other quoted line bears on the interpretation of the sentence quoted at the top of jimmy's post. Also, people have asked you several times to stop reminding them of the deleted post and the need for quotes proving that EY thinks highly of his intelligence can be satisfied without doing that. Seriously, they're everywhere.

-4[anonymous]15y

XiXiDu argues: "... your smart friends at Less Wrong and favorite rationalists like EY are not remotely close to the rationality standards of other people out there (yeah, there are other smart people, believe it or not), and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don't." You keep telling me that my arguments are no evidence for what I'm trying to prove. Other people asked me several times not to make up fantasies of AI-Gods kicking their testicles. But if you want to be upovted the winning move is just to go think about something else. So take my word for it, I know more than you do, no really I do, and SHUT UP.

3wedrifid15y

I actually feel embarrassed just from reading that.

0jimmy15y

See the edit to the original comment.

0[anonymous]15y

If only claims (1) and (2) had been critically analyzed in detail on Less Wrong or the SIAI website I would find your comment compelling. Given that such analysis has not been made or released, I interpret Eliezer's response as an argument by authority.

-1multifoliaterose15y

If only there had been detailed critical analysis of claims (1) and (2) on Less Wrong or the SIAI website I would find your comment compelling. But in light of the fact that detailed critical analysis of these significant claims has not taken place I believe that Eliezer's remarks are in fact properly conceptualized as an appeal to authority.

1jimmy15y

I totally agree that it's an appeal to authority. My point was that it's an appeal to a different and more relevant kind of authority.

0multifoliaterose15y

Do you disagree with ? If so, why?

3jimmy15y

Yes, I mostly disagree. The first part is giving an example of high IQ not leading to a good existential risk plan, and the second part is saying that you expect that high ability to weigh evidence won't lead to a good plan either. The counterexample proves that high IQ isn't everything one needs, but overall, I'd still expect it to help. I think "no bearing" is too strong even for an IQ->IQ comparison of that sort. If you're going to assume you've been exposed to all the plans that people have come up with, picking the right plan is more of a claim evaluation job than a novel hypothesis generation job. For this, you're going to want someone that can evaluate claims like MWI easily. I think that this is sufficiently close to the case to make your comparison a poor one. If I were going to make a comparison to make your point (to the degree which I agree with it), I'd use more than one person with more than one strength of intellect and instead ask "do we really think EY has shown enough to succeed where most talented people fail?". I'd also try to make it clear whether I'm arguing against him having a 'majority' of the probability mass in his favor vs having a 'plurality' of it going for him. It's a lot easier to argue against the former, but it's the latter that is more important if you have to pick someone to give money to.

0multifoliaterose15y

But how well does the ability to evaluate evidence connected with quantum mechanics correlate with ability to evaluate evidence connected with existential risk? See also the thread here

[-]khafra15y80

I agree with the overall point, here; but the "argument by authority" section is deeply flawed. In it, intelligence is consistently equated with rationality; and section's whole point seems to depend on that equation. As demonstrated in works like What Intelligence Tests Miss, G and rationality have markedly different effects. I don't think Eliezer would claim to be smarter than Grothendieck or Gödel or Erdős, but he could claim with some justification to be saner than them.

4multifoliaterose15y

It appears that what distinguished Grothendieck was not high g-factor. See Jordan Ellenberg's blog post titled The capacity to be alone. My point is that Grothendieck exhibited very high instrumental rationality with respect to mathematics but low instrumental rationality with respect to his efforts to ensure the survival of the human race, and that something analogous could very well be the case of Eliezer. What evidence is there that Eliezer is saner than Grothendieck? I don't have a strong opinion on this point, I'm just curious what you have in mind.

5Risto_Saarelma15y

It should perhaps be mentioned that the few accounts of encountering Grothendieck during the last 20 years describe someone who seems actually clinically insane, with delusions and extreme paranoia, not just someone with less than stellar rationality.

1multifoliaterose15y

Yes, I concur. But what about Grothendieck in the 1970s vs. Eliezer now? Or Gromov now vs. Eliezer now? It's not clear to me which way such comparisons go.

4khafra15y

Grothendieck's magnum opus was his contributions to pure mathematics. That requires very high intelligence and a willingness to, in hackneyed terms, think outside the box; or, in LW terms, go to school wearing a clown suit. Eliezer's magnum opus, so far, is the sequences. They combine a lot of pre-existing work and some of his own insights into a coherent whole that displays, I think, extraordinarily rare sanity. Pratchett's "First Sight," applied to a wide variety of fields. Going through accumulated human knowledge and picking out a framework that satisfies Occam's Razor better than any other I've seen is why I think he's very sane.

2Perplexed15y

It seems a little odd that the grandparent comment was about arguments from authority, but here we are talking about Grothendieck's work in pure math and Eliezer's on methods of rationality. Because the thing is, in neither area can an appeal to authority work. Regardless of how much G, or how much scholarship and expertise they have acquired, they both have to "win" by actually convincing ordinary people with their arguments rather than overawing them with their authority. On the other hand, when advocating anarchist political positions or prioritizing existential risks, authority helps. Trouble is, neither math skill nor {whatever it is that EY does so well} qualifies as a credential for the needed kind of authority.

0jimmy15y

There's a place for "argument from authority". The idea is that you don't, in general, have fully articulated proofs of the question in hand, and you're relying on some combination of heuristics to come to your conclusion. If you're allowed to hear other peoples answers, and a bit about the people making them, then you have a set of heuristics and answers, and you have to guess what the real answer is based on these. If you stick with your original answer, you're arbitrarily picking one heuristic to trust completely, which is clearly suboptimal. You want to discount like minded thinking (many people, one heuristic), weigh more heavily peoples views that you know were reached by thinking about the problem in different ways (again, weight the heuristic, not the person), and of course, more heavily weight heuristics that you expect to work. It's how to do this last part that we're talking about. High G people may have access to more complex heuristics that most could not come up with, but what's more important is having your heuristic free of errors that prevent its functioning. Knowing what a heuristic has to do in order to work is more important than having a lot of cognitive horsepower spent on coming up with fancy heuristics without a solid reason. Of course, in the end, if you spot a glaring error in someone's thinking, you don't trust him, even if he's an 'authority' (in other words: even if he has a track record of producing good heuristics, you condition on this one being bad and don't trust the output). And of course, the deeper into the object level you are able to dive, the more information you have on which to judge the credibility of heuristics. Perhaps it has better connotations whens stated as "Aumann agreement"?

0multifoliaterose15y

Agree with this.

[-]Daniel_Burfoot15y80

This post seems mis-named. I thought you were going to discuss "other existential risks" like nuclear war, global pandemic, environmental collapse, but mostly the discussion was about how to evaluate SIAI claims.

3multifoliaterose15y

A large portion of my post is about the idea that there's reason to doubt (1') on account of the existence of other existential risks. I do see what you mean though. I'm open to suggestions for how I might rename the post.

[-]Johnicholas15y70

SIAI's narrow focus on things that "look like HAL" neglects the risks of entities that are formed of humans and computers (and other objects) interacting. These entities already exist, they're already beyond human intelligence, and they're already existential risks.

Indeed, Lesswrong and SIAI are two obvious examples of these entities, and it's not clear at all how to steer them to become Friendly. Increasing individual rationality will help, but we also need to do social engineering - checks and balances and incentives (not just financial, but social incentives such as attention and praise) - and groupware research (e.g. karma and moderation systems, expert aggregation).

4[anonymous]15y

I don't think that "entities that are formed of humans and computers (and other objects) interacting" is sufficiently specific to be considered a type of existential risk. Any organization can be put into that category and unlike AGI, it's not true that most possible organizations have goal systems indifferent to human morals. Also, the fact that organizations can be dangerous is well known and there doesn't seem to be a simple solution to that or anything else a small organization could do. The problem isn't about coming up with checks and balances or incentive systems, it's about making people sane enough to use those solutions.

0torekp15y

True, but Johnicholas still has a point about "things that look like HAL," namely, that such scenarios presents the uFAI risk in an unconvincing manner. To most people, I suspect a scenario in which individuals and organizations gradually come to depend too much on AI would be more plausible.

0cousin_it15y

What makes you think LW is smarter than a human?

6Johnicholas15y

On some measures (breadth of knowledge, responsiveness at all hours, words-typed-per-month), LW is superhuman. On most other measures, LW can default to using one of its (human) component's capabilities, and thereby achieve human- comparable performance. I admit it has problems with cohesiveness and coherency.

[-]Simulation_Brain15y60

I think this is an excellent question. I'm hoping it leads to more actual discussion of the possible timeline of GAI.

Here's my answer, important points first, and not quite as briefly as I'd hoped.

1) even if uFAI isn't the biggest existential risk, the very low investment and interest in it might make it the best marginal value for investment of time or money. As someone noted, having at least a few people thinking about the risk far in advance seems like a great strategy if the risk is unknown.

2) No one but SIAI is taking donations to mitigate the risk ... (read more)

5Wei Dai15y

See Organizations formed to prevent or mitigate existential risks. (FHI isn't listed there for some reason.) Besides FHI, I know at least Lifeboat Foundation is also taking donations. They endorse SIAI, but have their separate plans.

[-]Paul Crowley15y60

With respect to point (E), in Astronomical Waste Bostrom writes:

a single percentage point of reduction of existential risks would be worth (from a utilitarian expected utility point-of-view) a delay of over 10 million years.

From this, if a near-existential disaster could cause a delay of, say, 10,000 years in reaching the stars, then a 10% reduction in the risk of such a disaster is worth the same as a 0.0001% reduction in existential risk.

0multifoliaterose15y

Yes, I appreciate that point, my concern is with permanent obstructions to technological development.

8Paul Crowley15y

Yes, a permanent obstruction is an existential risk. There is some discussion of ways in which a nuclear war could permanently obstruct our reaching the stars, but I'm not sure the risk is that high. Full-on runaway global warming is absolutely an existential risk - it will do more than delay us if the planet turns into Venus. Again this isn't considered a very likely outcome.

[-]JoshuaZ15y110

It seems extremely unlikely that we'll have Venusian style runaway global warming anytime in the next few thousand years assuming no major geoengineering occurs. A major part of why that happened on Venus is due to the lack of plate tectonics on Venus. Without that, there are serious limits. Earth could become much more inhospitable to humans but it would be very difficult to even have more than a 20 or 30 degree Farenheit increase. So humans would have to live near the poles, but it wouldn't be fatal.

A more serious long-term obstruction to going to the stars is that it isn't completely clear after a large-scale societal collapse that we will have the resources necessary to bootstrap back up to even current tech levels. Nick Bostrom has discussed this. Essentially, many of the resources we take for granted as necessary for developing a civilization (oil, coal, certain specific ores) have been consumed by civilization. We're already exhausting the easy to reach oil and have exhausted much of the easy to reach coal (we just don't notice it as much with coal because there's so much). A collapse back to bronze age tech, or even late Roman tech might not have enough easy energy sources ... (read more)

5CarlShulman15y

The big question for these issues is how much 'slack' we had over our development trajectory. A new civilization could cultivate biomass for energy, and hydropower provides a fair amount of electricity without steady use of consumables. I'd say probably but not very confidently we could recover after intense resource depletion and collapse.

4JoshuaZ15y

Right, and in some respects we'd actually have tiny advantages the second time around, in that a lot of metals which are hard to separate from ores are already separated so humans who know where to look will have easy sources of metal. This will be particularly relevant for copper and aluminum which are difficult to extract without large technological bases.

3ewbrownv15y

Yes, and let's keep in mind that no civilization with colonial-era tech has ever collapsed to a pre-industrial level, and it isn't at all clear that such an event is possible. You'd have to kill more than 99% of the population and keep the survivors from forming town-sized communities for a couple of generations, and even then the knowledge is still available in books. To me this just looks like reasoning from fictional evidence - there are lots of stories about primitive survivors of lost civilizations, so people assume that must be a plausible outcome.

2JoshuaZ15y

That may be using a bad reference class. We know that slides backwards have happened for other tech levels. I don't see an intrinsic reason to think it couldn't happen for a society at or near our tech level.

4ewbrownv15y

When low-tech societies collapse, the reason is typically that they lose access to some resource that’s essential to their way of life, and they can’t adapt because their technology base doesn’t include anything they can switch to as a substitute. Since the number of potential substitutes for any given resource grows steadily as technology advances we would expect more advanced societies to be more resistant to that type of problem, and indeed that’s what we see in the historical record. If you can’t keep the nuclear power plants working you can always fall back on oil, or natural gas, or coal, or hydro, or windmills, and so on all the way down the chain to bronze age power sources. Then, once you find a level you can sustain in your new situation, you can start rebuilding transportation and industry to get back to where you were before the disaster. Which is why I say that the “big disaster causes civilization to collapse” scenario is fictional evidence. AKAIK it has never happened to any society that had even colonial-era tech, and there are good reasons to think it can’t unless you posit such a high casualty rate (>99%) that instant extinction becomes an equally plausible outcome.

5ewbrownv15y

In evaluating existential risks it's essential to focus our attention on actual predictions and realistic scenarios, instead of fanciful 'worst imaginable case' scenarios. Earth could be destroyed by a giant asteroid made of antimatter moving at 99% C tomorrow, but since there's no reason to think such things actually exist it would be a waste of time to worry about them. Better to focus our attention on the scenarios that are actually plausible, so we don’t waste our efforts. With that in mind, I’ll point out that even the worst-case IPCC scenarios do not come remotely close to posing an existential risk. The predicted climate changes are only somewhat larger than what we experienced in the 20th century, and the predicted effects are mostly an increase of suffering in countries that are too poor to adapt easily. As near as I can tell global warming is only included in this kind of list because so many people have it in their mental ‘scary global bad stuff’ bucket, and don’t notice that crop failures and malaria outbreaks are in a completely different league than the end of all life on Earth.

0Paul Crowley15y

I would want to hear that from a climatology pro who acknowledged other existential risks to be really reassured, but thanks, and I hope you're right!

3multifoliaterose15y

I basically agree with your remarks. What about resource shortage? Edit: See Scott Aaronson's remarks under The Singularity Is Far for one perspective.

0Paul Crowley15y

Resource shortage (as JoshuaZ raises) is the discussion I was thinking of. Thanks for the link to that essay - I hadn't read it, and it's worth reading as is so often the case with him.

6multifoliaterose15y

I'd also remark that the asteroid risk seems like it might be worth thinking about, not because it's at all at the top of the list of things that might go wrong, but because it might be cheap to dispense with. I don't have relevant subject matter knowledge but am friends with an applied physics graduate student who suggested that it might cost 100 million dollars or less. Maybe even around a mere 10 million dollars. Carl expresses skepticism that working against asteroid strikes is cost-effective here.

7Paul Crowley15y

Asteroid risk is a good "poster child" for existential risk in general, since it's easily understood and doesn't provoke skepticism the way other risks can. To some extent, this means I'm less worried about it, since I'm more optimistic that if I don't campaign about it someone else will.

6John_Maxwell15y

I read in Influence that people are much more likely to identify with a cause once they've made a small commitment to it. Perhaps the best thing we can do for existential risk is to track down people who seem like intelligent, rational sorts and ask them to make very small contributions to preventing asteroid risk?

3XiXiDu15y

I wish. Last time I read about it the U.S. gov wasn't inclined to spend the few million necessary for an all-sky survey to register all potentially dangerous objects.

2mkehrt15y

Is it only expected to be a few million? This could easily be privately funded with a good advertising campaign. For example, a project which might have a similar audience, SETI, is entirely privately funded and has a budget of a few million a year.

2multifoliaterose15y

Part of why I mention asteroid risk is because it's a good poster child for existential risk in general. See the document which I emailed you.

0Paul Crowley15y

Not received it yet - what address did you mail it to? Try paul at ciphergoth dot org.

0multifoliaterose15y

Sent

0Paul Crowley15y

Still no joy I'm afraid. Is it possible your sender IP is listed by zen.spamhaus.org? I've checked my junk folder for things titled "asteroid" and found nothing. If you can tell me the sender address I can tell you if it's showing up in my mail logs. Sorry!

[-]orthonormal15y50

On the issue of AI timelines:

A quantitative analysis of the sort you seek is really not possible for the specifics of future technological development. If we knew exactly what obstacles stood in the way, we'd be all but there. Hence the reliance instead on antipredictions and disjunctions, which leave a lot of uncertainty but can still point strongly in one direction.

My own reasoning behind an "AI in the next few decades" position is that, even if every other approach people have thought of and will think of bogs down, there's always the ability ... (read more)

1timtyler15y

Surely the poster wasn't doing that!

0multifoliaterose15y

I was not citing Tim Tyler as a source for SIAI's views, I was addressing his argument as one of many in favor of short term focus on AI. Is there something that you would suggest that I do to make this more clear in the top level post?

[-]Jonathan_Graehl15y50

Suppose pro-friendly AI and anti-uncontrolled-AI advocacy and research is not at this point the most effective mitigation of x-risk. It doesn't follow that nothing at all should be done now.[1]

I would still want something like SIAI funded to some level (just like I would want a few competent people evaluating the utility of planning and preparing for other far-off high-leverage risks/opportunities).

Broadly, the question is: who should be funded, and for how much, to plan/act for our possible far-future benefit. Specifically: holding everything else const... (read more)

4rwallace15y

That's not the problem. The problem is that it's not mitigation at all, it's exacerbation. The current state of affairs is not stable (for that matter, it's not even in equilibrium); either we go up or we go down. If we snuff out real research in favor of hopeless feel-good programs to formalize Friendliness with pen and paper, we throw away chances of the former and thereby choose the latter by default. Remember, it is the way of extinction that what kills the last individual often has nothing to do with the factors that doomed the species. For all anyone knows, the last dodo may have died of old age. I'm confident there will still be at least some people alive in 2100. But whether there still exists a winning move for humanity at that stage may depend on what we choose to support now, in the early decades of the century.

1Jonathan_Graehl15y

I grant that if it (thinking about FAI) were certain to be harmful, then absolutely none of it should be done. I didn't even consider that possibility. I don't think it's certainly harmful, and I believe it has some expected (or at least diversification) benefit.

0John_Maxwell15y

What if we take a survivalist-type approach, putting a bunch of people and natural resources deep underground somewhere?

0rwallace15y

That would protect against certain straightforward kinds of disaster e.g. asteroid impact, but not against more subtle and more likely threats. Remember, people in a deep underground shelter will still die of old age just as they would have on the surface.

4John_Maxwell15y

OK, so what percentage of humanity's resources (human and natural) do you think should be kept in reserve underground? My guess is we both agree it should be way more than it is right now. Research "sex" :-P

2rwallace15y

I think we're better off spending the resources more proactively, unless and until we find evidence of an imminent threat of a variety against which that is a good defense. For example, instead of spending money preemptively populating underground shelters in case of an asteroid impact, I'd rather spend it extending our surveys of the sky to have a better chance of spotting an incoming asteroid in time to deflect it.

2John_Maxwell15y

This depends on being able to anticipate all significant threats. But I guess it's silly to debate what percentage of all humanity's resources should be devoted to various things--just what percentage of our resources.

0rwallace15y

Agreed on both counts; of course we have no way to know exactly what threats we face, let alone exactly how to deal with them. Except in this regard: however long or short our window of opportunity may be, any slowdown in technological progress increases the general threat that some specific threat will close that window on us while we are still vulnerable, while we still depend on nonrenewable resources, while all our eggs are still in one basket. The one form of protection we need above all else is speed, and that's how I believe we should be spending as much as possible of our resources.

[-]ata15y40

I definitely think that, alongside the introductory What is the Singularity? and Why work toward the Singularity? pages, SIAI should have a prominent page stating the basic case for donating to SIAI. Why work toward the Singularity? already explains why bringing about a positive Singularity would have a very high humanitarian impact, but it would probably be beneficial to make the additional case that SIAI's research program is likely increase the probability of that outcome, and that donations at its current funding level have a high marginal expected uti... (read more)

1Aleksei_Riikonen15y

I thought this was such a page: http://singinst.org/riskintro/index.html

0utilitymonster15y

I think the page makes a case that it is worth doing something about AI risk, and that SIAI is doing something. The page gives no one any reason to think that SIAI is doing better than anything else you could do about x-risk (there could be reasons elsewhere). In this respect, the page is similar to other non-profit pages: (i) argue that there is a problem, (ii) argue that you're doing something to solve the problem, but don't (iii) try to show that you're solving the problem better than others. Maybe that's reasonable, since that rubs some donors the wrong way and is hard to establish that you're the best; but it doesn't advance our discussion about the best way to reduce x-risk.

0ata15y

Ah, yes, I had forgotten about that. Thanks.

[-]rabidchicken15y40

Although there are an infinite number of existential risks which might cause human extinction, I still think that AI with a utility that conflicts with human existence is the one issue we should spend the most resources to fight. Why? First, an AI would be really useful, so you can be relatively sure that work on it will continue until the job is done. Other disasters like asteroid strikes, nuclear war, and massive pandemics are all possible, but at least they do not have a large economic and social incentive to get us closer to one.

Second, we have already... (read more)

[-][anonymous]15y40

Maybe I'm alone on this, but just to speak for the silent majority here:

Existential risk isn't that big a deal. The chances for any of the human civilizational failure modes are slim to none. It's really not something we as a society should be spending any time on.

That's not to say SIAI is a poor cause to contribute to. I've talked to some insiders who have assured me that SIAI has serious plans, over the span of decades, to really ramp up our productive capabilities and put them to good use, not wasteful or destructive use. To butter, not guns. To de... (read more)

4Larks15y

I can confirm that SIAI is very interested in the structural potential of paperclips. We will be sending photos soon to confirm our cooperation with other optimisation processes.

0Tyrrell_McAllister15y

Aren't paperclips bureaucracy goods to a significant extent? In the short term, you rely on humans to create most paperclips, and a decrease in bureaucracy would mean a decrease in the rate at which humans manufacture paperclips.

-1Clippy15y

I can manufacture paperclips without humans.

[-]Rain15y30

I'm surprised you bring up Mikhail Gromov as a counterexample to Eliezer, considering that Gromov's solution to existential risk, as presented in the quote above, can be paraphrased as: increase education so someone has a good idea on how to fix everything.

(Actual quote: "People must have ideas and they must prepare now. In two generations people must be educated. Teachers must be educated now, and then the teachers will educate a new generation. Then there will be sufficiently many people to face the difficulties. I am sure this will give a result.&q... (read more)

1multifoliaterose15y

I don't think that Gromov's views and Eliezer's views are necessarily incompatible. My reading of Gromov's quotation is that he does not have his eyes on a technological intelligence explosion and that the existential risk that he's presently most concerned about is natural resource shortage. This is in contrast with Eliezer who does have his eyes on a technological singularity and does not presently seem to be concerned about natural resource shortage. I would be very interested in seeing Gromov study the evidence for a near-term intelligence explosion and seeing how this affects his views. I may eventually approach him personally about this matter (although I hesitate to do so as I think that it's important that whoever approach him on this point make a good first impression and I'm not sure that I'm in a good position to do so at the moment).

[-]Jonathan_Graehl15y30

My own impression is that all existential risks are getting very little attention.

This is true, and indeed you refute (D) well with it. Although some particular risks, like cold-war era massive nuclear conflict (with or without the sexed-up nuclear winter scenarios), global warming, and medicine-resistant pandemic, have received magnitudes more serious consideration and media amplification than more things like nano and AI risks.

[-]Paul Crowley15y20

WRT point D, it should be possible to come up with some sort of formula that gives the relative utility according to maxipok of working on various risks. Something that takes into account

The current probability of a particular risk causing existential disaster
The total resources in dollars currently expended on that risk
The relative reduction in risk that a 1% increase in resources on that risk would bring

These I think are all that are needed when considering donations. When considering time rather than money, you also need to take into account:

... (read more)

-5taw15y

[-]mkehrt15y10

But I see no reason for assigning high probability to notion that a runaway superhuman intelligence will be developed within such a short timescale. In the bloggingheads diavlog Scott Aaronson challenges Eliezer on this point and Eliezer offers some throwaway remarks which I do not find compelling. As far as I know, neither Eliezer nor anybody else at SIAI have provided a detailed explanation for why we should expect runaway superhuman intelligence on such a short timescale.

I think this is a key point. While I think unFriendly AI could be a problem in ... (read more)

[-]PhilGoetz15y10

As I've commented elsewhere, any event which would permanently prevent humans from creating a transhuman paradise is properly conceived of as an existential risk on account of the astronomical waste which would result.

Is there no post somewhere on LW explaining why paradises are bad? A paradise must be all exploitation and no exploration; hence, they are static.

2multifoliaterose15y

I'm well aware of what you're talking about, when I referred to paradise I meant the word in a very broad sense.

0XiXiDu15y

Jehovah's Witnesses interpret a paradise as a CEV. For example that humans will never able to grasp the full complexity of God is a feature that will allow for infinite exploration and satisfaction of our curiosity. We'll never run out of fun and challenges. So I'm not sure what definition of paradise you had in mind. But even colloquial a paradise implies that which brings satisfaction. Even most religious people are not as naive to suggest that there won't be losers and winners. Or that a paradise would be static.

0mkehrt15y

Not voted, because I think this is utterly fascinating and entirely off topic!

-4PhilGoetz15y

I don't need a specific paradise in mind. Paradise means bad things don't happen, which means the entire society is highly optimized. Being highly-optimized requires being static. This is a general property of search/optimization algorithms.

[-]thomblake15y120

Being highly-optimized requires being static.

I'm not sure why I should believe this. Given that one of the properties that we're presumably optimizing over is 'not being static'.

[-]Jonathan_Graehl15y10

some LW posters are confident in both (1) and (2), some are confident in neither of (1) and (2) while others are confident in exactly one of (1) and (2)

Logically, this is tautological. I think you're saying that there don't seem to be many who are completely convinced that both (1) and (2) are untrue. I think that's right; both claims are somewhat plausible.

Curious: do people prefer "neither A nor B" or "neither of (A and B)"?

[-]Larks15y130

Nitpick: it's not quite tautological, as he asserts that at least one* person exists in each category. It is only a tautology that everyone fits into one of them, not that they're all non-empty.

*or two, depending on your interpretation of 'some'.

[-]bentarm15y140

I don't think this is a Nitpick - I think this explains why the statement is included in the original post in the first place - to point out that there is a wide variety of position that LW readers hold on these statements.

2Jonathan_Graehl15y

Nice subtlety (at least one).

2Dagon15y

The problem is that "confident in" has an ambiguous negation. "not confident in A" is different than "confident in not-A".

0Jonathan_Graehl15y

Right, but the quoted text is consistent, so, if you grant me that "some" means >=0, my original statement would have been correct. Of course, "some" implies >0, which I missed.

[-]pnrjulius13y00

We clearly don't focus enough on near-term existential risks that we already know about:

Nuclear war
Global warming
Asteroid impact
Supervolcano eruption

Compared to these (which already exist right now and are relatively well-understood), worrying about grey goo and unfriendly AIs does seem a bit beside the point.

[+]timtyler15y-190

LESSWRONG
LW

LESSWRONG
LW

40

Other Existential Risks

40

40